I recently upgraded a pfSense 2.2.x firewall to a 2.3.x version. I've done upgrades numerous times and they always go well. This time, it didn't.
I use the automatic upgrader, and duly click the backup tick box before setting the upgrade off. After a while of no Internet access I checked the console to find a dead firewall:
Loading /boot/defaults/loader.conf
can't find 'kernel'
Oh dear. A reboot didn't resolve it. Not to worry, I clicked backup. Just if I could work out what it did with said backup. Simple enough, the backup is a .tgz archive stored in the /root directory. Simple - apart from my device won't boot.
My action plan was to recover the backup, reinstall from scratch and overwrite the config with the one from the backup.
pfSense is based on FreeBSD, which is Unix in all but its name. Unix is fairly similar to Linux, but sufficiently different to cause headaches and confusion when trying to recover something. Most difficult of all is the Unix File System (ufs) file format. It's not straight foward to mount in Linux, so my attempts to recover the file after mounting the disk in Linux came to a swift end.
In the end, I used a FAT16 formatted drive and attached it to the firewall to act as a temporary store.
You can use the pfSense install media to boot and have the option of going into Recovery mode. Unfortunately it just gives you a console and no other information. A bit of fumbling around I issued the following commands to mount the existing disk as well as my temporary one:
mount /dev/da0s1a /mnt
mount /dev/da1s1 /media
Then copied the backup file to my temporary disk and shutdown the firewall:
cp /mnt/root/*.tgz /media
halt
I attached my temporary disk to another computer and extracted the /cf/conf/config.xml file and uploaded this to a clean pfSense install through the Diagnostics > Backup & Restore menu.
The moral of the story is backup your config, or snapshot your virtual machine before doing an upgrade.