How to back up a ZFS boot disk ?

Question

Hello all,
I have just installed Solaris 10 update 6 (10/08) on a Sparc machine (an Ultra 45 workstation) using ZFS for the boot disk.
Now I want to port a custom UFS boot disk backup script to ZFS.
Basically, this script copies the boot disk to a secondary disk and makes the secondary disk bootable.
With UFS, I had to play with the vfstab a bit to allow booting off the secondary disk, but this is not necessary with ZFS.
How can I perform such a backup of my ZFS boot disk ?
I tried the following (source disk: c1t0d0, target disk: c1t1d0):
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 110G 118G 94K /rpool
rpool/ROOT 4.58G 118G 18K legacy
rpool/ROOT/root 4.58G 25.4G 4.50G /
rpool/ROOT/root/var 79.2M 4.92G 79.2M /var
rpool/dump 16.0G 118G 16.0G -
rpool/export 73.3G 63.7G 73.3G /export
rpool/homelocal 21.9M 20.0G 21.9M /homelocal
rpool/swap 16G 134G 16K -
# zfs snapshot -r rpool@today
# zpool create -f -R /mnt rbackup c1t1d0
# zfs send -R rpool@today | zfs receive -F -d rbackup               <- This one fails (see below)
# installboot /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c1t1d0s0
The send/receive command fails after transfering the "/" filesystem (4.5 GB) with the following error message:
cannot mount '/mnt': directory is not empty
There may be some kind of unwanted recursion here (trying to back up the backup or something) but I cannot figure it out.
I tried a workaround: creating the mount point outside the snapshot:
zfs snapshot -r rpool@today
mkdir /var/tmp/mnt
zpool create -f -R /var/tmp/mnt rbackup c1t1d0
zfs send -R rpool@today | zfs receive -F -d rbackup
But it still fails, this time with mounting "/var/tmp/mnt".
So how does one back up the ZFS boot disk to a secondary disk in a live environment ?

Answer 1

OK, this post requires some clarification.
First, thanks to robert.cohen and rogerfujii for giving some elements.
The objective is to make a backup of the boot disk on another disk of the same machine. The backup must be bootable just like the original disk.
The reason for doing this instead of (or, even better, in addition to) mirroring the boot disk is to be able to quickly recover a stable operating system in case anything gets corrupted on the boot disk. Corruption includes hardware failures, but also any software corruption which could be caused by a virus, an attacker or an operator mistake (rm -rf ...).
After doing lots of experiments, I found two potential solutions to this need.
Solution 1 looks like what rogerfujii suggested, albeit with a few practical additions.
It consists in using ZFS mirroring and breaking up the mirror after resilvering:
     - Configure the backup disk as a mirror of the boot disk :
     zpool attach -f rpool <boot disk>s0 <backup disk>s0
     - Copy the boot block to the backup disk:
     installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/<backup disk>s0
     - Monitor the mirror resilvering:
     zpool status rpool
     - Wait until the "action" field disappears (this can be scripted).
     - Prevent any further resilvering:
     zpool offline rpool <backup disk>s0
     Note: this step is mandatory because detaching the disk without offlining it first results in a non bootable backup disk.
     - Detach the backup disk from the mirror:
     zpool detach rpool <backup disk>s0
     POST-OPERATIONS:
     After booting on the backup disk, assuming the main boot disk is unreachable:
     - Log in as super-user.
     - Detach the main boot disk from the mirror
     zpool detach rpool <boot disk>s0
This solution has many advantages, including simplicity and using no dirty tricks. However, it has two major drawbacks:
- When booting on the backup disk, if the main boot disk is online, it will be resilvered with the old data.
- There is no easy way to access the backup disk data without rebooting.
So if you accidentally lose one file on the boot disk, you cannot easily recover it from the backup.
This is because the pool name is the same on both disks, therefore effectively preventing any pool import.
Here is now solution 2, which I favor.
It is more complex and dependent on the disk layout and ZFS implementation changes, but overall offers more flexibility.
It may need some additions if there are other disks than the boot disk with ZFS pools (I have not tested that case yet).
***** HOW TO BACKUP A ZFS BOOT DISK TO ANOTHER DISK *****
1. Backup disk partitioning
- Clean up ZFS information from the backup disk:
The first and last megabyte of the backup disk, which hold ZFS information (plus other stuff) are erased:
dd if=/dev/zero seek=<backup disk #blocks minus 2048> count=2048 of=/dev/rdsk/<backup disk>s2
dd if=/dev/zero count=2048 of=/dev/rdsk/<backup disk>s2
- Label and partition the backup disk in SMI :
format -e <backup disk>
     label
     0          -> SMI label
     y
     (If more questions asked: press Enter 3 times.)
     partition
     (Create a single parition, number 0, filling the whole disk)
     label
     0
     y
     quit
     quit
2. Data copy
- Create the target ZFS pool:
zpool create -f -o failmode=continue -R /mnt -m legacy rbackup <backup disk>s0
Note: the chosen pool name is here "rbackup".
- Create a snapshot of the source pool :
zfs snapshot -r rpool@today
- Copy the data :
zfs send -R rpool@today | zfs receive -F -d rbackup
- Remove the snapshot, plus its copy on the backup disk :
zfs destroy -r rbackup@today
zfs destroy -r rpool@today
3. Backup pool reconfiguration
- Edit the following files:
/mnt/etc/vfstab
/mnt/etc/power.conf
/mnt/etc/dumpadm.conf
In these files, replace the source pool name "rpool" with the backup pool name "rbackup".
- Remove the ZFS mount list:
rm /mnt/etc/zfs/zpool.cache
4. Making the backup disk bootable
- Note the name of the current boot filesystem:
df -k /
E.g.:
# df -k /
Filesystem kbytes used avail capacity Mounted on
rpool/ROOT/root 31457280 4726390 26646966 16% /
- Configure the boot filesystem on the backup pool:
zpool set bootfs=rbackup/ROOT/root rbackup
Note: "rbackup/ROOT/root" is derived from the main boot filesystem name "rpool/ROOT/root".
- Copy the ZFS boot block to the backup disk:
installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/<backup disk>s0
5. Cleaning up
- Detach the target pool:
zpool export rbackup
I hope this howto will be useful to those like me who need to change all their habits while migrating to ZFS.
Regards.
HL