A mirrored ZFS storage pool can be quickly cloned as a backup pool by using the zpool split command. You can use this feature to split a mirrored root pool, but the pool that is split off is not bootable until you perform some additional steps. Some of the required actions are different for SPARC vs x86/x64 due to the difference of the OBP vs GRUB between the architectures. This article focuses on the steps required to split rpool in x86/x64 architecture.
Caveats and Assumptions
The new/split pool must be used on the same host or the same server family type and architecture. ie: If the source rpool is imported and split on a T5120, the new pool must also be used on the same server type (T5xx0).
Splitting the rpool
1. Verify the current status of the rpool. At minimum it should be a mirror. Depending on the number of available devices or desired end goal, an additional (3rd) mirror may been added and used to create the new rpool.
# zpool status rpool
pool: rpool
state: ONLINE
scan: resilvered 26.1G in 0h16m with 0 errors on Mon Mar 24 20:51:13 2014
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c7t0d0 ONLINE 0 0 0
c7t1d0 ONLINE 0 0 0
This post uses two-way-mirror example, though the procedure is no different for a 3-way mirror with the exception of the disk used to create the new pool.
2. According to the man page of zpool the last device in the pool will be used to create the new pool unless a device is specified. It is best practice to specify the device regardless to ensure the correct device is chosen.
# zpool split rpool newrpool c7t1d0
3. By default, any new pool created using ‘zpool split’ is automatically exported once the split completes. The new pool won’t appear in ‘zpool list’.
# zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT rpool 67.5G 26.1G 41.4G 38% 1.00x ONLINE -
4. Use ‘zpool import’ to confirm the new pool was created with the correct device.
# zpool import
pool: newrpool
id: 6439596171322377858
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
newrpool ONLINE
c7t1d0 ONLINE
5. Temporarily import the pool, ‘newrpool’, using a different mount point so modifications to critical files can be made before the pool is used. The following imports the pool and sets a temporary mountpoint, but does not mount any of the datasets.
# mkdir /newrpool # zpool import -N -R /newrpool newrpool # zfs list -r newrpool NAME USED AVAIL REFER MOUNTPOINT newrpool 26.8G 39.6G 4.58M /newrpool/newrpool newrpool/ROOT 2.09G 39.6G 31K legacy newrpool/ROOT/solaris 2.09G 39.6G 1.95G /newrpool newrpool/ROOT/solaris/var 138M 39.6G 136M /newrpool/var newrpool/VARSHARE 59K 39.6G 59K /newrpool/var/share newrpool/dump 20.6G 40.2G 20.0G - newrpool/export 98K 39.6G 32K /newrpool/export newrpool/export/home 66K 39.6G 32K /newrpool/export/home newrpool/export/home/jack 34K 39.6G 34K /newrpool/export/home/jack newrpool/swap 4.13G 39.7G 4.00G -
6. Mount the top-level filesystem where the ‘menu.conf’ and ‘grub.cfg’ files are located.
# zfs mount -vO -o mountpoint=/newrpool newrpool
7. Edit the menu.conf file to change the bootfs property value to point at the new pool name, ‘newrpool’.
# cp /newrpool/boot/grub/menu.conf /newrpool/boot/grub/menu.conf_bkp # vi /newrpool/boot/grub/menu.conf # This file is modified without notice. # DO NOT EDIT THIS FILE DIRECTLY. CHANGES MAY NOT BE PRESERVED. # Changes to this file can be made indirectly via bootadm(1M) subcommands. # [global] console = text timeout = 30 [meta] order = SolarisDiskBootInstance|0 [SolarisDiskBootInstance|0] kernel = /platform/i86pc/kernel/amd64/unix title = Oracle Solaris 11.1 default = True bootfs = newrpool/ROOT/solaris fstype = zfs boot_archive = /platform/i86pc/amd64/boot_archive
8. Solaris 11.1 introduces GRUB 1.99 (GRUB2) which works very differently than GRUB v0.97 used on Solaris 10 and 11.0. GRUB2 searches disks for boot environments with a specific UUID. The ‘zpool split’ command doesn’t update this UUID so it must be updated manually. Even if the boot disk order is changed, GRUB may still use the old rpool as both rpool and newrpool have the same UUID. There are two entries in grub.cfg, one for the 64-bit OS, the other for 32-bit.
# cp -p /newrpool/boot/grub/grub.cfg /newrpool/boot/grub/grub.cfg_bkp
search --no-floppy --fs-uuid --set=root a7df389a817b7ae1
--- and ---
if [ "$target" = "i386_pc" ]; then
search --no-floppy --fs-uuid --set=root_newrpool a7df389a817b7ae1
After updating the UUID, use grub-mkconfig command to generate one grub entry for each pool in grub.cfg.
# /usr/lib/grub2/bios/sbin/grub-mkconfig -o /newrpool/boot/grub/grub.cfg
9. Each entry has the same name (no way to change it using the grub-mkconfig command). Redundant entries for the rpool must be manually removed from /newrpool/boot/grub/grub.cfg to avoid boot confusion. Make the grub.cfg file writable.
# chmod u+w /newrpool/boot/grub/grub.cfg
10. Remove the relevent entries for the rpool highlighted in bold.
# cat /newrpool/boot/grub/grub.cfg
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /usr/lib/grub2/bios/etc/grub.d and settings from /usr/lib/grub2/bios/etc/default/grub
#
### BEGIN /usr/lib/grub2/bios/etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
load_env
fi
set default="0"
if [ "${prev_saved_entry}" ]; then
set saved_entry="${prev_saved_entry}"
save_env saved_entry
set prev_saved_entry=
save_env prev_saved_entry
set boot_once=true
fi
function savedefault {
if [ -z "${boot_once}" ]; then
saved_entry="${chosen}"
save_env saved_entry
fi
}
function load_video {
true
}
if loadfont "/@/boot/grub/unicode.pf2" ; then
set gfxmode="auto"
load_video
insmod gfxterm
fi
terminal_output gfxterm
set timeout=5
### END /usr/lib/grub2/bios/etc/grub.d/00_header ###
### BEGIN /usr/lib/grub2/bios/etc/grub.d/10_solaris ###
menuentry "Oracle Solaris 11.1" {
insmod part_msdos
insmod part_sunpc
insmod part_gpt
insmod zfs
search --no-floppy --fs-uuid --set=root 595e0a9970a2c682
zfs-bootfs /ROOT/solaris/@/ zfs_bootfs
load_video
set kern=/platform/i86pc/kernel/amd64/unix
echo -n "Loading ${root}/ROOT/solaris/@$kern: "
$multiboot /ROOT/solaris/@/$kern $kern -B $zfs_bootfs
insmod gzio
echo -n "Loading ${root}/ROOT/solaris/@/platform/i86pc/amd64/boot_archive: "
$module /ROOT/solaris/@/platform/i86pc/amd64/boot_archive
}
if [ "$target" = "i386_pc" ]; then
search --no-floppy --fs-uuid --set=root_newrpool 595e0a9970a2c682
if [ -s "($root_newrpool)/@/boot/grub/menu.lst" ]; then
submenu "Legacy GRUB Menu (from root pool newrpool)" "($root_newrpool)/@/boot/grub/menu.lst" {
extract_legacy_entries_source "$2"
}
fi
fi
menuentry "Oracle Solaris 11.1" {
insmod part_msdos
insmod part_sunpc
insmod part_gpt
insmod zfs
search --no-floppy --fs-uuid --set=root a7df389a817b7ae1
zfs-bootfs /ROOT/solaris/@/ zfs_bootfs
load_video
set kern=/platform/i86pc/kernel/amd64/unix
echo -n "Loading ${root}/ROOT/solaris/@$kern: "
$multiboot /ROOT/solaris/@/$kern $kern -B $zfs_bootfs
insmod gzio
echo -n "Loading ${root}/ROOT/solaris/@/platform/i86pc/amd64/boot_archive: "
$module /ROOT/solaris/@/platform/i86pc/amd64/boot_archive
}
if [ "$target" = "i386_pc" ]; then
search --no-floppy --fs-uuid --set=root_rpool a7df389a817b7ae1
if [ -s "($root_rpool)/@/boot/grub/menu.lst" ]; then
submenu "Legacy GRUB Menu (from root pool rpool)" "($root_rpool)/@/boot/grub/menu.lst" {
extract_legacy_entries_source "$2"
}
fi
fi
### END /usr/lib/grub2/bios/etc/grub.d/10_solaris ###
### BEGIN /usr/lib/grub2/bios/etc/grub.d/41_custom ###
if [ -f $prefix/custom.cfg ]; then
source $prefix/custom.cfg;
fi
### END /usr/lib/grub2/bios/etc/grub.d/41_custom ###
11. Make the file read-only again.
# chmod u-w /newrpool/boot/grub/grub.cfg
12. Push the new grub.cfg out to disk.
# bootadm install-bootloader -P newrpool
13. Mount the root filesystem (/).
# zfs mount -vO -o mountpoint=/newrpool newrpool/ROOT/solaris # cd /newrpool # ls bin devices home kernel mnt opt root share tmp ws boot etc homedir lib net platform rpool shared usr xfn dev export import media nfs4 proc sbin system var
14. Remove the zpool.cache file. This file maintains an on-disk view of the imported ZFS pools and is read at boot time to populate the in-core ZFS Storage Pool configs without having to delay the boot by scanning and reading all the disks presented to the host. The purpose of removing this file is to ensure that we do scan all the disks and re-create this file with the configuration of the new pool, ‘newrpool’ when the system is booted from the ‘newrpool’.
# rm /newrpool/etc/zfs/zpool.cache
15. Edit the vfstab and change the poolname for the swap device and any other entries referencing the rpool.
# grep rpool /newrpool/etc/vfstab /dev/zvol/dsk/newrpool/swap - - swap - no -
16. Create the /reconfigure file to perform a reconfiguration reboot when the system is next booted from ‘newrpool’.
# touch /newrpool/reconfigure
17. Update the boot archive for the newrpool.
# bootadm update-archive -v -R /newrpool
cannot find: /newrpool/etc/cluster/nodeid: No such file or directory
cannot find: /newrpool/etc/devices/mdi_ib_cache: No such file or directory
changed /newrpool/etc/devices/mdi_scsi_vhci_cache
cannot find: /newrpool/etc/devices/retire_store: No such file or directory
cannot find: /newrpool/etc/rtc_config: No such file or directory
updating /newrpool/platform/i86pc/boot_archive
Unable to extend /newrpool/platform/i86pc/boot_archive... rebuilding archive
Successfully created /newrpool/platform/i86pc/boot_archive
updating /newrpool/platform/i86pc/amd64/boot_archive
Unable to extend /newrpool/platform/i86pc/amd64/boot_archive... rebuilding archive
Successfully created /newrpool/platform/i86pc/amd64/boot_archive
18. Unmount the filesystems for the new pool and export the pool.
# cd / # zfs unmount /newrpool # umount /newrpool # zpool export newrpool
19. Reboot the system. At the BIOS screen, use (F8) to select the boot disk. Use the correct procedure for your system to select and change the boot device. You may need to enter the disk controller firmware menu (Adaptec, LSI, etc) to change boot device priority order. The GRUB menu for the default boot device has not been modified so it won’t show ‘newrpool’.
# reboot -p
20. When the system gets to the GRUB menu, press ‘e’ to view/edit the GRUB commands.
Verify the UUID seen on the ‘search –no-floppy –fs-uuid –set=root 595e0a9970a2c682’ line menu matches the expected UUID for ‘newrpool’. If it matches the UUID for the original rpool, either the system was booted from the wrong disk, or Step 12 failed. A workaround for this is to manually change the UUID, press F10 or Ctrl-X to boot the system, then re-run the bootadm command in Step 12 after Step 20. By manually changing just the UUID the system will boot from ‘newrpool’.
21. Watch the boot carefully for any errors or warnings. If the system boots and presents the login prompt with no errors, CONTINUE. If the system drops to maintenance mode and/or presents any SMF errors, STOP and resolve.