ZFS: adding disks and changing from non-redundant pools to mirrored

Lately I switched all (non-virtual) FreeBSD machines to use ZFS for their root file system, and it was a good choice!

Example 1:

On this example machine, I initially only had a single, relatively small HDD in there, running the root-fs on ZFS. The output of zpool status:

#zpool status

pool: firstpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
firstpool ONLINE 0 0 0
gpt/bootdisk-zfs ONLINE 0 0 0

errors: No known data errors

After a while, I had the chance to upgrade the system, so I added a second 500GB HDD.
As I wanted the data on that disk separated from the data on the first disk, I created a second zpool:

#zpool create -m /mnt/data secondpool /dev/ada1

The -m defines the mountpoint and mounts the disks. It was ready for use immediately. Too easy, just great.

After a while again, I got an additional 500GB HDD (yes it still fitted into my machine), and I decided to set up a ZFS mirror for my data dir, for safety against HDD failures.
Again just one simple command, and the mirror was working:

#zpool attach secondpool /dev/ada1 /dev/ada2

It took a while for resilvering the new disk, as I put already quite some data on it, but it was the easiest setting up of a mirrored RAID I ever did.

#zpool status

pool: secondpool
state: ONLINE
scan: resilvered 105G in 0h29m with 0 errors on Fri Aug 5 11:13:40 2011
config:

NAME STATE READ WRITE CKSUM
secondpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0

Example2:

On another machine I migrated from UFS to ZFS for the root-fs, but the HDD with data on it still remained UFS formatted, as I had no space to move the data around (the data HDD is 1TB and 80% full). As soon as I got the opportunity of borrowing a large HDD (2TB), I moved from UFS to ZFS.

The easiest way would have been to create a new zpool with the borrowed disk, copy the contents from the UFS disk over to the ZFS disk, then attach the UFS disk to the new ZFS pool, let it resilver, and finally remove the borrowed disk from the ZFS pool, leaving only the original disk, now ZFS formatted in the machine.

Well, unfortunately this only works with same size disks (of course). So I had to do it the long way round:

1. Create a new zpool with the borrowed disk
2. Move the content from the UFS disk to the ZFS disk
3. Create a new zpool with the old (and now empty) disk
4. Move the contents of the borrowed disk to the old disk (now both ZFS)
5. Remove the borrowed device from its pool using zpool remove
6. Destroy the pool using zpool destroy

Obviously you need to  do some planning first for this, in order to get the desired zpool name and mountpoint right for the second zpool which is the one you want, rather than the first zpool. Of course I didn't, but luckily it's so easy to just create and destroy zpools and add or remove devices in ZFS.

Next thing to do: setting up ZFS rolling snapshots. Recommended tool:

• sysutils/zfsnap

The following /etc/periodic.conf should do the trick (not tested yet)

 

hourly_zfsnap_enable="YES" 

hourly_zfsnap_recursive_fs="sting2-data sting2/usr sting2/var"

hourly_zfsnap_ttl="1m"

hourly_zfsnap_verbose="NO" 

# Don't snap on pools resilvering (-s) or scrubbing (-S)

hourly_zfsnap_flags="-s -S"                       

reboot_zfsnap_enable="YES"

reboot_zfsnap_recursive_fs="sting2-data sting2/usr sting2/var"

reboot_zfsnap_ttl="1m"

reboot_zfsnap_verbose="NO"

# Don't snap on pools resilvering (-s) or scrubbing (-S)

reboot_zfsnap_flags="-s -S"

I'm a bit confused about the zfsnap_delete directives, as I would suspect that the TTL bit would take care of deleting snapshots after the expiration time.

We'll see how it goes.

FreeBSD ARM on Qemu in a VirtualBox System

After lot's of macking around, and not being happy with the ports cross compile solution, I've been finally able to get FreeBSD-ARM running in Qemu running in a VirtualBox.

Here the steps I took to success:

I've installed FreeBSD 8.2 as VirtualBox guest, as described in my previous post. I've then installed the following ports:

• devel/subversion-freebsd
• emulators/qemu-devel

I've then checked out the latest FreeBSD-CURRENT source from svn.freebsd.org and put it into /usr/devel/:

cd /usr/devel
svn co http://svn.freebsd.org/base/head/ .

The next step was to patch the sources in order to make them work with Qemu and the supported Gumstix-connex architecture.

The patch can be downloaded here.

It has been created following the indications found here, here, here and here.

After patching the source:

patch -p0 < gumstix-qemu.patch

the GUMSTIX FreeBSD Kernel and the ARM world can be built and installed into /usr/armqemu/armworld:

make TARGET=arm TARGET_ARCH=arm KERNCONF=GUMSTIX DESTDIR=/usr/armqemu/armworld buildworld kernel installworld distrib-dirs distribution

I've then followed the suggestions on in this message again in order to build a flash image for Qemu to pass to the -pflash command. The necessary GUMSTIX-connex uboot image can be found here.
After obtaining it run:

dd of=flash bs=1k count=16k if=/dev/zero
dd of=flash bs=1k conv=notrunc if=u-boot-connex-400-r1604.bin
dd of=flash bs=1k conv=notrunc seek=256 if=/usr/armqemu/armworld/boot/kernel/kernel.gz.tramp

Now you are ready to boot a kernel, but ther will be no root system available. The solution is to have a diskless machine, using root via NFS.

This gets a bit tricky. In VirtualBox you do already have a DHCP server, but it won't assign any addresses to interfaces not created by VirtualBox itself. So you need to install an additional DHCP server, and run it. It is safer to use a NATed interface in VirtualBox for this case, unless you want the DHCP server to offer addresses outside of VirtualBox.
I used a single VBox interface, which shows up as em0 in the FreeBSD guest, and gets address 10.0.2.15 assigned by default (The VBox default)

Next step is to install package net/isc-dhcp42-server, and to configure it editing /usr/local/etc/dhcpd.conf. I used the following subnet settings (based on this post:

subnet 10.0.2.0 netmask 255.255.255.0 {
range 10.0.2.20 10.0.2.30;
option routers 10.0.2.2;
next-server 10.0.2.15;
option root-path "10.0.2.15:/usr/armqemu/armworld";
}

The next step is to set the NFS export up in /etc/exports:

/usr/armqemu/armworld -maproot=root -network 10.0.2/24

Now we need to make sure that the once the kernel hits the init of the NFS file systems, it is still able to read from the NFS file system, so we need to set the armworld's rc.conf straight.
Edit /usr/armqemu/armworld/etc/rc.conf and add:

hostname="qemu-arm"
ifconfig_smc0="DHCP"
sshd_enable="YES"
rpcbind_enable="YES"
nfs_client_enable="YES"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"

Then we need to make sure that the Qemu guest is capable of reaching the host system via the network interface. This is described in this wiki entry. In order to make it work you need to do the following:

kldload aio
kldload if_tap
kldload if_bridge
ifconfig tap0 create
ifconfig bridge0 create
ifconfig bridge0 addm tap0 em0 up
ifconfig tap0 up

In my case, tap0 would never stay up for long, unless it's used. I've read somewhere that there is a sysctl to fix that, but can't find the link anymore.
In order to make the tap bridge available on each reboot, edit /boot/loader.conf and add:

aio_load="YES"
if_tap_load="YES"
bridge_load="YES"

You also need to edit /etc/rc.conf on the server to get the tap and bridge set up properly and also to get the NFS server running.
The necessary additions to /etc/rc.conf are the following:

cloned_interfaces="tap0 bridge0"
ifconfig_bridge0="addm tap0 addm em0 up"
ifconfig_tap0="up"
rpcbind_enable="YES"
mountd_enable="YES"
mountd_flags="-r
nfs_server_enable="YES"
nfs_server_flags="-t -u -n4"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"
dhcpd_enable"YES"
dhcpd_ifaces="em0"

Now the DHCP and NFS servers can be started with:

/usr/local/etc/rc.d/isc-dhcpd start
/etc/rc.d/rpcbind start
/etc/rc.d/mountd start
/etc/rc.d/nfsserver start
/etc/rc.d/lockd start
/etc/rc.d/statd start

In my case /etc/rc.d/nfsserver start wouldn't do anything, so I had to start nfsd like that:

nfsd -t -u -n4

After setting up everything it should be possible to start Qemu, boot the kernel and run FreeBSD from the NFS mounted root.
The start command (from here):

qemu-system-arm -M connex -m 289 -pflash flash -nographic -serial tcp::4000,server -net nic,macaddr=23:32:43:34:54:45,vlan=0,model=smc91c111 -net tap,vlan=0,ifname=tap0,script=no

Then in a different terminal connect to the machine via telnet:

telnet localhost 4000

Qemu will then boot the GUMSTIX u-boot first and try to load the kernel from a wrong address. At the prompt enter:

GUM> bootelf 40000

You should see the kernel boot, acquire a DHCP address (10.0.2.20) with the root-path option and then boot init from the NFS root. If everything worked, you'll end up at a login screen.