ZFS: Setting up a ZFS Mirror

Lately I have been performing upgrades to the hard drives in my computers. The hard drive related project today is setting up a ZFS mirror on my desktop which will contain my /home directory.

ZFS Intro

I assume if you’re reading this then might you know was ZFS is, but in case you don’t, ZFS is an advanced file system developed by Sun Microsystems. It provides a lot of really attractive features for providing redundancy in your data. ZFS allows you to integrate many disks into a “zpool” for storage. There are many different configurations one could setup, but I’ll just focus on a simple ZFS mirror, where I have two disks, each of which will contain the same data, hence the name ‘mirror’.

This post is going to cover my experience setting up a ZFS mirror, if you want to know more about ZFS (and trust me, there is a lot more to know), I’ve really found this series from Aaron Toponce to be the best guide. That and the ArchWiki ZFS post of course.

Motivation

So why setup a ZFS mirror? Well, I’ve always been interested in setting up a RAID on my desktop. In doing some research on RAID I came across ZFS and it seemed to me like the better option. I’m pretty paranoid when it comes to keeping backups of my data, and while something like RAID or ZFS does not substitute for proper backups, the thought of redundant drives is appealing to me. Once setup, if we lose a disk due to hardware failure the pool will still run in a degraded state, giving us time to swap in a new drive.

Initial HDD Configuration

If you’re reading with the intention of setting up your own ZFS mirror we’re probably starting in different initial configurations. My setup at the start involves two SSDs and two HDDs. The two SSDs are 120 GB each; one boots Windows, the other contains /, /boot/ and swap for my Arch linux installation. (It probably shouldn’t have swap on it, but I’ll address that in another post.)

The two HDDs are different capacities. Their is a 640GB drive, which holds my /home directory, and a 2TB drive which is split into two 1TB partitions, one for Windows, one for linux, both for storage.

I did a lot of data shuffling to prep for setting up this ZFS mirror with the 2TB drive and a new 2TB drive I just purchased. So, I’m starting from a configuration where I’m booting from the linux SSD and I have no home directory (It’s backed up externally.) This gives us two empty 2TB drives to create the mirror on.

To get started we’ll need to install ZFS, and on Arch this is going to be from the AUR. Install how you want, but I use yaourt:

yaourt -S zfs-git

The kernel modules for ZFS are tied to a specific kernel, which required a downgrade for me at the time I installed. Once you exit dependency hell zfs will be installed. Before we jump into making a zpool we should take a second to think about the drives we’re using.

Advanced Format Drives

Something I learned about while setting this up (learning new things is another large part of the motivation for me here) is Advanced Format drives. Older drives will store data in 512 byte sectors. New “Advanced Format” drives store data in 4096 byte sectors.

Where this comes in with ZFS is a potential performance loss if you don’t explicitly define the sector size when making the zpool. This is pointed out in the ArchWiki, and can be done by providing the -o ashift=12 option.

Mixing Drives Types

Part of what made me read more and more about Advanced Format drives is the fact that I didn’t know how to tell if the drives I had were Advanced Format or not and if the two could be mixed in a zpool. They’re both WD Black drives but they were bought almost two years apart. Fortunately, I found this (pdf) model number format guide which details the letters in each model number.

I have a new WD2003FZEX drive and an old WD2002FAEX drive. The difference being the new drive is Advanced Format and the old drive isn’t. So, can we mix the two?

Several people on a few forums suggest forcing the 4K sectors when mixing drives, as not much performance is lost in forcing 512 byte sectors to align to 4096 byte sector, but there is definite performance lost the other way. So I’ll be throwing that -o ashift=12 option when I create my zpool. If you’re using Advanced Format drives you should do the same.

Setting Up the zpool

I wiped the data off my old drive and so both drives are unformatted (i.e. no file system on them). To see all your drives we can use parted,

[root@example ~]# parted --list

Model: ATA WDC WD2003FZEX-0 (scsi)
Disk /dev/sda: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  2000GB  2000GB               zfs
 9      2000GB  2000GB  8389kB


Model: ATA WDC WD6401AALS-0 (scsi)
Disk /dev/sdb: 640GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End    Size   Type     File system  Flags
 1      1049kB  640GB  640GB  primary  ntfs


Model: ATA WDC WD2002FAEX-0 (scsi)
Disk /dev/sdc: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  2000GB  2000GB               zfs
 9      2000GB  2000GB  8389kB


Model: ATA OCZ-VECTOR (scsi)
Disk /dev/sdd: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End    Size   Type     File system  Flags
 1      1049kB  128GB  128GB  primary  ntfs         boot


Model: ATA OCZ-VECTOR (scsi)
Disk /dev/sde: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system     Name  Flags
 1      1049kB  3146kB  2097kB                        bios_grub
 2      3146kB  527MB   524MB   ext4                  boot, esp
 3      527MB   17.7GB  17.2GB  linux-swap(v1)
 4      17.7GB  128GB   110GB   ext4                  msftdata

I’m looking for my only two 2TB drives, which for me were /dev/sda and /dev/sdc, your drives will probably be different. We’ll want the disk id’s to make the zpool, which we can get just by looking in /dev/,

[root@example ~]# ls -lah /dev/disk/by-id/
total 0
drwxr-xr-x 2 root root 660 Mar 24 22:43 .
drwxr-xr-x 7 root root 140 Mar 24 22:39 ..
lrwxrwxrwx 1 root root   9 Mar 24 22:04 ata-ASUS_DRW-24B1ST_a_B5D0CL354810 -> ../../sr0
lrwxrwxrwx 1 root root   9 Mar 24 22:43 ata-OCZ-VECTOR_OCZ-63C1889KG99WW96L -> ../../sdd
lrwxrwxrwx 1 root root  10 Mar 24 22:43 ata-OCZ-VECTOR_OCZ-63C1889KG99WW96L-part1 -> ../../sdd1
lrwxrwxrwx 1 root root   9 Mar 24 22:43 ata-OCZ-VECTOR_OCZ-8MGKY5RPKL9I42JV -> ../../sde
lrwxrwxrwx 1 root root  10 Mar 24 22:43 ata-OCZ-VECTOR_OCZ-8MGKY5RPKL9I42JV-part1 -> ../../sde1
lrwxrwxrwx 1 root root  10 Mar 24 22:43 ata-OCZ-VECTOR_OCZ-8MGKY5RPKL9I42JV-part2 -> ../../sde2
lrwxrwxrwx 1 root root  10 Mar 24 22:43 ata-OCZ-VECTOR_OCZ-8MGKY5RPKL9I42JV-part3 -> ../../sde3
lrwxrwxrwx 1 root root  10 Mar 24 22:43 ata-OCZ-VECTOR_OCZ-8MGKY5RPKL9I42JV-part4 -> ../../sde4
lrwxrwxrwx 1 root root   9 Mar 24 22:43 ata-WDC_WD2002FAEX-007BA0_WD-WCAY00770606 -> ../../sdc
lrwxrwxrwx 1 root root  10 Mar 24 22:43 ata-WDC_WD2002FAEX-007BA0_WD-WCAY00770606-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  10 Mar 24 22:43 ata-WDC_WD2002FAEX-007BA0_WD-WCAY00770606-part9 -> ../../sdc9
lrwxrwxrwx 1 root root   9 Mar 24 22:43 ata-WDC_WD2003FZEX-00Z4SA0_WD-WMC5H0DAU37A -> ../../sda
lrwxrwxrwx 1 root root  10 Mar 24 22:43 ata-WDC_WD2003FZEX-00Z4SA0_WD-WMC5H0DAU37A-part1 -> ../../sda1
lrwxrwxrwx 1 root root  10 Mar 24 22:43 ata-WDC_WD2003FZEX-00Z4SA0_WD-WMC5H0DAU37A-part9 -> ../../sda9
lrwxrwxrwx 1 root root   9 Mar 24 22:43 ata-WDC_WD6401AALS-00J7B0_WD-WMATV7730189 -> ../../sdb
lrwxrwxrwx 1 root root  10 Mar 24 22:43 ata-WDC_WD6401AALS-00J7B0_WD-WMATV7730189-part1 -> ../../sdb1
lrwxrwxrwx 1 root root   9 Mar 24 22:43 wwn-0x50014ee0579c8eef -> ../../sdb
lrwxrwxrwx 1 root root  10 Mar 24 22:43 wwn-0x50014ee0579c8eef-part1 -> ../../sdb1
lrwxrwxrwx 1 root root   9 Mar 24 22:43 wwn-0x50014ee0ae9998c9 -> ../../sda
lrwxrwxrwx 1 root root  10 Mar 24 22:43 wwn-0x50014ee0ae9998c9-part1 -> ../../sda1
lrwxrwxrwx 1 root root  10 Mar 24 22:43 wwn-0x50014ee0ae9998c9-part9 -> ../../sda9
lrwxrwxrwx 1 root root   9 Mar 24 22:43 wwn-0x50014ee25cdf92e3 -> ../../sdc
lrwxrwxrwx 1 root root  10 Mar 24 22:43 wwn-0x50014ee25cdf92e3-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  10 Mar 24 22:43 wwn-0x50014ee25cdf92e3-part9 -> ../../sdc9
lrwxrwxrwx 1 root root   9 Mar 24 22:43 wwn-0x5e83a972745093b5 -> ../../sde
lrwxrwxrwx 1 root root  10 Mar 24 22:43 wwn-0x5e83a972745093b5-part1 -> ../../sde1
lrwxrwxrwx 1 root root  10 Mar 24 22:43 wwn-0x5e83a972745093b5-part2 -> ../../sde2
lrwxrwxrwx 1 root root  10 Mar 24 22:43 wwn-0x5e83a972745093b5-part3 -> ../../sde3
lrwxrwxrwx 1 root root  10 Mar 24 22:43 wwn-0x5e83a972745093b5-part4 -> ../../sde4
lrwxrwxrwx 1 root root   9 Mar 24 22:43 wwn-0x5e83a97514d28d2a -> ../../sdd
lrwxrwxrwx 1 root root  10 Mar 24 22:43 wwn-0x5e83a97514d28d2a-part1 -> ../../sdd1

So what we want to note is that sda has the id ata-WDC_WD2003FZEX-00Z4SA0_WD-WMC5H0DAU37A and sdc has the id ata-WDC_WD2002FAEX-007BA0_WD-WCAY00770606.

We’ll need this info to make the zpool, which we can do with the zpool create command:

[koopman@honey etc]$ sudo zpool create -f -o ashift=12 -m /home lotus mirror ata-WDC_WD2003FZEX-00Z4SA0_WD-WMC5H0DAU37A ata-WDC_WD2002FAEX-007BA0_WD-WCAY00770606

The options I’ve used are -f, which forces the use of vdevs (virtual devices), ours will be a mirror, -o ashift=12, which forces the 4096 byte sectors for advanced format drives, and -m /home, which sets the mountpoint for the zpool. The rest of the command contains lotus, the name of my new pool, mirror, the type of ZFS pool we wanted to create, and the two disk id’s.

Making a dataset

Now that we have the zpool we will create a dataset for our home directory:

sudo zfs create lotus/koopman

Datasets allow you to do such things as setting disk usage quotas and maintaining individual snapshots per dataset.

Note that you’ll probably need to chown the new directory to use it,

sudo chown koopman koopman/

Now we can start using our new zpool!

Finishing Up

This was a lot easier than I pictured years ago when thinking I’d like to setup a RAID, granted we used ZFS, which seems to be a lot nicer in many respects.

To wrap up we’ll do some quick tweaks. The first of which will be auto-mounting at boot. The zfs daemon is capable of loading a zpool and mounting it at boot. In order to do this we need to setup a cache file, then enable the zfs daemon.

zpool set cachefile=/etc/zfs/zpool.cache lotus
systemctl enable zfs.target

Next, we’re going to enable relatime and turn on lz4 compression to decrease the number of superfluous writes we do to the zpool and to compress our data.

zfs set relatime=on lotus
zfs set compression=lz4 lotus

And that’s it, we’ve setup a ZFS mirror as our home directory! You can check out your new zpool with a simple zpool status:

$ sudo zpool status
[sudo] password for koopman: 
  pool: lotus
 state: ONLINE
  scan: scrub repaired 0 in 2h55m with 0 errors on Wed Apr 15 23:22:56 2015
config:

    NAME                                            STATE     READ WRITE CKSUM
    lotus                                           ONLINE       0     0     0
      mirror-0                                      ONLINE       0     0     0
        ata-WDC_WD2003FZEX-00Z4SA0_WD-WMC5H0DAU37A  ONLINE       0     0     0
        ata-WDC_WD2002FAEX-007BA0_WD-WCAY00770606   ONLINE       0     0     0

errors: No known data errors

(Note: On a fresh new pool you won’t see the “scan” line, this comes in from scrubbing your zpool, something that I’ll describe how to automate in a later post.)

Then all we need to do is rsync our data to our home directory from our backups and enjoy the new disk redundancy.

Further Adjustments

I really setup this pool about a month ago. There are many more things you can and should do with your zpool, such as having automated snapshots and scrubbing your zpool regularly. Setting all these things up at once takes a while on the first time through, and I was planning on including it all in this post. As a result, this already lengthy post was becoming way too long. So expect to see more ZFS related posts in the near future.