Recommended ZFS Storage Pool Practices

ZFS uses a pool storage model where storage devices are aggregated into a storage pool. File systems within the storage pool use all the storage in the pool. The following sections provide recommended practices for creating, monitoring, and troubleshooting ZFS storage pools.

ZFS Storage Pool Creation Practices

Root pool – Create pools with slices by using the s* identifier. Do not use the p* identifier. In general, a system's ZFS root pool is created when the system is installed. If you are creating a second root pool or re-creating a root pool, use syntax similar to the following:

# zpool create rpool c0t1d0s0
Or, create a mirrored root pool. For example:

# zpool create rpool mirror c0t1d0s0 c0t2d0s0
  •  Consider keeping root pool components separate from non-root pool data.
The root pool must be created as a mirrored configuration or as a single-disk configuration. A RAID-Z or a striped configuration is not supported. You cannot add additional disks to create multiple mirrored top-level virtual devices by using the zpooladd command, but you can expand a mirrored virtual device by using the zpool attach command.
  • The root pool cannot have a separate log device.
You can set pool properties during an AI installation by using the pool_options keyword syntax. The gzip compression algorithm is not supported on root pools.
Do not rename the root pool after it is created by an initial installation. Renaming the root pool might cause an system not to boot.
Non-root pools – Create non-root pools with whole disks by using the d* identifier. Do not use the p* identifier. 

  • ZFS works best without any additional volume management software.
For better performance, use individual disks or at least LUNs made up of just a few disks. By providing ZFS with more visibility into the LUN setup, ZFS is able to make better I/O scheduling decisions.
Mirrored storage pools – Consume more disk space but generally perform better with small random reads. For example:
# zpool create tank mirror c1d0 c2d0 mirror c3d0 c4d0
Mirrored storage pools are also more flexible in that you can detach, attach, and replace existing devices in the pool. 

RAID-Z storage pools – Can be created with 3 parity strategies, where parity equals 1 (raidz), 2 (raidz2), or 3 (raidz3).

A RAID-Z configuration maximizes disk space and generally performs well when data is written and read in large chunks (128K or more). Create a single-parity RAIDZ (raidz) configuration at 3 disks (2+1).

A RAIDZ-3 configuration maximizes disk space and offers excellent availability because it can withstand 3 disk failures. Create a triple-parity RAID-Z (raidz3) configuration at 8 disks (5+3). 

Non-redundant pools – If you create a non-redundant pool, you will see a message similar to the following:

# zpool create pond c8t2d0 c8t3d0  
'pond' successfully created, but with no redundancy; failure of one device will cause loss of the pool

Creating a pool with no redundancy is not recommended because a device failure could mean that the data is unrecoverable. Consider creating a ZFS storage pool with redundancy. For example:

# zpool create pond mirror c8t2d0 c8t3d0

ZFS Storage Pool Monitoring Practices

Make sure that pool capacity is below 80% for best performance. Monitor pool and file system space to make sure that they are not full. Consider using ZFS quotas and reservations to make sure file system space does not exceed 80% pool capacity.

Run zpool scrub on a regular basis to identify data integrity problems:
If you have datacenter-quality drives, consider a monthly scrubbing schedule.
You should also run a scrub prior to replacing devices to ensure that all devices are currently operational.
Use zpool status on a weekly basis to monitor pool and pool device status. Also use fmdump or fmdump -eV to see if any device faults or errors have occurred.

ZFS Storage Pool Troubleshooting Practices

Troubleshooting pool problems in Oracle Solaris 11 is similar to diagnosing problems in the Oracle Solaris 10 release, but review the following new diagnostic descriptions and features:

Failed devices – Review the zpool status -l output to identify the physical location of the failed device and replace it.

Failed device notification – The smtp-notify service can be configured to send electronic mail notifications in response to various fault management events, such as when a hardware component has been diagnosed as faulty.

By default, some notifications are set up automatically to be sent to the root user. If you add an alias for your user account as root in the /etc/aliases file, you will receive electronic mail notifications, similar to the following:

-------- Original Message --------
Subject: Fault Management Event: tardis:SMF-8000-YX
Date: Wed, 21 Sep 2011 11:11:27 GMT
From: No Access User <noaccess@tardis.drwho.COM>
Reply-To: root@tardis.drwho.COM
To: root@tardis.drwho.COM

SUNW-MSG-ID: ZFS-8000-D3, TYPE: Fault, VER: 1, SEVERITY: Major
EVENT-TIME: Wed Sep 21 11:11:27 GMT 2011
PLATFORM: Sun-Fire-X4140, CSN: 0904QAD02C, HOSTNAME: tardis
SOURCE: zfs-diagnosis, REV: 1.0
EVENT-ID: d9e3469f-8d84-4a03-b8a3-d0beb178c017
DESC: A ZFS device failed.  Refer to 
for more information.
AUTO-RESPONSE: No automated response will occur.
IMPACT: Fault tolerance of the pool may be compromised.
REC-ACTION: Run 'zpool status -x' and replace the bad device.

Moving devices – Devices that are part of a ZFS storage pool contain a device ID if the device driver creates or fabricates device IDs. Like all file systems, ZFS has a very close relationship with its underlying devices so if you attempt to upgrade a system's firmware, move a pool device to a different controller, or change a device's cabling, you might consider exporting the pool first. If the device ID does not follow the device change and this can happen with non-Oracle hardware, then the pool and pool data might become unavailable. In general, Oracle's Sun hardware can recover if a device is changed under a live pool because our drivers fully support device IDs, but you might consider exporting the pool before making any hardware changes.

For a full description of troubleshooting pool problems, see Chapter 11, Oracle Solaris ZFS Troubleshooting and Pool Recovery, in Oracle Solaris Administration: ZFS File Systems.
Powered by Blogger.