Solaris ZFS : How to Designate Hot Spares in a Storage Pool

What is a hot spare and how does it work?

The ZFS hot spares feature enables you to identify disks that can be used to replace a failed or faulted device in one or more storage pools. Designating a device as a hot spare means that the device is not an active device in a pool, but if an active device in the pool fails, the hot spare automatically replaces the failed device.

You can designate devices as hot spares with the zpool create command when you are creating a pool or with the zpool add command if the pool has already been created.

After a failed device has been replaced and resilvered, the spare is automatically detached and made available. An in-progress spare replacement can be canceled by detaching the spare. If the original faulted device is detached, the spare assumes its place in the configuration and is removed from the spare’s list of all active pools.

Designating Hot Spares in a Storage Pool

1. Adding spares while creating the storage pool

To designate hot spares to a pool that you are creating, use zpool create followed by the pool name, the configuration, the keyword spare, and the names of the spares. In the example below, a pool called appool is being created. Within this pool is a mirror that contains two disks: c2t1d0 c2t2d0. Two spares, c2t3d0 c2t4d0, have been designated.

# zpool create appool mirror c2t1d0 c2t2d0 spare c2t3d0 c2t4d0

If you look at the status of appool, you can see that the spares are part of the pool and that they have a status of available.

# zpool status appool
  pool: appool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        appool      ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c2t1d0  ONLINE       0     0     0
            c2t2d0  ONLINE       0     0     0
        spares
          c2t3d0    AVAIL   
          c2t4d0    AVAIL   

errors: No known data errors

In this scenario, if either or both of the mirrored disks were to fail, ZFS automatically replaces them with one or both of the available spares.

2. Adding spares to an existing storage pool

To designate hot spares by adding them to a pool after the pool is created, use the zpool add command followed by the pool name, the keyword spare, and the name of the disks that you want to designate as hot spares.

# zpool add appool spare c2t3d0 c2t4d0

In the example above, we have designated disks c2t3d0 and c2t4d0 as the spares and are adding them to the pool named appool. Then, you run the zpool status command for the pool to verify that the spares have been added successfully and they have. Notice that both spares have a status of available (AVAIL).

# zpool status appool
  pool: appool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        appool      ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c2t1d0  ONLINE       0     0     0
            c2t2d0  ONLINE       0     0     0
        spares
          c2t3d0    AVAIL   
          c2t4d0    AVAIL   

errors: No known data errors

Again, as you saw in the previous example, if either or both of the mirrored disks were to fail, ZFS automatically replaces them with one or both of the available spares.

Hot Spares in Action

Next, let us look at an example in which one of the active devices in appool has faulted and ZFS has automatically replaced the faulted device with one of the available spares.

# zpool status appool
  pool: appool
 state: DEGRADED
 status: One or more devices could not be opened.  Sufficient replicas
         exist for the pool to continue functioning in a degraded state.
 action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scrub: resilvered completed 0h12m with 0 errors on Tue Dec 13 14:16:0 2017
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        appool      ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c2t1d0  ONLINE       0     0     0
            c2t2d0  ONLINE       0     0     0
            spare-1 UNAVAIL      0     0     0
            c2t2d0  UNAVAIL      0     0     0  cannot open
            c2t4d0  ONLINE       0     0     0  58.5K resilvered

        spares
          c2t4d0    INUSE      currently in use   
          c2t3d0    AVAIL

In this example, disk c2t2d0 has faulted and is replaced automatically by the hot spare c2t4d0, which has been resilvered and now appears as an active device in the mirrored pool. Notice also that the status of the hot spare c2t4d0 has changed from available (AVAIL) to in use (INUSE).

Removing Hot Spares in a Storage Pool

To remove a hot spare from a storage pool, use the zpool remove command followed by the pool name and the name of the hot spare. In this example, you are removing the hot spare c2t3d0 from the pool named appool, leaving just one hot spare in the pool: c2t4d0.

# zpool remove appool c2t3d0

# zpool status appool
  pool: appool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        appool      ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c2t1d0  ONLINE       0     0     0
            c2t2d0  ONLINE       0     0     0
        spares
          c2t4d0    AVAIL   

errors: No known data errors

As you can see in this example, after it is removed, the hot spare no longer appears in the spares set.