Understanding Linux multipath (dm-multipath)

What is multipathing?

Multipathing allows the combination of multiple physical connections between a server and a storage array into one virtual device. This can be done to provide a more resilient connection to your storage (a path going down will not hamper connectivity), or to aggregate storage bandwidth for improved performance.

As an example, the server in the following diagram has two HBAs, each connected to a separate Fibre Channel switch, which in turn are connected to separate controllers on the storage array.

example multipath setup in linux

Red Hat Enterprise Linux 7 supports multipathing using the dm-multipath subsystem. This uses the kernel device mapper system to generate virtual devices, managed by the multipathd daemon and the multipath command-line tool.

The necessary binaries, daemons, and kernel modules are available in the device-mappermultipath package. Once device-mapper-multipath is installed, configured, and started, device nodes for multipathed devices will be created under two different locations.

For administrative purposes, multipath devices are created under /dev/mapper. They can be named mpathN[pM] if user-friendly names are chosen, or they can be named after the World Wide ID (WWID) of the storage device. An administrator can also set custom names for multipathed devices. These custom names are established using the alias option in the multipaths section of the multipath configuration file.

Multipath devices are also created under /dev in the form of /dev/dm-N to match those created under /dev/mapper. These devices are strictly for the system’s internal use and should therefore never be used directly for administrative purposes.

Note: Multipathing provides protection against a storage access path going down. If the storage itself becomes unavailable, access to the storage will be lost.

To create a multipath device, different paths will be combined into groups, based on settings in the /etc/multipath.conf configuration file. Typically only one group will be active at a time, but a group can consist of multiple paths. When a group fails, the multipath daemon will switch storage traffic to a different group.

Configuring multipathing

To configure multipathing, first make sure that the device-mapper-multipath package is installed.

# yum -y install device-mapper-multipath

Once the device-mapper-multipath package is installed, a configuration file must be created for the multipath daemon, /etc/multipath.conf. The easiest way to create this file is to use the mpathconf utility.

If there already is a file called /etc/multipath.conf, the mpathconf command will edit that file. If no such file exists, mpathconf will copy the default configuration from /usr/share/doc/device-mapper-multipath-*/multipath.conf. If that file does not exist, mpathconf will create a new configuration file from scratch.

To create a default configuration, and then start and enable the multipathd daemon, use the following command:

# mpathconf --enable --with_multipathd y --with_chkconfig y

Note: In the default configuration file created by mpathconf, user-friendly names are enabled with the user_friendly_names option. User-friendly names will result in multipathed devices being named mpathN. While this can be useful if there is only one multipathed device, it can become confusing when there are multiple multipathed devices. To disable user-friendly names, use the –user_friendly_names n option to mpathconf. This will result in multipathed devices being named after their WWIDs.

If fine-tuning multipath configuration is desired before starting the multipathd daemon, use the mpathconf command with just the –enable option:

# mpathconf --enable

After editing the configuration file, enable and start the multipathd daemon as normal with the systemctl command.

The multipath.conf configuration file

The multipath.conf configuration file consists of five sections:

multipath.conf section	Description
blacklist'{}	This sections defines which devices should be excluded from the multipath topology discovery.
blacklist_exceptions {}	This section defines which devices should be included in the multipath topology discovery, despite being listed in the blacklist section.
defaults {}	This section defines the default settings to be used for all multipaths, unless explicitly overridden in the devices {} or multipaths {} section.
devices {}	This section contains overrides for the defaults {} section for specific types of devices, unless overridden from the multipaths {} section. Devices are identified based on their vendor, product, and revision keywords (regular expressions matching information from sysfs).
multipaths {}	This section contains settings for specific multipaths. This section overrides what is defined in the defaults {} and devices {} section. Multipaths are identified based on their WWIDs (obtained using the getuid_callout function).

An easy way to remember overrides is: multipaths > devices > defaults.

Blacklisting

Devices can be blacklisted in the configuration file using the blacklist {} section of multipath.conf. If blacklisting using wildcards, individual devices can be exempted from the blacklist using the blacklist_exceptions {} section. Devices can be blacklisted using either their device node or their WWID. The following example shows an example of both:

blacklist {
    devnode "^cciss" 
    wwid 1234567890abcde
}

To determine the WWID of a disk device, use the scsi_id utility.

/usr/lib/udev/scsi_id -g -u /dev/sdN 
360014053bd9ea2a35914e39a556051cf

Defaults

Defaults for all multipaths can be set in the defaults {} section of multipath.conf. The complete list of all built-in defaults can be found in the file /usr/share/doc/devicemapper-multipath-*/multipath.conf.defaults. Some of the most interesting settings are:

path_selector: The algorithm that determines which path inside a priority group to use for the next I/O. The default of “round-robin 0” distributes I/O over all paths in the group. The number of request to be sent using one path before switching to the next is determined by the rr_min_io_rq setting. The alternatives are “queue-length 0“, which will send the next I/O request to the path with the shortest queue of outstanding requests, and “service-time 0“, which will send the next 1/0 request to the path that has the shortest estimated service time.
path_grouping_policy: This setting defines how multiple paths are combined into priority groups. In the default of failover, every path will be put into a separate group. On the other hand, with the multibus policy, all possible paths are aggregated into a single group. Before configuring a multipath device to use the multibus policy, make sure that the storage controller supports active-active connections.
path_checker: This setting determines how the multipathd daemon will determine if a path is healthy. Other than the hardware independent options of directio and readsector0, there are a number of hardware independent checkers. Although the default for this option is directio, it is typically overridden in one of the default devices specified in the devices {} section.
features: This option specifies the multipath features to enable. Syntax is the form of num list, where num represents the number of features being enabled and list represents list of features being enabled. The two available features are queue_if_no_path and no_partitions.
user_friendly_names: This setting determines whether multipaths without a defined alias will be named mpathN (when set to yes), or if they will be named after their WWID.

Note: If the queue_if_no_path feature is enabled with the setting, features “1 queue_if_no_path”, and paths fail, processes issuing I/O will hang until the paths are restored. This behavior is undesirable in cluster implementations since one node stuck blocking on 1/0 to a failed storage device can block the rest of the cluster from accessing the storage resource. To avoid this sitµation, specify a value of fail for the no-path_retry parameter. Doing so will fail I/O immediately back up to higher layers rather than blocking on 1/0 indefinitely until paths are recovered.

Note: The commented defaults {} section in the multipath.conf produced by mpathconf does not reflect the actual built-in defaults of the multipath daemon.

The devices {} section

In the devices {} section, the defaults for specific devices can be overridden. Inside the devices {} section, there are individual device {} subsections detailing settings for specific devices. Most common storage hardware already have their own section defined in the built-in defaults for the multipath daemon. If a hardware is not (yet) listed, a section for the hardware can be manually added. Following is an example definition for a nonexistent piece of storage hardware. The device itself is selected using a combination of vendor, product, and revision.

devices { 
    device { 
        vendor "MegaHyperSuperStorage" 
        product "BAS" 
        revision "513/B" 
        features "1 queue_if_no_path" 
        path_grouping_policy multibus 
        path_checker tur 
    }
}

In the preceding examples, the features line indicates that new I/O requests will be accepted and queued even when no paths are currently available.

The multipaths {} section

In the multipaths {} section, overrides can be defined for specific multipaths. This can be used to set different path_grouping policies for a specific multipath. One of the other common uses of the multipaths {} section is to define an alias for a multipath. When an alias is set. the name for the device node in /dev/mapper/ for this multipath will be based on the alias, making it easier to distinguish between different multipaths.

As an example, the following configuration will set an alias of clusterstorage for the multipath with a WWID of “1234567890abcdef”, as well as a path_selector of queue-length.

multipaths { 
    multipath { 
        wwid "1234567890abcdef" 
        alias "clusterstorage" 
        path_selector "queue-length 0"
    }
}

Adding partitions

To add a partition on a multipathed device, use the following steps:

Create the partition on the multipathed device using a partition editor, e.g., fdisk /dev/mapper/mpath0.
Run the partprobe command to update the kernel’s view of the partition table on all devices (including the devices collated into a multipath).
Run the command, kpartx -a, on the multipath device to create device mapper devices for the newly created partition(s).

Removing a multipath

After removing all paths for a multipath, remove the multipathed device by running the command multipath -f [device]. If the multipathd daemon has been stopped and there are still device nodes for multipathed devices, flush all multipathed devices by running multipath -F. This can be useful when testing out different configurations and see remnants of old configurations lingering around.