Multipathing Overview
A path is a connection between a server and the underlying storage. The path can be severed due to many reasons like faulty HBA, faulty cable etc. To avoid such single point of failures, multipathing exists. Multipathing ensures that the system uses multiple physical paths to provide redundancy and increased throughput. There are many vendor specific multipathing implementations like EMC’s powerpath and Symantecs VxDMP.
What is Device Mapper multipath
Device Mapper Multipathing (or DM-multipathing) is a Linux native multipath tool, which allows you to configure multiple I/O paths between server nodes and storage arrays into a single device. These I/O paths are physical SAN connections that can include separate cables, switches, and controllers. Multipathing aggregates the I/O paths, creating a new device that consists of the aggregated paths. Regardless of the vendor hardware in use, device mapper creates a block device under /dev/mapper/ for each LUN attached to the system.
Device Mapper components
The important components of Device Mapper multipathing are :
| Component | Description |
|---|---|
| dm-multipath | kernel module responsible for making routing decisions under normal/failure conditions |
| multipath | Command used for viewing/listing multipath devices and for initial configuration |
| multipathd | daemon that moitors path, marks failed paths, reactivates restored paths, adds/removes device files as needed. |
| kpartx | command used to create device mapper entries for partitions on multipathed LUN. It is invoked automatically when multipath command is used. |
How to verify if DMMP is installed and configured
1. Check whether device-mapper is installed.
# rpm -qa |grep device-mapper device-mapper-1.02.39-1.el5 device-mapper-multipath-0.4.7-34.el5 device-mapper-1.02.39-1.el5 device-mapper-event-1.02.39-1.el5
2. Check that the following device mapper modules are loaded.
# lsmod |grep dm_multipath dm_multipath 56921 2 dm_round_robin scsi_dh 42177 2 scsi_dh_rdac,dm_multipath dm_mod 101649 11 dm_mirror,dm_multipath,dm_raid45,dm_log
3. If above conditions are met, check whether the file /etc/multipath.conf is configured. Make sure the lines in bold are commented out in order to enable device mapper.
# This is a basic configuration file with some examples, for device mapper multipath
......
# Blacklist all devices by default. Remove this to enable multipathing
# on the default devices.
#blacklist {
# devnode "*"
#}
......
4. Check whether multipathd is running.
# /etc/init.d/multipathd status "multipathd (pid 11405) is running..."
5. If yes, check any devices listed using the command below.
# multipath -v2 or # multipath -ll mpath15 (3600a0b8000473abc0000bafc52fac127) dm-14 SUN,STK6580_6780 [size=10G][features=0][hwhandler=0][rw] _ round-robin 0 [prio=1][enabled] _ 8:0:0:2 sds 65:32 [active][ready] _ round-robin 0 [prio=0][enabled] _ 9:0:0:2 sdu 65:64 [active][faulty] mpath13 (3600a0b8000473abc0000bb74530aa7da) dm-12 SUN,STK6580_6780 [size=931G][features=0][hwhandler=0][rw] _ round-robin 0 [prio=1][enabled] _ 9:0:0:0 sdp 8:240 [active][ready] _ round-robin 0 [prio=0][enabled] _ 8:0:0:0 sdo 8:224 [active][faulty]
If all the above steps succeed, the system is configured for DMMP.
Multipathing Configuration
Before starting to configure the multipathing, make sure the device-mapper-multipath package is installed. If not installed, install it using yum :
# yum -y install device-mapper-multipath
The device mapper multipathing uses the configuration file /etc/multipath.conf for the configuration. If you make any changes to this file the multipath command must be run in order to reconfigure the multipathed devices. The easiest way to create this file is to use the mpathconf utility. If there is an existing configuration file mpathconf will edit it, if no such file exists it will copy /usr/share/doc/device-mapper-multipath-*/multipath.conf.
# mpathconf --enable --with_multipathd y --with_chkconfig y
The configuration file consists of 5 major sections as below :
| Section | Description |
|---|---|
| defaults | system-level default configuration |
| blacklist | Blacklisted devies. Devices that should not be configured under DMMP |
| blacklist_exceptions | Exceptions to the blacklisted devices |
| devices | settings for individual storage controller devices |
| multipaths | fine-tune configuration of individual LUNs |
Verifying Configuration
The multipath command can be used to verify the multipathinf configuration. To list the information about multipathed devices :
# multipath -ll mpath0 (3600a0b8000473abc0000bafc52fac127) dm-14 SUN,STK6580_6780 [size=10G][features=0][hwhandler=0][rw] _ round-robin 0 [prio=1][enabled] _ 8:0:0:2 sds 65:32 [active][ready] _ round-robin 0 [prio=0][enabled] _ 9:0:0:2 sdu 65:64 [active][faulty]
The output shows a multipathed LUN, mpath0. The number following it is the LUN’s WWID. The status active/ready indicates that the path is ready for I/O. If the path is showing faulty/failed then it needs to be repaired before using it for I/O. After the configuration is completed, we can start the multipathd persistently :
# /etc/init.d/multipathd start # chkconfig multipathd on
User Friendly Device Names
In order to troubleshoot efficiently, device-mapper can be configured to have human readable, user friendly device names under /dev/mapper instead of using the WWIDs. The user friendly names like /dev/mapper/mpath0 can be created by enabling the user_friendly_names option in /etc/multipath.conf file :
defaults {
user_friendly_names yes
}
You can also control the name for a particular LUN by using the alias option :
multipaths {
multipath {
wwid 3600a0b8000473abc0000bafc52fac127
alias mdisk001
}
}
Removing Multipath
After removing the all the paths for a multipathed device, run the below command to remove the multipath device completely :
# multipath -f [device]
To flush all the multipathed device after stopping the multipahtd daemon :
# multipath -F