What is SCSI reservation
SCSI Reservation is a mechanism which allows SCSI initiators to reserve a LUN for exclusive access and preventing other initiators from making changes. The feature is usually used in the cluster. This post elaborates the basic usage of SCSI reservation in CentOS/RHEL systems.
SCSI Reservation contains two stages: First, initiators must register a reservation key, then continue to reserve the device using the same reservation key when a host need exclusive access. Let’s see how we can use the SCSI reservation and necessary functions for SCSI Reservation.
Install the necessary utils
The sg_persist command provides all the necessary functions for SCSI Reservation, it’s contained in the package sg3_utils.
# yum install sg3_utils
View the registration
In the example shown below, no reservation key had been registered yet.
# sg_persist /dev/sdc >> No service action given; assume Persistent Reserve In command >> with Read Keys service action IET VIRTUAL-DISK 0001 Peripheral device type: disk PR generation=0x4, there are NO registered reservation keys
Register a reservation key
The reservation key must be hex strings, and up to 8 bytes long, here we use abc123 for example.
# sg_persist --out --register --param-sark=abc123 /dev/sdc
The output below show a SCSI LUN which have 2 reservation keys (or 2 hosts) registered.
# sg_persist /dev/sdc >> No service action given; assume Persistent Reserve In command >> with Read Keys service action IET VIRTUAL-DISK 0001 Peripheral device type: disk PR generation=0x6, 2 registered reservation keys follow: 0xabc123 0x123abc
Reserve a registered LUN on behalf of a given key
# sg_persist --out --reserve --param-rk=abc123 --prout-type=3 /dev/sdc IET VIRTUAL-DISK 0001 Peripheral device type: disk
The –prout-type parameter specified the reservation type, from manpage, valid types including:
- 1 : write exclusive
- 3 : exclusive access
- 5 : write exclusive – registrants only
- 6 : exclusive access – registrants only
- 7 : write exclusive – all registrants
- 8 : exclusive access – all registrants
View the reservation
The output indicated the server was reserved by key abc123, with type 3 (exclusive access):
# sg_persist -r /dev/sdc IET VIRTUAL-DISK 0001 Peripheral device type: disk PR generation=0x6, Reservation follows: Key=0xabc123 scope: LU_SCOPE, type: Exclusive Access
Verify the reservation
On node1 which have /dev/sdc1 reserved, it was able to mount the disk.
# mount /dev/sdc1 /mnt # mount | grep mnt /dev/sdc1 on /mnt type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered) # umount /mnt
In the meanwhile, it was not able to access the same disk on node2
# mount /dev/sdc1 /mnt mount: mount /dev/sdc1 on /mnt failed: Invalid exchange
The dmesg show reservation conflict
# dmesg | tail [6902380.608058] sd 11:0:0:1: [sdc] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK [6902380.608060] sd 11:0:0:1: [sdc] tag#16 CDB: Read(10) 28 00 00 1f ff 80 00 00 08 00 [6902380.608061] blk_update_request: critical nexus error, dev sdc, sector 2097024 [6902380.608064] Buffer I/O error on dev sdc1, logical block 261872, async page read [6902380.609007] sd 11:0:0:1: reservation conflict [6902380.609011] sd 11:0:0:1: [sdc] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK [6902380.609013] sd 11:0:0:1: [sdc] tag#14 CDB: Read(10) 28 00 00 00 08 00 00 00 01 00 [6902380.609015] blk_update_request: critical nexus error, dev sdc, sector 2048 [6902380.609523] sd 11:0:0:1: reservation conflict [6902380.609526] blk_update_request: critical nexus error, dev sdc, sector 0
Release the reservation
# sg_persist --out --release --param-rk=abc123 --prout-type=3 /dev/sdc
Unregister a reservation key
# sg_persist --out --register --param-rk=abc123 /dev/sdc