After reading this post, you should be able to:
- Tune NOP-Out interval and timeout settings for improved network monitoring.
- Manage recovery timeout setting on iSCSI initiators.
NOP-Out interval and timeout
Communication between iSCSI initiators and targets are sent in iSCSI protocol data units (PDUs). Request PDUs are sent by initiators, while response PDUs are sent by targets.
The Opcode field in the iSCSI POU header indicates the iSCSI POU type. There are two categories of opcodes: initiator opcodes and target opcodes. Initiator opcodes are used by initiators in request PDUs to targets, while target opcodes are used by targets in response PDUs to initiators.
One of the initiator opcodes is NOP-Out. Its target opcode counterpart is the NOP-In. The NOPOut opcode can be sent by initiators to targets as a “ping” request to verify that a connection or session is active and operational and that its components are operational. Upon receiving a NOP-Out POU from an initiator, a target responds with a NOP-In POU. If a NOP-In response is not received by the initiator within a certain amount of time, the running commands will be failed and re-queued for a later retry.
By default, in CentOS/RHEL 7, initiators send out NOP-Out requests every five seconds. Once sent, NOP-Out requests will time out in five seconds. Depending on the iSCSI implementation or the performance and utilization of the SAN, it may be desirable to tune these timeout settings.
In /etc/iscsi/iscsid.conf, NOP-Out request interval can be specified with the following entry:
node.conn.timeo.noop_out_interval = [interval value]
NOP-Out request timeout can be customized by editing the following entry in the same file.
node.conn.timeo.noop_out_timeout = [timeout value]
Once changes to /etc/iscsi/iscsid. conf are made, the iscsi service must be restarted for the changes to take effect.
# systemctl restart iscsi
NOP-Out interval and timeout setting changes will only be applied to future target interactions. iSCSI targets that are already discovered or logged into will still retain the old settings. To effect the new settings on these targets, use the update operator to the iscsiadm command.
# iscsiadm -m node -T TARGET_NAME -p PORTAL_ADDRESS -o update -n node.conn[0).timeo.noop_out_interval -v INTERVAL_VALUE.
Session replacement timeout
Once NOP-Out timeout has expired, running commands will be failed immediately. However, if the SCSI Error Handler is active, running commands are not failed immediately. Instead, when NOP-Out request timeout is reached, the iSCSI layer will wait for the disrupted session or connection to re-establish before failing the running commands. The wait time is controlled by the replacement_timeout setting.
In CentOS/RHEL 7, replacement_timeout is set by default to 120 seconds. It controls how long the iSCSI layer will wait before failing pending SCSI commands up to a higher level like multipath or to an application if multipath is not being used. When multipath is being used, it is desirable to lower the replacement_timeout value. By lowering replacement_timeout from 15 to 20 seconds, during error conditions, pending 1/0 commands will be promptly directed to a new physical path while the iSCSI layer works on session recovery.
Changes made to the replacement_timeout parameter will not affect iSCSI targets which are already discovered or logged into. To apply a new replacement_timeout setting on these targets, use the update operator to the iscsiadm command.
# iscsiadm -m node -T TARGET_NAME -p PORTAL_ADDRESS -o update -n node.session.timeo.replacernent_timeout -v TIMEOUT_ VALUE.
To customize the replacement_timeout value for future discovered targets, edit the following entry in /etc/iscsi/iscsid.conf and then restarting the iscsi service.
node.session.timeo.replacement_timeout = [timeout value]