The Problem
Oracle Real Application Clusters (RAC) servers run on CentOS/RHEL 7 and utilizing ASMLib-managed iSCSI multipathed disks. Upon server reboot, Oracle ASMLib disks are not automatically discovered at boot time causing Oracle Cluster Ready Services (CRS) to fail to start. ASM disks are, however, correctly discovered when manually running the oracleasm scandisks command soon after the server reboot.
Following is the oracleasm configuration
# cat /etc/sysconfig/oracleasm ... ORACLEASM_ENABLED=true ORACLEASM_UID=oracle ORACLEASM_GID=dba ORACLEASM_SCANBOOT=true ORACLEASM_SCANORDER="" ORACLEASM_SCANEXCLUDE="" ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false ...
# cat -n /usr/lib/systemd/system/oracleasm.service 1 [Unit] 2 Description=Load oracleasm Modules 3 Requires=multipathd.service 4 After=multipathd.service ...
The Solution
Reviewing system startup messages within the system log (/var/log/messages), it is evident that:
– oracleasm.service starts before iscsid.service completes making all iSCSI targets fully available/operational and before multipathd.service adds at least one path to iSCSI targets.
– the ohasd.service starts after oracleasm.service but no ASM disks are discovered.
Modify file /usr/lib/systemd/system/oracleasm.service and append both the ‘Requires‘ and ‘After‘ lines with iscsid.service multi-user.target to ensure the oracleasm.service starts after completion of iscsid.service and multipathd.service and after multi-user.target has been reached i.e.:
# cat -n /usr/lib/systemd/system/oracleasm.service 1 [Unit] 2 Description=Load oracleasm Modules 3 Requires=multipathd.service iscsid.service multi-user.target 4 After=multipathd.service iscsid.service multi-user.target ...
Modify file /etc/systemd/system/oracle-ohasd.service – append the ‘After‘ line with oracleasm.service to ensure the CRS ohasd.service starts after oracleasm.service has successfully discovered all iSCSI multipathed ASM disks i.e.:
# cat -n /etc/systemd/system/oracle-ohasd.service ... 5 [Unit] 6 Description=Oracle High Availability Services 7 After=syslog.target network-online.target remote-fs.target oracleasm.service ...