The Problem
The RDMA service fails to start on CentOS/RHEL 7 machine. ‘journalctl -r’ command shows the below errors:
Oct 11 11:01:15 geeklab systemd-modules-load: Failed to insert 'rds_rdma': Address family not supported by protocol Oct 11 11:01:15 geeklab systemd-modules-load: Failed to insert 'rds_rdma': Address family not supported by protocol Oct 11 11:01:15 geeklab systemd-modules-load: Failed to insert 'resilient_rdmaip': Address family not supported by protocol Oct 11 11:01:15 geeklab systemd-modules-load: Failed to insert 'resilient_rdmaip': Address family not supported by protocol Oct 11 11:01:15 geeklab systemd: rdma-load-modules@rdma.service: main process exited, code=exited, status=1/FAILURE Oct 11 11:01:15 geeklab systemd: rdma-load-modules@rdma.service: main process exited, code=exited, status=1/FAILURE Oct 11 11:01:15 geeklab systemd: Failed to start Load RDMA modules from /etc/rdma/modules/rdma.conf. Oct 11 11:01:15 geeklab systemd: Failed to start Load RDMA modules from /etc/rdma/modules/rdma.conf. Oct 11 11:01:15 geeklab systemd: Unit rdma-load-modules@rdma.service entered failed state. Oct 11 11:01:15 geeklab systemd: Unit rdma-load-modules@rdma.service entered failed state.
The Solution
The issue is caused by the following setup were the rds_rdma module was not being loaded to the kernel due to “ipv6.disable=1” parameter on /etc/default/grub/ and on /boot/grub2/grub.cfg
/etc/default/grub configuration file:
########### BEGIN DO NOT REMOVE Added by Oracle Exadata ###########
GRUB_DEFAULT=0
GRUB_DISABLE_OS_PROBER=true
GRUB_DISABLE_RECOVERY=true
GRUB_DISABLE_SUBMENU=y
GRUB_DISTRIBUTOR="Oracle Linux"
GRUB_TIMEOUT=5
GRUB_TIMEOUT_STYLE=menu
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
export EXADATA_IMAGE_TYPE=BareMetal
GRUB_CMDLINE_LINUX_DEFAULT="root=LABEL=DBSYS bootarea=dbsys bootfrom=BOOT ro loglevel=7 panic=60 pci=noaer log_buf_len=1m nmi_watchdog=0 transparent_hugepage=never rd_NO_PLYMOUTH audit=1 console=tty1 console=ttyS0,115200n8 crashkernel=448M processor.max_cstate=1 clocksource=tsc nohpet nopmtimer hda=noprobe hdb=noprobe ide0=noprobe pci=nocrs ifnames_skip=100 biosdevname=0 net.ifnames=0 "
########### END DO NOT REMOVE Added by Oracle Exadata ###########
#OS hardening start
GRUB_CMDLINE_LINUX='ipv6.disable=1'
#OS hardening end
/boot/grub2/grub.cfg configuration file:
linux16 /vmlinuz-4.14.35-1902.301.1.el7uek.x86_64 root=/dev/mapper/VGExaDb-LVDbSys1
ro ipv6.disable=1 root=LABEL=DBSYS bootarea=dbsys bootfrom=BOOT
ro loglevel=7 panic=60 pci=noaer log_buf_len=1m nmi_watchdog=0 transparent_hugepage=never rd_NO_PLYMOUTH audit=1
console=tty1 console=ttyS0,115200n8 crashkernel=448M processor.max_cstate=1 clocksource=tsc nohpet nopmtimer
hda=noprobe hdb=noprobe ide0=noprobe pci=nocrs ifnames_skip=100 biosdevname=0 net.ifnames=0
initrd16 /initramfs-4.14.35-1902.301.1.el7uek.x86_64.img
The RDMA is dependent to the ipv6 code to be available which is being prevented by this parameter “ipv6.disable=1”. In order to resolve the issue follow the steps outlined below:
1. Please back up the /etc/default/grub before any change. Please remove the following option (or line) from the /boot/grub2/grub.cfg boot command line:
# vi /boot/grub2/grub.cfg
Remove the below line:
ipv6.disable=1
It is because the RDMA depends on the IPv6 code to be available, but this prevents it from loading at all.
2. Please back up the /boot/grub2/grub.cfg. Please run the following command to rebuld the grub file.
# grub2-mkconfig -o /boot/grub2/grub.cfg
3. Please back up the current initramfs file. Then rebuild the initramfs.
# cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).bak.$(date +%m-%d-%H%M%S).img # dracut -f
4. Please reboot the server to check the result.