Oracle Clusterware may not startup (root.sh on second node fails with “Failure at final check of Oracle CRS stack.”), due to a firewall (such as iptables on Linux) configured on the cluster interconnect. Other symptoms include moderate to serious performance and stability issues, directly related to such firewalls. Examples are false evictions of nodes, ‘IPC SEND/RECEIVE TIMEOUT’ and slow/uneven performance.
Oracle RAC uses the cluster interconnect to send buffer cache blocks between instances running on different nodes. The cluster interconnect is also used for other critical smaller messages essential to the stability and scalability of the cluster. IPtables or similar firewall implementations are OSI Reference Model layer 3 (network) filtering method and should be disabled on the cluster interconnect since, by design, this network is trusted. The cluster interconnect should be a completely private/isolated (layer 2 packet processing), non-routable network (the only nodes connected to it are the cluster members themselves). It is therefore safe, and required, to disable any such software based firewalls for this network. If security is still a concern use a physically separate dedicated switch for the cluster interconnect, or configure the cluster interconnect to use a Virtual Local Area Network (VLAN; OSI network layer 2) dedicated to a private subnet.
This note applies to all platforms that run Oracle RAC, including Windows all Unix flavors and Linux. Sample commands on Linux are as follows:
1. To check if iptables and or ip6tables is enabled on any runlevel:
# chkconfig --list iptables iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
or on your current run level:
# service iptables status Firewall is stopped.
2. To disable iptables on all run levels (for next boot):
# chkconfig iptables off # chkconfig --list iptables iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off
3. To stop iptables on current running system:
# service iptables stop Flushing firewall rules: [ OK ] Setting chains to policy ACCEPT: filter [ OK ] Unloading iptables modules: [ OK ]
4. If you still need the iptables to control other interfaces you should flush all the rules:
# iptables --flush
and make sure you only create rules that do not affect the cluster interconnect interfaces, protocols and network address space. Repeat these steps on all RAC nodes.