In this post, we will configure a simple 2 node cluster with cluster nodes as nodea and nodeb. The cluster name will be geeklab.
Fencing for this cluster should be handled using the fencing daemon running on classroom.example.com in combination with the fence_rht fencing agent. Plug names for this fence device correspond to host names on the private. example. com network. In order to prevent fence races, nodeb should have a 10-second delay when being fenced.
To speed up resource recovery when the cluster is started, the cluster should not wait for all nodes to join before beginning resource recovery by fencing.
Installing the required packages
1. On both of your nodes, install the fence-agents-all, fence-agents-rht, and pcs packages.
# yum -Y install pcs fence-agents-all fence-agents-rht
2. On both nodes, allow cluster traffic to pass through the firewall.
# firewall-cmd --permanent --add-service=high-availability # firewall-cmd --reload
3. On both nodes, enable and start the pcsd service.
systemctl enable pcsd # systemctl start pcsd
4. On both nodes, set the hacluster password to root123.
# echo root123 I passwd --stdin hacluster
5. From your nodea machine, authenticate your nodea. private. example. com and nodeb. private. example. com machines for pcs.
# pcs cluster auth -u hacluster -p root123 \ > nodea.private.example.com \ > nodeb.private.example.com
6. Create a new two-node cluster called geeklab using your nodea.private.example.com and nodeb.private.example.com machines. Start and enable the cluster on both nodes:
# pcs cluster setup --start --enable \ > --name geeklab \ > nodea.private.example.com \ > nodeb.private.example.com
7. Verify the cluster is active.
# pcs status
8. When creating a new two-node cluster with pcs, the special two_node mode will automatically be enabled in corosync.conf. If extra nodes later join the cluster, this option will automatically be disabled again.
After creating a two-node cluster, the quorum block in /etc/corosync/corosync.conf will look like this:
quorum {
provider: corosync_votequorum
two_node: 1
}
Disabling wait_for _all
Even though the wait_for_all option is not specified explicitly, it is turned on automatically by the two_node option. To disable the wait_for _all option when creating a new cluster, the –wait_for_all=0 option can be added, like this:
# pcs cluster setup --start --enable \ > --name twonodecluster \ > --wait_for_all=e \ > nodea.private.example.com \ > nodeb.private.example.com
To disable the wait_for _all option on an already existing cluster, the following procedure can be used:
1. Stop the running cluster.
# pc cluster stop --all
2. Add a line wait_for_all: 0 to the quorum block in /etc/corosync/corosync.conf:
quorum {
provider: corosync_votequorum
two_node: 1
wait_for _all: 0
}
3. Sync corosync.conf to the other cluster node.
# pcs cluster sync
4. Start the cluster on all nodes.
# pc cluster start --all
Configuring delayed fencing
To prevent fence races, one of the two-nodes should be configured for delayed fencing. This will help in keeping the cluster stable, but it will not prevent reboot-then-fence cycles if communications between the cluster nodes has been compromised because of an external influence like a broken switch.
With delayed fencing, one of the fencing devices is configured with a delay=N option, where N is a timeout in seconds. When the fence device is called, it will log the beginning of the fencing operation immediately, but then wait for N seconds before actually starting the fencing operation. This gives the node that would be fenced by this device time to fence the other node before fencing can begin, effectively saving the node for which the delayed fencing device is configured from being fenced, and making it “win” the fence race.
Creating a delayed fence device
The procedure for creating a fence device with delayed fencing is exactly the same as the procedure for creating a regular fence device, but with an added delay=N option. For example, to create a fence device called fence_nodea_delayed, targeting nodea, and with a delay of 10 seconds, the following command can be used:
– For nodea cluster node:
# pcs stonith create fence_nodea_delayed \ > fence_rht ipaddr="classroom.example.com" \ > port="nodea.private.example.com" \ > pcmk_host_list="nodea.private.example.com" \ > delay=10
– For nodeb cluster node:
# pcs stonith create fence_nodeb_delayed \ > fence_rht ipaddr="classroom.example.com" \ > port="nodeb.private.example.com" \ > pcmk_host_list="nodeb.private.example.com"
If a fence device is created for the second node without a delay, this will prevent nodea from being fenced since it has a 10-second window to fence the second node. If the cluster communication network is still up, and nodea fails for another reason it will still be fenced, but with a 10-second delay.
Updating an existing fence device for delayed fencing
An existing fence device can be updated for delayed fencing as well, using the pcs stonith update command. Ensure that the fence device is only used for one host. then add the delay option.
For example, to update the fence device fence_mynodea to use a five-second delay, the following command can be used:
# pcs stonith update fence_mynodea delay=5
Delayed fencing and fencing levels
Since fencing levels are attempted sequentially, and the fence devices inside a fencing level are executed in a serial fashion as well, only the first fence device in the first fencing level needs a delay.
While extra delays can be added, they will slow down the overall fencing process without providing additional protection against fence races. It can even increase the risk of a partially fenced machine, since it provides a bigger window of opportunity for the second node to put in some fencing actions.