What is quorum?
In order for a cluster to work as expected, the nodes must be in agreement on certain facts, such as which machines are currently cluster members, where services are running, and which machines are using which resources.
The method in which this is implemented in the Red Hat High Availability Add-on is by the use of a majority voting scheme. Every cluster node casts one vote if it successfully joins the corosync network communication and is able to communicate with the other nodes that are already participating in the cluster.
The cluster is operational if more than half of all possible votes are successfully cast. The minimum number of votes needed to achieve more than half of the votes is called the quorum. If quorum is achieved, the cluster is considered quorate. A cluster loses quorum if half of the nodes or more cannot communicate with each other.
When a cluster gets started, all cluster nodes try to communicate with each other and aim to achieve quorum. As soon as a majority is formed, there is a quorate cluster. All other nodes that have not successfully joined the quorate cluster get fenced by a node that has quorum. If a node that is part of the quorate cluster is not able to communicate with the cluster anymore, the node gets fenced.
Why is quorum calculation required?
Quorum is required in situations where some nodes in a cluster cannot communicate with certain other nodes. The following illustration shows a five-node cluster consisting of nodes A, B, c, D, and E with a service that uses an ext4 file system on shared storage. This service is currently running on node A.
The nodes D and Eget split off from the main private network and cannot communicate with the nodes A, B, and c. Without quorum, those two nodes will decide that A, B, and C have all failed and need to be fenced (remotely powered off) so that their resources can be recovered. In this way, quorum acts as an important gate prior to fencing.
Were the cluster to be without a fencing device, nodes D and E would immediately begin recovering resources, resulting in the same ext4 file system being mounted in two places at once, which is not a healthy situation. This situation, where two halves of a cluster operate independently from each other, is referred to as a split-brain.
Split-brain is a particular concern in two-node clusters since if either node fails, the other node does not consist of a majority of the nodes in the cluster. Special steps need to be taken to allow two-node cluster operation and still avoid split-brain.”
Assuming that all nodes in this example have one vote, this situation cannot arise, since node D and node E only have two votes together, which is not more than half of the total votes (five). This would result in node D and node E ceasing to function until at least one other vote was added. Nodes A, B, and c, on the other hand, remain active providing service, since they have three votes combined, which is more than half of the total votes.
Quorum is calculated and managed in the Red Hat High Availability Add-on by the corosync component votequorum. The votequorum component uses two values to calculate if the cluster is quorate:
- Expected votes: The number of votes expected if all cluster nodes are fully operational and communicating with each other.
- Total votes: The number of votes currently present. This can be lower than the number of expected votes if some nodes are not up or not communicating with the cluster.
The number of votes required to achieve quorum is based on Expected Votes. The following formula shows how many votes are required for quorum. In this calculation, floor() means to always round down.
Quorum = floor(expected votes + 2 + 1)
Quorum calculation example
In the following example, a cluster with three nodes is assumed. A three-node cluster has an Expected Votes count of three.
Quorum= floor(expected votes + 2 + 1) Quorum = floor(3 + 2 + 1) Quorum = floor(1.5 + 1) Quorum = floor(2.5) Quorum = 2
In the three-node cluster, there needs to be a minimum of two nodes running to achieve quorum.
In the following example, a cluster with four nodes is assumed. A four-node cluster has an Expected Votes count of four.
Quorum = floor(expected votes + 2 + 1} Quorum = floor(4 + 2 + 1) Quorum = floor(2 + 1} Quorum = floor(3) Quorum = 3
In the four-node cluster, there needs to be a minimum of three nodes running to achieve quorum.
Displaying quorum status
The Red Hat High Availability Add-on provides a comprehensive utility to display the current state of quorum in a cluster: corosync-quorumtool. The corosync-quorumtool provides an overview of quorum-related information, such as Total Votes and Expected Votes, and shows if the cluster is quorate at a glance.
# corosync-quorumtool Quorum information ------------------ Date: Thu Sep 25 05:02:00 2014 Quorum provider: corosync_votequorum Nodes: 4  Node ID: 1  Ring ID: 4648 Quorate: Yes  Votequorum information ---------------------- Expected votes: 4  Highest expected: 4  Total votes: 4  Quorum: 3  Flags: Quorate  Membership information ---------------------- Nodeid Votes Name 1 1 nodea.private.example.com (local) 2 1 nodeb.private.example.com 3 1 nodec.private.example.com 4 1 noded.private.example:com
- The Nodes entry provides the node count of the nodes that belong to the cluster.
- The Node ID entry shows the ID of the node on which corosync-quorumtool was executed.
- The Quorate entry shows if the cluster is quorate.
- The Expected votes entry shows the number of votes that are present if all configured cluster members are active in the cluster.
- The Highest expected entry shows the largest count of expected votes seen in the cluster.
- The Total votes entry shows the count of votes currently present in the cluster.
- The Quorum entry shows how many votes need to be present at minimum so the cluster stays quorate.
- The Flags entry shows any quorum-related properties that are currently set on the cluster. If the cluster is quorate, then the Quorate property will show. Additional special features will also display in this field when set, such as LastManstanding or WaitForAll.
Managing Quorum Calculations in a Pacemaker Cluster