The Red Hat High Availability Add-on provides different ways for a system administrator to control membership of cluster nodes:
- Starting and stopping the cluster services control whether a cluster node is participating in the cluster at runtime.
- Enabling and disabling the cluster services control whether a cluster node automatically starts the cluster services and joins the cluster when it boots.
- Adding and removing a cluster node from the cluster permanently changes whether the node is a member of the cluster.
- The standby and unstandby modes control whether a cluster node is allowed to host resources in the cluster.
Managing cluster services
The systemd-managed cluster services corosync and pacemaker need to be running on the cluster node to join the cluster the node is part of. On an authenticated cluster node that runs pcsd, the cluster services can be controlled with pcs.
Starting and stopping cluster services
The pcs cluster start and pcs cluster stop commands start and stop the cluster services, respectively. The commands support starting or stopping cluster services on the local node, a remote node that is provided as a parameter, or on all nodes with the –all switch. To start the cluster services on the local node, execute:
# pcs cluster start
To stop cluster services on the remote node, geeklab.example.com, execute:
# pcs cluster stop geeklab.example.com
The pcs cluster start –all and pcs cluster stop –all commands start or stop all nodes that are in the same cluster as the local node. To stop the cluster services on all nodes in the same cluster as the local node, execute:
# pcs cluster stop --all
Enable and disable cluster services
The pcs cluster enable and pcs cluster disable commands enable or disable the systemd-managed cluster services corosync and pacemaker from starting or stopping automatically. when a cluster node boots up. A node automatically joins the cluster each time it is booted up if the cluster services are enabled on the node. Both commands control the services on the local node, a remote node that is provided as a parameter, or on all nodes in the same cluster as the local node with the –all switch. To enable automatic startup of the cluster services on the local node, execute:
# pcs cluster enable
To disable cluster services on the remote node, geeklab.example.com, execute:
# pcs cluster disable geeklab.exarnple.com
The pcs cluster enable –all and pcs cluster disable –all commands enable and disable the cluster services on all nodes respectively that are in the same cluster as the local node. To disable the cluster services on all nodes in the same cluster as the local node, execute:
# pcs cluster disable --all
Adding and removing a cluster node
The Red Hat High Availability Add-on allows for adding and removing cluster nodes on the fly. This allows for extending a cluster or replacing a cluster node without service downtime on the cluster. The pcs configuration utility allows a system administrator to add or remove cluster nodes. With pcs cluster node add node.fqdn, a new node is added to the cluster, and with pcs cluster node remove node.fqdn, a node is permanently removed.
Adding a new node to the cluster
Adding a new node to an existing cluster requires multiple configuration steps:
1. The new node that will join the cluster has to be configured to fulfill the following requirements:
- The firewall on the new cluster node is configured to allow cluster communication.
- The pcs and fence-agents-all packages and their dependencies are installed. In the classroom environment, the package fence-agents-rht needs to be used in place of the fence-agents-all package.
- The pcsd service is started and enabled.
- The password of the hacluster user is changed to match the password of that user on the existing nodes.
- The new cluster node was authenticated from all existing cluster members. This requires running pcs cluster auth newnode.fqdn from an existing cluster node. Unless the –local is used, the authentication token will be distributed across all cluster nodes automatically.
2. Once configured, the new cluster node can be added as a cluster member to the existing cluster and then authenticated with the previously existing cluster nodes.
- To add the prepared cluster node to the existing cluster, execute:
# pcs cluster node add newnode.fqdn oldnode1.fqdn: Corosync updated oldnode2.fqdn: Corosync updated oldnode3.fqdn: Corosync updated newnode.fqdn: Succeeded
- The existing nodes need to be authorized from the new node. The pcs cluster auth command authorizes all cluster members from the local node.
# pcs cluster auth Username: hacluster Password: redhat oldnode1.fqdn: Authorized oldnode2.fqdn: Authorized oldnode3.fqdn: Authorized newnode.fqdn: Already authorized
After the node was successfully added and authorized, a system administrator may enable and start the cluster services on the new node. Fencing must be configured and operational for the new cluster node before any services can be safely migrated to it.
Reviewing cluster status
For a system administrator, it is important to be able to retrieve the current status of the cluster, the cluster nodes, and the cluster resources.
A detailed overview of the cluster status, corosync status, configured resource groups, resources, and status of the cluster nodes is provided by pcs status.
The pcs status output can be limited with one of the following parameters:
|pcs status cluster||Show only information related to cluster status.|
|pcs status groups||Show only the configured resource groups and their resources.|
|pcs status resources||Show only the status of the resource groups and their individual resources in the cluster.|
|pcs status nodes||Show only the status of the configured cluster nodes.|
|pcs status corosync||Show only the status of corosync.|
|pcs status pcsd||Show only the status of pcsd on all configured cluster nodes.|
The pcs status command is a powerful utility that enables a system administrator to determine the status of cluster node membership and display all information related to the cluster and cluster nodes:
# pcs status Cluster name: cluster Last updated: Fri Sep 26 05:47:40 2014 Last change: Wed Sep 24 06:19:49 2014 via cibadmin on nodea.private.example.com Stack: corosync Current DC: nodeb.private.example.com (2) - partition with quorum Version: 1.1.10-29.el7-368c726 4 Nodes configured 6 Resources configured Node nodeb.private.example.com (2): standby Online: [ nodea.private.example.com nodec.private.example.com] OFFLINE: [ noded.private.example.com] Full list of resources: fence_nodea (stonith:fence_rht): Started nodea.private.example.com fence_nodeb (stonith:fence_rht}: Started nodeb.private.example.com fence_nodec (stonith:fence_rht): Started nodec.private.example.com fence_noded (stonith:fence_rht): Started noded.private.example.com Resource Group: web floatingip (ocf::heartbeat:IPaddr2): Started nodea.private.example.com website (ocf::heartbeat:apache) : Started nodea.private.example.com PCSD Status: nodea.private.example.com: Online nodeb.private.example.com: Online nodec.private.example.com: Online noded.private.example.com: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
In the previous example, the cluster consists of four cluster nodes with the following status:
- The cluster node nodeb.private.example.com is in standby mode.
- The cluster nodes nodea.private.example.com. and nodec.private.example.com are fully operational and participating in the cluster, and therefore marked as Online.
- The cluster node noded. private. example. com is running because the status of pcsd is Online. The cluster node is marked as OFFLINE because the cluster services have either been stopped on this cluster node or failed to communicate with the quorate part of the cluster.
Prohibiting a cluster node from hosting resources in pacemaker cluster (putting a node into standby mode)