Configuring MailTo notifications
Most system administrators like to be notified when a resource migrates to a different node since this is typically an indication of trouble with a node. One of the easiest ways to get notifications of this sort is to add a MailTo resource to a resource group. Whenever the resource group is started or stopped, the MailTo resource will send an email to the configured address.
For example, to add a Mail To resource to the important group resource group, sending emails to admin@exarnple.com with a subject prefix of importantgroup notification, the following command can be used:
# pcs resource create importantgroup-mailto MailTo \ > email=admin@example.com \ > subject="importantgroup notification" \ > --group importantgroup
The MailTo resource agents send messages using the mailx command. It is the responsibility of the system administrator to make sure that the mailx package is installed, and that outgoing SMTP connections are allowed from the cluster nodes.
Since mailx uses the local MTA for sending messages, the local MTA (postfix, for example) will have to be configured to use a smarthost where necessary.
Cluster monitoring with ClusterMon
For more detailed monitoring, the Red Hat High Availability Add-on ships with the crm_mon utility, and a resource agent built around that utility: ClusterMon.
crm_mon can generate cluster status reports in HTML, XML, HTML ready for use as a CGI script, Nagios-friendly formats, and more. It can be used in one-shot mode, where it generates one iteration of output and quits, or it can remain active, updating a status file on disk and optionally calling an external program whenever the status changes.
When using the ClusterMon resource agent, a number of important options are available:
| Option | Usage |
|---|---|
| user | The user to run the crm_mon utility. Defaults to root. |
| update | The interval in seconds between updates of the HTML file on disk. Defaults to 15 seconds. |
| extra_options | Any extra command-line options to pass to crrn_mon. See man crm_mon for possibilities. |
| htrnlfile | The full path of the desired output file. Defaults to /tmp/ClusterMon_[RESOURCENAME].html. |
One of the problems an administrator would face when creating a single ClusterMon resource is that it would only report the status of the cluster as seen by one node. A solution would be to create a single ClusterMon instance for each node and use resource constraints to fix each instance to a specific node. Not an ideal situation, but an easier solution does exist.
Resource clones
Whenever it is desirable to have a copy of a resource to run on each node, resource clones can be used. Whenever a resource is cloned, a copy of that resource will be started on every node in the cluster. To create a cloned resource, append the –clone option to the end of the pcs resource create command when creating a new resource.
To clone an existing resource, the command pcs resource clone [RESOURCE] command can be used. Existing clones can be removed with the pcs resource unclone command.
Using an external notification agent with ClusterMon
Configuring email or SNMP notifications with ClusterMon requires an external program or script that performs the actual notification. These scripts can then be called using the -E [SCRIPT] option in the extra_options field for ClusterMon.
Additionally, the -e [RECIPIENT] option can also be used to specify an intended recipient. This options is passed on to the script defined with -E.
Parameters are passed into the external agent script using environment variables called CRM_notify_ *.
The following is an example script that can be used to send an email on eve,y resource change. A script used in production will need to more aggressively filter which message to pass on, so as to not spam the notification mailbox.
#!/bin/bash
MAILTO=${CRM_notify_recipient:-root@localhost}
mutt -s "Cluster event from node ${CRM_notify_node}" ${MAILTO} << EOF
A cluster event was triggered by ${CRM_notify_rsc}
on ${CRM_notify_node}. This event was triggered
by a ${CRM_notify_task} event.
---
This mail has been generated automatically
EOF
With this script stored as /usr/local/bin/myagent.sh, a ClusterMon resource that uses this script can be configured as follows:
# pcs resource create myagent ClusterMon \ > extra_options="-E /usr/local/bin/myagent.sh \ > -e admin@example.com" \ > --clone