Monitoring the health of Hadoop cluster is an important aspect of Hadoop administration. Ambari provides us the centralized management of health alerts and checks for the services in your cluster. You can set thresholds and can disable/enable alerts using the ambari UI.
You can view all the alerts definations from the alerts page in the ambari dashboard.
The alerts which have breached the threshold will be shown in red color. The Alerts which are under the threshold value are shown in green color.
There are many filters available on the alerts page to filter out using some condition. You can filter alerts based on:
1. State (enabled/disabled)
2. Service (alerts for a specific service like YARN, HDFS)
3. Status (OK, WARNING, CRITICAL, UNKNOWN, NONE)
4. Last status changed.
5. Alert Definition Name.
You can also view the alert definitions with respect to individual service groups.
Disable/Enable an Alert
An alert can be easily enabled or disabled from the Alerts page in Ambari. In the example shown below, we are disabling the “Host Disk Usage” alert.
Manage alert group
Click on “Manage Alert Group” from “Actions” drop-down to view the options to manage alert groups. You can manage alert groups for each service in this dialog. View the list of alert groups and the alert definitions configured in them. You can also add/remove alert definitions, and pick notification for that alert group.
Each service will have a default alert group which can not be deleted.
We can add a new alert group click on the “+” icon at the bottom and provide the name of the alert group.
We can add the alert definitions to this new alert group.
We have added the 2 sample alerts to the new alert group as shown below.
Create Alert Notification
Creating alerts is not sufficient if the Hadoop admin does not get the email alerts from the ambari. Let’s see how we can create a notification in ambari. Goto the alerts page and click “Manage Notifications” under the “Actions” drop-down.
Here, Create the new Notification using the “+” icon at the bottom.
Provide the information such as Name, Alert groups, Severity and email address to receive the notification on this page.
Click save and verify if the Notification is created.
Manage Alert Settings (Alert Check Count)
Set the number of alert checks to perform before dispatching a notification. If during an alert check a state change occurs, Ambari will attempt to check this number of times before dispatching a notification. Increase this number if your environment experiences transient issues resulting in false alerts.
