We have seen in the last posts, how to install ambari-server and ambari-agent. Let us now see how we can install the HDP cluster using the ambari install wizard. We will be having the cluster topology as shown below.
The ambari-server install wizard will go ahead and install ambari-agent in all the nodes you mention during installation and configure the services.
1. Login to the ambari-server and launching install wizard
Login to the ambari-server using the URL : http://[ambari-server IP address]:8080. The default credentials will be admin/admin if you have not changed them. You would see the “launch install wizard” upon logging in.
2. Configuring Cluster Name
On the next screen, you can configure the cluster name without any spaces or special characters. I will name the cluster as “geeklab”.
3. Selecting HDP stack version
On the next screen, you can select the HDP stack version. The ambari server will also show the components that will get installed in the particular HDP stack. We will go with the HDP stack version 2.6.5.0
4. Install options
On the next screen, you will see the install options such as – Target Hosts, Host Registration Information etc. Before we can add the target hosts here make sure you have added the entries of all the nodes in the /etc/hosts file of ambari-server and all the nodes in the cluster. This is not required if you have setup the host FQDN in a centralized DNS server.
# cat /etc/hosts 192.168.1.6 ambari-server.localdomain ambari-server 192.168.1.2 nn1.localdomain nn1 192.168.1.3 dn1.localdomain dn1 192.168.1.4 dn2.localdomain dn2 192.168.1.5 dn3.localdomain dn3
Make sure you can see the FQDN in the command “hostname -f” on each node. For example:
[root@nn1 ~]# hostname -f nn1.localdomain
Configuring Passwordless ssh from ambari-server to all cluster nodes
We will use the automatic installation method which requires ambari-server to login to all the cluster nodes without a password. Follow the post below to configure passwordless ssh:
You will also have to generate the RSA private key if its not present already.
[root@ambari-server ~]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:XAHCgqR4yJsA4SANRnKDdb0DRHnbb8EWSn9wO1p6uwE root@ambari-server.localdomain The key's randomart image is: +---[RSA 2048]----+ |OX=+++. ... | |@+oo+ +.. o.. | |=o. + = +.+ . | |..o +.o.= = | | o .SoE* . | | =.. | | . ... | | .. | | .. | +----[SHA256]-----+
Place the RSA private key in the “install options” page and proceed for installation. On the next screen, you can check the progress of the installation and also view the log files for each node.
If everything goes alright, you would see the successful registration of all 4 hosts.
You may get some warning here. For example, I forgot to disable the firewalld service on all the datanodes.
You can correct the issues and re-run the prechecks again.
5. Choose services
On the next screen, you can choose the services that you want to run on your HDP cluster. For now, I will only install 4 important services i.e.
1. HDFS
2. YARN + MapReduce2
3. ZooKeeper
4. Ambari Metrics
You can enable a service later on after the installation as well.
6. Assign Masters
On the next screen, we will choose the master for the cluster of the primary/secondary namenodes for the cluster. We will choose nn1 as the primary and secondary namenode server. You should ideally have a separate secondary namenode in case of production environments. We will, later on, see how we can add a secondary namenode as well.
7. Assign slave and clients
On this screen, you would assign the slave components (DataNodes, NodeManagers, and RegionServers) to appropriate hosts in your cluster. I have kept all the settings as default except for the assignment of datanodes for the 3 nodes dn1,dn2, and dn3. Also, nn1 is made as the only client in the cluster.
8. Customize Services
Here ambari will show us the components with a red colored number which require our input. Such as in “SmartSense” section we would need to put the password for user ‘admin’ etc. Specify the required details like passwords etc for these components.
The ambari-server would also show you some recommendations like the DataNode maximum Java heap size should not be greater than 0.968GB. You can ignore these recommendations for our test setup and continue.
9. Review the configuration
On the next screen, you can review your configuration. You can modify any settings if required here.
10. Install, Start and Test
Once you have reviewed and started the deployment, the services will be deployed one by one. The service will be started and tested for proper functioning by the ambari server.
The installation will take a lot of time if you have selected more components to install. You would see that the installation is complete without any warnings.
11. Summary
You can review the summary page post instllation.
Click “complete” and review the cluster setup. The dashboard will show the number of nodes and the services that are currently configured on the cluster.