Hortonworks HDP

Note: This is post is part of the HDPCA exam objective series Starting from Haddop 2.4, HDFS can be configured with ACLs. These ACLs work very much the same way as extended ACLs in a Unix environment. This allows files and directories in HDFS to have more permissions than the basic POSIX permissions. To verify if you have already set the value, go to services > HDFS > config and search for the property "dfs.namenode.acls.enabled" in the search box. Enabling HDFS ACLs To use HDFS … [Read more...] about HDPCA Exam Objective – Configure HDFS ACLs

Note: This is post is part of the HDPCA exam objective series Apache Ranger is a security framework which lets you define the policies to control the data access in Hadoop. It provides a web-based console that can be used by the system administrators of the Hadoop cluster to define and activate the access policies. Apache Ranger understands how different tools interact with Hadoop and lets you define permissions accordingly. For example, for Hive data, you can define whether a user is allowed … [Read more...] about HDPCA Exam Objective – Install and configure Ranger

Note: This is post is part of the HDPCA exam objective series We mentioned earlier that HDFS replication alone is not a suitable backup strategy. In the Hadoop 2 filesystem, snapshots have been added, which brings another level of data protection to HDFS. As changes to the filesystem are made, any change that would affect the snapshot is treated specially. For example, if a file that exists in the snapshot is deleted then, even though it will be removed from the current state of the … [Read more...] about HDPCA Exam Objective – Recover a snapshot

Note: This is post is part of the HDPCA exam objective series HDFS Sanpshot In spite of having a replication factor of 3, there are chances of data loss in the Hadoop cluster due to human error or corruptions. Hadoop 2.0 added the capability of taking a snapshot (read-only copy and copy-on-write) of the filesystem (data blocks) stored on the data nodes. Using Snapshots, you can take a copy of directories seamlessly using the NameNode's metadata of the data blocks. Snapshot creation is … [Read more...] about HDPCA Exam Objective – Create a snapshot of an HDFS directory

Note: This is post is part of the HDPCA exam objective series When you install an HDP cluster using ambari, it selects an optimum value of the configuration parameter for each of the services in the cluster. But you may have a requirement to change these default values. You can use ambari to change these settings pretty easily. Viewing the current Service Config Properties To view the config properties of a particular service (HDFS for example), goto Services > HDFS > Configs. You … [Read more...] about HDPCA Exam Objective – Change the configuration of a service using Ambari

Note: This is post is part of the HDPCA exam objective series Ambari has made the job of hadoop admin much easier. With ambari you can start/stop/restart any service with a click of a button. With ambari you can see exactly what is happening behind the scenes when you start/stop a service. You can also troubleshoot a failed start/stop of a service by viewing the logs from the ambari dashboard itself. Managing Services using Amabari You can manage the services currently configured in you … [Read more...] about HDPCA Exam Objective – Restart an HDP service

Note: This is post is part of the HDPCA exam objective series In a Hadoop cluster, if the RM goes offline for any reason, all the jobs on the cluster will fail. In production, there will be critical jobs that might be running for a long time and it does not make sense to start them again due to the failure of RM. HA for ResourceManager was introduced in Hadoop 2.4 and it supports both manual and automatic failover. As the exam objective in HDPCA exam, we will see how to configure the … [Read more...] about HDPCA Exam Objective – Configure ResourceManager HA

Note: This is post is part of the HDPCA exam objective series NameNode High Availability Concepts In case of either accidental failures or regular maintenance of NameNode, the cluster will become unavailable. This is a big problem for a production Hadoop cluster. - In the cluster, there are two separate machines—the active state NameNode and standby state NameNode. - At any point in time, exactly one of the NameNodes is in an active state, and the other is in a standby state. - The … [Read more...] about HDPCA Exam Objective – Configure NameNode HA

Note: This is post is part of the HDPCA exam objective series Adding a new node to a Live cluster is almost similar to installing a new HDP cluster with ambari. To add a new host you must enable passwordless ssh from the ambari-server to the new host. How to setup passwordless SSH login in Linux Pre-requisites 1. Make sure you can do a passwordless ssh from ambari-server to the new host to be added. 2. You have generated the private ssh key in the ambari-server (this is already … [Read more...] about HDPCA Exam Objective – Add a new node to an existing cluster

Note: This is post is part of the HDPCA exam objective series HDFS HDFS (Hadoop Distributed File System) is the storage layer of the Hadoop cluster which stores the data. It is a distributed filesystem and it is very important for a Hadoop admin to know how to configure and manager HDFS inside out. For the purpose of the exam, we will see few of the basic commands to administer HDFS. This includes creating directories, managing owner & permissions, loading data into HDFS and copy data from … [Read more...] about HDPCA Exam Objective – Create a home directory for a user and configure permissions