Installing and setting HDFS cluster through Ambari

• Ambari provides a simple and attractive Web UI which helps in automating the cluster activities
• It simplifies the installation, management and configurations of all the services
• It simplifies the security and administration of all the services
• It monitors the cluster and provides a full visibility of the cluster

Minimum Requirement

Memory: Minimum 1GB (1024 MB) free for up to 10 hosts
Disk Size: Minimum 10GB
Software Requirement: Python 2.7.x and JDK 1.8.x

Note: If JDK is not installed then Ambari takes care of installing it while installing itself.

Installing Ambari on Ubuntu

Considering 2 hosts:
1.1.1.1 Acting as Ambari Server
2.2.2.2 Acting as Ambari Agent1
Make the following entry in /etc/hosts file of all the 2 hosts

• If you have not temporarily disabled SELinux, you may get a warning. Accept the default (y), and continue.
• Customize user account for ambari-server daemon prompt: ambari by default runs on root . If you want to configure it on root the select (n). If you want another user then select (y). If you select (y) then provide a username.
• Warning for disabling iptables: If you have not disabled it you may get a warning, so select (y).
• Select a JDK version. Select (1) to install JDK 1.8. If you select to choose the custom version of JDK then you must install JDK manually on all the servers and specify the Java_Home_Path.
• Note: Choosing option 1 would be good as it saves a lot of manual configurations and it makes the necessary configurations on its own related to Ambari.
• Accept the Oracle JDK license when prompted.
• To explicitly enable Ambari to install libraries related to LZO data compression, select (y). If you select (n) then Ambari will not install any libraries automatically on newly added hosts.
• Enter advanced database configuration: Select (n) to use the default, embedded PostgreSQL database. The default database name is Ambari. Its default username is ambari and password is bigdata. If you want to use an existing database of MySQL/MariaDB/Oracle/PostgreSQL the select (y)
• Proceed with configuring remote database connection properties [y/n]: y.
• Setup completes.

After the setup start the server using following command prompt command.
ambari-server start

Setup ambari-agent on both the servers

Run the following command on both server

apt-get install ambari-agent

Edit the ambari-agent.ini file on both the servers.

vi /etc/ambari-agent/conf/ambari-agent.ini
[server]
hostname=<your.ambari.server.hostname>
url_port=8440
secured_url_port=8441

On master-node change the value of hostname to localhost.
On node1 change the value of hostname to master-node.
Start the ambari-agent.
ambari-agent start

Ambari Server Web UI listens on 8080, it is accessible as:
http://<ambari-server-ip-address>:8080

It will ask for login username and password the default credentials are admin/admin.

Login Page

Launch the Ambari Cluster Install Wizard
From the login page you will be redirected to the Launch the Ambari Cluster Install Wizard page.

Launch Install Wizard

1. Using Public Repository: This will require internet connection to the server.
2. Using Local Repository: Requires software configured in a repository within a network.

Select Stack Version

If Ambari Server has access to the internet specific versions will be listed in the DropDown.
If you have a Version Definition File for a version that is not listed, you can click Add Version

You can edit the base URL of the repositories based on the OS.
BaseURL based on OS

1. In Target Host: Enter the list of hosts one per line.
2. If you want Ambari to automatically install the Ambari agent provide the SSH ppk or path or choose select file in Host Registration Information.
3. Provide username and password for the SSH key.
4. If you want to install Ambari Agent manually, select Perform Manual Registration: This step will require Ambari agent installed. We have already installed and setup ambari-agent above in the article.
   
   

   Install Option

5. Choose Register and Confirm.
   If you select to install Ambari Agent manually then it throws the below exception

Click OK.
If all the hosts are correctly configured the Confirm Host page will successfully register the hosts.

Confirm Hosts

It may show few warnings for host checks.
Warning in Host Health Check

The solutions for the warnings is given right there. If you resolve it then you can do a Rescan

Choose Services.

Based on the Stack chosen during the select stack step you are offered with the choice of services that you want to be installed through Ambari.

You can install/select the services now of add the services later as well.

Starting in Ambari 2.5, SmartSense deployment is mandatory. You cannot clear the option to install SmartSense using the Cluster Install wizard.

SmartSense Deployment Mandatory

1. Select the service you want to deploy and choose next.

Choose Services

After you select the services to be installed it asks us to verify the configurations and assign master and slaves.

If you select to create the HDFS cluster with YARN+MapReduce then it will ask the hostname for NodeName SecondaryNameNode and DataNode. It does the configurations according to the inputs provided by you.

The following are the snapshots that indicate the configuration details.

Step: Assign Masters: Asking the Secondary NameNode, NameNode Details

Step: Assign Slaves and Client

Step: Customize Service

Review the settings


Step: Install Start and Test


Summary of all the Services installed


Step: Monitoring


Logs


Starting and Stopping of Services