In Linux, the sudo command allows a user to execute a command as a superuser, or in other words an administrator of a windows system. The file /etc/sudoers contains a list of users who have the sudo permission. If you need to change any of the system properties or access any system file, you need to add sudo in the beginning of the command. To read more about the sudo command, you can refer to the blog at http://www.tutorialspoint.com/unix_commands/sudo.htm.

Nano is one of the editor tools for Linux. Its ease of use and simplicity allow beginners to handle files easily. To read more about the nano editor, you can refer to the documentation at http://www.nano-editor.org/dist/v2.0/nano.html.

The alternate to the nano editor is the default vi editor.

When you edit any of the environment setting files such as /etc/environment or ~/.bashrc, you need to refresh the file to apply the changes made in the file without restarting the system. To read more about the source command, you can refer to the blog at http://bash.cyberciti.biz/guide/Source_command.

Jps is a Java command used to list the Java processes running on a system. The output of the command contains the process ID and process name for all of the Java processes. Before using the jps command, you need to make sure that the bin directory of your JAVA_HOME command is set in the PATH variable for the user.

A sample output is as follows:

hduser@host:~$ jps
6690 Jps
2071 ResourceManager
2471 NameNode

To read more about the jps command, you can refer to the Oracle documentation at http://docs.oracle.com/javase/7/docs/technotes/tools/share/jps.html.

The netstat command is a utility to list the active ports on a system. It checks for TCP and UDP connections. This command will be helpful to get the list of ports being used by a process. To read more about the netstat command and its options, you can refer to the blog at http://www.c-jump.com/CIS24/Slides/Networking/html_utils/netstat.html.

You can use the netstat command with the grep command to get filtered results for a particular process:

netstat -nlp | grep <PID>

Most of the Linux commands have their documentations and user manuals that are also known as man pages. The man command is used to format and view these man pages through the command line interface. The basic syntax of the man command is as follows:

To read more about the man command, you can refer to the wiki page at http://en.wikipedia.org/wiki/Man_page

As mentioned in the software requirements for a cluster, all the nodes across the cluster must have Sun Java 1.6 or above and the SSH service installed. The Java version and JAVA_HOME should be consistent across all the nodes. If you want to read more regarding the Java compatibility with Hadoop, you can browse to a page on wiki at http://wiki.apache.org/hadoop/HadoopJavaVersions.

To install and configure Java on Ubuntu, you can refer to the blog at http://www.protechskills.com/linux/unix-commands/install-java-in-linux.

In the Hadoop cluster node installation, the Hadoop daemons run on multiple systems. The slave nodes run the DataNode and NodeManager services. All the nodes in a cluster must have a common user and a group. It is recommended to create a dedicated user for the Hadoop cluster on all the nodes of your cluster.

To create a user on Ubuntu, you can refer to the Ubuntu documentation at http://manpages.ubuntu.com/manpages/jaunty/man8/useradd.8.html.

Here is a sample command to create a new user hduser on Ubuntu:

sudo usersadd –m hduser

After creating a new user, you also need to set a password for the new user. Execute the following command to set a password for the newly created user hduser:

sudo passwd hduser

Hadoop daemons use a few ports for internal and client communication. The cluster administrator has an option to either disable the firewall of the nodes or allow traffic on the ports required by Hadoop. Hadoop has a list of default ports, but you can configure them as per your need.

To disable the firewall in Ubuntu, execute the following command:

sudo ufw disable

A Hadoop node is identified through its hostname. All the nodes in the cluster must have a unique hostname and IP address. Each Hadoop node should be able to resolve the hostname of the other nodes in the cluster.

If you are not using a DHCP server that manages your DNS hostname resolution, then you need to configure the /etc/hosts file on all the nodes. The /etc/hosts file of a system contains the IP addresses of the nodes specified with their hostname. You need to prepend the file with the IP address and hostname mapping. You can use the nano or vi editor with the sudo option to edit the file contents. Assuming the hostname of your nodes is master, slave1, slave2, and so on; the contents of the file will look similar to the following:

192.168.56.100 master
192.168.56.101 slave1
192.168.56.102 slave2
192.168.56.103 slave3

To view the system hostname, you can execute the hostname command.

hostname

To modify the system hostname in Linux, you can refer to the blogs here:

CentOS: https://www.centosblog.com/how-to-change-hostname-on-centos-linux/

Ubuntu: http://askubuntu.com/questions/87665/how-do-i-change-the-hostname-without-a-restart

After editing the file, you need to either restart your system network settings or reboot your system.

To restart networking on Ubuntu, execute the following command:

sudo /etc/init.d/networking restart

To verify that configuration is working properly, execute the following command:

ping slave1

To stop the command, press ctrl + c. The output of the command will look like this:

PING slave1 (slave1) 56(84) bytes of data.
64 bytes from slave1: icmp_req=1 ttl=64 time=0.025 ms
64 bytes from slave1: icmp_req=2 ttl=64 time=0.024 ms

The OpenSSH server and client packages should already be installed and the sshd service should be running on all the nodes. By default, the sshd service uses port 22.

To install these packages on Ubuntu, execute the following commands:

sudo apt-get install openssh-client
sudo apt-get install openssh-server

The Hadoop master node remotely manages all the Hadoop daemons running across the cluster. The master node creates a secure connection through SSH using the dedicated user group for the cluster. It is recommended that you allow the master node to create an SSH connection without a password. You need to configure a passwordless SSH from the master node to all slave nodes.

First you need to create SSH keys for the master node, then share the master's public key with the target slave node using the ssh-copy-id command.

Assuming that the user for the Hadoop-YARN cluster is hduser, the ssh-copy-id command will append the contents of the master node's public key file, /home/hduser/.ssh/ id_dsa.pub, to the /home/hduser/.ssh/authorized_keys file on the slave node.

To install the SSH service and configure a passwordless SSH on Ubuntu, you can refer to the Ubuntu documentation at https://help.ubuntu.com/lts/serverguide/openssh-server.html.

To verify that the sshd service is running, execute the following netstat command:

sudo netstat -nlp | grep sshd

The output will contain the service details if the service is running:

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 670/sshd

To verify the passwordless SSH connection, execute the ssh command from the master node and observe that the command will not prompt for a password now:

ssh hduser@slave1