An overview of web user interfaces
Similar to web interfaces available in the Hadoop 1.x version, the Hadoop 2.x version has web user interfaces for Hadoop services. Instead of the JobTracker service web UI, the web UI for ResourceManager is used to monitor applications and resources. A detailed explanation regarding the configurations related to the web UI's is given in the next chapter.
The following screenshot refers to the web-interface of Hadoop HDFS. Through the HDFS interface, you can:
- Explore the HDFS metrics
- View the list of DataNodes connected to the NameNode
- Browse the file system, and so on
The following screenshot refers to the web-interface of YARN ResourceManager. It shows:
- A summarized view of the cluster resource capabilities
- A list of applications related to the cluster
- A list of NodeManager nodes connected
- The scheduler details, and so on
You can refer to the .pid files for the Hadoop daemons that are created in the /tmp folder of your node. These files contain the current process ID for each daemon:
Testhdfs dfs -mkdir –p /user/hduser/input hdfs dfs -copyFromLocal /home/hduser/hadoop-2.5.1/etc/hadoop/* /user/hduser/input
To verify that your files are successfully copied into HDFS, execute the following command or browse to the HDFS filesystem through the HDFS web UI at http://localhost:50070/explorer.html:
hdfs dfs -ls /user/hduser/input
Run a sample application
Hadoop uses the yarn command to submit an application to the cluster. You can find the yarn command in the /home/hduser/hadoop-2.5.1/bin folder and the examples jar file containing sample MapReduce applications in the /home/hduser/hadoop-2.5.1/share/hadoop/mapreduce folder. Other than the examples jar path, specify three arguments to the command—the operation to perform (word count), the HDFS input directory, and an HDFS directory to store the output of the MapReduce application. Ensure that the HDFS output directory does not exist.
Execute the following command to run a word count example:
yarn jar /home/hduser/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar wordcount /user/hduser/input /user/hduser/output
Monitor the progress of the application through the YARN web user-interface at http://localhost:8088/cluster/apps