How to Install Hadoop on Windows and Linux - Step by Step Guide

Step 9: Downloading and Installing Java 8

• Click here to download the Java 8 Package. Save this file in your home directory
• Extract the Java tar file using the following command:

tar -xvf jdk-8u101-linux-i586.tar.gz

Step 10: Downloading and Installing Hadoop

• • Download a stable release packed as a zipped file from here and unpack it somewhere on your file system

• Extract the Hadoop file using the following command on the terminal:

tar -xvf hadoop-2.7.3.tar.gz

• • You will be directed to the following window:
    Step 11: Moving Hadoop to a Location
  • Use the following code to move your file to a particular location, here Hadoop:

    mv hadoop-2.7.3/home/intellipaaat/hadoop

    Note: The location of the file you want to change may differ. For demonstration purposes, I have used this location, and this will be the same throughout this tutorial. You can change it according to your choice.

• Here, Home will remain the same.
• Intellipaat is the user name I have used. You can change it according to your user name.
• Hadoop is the location where I want to save this file. You can change it as well if you want.

Step 12: Editing and Setting up HadoopFirst, you need to set the path in the ~/.bashrc file. You can set the path from the root user by using the command ~/.bashrc. Before you edit ~/.bashrc, you need to check your Java configurations.

Enter the command:

update-alternatives-config java

You will now see all the Java versions available on the machine. Here, since I have only one version of Java which is the latest one, it is shown below:

You can have multiple versions as well.

• Next, you need to select the version you want to work on. As you can see, there is a highlighted path in the above screenshot. Copy this path and place it in a gedit file. This path is just for being used in the upcoming steps
• Enter the selection number you have chosen. Here, I have chosen the number 1
• Now, open ~/.bashrc with the vi editor (the screen-oriented text editor in Linux)

Note: You have to become a root user first to be able to edit ~/.bashrc.

• • Enter the command: su
  • You will be prompted for the password. Enter your root password
    

• • When you get logged into your root user, enter the command: vi ~/.bashrc
    

• • The above command takes you to the vi editor, and you should be able to see the following screen:
    

• • To access this, press Insert on your keyboard, and then, start writing the following set of codes for setting a path for Java:

    fi
    #HADOOP VARIABLES START
    export JAVA_HOME= (path you copied in the previous step)
    export HADOOP_HOME=/home/(your username)/hadoop
    export PATH=$PATH:$HADOOP_INSTALL/bin
    export PATH=$PATH:$HADOOP_INSTALL/sbin
    export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
    export HADOOP_COMMON_HOME=$HADOOP_INSTALL
    export HADOOP_HDFS_HOME=$HADOOP_INSTALL
    export YARN_HOME=$HADOOP_INSTALL
    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_/INSTALL/lib/native
    export HADOOP_OPTS=”Djava.library.path”=$HADOOP_INSTALL/lib”
    #HADOOP VARIABLES END

    After writing the code, click on Esc on your keyboard and write the command:wq!
    This will save and exit you from the vi editor. The path has been set now as it can be seen in the image below:

    

    Step 13: Adding Configuration Files

  • Open hadoop-env.sh with the vi editor using the following command:

    vi /home/intellipaaat/hadoop/etc/hadoop/hadoop-env.sh

    

• • Replace this path with the Java path to tell Hadoop which path to use. You will see the following window coming up:
    

• • Change the JAVA_HOME variable to the path you had copied in the previous step
    Step 14:
    Now, several XML files need to be edited, and you need to set the property and the path for them.

• • Editing core-site.xml
    • • Use the same command as in the previous step and just change the last part to core-site.xml as given below:

        vi /home/intellipaaat/hadoop/etc/hadoop/core-site.xml

        Next, you will see the following window:

        

    • • Enter the following code in between the configuration tags as below:

        <configuration>
            <property>
                <name>fs.defaultFS</name>
                <value>hdfs://(your localhost):9000</value>
            </property>
        </configuration>
        

        

    • Now, exit from this window by entering the command:wq!

• • Editing yarn-site.xml
    • • Enter the command:

        vi /home/intellipaaat/hadoop/etc/hadoop/yarn-site.xml

        You will see the following window:

        

    • Enter the code in between the configuration tags as shown below:

      <configuration>
      <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
      </property>
      <property>
      <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
      <value>org.apache.hadoop.mapred.ShuffleHandler</value>
      </property>
      </configuration>

      

    • Exit from this window by pressing Esc and then writing the command:wq!

• • Editing mapred-site.xml
    • • Copy or rename a file mapred-site.xml.template with the name mapred-site.xml.Note: If you go to the following path, you will see that there is no file named mapred-site.xml:
        Home > intellipaaat > hadoop > hadoop-2.7.3 > etc > hadoop
        So, we will copy the contents of the mapred-site .xml.template to mapred-site.xml.
      • Use the following command to copy the contents:

        cp /home/intellipaaat/hadoop/hadoop-2.7.3/etc/hadoop/ mapred-site.xml.template /home/intellipaaat/hadoop/hadoop-2.7.3/etc/hadoop/ mapred-site.xml

        Once the contents have been copied to a new file named mapred-site.xml, you can verify it by going to the following path:
        Home > intellipaaat > hadoop > hadoop-2.7.3 > etc > hadoop

      • Now, use the following command to add configurations:

        vi/home/intellipaaat/hadoop/etc/hadoop/mapred-site.xml

        

    • In the new window, enter the following code in between the configuration tags as below:

      <configuration>
      <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
      </property>
      </configuration>

      

    • Exit using Esc and the command:wq!

• Editing hdfs-site.xml
  
  Before editing the hdfs-site.xml, two directories have to be created, which will contain the namenode and the datanode.
  • • Enter the following code for creating a directory, namenode:

      mkdir -p /home/intellipaaat/hadoop_store/hdfs/namenode

      
      Note: Here, mkdir means creating a new file.

  • • Similarly, to create the datanode directory, enter the following command:

      mkdir -p /home/intellipaaat/hadoop_store/hdfs/datanode

      

  • • Now, go to the following path to check both the files:
      Home > intellipaaat > hadoop_store > hdfs You can find both directories in the specified path as in the images below:
      
  • Now, to configure hdfs-site.xml, use the following command:

    vi /home/intellipaaat/hadoop/etc/hadoop/hdfs-site.xml

    

  • Enter the following code in between the configuration tags:

    <configuration>
    <property>
    <name>dfs.replication</name>
    <value>1</value>
    </property>
    <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/home/intellipaaat/hadoop_store/hdfs/namenode</value>
    </property>
    <property>
    <name>dfs.datanode.data.dir</name>
    <value> file:/home/intellipaaat/hadoop_store/hdfs/namenode</value>
    </property>
    </configuration>

    

  • Exit using Esc and the command:wq!

That’s all!
All your configurations are done. And Hadoop Installation is done now!

Step 15: Checking Hadoop
You will now need to check whether the Hadoop installation is successfully done on your system or not.

• Go to the location where you had extracted the Hadoop tar file, right-click on the bin, and open it in the terminal
• Now, write the command, ls
  Next, if you see a window as below, then it means that Hadoop is successfully installed!