Install Hadoop in standalone machine - java Hadoop

How to Install Hadoop in Standalone Machine?

Steps for Installing Hadoop in Standalone Machine

Description:
To install Hadoop on a standalone machine, ensure Java is installed, download the latest Hadoop version, and extract it to a directory. Set environment variables for JAVA_HOME and HADOOP_HOME in the .bashrc file. Configure Hadoop by editing core-site.xml (set fs.defaultFS to hdfs://localhost:9000) and hdfs-site.xml (set replication factor to 1). Format the Hadoop filesystem using hdfs namenode -format and start the Hadoop services with start-dfs.sh. Finally, verify the installation by checking the Hadoop processes with jps and accessing the web interface at http://localhost:50070.

Step 1: Update Package Lists for APT

Command:
sudo apt update

Step 2: Install Java JDK 8

Command:
sudo apt install openjdk-8-jdk

Step 3: Verify Java Installation

Command:
java -version

Step 4: Install OpenSSH

Command:
sudo apt install openssh-server openssh-client -y

Step 5: Create Hadoop User

Command:
sudo adduser hdoop

Step 6: Switch to the Hadoop User

Command:
su – hdoop

Step 7: Generate SSH Key Pair

Command:
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

Step 8: Store Public Key

Command:
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Step 9: Verify SSH Configuration

Command:
ssh localhost

Step 10: The steps outlined in this tutorial use the binary download for Hadoop Version 3.4.0. Select your preferred option, and you will be presented with a mirror link to download the Hadoop tar package.p

Step 11: Use the provided mirror link and download the Hadoop package using the wget command

wget:
https://dlcdn.apache.org/hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz

Step 12: Once the download completes, extract the .tar.gz file

Command:
tar xzf hadoop-3.4.0.tar.gz

Step 13: Edit the .bashrc shell configuration file using a text editor

Command:
nano ~/.bashrc

Step 14: Define the Hadoop environment variables by adding the following content at the end of the file

Configuration:

# Hadoop Related Options
export HADOOP_HOME=/home/hdoop/hadoop-3.4.0
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"

Step 15: Once you add the variables, save and exit the .bashrc file

Step 16: Run the command below to apply the changes to the current environment

Command:
source ~/.bashrc

Step 17: Use the previously created $HADOOP_HOME variable to access the hadoop-env.sh file:

Command:
nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh

Step 18: Uncomment the $JAVA_HOME variable and set the full path to OpenJDK

Configuration:

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Step 19: Open the core-site.xml file for editing

Command:
nano $HADOOP_HOME/etc/hadoop/core-site.xml

Step 20: Add the following configuration to the core-site.xml file

Configuration:

<configuration>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hdoop/tmpdata</value>
  </property>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://127.0.0.1:9000</value>
  </property>
</configuration>

Step 21: Put the core-site.xml file in $HADOOP_HOME/etc/hadoop

Step 22: Use the following command to open the hdfs-site.xml file for editing

Command:
nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml

Step 23: Add the following configuration to the file and adjust directories as needed

Configuration:

<configuration>
  <property>
    <name>dfs.data.dir</name>
    <value>/home/hdoop/dfsdata/namenode</value>
  </property>
  <property>
    <name>dfs.data.dir</name>
    <value>/home/hdoop/dfsdata/datanode</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

Step 24: Put the xml file in $HADOOP_HOME/etc/hadoop/hdfs-site.xml

Step 25: Use the following command to access the mapred-site.xml file and define MapReduce values

Command:
sudo nano $HADOOP_HOME/etc/hadoop/mapred-site.xml

Step 25 :Add the following configuration to change the default MapReduce framework name value to YARN

Configuration:

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

Step 27: Put the xml file in $HADOOP_HOME/etc/hadoop/mapred-site.xml

Step 28: Open the yarn-site.xml file in a text editor

Command:
nano $HADOOP_HOME/etc/hadoop/yarn-site.xml

Step 29: Append the following configuration to the yarn-site.xml file

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>127.0.0.1</value>
  </property>
  <property>
    <name>yarn.acl.enable</name>
    <value>0</value>
  </property>
  <property>
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PERPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
  </property>
</configuration>

Step 30: Put the xml file in $HADOOP_HOME/etc/hadoop/yarn-site.xml<

Step 31: Format HDFS NameNode

Command:
hdfs namenode -format

Step 32: Navigate to the hadoop-3.4.0/sbin directory and execute the following command to start the NameNode and DataNode:

Command:
./start-dfs.sh

Step 33: Once the namenode, datanodes, and secondary namenode are up and running, start the YARN resource and nodemanagers by typing:

Command:
./start-yarn.sh

Step 34: Run the following command to check if all the daemons are active and running as Java processes:

Command:
jps

Step 35:Use your preferred browser and navigate to your localhost URL or IP. The default port number 9870 gives you access to the Hadoop NameNode UI:

URL:
http://localhost:50070/

Step 36: The Resource Manager is an invaluable tool that allows you to monitor all running processes in your Hadoop cluster

Summary: You have configured Hadoop, formatted the NameNode, and started both the DFS and YARN services.

List

Office Address

How to Install Hadoop in Standalone Machine?

Steps for Installing Hadoop in Standalone Machine

Step 1: Update Package Lists for APT

Step 2: Install Java JDK 8

Step 3: Verify Java Installation

Step 4: Install OpenSSH

Step 5: Create Hadoop User

Step 6: Switch to the Hadoop User

Step 7: Generate SSH Key Pair

Step 8: Store Public Key

Step 9: Verify SSH Configuration

Step 10: The steps outlined in this tutorial use the binary download for Hadoop Version 3.4.0. Select your preferred option, and you will be presented with a mirror link to download the Hadoop tar package.p

Step 11: Use the provided mirror link and download the Hadoop package using the wget command

Step 12: Once the download completes, extract the .tar.gz file

Step 13: Edit the .bashrc shell configuration file using a text editor

Step 14: Define the Hadoop environment variables by adding the following content at the end of the file

Step 15: Once you add the variables, save and exit the .bashrc file

Step 16: Run the command below to apply the changes to the current environment

Step 17: Use the previously created $HADOOP_HOME variable to access the hadoop-env.sh file:

Step 18: Uncomment the $JAVA_HOME variable and set the full path to OpenJDK

Step 19: Open the core-site.xml file for editing

Step 20: Add the following configuration to the core-site.xml file

Step 21: Put the core-site.xml file in $HADOOP_HOME/etc/hadoop

Step 22: Use the following command to open the hdfs-site.xml file for editing

Step 23: Add the following configuration to the file and adjust directories as needed

Step 24: Put the xml file in $HADOOP_HOME/etc/hadoop/hdfs-site.xml

Step 25: Use the following command to access the mapred-site.xml file and define MapReduce values

Step 25 :Add the following configuration to change the default MapReduce framework name value to YARN

Step 27: Put the xml file in $HADOOP_HOME/etc/hadoop/mapred-site.xml

Step 28: Open the yarn-site.xml file in a text editor

Step 29: Append the following configuration to the yarn-site.xml file

Step 30: Put the xml file in $HADOOP_HOME/etc/hadoop/yarn-site.xml<

Step 31: Format HDFS NameNode

Step 32: Navigate to the hadoop-3.4.0/sbin directory and execute the following command to start the NameNode and DataNode:

Step 33: Once the namenode, datanodes, and secondary namenode are up and running, start the YARN resource and nodemanagers by typing:

Step 34: Run the following command to check if all the daemons are active and running as Java processes:

Step 35:Use your preferred browser and navigate to your localhost URL or IP. The default port number 9870 gives you access to the Hadoop NameNode UI:

Step 36: The Resource Manager is an invaluable tool that allows you to monitor all running processes in your Hadoop cluster

Related Links