Pre-requirements

• Ubuntu 18.04 installed on a virtual machine.

Create A New User

We begin with adding a new user , open a new terminal (Ctrl+Alt+T) & type the following commands. Firstly, create a group hadoop and add a new user hduser within the same hadoop group.

sudo addgroup hadoop
sudo adduser — ingroup hadoop hduser

The newly created user will require necessary root permissions for custom files installation. To do so we have to open the sudoers file (sudo visudo) and provide created user hduser root user privileges by updating the file like below

sudo visudo
hduser ALL=(ALL:ALL) ALL

Note: To exit from ‘nano’ editor window (Ctrl+X) to save changes press Y press enter to exit back to working terminal.

So, Now our new user is ready lets switch to hduser for installing hadoop framework environment.

su – hduser

Java Environment ..

Surprise! Before we install hadoop we have understand a quick simple concept: Hadoop framework requires java environment for running all its services on separate Java Virtual Machine (true only for pseudo distributed mode). Download JDK 8 or above, with java 7 attaining end of life in 2015, Hadoop 3.x revised the min. run time version to Java 8 or above.

Make sure the user changed from root user to hduser (su – hduser). Specify a preferred directory for storing all files for hadoop I always prefer /usr/local.

cd /usr/local

sudo wget –no-cookies –no-check-certificate –header “Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie” “http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz”

Extract the downloaded jdk file jdk-8u181-linux-x64.tar.gz in same preferred location /usr/local. Rename the extracted file from jdk1.8.0_181 to java.

sudo tar xvzf jdk-8u181-linux-x64.tar.gz
sudo mv jdk1.8.0_181 java

Now we have an extracted java file in /usr/local. But ubuntu 18.04 is unaware of the java environment. Lets set the java environment in .bashrc file or .profile file (for future stories I will prefer bashrc).

Doing so is compulsory, first change the working directory from /usr/local to home directory, open the .bashrc file and add following.

(check if java file is present or not!)
ls
cd ~
sudo nano ~/.bashrc

Set up java environment in .bashrc file by setting up java_home location and java bin path directory.

export JAVA_HOME=/usr/local/java
export PATH=$PATH:/usr/local/java/bin

Note : Whenever you update .bashrc file it apply changes in ubuntu environment either by rebooting the system or the better option applying source ~/.bashrc

Setting Oracle JDK as the default JVM…

In our case, the java executable is located under /usr/local/java . To set the JVM environment location in your machine run following commands. Let us update the environment with the setting the java (java executable), javac (java compiler), javaws (java web start).

sudo update-alternatives –install “/usr/bin/java” “java” “/usr/local/java/bin/java” 1
sudo update-alternatives –install “/usr/bin/javac” “javac” “/usr/local/java/bin/javac” 1
sudo update-alternatives –install “/usr/bin/javaws” “javaws” “/usr/local/java/bin/javaws” 1

Verify that java has been successfully configured by executing:

sudo update-alternatives –set java /usr/local/java/bin/java
sudo update-alternatives –set javac /usr/local/java/bin/javac
sudo update-alternatives –set javaws /usr/local/java/bin/javaws

Verify java environment manual installation is completed in the ubuntu system.

java -version

If you have completed till here, Congratulations !! You have just learned how to do manual installation oracle jdk8 on linux environment…

Setting Up Hduser with SSH Keys & Enable Localhost

Hadoop services like Resource Manager & Node Manager uses ssh to share the status of nodes like job scheduling, running status of job, status of DataNodes (alive or not) with all services that running on localhost on separate jvm they exchanges messages b/w slave to master & master to master via. ssh.

SSH(Secure Protocol) is used as the primary means of connecting to Linux servers remotely. In this case ssh localhost is about doing secure ssh to host i.e. localhost.

Try starting your localhost using command: ssh localhost. Doing so initially may throw connection refused error message…
ssh: connect to host localhost port 22: Connection refused

Note: If it doesn’t throw any error & succeeds in doing ssh to localhost you can skip these steps & jump to Hadoop Installation.

To handle connection refused error Follow the commands step by step …Remove older version SSH:
sudo apt-get remove openssh-server openssh-client

Install SSH Again:
sudo apt-get update
sudo apt-get install openssh-server openssh-client
sudo ufw allow 22

sudo systemctl restart ssh
sudo apt-get install ssh
sudo apt-get install rsync

After installing ssh (secure protocol)lets make ssh a secure communication by generating ssh keys and copying the key value in ~/.ssh/authorized_keys.

Generate Keys for secure communication:
ssh-keygen -t dsa -P ‘’ -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh-keygen -t rsa

Try ssh localhost again if it asks for password-login as shown below follow the next… Otherwise jump to hadoop installation.

Generate the ssh key again, store key in ~/.ssh/id_rsa.pub. Now,copy the keys to ~/.ssh/authorized_keys using cat command & change permissions of authorized_keys.

ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod og-wx ~/.ssh/authorized_keys
sudo apt-get update
ssh localhost

SSH to localhost is now secured & remotely accessible without password login requirement. Now our prerequisites to hadoop installation been completed successfully.
Note: Don’t forget to exit from localhost before proceeding to next.

Hadoop 3.x Installation Step By Step

Now, lets proceed towards major step of hadoop installation with downloading the latest stable release of hadoop 3.0.2. First get back to location /usr/local & download the stable release[When I publishing this hadoop3.0.2 is the stable version].

cd /usr/local
sudo wget http://redrockdigimark.com/apachemirror/hadoop/common/hadoop-3.0.2/hadoop-3.0.2.tar.gz

Extract the the hadoop file in same location and rename the extracted folder hadoop-3.0.2 to hadoop (just like we did for java).

sudo tar xvzf hadoop-3.0.2.tar.gz
sudo mv hadoop-3.0.2 hadoop

Now we has a folder hadoop framework installed in ubuntu in an embryonic stage.

Setting Up Hadoop 3.x in Pseudo Distributed Mode

Hadoop Framework supports three working modes : Standalone Mode, Pesudo-Distributed Mode, Fully-Distributed Mode. Where, standalone runs all its services in single machine & single JVM, pseudo-distributed mode runs all in one system but on separate JVMs (Best suitable for learning hadoop). Where enterprises set up hadoop in Fully-Distributed Mode each master & slave services running in separate systems & different JVMs. In this story I will be covering pseudo distributed mode.

First, provide the ownership of hadoop to ‘user’ hduser [“ This will give ownership only to hduser for running hadoop services ”] using chmod & change the mode of hadoop folder to read, write & execute modes of working.

sudo chown -R hduser:hadoop /usr/local/hadoop
sudo chmod -R 777 /usr/local/hadoop

Disable IPV6

Hadoop & IPV6 does not agrees on the meaning of address 0.0.0.0 so we need to disable IPV6 editing the file…
sudo nano /etc/sysctl.conf
with…
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1

For confirming if IPV6 is disable or not! execute the command.
cat /proc/sys/net/ipv6/conf/all/disable_ipv6

Setting Hadoop Environment

Apply changes in .bashrc file for setting the necessary hadoop environment. Setting changes with hadoop path. Locations of sbin[ “It stores hadoop’s necessary command location” ] & bin directory path are essential otherwise as user you have to always change location to hadoop’s sbin or bin to run required commands.

sudo nano ~/.bashrc

#HADOOP ENVIRONMENT
export HADOOP_PREFIX=/usr/local/hadoop
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export HADOOP_MAPRED_HOME=/usr/local/hadoop
export HADOOP_COMMON_HOME=/usr/local/hadoop
export HADOOP_HDFS_HOME=/usr/local/hadoop
export YARN_HOME=/usr/local/hadoop
export PATH=$PATH:/usr/local/hadoop/bin
export PATH=$PATH:/usr/local/hadoop/sbin

#HADOOP NATIVE PATH:
export “HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS=“-Djava.library.path=$HADOOP_PREFIX/lib”

After completing native directory path setting & configuring. Proceed the next step to hadoop configuration file setting …

Setting up Hadoop Configurations …

Lets change the working directory to hadoop configurations location
cd /usr/local/hadoop/etc/hadoop/

1. hadoop-env.sh
Open hadoop-env file & change the configurations…

sudo nano hadoop-env.sh

In Hadoop, by default logging level will be determined by logging
configuration variable HADOOP_ROOT_LOGGER in hadoop-env.sh file.
Set new configurations in hadoop environment file…

export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
export JAVA_HOME=/usr/local/java
export HADOOP_HOME_WARN_SUPPRESS=”TRUE”
export HADOOP_ROOT_LOGGER=”WARN,DRFA”

2. yarn-site.xml
Open hadoop-env file & set the configurations …

sudo nano yarn-site.xml

yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.nodemanager.aux-services.mapreduce.shuffle.class
org.apache.hadoop.mapred.ShuffleHandler

Note : Do exactly same to set all other configuration files..

3. hdfs-site.xml

sudo nano hdfs-site.xml

dfs.replication
1
dfs.namenode.name.dir
file:/usr/local/hadoop/yarn_data/hdfs/namenode
dfs.datanode.data.dir
file:/usr/local/hadoop/yarn_data/hdfs/datanode

4. core-site.xml

sudo nano core-site.xml

hadoop.tmp.dir
/app/hadoop/tmp
fs.default.name
hdfs://localhost:9000

5. mapred-site.xml

sudo nano mapred-site.xml

mapred.framework.name
yarn
mapreduce.jobhistory.address
localhost:10020

Create a temp directory which will be used as base location for DFS…

Now we have updated all hadoop configuration files, first get back to home directory & create temp directory.

sudo mkdir -p /app/hadoop/tmp
sudo chown -R hduser:hadoop /app/hadoop/tmp
sudo chmod -R 777 /app/hadoop/tmp

Create Directory where hadoop will store its yarn data work and give good permission to it “path set for namenode & datanode in hdfs-site.xml”

sudo mkdir -p /usr/local/hadoop/yarn_data/hdfs/namenode
sudo mkdir -p /usr/local/hadoop/yarn_data/hdfs/datanode
sudo chmod -R 777 /usr/local/hadoop/yarn_data/hdfs/namenode
sudo chmod -R 777 /usr/local/hadoop/yarn_data/hdfs/datanode
sudo chown -R hduser:hadoop /usr/local/hadoop/yarn_data/hdfs/namenode
sudo chown -R hduser:hadoop /usr/local/hadoop/yarn_data/hdfs/datanode

Format, hadoop namenode to clean the cluster before we start all hadoop framework’s core services.

hdfs namenode -format

Start all hadoop services by executing command one by one.
start-dfs.sh
start-yarn.sh

With all Hadoop services running in pesudo-distributed mode.