See this page for more details about Apache Hadoop:
https://hadoop.apache.org/docs/current/

$ sudo addgroup hadoop $ sudo adduser --ingroup hadoop hadoop

Download Apache Hadoop: http://hadoop.apache.org/releases.html

Extract the file "hadoop-3.3.0.tar.gz" in the folder you want to install Hadoop: e.g. '/opt/hadoop-3.3.0'
$ tar -xf ~/Downloads/hadoop-3.3.0.tar.gz -C /opt/ $ chmod -R 755 /opt/hadoop-3.3.0 $ sudo chown -R hadoop:hadoop /opt/hadoop-3.3.0
Note: In the following sections, the environment variable ${HADOOP_HOME} will refer to this location '/opt/hadoop-3.3.0'.

$ su - hadoop

$ vi ~/.profile export JAVA_HOME="/opt/jdk1.8.0_172" export HADOOP_HOME="/opt/hadoop-3.3.0" export CLASSPATH=$CLASSPATH:$HADOOP_HOME/share/hadoop/common/lib PATH="$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH"
Load ".profile" environment variables: $ source ~/.profile
Print Hadoop version: $ hadoop version Hadoop 3.3.0

$ mkdir /home/hadoop/hadoop-tmp-dir

See this page for more detail:
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/core-default.xml

$ vi ${HADOOP_HOME}/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/hadoop-tmp-dir</value> </property> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:8020</value> </property> <property> <name>hadoop.proxyuser.hadoop.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.hadoop.hosts</name> <value>*</value> </property> </configuration>

See this page for more detail:
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

$ vi ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file://${hadoop.tmp.dir}/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file://${hadoop.tmp.dir}/dfs/data</value> </property> </configuration>

See this page for more detail:
https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

$ vi ${HADOOP_HOME}/etc/hadoop/mapred-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>mapreduce.jobtracker.address</name> <value>localhost:54311</value> </property> </configuration>

Edit file "hadoop-env.sh" and export "JAVA_HOME" environment variable. $ vi ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh export JAVA_HOME="/opt/jdk1.8.0_172"

$ hadoop namenode -format

$ ${HADOOP_HOME}/sbin/start-all.sh Starting namenodes on [localhost] Starting datanodes Starting secondary namenodes [mtitek] Starting resourcemanager Starting nodemanagers
You might get this error "Permission denied (publickey, password)" when you start Hadoop.
To fix this error, see Hadoop "start-all.sh" permission denied: "ssh localhost: Permission denied (publickey, password)".

Check permission: $ hdfs dfs -getfacl / # file: / # owner: hadoop # group: supergroup user::rwx group::r-x other::r-x
Set permission: $ hdfs dfs -chmod -R 775 / $ hdfs dfs -chown -R hadoop:hadoop /

Ports Used by Hadoop: DataNode: 50010 (dfs.datanode.address) NameNode: 8020 (fs.defaultFS) Secondary NameNode: 50090 (dfs.namenode.secondary.http-address)

• HDFS Web UI: http://localhost:9870
  
  
  
  
• Resource Manager: http://localhost:8088
  
  
  
  
• Node Manager: http://localhost:8042
  
  

Hadoop processes info:

• Java virtual machine process status tool: jps
  $ jps -ml 12422 org.apache.hadoop.hdfs.server.namenode.NameNode 12640 org.apache.hadoop.hdfs.server.datanode.DataNode 13524 org.apache.hadoop.yarn.server.nodemanager.NodeManager 13156 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager 12901 org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
  
• Display process info: ps -fp <pid> | less
  $ ps -fp 6476 | less UID PID PPID C STIME TTY TIME CMD hadoop 6476 1 0 09:55 ? 00:00:09 /opt/jdk1.8.0_172/bin/java -Dproc_datanode -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=ERROR,RFAS -Dyarn.log.dir=/opt/hadoop-3.3.0/logs -Dyarn.log.file=hadoop-hadoop-datanode-mtitek.log -Dyarn.home.dir=/opt/hadoop-3.3.0 -Dyarn.root.logger=INFO,console -Djava.library.path=/opt/hadoop-3.3.0/lib/native -Dhadoop.log.dir=/opt/hadoop-3.3.0/logs -Dhadoop.log.file=hadoop-hadoop-datanode-mtitek.log -Dhadoop.home.dir=/opt/hadoop-3.3.0 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.datanode.DataNode
  
• Display active TCP connections: sudo netstat -plten
  

Hadoop log files:

• Hadoop log files can be found in "${HADOOP_HOME}/logs/"
  
• Hadoop jetty web app in "/tmp/jetty*"
  
• Hadoop pid files: "/tmp/hadoop-*.pid"
  

$ ls -al ${HADOOP_HOME}/logs/ -rw-rw-r-- hadoop hadoop hadoop-hadoop-namenode-mtitek.log -rw-rw-r-- hadoop hadoop hadoop-hadoop-namenode-mtitek.out -rw-rw-r-- hadoop hadoop hadoop-hadoop-datanode-mtitek.log -rw-rw-r-- hadoop hadoop hadoop-hadoop-datanode-mtitek.out -rw-rw-r-- hadoop hadoop hadoop-hadoop-secondarynamenode-mtitek.log -rw-rw-r-- hadoop hadoop hadoop-hadoop-secondarynamenode-mtitek.out -rw-rw-r-- hadoop hadoop hadoop-hadoop-nodemanager-mtitek.log -rw-rw-r-- hadoop hadoop hadoop-hadoop-nodemanager-mtitek.out -rw-rw-r-- hadoop hadoop hadoop-hadoop-resourcemanager-mtitek.log -rw-rw-r-- hadoop hadoop hadoop-hadoop-resourcemanager-mtitek.out -rw-rw-r-- hadoop hadoop SecurityAuth-hadoop.audit drwxr-xr-x hadoop hadoop userlogs
$ ls -al /tmp/ | grep hadoop -rw-rw-r-- hadoop hadoop hadoop-hadoop-namenode.pid -rw-rw-r-- hadoop hadoop hadoop-hadoop-datanode.pid -rw-rw-r-- hadoop hadoop hadoop-hadoop-secondarynamenode.pid -rw-rw-r-- hadoop hadoop hadoop-hadoop-nodemanager.pid -rw-rw-r-- hadoop hadoop hadoop-hadoop-resourcemanager.pid drwxrwxr-x hadoop hadoop jetty-0.0.0.0-9870-hdfs-_-any-4404239779972224131.dir drwxrwxr-x hadoop hadoop jetty-localhost-34228-datanode-_-any-3510019043507033994.dir drwxrwxr-x hadoop hadoop jetty-0.0.0.0-9868-secondary-_-any-5247285317055775022.dir drwxrwxr-x hadoop hadoop jetty-0.0.0.0-8042-node-_-any-3822503648428813577.dir drwxrwxr-x hadoop hadoop jetty-0.0.0.0-8088-cluster-_-any-3021095246783254089.dir

$ ${HADOOP_HOME}/sbin/stop-all.sh

Make sure that Hadoop is not running (see above how to stop Hadoop).

$ hadoop namenode -format $ sudo rm -rf /opt/hadoop-3.3.0 $ sudo rm -rf /home/hadoop/hadoop-tmp-dir $ sudo userdel hadoop $ sudo groupdel hadoop $ sudo rm -rf /home/hadoop/
Note: You also need to delete Hadoop environment variables from "~/.profile" file.

To fix this issue you have to generate an SSH key: $ cd ~/.ssh/ $ ssh-keygen -t rsa -P "" $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Optional: You may also need to edit the file "sshd_config" and update "PubkeyAuthentication" and "AllowUsers" variables. $ sudo vi /etc/ssh/sshd_config PubkeyAuthentication yes AllowUsers hadoop
Reload SSH configs. $ sudo /etc/init.d/ssh reload
To test SSH connection: $ ssh localhost
To debug SSH connection: $ ssh -v localhost