準備

$ sudo apt-get install ssh
$ sudo apt-get install pdsh

我們還需要設定pdsh的rcmd，執行下面內容。或建立/etc/pdsh/rcmd_default，裡面填入ssh

echo "ssh" > /etc/pdsh/rcmd_default

如果這步驟不設定，我們在啟動hdfs時，可能會報告

[email protected]:~/wei/hadoop/hadoop-3.1.0/sbin$ start-dfs.sh
Starting namenodes on [gsta005]
[email protected]: gsta005: rcmd: socket: Permission denied
Starting datanodes
 
[email protected]: localhost: rcmd: socket: Permission denied
Starting secondary namenodes [gsta005]
[email protected]: gsta005: rcmd: socket: Permission denied

設定免密碼登入的ssh

$ ssh-keygen -t rsa -P  -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
驗證
$ ssh localhost
 

配置和啟動hdfs

etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://191.8.2.45:9000</value>
    </property>
    <!-- 因為我在一臺機器內同時安裝了2.9.0版本和3.1.0版本，不採用預設的/tmp目錄，採用指定目錄方式-->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/gsta/wei/hadoop/hadoop-3.1.0/tmp</value>
    </property>
</configuration>
 

etc/hadoop/hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

格式化hdfs

$ hdfs namenode -format

啟動hdfs

$ start-hdfs.sh
設定相關的資料夾 /user,/user/<username>，命令同v2版本

開啟web監控

http://191.8.2.45:9870

如果啟動的過程中報錯：

[email protected]:~/wei/hadoop$ start-dfs.sh
Starting namenodes on [gsta005]
gsta005: ERROR: JAVA_HOME is not set and could not be found.
[email protected]: gsta005: ssh exited with exit code 1
Starting datanodes
localhost: ERROR: JAVA_HOME is not set and could not be found.
[email protected]: localhost: ssh exited with exit code 1
Starting secondary namenodes [gsta005]
gsta005: ERROR: JAVA_HOME is not set and could not be found.
[email protected]: gsta005: ssh exited with exit code 1

則在hadoop-env.sh中指定JAVA_HOME

export JAVA_HOME=/home/wei/jdk1.8.0_66

配置和啟動yarn

etc/hadoop/mapred-site.xml：

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>

        <property>
                <name>mapreduce.application.classpath</name>
                <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
        </property>
</configuration>

etc/hadoop/yarn-site.xml：

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
  </property>
</configuration>

啟動yarn

start-yarn.sh

web監控檢視

http://191.8.2.45:8088/