Hadoop部署（六）——CentOS 7部署Hadoop叢集（HA高可用叢集）
阿新 • • 發佈：2018-12-14
目錄
測試環境
Linux系統版本：CentOS 7 64位
Hadoop 組織框架
Hadoop主要包括兩部分：
一部分是HDFS（Hadoop Distributed File System），主要負責分散式儲存和計算；
另一部分是YARN（Yet Another Resource Negotiator， 從Hadoop2.0開始引入），主要負責叢集的資源管理和排程。
HDFS架構
架構圖：
1. Active Name Node
主Master，整個Hadoop叢集只能有一個
管理HDFS檔案系統的名稱空間
維護元資料資訊
管理副本的配置和資訊（預設三個副本）
處理客戶端讀寫請求
2. Standby Name Node
Active Name Node的熱備節點
Active Name Node故障時可快速切換成新的Active Name Node
週期性同步edits編輯日誌，定期合併fsimage與edits到本地磁碟
3. Journal Node
可以被Active Name Node和StandBy Name Node同時訪問，用以支援Active Name Node高可用
Active Name Node在檔案系統被修改時，會向Journal Node寫入操作日誌（edits）
Standby Name Node同步Journal Node edits日誌，使叢集中的更新操作可以被共享和同步。
4. Data Node
Slave 工作節點，叢集一般會啟動多個
負責儲存資料塊和資料塊校驗
執行客戶端的讀寫請求
通過心跳機制定期向NameNode彙報執行狀態和本地所有塊的列表資訊
在叢集啟動時DataNode項NameNode提供儲存Block塊的列表資訊
5. Block資料塊
HDSF固定的最小的儲存單元（預設128M，可配置修改）
寫入到HDFS的檔案會被切分成Block資料塊（若檔案大小小於資料塊大小，則不會佔用整個資料塊）
預設配置下，每個block有三個副本
6. Client
與Name Node互動獲取檔案的元資料資訊
與Data Node，讀取或者寫入資料
通過客戶端可以管理HDFS
YARN架構
架構圖：
1. Resource Manager
整個叢集只有一個Master。Slave可以有多個，支援高可用
處理客戶端Client請求
啟動／管理／監控ApplicationMaster
監控NodeManager
資源的分配和排程
2. Node Manager
每個節點只有一個，一般與Data Node部署在同一臺機器上且一一對應
定時向Resource Manager彙報本機資源的使用狀況
處理來自Resource Manager的作業請求，為作業分配Container
處理來自Application Master的請求，啟動和停止Container
3. Application Master
每個任務只有一個，負責任務的管理，資源的申請和任務排程
與Resource Manager協商，為任務申請資源
與Node Manager通訊，啟動／停止任務
監控任務的執行狀態和失敗處理
4. Container
任務執行環境的抽象，只有在分配任務時才會抽象生成一個Container
負責任務執行資源和環境的維護（節點，記憶體，CPU）
負責任務的啟動
雖然在架構圖中沒有畫出，但Hadoop高可用都是基於Zookeeper來實現的。如NameNode高可用，Block高可用，ResourceManager高可用等
HA叢集部署規劃
| 主機名稱 | IP地址 | 使用者名稱稱 | 程序 | 安裝的軟體 |
| node200 | 192.168.33.200 | hadoop | NameNode（Active）、ResourceManager（StandBy）、ZKFC、JobHistoryServer | JDK、Hadoop |
| node201 | 192.168.33.201 | hadoop | NameNode（StandBy）、ResourceManager（Active）、ZKFC、WebProxyServer | JDK、Hadoop |
| node202 | 192.168.33.202 | hadoop | DataNode、NodeManager、JournalNode、QuorumPeerMain | JDK、Hadoop、Zookeepe |
| node203 | 192.168.33.203 | hadoop | DataNode、NodeManager、JournalNode、QuorumPeerMain | JDK、Hadoop、Zookeepe |
| node204 | 192.168.33.204 | hadoop | DataNode、NodeManager、JournalNode、QuorumPeerMain | JDK、Hadoop、Zookeepe |
規劃說明：
HDFS HA通常由兩個NameNode組成，一個處於Active狀態，另一個處於Standby狀態。
Active NameNode對外提供服務，而Standby NameNode則不對外提供服務，僅同步Active NameNode的狀態，以便能夠在它失敗時快速進行切換。
Hadoop 2.0官方提供了兩種HDFS HA的解決方案，一種是NFS，另一種是QJM。這裡我們使用簡單的QJM。在該方案中，主備NameNode之間通過一組JournalNode同步元資料資訊，一條資料只要成功寫入多數JournalNode即認為寫入成功。通常配置奇數個JournalNode，這裡還配置了一個Zookeeper叢集，用於ZKFC故障轉移，當Active NameNode掛掉了，會自動切換Standby NameNode為Active狀態。
YARN的ResourceManager也存在單點故障問題，這個問題在hadoop-2.4.1得到了解決：有兩個ResourceManager，一個是Active，一個是Standby，狀態由Zookeeper進行協調。
YARN框架下的MapReduce可以開啟JobHistoryServer來記錄歷史任務資訊，否則只能檢視當前正在執行的任務資訊。
Zookeeper的作用是負責HDFS中NameNode主備節點的選舉，和YARN框架下ResourceManaer主備節點的選舉。
自動故障轉移
Zookeeper叢集作用：
一是故障監控。每個NameNode將會和Zookeeper建立一個持久session，如果NameNode失效，那麼此session將會過期失效，此後Zookeeper將會通知另一個Namenode，然後觸發Failover；
二是NameNode選舉。ZooKeeper提供了簡單的機制來實現Acitve Node選舉，如果當前Active失效，Standby將會獲取一個特定的排他鎖，那麼獲取鎖的Node接下來將會成為Active。
ZKFC：
ZKFC是一個Zookeeper的客戶端，它主要用來監測和管理NameNodes的狀態，每個NameNode機器上都會執行一個ZKFC程式
主要職責：
一是健康監控。ZKFC間歇性的ping NameNode，得到NameNode返回狀態，如果NameNode失效或者不健康，那麼ZKFS將會標記其為不健康；
二是Zookeeper會話管理。當本地NaneNode執行良好時，ZKFC將會持有一個Zookeeper session，如果本地NameNode為Active，它同時也持有一個“排他鎖”znode，如果session過期，那麼次lock所對應的znode也將被刪除；
三是選舉。當叢集中其中一個NameNode宕機，Zookeeper會自動將另一個啟用。
關於叢集主機時間
因為高可用叢集的機制，各主機在叢集中的時間需一致。
在下面Linux搭建前將虛擬機器進行設定，設定方法如下：
安裝完成後對每臺主機的時間進行確認，確保每臺主機時間一致。
Linux環境搭建
按如下方法部署五臺主機，主機名與IP地址的對應關係見上文叢集部署規劃
配置完成之後各主機IP、主機名與時間資訊如下：（時間不一致的自己百度同步叢集時間的方法）
命令：
#檢視系統ip資訊
ip a
#檢視系統時間
date執行結果：
[[email protected] ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:6a:0e:74 brd ff:ff:ff:ff:ff:ff
inet 192.168.33.200/24 brd 192.168.33.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe6a:e74/64 scope link
valid_lft forever preferred_lft forever
[[email protected] ~]# date
2018年 10月 11日 星期四 16:57:56 CST
[[email protected] ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:60:36:3c brd ff:ff:ff:ff:ff:ff
inet 192.168.33.201/24 brd 192.168.33.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe60:363c/64 scope link
valid_lft forever preferred_lft forever
[[email protected] ~]# date
2018年 10月 11日 星期四 16:57:48 CST
[[email protected] ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:41:4a:f4 brd ff:ff:ff:ff:ff:ff
inet 192.168.33.202/24 brd 192.168.33.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe41:4af4/64 scope link
valid_lft forever preferred_lft forever
[[email protected] ~]# date
2018年 10月 11日 星期四 16:58:00 CST
[[email protected] ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:53:27:40 brd ff:ff:ff:ff:ff:ff
inet 192.168.33.203/24 brd 192.168.33.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe53:2740/64 scope link
valid_lft forever preferred_lft forever
[[email protected] ~]# date
2018年 10月 11日 星期四 16:58:01 CST
[[email protected] ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:6e:ad:fa brd ff:ff:ff:ff:ff:ff
inet 192.168.33.204/24 brd 192.168.33.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe6e:adfa/64 scope link
valid_lft forever preferred_lft forever
[[email protected] ~]# date
2018年 10月 11日 星期四 16:57:54 CST
網路測試：
配置完成之後測試各主機網路互通情況，在每臺主機上執行下面兩條命令，執行過程中按Ctrl+C可以終止程序，下面就不貼測試效果了
ping 192.168.33.1
ping www.baidu.com配置Java環境
node200、node201、node202、node203、node204都需要安裝
為上面安裝的系統配置Java環境變數，本文中就寫關鍵配置步驟與執行命令了，想了解詳細的配置過程可以檢視：
為了方便，本文就直接使用rpm包安裝了，/etc/profile檔案暫時不進行配置，到後面配置hadoop單機版時再進行配置
[1-3]均使用root使用者執行
1、將安裝包jdk-8u181-linux-x64.rpm上傳到/usr/local目錄下
2、安裝rpm包，先設定許可權，然後執行rpm命令安裝
chmod 755 /usr/local/jdk-8u181-linux-x64.rpm
rpm -ivh /usr/local/jdk-8u181-linux-x64.rpm3、校驗安裝情況
java -version安裝單機版Hadoop
node200、node201、node202、node203、node204都需要安裝
[1-5]均使用root使用者執行
1、將壓縮包hadoop-2.7.3.tar.gz上傳到/usr/local目錄下
2、解壓壓縮包，進入/usr/local目錄，對資料夾重新命名
tar -zxvf /usr/local/hadoop-2.7.3.tar.gz
cd /usr/local
mv hadoop-2.7.3 hadoop3、建立hadoop使用者和hadoop使用者組，並設定hadoop使用者密碼
useradd hadoop
passwd hadoop4、為hadoop使用者新增sudo許可權
vi /etc/sudoers在root使用者下面一行加上hadoop ALL=(ALL) ALL，儲存並退出（這裡需要用wq!強制儲存退出）
## Next comes the main part: which users can run what software on
## which machines (the sudoers file can be shared between multiple
## systems).
## Syntax:
##
## user MACHINE=COMMANDS
##
## The COMMANDS section may have other options added to it.
##
## Allow root to run any commands anywhere
root ALL=(ALL) ALL
hadoop ALL=(ALL) ALL5、將hadoop資料夾的主：組設定成hadoop，/usr目錄與/usr/local目錄所屬主：組均為root，預設許可權為755，也就是說其他使用者（hadoop）沒有寫入（w）許可權，在這裡我們需要將這兩個目錄其他使用者的許可權設定為7
chown -R hadoop:hadoop hadoop
chmod 757 /usr
chmod 757 /usr/localZookeeper叢集安裝
在node202、node203、node204上安裝
Zookeeper是一個開源分散式協調服務，其獨特的Leader-Follower叢集結構，很好的解決了分散式單點問題。目前主要用於諸如：統一命名服務、配置管理、鎖服務、叢集管理等場景。大資料應用中主要使用Zookeeper的叢集管理功能。
[1-5]均使用root使用者執行
1、將壓縮包zookeeper-3.4.10.tar.gz上傳到node202的/usr/local目錄下
2、解壓壓縮包，進入/usr/local目錄，對資料夾重新命名
tar -zxvf /usr/local/zookeeper-3.4.10.tar.gz
cd /usr/local
mv zookeeper-3.4.10 zookeeper3、修改zookeeper的配置檔案，命令如下：
cd /usr/local/zookeeper/conf/
cp zoo_sample.cfg zoo.cfg
vi zoo.cfg配置檔案如下：
# 客戶端心跳時間(毫秒) tickTime=2000 # 允許心跳間隔的最大時間 initLimit=10 # 同步時限 syncLimit=5 # 資料儲存目錄 dataDir=/usr/local/zookeeperdata # 資料日誌儲存目錄 dataLogDir=/usr/local/tmp/zookeeperlogs # 埠號 clientPort=2181 # 叢集節點和服務埠配置 server.1=node202:2888:3888 server.2=node203:2888:3888 server.3=node204:2888:3888 # 以下為優化配置 # 伺服器最大連線數，預設為10，改為0表示無限制 maxClientCnxns=0 # 快照數 autopurge.snapRetainCount=3 # 快照清理時間，預設為0 autopurge.purgeInterval=1
3、修改完zookeeper的配置檔案，將node202上的zookeeper上傳到其他兩臺伺服器：
[[email protected] conf]# scp -r /usr/local/zookeeper 192.168.33.203:/usr/local/
[[email protected] conf]# scp -r /usr/local/zookeeper 192.168.33.204:/usr/local/
#這兒輸入yes回車
Are you sure you want to continue connecting (yes/no)? yes
#這裡輸入遠端主機的root密碼
[email protected]'s password:4、在三臺主機上建立 zookeeper資料儲存目錄：
mkdir /usr/local/zookeeperdata在資料夾下面建立一個檔案，叫myid，並且在檔案裡寫入server.X對應的X
# 叢集節點和服務埠配置
server.1=node202:2888:3888
server.2=node203:2888:3888
server.3=node204:2888:3888上述配置中node202是1，node203是2，node204是3，所以按如下操作：
在node202 上執行：
echo "1" > /usr/local/zookeeperdata/myid在node203 上執行：
echo "2" > /usr/local/zookeeperdata/myid在node204 上執行：
echo "3" > /usr/local/zookeeperdata/myid5、將zookeeper資料夾的主：組設定成hadoop
chown -R hadoop:hadoop /usr/local/zookeeper
chown -R hadoop:hadoop /usr/local/zookeeperdata配置環境變數
node200、node201、node202、node203、node204都需要配置，第2步有所區別
[1-3]均使用root使用者執行
1、編輯/etc/profile檔案
vi /etc/profile2、在末尾加上如下程式碼
node200、node201新增如下程式碼：
export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64
export HADOOP_HOME=/usr/local/hadoop
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JRE_HOME=$JAVA_HOME/jrenode202、node203、node204新增如下程式碼：
export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64
export HADOOP_HOME=/usr/local/hadoop
export ZOOKEEPER_HOME=/usr/local/zookeeper
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JRE_HOME=$JAVA_HOME/jre3、配置完環境變數之後儲存退出，讓環境變數立即生效
source /etc/profile關閉防火牆
node200、node201、node202、node203、node204都需要配置
CentOS 7 使用的是firewalld作為防火牆，與CentOS 6 有所不同
下面三步均使用root使用者執行
檢視防火牆狀態：
systemctl status firewalld關閉防火牆：
systemctl stop firewalld關閉防火牆開機自動啟動：
systemctl disable firewalld修改hosts檔案
修改所有主機的/etc/hosts檔案，這裡使用root使用者操作
vi /etc/hosts在檔案最後面加上：
192.168.33.200 node200
192.168.33.201 node201
192.168.33.202 node202
192.168.33.203 node203
192.168.33.204 node204
注意：IP後面是Tab製表符，而不是空格，這裡配置完後最好測試一下網路。如果複製貼上配置完後無法ping通，可能是IP地址後面的空格問題。
以下為測試方法：
ping node200
ping node201
ping node202
ping node203
ping node204配置到這裡，我們切換到hadoop使用者，使用hadoop使用者進行下面的操作
配置SSH免密登入
具體方法參照：
這裡貼關鍵程式碼，不展示操作過程：
在node200、node201、node202、node203、node204上分別操作，一路回車就可以了：
[[email protected] ~]$ ssh-keygen -t rsa
[[email protected] ~]$ ssh-keygen -t rsa
[[email protected] ~]$ ssh-keygen -t rsa
[[email protected] ~]$ ssh-keygen -t rsa
[[email protected] ~]$ ssh-keygen -t rsa
在node200上操作：
[[email protected] ~]$ ssh-copy-id localhost
[[email protected] ~]$ scp ~/.ssh/* node201:~/.ssh
[[email protected] ~]$ scp ~/.ssh/* node202:~/.ssh
[[email protected] ~]$ scp ~/.ssh/* node203:~/.ssh
[[email protected] ~]$ scp ~/.ssh/* node204:~/.ssh修改Hadoop配置檔案
均使用hadoop使用者操作，只需要在node200上修改即可
先進入/usr/local/hadoop/etc/hadoop/檔案
[[email protected] ~]$ cd /usr/local/hadoop/etc/hadoop/1、修改hadoop-env.sh檔案
[[email protected] hadoop]$ vi hadoop-env.sh找到export JAVA_HOME=${JAVA_HOME}，在前面加個#註釋掉，將JAVA_HOME用路徑代替，如下：
#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/usr/java/jdk1.8.0_181-amd642、修改core-site.xml檔案
[[email protected] hadoop]$ vi core-site.xml配置檔案如下：
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <!-- 指定hdfs的nameservices名稱為mycluster，與hdfs-site.xml的HA配置相同 --> <property> <name>fs.defaultFS</name> <value>hdfs://mycluster</value> </property> <!-- 指定快取檔案儲存的路徑 --> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/tmp/hadoop/data</value> </property> <!-- 配置hdfs檔案被永久刪除前保留的時間（單位：分鐘），預設值為0表明垃圾回收站功能關閉 --> <property> <name>fs.trash.interval</name> <value>1440</value> </property> <!-- 指定zookeeper地址，配置HA時需要 --> <property> <name>ha.zookeeper.quorum</name> <value>node202:2181,node203:2181,node204:2181</value> </property> </configuration>
3、修改hdfs-site.xml
[[email protected] hadoop]$ vi hdfs-site.xml配置檔案如下：
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <!-- 指定hdfs元資料儲存的路徑 --> <property> <name>dfs.namenode.name.dir</name> <value>/usr/local/hadoopdata/namenode</value> </property> <!-- 指定hdfs資料儲存的路徑 --> <property> <name>dfs.datanode.data.dir</name> <value>/usr/local/hadoopdata/datanode</value> </property> <!-- 資料備份的個數 --> <property> <name>dfs.replication</name> <value>3</value> </property> <!-- 關閉許可權驗證 --> <property> <name>dfs.permissions.enabled</name> <value>false</value> </property> <!-- 開啟WebHDFS功能（基於REST的介面服務） --> <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> <!-- //////////////以下為HDFS HA的配置////////////// --> <!-- 指定hdfs的nameservices名稱為mycluster --> <property> <name>dfs.nameservices</name> <value>mycluster</value> </property> <!-- 指定mycluster的兩個namenode的名稱分別為nn1,nn2 --> <property> <name>dfs.ha.namenodes.mycluster</name> <value>nn1,nn2</value> </property> <!-- 配置nn1,nn2的rpc通訊埠 --> <property> <name>dfs.namenode.rpc-address.mycluster.nn1</name> <value>node200:9000</value> </property> <property> <name>dfs.namenode.rpc-address.mycluster.nn2</name> <value>node201:9000</value> </property> <!-- 配置nn1,nn2的http通訊埠 --> <property> <name>dfs.namenode.http-address.mycluster.nn1</name> <value>node200:50070</value> </property> <property> <name>dfs.namenode.http-address.mycluster.nn2</name> <value>node201:50070</value> </property> <!-- 指定namenode元資料儲存在journalnode中的路徑 --> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://node202:8485;node203:8485;node204:8485/mycluster</value> </property> <!-- 指定journalnode日誌檔案儲存的路徑 --> <property> <name>dfs.journalnode.edits.dir</name> <value>/usr/local/tmp/journalnodelogs</value> </property> <!-- 指定HDFS客戶端連線active namenode的java類 --> <property> <name>dfs.client.failover.proxy.provider.mycluster</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <!-- 配置隔離機制為ssh --> <property> <name>dfs.ha.fencing.methods</name> <value>sshfence</value> </property> <!-- 指定祕鑰的位置 --> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/home/hadoop/.ssh/id_rsa</value> </property> <!-- 開啟自動故障轉移 --> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> </configuration>
4、修改mapred-site.xml
/usr/local/hadoop/etc/hadoop資料夾中並沒有mapred-site.xml檔案，但提供了模板mapred-site.xml.template，將其複製一份重新命名為mapred-site.xml 即可
[[email protected] hadoop]$ cp mapred-site.xml.template mapred-site.xml
[[email protected] hadoop]$ vi mapred-site.xml配置檔案如下：
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <!-- 指定MapReduce計算框架使用YARN --> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <!-- 指定jobhistory server的rpc地址 --> <property> <name>mapreduce.jobhistory.address</name> <value>node200:10020</value> </property> <!-- 指定jobhistory server的http地址 --> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>node200:19888</value> </property> <!-- 開啟uber模式（針對小作業的優化） --> <property> <name>mapreduce.job.ubertask.enable</name> <value>true</value> </property> <!-- 配置啟動uber模式的最大map數 --> <property> <name>mapreduce.job.ubertask.maxmaps</name> <value>9</value> </property> <!-- 配置啟動uber模式的最大reduce數 --> <property> <name>mapreduce.job.ubertask.maxreduces</name> <value>5</value> </property> </configuration>
5、修改yarn-site.xml
[[email protected] hadoop]$ vi yarn-site.xml配置檔案如下：
<?xml version="1.0"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <configuration> <!-- Site specific YARN configuration properties --> <!-- NodeManager上執行的附屬服務，需配置成mapreduce_shuffle才可執行MapReduce程式 --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <!-- 配置Web Application Proxy安全代理（防止yarn被攻擊） --> <property> <name>yarn.web-proxy.address</name> <value>node201:8888</value> </property> <!-- 開啟日誌 --> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <!-- 配置日誌刪除時間為7天，-1為禁用，單位為秒 --> <property> <name>yarn.log-aggregation.retain-seconds</name> <value>604800</value> </property> <!-- 修改日誌目錄 --> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/usr/local/tmp/hadooplogs</value> </property> <!--配置nodemanager可用的資源記憶體 <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>2048</value> </property> 配置nodemanager可用的資源CPU <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>2</value> </property> --> <!-- //////////////以下為YARN HA的配置////////////// --> <!-- 開啟YARN HA --> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <!-- 啟用自動故障轉移 --> <property> <name>yarn.resourcemanager.ha.automatic-failover.enabled</name> <value>true</value> </property> <!-- 指定YARN HA的名稱 --> <property> <name>yarn.resourcemanager.cluster-id</name> <value>yarncluster</value> </property> <!-- 指定兩個resourcemanager的名稱 --> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <!-- 配置rm1，rm2的主機 --> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>node201</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>node200</value> </property> <!-- 配置YARN的http埠 --> <property> <name>yarn.resourcemanager.webapp.address.rm1</name> <value>node201:8088</value> </property> <property> <name>yarn.resourcemanager.webapp.address.rm2</name> <value>node200:8088</value> </property> <!-- 配置zookeeper的地址 --> <property> <name>yarn.resourcemanager.zk-address</name> <value>node202:2181,node203:2181,node204:2181</value> </property> <!-- 配置zookeeper的儲存位置 --> <property> <name>yarn.resourcemanager.zk-state-store.parent-path</name> <value>/usr/local/zookeeperdata/rmstore</value> </property> <!-- 開啟yarn resourcemanager restart --> <property> <name>yarn.resourcemanager.recovery.enabled</name> <value>true</value> </property> <!-- 配置resourcemanager的狀態儲存到zookeeper中 --> <property> <name>yarn.resourcemanager.store.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value> </property> <!-- 開啟yarn nodemanager restart --> <property> <name>yarn.nodemanager.recovery.enabled</name> <value>true</value> </property> <!-- 配置nodemanager IPC的通訊埠 --> <property> <name>yarn.nodemanager.address</name> <value>0.0.0.0:45454</value> </property> </configuration>
6、修改slaves檔案
[[email protected] hadoop]$ vi slaves配置檔案如下：
node202 node203 node204
可以直接下載下來，通過上傳到Linux直接覆蓋原來檔案的方式進行配置
7、通過scp將配置檔案上傳到其他主機
[[email protected] hadoop]$ scp -r /usr/local/hadoop/etc/hadoop/* node201:/usr/local/hadoop/etc/hadoop
[[email protected] hadoop]$ scp -r /usr/local/hadoop/etc/hadoop/* node202:/usr/local/hadoop/etc/hadoop
[[email protected] hadoop]$ scp -r /usr/local/hadoop/etc/hadoop/* node203:/usr/local/hadoop/etc/hadoop
[[email protected] hadoop]$ scp -r /usr/local/hadoop/etc/hadoop/* node204:/usr/local/hadoop/etc/hadoopHadoop叢集的初始化
1、啟動zookeeper叢集（分別在node202、node203和node204上執行）
zkServer.sh startps：zookeeper其他命令
#檢視狀態
zkServer.sh status
#關閉
zkServer.sh stop啟動後檢視狀態如下：
#node202 [[email protected] ~]$ zkServer.sh status ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Mode: follower #node203 [[email protected] ~]$ zkServer.sh status ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Mode: leader #node204 [[email protected] ~]$ zkServer.sh status ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Mode: follower #只有一臺主機為leader，剩下的都是follower #zookeeper配置的主機數一般為2n+1臺，且最少需要3臺
2、格式化ZKFC（在node200上執行）
[[email protected] ~]$ hdfs zkfc -formatZK出現如下程式碼說明執行成功：
18/10/12 09:19:11 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at node200/192.168.33.200:9000 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:host.name=node200 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_181 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_181-amd64/jre 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/usr/local/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.7.3.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/usr/local/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/usr/local/hadoop/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/usr/local/hadoop/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-annotations-2.7.3.jar:/usr/local/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.3.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.3-tests.jar:/usr/local/hadoop/share/hadoop/common/hadoop-nfs-2.7.3.jar:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.3.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.3-tests.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/usr/local/hadoop/contrib/capacity-scheduler/*.jar 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/hadoop/lib/native 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.0-862.el7.x86_64 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=node202:2181,node203:2181,node204:2181 sessionTimeout=5000 watcher[email protected]57a3af25 18/10/12 09:19:12 INFO zookeeper.ClientCnxn: Opening socket connection to server node202/192.168.33.202:2181. Will not attempt to authenticate using SASL (unknown error) 18/10/12 09:19:12 INFO zookeeper.ClientCnxn: Socket connection established to node202/192.168.33.202:2181, initiating session 18/10/12 09:19:12 INFO zookeeper.ClientCnxn: Session establishment complete on server node202/192.168.33.202:2181, sessionid = 0x16665c138780001, negotiated timeout = 5000 18/10/12 09:19:12 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/mycluster in ZK. 18/10/12 09:19:12 INFO zookeeper.ZooKeeper: Session: 0x16665c138780001 closed 18/10/12 09:19:12 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x16665c138780001 18/10/12 09:19:12 INFO zookeeper.ClientCnxn: EventThread shut down
3、啟動journalnode（分別在node202、node203和node204上執行）
hadoop-daemon.sh start journalnode
執行過程如下：
[[email protected] ~]$ hadoop-daemon.sh start journalnode starting journalnode, logging to /usr/local/hadoop/logs/hadoop-hadoop-journalnode-node202.out [[email protected] ~]$ jps 1508 QuorumPeerMain 3239 Jps 3182 JournalNode [[email protected] ~]$ hadoop-daemon.sh start journalnode starting journalnode, logging to /usr/local/hadoop/logs/hadoop-hadoop-journalnode-node203.out [[email protected] ~]$ jps 3289 Jps 1484 QuorumPeerMain 3231 JournalNode [[email protected] ~]$ hadoop-daemon.sh start journalnode starting journalnode, logging to /usr/local/hadoop/logs/hadoop-hadoop-journalnode-node204.out [[email protected] ~]$ jps 3296 Jps 1490 QuorumPeerMain 3237 JournalNode #這裡用jps命令確認JournalNode程序啟動情況 #QuorumPeerMain是第1步操作啟動的zookeeper的程序
4、格式化HDFS（在node200上執行）
[[email protected] ~]$ hdfs namenode -format執行出現如下程式碼說明執行成功：
[[email protected] ~]$ hdfs namenode -format 18/10/12 09:36:22 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = node200/192.168.33.200 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 2.7.3 STARTUP_MSG: classpath = /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/usr/local/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/usr/loc