Kafka：ZK+Kafka+Spark Streaming叢集環境搭建（十）安裝hadoop2.9.0搭建HA

如何搭建配置centos虛擬機器請參考《Kafka：ZK+Kafka+Spark Streaming叢集環境搭建（一）VMW安裝四臺CentOS，並實現本機與它們能互動，虛擬機器內部實現可以上網。》

如何安裝hadoop2.9.0請參考《Kafka：ZK+Kafka+Spark Streaming叢集環境搭建（二）安裝hadoop2.9.0》

如何配置zookeeper3.4.12 請參考《Kafka：ZK+Kafka+Spark Streaming叢集環境搭建（八）安裝zookeeper-3.4.12》

安裝hadoop的伺服器：

192.168.0.120      master

192.168.0.121      slave1

192.168.0.122      slave2

192.168.0.123      slave3

前言：

配置高可用（HA）叢集在實際生產中可保證叢集安全穩定高效的執行，下面講解一下HA中的各個節點的作用。

1，zookeeper的java程序：QuorumPeermain，各種分散式服務（如hdfs，hbase，resourcemanager）註冊到zookeeper上去，即可啟動2個以上的相同程序（其中一個狀態為active，其他為standby），那麼，其中active節點掛了，可通過zookeeper進行切換到standby，讓他active，保證程序正常執行。

2，namenode：hdfs的主節點，儲存hdfs正常執行的各種必要元資料（儲存在edits和fsimage檔案中），萬一掛了，整個叢集就掛了，所以要配置多namenode。

3，datanode：hdfs的資料節點，儲存真實資料的節點。

4，journalnode：日誌節點，將namenode上的edits檔案分離出來，弄成一個叢集，保證namenode不被掛。

5，zkfc：全稱DFSZKFailoverController，監控並管理namenode的狀態和切換。

6，Resourcemanager：管理並分配叢集的資源，如為nodemanager分配計算資源等。

7，nodemanager：管理datanode，並隨datanode的啟動而啟動。

其實，最好每個節點跑一個程序，奈何機器效能不足，跑不了那麼多虛擬機器，就將幾個節點放在同一個節點上，只將重要的程序（namenode和resourcemanager）放在不同的節點。

搭建叢集前的工作：

首先是系統環境：

如何安裝hadoop2.9.0請參考《Kafka：ZK+Kafka+Spark Streaming叢集環境搭建（二）安裝hadoop2.9.0》

然後搭建zookeeper3.4.12請參考《Kafka：ZK+Kafka+Spark Streaming叢集環境搭建（八）安裝zookeeper-3.4.12》

三，修改配置檔案

先在master節點上修改配置檔案，然後傳到其他節點。

1）停掉hadoop服務並清除slaves上hadoop安裝目錄

master上：關閉hadoop服務，並清除dfs、logs、tmp目錄

cd /opt/hadoop-2.9.

sbin/stop-all.sh

rm -r dfs

rm -r logs

rm -r tmp

slaves上：關閉hadoop服務，清除hadoop安裝目錄

cd /opt/hadoop-2.9.

sbin/stop-all.sh

rm -r *

2）etc/hadoop/hadoop-env.sh

cd /opt/hadoop-2.9./etc/hadoop

vi hadoop-env.sh

export JAVA_HOME = path_to_jdk  #新增配置

3）core-site.xml修改

cd /opt/hadoop-2.9.0/etc/hadoop

vi core-site.xml

<configuration>

    <!-- 指定hdfs的nameservice為ns1 -->

    <property>

        <name>fs.defaultFS</name>

       <!-- <value>hdfs://master:9000/</value>-->

        <value>hdfs://HA</value>

    </property>

    <!-- 指定hadoop臨時目錄 -->

    <property>

         <name>hadoop.tmp.dir</name>

         <value>/opt/hadoop-2.9.0/tmp</value>

         <description>這裡的路徑預設是NameNode、DataNode、JournalNode等存放資料的公共目錄</description>

    </property>

    <property>

        <name>io.file.buffer.size</name>

        <value>131702</value>

    </property>

    <!-- 指定zookeeper地址 -->

    <property>

        <name>ha.zookeeper.quorum</name>

        <value>master:2181,slave1:2181,slave2:2181</value>

        <description>這裡是ZooKeeper叢集的地址和埠。注意，數量一定是奇數，且不少於三個節點</description>

    </property>

    <!-- 下面的配置可解決NameNode連線JournalNode超時異常問題-->

    <property>

        <name>ipc.client.connect.retry.interval</name>

        <value>10000</value>

        <description>Indicates the number of milliseconds a client will wait for before retrying to establish a server connection.

        </description>

    </property>

</configuration>

4）hdfs-site.xml修改

cd /opt/hadoop-2.9.0/etc/hadoop

vi hdfs-site.xml

<configuration>

    <!--指定hdfs的nameservice為HA，需要和core-site.xml中的保持一致 -->

    <property>

        <name>dfs.nameservices</name>

        <value>HA</value>

    </property>

    <!-- HA下面有兩個NameNode，分別是nn1，nn2 -->

    <property>

        <name>dfs.ha.namenodes.HA</name>

        <value>nn1,nn2</value>

    </property>

    <!-- nn1的RPC通訊地址 -->

    <property>

        <name>dfs.namenode.rpc-address.HA.nn1</name>

        <value>master:9000</value>

        <description>9000為HDFS 客戶端接入地址（包括命令列與程式），有的使用8020</description>

    </property>

    <!-- nn1的http通訊地址 -->

    <property>

        <name>dfs.namenode.http-address.HA.nn1</name>

        <value>master:50070</value>

        <description> namenode web的接入地址</description>

    </property>

    <!-- nn2的RPC通訊地址 -->

    <property>

        <name>dfs.namenode.rpc-address.HA.nn2</name>

        <value>slave1:9000</value>

        <description>9000為HDFS 客戶端接入地址（包括命令列與程式），有的使用8020</description>

    </property>

    <!-- nn2的http通訊地址 -->

    <property>

        <name>dfs.namenode.http-address.HA.nn2</name>

        <value>slave1:50070</value>

        <description> namenode web的接入地址</description>

    </property>

    <!-- 指定NameNode的edits元資料在JournalNode上的存放位置 -->

    <property>

        <name>dfs.namenode.shared.edits.dir</name>

        <value>qjournal://master:8485;slave1:8485;slave2:8485/HA</value>

        <description>指定 nn1 的兩個NameNode共享edits檔案目錄時，使用的JournalNode叢集資訊。master\slave1主機中使用這個配置</description>

    </property>

    <!-- 指定JournalNode在本地磁碟存放資料的位置 -->

    <property>

        <name>dfs.journalnode.edits.dir</name>

        <value>/opt/hadoop-2.9.0/tmp/journal</value>

    </property>

    <!-- 開啟NameNode失敗自動切換 -->

    <property>

        <name>dfs.ha.automatic-failover.enabled</name>

        <value>true</value>

    </property>

    <!-- 配置失敗自動切換實現方式 -->

    <property>

        <name>dfs.client.failover.proxy.provider.HA</name>

        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

    </property>

    <!-- 配置隔離機制方法，多個機制用換行分割，即每個機制暫用一行-->

    <property>

        <name>dfs.ha.fencing.methods</name>

        <value>

        sshfence

        shell(/bin/true)

        </value>

    </property>

    <!-- 使用sshfence隔離機制時需要ssh免登陸 -->

    <property>

        <name>dfs.ha.fencing.ssh.private-key-files</name>

        <value>/home/spark/.ssh/id_rsa</value>

        <description>

        [spark@master hadoop]$ cd /home/spark/.ssh/

        [spark@master .ssh]$ ls

        authorized_keys  id_rsa.pub         id_rsa.pub.slave2  known_hosts

        id_rsa           id_rsa.pub.slave1  id_rsa.pub.slave3

        </description>

    </property>

    <!-- 配置sshfence隔離機制超時時間 -->

    <property>

        <name>dfs.ha.fencing.ssh.connect-timeout</name>

        <value>30000</value>

    </property>

        <!--

    <property>

        <name>dfs.namenode.secondary.http-address</name>

        <value>master:9001</value>

    </property>

    <property>

        <name>dfs.namenode.name.dir</name>

        <value>/opt/hadoop-2.9.0/dfs/name</value>

    </property>

    <property>

        <name>dfs.datanode.data.dir</name>

        <value>/opt/hadoop-2.9.0/dfs/data</value>

    </property>

    -->

    <property>

        <name>dfs.replication</name>

        <value>3</value>

        <description>指定DataNode儲存block的副本數量。預設值是3個，我們現在有4個DataNode，該值不大於4即可</description>

    </property>

    <property>

      <name>dfs.blocksize</name>

      <value>134217728</value>

      <description>

          The default block size for new files, in bytes.

          You can use the following suffix (case insensitive):

          k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.),

          Or provide complete size in bytes (such as 134217728 for 128 MB).

          注：1.X及以前版本預設是64M，而且配置項名為dfs.block.size

      </description>

    </property>

    <property>

         <name>dfs.permissions.enabled</name>

         <value>false</value>

         <description>注：如果還有許可權問題，請執行下“/opt/hadoop-2.9.0/bin/hdfs dfs -chmod -R 777 /”命令</description>

    </property>

    <property>

        <name>dfs.webhdfs.enabled</name>

        <value>true</value>

    </property>

</configuration>

注意：這裡可能出現問題：The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe mode will be turned off automatically in 27 seconds.

原因：

hadoop預設情況下在安全模式執行。可以通過下面hadoop dfsadmin -safemode 引數,檢視相關的狀態和設定安全模組是否啟用。

enter 進入安全模式
leave 強制NameNode離開安全模式
get 返回安全模式是否開啟的資訊

wait 等待，一直到安全模式結束。

解決辦法有二種：

《1》可能執行命令hadoop dfsadmin -safemode leave 離開安全模式，但是每次都需要手動去設定。

《2》通過配置dfs.safemode.threshold.pct的引數。預設情況下是0.9990f。這個配置可以在hdfs-defalut.xml中找到。我們可以把這個引數配置為0，永久關閉安全模式。

在hadoop中的hdfs-site.xml新增如下配置：

        <property>

                <name>dfs.safemode.threshold.pct</name>

                <value>0f</value>

                <description>

                        Specifies the percentage of blocks that should satisfy

                        the minimal replication requirement defined by dfs.replication.min.

                        Values less than or equal to 0 mean not to wait for any particular

                        percentage of blocks before exiting safemode.

                        Values greater than 1 will make safe mode permanent.

                </description>

        </property>

重啟NameNode就可以了。

5）mapred-site.xml 修改

cd /opt/hadoop-2.9.0/etc/hadoop

vi mapred-site.xml

<configuration>

    <property>

        <name>mapreduce.framework.name</name>

        <value>yarn</value>

    </property>

    <property>

        <name>mapreduce.jobtracker.http.address</name>

        <value>master:50030</value>

        <description>注：每臺機器上配置都不一樣，需要修改成對應的主機名，埠不用修改，比如 slave1:50030 slave2:50030 slave3:50030 ，拷貝過去後請做相應修改</description>

    </property>

    <property>

        <name>mapreduce.jobhistory.address</name>

        <value>master:10020</value>

        <description>注：每臺機器上配置都不一樣，需要修改成對應的主機名，埠不用修改，比如 slave1:10020、 slave2:10020、 slave3:10020 ，拷貝過去後請做相應修改</description>

    </property>

    <property>

        <name>mapreduce.jobhistory.webapp.address</name>

        <value>master:19888</value>

        <description>注：每臺機器上配置都不一樣，需要修改成對應的主機名，埠不用修改，比如 slave1:19888、 slave2:19888、 slave3:19888 ，拷貝過去後請做相應修改</description>

    </property>

    <property>

        <name>mapred.job.tracker</name>

        <value>http://master:9001</value>

        <description>注：每臺機器上配置都不一樣，需要修改成對應的主機名，埠不用修改，比如 http://slave1:9001 、 http://slave2:9001 、 http://slave3:9001 ，拷貝過去後請做相應修改</description>

    </property>

</configuration>

6）yarn-site.xml 修改

cd /opt/hadoop-2.9.0/etc/hadoop

vi yarn-site.xml

<configuration>

    <!--

    <property>

        <name>yarn.resourcemanager.address</name>

        <value>master:8032</value>

    </property>

    <property>

        <name>yarn.resourcemanager.scheduler.address</name>

        <value>master:8030</value>

    </property>

    <property>

        <name>yarn.resourcemanager.resource-tracker.address</name>

        <value>master:8035</value>

    </property>

    <property>

        <name>yarn.resourcemanager.admin.address</name>

        <value>master:8033</value>

    </property>

    <property>

        <name>yarn.resourcemanager.webapp.address</name>

        <value>master:8088</value>

    </property>

    <property>

        <name>yarn.resourcemanager.hostname</name>

        <value>master</value>

    </property>

    <property>

        <name>yarn.nodemanager.resource.memory-mb</name>

        <value>2048</value>

    </property>

    <property>

        <name>yarn.nodemanager.pmem-check-enabled</name>

        <value>false</value>

    </property>

    <property>

        <name>yarn.nodemanager.vmem-check-enabled</name>

        <value>false</value>

        <description>Whether virtual memory limits will be enforced for containers</description>

    </property>

    <property>

        <name>yarn.nodemanager.vmem-pmem-ratio</name>

        <value>4</value>

        <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>

    </property>

    -->

    <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

    <property>

        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>

        <value>org.apache.hadoop.mapred.ShuffleHandler</value>

    </property>

    <!-- 開啟RM高可用 -->

    <property>

        <name>yarn.resourcemanager.ha.enabled</name>

        <value>true</value>

    </property>

    <!-- 指定RM的cluster id -->

    <property>

        <name>yarn.resourcemanager.cluster-id</name>

        <value>yarn-cluster</value>

    </property>

    <!-- 分別指定RM的地址 -->

    <property>

        <name>yarn.resourcemanager.ha.rm-ids</name>

        <value>rm1,rm2</value>

    </property>

    <property>

        <name>yarn.resourcemanager.hostname.rm1</name>

        <value>master</value>

    </property>

    <property>

        <name>yarn.resourcemanager.hostname.rm2</name>

        <value>slave1</value>

    </property>

    <property>

        <name>yarn.resourcemanager.webapp.address.rm1</name>

        <value>master:8088</value>

    </property>

    <property>

        <name>yarn.resourcemanager.webapp.address.rm2</name>

        <value>slave1:8088</value>

    </property>

    <!-- 指定zk叢集地址 -->

    <property>

        <name>yarn.resourcemanager.zk-address</name>

        <value>master:2181,slave1:2181,slave2:2181</value>

    </property>

    <property>

        <name>yarn.resourcemanager.recovery.enabled</name>

        <value>true</value>

    </property>

    <property>

        <name>yarn.resourcemanager.store.class</name>

        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>

        <description>RM的資料預設存放在ZK上的/rmstore中，可通過yarn.resourcemanager.zk-state-store.parent-path 設定</description>

    </property>

    <property>

        <name>yarn.log-aggregation-enable</name>

        <value>true</value>

        <description>開啟日誌收集，這樣會將每臺執行任務的機上產生的本地日誌檔案集中拷貝到HDFS的某個地方，這樣就可以在任何一臺叢集中的機器上集中檢視作業日誌了</description>

    </property>

    <property>

        <name>yarn.log.server.url</name>

        <value>http://master:19888/jobhistory/logs</value>

        <description>注：每臺機器上配置都不一樣，需要修改成對應的主機名，埠不用修改，比如http://slave1:19888/jobhistory/logs、http://slave2:19888/jobhistory/logs、http://slave3:19888/jobhistory/logs，拷貝過去後請做相應修改</description>

    </property>

</configuration>

7） slaves 修改

cd /opt/hadoop-2.9./etc/hadoop

vi slaves

[spark@master hadoop]$ more slaves

#localhost

slave1

slave2

slave3

8）配置好後，將配置檔案傳到其他節點上

scp -r /opt/hadoop-2.9. spark@slave1:/opt/

scp -r /opt/hadoop-2.9. spark@slave2:/opt/

scp -r /opt/hadoop-2.9. spark@slave3:/opt/

分別在slave1,slave2,slave3上修改配置：

修改yarn-site.xml

cd /opt/hadoop-2.9./etc/hadoop

vi yarn-site.xml

修改內容

    <property>

        <name>yarn.log.server.url</name>

        <value>http://master:19888/jobhistory/logs</value>

        <description>注：每臺機器上配置都不一樣，需要修改成對應的主機名，埠不用修改，比如http://slave1:19888/jobhistory/logs、http://slave2:19888/jobhistory/logs、http://slave3:19888/jobhistory/logs，拷

貝過去後請做相應修改</description>

    </property>

修改mapred-site.xml

cd /opt/hadoop-2.9.

vi mapred-site.xml

修改內容

<configuration>

    <property>

        <name>mapreduce.jobtracker.http.address</name>

        <value>master:50030</value>

        <description>注：每臺機器上配置都不一樣，需要修改成對應的主機名，埠不用修改，比如 slave1:50030 slave2:50030 slave3:50030 ，拷貝過去後請做相應修改</description>

    </property>

    <property>

        <name>mapreduce.jobhistory.address</name>

        <value>master:10020</value>

        <description>注：每臺機器上配置都不一樣，需要修改成對應的主機名，埠不用修改，比如 slave1:10020、 slave2:10020、 slave3:10020 ，拷貝過去後請做相應修改</description>

    </property>

    <property>

        <name>mapreduce.jobhistory.webapp.address</name>

        <value>master:19888</value>

        <description>注：每臺機器上配置都不一樣，需要修改成對應的主機名，埠不用修改，比如 slave1:19888、 slave2:19888、 slave3:19888 ，拷貝過去後請做相應修改</description>

    </property>

    <property>

        <name>mapred.job.tracker</name>

        <value>http://master:9001</value>

        <description>注：每臺機器上配置都不一樣，需要修改成對應的主機名，埠不用修改，比如 http://slave1:9001 、 http://slave2:9001 、 http://slave3:9001 ，拷貝過去後請做相應修改</description>

    </property>

</configuration>

初始化叢集，嚴格按照一下步驟！

1，在3臺節點（master，slave1，slave2）啟動zookeeper叢集，jps檢視有沒有啟動。

cd /opt/zookeeper-3.4./bin

./zkServer.sh start

./zkServer.sh status

master

[spark@master hadoop-2.9.]$ cd /opt/zookeeper-3.4./bin

[spark@master bin]$ ./zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /opt/zookeeper-3.4./bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

[spark@master bin]$

[spark@master bin]$ ./zkServer.sh status

ZooKeeper JMX enabled by default

Using config: /opt/zookeeper-3.4./bin/../conf/zoo.cfg

Mode: follower

slave1

[spark@slave1 hadoop]$ cd /opt/zookeeper-3.4./bin

[spark@slave1 bin]$ ./zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /opt/zookeeper-3.4./bin/../conf/zoo.cfg

./zkServer.sh status

Starting zookeeper ... STARTED

[spark@slave1 bin]$

[spark@slave1 bin]$ ./zkServer.sh status

ZooKeeper JMX enabled by default

Using config: /opt/zookeeper-3.4./bin/../conf/zoo.cfg

Mode: leader

slave2

[spark@slave2 hadoop]$ cd /opt/zookeeper-3.4./bin

[spark@slave2 bin]$ ./zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /opt/zookeeper-3.4./bin/../conf/zoo.cfg

Starting zookeeper ... ./zkServer.sh status

STARTED

[spark@slave2 bin]$

[spark@slave2 bin]$ ./zkServer.sh status

ZooKeeper JMX enabled by default

Using config: /opt/zookeeper-3.4./bin/../conf/zoo.cfg

Mode: follower

2，在3臺節點（master，slave1，slave2）啟動journalnode叢集，jps檢視有沒有啟動。

cd /opt/hadoop-2.9.

sbin/hadoop-daemon.sh start journalnode

master

[spark@master hadoop]$ cd /opt/hadoop-2.9.

[spark@master hadoop-2.9.]$ sbin/hadoop-deamon.sh start journalnode

starting journalnode, logging to /opt/hadoop-2.9./logs/hadoop-spark-journalnode-master.out

[spark@master hadoop-2.9.]$ jps

 Jps

 JournalNode

 QuorumPeerMain

[spark@master hadoop-2.9.]$

slave1

[spark@slave1 bin]$ cd /opt/hadoop-2.9.

[spark@slave1 hadoop-2.9.]$ sbin/hadoop-daemon.sh start journalnode

starting journalnode, logging to /opt/hadoop-2.9./logs/hadoop-spark-journalnode-slave1.out

[spark@slave1 hadoop-2.9.]$ jps

 JournalNode

 Jps

 QuorumPeerMain

[spark@slave1 hadoop-2.9.]$

slave2

[spark@slave2 bin]$ cd /opt/hadoop-2.9.

[spark@slave2 hadoop-2.9.]$ sbin/hadoop-daemon.sh start journalnode

starting journalnode, logging to /opt/hadoop-2.9./logs/hadoop-spark-journalnode-slave2.out

[spark@slave2 hadoop-2.9.]$ jps

 QuorumPeerMain

 JournalNode

 Jps

[spark@slave2 hadoop-2.9.]$

3，在 master 上格式化hdfs

cd /opt/hadoop-2.9.

bin/hdfs namenode -format   #等同 bin/hadoop namenode -format

[spark@master HA]$ cd /opt/hadoop-2.9.

[spark@master hadoop-2.9.]$ bin/hdfs namenode -format

// :: INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = master/192.168.0.120

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 2.9.0

STARTUP_MSG:   classpath = /opt/hadoop-2.9.0/etc/hadoop:/opt/hadoop-2.9.0/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/opt/hadoop-2.9.0/share/hadoop/common/li....

STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 756ebc8394e473ac25feac05fa493f6d612e6c50; compiled by 'arsuresh' on 2017-11-13T23:15Z

STARTUP_MSG:   java = 1.8.0_171

************************************************************/

// :: INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]

// :: INFO namenode.NameNode: createNameNode [-format]

Formatting using clusterid: CID-01f8ecdf-532b--807e-90b4aa179e29

// :: INFO namenode.FSEditLog: Edit logging is async:true

// :: INFO namenode.FSNamesystem: KeyProvider: null

// :: INFO namenode.FSNamesystem: fsLock is fair: true

// :: INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false

// :: INFO namenode.FSNamesystem: fsOwner             = spark (auth:SIMPLE)

// :: INFO namenode.FSNamesystem: supergroup          = supergroup

// :: INFO namenode.FSNamesystem: isPermissionEnabled = false

// :: INFO namenode.FSNamesystem: Determined nameservice ID: HA

// :: INFO namenode.FSNamesystem: HA Enabled: true

// :: INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to . Disabling file IO profiling

// :: INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=, counted=, effected=

// :: INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true

// :: INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to :::00.000

// :: INFO blockmanagement.BlockManager: The block deletion will start around  Jul  ::

// :: INFO util.GSet: Computing capacity for map BlocksMap

// :: INFO util.GSet: VM type       = -bit

// :: INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB

// :: INFO util.GSet: capacity      = ^ =  entries

// :: INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false

// :: WARN conf.Configuration: No unit for dfs.namenode.safemode.extension() assuming MILLISECONDS

// :: INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033

// :: INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes =

// :: INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension =

// :: INFO blockmanagement.BlockManager: defaultReplication         =

// :: INFO blockmanagement.BlockManager: maxReplication             =

// :: INFO blockmanagement.BlockManager: minReplication             =

// :: INFO blockmanagement.BlockManager: maxReplicationStreams      =

// :: INFO blockmanagement.BlockManager: replicationRecheckInterval =

// :: INFO blockmanagement.BlockManager: encryptDataTransfer        = false

// :: INFO blockmanagement.BlockManager: maxNumBlocksToLog          =

// :: INFO namenode.FSNamesystem: Append Enabled: true

// :: INFO util.GSet: Computing capacity for map INodeMap

// :: INFO util.GSet: VM type       = -bit

// :: INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB

// :: INFO util.GSet: capacity      = ^ =  entries

// :: INFO namenode.FSDirectory: ACLs enabled? false

// :: INFO namenode.FSDirectory: XAttrs enabled? true

// :: INFO namenode.NameNode: Caching file names occurring more than  times

// :: INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: falseskipCaptureAccessTimeOnlyChange: false

// :: INFO util.GSet: Computing capacity for map cachedBlocks

// :: INFO util.GSet: VM type       = -bit

// :: INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB

// :: INFO util.GSet: capacity      = ^ =  entries

// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets =

// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users =

// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = ,,

// :: INFO namenode.FSNamesystem: Retry cache on namenode is enabled

// :: INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is  millis

// :: INFO util.GSet: Computing capacity for map NameNodeRetryCache

// :: INFO util.GSet: VM type       = -bit

// :: INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB

// :: INFO util.GSet: capacity      = ^ =  entries

Re-format filesystem in Storage Directory /opt/hadoop-2.9./tmp/dfs/name ? (Y or N) y

Re-format filesystem in QJM to [192.168.0.120:, 192.168.0.121:, 192.168.0.122:] ? (Y or N) y

// :: INFO namenode.FSImage: Allocated new BlockPoolId: BP--192.168.0.120-

// :: INFO common.Storage: Will remove files: [/opt/hadoop-2.9./tmp/dfs/name/current/VERSION, /opt/hadoop-2.9./tmp/dfs/name/current/seen_txid]

// :: INFO common.Storage: Storage directory /opt/hadoop-2.9./tmp/dfs/name has been successfully formatted.

// :: INFO namenode.FSImageFormatProtobuf: Saving image file /opt/hadoop-2.9./tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression

// :: INFO namenode.FSImageFormatProtobuf: Image file /opt/hadoop-2.9./tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size  bytes saved in  seconds.

// :: INFO namenode.NNStorageRetentionManager: Going to retain  images with txid >=

// :: INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at master/192.168.0.120

************************************************************/

4，配置 slave1 上的namenode

先在 master 上啟動namenode

cd /opt/hadoop-2.9.

sbin/hadoop-daemon.sh start namenode

[spark@master hadoop-2.9.]$ cd /opt/hadoop-2.9.

[spark@master hadoop-2.9.]$ sbin/hadoop-daemon.sh start namenode

starting namenode, logging to /opt/hadoop-2.9./logs/hadoop-spark-namenode-master.out

[spark@master hadoop-2.9.]$ jps

 Jps

 NameNode

 JournalNode

 QuorumPeerMain

[spark@master hadoop-2.9.]$

再在 slave1 上執行命令：

cd /opt/hadoop-2.9.

bin/hdfs namenode -bootstrapStandby

[spark@slave1 HA]$ cd /opt/hadoop-2.9.

[spark@slave1 hadoop-2.9.]$ bin/hdfs namenode -bootstrapStandby

// :: INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = slave1/192.168.0.121

STARTUP_MSG:   args = [-bootstrapStandby]

STARTUP_MSG:   version = 2.9.0

STARTUP_MSG:   classpath = /opt/hadoop-2.9.0/etc/hadoop:/opt/hadoop-2.9.0/share/hadoop/common/lib...

STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 756ebc8394e473ac25feac05fa493f6d612e6c50; compiled by 'arsuresh' on 2017-11-13T23:15Z

STARTUP_MSG:   java = 1.8.0_171

************************************************************/

// :: INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]

// :: INFO namenode.NameNode: createNameNode [-bootstrapStandby]

=====================================================

About to bootstrap Standby ID nn2 from:

           Nameservice ID: HA

        Other Namenode ID: nn1

  Other NN's HTTP address: http://master:50070

  Other NN's IPC  address: master/192.168.0.120:9000

             Namespace ID:

            Block pool ID: BP--192.168.0.120-

               Cluster ID: CID-01f8ecdf-532b--807e-90b4aa179e29

           Layout version: -

       isUpgradeFinalized: true

=====================================================

// :: INFO common.Storage: Storage directory /opt/hadoop-2.9./tmp/dfs/name has been successfully formatted.

// :: INFO namenode.FSEditLog: Edit logging is async:true

// :: INFO namenode.TransferFsImage: Opening connection to http://master:50070/imagetransfer?getimage=1&txid=0&storageInfo=-63:1875840257:1530455179314:CID-01f8ecdf-532b-4415-807e-90b4aa179e29&bootstrapstandby=true

// :: INFO namenode.TransferFsImage: Image Transfer timeout configured to  milliseconds

// :: INFO namenode.TransferFsImage: Combined time for fsimage download and fsync to all disks took .02s. The fsimage download took .01s at 0.00 KB/s. Synchronous (fsync) write to disk of /opt/hadoop-2.9./tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 took .00s.

// :: INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size  bytes.

// :: INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at slave1/192.168.0.121

************************************************************/

5，格式化ZKFC（在 master 上執行一次即可）

cd /opt/hadoop-2.9.

bin/hdfs zkfc -formatZK

[spark@master hadoop-2.9.]$ cd /opt/hadoop-2.9.

[spark@master hadoop-2.9.]$ bin/hdfs zkfc -formatZK

// :: INFO tools.DFSZKFailoverController: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting DFSZKFailoverController

STARTUP_MSG:   host = master/192.168.0.120

STARTUP_MSG:   args = [-formatZK]

STARTUP_MSG:   version = 2.9.0

STARTUP_MSG:   classpath = /opt/hadoop-2.9.0/etc/hadoop:/opt/hadoop-2.9.0/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/opt/hadoop-2.9.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar....

STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 756ebc8394e473ac25feac05fa493f6d612e6c50; compiled by 'arsuresh' on 2017-11-13T23:15Z

STARTUP_MSG:   java = 1.8.0_171

************************************************************/

// :: INFO tools.DFSZKFailoverController: registered UNIX signal handlers for [TERM, HUP, INT]

// :: INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at master/192.168.0.120:

// :: INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.-, built on // : GMT

// :: INFO zookeeper.ZooKeeper: Client environment:host.name=master

// :: INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_171

// :: INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation

// :: INFO zookeeper.ZooKeeper: Client environment:java.home=/opt/jdk1..0_171/jre

// :: INFO zookeeper.ZooKeeper: Client environment:java.class.path=/opt/hadoop-2.9./etc/hadoop:/opt/hadoop-2.9./share/hadoop/common/lib/nimbu.....

// :: INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/hadoop-2.9./lib/native

// :: INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp

// :: INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>

// :: INFO zookeeper.ZooKeeper: Client environment:os.name=Linux

// :: INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64

// :: INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.-.el7.x86_64

// :: INFO zookeeper.ZooKeeper: Client environment:user.name=spark

// :: INFO zookeeper.ZooKeeper: Client environment:user.home=/home/spark

// :: INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/hadoop-2.9.

// :: INFO zookeeper.ZooKeeper: Initiating client connection, connectString=master:,slave1:,slave2: sessionTimeout= watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@6ca8564a

// :: INFO zookeeper.ClientCnxn: Opening socket connection to server master/192.168.0.120:. Will not attempt to authenticate using SASL (unknown error)

// :: INFO zookeeper.ClientCnxn: Socket connection established to master/192.168.0.120:, initiating session

// :: INFO zookeeper.ClientCnxn: Session establishment complete on server master/192.168.0.120:, sessionid = 0x100006501250000, negotiated timeout =

// :: INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/HA in ZK.

// :: INFO zookeeper.ZooKeeper: Session: 0x100006501250000 closed

// :: WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x100006501250000

// :: INFO zookeeper.ClientCnxn: EventThread shut down

// :: INFO tools.DFSZKFailoverController: SHUTDOWN_MSG:

/************************************************************

現在可以啟動叢集啦

1，首先啟動zookeeper（在master,slave1,slave2）

cd /opt/zookeeper-3.4./bin

./zkServer.sh start

2，在master上啟動hdfs和yarn

cd /opt/hadoop-2.9.

sbin/start-dfs.sh

sbin/start-yarn.sh

[spark@master sbin]$ cd /opt/hadoop-2.9.

[spark@master hadoop-2.9.]$ sbin/start-dfs.sh

sbin/start-yarn.shStarting namenodes on [master slave1]

slave1: starting namenode, logging to /opt/hadoop-2.9./logs/hadoop-spark-namenode-slave1.out

master: namenode running as process . Stop it first.

slave3: starting datanode, logging to /opt/hadoop-2.9./logs/hadoop-spark-datanode-slave3.out

slave2: starting datanode, logging to /opt/hadoop-2.9./logs/hadoop-spark-datanode-slave2.out

slave1: starting datanode, logging to /opt/hadoop-2.9./logs/hadoop-spark-datanode-slave1.out

Starting journal nodes [master slave1 slave2]

slave2: journalnode running as process . Stop it first. # 前邊我們已經啟動過了，是手動啟動的。

master: journalnode running as process . Stop it first.    # 前邊我們已經啟動過了，是手動啟動的。

slave1: journalnode running as process . Stop it first.    # 前邊我們已經啟動過了，是手動啟動的。

Starting ZK Failover Controllers on NN hosts [master slave1]

master: starting zkfc, logging to /opt/hadoop-2.9./logs/hadoop-spark-zkfc-master.out

slave1: starting zkfc, logging to /opt/hadoop-2.9./logs/hadoop-spark-zkfc-slave1.out

[spark@master hadoop-2.9.]$ sbin/start-yarn.sh

starting yarn daemons

starting resourcemanager, logging to /opt/hadoop-2.9./logs/yarn-spark-resourcemanager-master.out

slave1: starting nodemanager, logging to /opt/hadoop-2.9./logs/yarn-spark-nodemanager-slave1.out

slave2: starting nodemanager, logging to /opt/hadoop-2.9./logs/yarn-spark-nodemanager-slave2.out

slave3: starting nodemanager, logging to /opt/hadoop-2.9./logs/yarn-spark-nodemanager-slave3.out

[spark@master hadoop-2.9.]$ jps

 DFSZKFailoverController

 NameNode

 ResourceManager

 Jps

 JournalNode

 QuorumPeerMain

[spark@master hadoop-2.9.]$

3，slave1 上的resourcemanager需要手動啟動

cd /opt/hadoop-2.9.

sbin/yarn-deamon.sh start resourcemanager

[spark@slave1 hadoop-2.9.]$ jps

 JournalNode

 DataNode

 NodeManager

 Jps

 DFSZKFailoverController

 QuorumPeerMain

 NameNode

[spark@slave1 sbin]$ cd /opt/hadoop-2.9.

[spark@slave1 hadoop-2.9.]$ cd sbin/

[spark@slave1 sbin]$ ls

distribute-exclude.sh hadoop-daemons.sh httpfs.sh refresh-namenodes.sh start-all.sh start-dfs.sh start-yarn.sh stop-balancer.sh stop-secure-dns.sh yarn-daemon.sh

FederationStateStore hdfs-config.cmd kms.sh slaves.sh start-balancer.sh start-secure-dns.sh stop-all.cmd stop-dfs.cmd stop-yarn.cmd yarn-daemons.sh

hadoop-daemon.sh hdfs-config.sh mr-jobhistory-daemon.sh start-all.cmd start-dfs.cmd start-yarn.cmd stop-all.sh stop-dfs.sh stop-yarn.sh

[spark@slave1 sbin]$ ./yarn-daemon.sh start resourcemanager

starting resourcemanager, logging to /opt/hadoop-2.9./logs/yarn-spark-resourcemanager-slave1.out

[spark@slave1 sbin]$ jps

 ResourceManager

 JournalNode

 DataNode

 Jps

 NodeManager

 DFSZKFailoverController

 QuorumPeerMain

 NameNode

[spark@slave1 sbin]$

此時slave2

[spark@slave2 HA]$ jps

 DataNode

 Jps

 NodeManager

 QuorumPeerMain

 JournalNode

[spark@slave2 HA]$

此時slave3

[spark@slave3 hadoop-2.9.]$ jps

 Jps

 NodeManager

 DataNode

[spark@slave3 hadoop-2.9.]$

至此，高可用hadoop配置完畢，可訪問瀏覽器：

http://192.168.0.120:50070 //namenode active

http://192.168.0.121:50070 //namenode standby

http://192.168.0.120:8088 //resourcemanager active

http://192.168.0.121:8088 //resourcemanager standby

此時 http://192.168.0.120:50070/可以訪問（顯示為： 'master:9000' (active)）,http://192.168.0.121:50070/可以訪問（顯示為： 'slave1:9000' (standby)）

此時 http://192.168.0.121:8088/ 不可以訪問，而http://192.168.0.120:8088/ 可以訪問。

關閉的時候按相反的順序關閉

高可用驗證

1，驗證hdfs

首先kill掉 master 上active的namenode
kill -9 <pid of NN>

[spark@master hadoop-2.9.]$ jps

 DFSZKFailoverController

 NameNode

 ResourceManager

 JournalNode

 QuorumPeerMain

 Jps

[spark@master hadoop-2.9.]$ kill -

[spark@master hadoop-2.9.]$ jps

 DFSZKFailoverController

 ResourceManager

 Jps

 JournalNode

 QuorumPeerMain

[spark@master hadoop-2.9.]$

然後檢視 slave1 上狀態為standby的namenode狀態變成active。

此時 slave1 jps

[spark@slave1 sbin]$ jps

 ResourceManager

 Jps

 JournalNode

 DataNode

 NodeManager

 DFSZKFailoverController

 QuorumPeerMain

 NameNode

[spark@slave1 sbin]$

此時 http://192.168.0.120:50070/ 不可以訪問,http://192.168.0.121:50070/可以訪問（顯示為： 'slave1:9000' (active)）
此時 http://192.168.0.121:8088/ 不可以訪問，而http://192.168.0.120:8088/ 可以訪問。

2，驗證yarn

同上，kill掉active狀態的rm後，standby狀態下的rm即變成active，繼續工作。

[spark@master hadoop-2.9.]$ jps

 DFSZKFailoverController

 Jps

 ResourceManager

 JournalNode

 QuorumPeerMain

[spark@master hadoop-2.9.]$ kill -

[spark@master hadoop-2.9.]$ jps

 DFSZKFailoverController

 JournalNode

 QuorumPeerMain

 Jps

此時 http://192.168.0.121:8088/ 可以訪問，而http://192.168.0.120:8088/ 不可以訪問。

http://192.168.0.120:50070/dfshealth.html#tab-overview

http://192.168.0.121:50070/dfshealth.html#tab-overview

http://192.168.0.120:8088/cluster/apps/RUNNING

http://192.168.0.121:8088/cluster/apps/ACCEPTED

參考《https://www.cnblogs.com/jiangzhengjun/p/6347916.html》

《https://blog.csdn.net/qq_32166627/article/details/51553394》

Kafka：ZK+Kafka+Spark Streaming叢集環境搭建（十）安裝hadoop2.9.0搭建HA