1. 程式人生 > >YARN 高可用(HA)叢集搭建(CDH)

YARN 高可用(HA)叢集搭建(CDH)

HDFS 高可用叢集安裝完成之後,緊接著快速搞定YARN的叢集搭建。

1.修改YARN配置檔案

1.1修改yarn-site.xml

<configuration>
<property>
	<name>yarn.resourcemanager.connect.retry-interval.ms</name>
	<value>2000</value>
</property>
<property>
	<name>yarn.resourcemanager.ha.enabled</name>
	<value>true</value>
</property>
<!--開啟高可用-->
<property>
	<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
	<value>true</value>
</property>
<!--啟動故障自動恢復-->
<property>
	<name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
	<value>true</value>
</property>
<!--rm啟動內建選舉active-->
<property>
	<name>yarn.resourcemanager.cluster-id</name>
	<value>yarn-rm-cluster</value>
</property>
<!--給yarn cluster 取個名字yarn-rm-cluster-->
<property>
	<name>yarn.resourcemanager.ha.rm-ids</name>
	<value>rm1,rm2</value>
</property>
<!--ResourceManager高可用 rm1,rm2-->
<property>
	<name>yarn.resourcemanager.hostname.rm1</name>
	<value>hadoop01</value>
</property>
<property>
	<name>yarn.resourcemanager.hostname.rm2</name>
	<value>hadoop02</value>
</property>
<property>
	<name>yarn.resourcemanager.recovery.enabled</name>
	<value>true</value>
</property>
<!--啟用resourcemanager 自動恢復-->
<property>
	<name>yarn.resourcemanager.zk.state-store.address</name>
	<value>hadoop01:2181,hadoop02:2181,hadoop03:2181</value>
</property>
<!--狀態儲存地址-->
<property>
	<name>yarn.resourcemanager.zk-address</name>
	<value>hadoop01:2181,hadoop02:2181,hadoop03:2181</value>
</property>
<!--配置Zookeeper地址-->
<property>
	<name>yarn.resourcemanager.address.rm1</name>
	<value>hadoop01:8032</value>
</property>
<!--rm1埠號-->
<property>
	<name>yarn.resourcemanager.scheduler.address.rm1</name>
	<value>hadoop01:8034</value>
</property>
<!-- rm1排程器的埠號-->
<property>
	<name>yarn.resourcemanager.webapp.address.rm1</name>
	<value>hadoop01:8088</value>
</property>
<!-- rm1 webapp埠號-->
<property>
	<name>yarn.resourcemanager.address.rm2</name>
	<value>hadoop02:8032</value>
</property>
<property>
	<name>yarn.resourcemanager.scheduler.address.rm2</name>
	<value>hadoop02:8034</value>
</property>
<property>
	<name>yarn.resourcemanager.webapp.address.rm2</name>
	<value>hadoop02:8088</value>
</property>
<property>
	<name>yarn.nodemanager.aux-services</name>
	<value>mapreduce_shuffle</value>
</property>
<property>
	<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
	<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<!--執行MapReduce需要配置的shuffle過程-->
</configuration>

1.2修改mapred-site.xml

<configuration>
<property>
	<name>mapreduce.framework.name</name>
	<value>yarn</value>
</property>
<!--MapReduce以yarn模式執行-->
</configuration>

2.修改過的配置檔案同步到其他節點

通過如下命令,將yarn相關配置同步到slave節點。

deploy.sh mapred-site.xml /home/hadoop/app/hadoop-2.6.0-hadoop5.10.0/etc/hadoop slave

deploy.sh yarn-site.xml /home/hadoop/app/hadoop-2.6.0-hadoop5.10.0/etc/hadoop slave

3.一鍵啟動YARN叢集

主節點hadoop01通過如下命令一鍵啟動YARN 叢集。

sbin/start-yarn.sh

4.啟動備用節點RM

在備用節點hadoop02單獨啟動ResourceManager程序。

sbin/yarn-daemon.sh start resourcemanager

5.檢視RM狀態

通過如下命令分別檢視rm1 rm2的狀態。

bin/yarn rmadmin -getServiceState rm1

bin/yarn rmadmin -getServiceState rm2

6.web介面檢視yarn

通過如下地址訪問web介面,瞭解YARN相關作業執行資訊。

http://hadoop01:8088

或者

http://hadoop02:8088

7.測試執行YARN

在YARN上面執行Hadoop自帶Wordcount程式

bin/hadoop jar share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.10.0.jar wordcount /test/wd.txt /test/output

通過命令列檢視執行結果

bin/hdfs dfs -cat /test/output/*

如果Wordcount程式能在YARN上成功執行,說明整個Hadoop叢集安裝成功。