27課 ：SPARK 執行在yarn資源排程框架 client 、cluster方式 ！！
阿新 • • 發佈：2018-12-29
分散式叢集
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# vi /etc/hosts
127.0.0.1 localhost
192.168.189.1 master
192.168.189.2 worker1
192.168.189.3 worker2
192.168.189.4 worker3
192.168.189.5 worker4
192.168.189.6 worker5
192.168.189.7 worker6
192.168.189.8 worker7
192.168.189.9 worker8
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# cat core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-2.6.0/tmp</value>
<description>hadoop.tmp.dir</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
</configuration>
worker1
worker2
worker3
worker4
worker5
worker6
worker7
worker8
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# cat hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/hadoop-2.6.0/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value> /usr/local/hadoop-2.6.0/tmp/dfs/data</value>
</property>
</configuration>
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# ls | grep map
mapred-env.cmd
mapred-env.sh
mapred-queues.xml.template
mapred-site.xml
mapred-site.xml.template
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop#
修改的 地方
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# cat mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# cat yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop#
[email protected]:/usr/local/setup_scripts# cat yarn_scp.sh
#!/bin/sh
for i in 2 3 4 5 6 7 8 9
do
scp -rq /usr/local/hadoop-2.6.0/etc/hadoop/mapred-site.xml [email protected]$i:/usr/local/hadoop-2.6.0/etc/hadoop/
scp -rq /usr/local/hadoop-2.6.0/etc/hadoop/yarn-site.xml [email protected]$i:/usr/local/hadoop-2.6.0/etc/hadoop/
done
[email protected]:/usr/local/setup_scripts#
[email protected]:/usr/local/setup_scripts# ./yarn_scp.sh
啟動hdfs叢集
[email protected]:/usr/local/hadoop-2.6.0/sbin# start-dfs.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-namenode-master.out
worker5: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker5.out
worker8: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker8.out
worker4: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker4.out
worker3: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker3.out
worker2: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker2.out
worker7: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker7.out
worker6: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker6.out
worker1: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker1.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-secondarynamenode-master.out
[email protected]:/usr/local/hadoop-2.6.0/sbin# jps
4080 SecondaryNameNode
4182 Jps
3863 NameNode
[email protected]:/usr/local/hadoop-2.6.0/sbin#
[email protected]:~# jps
2916 DataNode
2983 Jps
[email protected]:~#
啟動yarn叢集
[email protected]:/usr/local/hadoop-2.6.0/sbin# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-resourcemanager-master.out
worker2: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker2.out
worker7: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker7.out
worker4: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker4.out
worker3: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker3.out
worker6: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker6.out
worker5: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker5.out
worker8: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker8.out
worker1: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker1.out
[email protected]:/usr/local/hadoop-2.6.0/sbin# jps
4080 SecondaryNameNode
3863 NameNode
4234 ResourceManager
4493 Jps
[email protected]:/usr/local/hadoop-2.6.0/sbin#
[email protected]:~# jps
2916 DataNode
2983 Jps
[email protected]:~# jps
3171 Jps
2916 DataNode
3053 NodeManager
[email protected]:~#
啟動歷史伺服器
[email protected]:/usr/local/hadoop-2.6.0/sbin# mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /usr/local/hadoop-2.6.0/logs/mapred-root-historyserver-master.out
[email protected]:/usr/local/hadoop-2.6.0/sbin#
[email protected]:/usr/local/hadoop-2.6.0/sbin# jps
4080 SecondaryNameNode
4609 Jps
3863 NameNode
4537 JobHistoryServer
4234 ResourceManager
[email protected]:/usr/local/hadoop-2.6.0/sbin#
[email protected]:~# jps
2916 DataNode
3053 NodeManager
3183 Jps
[email protected]:~#
啟動SPARK
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# ./start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.out
worker3: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker3.out
worker5: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker5.out
worker7: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker7.out
worker6: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker6.out
worker1: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker1.out
worker4: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker4.out
worker2: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker2.out
worker8: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker8.out
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# jps
4080 SecondaryNameNode
4802 Jps
3863 NameNode
4537 JobHistoryServer
4234 ResourceManager
4747 Master
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin#
[email protected]:~# jps
2916 DataNode
3286 Worker
3337 Jps
3053 NodeManager
[email protected]:~#
啟動spark history
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# ./start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-master.out
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# jps
4080 SecondaryNameNode
4930 Jps
4883 HistoryServer
3863 NameNode
4537 JobHistoryServer
4234 ResourceManager
4747 Master
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin#
http://192.168.189.1:19888/jobhistory/app
http://192.168.189.1:8088/cluster/nodes
http://192.168.189.1:18080/
http://192.168.189.1:50070/dfshealth.html#tab-overview
關掉SPARK叢集，只執行yarn叢集，SPARK只做計算
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# jps
4080 SecondaryNameNode
4930 Jps
4883 HistoryServer
3863 NameNode
4537 JobHistoryServer
4234 ResourceManager
4747 Master
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# ./stop-all.sh
worker7: stopping org.apache.spark.deploy.worker.Worker
worker2: stopping org.apache.spark.deploy.worker.Worker
worker6: stopping org.apache.spark.deploy.worker.Worker
worker4: stopping org.apache.spark.deploy.worker.Worker
worker5: stopping org.apache.spark.deploy.worker.Worker
worker8: stopping org.apache.spark.deploy.worker.Worker
worker3: stopping org.apache.spark.deploy.worker.Worker
worker1: stopping org.apache.spark.deploy.worker.Worker
stopping org.apache.spark.deploy.master.Master
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# jps
4080 SecondaryNameNode
4883 HistoryServer
3863 NameNode
4537 JobHistoryServer
4234 ResourceManager
5055 Jps
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin#
[email protected]:/usr/local/hadoop-2.6.0/sbin# jps
3645 NodeManager
4093 Jps
3517 DataNode
[email protected]:/usr/local/hadoop-2.6.0/sbin#
執行SPARKPI
CLINET模式
./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client /usr/local/spark-1.6.0-bin-hadoop2.6/lib/spark-examples-1.6.0-hadoop2.6.0.jar 1000
cluster模式
./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /usr/local/spark-1.6.0-bin-hadoop2.6/lib/spark-examples-1.6.0-hadoop2.6.0.jar 1000
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/bin# ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /usr/local/spark-1.6.0-bin-hadoop2.6/lib/spark-examples-1.6.0-hadoop2.6.0.jar 1000
16/03/05 20:56:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/03/05 20:56:52 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.189.1:8032
16/03/05 20:56:53 INFO yarn.Client: Requesting a new application from cluster with 8 NodeManagers
16/03/05 20:56:53 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
16/03/05 20:56:53 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
16/03/05 20:56:53 INFO yarn.Client: Setting up container launch context for our AM
16/03/05 20:56:53 INFO yarn.Client: Setting up the launch environment for our AM container
16/03/05 20:56:53 INFO yarn.Client: Preparing resources for our AM container
16/03/05 20:56:55 INFO yarn.Client: Uploading resource file:/usr/local/spark-1.6.0-bin-hadoop2.6/lib/spark-assembly-1.6.0-hadoop2.6.0.jar -> hdfs://master:9000/user/root/.sparkStaging/application_1457178371090_0002/spark-assembly-1.6.0-hadoop2.6.0.jar
16/03/05 20:57:02 INFO yarn.Client: Uploading resource file:/usr/local/spark-1.6.0-bin-hadoop2.6/lib/spark-examples-1.6.0-hadoop2.6.0.jar -> hdfs://master:9000/user/root/.sparkStaging/application_1457178371090_0002/spark-examples-1.6.0-hadoop2.6.0.jar
16/03/05 20:57:07 INFO yarn.Client: Uploading resource file:/tmp/spark-fd52bb0e-ff6b-4587-a407-254abbb52aac/__spark_conf__7795016576129936125.zip -> hdfs://master:9000/user/root/.sparkStaging/application_1457178371090_0002/__spark_conf__7795016576129936125.zip
16/03/05 20:57:07 INFO spark.SecurityManager: Changing view acls to: root
16/03/05 20:57:07 INFO spark.SecurityManager: Changing modify acls to: root
16/03/05 20:57:07 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
16/03/05 20:57:08 INFO yarn.Client: Submitting application 2 to ResourceManager
16/03/05 20:57:08 INFO impl.YarnClientImpl: Submitted application application_1457178371090_0002
16/03/05 20:57:09 INFO yarn.Client: Application report for application_1457178371090_0002 (state: ACCEPTED)
16/03/05 20:57:09 INFO yarn.Client:
./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /usr/local/spark-1.6.0-bin-hadoop2.6/lib/spark-examples-1.6.0-hadoop2.6.0.jar 1000
16/03/05 21:32:16 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.189.4
ApplicationMaster RPC port: 0
queue: default
start time: 1457184716828
final status: UNDEFINED
tracking URL: http://master:8088/proxy/application_1457184438410_0001/
user: root
16/03/05 21:32:17 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:18 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:19 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:20 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:21 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:22 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:23 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:24 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:25 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:26 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:27 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:28 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:29 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:30 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:31 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:32 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:33 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:34 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:35 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:36 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:37 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:38 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:39 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:40 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:41 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:42 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:43 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:44 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:45 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:46 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:47 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:48 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:49 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:50 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:51 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:52 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:53 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:54 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:55 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:56 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:57 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:58 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:59 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:33:00 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:33:01 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:33:02 INFO yarn.Client: Application report for application_1457184438410_0001 (state: FINISHED)
16/03/05 21:33:02 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.189.4
ApplicationMaster RPC port: 0
queue: default
start time: 1457184716828
final status: SUCCEEDED
tracking URL: http://master:8088/proxy/application_1457184438410_0001/A
user: root
16/03/05 21:33:02 INFO util.ShutdownHookManager: Shutdown hook called
16/03/05 21:33:02 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-99c20d66-4fe3-4229-bac3-45affca56ce1
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/bin#
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# vi /etc/hosts
127.0.0.1 localhost
192.168.189.1 master
192.168.189.2 worker1
192.168.189.3 worker2
192.168.189.4 worker3
192.168.189.5 worker4
192.168.189.6 worker5
192.168.189.7 worker6
192.168.189.8 worker7
192.168.189.9 worker8
yarn執行情況
yarn 配置
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# cat core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-2.6.0/tmp</value>
<description>hadoop.tmp.dir</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
</configuration>
worker1
worker2
worker3
worker4
worker5
worker6
worker7
worker8
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# cat hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/hadoop-2.6.0/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value> /usr/local/hadoop-2.6.0/tmp/dfs/data</value>
</property>
</configuration>
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# ls | grep map
mapred-env.cmd
mapred-env.sh
mapred-queues.xml.template
mapred-site.xml
mapred-site.xml.template
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop#
修改的 地方
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# cat mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# cat yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop#
[email protected]:/usr/local/setup_scripts# cat yarn_scp.sh
#!/bin/sh
for i in 2 3 4 5 6 7 8 9
do
scp -rq /usr/local/hadoop-2.6.0/etc/hadoop/mapred-site.xml [email protected]$i:/usr/local/hadoop-2.6.0/etc/hadoop/
scp -rq /usr/local/hadoop-2.6.0/etc/hadoop/yarn-site.xml [email protected]$i:/usr/local/hadoop-2.6.0/etc/hadoop/
done
[email protected]:/usr/local/setup_scripts#
[email protected]:/usr/local/setup_scripts# ./yarn_scp.sh
啟動hdfs叢集
[email protected]:/usr/local/hadoop-2.6.0/sbin# start-dfs.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-namenode-master.out
worker5: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker5.out
worker8: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker8.out
worker4: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker4.out
worker3: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker3.out
worker2: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker2.out
worker7: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker7.out
worker6: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker6.out
worker1: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-datanode-worker1.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-root-secondarynamenode-master.out
[email protected]:/usr/local/hadoop-2.6.0/sbin# jps
4080 SecondaryNameNode
4182 Jps
3863 NameNode
[email protected]:/usr/local/hadoop-2.6.0/sbin#
[email protected]:~# jps
2916 DataNode
2983 Jps
[email protected]:~#
啟動yarn叢集
[email protected]:/usr/local/hadoop-2.6.0/sbin# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-resourcemanager-master.out
worker2: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker2.out
worker7: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker7.out
worker4: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker4.out
worker3: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker3.out
worker6: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker6.out
worker5: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker5.out
worker8: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker8.out
worker1: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-root-nodemanager-worker1.out
[email protected]:/usr/local/hadoop-2.6.0/sbin# jps
4080 SecondaryNameNode
3863 NameNode
4234 ResourceManager
4493 Jps
[email protected]:/usr/local/hadoop-2.6.0/sbin#
[email protected]:~# jps
2916 DataNode
2983 Jps
[email protected]:~# jps
3171 Jps
2916 DataNode
3053 NodeManager
[email protected]:~#
啟動歷史伺服器
[email protected]:/usr/local/hadoop-2.6.0/sbin# mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /usr/local/hadoop-2.6.0/logs/mapred-root-historyserver-master.out
[email protected]:/usr/local/hadoop-2.6.0/sbin#
[email protected]:/usr/local/hadoop-2.6.0/sbin# jps
4080 SecondaryNameNode
4609 Jps
3863 NameNode
4537 JobHistoryServer
4234 ResourceManager
[email protected]:/usr/local/hadoop-2.6.0/sbin#
[email protected]:~# jps
2916 DataNode
3053 NodeManager
3183 Jps
[email protected]:~#
啟動SPARK
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# ./start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.out
worker3: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker3.out
worker5: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker5.out
worker7: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker7.out
worker6: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker6.out
worker1: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker1.out
worker4: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker4.out
worker2: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker2.out
worker8: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-worker8.out
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# jps
4080 SecondaryNameNode
4802 Jps
3863 NameNode
4537 JobHistoryServer
4234 ResourceManager
4747 Master
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin#
[email protected]:~# jps
2916 DataNode
3286 Worker
3337 Jps
3053 NodeManager
[email protected]:~#
啟動spark history
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# ./start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to /usr/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-master.out
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# jps
4080 SecondaryNameNode
4930 Jps
4883 HistoryServer
3863 NameNode
4537 JobHistoryServer
4234 ResourceManager
4747 Master
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin#
http://192.168.189.1:19888/jobhistory/app
http://192.168.189.1:8088/cluster/nodes
http://192.168.189.1:18080/
http://192.168.189.1:50070/dfshealth.html#tab-overview
關掉SPARK叢集，只執行yarn叢集，SPARK只做計算
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# jps
4080 SecondaryNameNode
4930 Jps
4883 HistoryServer
3863 NameNode
4537 JobHistoryServer
4234 ResourceManager
4747 Master
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# ./stop-all.sh
worker7: stopping org.apache.spark.deploy.worker.Worker
worker2: stopping org.apache.spark.deploy.worker.Worker
worker6: stopping org.apache.spark.deploy.worker.Worker
worker4: stopping org.apache.spark.deploy.worker.Worker
worker5: stopping org.apache.spark.deploy.worker.Worker
worker8: stopping org.apache.spark.deploy.worker.Worker
worker3: stopping org.apache.spark.deploy.worker.Worker
worker1: stopping org.apache.spark.deploy.worker.Worker
stopping org.apache.spark.deploy.master.Master
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# jps
4080 SecondaryNameNode
4883 HistoryServer
3863 NameNode
4537 JobHistoryServer
4234 ResourceManager
5055 Jps
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin#
[email protected]:/usr/local/hadoop-2.6.0/sbin# jps
3645 NodeManager
4093 Jps
3517 DataNode
[email protected]:/usr/local/hadoop-2.6.0/sbin#
執行SPARKPI
CLINET模式
./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client /usr/local/spark-1.6.0-bin-hadoop2.6/lib/spark-examples-1.6.0-hadoop2.6.0.jar 1000
cluster模式
./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /usr/local/spark-1.6.0-bin-hadoop2.6/lib/spark-examples-1.6.0-hadoop2.6.0.jar 1000
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/bin# ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /usr/local/spark-1.6.0-bin-hadoop2.6/lib/spark-examples-1.6.0-hadoop2.6.0.jar 1000
16/03/05 20:56:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/03/05 20:56:52 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.189.1:8032
16/03/05 20:56:53 INFO yarn.Client: Requesting a new application from cluster with 8 NodeManagers
16/03/05 20:56:53 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
16/03/05 20:56:53 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
16/03/05 20:56:53 INFO yarn.Client: Setting up container launch context for our AM
16/03/05 20:56:53 INFO yarn.Client: Setting up the launch environment for our AM container
16/03/05 20:56:53 INFO yarn.Client: Preparing resources for our AM container
16/03/05 20:56:55 INFO yarn.Client: Uploading resource file:/usr/local/spark-1.6.0-bin-hadoop2.6/lib/spark-assembly-1.6.0-hadoop2.6.0.jar -> hdfs://master:9000/user/root/.sparkStaging/application_1457178371090_0002/spark-assembly-1.6.0-hadoop2.6.0.jar
16/03/05 20:57:02 INFO yarn.Client: Uploading resource file:/usr/local/spark-1.6.0-bin-hadoop2.6/lib/spark-examples-1.6.0-hadoop2.6.0.jar -> hdfs://master:9000/user/root/.sparkStaging/application_1457178371090_0002/spark-examples-1.6.0-hadoop2.6.0.jar
16/03/05 20:57:07 INFO yarn.Client: Uploading resource file:/tmp/spark-fd52bb0e-ff6b-4587-a407-254abbb52aac/__spark_conf__7795016576129936125.zip -> hdfs://master:9000/user/root/.sparkStaging/application_1457178371090_0002/__spark_conf__7795016576129936125.zip
16/03/05 20:57:07 INFO spark.SecurityManager: Changing view acls to: root
16/03/05 20:57:07 INFO spark.SecurityManager: Changing modify acls to: root
16/03/05 20:57:07 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
16/03/05 20:57:08 INFO yarn.Client: Submitting application 2 to ResourceManager
16/03/05 20:57:08 INFO impl.YarnClientImpl: Submitted application application_1457178371090_0002
16/03/05 20:57:09 INFO yarn.Client: Application report for application_1457178371090_0002 (state: ACCEPTED)
16/03/05 20:57:09 INFO yarn.Client:
./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /usr/local/spark-1.6.0-bin-hadoop2.6/lib/spark-examples-1.6.0-hadoop2.6.0.jar 1000
16/03/05 21:32:16 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.189.4
ApplicationMaster RPC port: 0
queue: default
start time: 1457184716828
final status: UNDEFINED
tracking URL: http://master:8088/proxy/application_1457184438410_0001/
user: root
16/03/05 21:32:17 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:18 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:19 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:20 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:21 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:22 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:23 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:24 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:25 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:26 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:27 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:28 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:29 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:30 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:31 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:32 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:33 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:34 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:35 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:36 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:37 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:38 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:39 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:40 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:41 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:42 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:43 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:44 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:45 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:46 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:47 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:48 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:49 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:50 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:51 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:52 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:53 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:54 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:55 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:56 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:57 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:58 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:32:59 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:33:00 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:33:01 INFO yarn.Client: Application report for application_1457184438410_0001 (state: RUNNING)
16/03/05 21:33:02 INFO yarn.Client: Application report for application_1457184438410_0001 (state: FINISHED)
16/03/05 21:33:02 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.189.4
ApplicationMaster RPC port: 0
queue: default
start time: 1457184716828
final status: SUCCEEDED
tracking URL: http://master:8088/proxy/application_1457184438410_0001/A
user: root
16/03/05 21:33:02 INFO util.ShutdownHookManager: Shutdown hook called
16/03/05 21:33:02 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-99c20d66-4fe3-4229-bac3-45affca56ce1
[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/bin#