Hadoop2.X分布式集群部署
阿新 • • 發佈：2018-10-16
view last linux ntp 啟動 abi without namenode ssp 命令行
本博文集群搭建沒有實現Hadoop HA，詳細文檔在後續給出，本次只是先給出大概邏輯思路。
（一）hadoop2.x版本下載及安裝
Hadoop 版本選擇目前主要基於三個廠商（國外）如下所示：
- 基於Apache廠商的最原始的hadoop版本， 所有發行版均基於這個版本進行改進。
- 基於HortonWorks廠商的開源免費的hdp版本。
- 基於Cloudera廠商的cdh版本，Cloudera有免費版和企業版， 企業版只有試用期。不過cdh大部分功能都是免費的。
（二）hadoop2.x分布式集群配置
1.集群資源規劃設計
2.hadoop2.x分布式集群配置
1）Hadoop安裝配置
先上傳資源，並解壓。
[kfk@bigdata-pro01 softwares]$ tar -zxf hadoop-2.6.0.tar.gz -C /opt/momdules/ [kfk@bigdata-pro01 softwares]$ cd ../momdules/ [kfk@bigdata-pro01 momdules]$ ll total 8 drwxr-xr-x 9 kfk kfk 4096 Nov 14 2014 hadoop-2.6.0 drwxr-xr-x 8 kfk kfk 4096 Aug 5 2015 jdk1.8.0_60 [kfk@bigdata-pro01 momdules]$ cd hadoop-2.6.0/ [kfk@bigdata-pro01 hadoop-2.6.0]$ ls bin etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin share
接下來對hadoop進行一個瘦身（刪除不必要的文件，減小其大小）
[kfk@bigdata-pro01 hadoop-2.6.0]$ cd share/ [kfk@bigdata-pro01 share]$ ls doc hadoop [kfk@bigdata-pro01 share]$ rm -rf ./doc/ [kfk@bigdata-pro01 share]$ ls hadoop [kfk@bigdata-pro01 share]$ cd .. [kfk@bigdata-pro01 hadoop-2.6.0]$ ls bin etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin share [kfk@bigdata-pro01 hadoop-2.6.0]$ cd etc/hadoop/ [kfk@bigdata-pro01 hadoop]$ ls capacity-scheduler.xml hadoop-env.sh httpfs-env.sh kms-env.sh mapred-env.sh ssl-server.xml.example configuration.xsl hadoop-metrics2.properties httpfs-log4j.properties kms-log4j.properties mapred-queues.xml.template yarn-env.cmd container-executor.cfg hadoop-metrics.properties httpfs-signature.secret kms-site.xml mapred-site.xml.template yarn-env.sh core-site.xml hadoop-policy.xml httpfs-site.xml log4j.properties slaves yarn-site.xml hadoop-env.cmd hdfs-site.xml kms-acls.xml mapred-env.cmd ssl-client.xml.example [kfk@bigdata-pro01 hadoop]$ rm -rf ./*.cmd //.cmd為Windows下的命令，所以不需要，可以刪掉。
2）hadoop2.x分布式集群配置-HDFS
安裝hdfs需要修改4個配置文件：hadoop-env.sh、core-site.xml、hdfs-site.xml和slaves
註意：為了方便和正確性的保證，以後Linux中的配置都使用外部工具Notepad++進行（連接之前請保證Windows下的Hosts文件已經添加了網絡映射），連接方式如下：
註：如果出現的目錄和我的不同，請雙擊根目錄（/）。
在配置的時候再教大家一個小技巧：能夠復制粘貼的盡量復制粘貼，這樣能盡量避免拼寫錯誤。比如配置hadoop-env.sh文件時可以如下操作:
然後Ctrl+Ins組合鍵可以實現Linux下的復制操作，粘貼操作用Shift+Ins組合鍵。
該文件只需配置JAVA_HOME目錄即可。
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
配置Namenode
<property>
<name>fs.default.name</name>
<value>hdfs://bigdata-pro01.kfk.com:9000</value>
<description>The name of the default file system, using 9000 port.</description>
</property>
配置Datanode
格式化Namenode
[kfk@bigdata-pro01 hadoop]$ cd .. [kfk@bigdata-pro01 etc]$ cd .. [kfk@bigdata-pro01 hadoop-2.6.0]$ bin/hdfs namenode –format
啟動Namenode和Datanode
[kfk@bigdata-pro01 hadoop-2.6.0]$ sbin/hadoop-daemon.sh start namenode starting namenode, logging to /opt/momdules/hadoop-2.6.0/logs/hadoop-kfk-namenode-bigdata-pro01.kfk.com.out [kfk@bigdata-pro01 hadoop-2.6.0]$ sbin/hadoop-daemon.sh start datanode starting datanode, logging to /opt/momdules/hadoop-2.6.0/logs/hadoop-kfk-datanode-bigdata-pro01.kfk.com.out [kfk@bigdata-pro01 hadoop-2.6.0]$ jps 21778 NameNode 21927 Jps 21855 DataNode
進入網址：http://bigdata-pro01.kfk.com:50070/dfshealth.html#tab-overview
以上結果表明配置是成功的，然後發送到其他節點。
[kfk@bigdata-pro01 momdules]$ scp -r hadoop-2.6.0/ bigdata-pro02.kfk.com:/opt/momdules/ The authenticity of host ‘bigdata-pro02.kfk.com (192.168.86.152)‘ can‘t be established. RSA key fingerprint is b5:48:fe:c4:80:24:0c:aa:5c:f5:6f:82:49:c5:f8:8e. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added ‘bigdata-pro02.kfk.com,192.168.86.152‘ (RSA) to the list of known hosts. kfk@bigdata-pro02.kfk.com‘s password: [kfk@bigdata-pro01 momdules]$ scp -r hadoop-2.6.0/ bigdata-pro03.kfk.com:/opt/momdules/
然後啟動兩個子節點的DataNode並刷新網頁看看有什麽變化。
[kfk@bigdata-pro02 hadoop-2.6.0]$ sbin/hadoop-daemon.sh start datanode starting datanode, logging to /opt/momdules/hadoop-2.6.0/logs/hadoop-kfk-datanode-bigdata-pro02.kfk.com.out [kfk@bigdata-pro02 hadoop-2.6.0]$ jps 21655 DataNode 21723 Jps [kfk@bigdata-pro03 ~]$ cd /opt/momdules/hadoop-2.6.0/ [kfk@bigdata-pro03 hadoop-2.6.0]$ sbin/hadoop-daemon.sh start datanode starting datanode, logging to /opt/momdules/hadoop-2.6.0/logs/hadoop-kfk-datanode-bigdata-pro03.kfk.com.out [kfk@bigdata-pro03 hadoop-2.6.0]$ jps 21654 DataNode 21722 Jps
接下來，我們在dfs上創建一個目錄並上傳一個文件：
[kfk@bigdata-pro01 hadoop-2.6.0]$ bin/hdfs dfs -mkdir -p /user/kfk/data/ 18/10/16 09:21:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [kfk@bigdata-pro01 hadoop-2.6.0]$ bin/hdfs dfs -put /opt/momdules/hadoop-2.6.0/etc/hadoop/core-site.xml /user/kfk/data
創建和上傳都成功！
[kfk@bigdata-pro01 hadoop-2.6.0]$ bin/hdfs dfs -text /user/kfk/data/core-site.xml 18/10/16 13:43:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>fs.default.name</name> <value>hdfs://bigdata-pro01.kfk.com:9000</value> <description>The name of the default file system, using 9000 port.</description> </property> </configuration>
文件內容也是完全一致的！
3）hadoop2.x分布式集群配置-YARN
安裝yarn需要修改4個配置文件：yarn-env.sh、mapred-env.sh、yarn-site.xml和mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>bigdata-pro01.kfk.com:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>bigdata-pro01.kfk.com:19888</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>bigdata-pro01.kfk.com</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>10000</value>
</property>
（三）分發到其他各個機器節點
hadoop相關配置在第一個節點配置好之後，可以通過腳本命令分發給另外兩個節點即可，具體操作如下所示。
#將安裝包分發給第二個節點
[kfk@bigdata-pro01 hadoop]$ scp -r ./* [email protected]:/opt/momdules/hadoop-2.6.0/etc/hadoop/
#將安裝包分發給第三個節點
[kfk@bigdata-pro01 hadoop]$ scp -r ./* [email protected]:/opt/momdules/hadoop-2.6.0/etc/hadoop/
（四）HDFS啟動集群運行測試
[kfk@bigdata-pro01 hadoop]$ cd .. [kfk@bigdata-pro01 etc]$ cd .. [kfk@bigdata-pro01 hadoop-2.6.0]$ cd .. [kfk@bigdata-pro01 momdules]$ cd .. [kfk@bigdata-pro01 opt]$ cd datas/ [kfk@bigdata-pro01 datas]$ touch wc.input [kfk@bigdata-pro01 datas]$ vi wc.input hadoop hive hive spark hbase java [kfk@bigdata-pro01 datas]$ cd ../momdules/hadoop-2.6.0/ [kfk@bigdata-pro01 hadoop-2.6.0]$ bin/hdfs dfs -put /opt/datas/wc.input /user/kfk/data 18/10/16 14:11:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hdfs相關配置好之後，可以啟動resourcemanager和nodemanager。
[kfk@bigdata-pro01 hadoop-2.6.0]$ sbin/yarn-daemon.sh resourcemanager Usage: yarn-daemon.sh [--config <conf-dir>] [--hosts hostlistfile] (start|stop) <yarn-command> [kfk@bigdata-pro01 hadoop-2.6.0]$ sbin/yarn-daemon.sh start resourcemanager starting resourcemanager, logging to /opt/momdules/hadoop-2.6.0/logs/yarn-kfk-resourcemanager-bigdata-pro01.kfk.com.out [kfk@bigdata-pro01 hadoop-2.6.0]$ sbin/yarn-daemon.sh start nodemanager starting nodemanager, logging to /opt/momdules/hadoop-2.6.0/logs/yarn-kfk-nodemanager-bigdata-pro01.kfk.com.out [kfk@bigdata-pro01 hadoop-2.6.0]$ sbin/mr-jobhistory-daemon.sh start historyserver starting historyserver, logging to /opt/momdules/hadoop-2.6.0/logs/mapred-kfk-historyserver-bigdata-pro01.kfk.com.out [kfk@bigdata-pro01 hadoop-2.6.0]$ jps 21778 NameNode 23668 NodeManager 23701 Jps 23431 ResourceManager 21855 DataNode 23835 JobHistoryServer [kfk@bigdata-pro02 hadoop-2.6.0]$ sbin/yarn-daemon.sh start nodemanager starting nodemanager, logging to /opt/momdules/hadoop-2.6.0/logs/yarn-kfk-nodemanager-bigdata-pro02.kfk.com.out [kfk@bigdata-pro02 hadoop-2.6.0]$ jps 22592 NodeManager 21655 DataNode 22622 Jps [kfk@bigdata-pro03 hadoop-2.6.0]$ sbin/yarn-daemon.sh start nodemanager starting nodemanager, logging to /opt/momdules/hadoop-2.6.0/logs/yarn-kfk-nodemanager-bigdata-pro03.kfk.com.out [kfk@bigdata-pro03 hadoop-2.6.0]$ jps 22566 Jps 21654 DataNode 22536 NodeManager
進入網址：http://bigdata-pro01.kfk.com:8088/cluster/nodes
接下來配置一下DataNode的日誌目錄。
<property>
<name>dfs.permissions.enable</name>
<value>false</value>
</property>
<property>
<name>hadoop.http.staticuser.user</name>
<value>kfk</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/momdules/hadoop-2.6.0/data/tmp</value>
</property>
創建目錄：
[kfk@bigdata-pro01 hadoop-2.6.0]$ mkdir -p data/tmp [kfk@bigdata-pro01 hadoop-2.6.0]$ cd data/tmp/ [kfk@bigdata-pro01 tmp]$ pwd /opt/momdules/hadoop-2.6.0/data/tmp
然後分發配置到其他節點：
由於修改東西並且新建了路徑，為了安全起見，先刪掉兩個節點的hadoop文件夾，全部重發一次吧。
[kfk@bigdata-pro02 momdules]$ rm -rf hadoop-2.6.0/ //註意刪除的是02和03節點，別刪錯了。
然後分發：
scp -r ./hadoop-2.6.0/ [email protected]:/opt/momdules/ scp -r ./hadoop-2.6.0/ [email protected]:/opt/momdules/
格式化NameNode
格式化之前要先停掉所有服務：
[kfk@bigdata-pro01 hadoop-2.6.0]$ sbin/yarn-daemon.sh stop resourcemanager stopping resourcemanager [kfk@bigdata-pro01 hadoop-2.6.0]$ sbin/yarn-daemon.sh stop nodemanager stopping nodemanager [kfk@bigdata-pro01 hadoop-2.6.0]$ sbin/mr-jobhistory-daemon.sh stop historyserver stopping historyserver [kfk@bigdata-pro01 hadoop-2.6.0]$ sbin/hadoop-daemon.sh stop namenode stopping namenode [kfk@bigdata-pro01 hadoop-2.6.0]$ sbin/hadoop-daemon.sh stop datanode stopping datanode [kfk@bigdata-pro01 hadoop-2.6.0]$ jps 24207 Jps
格式化：
[kfk@bigdata-pro01 hadoop-2.6.0]$ bin/hdfs namenode -format
啟動各個節點機器服務
1）啟動NameNode命令：
sbin/hadoop-daemon.sh start namenode（01節點）
2) 啟動DataNode命令：
sbin/hadoop-daemon.sh start datanode（01/02/03節點）
格式化Namenode之後之前建立的路徑也就沒有了，所有我們要重新創建。
[kfk@bigdata-pro01 hadoop-2.6.0]$ bin/hdfs dfs -mkdir -p /user/kfk/data
3）啟動ResourceManager命令：
sbin/yarn-daemon.sh start resourcemanager（01節點）
4）啟動NodeManager命令：
sbin/yarn-daemon.sh start nodemanager(01/02/03節點)
5）啟動log日誌命令：
sbin/mr-jobhistory-daemon.sh start historyserver(01節點)
（五）YARN集群運行MapReduce程序測試
前面hdfs和yarn都啟動起來之後，可以通過運行WordCount程序檢測一下集群是否能run起來。
重新上傳測試文件：
[kfk@bigdata-pro01 hadoop-2.6.0]$ bin/hdfs dfs -put /opt/datas/wc.input /user/kfk/data/
然後創建一個輸出目錄：
[kfk@bigdata-pro01 hadoop-2.6.0]$ bin/hdfs dfs -mkdir -p /user/kfk/data/output/
使用集群自帶的WordCount程序執行命令：
[kfk@bigdata-pro01 hadoop-2.6.0]$ bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /user/kfk/data/wc.input /user/kfk/data/output 18/10/16 15:19:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 18/10/16 15:19:41 INFO client.RMProxy: Connecting to ResourceManager at bigdata-pro01.kfk.com/192.168.86.151:8032 org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://bigdata-pro01.kfk.com:9000/user/kfk/data/output already exists at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:562) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314) at org.apache.hadoop.examples.WordCount.main(WordCount.java:87) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
運行報錯。原因是輸出目錄已經存在，而MapReduce執行時會檢測輸出目錄是否存在，不存在則自動創建並正常執行；否則報錯。所以我們重新運行，在輸出目錄後再追加一個目錄即可。
[kfk@bigdata-pro01 hadoop-2.6.0]$ bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /user/kfk/data/wc.input /user/kfk/data/output/1
點擊History可以查看日誌
這樣就能很方便地查看日誌，而不用在命令行進hadoop的logs/目錄下去查看了。我們查看一下運行結果：
[kfk@bigdata-pro01 hadoop-2.6.0]$ bin/hdfs dfs -text /user/kfk/data/output/1/par* 18/10/16 16:00:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable hadoop 1 hbase 1 hive 2 java 1 spark 1
很明顯結果是正確的（上圖中黃色為行號，而非文件內容）！
（六）ssh無秘鑰登錄
在集群搭建的過程中，需要不同節點分發文件，那麽節點間分發文件每次都需要輸入密碼，比較麻煩。另外在hadoop 集群啟動過程中，也需要使用批量腳本統一啟動各個節點服務，此時也需要節點之間實現無秘鑰登錄。具體操作步驟如下所示：
1.主節點上創建 .ssh 目錄，然後生成公鑰文件id_rsa.pub和私鑰文件[kfk@bigdata-pro01 datas]$ cd
[kfk@bigdata-pro01 ~]$ cd .ssh [kfk@bigdata-pro01 .ssh]$ ls known_hosts [kfk@bigdata-pro01 .ssh]$ cat known_hosts bigdata-pro02.kfk.com,192.168.86.152 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAsHpzF1vSSqZPIbTKrhsxKGqofgngHbm5MdXItaSEJ8JemIuWrMo5++0g3QG/m/DRW8KqjXhnBO819tNIqmVNeT+0cH7it9Nosz1NWfwvXyNy+lbxdjfqSs+DvMh0w5/ZoiXVdqWmPAh2u+CP4BKbHS4VKRNoZk42B+1+gzXxN6Gt1kxNemLsLw6251IzmsX+dVr8iH493mXRwE9dv069uKoA0HVwn6FL51D8c1H1v1smD/EzUsL72TUknz8DV43iawIBDMSw4GQJFoZtm2ogpCuIhBfLwTfl+5yyzjY8QdwH5sDiKFlPX476M+A1s+mneyQtaaRwORIiOvs7TgtSTw== bigdata-pro03.kfk.com,192.168.86.153 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAsHpzF1vSSqZPIbTKrhsxKGqofgngHbm5MdXItaSEJ8JemIuWrMo5++0g3QG/m/DRW8KqjXhnBO819tNIqmVNeT+0cH7it9Nosz1NWfwvXyNy+lbxdjfqSs+DvMh0w5/ZoiXVdqWmPAh2u+CP4BKbHS4VKRNoZk42B+1+gzXxN6Gt1kxNemLsLw6251IzmsX+dVr8iH493mXRwE9dv069uKoA0HVwn6FL51D8c1H1v1smD/EzUsL72TUknz8DV43iawIBDMSw4GQJFoZtm2ogpCuIhBfLwTfl+5yyzjY8QdwH5sDiKFlPX476M+A1s+mneyQtaaRwORIiOvs7TgtSTw== [kfk@bigdata-pro01 .ssh]$ rm -f known_hosts //保證.ssh目錄是幹凈的 [kfk@bigdata-pro01 .ssh]$ ls [kfk@bigdata-pro01 .ssh]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/kfk/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/kfk/.ssh/id_rsa. Your public key has been saved in /home/kfk/.ssh/id_rsa.pub. The key fingerprint is: 6f:a2:83:da:9d:77:71:e5:29:71:a1:27:0c:7a:8d:b8 kfk@bigdata-pro01.kfk.com The key‘s randomart image is: +--[ RSA 2048]----+ | | | . . | | o = . . | | o o * + | | So B . | | E.. o o | | . . oo . | | ....o.o. | | ... +o . | +-----------------+
2.拷貝公鑰到各個機器
ssh-copy-id bigdata-pro1.kfk.com ssh-copy-id bigdata-pro2.kfk.com ssh-copy-id bigdata-pro3.kfk.com
3.測試ssh連接
ssh bigdata-pro1.kfk.com ssh bigdata-pro2.kfk.com ssh bigdata-pro3.kfk.com [kfk@bigdata-pro01 .ssh]$ ssh bigdata-pro02.kfk.com Last login: Tue Oct 16 13:17:38 2018 from 192.168.86.1
4.測試hdfs
ssh無秘鑰登錄做好之後，可以在主節點通過一鍵啟動/停止命令，啟動/停止hdfs各個節點的服務，具體操作如下所示：
[kfk@bigdata-pro01 hadoop-2.6.0]$ sbin/stop-dfs.sh 18/10/16 15:52:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Stopping namenodes on [bigdata-pro01.kfk.com] bigdata-pro01.kfk.com: stopping namenode bigdata-pro03.kfk.com: stopping datanode bigdata-pro01.kfk.com: stopping datanode bigdata-pro02.kfk.com: stopping datanode Stopping secondary namenodes [0.0.0.0] The authenticity of host ‘0.0.0.0 (0.0.0.0)‘ can‘t be established. RSA key fingerprint is b5:48:fe:c4:80:24:0c:aa:5c:f5:6f:82:49:c5:f8:8e. Are you sure you want to continue connecting (yes/no)? yes 0.0.0.0: Warning: Permanently added ‘0.0.0.0‘ (RSA) to the list of known hosts. 0.0.0.0: no secondarynamenode to stop 18/10/16 15:53:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [kfk@bigdata-pro01 hadoop-2.6.0]$ sbin/stop-yarn.sh stopping yarn daemons stopping resourcemanager bigdata-pro03.kfk.com: stopping nodemanager bigdata-pro02.kfk.com: stopping nodemanager bigdata-pro01.kfk.com: stopping nodemanager no proxyserver to stop
如果yarn和hdfs主節點共用，配置一個節點即可。否則，yarn也需要單獨配置ssh無秘鑰登錄。
（七）配置集群內機器時間同步（使用Linux ntp進行）
參考博文：https://www.cnblogs.com/zimo-jing/p/8892697.html
註：在三個節點上都要進行操作，還有最後一個命令使用sudo。
以上就是博主為大家介紹的這一板塊的主要內容，這都是博主自己的學習過程，希望能給大家帶來一定的指導作用，有用的還望大家點個支持，如果對你沒用也望包涵，有錯誤煩請指出。如有期待可關註博主以第一時間獲取更新哦，謝謝！同時也歡迎轉載，但必須在博文明顯位置標註原文地址，解釋權歸博主所有！
Hadoop2.X分布式集群部署