曾經的你驕橫跋扈，如今你審視人生，重新來過，重新選擇，相比同齡你可能晚了一些，遙看人生路路，你沒有輸，勞動可以改造一個人，堅持勞動，就可以改變人心！無論是腦力勞動還是體力勞動。
————————————-前言送給努力改變自己的人！
Hadoop:
廣義: 以hadoop軟體為主的生態圈
狹義: hadoop軟體

hadoop.apache.org
hive.apache.org
spark.apache.org
flink.apache.org

hadoop軟體:
1.x
2.x 生產 2.6
3.x

hadoop2.x元件:
hdfs: 儲存 分散式檔案系統 底層 生產
hive/hbase
mapreduce: 分散式計算 --》開發難度高、計算慢(shuffle 磁碟)
hive sql/spark
yarn: 資源(記憶體+core)+作業(job)排程管理系統 生產

但:
apache hadoop 不選擇部署
企業一般選擇CDH、Ambari、hdp部署
CDH:
cloudera公司 將Apache hadoop-2.6.0原始碼，
修復bug,新功能，編譯為自己的版本cdh5.7.0

Apache hadoop-2.6.0 --》hadoop-2.6.0-cdh5.7.0

部署:

1.新增sudo許可權的無密碼訪問的hadoop使用者
[[email protected] ~]# useradd hadoop
[[email protected] ~]# cat /etc/sudoers |grep hadoop
hadoop ALL=(ALL) NOPASSWD: ALL
[

[email protected] ~]#
[[email protected] ~]# su - hadoop
[[email protected] ~]$

2.下載
[[email protected] ~]$ mkdir app
[[email protected]02 ~]$ cd app
[[email protected] app]$ wget http://archive-primary.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz

[[email protected] app]$ tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz
[

[email protected] app]$ cd hadoop-2.6.0-cdh5.7.0
[[email protected] hadoop-2.6.0-cdh5.7.0]$

Required software for Linux include:
Java™ must be installed. Recommended Java versions are described at HadoopJavaVersions.
ssh must be installed and sshd must be running to use the Hadoop scripts that manage remote Hadoop daemons.

3.JAVA1.7部署
[[email protected] hadoop-2.6.0-cdh5.7.0]$ ll /usr/java/
total 319160
drwxr-xr-x 8 root root 4096 Apr 11 2015 jdk1.7.0_80
drwxr-xr-x 8 root root 4096 Apr 11 2015 jdk1.8.0_45
-rw-r–r-- 1 root root 153530841 Jul 8 2015 jdk-7u80-linux-x64.tar.gz
-rw-r–r-- 1 root root 173271626 Sep 19 11:49 jdk-8u45-linux-x64.gz
[[email protected] hadoop-2.6.0-cdh5.7.0]$ echo $JAVA_HOME /usr/java/jdk1.7.0_80 [[email protected] hadoop-2.6.0-cdh5.7.0]$

[[email protected] hadoop-2.6.0-cdh5.7.0]$ which java
/usr/java/jdk1.7.0_80/bin/java
[[email protected] hadoop-2.6.0-cdh5.7.0]$ java -version
java version “1.7.0_80”
Java™ SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot™ 64-Bit Server VM (build 24.80-b11, mixed mode)
[[email protected] hadoop-2.6.0-cdh5.7.0]$

4.準備
[[email protected] hadoop-2.6.0-cdh5.7.0]$ cd etc/hadoop
[[email protected] hadoop]$ vi hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.7.0_80
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
[[email protected] hadoop-2.6.0-cdh5.7.0]$ bin/hadoop
Usage: hadoop [–config confdir] COMMAND
where COMMAND is one of:

啟動三種模式
Local (Standalone) Mode: 單機 沒有程序 不用
Pseudo-Distributed Mode: 偽分散式 1臺機器 程序 學習
Fully-Distributed Mode: 分散式 程序 生產

5.配置檔案
[[email protected] hadoop]$ vi core-site.xml

<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> fs.defaultFS hdfs://hadoop002:9000 "core-site.xml" 24L, 884C written [[email protected] hadoop]$ vi hdfs-site.xml <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an “AS IS” BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
–>

dfs.replication 1 "hdfs-site.xml" 23L, 866C written [[email protected] hadoop]$ cd

6.無密碼ssh
[[email protected] hadoop]$ cd
[[email protected] ~]$
[[email protected] ~]$
[[email protected] ~]$ rm -rf .ssh
[[email protected] ~]$ ssh-keygen -t dsa -P ‘’ -f ~/.ssh/id_dsa
Generating public/private dsa key pair.
Created directory ‘/home/hadoop/.ssh’.
Your identification has been saved in /home/hadoop/.ssh/id_dsa.
Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.
The key fingerprint is:
a3:c7:ba:e9:2e:77:ff:6f:50:bd:bc:f7:1b:1d:a6:e1 [email protected]
The key’s randomart image is:
±-[ DSA 1024]----+
| |
| |
| . |
| . .|
| S o.o.|
| o . o +oo|
| . o E .o|
| . .+. …o|
| =*o …o…=|
±----------------+
[[email protected] ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[[email protected] ~]$ cd .ssh
[[email protected] .ssh]$ ll
total 12
-rw-rw-r-- 1 hadoop hadoop 606 Sep 19 23:16 authorized_keys
-rw------- 1 hadoop hadoop 668 Sep 19 23:16 id_dsa
-rw-r–r-- 1 hadoop hadoop 606 Sep 19 23:16 id_dsa.pub

[[email protected] .ssh]$ chmod 600 authorized_keys
[[email protected] .ssh]$

[[email protected] .ssh]$ ssh hadoop002
The authenticity of host ‘hadoop002 (172.31.236.240)’ can’t be established.
RSA key fingerprint is b1:94:33:ec:95:89:bf:06:3b:ef:30:2f:d7:8e:d2:4c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘hadoop002,172.31.236.240’ (RSA) to the list of known hosts.
Last login: Wed Sep 19 18:21:09 2018 from 172.31.236.240

Welcome to Alibaba Cloud Elastic Compute Service !

[[email protected] ~]$

7.環境變數
[[email protected] ~]$ vi .bash_profile
export MVN_HOME=/home/hadoop/app/apache-maven-3.3.9
export PROTOC_HOME=/home/hadoop/app/protobuf
export FINDBUGS_HOME=/home/hadoop/app/findbugs-1.3.9

.bash_profile

Get the aliases and functions

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

User specific environment and startup programs

export JAVA_HOME=/usr/java/jdk1.7.0_80
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0

export PATH= $HADOOP_PREFIX/bin:$JAVA_HOME/bin: $PATH ~ ~ &quot;.bash_profile&quot; 12L, 293C written [[email protected] ~]$
[[email protected] ~]$
[[email protected] ~]$
[[email protected] ~]$ ssh hadoop002
Last login: Wed Sep 19 23:18:35 2018 from 172.31.236.240

Welcome to Alibaba Cloud Elastic Compute Service !

[[email protected] ~]$ which hdfs
~/app/hadoop-2.6.0-cdh5.7.0/bin/hdfs
[[email protected] ~]$
[[email protected] ~]$ cd ~/app/hadoop-2.6.0-cdh5.7.0
[[email protected] hadoop-2.6.0-cdh5.7.0]$

[[email protected] hadoop-2.6.0-cdh5.7.0]$ bin/hdfs namenode -format

[[email protected] hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
[[email protected] hadoop-2.6.0-cdh5.7.0]$ jps
27707 SecondaryNameNode
27820 Jps
27432 NameNode

發現DN程序有問題，重新部署
[[email protected] tmp]# rm -rf /tmp/hadoop-hadoop
[[email protected] tmp]#
[[email protected] hadoop]$ vi slaves
hadoop002

[[email protected] hadoop]$ cd …/…/
[[email protected] hadoop-2.6.0-cdh5.7.0]$
[[email protected] hadoop-2.6.0-cdh5.7.0]$ bin/hdfs namenode -format

[[email protected] hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
18/09/19 23:29:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Starting namenodes on [hadoop002]
hadoop002: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop002.out
hadoop002: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop002.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop002.out
18/09/19 23:29:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
[[email protected] hadoop-2.6.0-cdh5.7.0]$
[[email protected] hadoop-2.6.0-cdh5.7.0]$
[[email protected] hadoop-2.6.0-cdh5.7.0]$ jps
28288 NameNode
28686 Jps
28410 DataNode
28575 SecondaryNameNode
[[email protected] hadoop-2.6.0-cdh5.7.0]$

雲主機，開啟防火牆
結束語：我們努力不是為了改變這個世界，而是不被這個世界所改變，所淘汰，安逸的暖巢永遠不適合迎風的浪人，安穩的港灣永遠不適合追逐的海盜，每個人都有適合自己的過程，既然選擇，何懼風險！
------------------送給在風雨中為夢想打拼的人們！