本次搭建的偽分散式hadoop叢集所使用的作業系統是紅帽5,64位系統。
所以,需要注意以下幾點:
1、jdk和hadoop安裝包也應該是64位的
2、64位的jdk是從檔名可以直接看出,例如:jdk-8u172-linux-x64.tar.gz
3、而hadoop的可執行安裝包(非原始碼包)是看不出多少位的,例如:hadoop-2.6.5.tar.gz,需要解壓後使用file ${HADOOP_HOME}/lib/native/libhadoop.so.1.0.0來確認。

[root@hadoop02 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 5 (Tikanga)
[root@hadoop02 ~]# 
[root@hadoop02 ~]# uname -a
Linux hadoop02 2.6.18-8.el5 #1 SMP Fri Jan 26 14:15:14 EST 2007 x86_64 x86_64 x86_64 GNU/Linux
[root@hadoop02 ~]# 

修改hosts檔案以及建立hadoop使用者。

[root@hadoop02 ~]# hostname
hadoop02
[root@hadoop02 ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:15:C5:75  
          inet addr:192.168.16.24  Bcast:192.168.16.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe15:c575/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:156 errors:0 dropped:0 overruns:0 frame:0
          TX packets:118 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:15364 (15.0 KiB)  TX bytes:18713 (18.2 KiB)
          Interrupt:19 Base address:0x2000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:12 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:720 (720.0 b)  TX bytes:720 (720.0 b)

[root@hadoop02 ~]# vi /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.16.24   hadoop02

[root@hadoop02 ~]# useradd hadoop
[root@hadoop02 ~]# passwd hadoop
Changing password for user hadoop.
New password: 
BAD PASSWORD: it is based on a dictionary word
BAD PASSWORD: is too simple
Retype new password: 
passwd: all authentication tokens updated successfully.
[root@hadoop02 ~]# id hadoop
uid=500(hadoop) gid=500(hadoop) groups=500(hadoop)
[root@hadoop02 ~]# 

上傳jdk-8u172-linux-x64.tar.gz到Linux伺服器上解壓配置。

[hadoop@hadoop02 ~]$ ls
hadoop-2.6.5.tar.gz  jdk-8u172-linux-x64.tar.gz
[hadoop@hadoop02 ~]$ 
[hadoop@hadoop02 ~]$ mkdir app
[hadoop@hadoop02 ~]$ 
[hadoop@hadoop02 ~]$ tar -zxvf jdk-8u172-linux-x64.tar.gz -C app
[hadoop@hadoop02 ~]$ cd app
[hadoop@hadoop02 app]$ cd jdk1.8.0_172/
[hadoop@hadoop02 jdk1.8.0_172]$ pwd
/home/hadoop/app/jdk1.8.0_172
[hadoop@hadoop02 jdk1.8.0_172]$ 

配置JAVA_HOME環境變數。

[hadoop@hadoop02 jdk1.8.0_172]$ cd
[hadoop@hadoop02 ~]$ vi .bash_profile
#追加下面兩行
export JAVA_HOME=/home/hadoop/app/jdk1.8.0_172

export PATH=$JAVA_HOME/bin:$PATH

重新整理配置檔案

[hadoop@hadoop02 ~]$ source .bash_profile
[hadoop@hadoop02 ~]$ java -version
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)
[hadoop@hadoop02 ~]$ 
[hadoop@hadoop02 ~]$ ls
app  hadoop-2.6.5.tar.gz  jdk-8u172-linux-x64.tar.gz
[hadoop@hadoop02 ~]$ 
[hadoop@hadoop02 ~]$ tar -zxvf hadoop-2.6.5.tar.gz -C app

[hadoop@hadoop02 app]$ cd hadoop-2.6.5/
[hadoop@hadoop02 hadoop-2.6.5]$ ll
total 132
drwxrwxr-x. 2 hadoop hadoop  4096 Oct  3  2016 bin
drwxrwxr-x. 3 hadoop hadoop  4096 Oct  3  2016 etc
drwxrwxr-x. 2 hadoop hadoop  4096 Oct  3  2016 include
drwxrwxr-x. 3 hadoop hadoop  4096 Oct  3  2016 lib
drwxrwxr-x. 2 hadoop hadoop  4096 Oct  3  2016 libexec
-rw-rw-r--. 1 hadoop hadoop 84853 Oct  3  2016 LICENSE.txt
-rw-rw-r--. 1 hadoop hadoop 14978 Oct  3  2016 NOTICE.txt
-rw-rw-r--. 1 hadoop hadoop  1366 Oct  3  2016 README.txt
drwxrwxr-x. 2 hadoop hadoop  4096 Oct  3  2016 sbin
drwxrwxr-x. 4 hadoop hadoop  4096 Oct  3  2016 share

這裡順便說一下,hadoop官方文件就在上面的解壓目錄裡。
/home/hadoop/app/hadoop-2.6.5/share/doc
配置HADOOP_HOME環境變數:

[hadoop@hadoop02 ~]$ cd app/hadoop-2.6.5/
[hadoop@hadoop02 hadoop-2.6.5]$ pwd
/home/hadoop/app/hadoop-2.6.5
[hadoop@hadoop02 hadoop-2.6.5]$ cd
[hadoop@hadoop02 ~]$ vi .bash_profile
#追加下面兩行
export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.5

export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

重新整理配置檔案:

[hadoop@hadoop02 ~]$ source .bash_profile
[hadoop@hadoop02 ~]$ hadoop version
Hadoop 2.6.5
Subversion https://github.com/apache/hadoop.git -r e8c9fe0b4c252caf2ebf1464220599650f119997
Compiled by sjlee on 2016-10-02T23:43Z
Compiled with protoc 2.5.0
From source with checksum f05c9fa095a395faa9db9f7ba5d754
This command was run using /home/hadoop/app/hadoop-2.6.5/share/hadoop/common/hadoop-common-2.6.5.jar
[hadoop@hadoop02 ~]$ 

下面開始配置偽分散式hadoop。

Pseudo-Distributed Operation
Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process.

  • ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh:
 24 # The java implementation to use.
 把第25${JAVA_HOME}替換成/home/hadoop/app/jdk1.8.0_172
 25 export JAVA_HOME=/home/hadoop/app/jdk1.8.0_172
  • ${HADOOP_HOME}/etc/hadoop/core-site.xml:
<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://hadoop02:9000</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/home/hadoop/app/hadoop-2.6.5/tmp</value>
        </property>
</configuration>
  • ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml:
<configuration>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
</configuration>
  • ${HADOOP_HOME}/etc/hadoop/mapred-site.xml:
[[email protected] hadoop]$ mv mapred-site.xml.template mapred-site.xml
[[email protected] hadoop]$ vi mapred-site.xml
<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
</configuration>
  • ${HADOOP_HOME}/etc/hadoop/yarn-site.xml:
<configuration>
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>hadoop02</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
</configuration>

配置ssh免密碼登入

    #生成ssh免登陸金鑰
    #進入到我的home目錄
    cd ~/.ssh

    ssh-keygen -t rsa (四個回車)
    執行完這個命令後,會生成兩個檔案id_rsa(私鑰)、id_rsa.pub(公鑰)
    將公鑰拷貝到要免密登陸的目標機器上
    [[email protected] .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub localhost
    [[email protected] .ssh]$ ssh localhost
    Last login: Tue May 15 19:24:12 2018 from hadoop02
    [[email protected] ~]$ exit
    logout
    Connection to localhost closed.
    [[email protected] .ssh]$ 

見證奇蹟的時刻到了,下面開始啟動hdfs和yarn。
Format the filesystem:

[hadoop@hadoop02 ~]$ hdfs namenode -format

Start NameNode daemon and DataNode daemon:

[hadoop@hadoop02 hadoop]$ start-dfs.sh

Start ResourceManager daemon and NodeManager daemon:

[hadoop@hadoop02 hadoop]$ start-yarn.sh

檢視後臺程序啟動情況:

 [hadoop@hadoop02 hadoop]$ jps
4612 Jps
3798 NameNode
4216 ResourceManager
4074 SecondaryNameNode
3917 DataNode
4509 NodeManager

驗證:

[[email protected] ~]$ hadoop fs -put hadoop-2.6.5.tar.gz /hadoop-2.6.5.tar.gz 
18/05/15 19:30:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[[email protected] ~]$ hadoop fs -ls /
18/05/15 19:30:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r--   1 hadoop supergroup  199635269 2018-05-15 19:30 /hadoop-2.6.5.tar.gz
[[email protected] ~]$ 

這裡這個警告:18/05/15 19:30:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable 目前我也知道什麼原因了,如果哪位高手知道,幫忙指導下哈。

.