一 、centos叢集環境配置
1.建立一個namenode節點，5個datanode節點

2.關閉防火牆，設定selinux為disabled
#service iptables stop
#chkconfig iptables off

重啟後生效，建議全部配置完成後再全部重啟。
3.配置hosts檔案
將各節點對應的主機名與IP地址記錄在hosts檔案中
vi /etc/hosts

本文所用java安裝包為jdk-7u25-Linux-x64.tar.gz(百度上各種版本的都有)
解壓到/opt目錄下
修改環境變數：
#vi /etc/profile
在檔案末尾新增以下內容
export JAVA_HOME=/opt/jdk1.7.0_25
export JAR_HOME=/opt/jdk1.7.0_25/jre
export ClASSPATH=$

顯示上圖結果說明配置成功
注意：使用source \etc\profile命令盡在本終端有效，重啟後才全部生效
提示：namenode節點和datanode節點上述配置都相同
二、設定各節點間SSH無密碼通訊
在一個節點namenodezsw上操作
1.生成金鑰對
#ssh-keygen -t rsa
按三次回車，結果如下圖

2.將各個節點生成的公有金鑰新增到authorized_keys
#cat .ssh/id_rsa.pub > .ssh/authorized_keys

#ssh 192.168.129.159 ssh-keygen -t rsa
#ssh 192.168.129.159 cat ~/.ssh/id_rsa.pub >>.ssh/authorized_keys
#ssh 192.168.129.160 ssh-keygen -t rsa
#ssh 192.168.129.160 cat ~/.ssh/id_rsa.pub >>.ssh/authorized_keys
#ssh 192.168.129.161 ssh-keygen -t rsa
#ssh 192.168.129.161 cat ~/.ssh/id_rsa.pub >>.ssh/authorized_keys
#ssh 192.168.129.162 ssh-keygen -t rsa
#ssh 192.168.129.162 cat ~/.ssh/id_rsa.pub >>.ssh/authorized_keys
#ssh 192.168.129.163 ssh-keygen -t rsa
#ssh 192.168.129.163 cat ~/.ssh/id_rsa.pub >>.ssh/authorized_keys

3.將authorized_keys檔案傳到各個節點
#scp /root/.ssh/authorized_keys [email protected]:/root/.ssh/authorized_keys
#scp /root/.ssh/authorized_keys [email protected]:/root/.ssh/authorized_keys
#scp /root/.ssh/authorized_keys [email protected]:/root/.ssh/authorized_keys
#scp /root/.ssh/authorized_keys [email protected]:/root/.ssh/authorized_keys
#scp /root/.ssh/authorized_keys [email protected]:/root/.ssh/authorized_keys

說明：每個節點上都要ssh命令其他的節點主機名和IP一下，檢測是否可以

1.去hadoop官網或者百度下載hadoop-2.6.0-tar.gz，然後解壓到namenode的/opt目錄下
2.修改配置檔案
#cd /opt/hadoop-2.6.0/etc/hadoop

①修改 hadoop-env.sh和yarn-env.sh
export JAVA_HOME=/opt/jdk1.7.0_25
②修改core-site.xml

<configuration>

         <property>

               <name>hadoop.tmp.dir</name>

                <value>/opt/hadoop-2.6.0/tmp</value>

               <description>Abase for other temporarydirectories.</description>

       </property>

       <property>

               <name>fs.defaultFS</name>

               <value>hdfs://namenodezsw:9000</value>

        </property>

       <property>

               <name>io.file.buffer.size</name>

               <value>4096</value>

       </property>

</configuration>

configuration>

   <property>

         <name>dfs.namenode.name.dir</name>

         <value>file:///opt/hadoop-2.6.0/dfs/name</value>

   </property>

       <property>

        <name>dfs.datanode.data.dir</name>

        <value>file:///opt/hadoop-2.6.0/dfs/data</value>

   </property>

   <property>

        <name>dfs.replication</name>

        <value>2</value>

   </property>

   <property>

       <name>dfs.nameservices</name>

       <value>h1</value>

   </property>

   <property>

       <name>dfs.namenode.secondary.http-address</name>

       <value>namenodezsw:50090</value>

   </property>

    <property>

       <name>dfs.webhdfs.enabled</name>

       <value>true</value>

   </property>

</configuration>

④修改mapred-site.xml
cp mapred-site.xml.template mapred-site.xml

<configuration>

   <property>

       <name>mapreduce.framework.name</name>

       <value>yarn</value>

       <final>true</final>

   </property>



   <property>

       <name>mapreduce.jobtracker.http.address</name>

       <value>namenodezsw:50030</value>

   </property>

   <property>

       <name>mapreduce.jobhistory.address</name>

       <value>namenodezsw:10020</value>

   </property>

   <property>

       <name>mapreduce.jobhistory.webapp.address</name>

       <value>namenodezsw:19888</value>

   </property>

   <property>

       <name>mapred.job.tracker</name>

        <value>http://namenodezsw:9001</value>

   </property>

</configuration>

⑤修改yarn-site.xml

<configuration> 

<!-- Site specific YARN configuration properties-->

   <property>

       <name>yarn.resourcemanager.hostname</name>

       <value>namenodezsw</value>

    </property>



   <property>

       <name>yarn.nodemanager.aux-services</name>

       <value>mapreduce_shuffle</value>

   </property>

   <property>

       <name>yarn.resourcemanager.address</name>

       <value>namenodezsw:8032</value>

   </property>

   <property>

       <name>yarn.resourcemanager.scheduler.address</name>

        <value>namenodezsw:8030</value>

   </property>

   <property>

       <name>yarn.resourcemanager.resource-tracker.address</name>

       <value>namenodezsw:8031</value>

   </property>

   <property>

       <name>yarn.resourcemanager.admin.address</name>

       <value>namenodezsw:8033</value>

   </property>

   <property>

       <name>yarn.resourcemanager.webapp.address</name>

       <value>namenodezsw:8088</value>

   </property>

</configuration>

⑥修改slaves檔案

⑦配置環境變數
# vi /etc/profile
export JAVA_HOME=/opt/jdk1.7.0_25
export HADOOP_HOME=/opt/hadoop-2.6.0
export JAR_HOME=/opt/jdk1.7.0_25/jre
export CLASSPATH=$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$JAR_HOME/lib
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
環境變數配置同樣source /etc/profile命令本終端有效，重啟後都有效
至此，修改完成
3.啟動hadoop
使用scp -r /opt/hadoop-2.6.0/  hostname:/opt/命令將hadoop檔案依次拷貝到5個datanode節點上。
在namenode上操作：
首先格式化
#hadoop namenode -format
啟動所有服務
#start-all.sh

4.驗證
namenode上：
#jps

datanode上：
#jps

也可以使用 hadoop dfsadmin -report命令檢視總體情況
至此叢集搭建全部完成

遇到的問題

原因是hadoop格式化多次，導致namenode和datanode的clusterid不一樣

解決方法：在hdfs-site.xml中有如下配置

在hadoop-2.6.5/dfs/data下有current/version，開啟version，找到裡面的clusterId

找到log檔案（在hadoop-2.6.5中的logs下），找到clusterId，並用該clusterId替換掉version中的clusterId

注意：改完後注意重啟所有節點，改完後如果直接start-all.sh，可能又會生成不一樣的clusterId

轉載自：http://blog.csdn.net/zsw_2015/article/details/51406644