hadoop分布式集群搭建
hadoop分布式集群搭建:
1. 集群規劃:
2.具體步驟:
(1)上傳安裝包,並解壓到相關目錄:
[hadoop@hadoop01 home]$ put c:/hadoop-2.6.5-centos-6.7.tar.gz
[hadoop@hadoop01 home]$tar -zxvf hadoop-2.6.5-centos-6.7.tar.gz -C /home/hadoop/apps
(2)修改配置文件:
[hadoop@hadoop01 home]$ cd /home/hadoop/apps/hadoop-2.6.5/hadoop/etc
hadoop-env.sh:
加入:export JAVA_HOME=/usr/java/jdk1.8.0_73
core-site.xml:
<property> <name>fs.defaultFS</name> <value>hdfs://hadoop01:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/hadoopdata</value> </property>
hdfs-site.xml:
<property> <name>dfs.namenode.name.dir</name> <value>/home/hadoop/hadoopdata/name</value> <description>為了保證元數據的安全一般配置多個不同目錄</description> </property> <property> <name>dfs.datanode.data.dir</name> <value>/home/hadoop/hadoopdata/data</value> <description>datanode 的數據存儲目錄</description> </property> <property> <name>dfs.replication</name> <value>2</value> <description>HDFS 的數據塊的副本存儲個數</description> </property> <property> <name>dfs.secondary.http.address</name> <value>hadoop02:50090</value> <description>secondarynamenode 運行節點的信息,和 namenode 不同節點</description> </property>
mapred-site.xml:
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
yarn-site.xml:
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop03</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>YARN 集群為 MapReduce 程序提供的 shuffle 服務</description>
</property>
slaves:
hadoop01
hadoop02
hadoop03
(3)分發安裝包到各個節點,Hadoop 集群的每個節點都需要安裝 Hadoop 安裝包:
[hadoop@hadoop01 etc]$scp -r hadoop-2.6.5 hadoop02:$PWD
[hadoop@hadoop01 etc]$scp -r hadoop-2.6.5 hadoop03:$PWD
(4)配置hadoop的環境變量
[hadoop@hadoop01 etc]$sudo vim /etc/profile:
加入:
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.6.5/
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
[hadoop@hadoop01 etc]$source /etc/profile
(5)在 HDFS 主節點上執行命令進行初始化 namenode
[hadoop@hadoop01 etc]$hadoop namenode -format
(6)在 HDFS 上的主節點啟動 HDFS
[hadoop@hadoop01 etc]$start-dfs.sh
(7)在 HDFS 上的主節點啟動 YARN
[hadoop@hadoop01 etc]$sbin/start-yarn.sh
補充:
hdfs的web界面是: http://hadoop01:50070
yarn的web界面是: http://hadoop03:8088
查看集群的狀態:hdfs dfsadmin -report
1、啟動 namenode 或者 datenode
sbin/hadoop-daemon.sh start datanode
sbin/hadoop-daemon.sh start namenode
sbin/hadoop-daemon.sh start secondarynamenode
2、啟動 yarn nodemanager
sbin/yarn-daemon.sh start nodemanager
sbin/yarn-daemon.sh start resourcemanager
hadoop分布式集群搭建