Spark2.2.0叢集搭建部署之【HADOOP叢集篇】
阿新 • • 發佈:2018-11-23
軟體準備資訊,詳見Spark2.2.0叢集搭建部署之【軟體準備篇】
基礎配置資訊,詳見Spark2.2.0叢集搭建部署之【基礎配置篇】
SSH無密訪問,詳見park2.2.0叢集搭建部署之【無密訪問篇】
將hadoop-2.7.7.tar.gz 進行解壓
tar -zxvf hadoop-2.7.7.tar.gz
配置環境變數資訊,vi /etc/profile
export HADOOP_HOME=/root/xdb/hadoop-2.7.7 export PATH=$PATH:$HADOOP_HOME/bin export PATH=$PATH:$HADOOP_HOME/sbin export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_ROOT_LOGGER=INFO,console export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
重新整理配置, source /etc/profile,以上配置在各個機器均做此操作。
修改$HADOOP_HOME/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
即使環境變數中已經配置,此處必須修改,否則會報“JAVA_HOME is not set and could not be found.”。
修改$HADOOP_HOME/etc/hadoop/slaves
slave1
修改$HADOOP_HOME/etc/hadoop/core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://master:9000</value> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/root/xdb/hadoop-2.7.7/tmp</value> </property> </configuration>
修改$HADOOP_HOME/etc/hadoop/hdfs-site.xml
<configuration> <property> <name>dfs.namenode.secondary.http-address</name> <value>master:50090</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/root/xdb/hadoop-2.7.7/hdfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/root/xdb/hadoop-2.7.7/hdfs/data</value> </property> <property> <name>dfs.namenode.datanode.registration.ip-hostname-check</name> <value>false</value> </property> </configuration>
修改$HADOOP_HOME/etc/hadoop/mapred-site.xml (cp mapred-site.xml.template mapred-site.xml)
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:19888</value>
</property>
</configuration>
修改$HADOOP_HOME/etc/hadoop/yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>
複製master節點的hadoop資料夾到slave1上。
scp -r hadoop-2.7.7 slave1:/root/xdb/
在master節點啟動叢集,啟動之前格式化一下namenode
hadoop namenode -format
啟動:
$HADOOP_HOME/sbin/start-all.sh
檢查,各節點執行 jps
NameNode顯示
DataNode顯示
Hadoop管理介面, http://192.168.195.129:8088即可訪問,可檢視到active nodes節點有一條資料,即slave1.