楔子

學習 hadoop(基於hadoop3.1.1)，學習視訊可以參看大資料第二階段：hadoop體系之離線計算

1 前期準備

1.1 hosts修改

win10修改hosts許可權

虛擬機器4 臺 ，分配如下

整個過程使用的使用者為root。

1.2 安裝java，配置HADOOP_HOME環境變數

為了方便，我配置了 HADOOP_HOME 到/etc/profile

export HADOOP_HOME=/opt/hadoop-3.1.1
PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
 

2分散式搭建

2.1配置檔案

hadoop-env.sh

末尾新增如下內容

export JAVA_HOME=/usr/local/soft/jdk
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root

core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
 

        <value>hdfs://had1:9820</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/data/hadoop</value>
    </property>
</configuration>

hdfs-site.xml

<configuration>
   <!--檔案副本數量  -->
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>had2:9868</value>
    </property>
</configuration>

workers

此檔案中需要配置datanode節點

had2
had3
had4

2.2 格式化namenode，啟動

-------格式化namenode----------------------------------
[[email protected] hadoop-3.1.1]# hdfs namenode -format
-------啟動--------------------------------------------
[[email protected] hadoop-3.1.1]# start-dfs.sh 
Starting namenodes on [had1]
Starting datanodes
had2: WARNING: /opt/hadoop-3.1.1/logs does not exist. Creating.
had4: WARNING: /opt/hadoop-3.1.1/logs does not exist. Creating.
had3: WARNING: /opt/hadoop-3.1.1/logs does not exist. Creating.
Starting secondary namenodes [had2]
2018-12-20 06:25:01,882 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[[email protected] hadoop-3.1.1]# 

2.3檢視程序

jps檢視程序

2.4 web頁面

http://had1:9870/dfshealth.html