1、準備環境

作業系統：Mac OS X 10.11.6 
JDK:1.8.0_111 
Hadoop:2.7.3

2、配置ssh

首先確認能夠遠端登入 
系統偏好設定－共享 
 
在終端執行

ssh-keygen -t rsa

一路回車就行 
會在~/.ssh目錄下生成兩個檔案 
 
然後執行

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

3、驗證ssh

ssh localhost

4、安裝hadoop

下載連結：http://hadoop.apache.org/releases.html

解壓到相應目錄

tar -zxf hadoop-2.7
 
.3.tar.gz
ln -s hadoop-2.7.3 hadoop

• 1
• 2

• 1
• 2

5、修改配置檔案

如果沒有配置Java環境變數，需要在hadoop-env.sh增加

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home

core-site.xml 
該配置檔案用於指明namenode的主機名和埠，hadoop臨時目錄

<configuration>
    <!-- 指定HDFS老大（namenode）的通訊地址 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://0.0.0.0:9000</value>
    </property>
    <!-- 指定hadoop執行時產生檔案的儲存路徑 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/Users/eleme/Documents/ProgramFiles/apache-software-foundation/hadoop-2.7.3/temp</value>
    </property>
</configuration>
 

預設副本數3，修改為1，dfs.namenode.name.dir指明fsimage存放目錄，多個目錄用逗號隔開。dfs.datanode.data.dir指定塊檔案存放目錄，多個目錄逗號隔開

<property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
       <value>file:/Users/eleme/Documents/ProgramFiles/apache-software-foundation/hadoop-2.7.3/tmp/hdfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
          <value>file://Users/eleme/Documents/ProgramFiles/apache-software-foundation/hadoop-2.7.3/tmp/hdfs/data</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>localhost:9001</value>
    </property>
    <property>
      <name>dfs.webhdfs.enabled</name>
      <value>true</value>
    </property>
 

• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17

yarn配置 
mapred-site.xml

<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.admin.user.env</name>
        <value>HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME</value>
    </property>
    <property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME</value>
    </property>

yarn-site.xml

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

sudo vim /etc/profile

export HADOOP_HOME=/Users/eleme/Documents/ProgramFiles/apache-software-foundation/hadoop-2.7.3/hadoop
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

source /etc/profile

7、執行Hadoop

格式化HDFS

hdfs namenode -format

啟動hdfs

start-dfs.sh

啟動yarn

start-yarn.sh

執行測試例子

建立hdfs目錄

hdfs dfs -mkdir -p /user/yonglu/input

上傳測試檔案

hdfs dfs -put $HADOOP_HOME/etc/hadoop/*.xml input

執行測試jar

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-alpha1.jar grep input output ‘dfs[a-z.]+’

檢視執行結果

hdfs dfs -cat output/part-r-00000

16/12/17 12:56:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
dfsadmin
dfs.webhdfs.enabled
dfs.replication
dfs.namenode.secondary.http
dfs.namenode.name.dir
dfs.datanode.data.dir