環境：阿里雲伺服器 CentOS 7 x86_64

安裝介質：jdk-7u75-linux-i586.tar.gz，hadoop-2.4.1.tar.gz

安裝jdk

tar -zxvf jdk-7u75-linux-i586.tar.gz

配置環境變數：

# vi .bash_profile

JAVA_HOME=/root/training/jdk1.7.0_75
export JAVA_HOME

PATH=$JAVA_HOME/bin:$PATH
export PATH

# source .bash_profile
# which java
# java -version
 

bug解決：64bit的作業系統，無法執行32bit的應用程式，需要安裝32bit的glibc庫。

-bash: /root/training/jdk1.7.0_75/bin/java: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory

# yum install glibc*.i686
# locate /lib/ld-linux.so.2
# rpm -qf /lib/ld-linux.so.2

安裝Hadoop

tar -zxvf hadoop-2.4.1.tar.gz

配置環境變數：

# vi .bash_profile
 


HADOOP_HOME=/root/training/hadoop-2.4.1
export HADOOP_HOME

PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export PATH

# source .bash_profile

本地模式的配置

# vi hadoop-env.sh

export JAVA_HOME=/root/training/jdk1.7.0_75

修改hostname，/etc/hosts下的地址必須使用私有地址。

# vi /etc/hosts

192.168.1.107 izwz985sjvpoji48moqz01z

驗證mapreduce

# hadoop jar hadoop-mapreduce-examples-2.4.1.jar wordcount ~/training/data/input/data.txt ~/training/data/output/
# more part-r-00000

偽分散式模式配置

hdfs-site.xml

<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>

core-site.xml

<property>
    <name>fs.defaultFS</name>
    <value>hdfs://192.168.1.107:9000</value>
</property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/root/training/hadoop-2.4.1/tmp</value>
</property>

mapred-site.xml，cp mapred-site.xml.template mapred-site.xml

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

yarn-site.xml

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>192.168.1.107</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

驗證HDFS和mapreduce

# cd ~/trainging
# ls hadoop-2.4.1/tmp/
# hdfs namenode -format
# start-all.sh
# jps
5828 NodeManager
6284 Jps
5438 SecondaryNameNode
5288 DataNode
5579 ResourceManager
5172 NameNode
# hdfs dfsadmin -report
# hdfs dfs -mkdir /input
# hdfs dfs -put data/input/data.txt /input/data.txt
# hdfs dfs -lsr /
# hadoop jar hadoop-mapreduce-examples-2.4.1.jar wordcount /input/data.txt /output
# hdfs dfs -cat /output/part-r-00000
# stop-all.sh
# jps

配置Hadoop的SSH免密碼登入

# cd ~
# ls .ssh/
hnown_hosts
# ssh-keygen -t rsa
# ssh-copy-id -i .ssh/id_rsa.pub [email protected]
# more .ssh/authorized_keys

微信公眾號「資料分析」，分享資料科學家的自我修養，既然遇見，不如一起成長。