學習筆記:從0開始學習大資料-2.hadoop安裝
在膝上型電腦安裝學習環境,採用all in one的偽分散式,所有都在一臺電腦部署。
1.下載 hadoop
wget http://archive-primary.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.16.0.tar.gz
2. 解壓 tar -zxvf hadoop-2.6.0-cdh5.16.0.tar.gz
3. 修改環境變數,加入hadoop 和 路徑
nano /etc/profile 在檔案後面加入如下
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-2.b14.el7.x86_64
export HADOOP_HOME=/root/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin
其中如果不知道java的路徑,可以用如下方法定位
source /etc/profile 或重啟電腦
4. 修改 /etc/hosts
加入 192.168.0.101 centos7 根據自己的主機名稱和ip修改
5.設定SSH免密碼登入
ssh-keygen -t rsa
cd .ssh
cp id_rsa.pub authorized_keys
6.關閉防火牆
systemctl stop firewalld
7. 修改相關hadoop配置檔案
如下5個檔案
hadoop-env.sh
yarn-site.xml
mapred-site.xml
hdfs-site.xml
core-site.xml
(1)hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-2.b14.el7.x86_64
(2)yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
(3)mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
(4)hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/root/hadoop/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/root/hadoop/tmp/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>
(5) core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://centos7:8020</value>
</property>
</configuration>
8. format hdfs
#hadoop namenode -format
9.到此hadoop配置完成,啟動測試
[[email protected] hadoop]#start-dfs.sh
[[email protected] hadoop]# start-yarn.sh
[[email protected] hadoop]# jps
8673 DataNode
9378 ResourceManager
7907 SecondaryNameNode
8549 NameNode
9685 Jps
9502 NodeManager
輸入jps 顯示如上說明hadoop啟動成功
may be must install jps
yum install java-1.8.0-openjdk-devel.x86_64
9. 可以測試hadoop命令了
hadoop fs -ls /
hadoop fs -mkdir /test
hadoop fs -put hello.txt /test //傳送當前目錄下的hello.txt到hdfs
hadoop fs -txt /test/hello.txt //等於 cat hello.txt
10.可以通過瀏覽器檢視
http://centos7:50070
http://centos7:8088