3.4 HDFS叢集模式
阿新 • • 發佈：2018-12-04
第3章 HDFS：分散式檔案系統
3.4 HDFS叢集模式
| 節點 | IP | 角色 |
|---|---|---|
| node1 | 192.168.80.131 | NameNode，DataNode |
| node2 | 192.168.80.132 | SecondaryNameNode，DataNode |
| node3 | 192.168.80.133 | DataNode |
3.4.1 Hadoop環境變數
[root@node1 ~]# vi /etc/profile.d/custom.sh
- 1
- 1
-
#Hadoop path
-
export HADOOP_HOME=/opt/hadoop-
2.7.
3
-
export PATH=
$PATH:
$HADOOP_HOME/bin:
$HADOOP_HOME/sbin
- 1
- 2
- 3
[root@node1 ~]# source /etc/profile.d/custom.sh
- 1
【2018-01-27補充】
感謝吳家行hang的提醒，這裡node2和node3也要進行類似的環境變數配置。
3.4.2 準備工作
由於前面在node1上部署了Hadoop單機模式，需要停止Hadoop所有服務並清除資料目錄。順便檢驗一下設定的Hadoop環境變數。
清除Hadoop資料目錄
[root@node1 ~]# rm -rf /tmp/hadoop-root/
- 1
3.4.2 core-site.xml
-
[root
@node1 ~]
# cd /opt/hadoop-2.7.3/etc/hadoop/
-
[root
@node1 hadoop]
# vi core-site.xml
- 1
- 2
core-site.xml檔案內容如下：
-
<?xml version="1.0" encoding="UTF-8"?>
-
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-
<configuration>
-
<property>
-
<name>fs.defaultFS
</name>
-
<value>hdfs://node1:9000
</value>
-
</property>
-
<property>
-
<name>hadoop.tmp.dir
</name>
-
<value>/var/data/hadoop
</value>
-
</property>
-
<property>
-
<name>io.file.buffer.size
</name>
-
<value>65536
</value>
-
</property>
-
</configuration>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
3.4.3 hdfs-site.xml
[root@node1 hadoop]# vi hdfs-site.xml
- 1
hdfs-site.xml檔案內容如下：
-
<?xml version="1.0" encoding="UTF-8"?>
-
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-
<configuration>
-
<property>
-
<name>dfs.replication
</name>
-
<value>3
</value>
-
</property>
-
<property>
-
<name>dfs.namenode.secondary.http-address
</name>
-
<value>node2:50090
</value>
-
</property>
-
<property>
-
<name>dfs.namenode.secondary.https-address
</name>
-
<value>node2:50091
</value>
-
</property>
-
</configuration>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
3.4.4 slaves
編輯slaves檔案
[root@node1 hadoop]# vi slaves
- 1
slaves檔案內容設定為：
- node1
- node2
- node3
- 1
- 2
- 3
3.4.5 分發檔案
將Hadoop軟體包複製到node2和node3節點上
[root@node1 ~]# scp -r /opt/hadoop-2.7.3/ node2:/opt
- 1
[root@node1 ~]# scp -r /opt/hadoop-2.7.3/ node3:/opt
- 1
將環境變數檔案複製到node2和node3節點上
[root@node1 ~]# scp /etc/profile.d/custom.sh node2:/etc/profile.d
- 1
[root@node1 ~]# scp /etc/profile.d/custom.sh node3:/etc/profile.d
- 1
最後source一下
[root@node2 ~]# source /etc/profile.d/custom.sh
- 1
[root@node3 ~]# source /etc/profile.d/custom.sh
- 1
3.4.6 NameNode格式化
[root@node1 ~]# hdfs namenode -format
- 1
-
************************************************************/
-
17/
05/
14
09:
17:
28
INFO namenode
.NameNode: registered
UNIX signal handlers
for [
TERM,
HUP,
INT]
-
17/
05/
14
09:
17:
28
INFO namenode
.NameNode: createNameNode [-format]
-
Formatting using clusterid:
CID-
29bae3d3-
1786-
4428-
8359-
077976fe15e5
-
17/
05/
14
09:
17:
30
INFO namenode
.FSNamesystem:
No
KeyProvider found.
-
17/
05/
14
09:
17:
30
INFO namenode
.FSNamesystem: fsLock
is fair:
true
-
17/
05/
14
09:
17:
30
INFO blockmanagement
.DatanodeManager: dfs
.block
.invalidate
.limit=
1000
-
17/
05/
14
09:
17:
30
INFO blockmanagement
.DatanodeManager: dfs
.namenode
.datanode
.registration
.ip-hostname-check=
true
-
17/
05/
14
09:
17:
30
INFO blockmanagement
.BlockManager: dfs
.namenode
.startup
.delay
.block
.deletion
.sec
is
set to
000:
00:
00:
00.000
-
17/
05/
14
09:
17:
30
INFO blockmanagement
.BlockManager:
The block deletion will start around
2017
May
14
09:
17:
30
-
17/
05/
14
09:
17:
30
INFO util
.GSet:
Computing capacity
for
map
BlocksMap
-
17/
05/
14
09:
17:
30
INFO util
.GSet:
VM type =
64-bit
-
17/
05/
14
09:
17:
30
INFO util
.GSet:
2.0%
max memory
966.7
MB =
19.3
MB
-
17/
05/
14
09:
17:
30
INFO util
.GSet: capacity =
2^
21 =
2097152 entries
-
17/
05/
14
09:
17:
30
INFO blockmanagement
.BlockManager: dfs
.block
.access
.token
.enable=
false
-
17/
05/
14
09:
17:
30
INFO blockmanagement
.BlockManager: defaultReplication =
3
-
17/
05/
14
09:
17:
30
INFO blockmanagement
.BlockManager: maxReplication =
512
-
17/
05/
14
09:
17:
30
INFO blockmanagement
.BlockManager: minReplication =
1
-
17/
05/
14
09:
17:
30
INFO blockmanagement
.BlockManager: maxReplicationStreams =
2
-
17/
05/
14
09:
17:
30
INFO blockmanagement
.BlockManager: replicationRecheckInterval =
3000
-
17/
05/
14
09:
17:
30
INFO blockmanagement
.BlockManager: encryptDataTransfer =
false
-
17/
05/
14
09:
17:
30
INFO blockmanagement
.BlockManager: maxNumBlocksToLog =
1000
-
17/
05/
14
09:
17:
30
INFO namenode
.FSNamesystem: fsOwner = root (auth:
SIMPLE)
-
17/
05/
14
09:
17:
30
INFO namenode
.FSNamesystem: supergroup = supergroup
-
17/
05/
14
09:
17:
30
INFO namenode
.FSNamesystem: isPermissionEnabled =
true
-
17/
05/
14
09:
17:
30
INFO namenode
.FSNamesystem:
HA
Enabled:
false
-
17/
05/
14
09:
17:
30
INFO namenode
.FSNamesystem:
Append
Enabled:
true
-
17/
05/
14
09:
17:
31
INFO util
.GSet:
Computing capacity
for
map
INodeMap
-
17/
05/
14
09:
17:
31
INFO util
.GSet:
VM type =
64-bit
-
17/
05/
14
09:
17:
31
INFO util
.GSet:
1.0%
max memory
966.7
MB =
9.7
MB
-
17/
05/
14
09:
17:
31
INFO util
.GSet: capacity =
2^
20 =
1048576 entries
-
17/
05/
14
09:
17:
31
INFO namenode
.FSDirectory:
ACLs enabled?
false
-
17/
05/
14
09:
17:
31
INFO namenode
.FSDirectory:
XAttrs enabled?
true
-
17/
05/
14
09:
17:
31
INFO namenode
.FSDirectory:
Maximum size of an xattr:
16384
-
17/
05/
14
09:
17:
31
INFO namenode
.NameNode:
Caching file names occuring more than
10 times
-
17/
05/
14
09:
17:
31
INFO util
.GSet:
Computing capacity
for
map cachedBlocks
-