1、HDFS

Hadoop分散式檔案系統，為海量的資料提供了儲存。

2、HDFS基本概念

（1）塊

HDFS的檔案被分成塊進行儲存，HDFS塊的預設大小是64MB，塊是檔案儲存處理的邏輯單元。

（2）節點

HDFS叢集有兩類節點，並以管理者-工作者模式執行，即一個NameNode（管理者，管理節點，存放元資料）和多個DataNode（工作者，工作節點，存放資料塊）。

一個HDFS cluster包含一個NameNode和若干的DataNode，NameNode主要負責管理hdfs檔案系統，DataNode主要是用來儲存資料檔案。

3、命令列操作

（1）建立目錄，hadoop fs -mkdir /test

[[email protected] sbin]# hadoop fs -mkdir /test

（2）列出檔案，hadoop fs -ls /

[[email protected] sbin]# hadoop fs -ls /
Found 1 items
drwxr-xr-x   - root supergroup          0 2016-11-25 18:46 /test

（3）從本地系統拷貝檔案到DFS，hadoop fs -put hadoop-env.sh /test/

[[email protected] hadoop]# hadoop fs -put hadoop-env.sh /test/
 

（4）顯示檔案內容，hadoop fs -cat /test/hadoop-env.sh

[[email protected] hadoop]# hadoop fs -cat /test/hadoop-env.sh

（5）從DFS拷貝檔案到本地檔案系統，hadoop fs -get /test/hadoop-env.sh hadoop-env2.sh

[[email protected] hadoop]# hadoop fs -get /test/hadoop-env.sh hadoop-env2.sh
16/11/25 18:57:02 WARN hdfs.DFSClient: DFSInputStream has been closed already
[
 
[email protected] hadoop]# ls
capacity-scheduler.xml      httpfs-env.sh            mapred-queues.xml.template
configuration.xsl           httpfs-log4j.properties  mapred-site.xml
container-executor.cfg      httpfs-signature.secret  mapred-site.xml.template
core-site.xml               httpfs-site.xml          slaves
hadoop-env2.sh              kms-acls.xml             ssl-client.xml.example
hadoop-env.cmd              kms-env.sh               ssl-server.xml.example
hadoop-env.sh               kms-log4j.properties     yarn-env.cmd
hadoop-metrics2.properties  kms-site.xml             yarn-env.sh
hadoop-metrics.properties   log4j.properties         yarn-site.xml
hadoop-policy.xml           mapred-env.cmd
hdfs-site.xml               mapred-env.sh

（6）顯示檔案系統的基本資料，hadoop dfsadmin -report

[[email protected] hadoop]# hadoop dfsadmin -report

DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.


Configured Capacity: 18746441728 (17.46 GB)
Present Capacity: 12141334709 (11.31 GB)
DFS Remaining: 12141318144 (11.31 GB)
DFS Used: 16565 (16.18 KB)
DFS Used%: 0.00%
Under replicated blocks: 1
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0


-------------------------------------------------
Live datanodes (1):


Name: 127.0.0.1:50010 (localhost)
Hostname: localhost
Decommission Status : Normal
Configured Capacity: 18746441728 (17.46 GB)
DFS Used: 16565 (16.18 KB)
Non DFS Used: 6605107019 (6.15 GB)
DFS Remaining: 12141318144 (11.31 GB)
DFS Used%: 0.00%
DFS Remaining%: 64.77%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Nov 25 18:49:47 CST 2016

（7）刪除操作，hadoop fs -rm /test/hadoop-env.sh

[[email protected] sbin]# hadoop fs -rm /test/hadoop-env.sh
16/11/26 12:02:00 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /test/hadoop-env.sh