Linux執行 Hadoop WordCount

ubuntu 終端進入快捷鍵 ：ctrl + Alt +t

hadoop啟動命令：start-all.sh

正常執行效果如下：

[email protected]:~$ start-all.sh

Warning: $HADOOP_HOME is deprecated.

starting namenode, logging to /home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-namenode-HADOOP.MAIN.out

HADOOP.MAIN: starting datanode, logging to/home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-datanode-HADOOP.MAIN.out

HADOOP.MAIN: starting secondarynamenode,logging to/home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-secondarynamenode-HADOOP.MAIN.out

starting jobtracker, logging to/home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-jobtracker-HADOOP.MAIN.out

HADOOP.MAIN: starting tasktracker, loggingto /home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-tasktracker-HADOOP.MAIN.out

jps命令檢視啟動的hadoop服務

[email protected]:~$ jps

3615 Jps

2699 NameNode

3461 TaskTracker

2922 DataNode

3137 SecondaryNameNode

3231 JobTracker

本地建立一個資料夾

[email protected]:~$ mkdir ~/file

在file檔案建立兩個txt檔案

[email protected]:~$ cd file

[email protected]:~/file$ echo "Hello World" > file1.txt

[email protected]:~/file$ echo "Hello Hadoop" > file2.txt

[email protected]:~/file$ ls

file1.txt file2.txt

[email protected]:~/file$

在HDFS上建立一個輸入資料夾

[email protected]:~/file$ hadoop fs -mkdir input

檢視建立的input資料夾路徑

[email protected]:~$ hadoop fs -ls

Warning: $HADOOP_HOME is deprecated.

Found 5 items

-rw-r--r--  3 Administrator supergroup   6296230 2014-09-03 10:38 /user/hadoop/cloud.txt

drwxr-xr-x  - hadoop        supergroup          0 2014-09-02 16:31/user/hadoop/hadi_curbm

drwxr-xr-x  - hadoop        supergroup          0 2014-09-04 09:59 /user/hadoop/input

drwxr-xr-x  - hadoop        supergroup          0 2014-09-02 16:31/user/hadoop/pegasus

[email protected]:~$

可以看到目錄被建立到 /user/hadoop/input 目錄

上傳本地file檔案到input目錄

[email protected]:~$hadoop fs put ~/file/*.txt /user/hadoop/input

找到hadoop目錄下的examples.jar 程式包

[email protected]:~$ cd hadoop-1.1.2

[email protected]:~/hadoop-1.1.2$ ls

bin         docs                          hadoop-test-1.1.2.jar   LICENSE.txt src

build.xml   hadoop-ant-1.1.2.jar         hadoop-tools-1.1.2.jar  logs         webapps

c++         hadoop-client-1.1.2.jar      ivy                    NOTICE.txt   wordcount.jar

CHANGES.txt hadoop-core-1.1.2.jar        ivy.xml                 README.txt

conf        hadoop-examples-1.1.2.jar     lib                     sbin

contrib     hadoop-minicluster-1.1.2.jar libexec                 share

[email protected]:~/hadoop-1.1.2$

執行jar程式程式碼 統計input目錄下檔案的Wordcount

[email protected]:~$ hadoop jar /home/hadoop/hadoop-1.1.2/hadoop-examples-1.1.2.jar wordcount  /user/hadoop/input output

Warning: $HADOOP_HOME is deprecated.

14/09/04 10:10:44 INFOinput.FileInputFormat: Total input paths to process : 0

14/09/04 10:10:45 INFO mapred.JobClient:Running job: job_201409040943_0001

14/09/04 10:10:46 INFOmapred.JobClient:  map 0% reduce 0%

14/09/04 10:10:54 INFOmapred.JobClient:  map 0% reduce 100%

14/09/04 10:10:55 INFO mapred.JobClient:Job complete: job_201409040943_0001

14/09/04 10:10:55 INFO mapred.JobClient:Counters: 18

14/09/04 10:10:55 INFOmapred.JobClient:   Job Counters

14/09/04 10:10:55 INFOmapred.JobClient:     Launched reducetasks=1

14/09/04 10:10:55 INFOmapred.JobClient:    SLOTS_MILLIS_MAPS=4087

14/09/04 10:10:55 INFOmapred.JobClient:     Total time spent byall reduces waiting after reserving slots (ms)=0

14/09/04 10:10:55 INFOmapred.JobClient:     Total time spent byall maps waiting after reserving slots (ms)=0

14/09/04 10:10:55 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=4068

14/09/04 10:10:55 INFOmapred.JobClient:   File Output FormatCounters

14/09/04 10:10:55 INFOmapred.JobClient:     Bytes Written=0

14/09/04 10:10:55 INFOmapred.JobClient:   FileSystemCounters

14/09/04 10:10:55 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=55309

14/09/04 10:10:55 INFOmapred.JobClient:   Map-Reduce Framework

14/09/04 10:10:55 INFOmapred.JobClient:     Reduce inputgroups=0

14/09/04 10:10:55 INFOmapred.JobClient:     Combine outputrecords=0

14/09/04 10:10:55 INFOmapred.JobClient:     Reduce shufflebytes=0

14/09/04 10:10:55 INFOmapred.JobClient:     Physical memory(bytes) snapshot=35037184

14/09/04 10:10:55 INFOmapred.JobClient:     Reduce outputrecords=0

14/09/04 10:10:55 INFO mapred.JobClient:     Spilled Records=0

14/09/04 10:10:55 INFOmapred.JobClient:     CPU time spent(ms)=120

14/09/04 10:10:55 INFOmapred.JobClient:     Total committedheap usage (bytes)=15925248

14/09/04 10:10:55 INFOmapred.JobClient:     Virtual memory(bytes) snapshot=377499648

14/09/04 10:10:55 INFOmapred.JobClient:     Combine inputrecords=0

14/09/04 10:10:55 INFOmapred.JobClient:     Reduce inputrecords=0

[email protected]:~$

顯示結果

[email protected]:~$ hadoop fs -ls output

Warning: $HADOOP_HOME is deprecated.

Found 3 items

-rw-r--r--  1 hadoop supergroup          02014-09-04 10:10 /user/hadoop/output/_SUCCESS

drwxr-xr-x  - hadoop supergroup          02014-09-04 10:10 /user/hadoop/output/_logs

-rw-r--r--  1 hadoop supergroup          02014-09-04 10:10 /user/hadoop/output/part-r-00000

[email protected]:~$

檢視執行結果

[email protected]:~$ hadoop fs -cat output/part-r-00000

Hadoop 1

Hello   2

World  1