Map-Reduce測試（分散式運算）

[[email protected] ~]$ hadoop dfs -ls dir1
Found 3 items
-rw-r–r-- 2 hadoop supergroup 13 2013-10-30 14:33 /user/hadoop/dir1/file1
-rw-r–r-- 2 hadoop supergroup 13 2013-10-30 14:33 /user/hadoop/dir1/file2
[[email protected] ~]$ hadoop dfs -cat dir1/file1
hello slave1
[[email protected]

~]$ hadoop dfs -cat dir1/file2
hello slave2
[[email protected] ~]$ ls
Desktop dir1 Documents Downloads Music name Pictures Public Templates tmp Videos
[[email protected] ~]$ ls /tmp/hadoop-0.20.2/
bin CHANGES.txt docs hadoop-0.20.2-examples.jar ivy librecordio NOTICE.txt webapps
build.xml conf hadoop-0.20.2-ant.jar hadoop-0.20.2-test.jar ivy.xml LICENSE.txt README.txt
c++ contrib hadoop-0.20.2-core.jar hadoop-0.20.2-tools.jar lib logs src
[

[email protected] ~]$ hadoop jar /tmp/hadoop-0.20.2/hadoop-0.20.2-examples.jar wordcout dir1 out1
Unknown program ‘wordcout’ chosen.
Valid program names are:
aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
dbcount: An example job that count the pageview counts from a database.
grep: A map/reduce program that counts the matches of a regex in the input.
join: A job that effects a join over sorted, equally partitioned datasets
multifilewc: A job that counts words from several files.
pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
pi: A map/reduce program that estimates Pi using monte-carlo method.
randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
randomwriter: A map/reduce program that writes 10GB of random data per node.
secondarysort: An example defining a secondary sort to the reduce.
sleep: A job that sleeps at each map and reduce task.
sort: A map/reduce program that sorts the data written by the random writer.
sudoku: A sudoku solver.
teragen: Generate data for the terasort
terasort: Run the terasort
teravalidate: Checking results of terasort
wordcount: A map/reduce program that counts the words in the input files.
[

[email protected] ~]$ hadoop jar /tmp/hadoop-0.20.2/hadoop-0.20.2-examples.jar wordcount dir1 out1
13/10/30 14:52:57 INFO input.FileInputFormat: Total input paths to process : 3
13/10/30 14:52:58 INFO mapred.JobClient: Running job: job_201310301418_0001
13/10/30 14:52:59 INFO mapred.JobClient: map 0% reduce 0%
13/10/30 14:53:07 INFO mapred.JobClient: map 66% reduce 0%
13/10/30 14:53:10 INFO mapred.JobClient: map 100% reduce 0%
13/10/30 14:53:19 INFO mapred.JobClient: map 100% reduce 100%
13/10/30 14:53:21 INFO mapred.JobClient: Job complete: job_201310301418_0001
13/10/30 14:53:21 INFO mapred.JobClient: Counters: 17
13/10/30 14:53:21 INFO mapred.JobClient: Job Counters
13/10/30 14:53:21 INFO mapred.JobClient: Launched reduce tasks=1
13/10/30 14:53:21 INFO mapred.JobClient: Launched map tasks=3
13/10/30 14:53:21 INFO mapred.JobClient: Data-local map tasks=3
13/10/30 14:53:21 INFO mapred.JobClient: FileSystemCounters
13/10/30 14:53:21 INFO mapred.JobClient: FILE_BYTES_READ=81
13/10/30 14:53:21 INFO mapred.JobClient: HDFS_BYTES_READ=39
13/10/30 14:53:21 INFO mapred.JobClient: FILE_BYTES_WRITTEN=270
13/10/30 14:53:21 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=26
13/10/30 14:53:21 INFO mapred.JobClient: Map-Reduce Framework
13/10/30 14:53:21 INFO mapred.JobClient: Reduce input groups=3
13/10/30 14:53:21 INFO mapred.JobClient: Combine output records=6
13/10/30 14:53:21 INFO mapred.JobClient: Map input records=3
13/10/30 14:53:21 INFO mapred.JobClient: Reduce shuffle bytes=62
13/10/30 14:53:21 INFO mapred.JobClient: Reduce output records=3
13/10/30 14:53:21 INFO mapred.JobClient: Spilled Records=12
13/10/30 14:53:21 INFO mapred.JobClient: Map output bytes=63
13/10/30 14:53:21 INFO mapred.JobClient: Combine input records=6
13/10/30 14:53:21 INFO mapred.JobClient: Map output records=6
13/10/30 14:53:21 INFO mapred.JobClient: Reduce input records=6
[[email protected] ~]$ hadoop dfs -ls
Found 3 items
drwxr-xr-x - hadoop supergroup 0 2013-10-30 14:33 /user/hadoop/dir1
drwxr-xr-x - hadoop supergroup 0 2013-10-30 14:53 /user/hadoop/out1
-rw-r–r-- 2 hadoop supergroup 1761 2013-10-30 14:31 /user/hadoop/passwd.new
[[email protected] ~]$ hadoop dfs -ls out1
Found 2 items
drwxr-xr-x - hadoop supergroup 0 2013-10-30 14:52 /user/hadoop/out1/_logs
-rw-r–r-- 2 hadoop supergroup 26 2013-10-30 14:53 /user/hadoop/out1/part-r-00000
[[email protected] ~]$ hadoop dfs -cat out1/part-r-00000
hello 2
slave1 1
slave2 1

從jobTracker提供的web介面檢視資訊（Map-Reduce）

[[email protected] ~]$ firefox http://master:50030 &