學習Hadoop權威指南之Hadoop執行MapReduce日誌檢視
修改map配置檔案 mapred-site.xml
[[email protected] yar]# vim /opt/hadoop-2.8.3/etc/hadoop/mapred-site.xml
<property>
<name>mapreduce.jobhistory.address </name>
<value>bigdata.cqmfin.com:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>bigdata.cqmfin.com:19888</value>
</property>
修改yarn配置檔案 yarn-site.xml
yarn.nodemanager.delete.debug-delay-sec
預設值:0,app執行完之後立即刪除本地檔案
desc:應用程式完成之後 NodeManager 的 DeletionService 刪除應用程式的本地化檔案和日誌目錄之前的時間(秒數)。要診斷 YARN 應用程式問題,請將此屬性的值設為足夠大(例如,設為 600 秒,即 10 分鐘)以允許檢查這些目錄。
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/opt/hadoop/logs/yar/local</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/opt/hadoop/logs/yar/log</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://bigdata.cqmfin.com:19888/jobhistory/logs</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.delete.debug-delay-sec</name>
<value>60000</value>
</property>
啟動服務
[[email protected] yar]mr-jobhistory-daemon.sh start historyserver
檢視是否有JobHistoryServer
[[email protected] yar]# jps
15098 Jps
13338 NodeManager
12875 DataNode
13052 SecondaryNameNode
13229 ResourceManager
8765 JobHistoryServer
12735 NameNode
執行MapReducer
檢視輸入檔案
[[email protected] yar]# hadoop dfs -cat /user/input/sample.txt
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=512M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1024m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=512M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1024m; support was removed in 8.0
18/08/08 22:47:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
0067011990999991950051507004+68750+023550FM-12+038299999V0203301N00671220001CN9999999N9+00001+99999999999
0043011990999991950051512004+68750+023550FM-12+038299999V0203201N00671220001CN9999999N9+00221+99999999999
0043011990999991950051518004+68750+023550FM-12+038299999V0203201N00261220001CN9999999N9-00111+99999999999
0043012650999991949032412004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+01111+99999999999
0043012650999991949032418004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+00781+99999999999
設定環境
[[email protected] jar]# export HADOOP_CLASSPATH=/opt/jar/hadoop-examples-mapReducer.jar
刪除輸出檔案
[[email protected] jar]# hadoop dfs -rm -f -R /user/input/outputMax
執行測試
[[email protected] opt]# hadoop MaxTemperature /user/input/sample.txt /user/input/outputMax
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=512M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1024m; support was removed in 8.0
18/08/08 20:46:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/08/08 20:46:47 INFO client.RMProxy: Connecting to ResourceManager at bigdata.cqmfin.com/192.168.100.131:8032
18/08/08 20:46:47 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
18/08/08 20:46:48 INFO input.FileInputFormat: Total input files to process : 1
18/08/08 20:46:48 INFO mapreduce.JobSubmitter: number of splits:1
18/08/08 20:46:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1533786361328_0001
18/08/08 20:46:49 INFO impl.YarnClientImpl: Submitted application application_1533786361328_0001
18/08/08 20:46:49 INFO mapreduce.Job: The url to track the job: http://bigdata.cqmfin.com:8088/proxy/application_1533786361328_0001/
18/08/08 20:46:49 INFO mapreduce.Job: Running job: job_1533786361328_0001
18/08/08 20:46:56 INFO mapreduce.Job: Job job_1533786361328_0001 running in uber mode : false
18/08/08 20:46:56 INFO mapreduce.Job: map 0% reduce 0%
18/08/08 20:47:00 INFO mapreduce.Job: map 100% reduce 0%
18/08/08 20:47:05 INFO mapreduce.Job: map 100% reduce 100%
18/08/08 20:47:06 INFO mapreduce.Job: Job job_1533786361328_0001 completed successfully
18/08/08 20:47:06 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=61
FILE: Number of bytes written=315249
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=650
HDFS: Number of bytes written=17
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=2212
Total time spent by all reduces in occupied slots (ms)=2402
Total time spent by all map tasks (ms)=2212
Total time spent by all reduce tasks (ms)=2402
Total vcore-milliseconds taken by all map tasks=2212
Total vcore-milliseconds taken by all reduce tasks=2402
Total megabyte-milliseconds taken by all map tasks=2265088
Total megabyte-milliseconds taken by all reduce tasks=2459648
Map-Reduce Framework
Map input records=5
Map output records=5
Map output bytes=45
Map output materialized bytes=61
Input split bytes=117
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=61
Reduce input records=5
Reduce output records=2
Spilled Records=10
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=108
CPU time spent (ms)=950
Physical memory (bytes) snapshot=453992448
Virtual memory (bytes) snapshot=4193198080
Total committed heap usage (bytes)=323485696
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=533
File Output Format Counters
Bytes Written=17
[[email protected] yar]# pwd
/opt/hadoop/logs/yar
查詢日誌檔案
[[email protected] yar]# grep -R 'write,key' *
log/application_1533786361328_0001/container_1533786361328_0001_01_000002/stdout:--------------------->>>>>write,key:1950,value:0
log/application_1533786361328_0001/container_1533786361328_0001_01_000002/stdout:--------------------->>>>>write,key:1950,value:22
log/application_1533786361328_0001/container_1533786361328_0001_01_000002/stdout:--------------------->>>>>write,key:1950,value:-11
log/application_1533786361328_0001/container_1533786361328_0001_01_000002/stdout:--------------------->>>>>write,key:1949,value:111
log/application_1533786361328_0001/container_1533786361328_0001_01_000002/stdout:--------------------->>>>>write,key:1949,value:78
[[email protected] yar]# grep -R 'write,key' *
log/application_1533786361328_0001/container_1533786361328_0001_01_000002/stdout:--------------------->>>>>write,key:1950,value:0
log/application_1533786361328_0001/container_1533786361328_0001_01_000002/stdout:--------------------->>>>>write,key:1950,value:22
log/application_1533786361328_0001/container_1533786361328_0001_01_000002/stdout:--------------------->>>>>write,key:1950,value:-11
log/application_1533786361328_0001/container_1533786361328_0001_01_000002/stdout:--------------------->>>>>write,key:1949,value:111
log/application_1533786361328_0001/container_1533786361328_0001_01_000002/stdout:--------------------->>>>>write,key:1949,value:78
[[email protected] yar]# grep -R 'maxValue' *
log/application_1533786361328_0001/container_1533786361328_0001_01_000003/stdout:--------reducer>> maxValue:1949,maxValue:111
log/application_1533786361328_0001/container_1533786361328_0001_01_000003/stdout:--------reducer>> maxValue:1950,maxValue:22
[[email protected] yar]# ^C
[[email protected] yar]# ^C
檢視日誌檔案
[[email protected] yar]# cat log/application_1533786361328_0001/container_1533786361328_0001_01_000003/stdout
--------reducer>> key:1949,values:o[email protected]56f6d40b
--------reducer>> maxValue:1949,maxValue:111
--------reducer>> key:1950,values:o[email protected]56f6d40b
--------reducer>> maxValue:1950,maxValue:22
[[email protected] yar]# cat log/application_1533786361328_0001/container_1533786361328_0001_01_000002/stdout
--------------------->>>>>,key:0,value:0067011990999991950051507004+68750+023550FM-12+038299999V0203301N00671220001CN9999999N9+00001+99999999999
--------------------->>>>>write,key:1950,value:0
--------------------->>>>>,key:107,value:0043011990999991950051512004+68750+023550FM-12+038299999V0203201N00671220001CN9999999N9+00221+99999999999
--------------------->>>>>write,key:1950,value:22
--------------------->>>>>,key:214,value:0043011990999991950051518004+68750+023550FM-12+038299999V0203201N00261220001CN9999999N9-00111+99999999999
--------------------->>>>>write,key:1950,value:-11
--------------------->>>>>,key:321,value:0043012650999991949032412004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+01111+99999999999
--------------------->>>>>write,key:1949,value:111
--------------------->>>>>,key:428,value:0043012650999991949032418004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+00781+99999999999
--------------------->>>>>write,key:1949,value:78