flume例項(二):監控伺服器日誌
阿新 • • 發佈:2018-12-14
1.實現功能:
監控日誌增加,並且將日誌資訊儲存到hdfs上
2.flume和hdfs整合
將hadoop對應jar包放到flume的lib目錄下
htrace-core-3.1.0-incubating.jar
hadoop-hdfs-2.7.3.jar
hadoop-common-2.7.3.jar
hadoop-auth-2.7.3.jar
commons-io-2.4.jar
commons-configuration-1.6.jar
3.配置檔案
#a1是代理 # Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = exec a1.sources.r1.command = tail -f /var/log/httpd/access_log a1.sources.r1.channels = c1 # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Describe the sink a1.sinks.k1.type=hdfs a1.sinks.k1.hdfs.path=hdfs://bigdata.ibeifeng.com:8020/flume02 a1.sinks.k1.channel = c1
4.啟動命令
(1)啟動hdfs
(2)啟動flume
bin/flume-ng agent --name a1 --conf conf --conf-file conf/apache-sink-hdfs.conf