Flume+hbase 日誌資料採集與儲存
阿新 • • 發佈：2018-12-31
瞭解過flume的人，差不多都看過這張或則類似的圖片，本文即實現上圖部分內容。（由於條件有限，目前是單機上實現）
flume-agent配置檔案
#flume agent conf source_agent.sources = server source_agent.sinks = avroSink source_agent.channels = memoryChannel source_agent.sources.server.type = exec source_agent.sources.server.command = tail -F /data/hudonglogs/self/channel.log source_agent.sources.server.channels= memoryChannel source_agent.channels.memoryChannel.type = memory source_agent.channels.memoryChannel.capacity = 1000 source_agent.channels.memoryChannel.transactionCapacity = 100 source_agent.sinks = avroSink source_agent.sinks.avroSink.type = avro source_agent.sinks.avroSink.hostname = 127.0.0.1 source_agent.sinks.avroSink.port= 41414 source_agent.sinks.avroSink.channel = memoryChannel
flume-hbase 配置檔案
#hbase flume conf sinks collector.sources = avroSource collector.sinks = hbaseSink collector.channels =memChannel collector.sources.avroSource.type = avro collector.sources.avroSource.bind = 127.0.0.1 collector.sources.avroSource.port= 41414 collector.sources.avroSource.channels = memChannel collector.channels.memChannel.type = memory collector.channels.memChannel.capacity = 1000 collector.sinks.hbaseSink.type = asynchbase collector.sinks.hbaseSink.channel = memChannel collector.sinks.hbaseSink.table = logs collector.sinks.hbaseSink.columnFamily = content collector.sinks.hbaseSink.batchSize = 5
hbase中建立表logs 以及列族content
create 'logs','content'
啟動flume
nohup $FLUME_HOME/bin/flume-ng agent -c $FLUME_HOME/conf -f $FLUME_HOME/conf/flume-hbase.conf -n collector &
nohup $FLUME_HOME/bin/flume-ng agent -c $FLUME_HOME /conf -f $FLUME_HOME/conf/flume-agent.conf -n source_agent &
注：這兩條命令啟動是有順序的，先啟動collector，否則會報錯
在不斷向檔案中寫入日誌後，可以使用scan命令檢視hbase
scan 'logs'
ROW COLUMN+CELL default09626ade-3f37-49c3-a930-270ef7119dd3 column=content:pCol, timestamp=1454480049481, value=2016-02-03 12:19:36,260 com.hudong.test.Test01 [main] [INFO]-(Test01.java:22) 145 4473176260 default09ba09cc-326c-465c-90f9-533c923923a0 column=content:pCol, timestamp=1454480074182, value=2016-02-03 14:14:22,179 com.hudong.test.Test01 [main] [INFO]-(Test01.java:22) 145 4480062179 default0c199142-05f8-49c6-b341-8158bb445861 column=content:pCol, timestamp=1454480113199, value=2016-02-03 14:15:07,196 com.hudong.test.Test01 [main] [INFO]-(Test01.java:22) 145 4480107196 default0cbb050e-3756-44f2-8c66-419a6e52fd97 column=content:pCol, timestamp=1454480049493, value=2016-02-03 12:19:42,263 com.hudong.test.Test01 [main] [INFO]-(Test01.java:22) 145 4473182263 default0f8eb841-7ccd-4abf-9dcf-4fc26d3f55f4 column=content:pCol, timestamp=1454480079188, value=2016-02-03 14:14:34,185 com.hudong.test.Test01 [main] [INFO]-(Test01.java:22) 145 4480074185 default133b7acf-ca3c-46e9-90a8-27d553721900 column=content:pCol, timestamp=1454480121214, value=2016-02-03 14:15:18,200 com.hudong.test.Test01 [main] [INFO]-(Test01.java:22) 145 4480118200 default13e2a5bc-f79a-4ffe-86a0-bf0ea37d8f35 column=content:pCol, timestamp=1454480139209, value=2016-02-03 14:15:35,206 com.hudong.test.Test01 [main] [INFO]-(Test01.java:22) 145 4480135206 default19cae7ca-87ba-42b5-9f13-136c89c0c20e column=content:pCol, timestamp=1454480094196, value=2016-02-03 14:14:50,190 com.hudong.test.Test01 [main] [INFO]-(Test01.java:22) 145 4480090190 default1aba06a5-973d-403a-91de-afacc9536172 column=content:pCol, timestamp=1454480139209, value=2016-02-03 14:15:34,206 com.hudong.test.Test01 [main] [INFO]-(Test01.java:22) 145 4480134206 default1b240a9c-2b35-4649-b199-29a27a0c4db2 column=content:pCol, timestamp=1454480049481, value=2016-02-03 12:19:40,262 com.hudong.test.Test01 [main] [INFO]-(Test01.java:22) 145 4473180262 default1d92a0e3-b427-4958-bae7-c7bb1e9d20ef column=content:pCol, timestamp=1454480106197, value=2016-02-03 14:14:59,193 com.hudong.test.Test01 [main] [INFO]-(Test01.java:22) 145 4480099193 default25ef2b1b-6d6c-4431-885e-c2e4d2297f59 column=content:pCol, timestamp=1454480085201, value=2016-02-03 14:14:42,187 com.hudong.test.Test01 [main] [INFO]-(Test01.java:22) 145 4480082187
另：
對於現在的日誌格式可能不滿意，flume支援自定義Serializer，用於日誌的清洗，實現AsyncHbaseEventSerializer介面即可，並將flume重新打包，然後將相應的jar複製到flume home目錄下lib資料夾下，舊可以！
介面的實現，參照博文：http://blog.csdn.net/yaoyasong/article/details/39400829