flume

安裝與配置

0、按照jdk。

官網分別下載jdk-1.8.0和apache-flume二進位制包
設定軟體路徑如下
jdk: /usr/local/jdk-1.8.0
flume: /opt/apache-flume

1、配置flume-env.sh

cd /opt/apache-flume-1.8.0/conf
cp flume-env.sh.template flume-env.sh
// 指向JAVA
echo "export JAVA_HOME=/usr/local/jdk1.8.0_162/" >> flume-env.sh

2、編輯配置檔案(vim default.conf)

# 設定配置組log
log.sinks    = sk
log.sources  = sc
log.channels = ch

# 配置sources
log.sources.sc.type = exec  # source型別:命令
log.sources.sc.command = tail -F /data/log/access.log

# 配置sinks
log.sinks.sk.type = org.apache.flume.sink.kafka.KafkaSink
log.sinks.sk.kafka.topic = test
log.sinks.sk.kafka.bootstrap
 
.servers = localhost:9092

# 配置channels。記錄checkpoint
log.channels.ch.type = file 
log.channels.ch.checkpointDir= /data/flume/check
log.channels.ch.dataDirs = /data/flume/data

# 將source和sink繫結到channle
log.sinks.sk.channel = ch
log.sources.sc.channel = ch

3、啟動flume

cd /opt/apache-flume-1.8.0
bin/flume-ng agent --
 
conf conf --conf-file conf/default.conf -name log

4、啟動kafka，監聽輸出資料