需求: 將131上產生的日誌文件負載均衡到 132 和133機器上

如圖:

配置132 機器和133機器 (相同))

arov.conf

a1.sources = r1
a1.sinks = k1
a1.channels = c1
   
# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind =0.0.0.0
a1.sources.r1.port =41414
 
   
# Describe the sink
a1.sinks.k1.type = logger
   
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
   
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

啟動132 和133 該flume進程 命令 註意一定要先啟動不然在131訪問的時候會說連接被拒絕

bin/flume-ng agent --name a1 --conf-file conf/avro.conf --conf conf/ -Dflume.root.logger=INFO,console -Dflume.monitoring.type=http -Dflume.monitoring.port=34343 &

配置131機器的flume

load_balance.conf

a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1
   
# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /opt/test.log
   
# Describe the sink 1
a1.sinks.k1.type = avro
a1.sinks.k1.hostname=192.168.13.132
a1.sinks.k1.port=41414

# Describe the sink 2
a1.sinks.k2.type = avro
a1.sinks.k2.hostname=192.168.13.133
a1.sinks.k2.port=41414
   
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
   
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c1

啟動命令

bin/flume-ng agent --name a1 --conf-file conf/load_balance.conf --conf conf/ -Dflume.root.logger=INFO,console &

測試:

在131機器上創建文件

執行命令查看132 和133機器上收到的信息.是否負載成功

echo ‘adddl‘ >> /opt/test.log

案例15:avro 負載均衡的方式動態獲取文件中內容