net %d 註意 path 文件 ast and bubuko ans

配置好Nginx後，通過flume收集日誌到hdfs

可參考flume的文件

用flume的案例二

執行的註意點

avro和exec聯合用法

https://blog.csdn.net/HG_Harvey/article/details/78358304

exec實質是收集文件

spool用法

https://blog.csdn.net/a_drjiaoda/article/details/84954593

或者下面這個代碼

名字為

conf/job/project/flume-hdfs.conf

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /opt/data/access.log

# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://master:9000/project/log/%Y%m%d
a1.sinks.k1.hdfs.filePrefix = events-

a1.sinks.k1.hdfs.rollInterval = 0
a1.sinks.k1.hdfs.rollSize = 10240000
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.useLocalTimeStamp = true
a1.sinks.k1.hdfs.callTimeout = 60000
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.idleTimeout = 10

# Use a channel which buffers events in memory
a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

啟動hdfs的前提下

start-all.sh

執行

flume-ng agent --conf conf/ --name a1 --conf-file conf/job/project/flume-hdfs.conf

配置好Nginx後，通過flume收集日誌到hdfs