flume的source和sink
周海漢 2013.1.21
http://abloz.com
常用source源:
-
console
-
標準輸入控制檯
-
text(“filename”)
-
單文字檔案源,一行一事件
-
tail(“filename”)
-
和 Unix 的tail -F 類似。一行一事件。一直開啟等待資料,會跟蹤檔案切換。
-
multitail(“file1″[, ”file2”[, …]])
-
同 tail 源類似,但可以跟蹤多檔案。
-
asciisynth(msg_count,msg_size)
-
一個源,用於產生msg_count 個msg_size大小的隨機訊息,轉成可列印 ASCII字元。
-
syslogUdp(port)
-
UDP 埠上的 Syslog,和syslog相容。
-
syslogTcp(port)
-
TCP 埠上的 Syslog,和syslog-ng相容。
常用sink:
null
Null sink. Events are dropped.
console[("format")]
Console sink. Display to console’s stdout. The “format” argument is optional and defaults to the “debug” output format.
text("_txtfile_"[,"format"])
Textfile sink. Write the events to text file _txtfile_
using output format “format”. The default format is “raw” event bodies with no metadata.
dfs("_dfsfile_")
DFS seqfile sink. Write serialized Flume events to a dfs path such as hdfs://namenode/file
or file:///file
in Hadoop’s seqfile format. Note that because of the HDFS write semantics, no data for this sink write until the sink is closed.
syslogTcp("_host_",_port_)
Syslog TCP sink. Forward to events to host
on TCP port port
in syslog wire format (syslog-ng compatible), or to other Flume nodes setup to listen for syslogTcp.
附:所有flume的sinks,sources和decorators.
Sinks
accumulator |
agentBEChain |
agentBESink |
agentBestEffortSink |
agentDFOChain |
agentDFOSink |
agentE2EChain |
agentE2ESink |
agentFailoverSink |
agentSink |
autoBEChain |
autoDFOChain |
autoE2EChain |
avroSink |
collectorSink |
console |
counter |
counterHistory |
customdfs |
dfs |
escapedCustomDfs |
escapedFormatDfs |
fail |
failChain |
formatDfs |
ganglia |
irc |
logicalSink |
multigrep |
multigrepspec |
null |
regexhisto |
regexhistospec |
rpcSink |
seqfile |
syslogTcp |
text |
thriftSink |
Sources
asciisynth |
autoCollectorSource |
avroSource |
collectorSource |
console |
exec |
execPeriodic |
execStream |
fail |
irc |
log4jfile |
logicalSource |
multitail |
nonlsynth |
null |
report |
rpcSource |
scribe |
seqfile |
stdin |
synth |
synthrndsize |
syslogTcp |
syslogTcp1 |
syslogUdp |
tail |
tailDir |
text |
thriftSource |
tpriosource |
Decorators
ackChecker |
ackInjector |
ackedWriteAhead |
batch |
benchinject |
benchreport |
bloomCheck |
bloomGen |
choke |
delay |
digest |
diskFailover |
exDate |
flakeyAppend |
format |
gunzip |
gzip |
inmem |
insistentAppend |
insistentOpen |
intervalDroppyAppend |
intervalFlakeyAppend |
intervalSampler |
lazyOpen |
mask |
mult |
nullDeco |
probSampler |
regex |
regexAll |
reservoirSampler |
select |
split |
stubbornAppend |
unbatch |
value |
如非註明轉載, 均為原創. 本站遵循知識共享CC協議,轉載請註明來源