Flume監控指標項
阿新 • • 發佈：2018-11-28
配置監控
修改flume.env.sh
export JAVA_OPTS=”-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5445 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false”使用以下指令碼
flume-ng agent \ --name a1 \ --conf $FLUME_HOME/conf \ --conf-file $FLUME_HOME/conf/exec-memory-hdfs-partition.conf \ -Dflume.root.logger=INFO,console \ -Dflume.monitoring.type=http \ -Dflume.monitoring.port=1234
指標項整理
OS監控項
| objectName | 指標項 | 說明 |
|---|---|---|
| java.lang:type=OperatingSystem | FreePhysicalMemorySize | 空閒實體記憶體 |
| java.lang:type=OperatingSystem | SystemCpuLoad | 系統CPU利用率 |
| java.lang:type=OperatingSystem | ProcessCpuLoad | 程序CPU利用率 |
| java.lang:type=GarbageCollector,name=PS Scavenge | CollectionCount | GC次數 |
json資料格式
{ "SOURCE.src-1":{ "OpenConnectionCount":"0", "Type":"SOURCE", "AppendBatchAcceptedCount":"1355", "AppendBatchReceivedCount":"1355", "EventAcceptedCount":"28286", "AppendReceivedCount":"0", "StopTime":"0", "StartTime":"1442566410435", "EventReceivedCount":"28286", "AppendAcceptedCount":"0" }, "CHANNEL.ch-1":{ "EventPutSuccessCount":"28286", "ChannelFillPercentage":"0.0", "Type":"CHANNEL", "StopTime":"0", "EventPutAttemptCount":"28286", "ChannelSize":"0", "StartTime":"1442566410326", "EventTakeSuccessCount":"28286", "ChannelCapacity":"1000000", "EventTakeAttemptCount":"313734329512" }, "SINK.sink-1":{ "Type":"SINK", "ConnectionClosedCount":"0", "EventDrainSuccessCount":"28286", "KafkaEventSendTimer":"482493", "BatchCompleteCount":"0", "ConnectionFailedCount":"0", "EventDrainAttemptCount":"0", "ConnectionCreatedCount":"0", "BatchEmptyCount":"0", "StopTime":"0", "RollbackCount":"9", "StartTime":"1442566411897", "BatchUnderflowCount":"0" } }
指標項說明
source監控項
| objectName (會隨實際情況而變化) |
指標項 | 說明 |
|---|---|---|
| org.apache.flume.source:type=r1 | OpenConnectionCount | 目前與客戶端或sink保持連線的總數量 |
| org.apache.flume.source:type=r1 | AppendBatchAcceptedCount | 成功提交到channel的批次的總數量 |
| org.apache.flume.source:type=r1 | AppendBatchReceivedCount | 接收到事件批次的總數量 |
| org.apache.flume.source:type=r1 | AppendAcceptedCount | 逐條錄入的次數 |
| org.apache.flume.source:type=r1 | AppendReceivedCount | 每批只有一個事件的事件總數量 |
| org.apache.flume.source:type=r1 | EventAcceptedCount | 成功寫出到channel的事件總數量 |
| org.apache.flume.source:type=r1 | EventReceivedCount | 目前為止source已經接收到的事件總數量 |
| org.apache.flume.source:type=r1 | StartTime | source啟動時的毫秒值時間 |
| org.apache.flume.source:type=r1 | StopTime | source停止時的毫秒值時間，為0表示一直在執行 |
channel監控項
| objectName (會隨實際情況而變化) |
指標項 | 說明 |
|---|---|---|
| org.apache.flume.channel:type=c1 | EventPutAttemptCount | Source嘗試寫入Channe的事件總次數 |
| org.apache.flume.channel:type=c1 | EventPutSuccessCount | 成功寫入channel且提交的事件總次數 |
| org.apache.flume.channel:type=c1 | EventTakeAttemptCount | sink嘗試從channel拉取事件的總次數。 |
| org.apache.flume.channel:type=c1 | EventTakeSuccessCount | sink成功從channel讀取事件的總數量 |
| org.apache.flume.channel:type=c1 | ChannelSize | 目前channel中事件的總數量 |
| org.apache.flume.channel:type=c1 | ChannelCapacity | channel的容量 |
| org.apache.flume.channel:type=c1 | ChannelFillPercentage | channel已填入的百分比 |
| org.apache.flume.channel:type=c1 | StartTime | channel啟動時的毫秒值時間 |
| org.apache.flume.channel:type=c1 | StopTime | channel停止時的毫秒值時間，為0表示一直在執行 |
sink監控項
| objectName (會隨實際情況而變化) |
指標項 | 說明 |
|---|---|---|
| org.apache.flume.sink:type=k1 | ConnectionCreatedCount | 建立的連線數量 |
| org.apache.flume.sink:type=k1 | ConnectionClosedCount | 關閉的連線數量 |
| org.apache.flume.sink:type=k1 | ConnectionFailedCount | 由於錯誤關閉的連線數量 |
| org.apache.flume.sink:type=k1 | BatchEmptyCount | 批量處理event的個數為0的數量-表示source寫入資料的速度比sink處理資料的速度慢 |
| org.apache.flume.sink:type=k1 | BatchUnderflowCount | 批量處理event的個數小於批處理大小的數量 |
| org.apache.flume.sink:type=k1 | BatchCompleteCount | 批量處理event的個數等於批處理大小的數量 |
| org.apache.flume.sink:type=k1 | EventDrainAttemptCount | sink嘗試寫出到儲存的事件總數量 |
| org.apache.flume.sink:type=k1 | EventDrainSuccessCount | sink成功寫出到儲存的事件總數量 |
| org.apache.flume.sink:type=k1 | StartTime | channel啟動時的毫秒值時間 |
| org.apache.flume.sink:type=k1 | StopTime | channel停止時的毫秒值時間，為0表示一直在執行 |