1.問題描述

自己採用flume監控log檔案，結果報錯

2018-11-26 19:32:04,687 (pool-5-thread-1) [ERROR - org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:353)] Failed while running command: tail -F /opt/datas/streamingproject/access.log
org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name: memory-channel}
        at org.apache.flume.channel.ChannelProcessor.executeChannelTransaction(ChannelProcessor.java:253)
        at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:191)
        at org.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:382)
        at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:342)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight
        at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:130)
        at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
        at org.apache.flume.channel.ChannelProcessor.executeChannelTransaction(ChannelProcessor.java:245)
        ... 8 more
2018-11-26 19:32:04,690 (timedFlushExecService19-0) [ERROR - org.apache.flume.source.ExecSource$ExecRunnable$1.run(ExecSource.java:328)] Exception occured when processing event batch
org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name: memory-channel}
        at org.apache.flume.channel.ChannelProcessor.executeChannelTransaction(ChannelProcessor.java:253)
        at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:191)
        at org.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:382)
        at org.apache.flume.source.ExecSource$ExecRunnable.access$100(ExecSource.java:255)
        at org.apache.flume.source.ExecSource$ExecRunnable$1.run(ExecSource.java:324)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flume.ChannelException: java.lang.InterruptedException
        at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:154)
        at org.apache.flume.channel.ChannelProcessor.executeChannelTransaction(ChannelProcessor.java:245)
        ... 11 more
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582)
        at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
        at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
        ... 12 more
2018-11-26 19:32:04,690 (pool-5-thread-1) [INFO - org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:376)] Command [tail -F /opt/datas/streamingproject/access.log] exited with 143
 

2.原因

參看Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the，是由於memory channel導致的。對於記憶體通道，當輸入過多，而無法全部輸出的時候，會導致記憶體通道崩潰，從而報錯。

3.解決

將記憶體通道換為file通道。生產中多用file通道，安全穩定。但是缺點是慢。