flume蒐集資料到hadoop HA可能出現的問題及解決辦法
把Hadoop叢集的hdfs-site.xml、core-site.xml兩個配置檔案複製到 flume安裝目錄的conf目錄去，把hadoop-hdfs-2.7.1.jar複製到 Flume lib目錄。
ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed
java.lang.IllegalArgumentException: java.net.UnknownHostException: ha1
解決方法
ha1公司Hadoop叢集NameService的名字，這個錯誤是由於找不到Hadoop叢集NameService造成的，所以需要把hdfs-site.xml複製到flume/conf目錄。
[email protected]:~# cp /etc/hadoop/conf/hdfs-site.xml /home//flume/conf/
java.io.IOException: Mkdirs failed to create /test/flume/16-09-19 (exists=false, cwd=file:/data/apache-flume-1.6.0-bin)
解決方法
[email protected]:~# cp/etc/hadoop/conf/core-site.xml /home/flume/conf/
java.io.IOException: No FileSystem for scheme: hdfs
解決方法把hadoop-hdfs-*.jar複製到flume/lib目錄下
[email protected]:~# cp /opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hadoop/client/hadoop-hdfs.jar /home/worker/flume/lib/
java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
原因是Event物件headers沒有設定timestamp造成的，解決辦法：設定a1.sinks.k1.hdfs.useLocalTimeStamp=true，使用本地時間戳。