1. 程式人生 > >Filebeat 原理詳解/配置檔案分析

Filebeat 原理詳解/配置檔案分析

  1. 配置檔案位置

對於rpmdeb,您將在以下位置找到配置檔案/etc/filebeat/filebeat.yml。在Docker下,它位於/usr/share/filebeat/filebeat.yml。對於macwin以及zip文件,請檢視剛剛提取的存檔。相同路徑下還有一個名為的完整示例配置檔案filebeat.reference.yml,顯示了所有未棄用的配置選項。

  1. 讀取日誌配置:
  • 專案日誌檔案

    利用 Filebeat 去讀取檔案,paths 下面配置路徑地址。 /data/share/business_log/TA-*/debug.log Filebeat 會自動去讀取business_log裡面的TA開頭的檔案。可以使用 Linux 的

    萬用字元 對檔名進行匹配,找到需要的檔名

    #=========================== Filebeat prospectors =============================
     
    filebeat.prospectors:
     
    # Each - is a prospector. Most options can be set at the prospector level, so
    # you can use different prospectors for various configurations.
    # Below are the prospector specific configurations.
    # 設定輸入的type為log - type: log # Change to true to enable this prospector configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: #- /usr/local/server/openresty/nginx/logs/*.log - /data/share/business_log/TA-*/debug.log #- c:\programdata\elasticsearch\logs\*

    filebeat 對於多行日誌的處理,需要處理多行日誌的情況下

    # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
    multiline:
        pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
        negate: true
        match: after
    

    上面配置的意思是:不以時間格式開頭的行都合併到上一行的末尾(正則寫的不好,忽略忽略) pattern:正則表示式 negate:true 或 false;預設是false,匹配pattern的行合併到上一行;true,不匹配pattern的行合併到上一行 match:after 或 before,合併到上一行的末尾或開頭 還有更多兩個配置,預設也是註釋的,沒特殊要求可以不管它 max_lines: 500 timeout: 5s max_lines:合併最大行,預設500 timeout:一次合併事件的超時時間,預設5s,防止合併消耗太多時間甚至卡死

  • nginx日誌檔案

    #=========================== Filebeat prospectors =============================
     
    filebeat.prospectors:
     
    # Each - is a prospector. Most options can be set at the prospector level, so
    # you can use different prospectors for various configurations.
    # Below are the prospector specific configurations.
     
     - type: log
     
      # Change to true to enable this prospector configuration.
      enabled: true
     
      # Paths that should be crawled and fetched. Glob based paths.
      paths:
        - /usr/local/server/openresty/nginx/logs/access.log
        - /usr/local/server/openresty/nginx/logs/error.log
        #- /data/share/business_log/TA-*/debug.log
        #- c:\programdata\elasticsearch\logs\*
    
  • 輸出配置

    我們需要輸出到 Logstash 裡面,註釋掉 Elasticsearch 下面的配置項,並配置 Logstash 下面的配置,會將 Filebeat 讀取到的日誌檔案傳送到 hosts 裡面配置的 Logstash 伺服器上面去

    #----------------------------- Logstash output --------------------------------
    output.logstash:
      # The Logstash hosts
      # Logstash 不會組成叢集,但是 Filebeat 會自己去輪詢 Logstash 的伺服器,去找到可用的 Logstash 伺服器傳送過去
      hosts: ["172.18.1.152:5044","172.18.1.153:5044","172.18.1.154:5044"]
      index: "logstash-%{+yyyy.MM.dd}"
     
      # Optional SSL. By default is off.
      # List of root certificates for HTTPS server verifications
      #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
     
      # Certificate for SSL client authentication
      #ssl.certificate: "/etc/pki/client/cert.pem"
     
      # Client Certificate Key
      #ssl.key: "/etc/pki/client/cert.key"
    

Filebeat 啟動命令:nohup ./filebeat -e -c filebeat-TA.yml >/dev/null 2>&1 & Filebeat 可以啟動多個,通過不同的 *-Filebeat.yml 配置檔案啟動