一般模式 FileBeat(更輕量化) --> Logstash

1、Logstash是什麼？

Logstash是一個日誌收集、過濾、轉發的中介軟體，主要負責將各條業務線的各類日誌統一收集、過濾後，轉發給 Elasticsearch 進行下一步處理

2、安裝Logstash

logstash無需安裝，直接解壓就行。收集日誌需要寫符合業務需求的配置檔案。下面從最基本的控制檯輸入輸出開始，一步步配置符合內心期望的收集、過濾等配置。

1）簡單安裝

# cd /usr/local/src

# tar -zxvf logstash-5.2.2.tar.gz

# mv logstash-5.2.2 /usr/local/logstash

 

2）為logstash安裝x-pack

# cd /usr/local/logstash/bin

# ./logstash-plugin install x-pack

 

[[email protected] bin]# ./logstash-plugin install x-pack

Downloading file: https://artifacts.elastic.co/downloads/logstash-plugins/x-pack/x-pack-5.2.2.zip

Downloading [=============================================================] 100%

Installing file: /tmp/studtmp-e97bf0800211a6de985aa19c225dd2a5dce0d369da9f5eb975df52c0adee/x-pack-5.2.2.zip

Install successful

[[email protected] bin]#

3）建立配置檔案，指定資料流向，在目錄下建立一個simple.conf，內容如下

input { beats { port => "5044" } } # 資料過濾 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } geoip { source => "clientip" } } # 輸出到本機的 ES output { elasticsearch { hosts => [ "127.0.0.1:9200" ] } }

 

3）上述配置輸出日誌到ES，啟動命令如下：

# cd /usr/local/logstash/bin

# ./logstash -f /usr/local/logstash/config/simple.conf --config.reload.automatic

 

啟動時報錯：LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://127.0.0.1:9200/'"}

原因：之前給ES 安裝了x-pack外掛，訪問需要登入賬號和密碼

解決：調整logstash配置檔案，在output中 訪問es時增加 user、password

 

# cd /usr/local/logstash/config

# vi + simple.conf output { elasticsearch { hosts => [ "127.0.0.1:9200" ]

user => elastic

password=>changeme } }

# cd /usr/local/logstash/bin

# ./logstash -f /usr/local/logstash/config/simple.conf --config.reload.automatic 重新啟動

...

[2018-08-03T19:08:53,852][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/local/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.0.4-java/vendor/GeoLite2-City.mmdb"}

[2018-08-03T19:08:53,879][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}

[2018-08-03T19:08:54,307][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}

[2018-08-03T19:08:54,343][INFO ][logstash.pipeline ] Pipeline main started

[2018-08-03T19:08:54,408][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

 

參考地址https://blog.csdn.net/Ahri_J/article/details/79609444

 

3、安裝FileBeats

下載地址： https://www.elastic.co/downloads/beats/filebeat 

安裝檔案：filebeat-5.2.2-linux-x86_64.tar.gz

1）簡單安裝FileBeats

# cd /usr/local/src

# tar -zxvf filebeat-5.2.2-linux-x86_64.tar.gz

# mv filebeat-5.2.2-linux-x86_64 /usr/local/filebeat

 

2) 配置 filebeats.yml ，指定讀取日誌目錄，指定輸出到本機logstash

- input_type: log

# Paths that should be crawled and fetched. Glob based paths.

paths:

- /home/daxiang/logs/mobile/*.log

 

#----------------------------- Logstash output --------------------------------

output.logstash:

# The Logstash hosts

hosts: ["localhost:5044"]

 

3）啟動

測試啟動命令

# ./filebeat -e -c filebeat.yml -d "publish"

正常後臺執行啟動，日誌開始採集

# nohup ./filebeat -e -c filebeat.yml >/dev/null 2>&1 &

 

4、Logstash基本學習

1)執行最基本的logstash管道

# cd /usr/local/logstash/bin

# ./logstash -e 'input { stdin {} } output { stdout {}}'

執行效果如下

[[email protected] bin]# ./logstash -e 'input { stdin {} } output { stdout {}}'

Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties

[2018-07-25T16:18:00,220][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/local/logstash/data/queue"}

[2018-07-25T16:18:00,238][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"1850a7c6-e6b8-43ae-9079-6e8d27be4173", :path=>"/usr/local/logstash/data/uuid"}

[2018-07-25T16:18:00,410][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}

[2018-07-25T16:18:00,433][INFO ][logstash.pipeline ] Pipeline main started

The stdin plugin is now waiting for input:

[2018-07-25T16:18:00,506][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

 

hello (在控制檯輸入hello，可看到如下輸出)

2018-07-25T08:18:34.363Z test hello

hello world

2018-07-25T08:19:15.051Z test hello world (當前輸出格式 時間戳 主機名 輸入內容)

 

2)配置輸出時，指定使用codec

建立一個logstash-simple.conf

# cd /usr/local/

# touch logstash-simple.conf

# vi logstash-simple.conf

#輸入來源

input { stdin {} }

#指定輸出格式

output {

stdout{

codec=>rubydebug

}

}

 

# cd /usr/local/logstash/bin

# ./logstash -f /usr/local/logstash/config/logstash-simple.conf

 

執行結果如下

[[email protected] bin]# ./logstash -f /usr/local/logstash/config/logstash-simple.conf

Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties

[2018-07-25T16:53:33,755][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}

[2018-07-25T16:53:33,794][INFO ][logstash.pipeline ] Pipeline main started

The stdin plugin is now waiting for input:

[2018-07-25T16:53:33,849][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

 

hello 輸入hello，返回了JSON

{

"@timestamp" => 2018-07-25T08:53:45.928Z,

"@version" => "1",

"host" => "test",

"message" => "hello"

}

 

5、Logstash配置指定日誌的切割正則，需用到ruby做正則，待後續完善。