1. 程式人生 > >CentOS7搭建ELK-6.2.4版本

CentOS7搭建ELK-6.2.4版本

java centos7 elk

CentOS7搭建ELK-6.2.4版本


ELK是ElasticSerach、Logstash、Kibana三款產品名稱的首字母集合,用於日誌的搜集和搜索,今天我們一起搭建和體驗基於ELK的日誌服務;

環境規劃

本次實戰需要兩臺電腦(或者vmware下的兩個虛擬機),操作系統都是CentOS7,它們的身份、配置、地址等信息如下:

hostnameIP地址身份配置
elk-server192.168.119.132ELK服務端,接收日誌,提供日誌搜索服務雙核,4G內存
nginx-server192.168.119.133Nginx服務端,產生的訪問日誌通過上報到Logstash雙核,2G內存

部署情況簡介

運行時的部署情況如下圖所示:

技術分享圖片

  1. 業務請求到達nginx-server機器上的Nginx;

  2. Nginx響應請求,並在access.log文件中增加訪問記錄;

  3. FileBeat搜集新增的日誌,通過LogStash的5044端口上傳日誌;

  4. LogStash將日誌信息通過本機的9200端口傳入到ElasticSerach;

  5. 搜索日誌的用戶通過瀏覽器訪問Kibana,服務器端口是5601;

  6. Kibana通過9200端口訪問ElasticSerach;

關於Nginx安裝

請在nginx-server機器上安裝nginx1.14.1或者更高版本,安裝的詳細步驟請參考《 CentOS7安裝Nginx1.10.1》;

接下來安裝ELK吧;

安裝JDK

首先請在elk-server機器上JDK8;

在ELK官方文檔中(https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/requirements.html),推薦的JDK版本為8,如下圖所示:
技術分享圖片

在CentOS7安裝JDK8的步驟請參考《CentOS7安裝JDK8》;

創建用戶

ElasticSerach要求以非root身份啟動,所以我們要創建一個用戶:
1. 創建用戶組:groupadd elasticsearch
2. 創建用戶加入用戶組:useradd elasticsearch -g elasticsearch
3. 設置ElasticSerach文件夾為用戶elasticsearch所有:chown -R elasticsearch.elasticsearch /usr/local/work/elasticsearch-6.2.4

系統設置

  1. 設置hostname,打開文件/etc/hostname,將內容改為elk-server

  2. 關閉防火墻(如果因為其他原因不能關閉防火墻,也請不要禁止80端口):systemctl stop firewalld.service

  3. 禁止防火墻自動啟動:systemctl disable firewalld.service

  4. 打開文件/etc/security/limits.conf,添加下面四行內容:

* soft nofile 65536* hard nofile 131072* soft nproc 2048* hard nproc 40961234


5. 打開文件/etc/sysctl.conf,添加下面一行內容:

vm.max_map_count=6553601


6. 加載sysctl配置,執行命令:sysctl -p
7. 重啟電腦;

elk-server:安裝文件準備

請在ELK官網https://www.elastic.co/downloads下載以下文件:
1. elasticsearch-6.2.4.tar.gz;
2. logstash-6.2.4.tar.gz;
3. kibana-6.2.4-linux-x86_64.tar.gz;

上述三個文件,推薦在CentOS7的命令行輸入以下四個命令下載:

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gzwget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.tar.gzwget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-linux-x86_64.tar.gz123

下載完畢後,創建目錄/usr/local/work,將剛剛下載的三個文件全部在這個目錄下解壓,得到以下三個文件夾:
1. /usr/local/work/elasticsearch-6.2.4
2. /usr/local/work/logstash-6.2.4
3. kibana-6.2.4-linux-x86_64

啟動ElasticSerach

  1. 切換到用戶elasticsearch:su elasticsearch

  2. 進入目錄/usr/local/work/elasticsearch-6.2.4

  3. 執行啟動命令:bin/elasticsearch -d,此時會在後臺啟動elasticsearch;

  4. 查看啟動日誌可執行命令:tail -f /usr/local/work/elasticsearch-6.2.4/logs/elasticsearch.log,大約五到十分鐘後啟動成功,提示如下:

[2018-04-07T10:12:27,392][INFO ][o.e.n.Node               ] initialized
[2018-04-07T10:12:27,392][INFO ][o.e.n.Node               ] [MNb1nGq] starting ...
[2018-04-07T10:12:39,676][INFO ][o.e.t.TransportService   ] [MNb1nGq] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-04-07T10:12:42,772][INFO ][o.e.c.s.MasterService    ] [MNb1nGq] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {MNb1nGq}{MNb1nGq6Tn6VskdKFQckow}{_DglQhgmRsGAF2D7eTfVfg}{127.0.0.1}{127.0.0.1:9300}
[2018-04-07T10:12:42,776][INFO ][o.e.c.s.ClusterApplierService] [MNb1nGq] new_master {MNb1nGq}{MNb1nGq6Tn6VskdKFQckow}{_DglQhgmRsGAF2D7eTfVfg}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {MNb1nGq}{MNb1nGq6Tn6VskdKFQckow}{_DglQhgmRsGAF2D7eTfVfg}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-04-07T10:12:42,817][INFO ][o.e.g.GatewayService     ] [MNb1nGq] recovered [0] indices into cluster_state
[2018-04-07T10:12:42,821][INFO ][o.e.h.n.Netty4HttpServerTransport] [MNb1nGq] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-04-07T10:12:42,821][INFO ][o.e.n.Node               ] [MNb1nGq] starte12345678


5. 執行curl命令檢查服務是否正常響應:curl 127.0.0.1:9200,收到響應如下:

[elasticsearch@elk-server work]$ curl 127.0.0.1:9200{  "name" : "MNb1nGq",  "cluster_name" : "elasticsearch",  "cluster_uuid" : "ZHkI7PCQTnCqMBM6rhyT5g",  "version" : {    "number" : "6.2.3",    "build_hash" : "c59ff00",    "build_date" : "2018-03-13T10:06:29.741383Z",    "build_snapshot" : false,    "lucene_version" : "7.2.1",    "minimum_wire_compatibility_version" : "5.6.0",    "minimum_index_compatibility_version" : "5.0.0"
  },  "tagline" : "You Know, for Search"}12345678910111213141516

至此,ElasticSerach服務啟動成功,接下來是Logstash;

配置和啟動Logstash

  1. 在目錄/usr/local/work/logstash-6.2.4下創建文件default.conf,內容如下:

# 監聽5044端口作為輸入input {
    beats {
        port => "5044"
    }
}# 數據過濾filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    geoip {
        source => "clientip"
    }
}# 輸出配置為本機的9200端口,這是ElasticSerach服務的監聽端口output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
    }
}123456789101112131415161718192021


2. 後臺啟動Logstash服務:nohup bin/logstash -f default.conf --config.reload.automatic &
3. 查看啟動日誌:tail -f logs/logstash-plain.log,啟動成功的信息如下:

[2018-04-07T10:56:28,143][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.3"}
[2018-04-07T10:56:28,870][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-04-07T10:56:33,639][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-04-07T10:56:34,628][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}
[2018-04-07T10:56:34,650][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2018-04-07T10:56:35,147][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"}
[2018-04-07T10:56:35,245][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-04-07T10:56:35,248][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-04-07T10:56:35,304][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-04-07T10:56:35,333][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-04-07T10:56:35,415][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1:9200"]}
[2018-04-07T10:56:35,786][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/local/work/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2018-04-07T10:56:36,727][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-04-07T10:56:36,902][INFO ][logstash.pipeline        ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x427aed17 run>"}
[2018-04-07T10:56:36,967][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-04-07T10:56:37,083][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}12345678910111213141516

Kibana

  1. 打開Kibana的配置文件/usr/local/work/kibana-6.2.4-linux-x86_64/config/kibana.yml,找到下面這行:

#server.host: "localhost"1

改成如下內容:

server.host: "192.168.119.132"1

這樣其他電腦就能用瀏覽器訪問Kibana的服務了;
2. 進入Kibana的目錄:/usr/local/work/kibana-6.2.4-linux-x86_64
3. 執行啟動命令:nohup bin/kibana &
4. 查看啟動日誌:tail -f nohup.out
5. 以下信息表示啟動成功:

{"type":"log","@timestamp":"2018-04-07T04:44:59Z","tags":["status","plugin:[email protected]","info"],"pid":3206,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-04-07T04:44:59Z","tags":["status","plugin:[email protected]","info"],"pid":3206,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-04-07T04:45:01Z","tags":["status","plugin:[email protected]","info"],"pid":3206,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-04-07T04:45:01Z","tags":["status","plugin:[email protected]","info"],"pid":3206,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-04-07T04:45:01Z","tags":["listening","info"],"pid":3206,"message":"Server running at http://localhost:5601"}
{"type":"log","@timestamp":"2018-04-07T04:45:01Z","tags":["status","plugin:[email protected]","info"],"pid":3206,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}123456


6. 在瀏覽器訪問http://192.168.119.132:5601,看到如下頁面:
技術分享圖片

至此,ELK服務啟動成功,接下來我們將業務日誌上報上來,需要操作另一臺電腦:nginx-server;

防火墻

首先,請關閉nginx-server的防火墻:

systemctl stop firewalld.service && systemctl disable firewalld.service1

安裝Nginx

在nginx-server上安裝並啟動nginx服務,可以參考《 CentOS7安裝Nginx1.10.1》;

FileBeat

  1. 在nginx-server電腦創建目錄/usr/local/work

  2. 在/usr/local/work目錄下執行以下命令,下載FileBeat安裝包:

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-linux-x86_64.tar.gz**重點內容**1


3. 解壓:tar -zxvf filebeat-6.2.4-linux-x86_64.tar.gz
4. 打開文件/usr/local/work/filebeat-6.2.4-linux-x86_64/filebeat.yml,找到如下圖的位置:
技術分享圖片
5. 首先,將上圖綠框中的enabled: false改為enabled: true
6. 其次,將上圖紅框中的- /var/log/*.log改為- /usr/local/nginx/logs/*.log
7. 繼續修改filebeat.yml文件,找到下圖兩個紅框中的內容,在每行的左側添加”#”,將這兩行內容註釋掉:
技術分享圖片

8. 繼續修改filebeat.yml文件,找到下圖中的內容:
技術分享圖片
首先,將上圖紅框中的”#”去掉;
其次,將上圖綠框那一行的左側”#”去掉;
最後,將上圖綠框中的內容從[“localhost:5044”]改為[“192.168.119.132:5044”](連接ElasticSerach);
改好的內容如下圖;
技術分享圖片

9. 啟動FileBeat:./filebeat -e -c filebeat.yml -d “publish”

至此,FileBeat也啟動成功了,接下來驗證服務;

創建Index Patterns

  1. 通過瀏覽器多訪問幾次nginx服務,這樣能多制造一些訪問日誌,訪問地址:http://192.168.119.133

  2. 訪問Kibana:http://192.168.119.132:5601,點擊左上角的Discover,如下圖紅框,可以看到訪問日誌已經被ELK搜集了:
    技術分享圖片

  3. 如下圖,輸入logstash-*,點擊”Next step”:
    技術分享圖片

  4. 如下圖,選擇Time Filter,再點擊“Create index pattern”:
    技術分享圖片

  5. 頁面提示創建Index Patterns成功:
    技術分享圖片

  6. 點擊左上角的”Discover”按鈕,即可看到最新的日誌信息,如下圖:
    技術分享圖片

至此,我們已經可以在ELK上查到Nginx的訪問日誌了,接下來將Tomcat的日誌也接進來;

安裝和啟動Tomcat

  1. 確保nginx-server電腦上已經安裝了JDK8;

  2. /usr/local/work/目錄下執行以下命令,下載Tomcat:

wget http://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-7/v7.0.85/bin/apache-tomcat-7.0.85.zip1


3. 解壓縮:unzip apache-tomcat-7.0.85.zip
4. 給腳本賦予可執行權限:chmod a+x /usr/local/work/apache-tomcat-7.0.85/bin/*.sh
5. 啟動:/usr/local/work/apache-tomcat-7.0.85/bin/startup.sh
6. 瀏覽器訪問:http://192.168.119.133:8080,看到啟動成功,如下圖:
技術分享圖片

7. 訪問Tomcat提供的example服務的子頁面:http://192.168.119.133:8080/examples/servlets/servlet/RequestInfoExample,如下圖:
技術分享圖片

至此,Tomcat已經啟動成功,接下來將Tomcat的訪問日誌接入ELK;

Tomcat訪問日誌接入ELK

  1. 打開FileBeat的配置文件/usr/local/work/filebeat-6.2.4-linux-x86_64/filebeat.yml,在”filebeat.prospectors:”下面新增一個配置節點,內容如下:

- type: log
  enabled: true
  paths:    - /usr/local/work/apache-tomcat-7.0.85/logs/localhost_access_log.*.txt1234

配置好的filebeat.yml有兩個type節點了,如下圖:
技術分享圖片

2. 停掉filebeat服務,ps -ef |grep filebeat, kill -9 pid

再用./filebeat -e -c filebeat.yml -d “publish”命令啟動filebeat服務;
3. 此時在Kibana頁面已經可以搜索到Tomcat的訪問日誌,以“RequestInfoExample”作為關鍵詞搜索也能搜到對應的訪問日誌:
技術分享圖片

至此,ELK-6.2.4版本的服務和日誌上報的搭建已經完成,後續如果還有業務服務器要上報日誌,只需按照上述步驟安裝和配置FileBeat即可;

版權聲明:歡迎轉載,請註明出處,謝謝。 https://blog.csdn.net/boling_cavalry/article/details/79836171


CentOS7搭建ELK-6.2.4版本