CentOS 6.5搭建ELK環境ElasticSearch+Kibana+Logstash
阿新 • • 發佈：2019-01-10
ELK工作流程
-
多個獨立的Agent(Shipper)負責收集不同來源的資料，一箇中心Agent(Indexer)負責彙總和分析資料，在中心Agent前的Broker(使用Redis實現)作為緩衝區，中心Agent後的ElasticSearch用於儲存和搜尋資料，前端的Kibana提供豐富的圖表展示。
-
Shipper表示日誌收集，使用LogStash收集各種來源的日誌資料，可以是系統日誌、檔案、Redis、mq等等；
-
Broker作為遠端Agent與中心Agent之間的緩衝區，使用Redis實現，一是可以提高系統的效能，二是可以提高系統的可靠性，當中心Agent提取資料失敗時，資料儲存在Redis中，而不至於丟失；
-
中心Agent(Indexer)也是LogStash，從Broker中提取資料，可以執行相關的分析和處理(Filter)；
-
ElasticSearch用於儲存最終的資料，並提供搜尋功能；
-
Kibana提供一個簡單、豐富的Web介面，資料來自於ElasticSearch，支援各種查詢、統計和展示
機器部署
| 系統 | IP | 配置 |
|---|---|---|
| CentOS 6.5 | 192.168.123.2 | Logstash |
| CentOS 6.5 | 192.168.123.3 | ES+Kibana |
Logstash
（Logstash部署在IP為192.168.123.2的機器上。）
資料流
input|decode|filter|encode|output
安裝和配置
1.安裝Java環境
[[email protected] ~]# yum install java-1.8.0-openjdk [[email protected] ~]# export JAVA_HOME=/usr/java [[email protected] bin]# which java /usr/bin/java
2.下載並安裝GPG key
[[email protected] ~]# rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch
3.yum源配置
[[email protected] ~]# cat > /etc/yum.repos.d/logstash.repo <<EOF [logstash-5.0] name=logstash repository for 5.0.x packages baseurl=http://packages.elasticsearch.org/logstash/5.0/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1 EOF [[email protected] ~]# yum clean all
3.安裝Logstash
[[email protected] ~]# yum install logstash
4.安裝目錄
[[email protected] ~]# whereis logstash logstash: /etc/logstash /usr/share/logstash /etc/logstash/conf.d #配置檔案目錄，預設是空的，需要自己寫 /usr/share/logstash/bin/logstash #執行檔案
5.編輯一個簡單的配置檔案
[[email protected] bin]# cd /etc/logstash/conf.d [[email protected] bin]# vim 1.conf #1.conf檔案內容 input{ stdin{} } output{ stdout{} } ################################ input外掛stdin（標準輸入） output外掛 stdout（標準輸出） /opt/logstash/bin/logstash 前臺啟動 提示：標準輸入和標準輸出的意思就是我們輸入什麼就會輸出什麼
將日誌儲存到ES中的配置：
input { sdin {} } filter{ } output{ elasticsearch { hosts => ["192.168.123.3:9200"] index => "logstash-%{+YYYY.MM.dd}" } stdout{ codec => rubydebug } }
6.執行測試
[[email protected] ~]# cd /usr/share/logstash/bin [[email protected] ~]# ./logstash -e 'input{stdin{}}output{stdout{} }' #-e引數表示執行，然後會發現終端在等待你的輸入。沒問題，敲入 hello,ichunqiu! ，回車，然後看看會返回什麼結果。 結果： OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N hello,ichunqiu! --- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.0 --- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.1-1 Sending logstash logs to /var/log/logstash/logstash.log. Pipeline main started 2017-12-16T13:23:56.586Z localhost.localdomain hello,ichunqiu! 2017-12-16T13:23:56.586Z localhost.localdomain hello,ichunqiu!
7.遇到的問題：
[[email protected] bin]# ./logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug} }' OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N --- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.0 --- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.1-1 Logstash has a new settings file which defines start up time settings. This file is typically located in $LS_HOME/config or /etc/logstash. If you installed Logstash through a package and are starting it manually please specify the location to this settings file by passing in "--path.settings=/path/.." in the command line options {:level=>:warn} Failed to load settings file from "path.settings". Aborting... {"path.settings"=>"/usr/share/logstash/config", "exception"=>Errno::ENOENT, "message"=>"No such file or directory - /usr/share/logstash/config/logstash.yml", :level=>:fatal}
解決方法： [[email protected] bin]# whereis logstash logstash: /etc/logstash /usr/share/logstash [[email protected] bin]# cp -r /etc/logstash/logstash.yml /usr/share/logstash/config/logstash.yml
ElasticSearch
資料流
安裝和配置
如果是在不同機器上安裝，則需要像Logstash的步驟1一樣配置好Java環境。
（本文在不同機器上部署，以下配置在IP為192.168.123.3的機器上進行。）
1.下載並安裝GPG key
[[email protected] ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
2.yum源配置
[[email protected] ~]# cat >/etc/yum.repos.d/elasticsearch.repo<<EOF [elasticsearch-2.x] name=Elasticsearch repository for 2.x packages baseurl=http://packages.elastic.co/elasticsearch/2.x/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 EOF [[email protected] ~]# yum clean all
3.安裝ElasticSearch
[[email protected] ~]# yum install elasticsearch
4.安裝目錄
[[email protected] ~]# whereis elasticsearch elasticsearch: /etc/elasticsearch /usr/share/elasticsearch /etc/elasticsearch/elasticsearch.yml #配置檔案目錄，預設是空的，需要自己寫 /usr/share/elasticsearch/bin/elasticsearch #執行檔案
5.修改 limits.conf
#在結尾新增如下兩行程式碼 [[email protected] ~]#vi /etc/security/limits.conf elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited #檢查 [[email protected] ~]# tail -2 /etc/security/limits.conf elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited
6.建立目錄並授權
[[email protected] ~]# mkdir -p /data/es-data [[email protected] ~]# chown -R elasticsearch.elasticsearch /data/es-data/
7.配置elasticsearch.yml
[[email protected] bin]# vim /etc/elasticsearch/elasticsearch.yml cluster.name: elk-cluster # 叢集的名稱 node.name: linux-node1 # 節點的名稱 path.data: /data/es-data # 資料儲存的目錄（多個目錄使用逗號分隔） path.logs: /var/log/elasticsearch # 日誌路徑 bootstrap.mlockall: true # 鎖住記憶體，使記憶體不會分配至交換區（swap） network.host:192.168.123.3 # 本機IP地址 http.port: 9200 # 埠預設9200
#檢視配置檔案 [[email protected] bin]# grep '^[a-z]' /etc/elasticsearch/elasticsearch.yml cluster.name: elk-cluster path.data: /data/es-data path.logs: /var/log/elasticsearch bootstrap.memory_lock: true network.host: 192.168.123.3 http.port: 9200
8.啟動ElasticSearch
[[email protected] elasticsearch]# /etc/init.d/elasticsearch start 正在啟動 elasticsearch：OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N [確定]
8.檢查啟動
[[email protected] init.d]# ps -ef|grep elasticsearch 498 4094 1 42 22:46 ? 00:00:20 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.4.6.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -p /var/run/elasticsearch/elasticsearch.pid -d -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.conf=/etc/elasticsearch root 4142 1524 0 22:47 pts/0 00:00:00 grep elasticsearch
9.訪問測試
#Linux下訪問： [[email protected] init.d]# curl 192.168.123.3:9200 { "name" : "node-1", "cluster_name" : "elk-cluster", "cluster_uuid" : "xJO564iaTpG7Z8WlaJ1x2Q", "version" : { "number" : "2.4.6", "build_hash" : "5376dca9f70f3abef96a77f4bb22720ace8240fd", "build_timestamp" : "2017-07-18T12:17:44Z", "build_snapshot" : false, "lucene_version" : "5.5.4" }, "tagline" : "You Know, for Search" }
windows下訪問:
10.遇到的問題;
[[email protected] elasticsearch]# /etc/init.d/elasticsearch start #執行該命令時失敗
解決方法： 在錯誤提示的地方加空格。
11.安裝Elasticsearch外掛
-
Head
-
外掛作用：主要是做ES叢集管理。
-
[[email protected] init.d]# /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head -> Installing mobz/elasticsearch-head... Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ... Downloading ......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ... NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify) Installed head into /usr/share/elasticsearch/plugins/head #安裝路徑 #訪問 在瀏覽器中訪問地址：http://ip地址:9200/_plugin/head/
Kibana
安裝和配置
1.下載並安裝GPG Key:
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
2.yum源配置
[[email protected] ~]# vim /etc/yum.repos.d/kibana.repo [kibana-4.5] name=Kibana repository for 4.5.x packages baseurl=http://packages.elastic.co/kibana/4.5/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 [[email protected] ~]# yum clean all
3.安裝Logstash
[[email protected] ~]# yum install kibana
4.安裝目錄
[[email protected] ~]# whereis kibana kibana: /opt/kibana/bin/kibana /opt/kibana/bin/kibana.bat
5.修改配置檔案
[[email protected] ~]# vim /opt/kibana/config/kibana.yml [[email protected] ~]# grep "^[a-Z]" /opt/kibana/config/kibana.yml server.port: 5601 #埠，預設5601 server.host: "0.0.0.0" #主機 elasticsearch.url: "http://192.168.123.3:9200" #es地址 kibana.index: ".kibana" #kibana是一個小系統，自己也需要儲存資料(將kibana的資料儲存到.kibana的索引中，會在ES裡面建立一個.kibana) # elasticsearch.username: "user" kibana中的es外掛是需要收費的，所以無法使用 # elasticsearch.password: "pass"
6.啟動Kibana
[[email protected] ~]# /etc/init.d/kibana start #ES也要開啟。
7.檢查埠
[[email protected] ~]# netstat -lntup Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1104/rpcbind tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 1321/vsftpd tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1310/sshd tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1180/cupsd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1397/master tcp 0 0 0.0.0.0:37212 0.0.0.0:* LISTEN 1122/rpc.statd tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 1671/node tcp 0 0 :::111 :::* LISTEN 1104/rpcbind tcp 0 0 :::22 :::* LISTEN 1310/sshd tcp 0 0 ::1:631 :::* LISTEN 1180/cupsd tcp 0 0 ::1:25 :::* LISTEN 1397/master tcp 0 0 :::49031 :::* LISTEN 1122/rpc.statd udp 0 0 0.0.0.0:111 0.0.0.0:* 1104/rpcbind udp 0 0 0.0.0.0:631 0.0.0.0:* 1180/cupsd udp 0 0 0.0.0.0:55684 0.0.0.0:* 1122/rpc.statd udp 0 0 0.0.0.0:855 0.0.0.0:* 1104/rpcbind udp 0 0 0.0.0.0:874 0.0.0.0:* 1122/rpc.statd udp 0 0 :::111 :::* 1104/rpcbind udp 0 0 :::48298 :::* 1122/rpc.statd udp 0 0 :::855 :::* 1104/rpcbind
8.訪問
訪問地址：http://192.168.123.3:5601
