一步一下搭建ELK叢集
ELK叢集搭建手冊
一、 環境準備：
三臺Linux伺服器，ip地址分別為：
192.168.25.30
192.168.25.31
192.168.25.32
角色劃分：
3臺機器全部安裝jdk1.8，因為elasticsearch是java開發的
3臺全部安裝elasticsearch (後續都簡稱為es)
192.168.25.30作為主節點
192.168.25.31以及192.168.25.32作為資料節點
主節點上需要安裝kibana
在192.168.77.130上安裝 logstash
ELK版本資訊：
Elasticsearch-6.4.2
logstash-6.4.2
kibana-6.4.2
filebeat-6.4.2
配置三臺機器的hosts檔案內容如下：
$ vim /etc/hosts
192.168.25.30 data-node-0
192.168.25.31 data-node-1
192.168.25.32 data-node-2
然後三臺機器都得關閉防火牆或清空防火牆規則。
二、 安裝java環境
| 安裝包版本：jdk-8u25-linux-x64.tar.gz #tar -zxvf jdk-8u25-linux-x64.tar.gz #cd jdk1.8.0_25/ #mkdir –p /app/jdk #cp -r ../jdk1.8.0_25 /app/jdk #vim /etc/profile 在最後插入如下幾行： export JAVA_HOME=/app/jdk/jdk1.8.0_25 export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH:$HOME/bin export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/jre/lib/tools.jar 檢查安裝情況： # source /etc/profile # java –version java version "1.8.0_25" Java(TM) SE Runtime Environment (build 1.8.0_25-b17) Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode) #javac
Usage: javac <options> <source files> where possible options include: -g Generate all debugging info -g:none Generate no debugging info -g:{lines,vars,source} Generate only some debugging info -nowarn Generate no warnings -verbose Output messages about what the compiler is doing -deprecation Output source locations where deprecated APIs are used -classpath <path> Specify where to find user class files and annotation processors -cp <path> Specify where to find user class files and annotation processors -sourcepath <path> Specify where to find input source files -bootclasspath <path> Override location of bootstrap class files -extdirs <dirs> Override location of installed extensions -endorseddirs <dirs> Override location of endorsed standards path -proc:{none,only} Control whether annotation processing and/or compilation is done. -processor <class1>[,<class2>,<class3>...] Names of the annotation processors to run; bypasses default discovery process -processorpath <path> Specify where to find annotation processors -parameters Generate metadata for reflection on method parameters -d <directory> Specify where to place generated class files -s <directory> Specify where to place generated source files -h <directory> Specify where to place generated native header files -implicit:{none,class} Specify whether or not to generate class files for implicitly referenced files -encoding <encoding> Specify character encoding used by source files -source <release> Provide source compatibility with specified release -target <release> Generate class files for specific VM version -profile <profile> Check that API used is available in the specified profile -version Version information -help Print a synopsis of standard options -Akey[=value] Options to pass to annotation processors -X Print a synopsis of nonstandard options -J<flag> Pass <flag> directly to the runtime system -Werror Terminate compilation if warnings occur @<filename> Read options and filenames from file |
安裝java成功
三、 安裝Elasticsearch（簡稱ES）
| 安裝ES： 下載安裝包elasticsearch-6.4.2.rpm https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.2.rpm
#wget –O /app/elasticsearch-6.4.2.rpm https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.2.rpm #cd /app #rpm -ivh elasticsearch-6.4.2.rpm
warning: elasticsearch-6.4.2.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY Preparing... ################################# [100%] Creating elasticsearch group... OK Creating elasticsearch user... OK Updating / installing... 1:elasticsearch-0:6.4.2-1 ################################# [100%] ### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd sudo systemctl daemon-reload sudo systemctl enable elasticsearch.service ### You can start elasticsearch service by executing sudo systemctl start elasticsearch.service Created elasticsearch keystore in /etc/elasticsearch
配置ES： elasticsearch配置檔案在/etc/elasticsearch/下和/etc/sysconfig/elasticsearch這個檔案，其中elasticsearch.yml 檔案用於配置叢集節點等相關資訊的，elasticsearch 檔案則是配置服務本身相關的配置，例如某個配置檔案的路徑以及java的一些路徑配置什麼的。 # cd /etc/elasticsearch/ # ll total 28 -rw-rw---- 1 root elasticsearch 207 Nov 5 11:48 elasticsearch.keystore -rw-rw---- 1 root elasticsearch 2869 Sep 26 21:39 elasticsearch.yml -rw-rw---- 1 root elasticsearch 3009 Sep 26 21:39 jvm.options -rw-rw---- 1 root elasticsearch 6380 Sep 26 21:39 log4j2.properties -rw-rw---- 1 root elasticsearch 473 Sep 26 21:39 role_mapping.yml -rw-rw---- 1 root elasticsearch 197 Sep 26 21:39 roles.yml -rw-rw---- 1 root elasticsearch 0 Sep 26 21:39 users -rw-rw---- 1 root elasticsearch 0 Sep 26 21:39 users_roles
# ll /etc/sysconfig/elasticsearch -rw-rw---- 1 root elasticsearch 1613 Sep 26 21:39 /etc/sysconfig/elasticsearch
在每個節點上建立資料data和logs目錄： #mkdir -p /app/elk/elasticsearch/data #mkdir -p /app/elk/elasticsearch/logs #chown -R elasticsearch /app/elk/elasticsearch/
開始配置叢集節點，在主節點 192.168.25.30 上編輯配置檔案： # vim /etc/elasticsearch/elasticsearch.yml 新增或修改以下內容（沒有的增加，存在的修改）： path.data: /app/elk/elasticsearch/data path.logs: /app/elk/elasticsearch/logs cluster.name: elk-test # 叢集中的名稱 node.name: data-node-0 # 該節點名稱 node.master: true # 意思是該節點是否可選舉為主節點 node.data: true # 表示這不是資料節點 network.host: 0.0.0.0 # 監聽全部ip，在實際環境中應為一個安全的ip http.port: 9200 # es服務的埠號 discovery.zen.ping.unicast.hosts: ["192.168.25.30", "192.168.25.31", "192.168.25.32"] # 配置自動發現
然後在從節點192.168.25.31、32上編輯配置檔案，新增或修改如下內容： path.data: /app/elk/elasticsearch/data path.logs: /app/elk/elasticsearch/logs cluster.name: elk-test # 叢集中的名稱 node.name: data-node-? # 該節點名稱，與前面配置hosts保持一致 node.master: true # 意思是該節點是否可選舉為主節點 node.data: true # 表示這不是資料節點 network.host: 0.0.0.0 # 監聽全部ip，在實際環境中應為一個安全的ip http.port: 9200 # es服務的埠號 discovery.zen.ping.unicast.hosts: ["192.168.25.30", "192.168.25.31", "192.168.25.32"] # 配置自動發現
修改 /etc/sysconfig/elasticsearch中的java路徑 # vim /etc/sysconfig/elasticsearch JAVA_HOME=/app/jdk/jdk1.8.0_25
完成以上的配置之後，到主節點上，啟動es服務, 主節點啟動完成之後，再啟動其他節點的es服務: # systemctl start elasticsearch.service # systemctl status elasticsearch.service ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled) Active: active (running) since Mon 2018-11-05 14:30:56 CST; 2s ago Docs: http://www.elastic.co Main PID: 522372 (java) CGroup: /system.slice/elasticsearch.service ├─522372 /app/jdk/jdk1.8.0_25/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -... └─522574 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
Nov 05 14:30:56 cnsz22pl1030 systemd[1]: Started Elasticsearch. Nov 05 14:30:56 cnsz22pl1030 systemd[1]: Starting Elasticsearch...
安裝成功 檢查安裝好的叢集健康狀態： # curl '192.168.25.30:9200/_cluster/health?pretty' { "cluster_name" : "master-node", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 2, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }
檢視叢集的詳細資訊： # curl '192.168.25.30:9200/_cluster/state?pretty' |
四、 安裝kibana
| Kibana只需要在主節點192.168.25.30上安裝即可，由於kibana是使用node.js開發的，所以程序名稱為node。 下載RPM安裝包：kibana-6.4.2-x86_64.rpm 下載地址：https://artifacts.elastic.co/downloads/kibana/kibana-6.4.2-x86_64.rpm 如果主機可以上外網，也可以執行以下命令： #wget –O /app/ kibana-6.4.2-x86_64.rpm https://artifacts.elastic.co/downloads/kibana/kibana-6.4.2-x86_64.rpm
# cd /app # rpm -ivh kibana-6.4.2-x86_64.rpm warning: kibana-6.4.2-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY Preparing... ################################# [100%] Updating / installing... 1:kibana-6.4.2-1 ################################# [100%]
配置kibana # vim /etc/kibana/kibana.yml 新增或修改如下項： server.port: 5601 # 配置kibana的埠 server.host: 192.168.25.30 # 配置監聽ip elasticsearch.url: "http://192.168.25.30:9200" # 配置es伺服器的ip，如果是叢集則配置該叢集中主節點的ip logging.dest: /var/log/kibana.log # 配置kibana的日誌檔案路徑，不然預設是messages裡記錄日誌
由於我們配置了日誌路徑，所以需要建立日誌檔案： # touch /var/log/kibana.log # chmod 777 /var/log/kibana.log
啟動kibana服務，並檢查程序和監聽埠： # systemctl start kibana # systemctl status kibana ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: disabled) Active: active (running) since Mon 2018-11-05 15:09:00 CST; 4s ago Main PID: 146989 (node) CGroup: /system.slice/kibana.service └─146989 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
Nov 05 15:09:00 cnsz22pl1030 systemd[1]: Started Kibana. Nov 05 15:09:00 cnsz22pl1030 systemd[1]: Starting Kibana...
# ps aux |grep kibana kibana 146989 47.0 0.0 1349520 269736 ? Ssl 15:09 0:29 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml root 150923 0.0 0.0 112644 952 pts/1 R+ 15:10 0:00 grep --color=auto kibana
#netstat -lntp |grep 5601 tcp 0 0 127.0.0.1:5601 0.0.0.0:* LISTEN 146989/node |
到此我們的kibana就安裝完成了，很簡單，接下來就是安裝logstash，不然kibana是沒法用的。
五、 安裝logstash
| 在192.168.25.31上安裝logstash，注意目前logstash不支援JDK1.9： 下載RPM安裝包logstash-6.4.2.rpm，下載地址如下： https://artifacts.elastic.co/downloads/logstash/logstash-6.4.2.rpm 如果主機支援外網，可直接執行以下命令下載： wget –O /app/ logstash-6.4.2.rpm https://artifacts.elastic.co/downloads/logstash/logstash-6.4.2.rpm
# rpm -ivh logstash-6.4.2.rpm warning: logstash-6.4.2.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY Preparing... ################################# [100%] Updating / installing... 1:logstash-1:6.4.2-1 ################################# [100%] Using provided startup.options file: /etc/logstash/startup.options Successfully created system startup script for Logstash
修改環境變數 # vim /etc/default/logstash 新增以下項： JAVA_HOME=/app/jdk/jdk1.8.0_25
修改日誌儲存路徑： #mkdir -p /app/elk/logstash/data #mkdir -p /app/elk/logstash/logs #chown -R logstash /app/elk/logstash/
修改配置檔案 # vim /etc/logstash/logstash.yml 將如下項的值修改為如下： path.data: /app/elk/logstash/data http.host: "192.168.25.31" path.logs: /app/elk/logstash/logs #
安裝完之後，先不要啟動服務，先配置logstash收集syslog日誌： #vim /etc/logstash/conf.d/syslog.conf 加入如下內容： input { # 定義日誌源 syslog { type => "system-syslog" # 定義型別 port => 10514 # 定義監聽埠 } }
elasticsearch { hosts => ["192.168.25.30:9200","192.168.25.31:9200","192.168.25.32:9200"] # 定義es伺服器的ip index => "system-syslog-%{+YYYY.MM.dd}" # 定義索引 } }
檢測配置檔案是否有錯： # cd /usr/share/logstash/bin # ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties [2018-11-05T16:20:07,997][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [2018-11-05T16:20:09,448][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash Configuration OK # 為ok則代表配置檔案沒有問題 命令說明：
配置logstash伺服器的ip以及配置的監聽埠： # vim /etc/rsyslog.conf #### RULES #### *.* @@192.168.25.31:10514
重啟rsyslog，讓配置生效： # systemctl restart rsyslog
啟動logstash並檢查服務狀態： # systemctl start logstash # systemctl status logstash
|
六、 安裝filebeats
| 在192.168.25.32上安裝filebeat。 下載RPM包filebeat-6.4.2-x86_64.rpm，下載地址： https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.2-x86_64.rpm 如果安裝的主機可以直接上外網，也可以使用如下命令下載： wget –O /app/filebeat-6.4.2-x86_64.rpm https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.2-x86_64.rpm
下載完成執行命令安裝 #rpm -ivh filebeat-6.4.2-x86_64.rpm warning: filebeat-6.4.2-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY Preparing... ################################# [100%] Updating / installing... 1:filebeat-6.4.2-1 ################################# [100%] 安裝完成後編輯配置檔案： # vim /etc/filebeat/filebeat.yml - type: log # Change to true to enable this input configuration. enabled: true #================== Kibana===================================== setup.kibana: host: "192.168.25.30:5601" #==================== Outputs ================================= # Configure what output to use when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: ["192.168.25.30:9200","192.168.25.31:9200","192.168.25.32:9200"] 以下配置可選，根據實際需要配置 #----------------------------- Logstash output -------------------------------- #output.logstash: # The Logstash hosts #hosts: ["192.168.25.31:5044"]
啟動服務： #systemctl start filebeat.service 檢視服務啟動狀態 #systemctl status filebeat.service
檢視elasticsearch # curl '192.168.25.30:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open system-syslogs-2018.11.06 9-WQSrX7Su2FeORk5XM5-w 5 1 614 0 924.1kb 406.5kb green open filebeat-6.4.2-2018.11.06 gYOcxCK8THaJ57AWAUbK3Q 3 1 8039 0 2.7mb 1.3mb
|