elk通過kafka收集處理Tomcat日誌
阿新 • • 發佈：2019-01-12
環境說明
| server | 安裝軟體 |
| 10.16.0.15 | elasticsearch+logstash+kibana+kafka |
| 10.16.0.14 | tomcat+logstash+mysql |
實現原理
業務主機
部署jsp業務環境[jdk8、tomcat、mysql(mariadb)]
釋出一個簡單的程式測試
部署logstash收集tomcat日誌推送到kafka
分析主機
部署kafka訊息伺服器
部署ELK服務
logstash拉取kafka中json格式日誌到elasticserach
kibana展示elasticsearch收集到的資料
業務主機-安裝配置tomcat
1、jdk安裝
[[email protected] ~]# tar fx jdk-8u161-linux-x64.tar.gz -C /usr/local/jdk1.8 #配置Java環境變數 [[email protected] ~]# vim /etc/profile export JAVA_HOME=/usr/local/jdk1.8 export JRE_HOME=/usr/local/jdk1.8/jre export CLASSPATH=.:/lib:/lib: export PATH=/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin #使變數生效 [[email protected] ~]# sourcr /etc/profile #檢視Java版本 [[email protected] ~]# java -version java version "1.8.0_161" Java(TM) SE Runtime Environment (build 1.8.0_161-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
2、安裝Tomcat
安裝tomcat
[[email protected] ~]# wget http://apache.mirrors.hoobly.com/tomcat/tomcat-9/v9.0.14/bin/apache-tomcat-9.0.14.tar.gz [[email protected] ~]# tar fx apache-tomcat-9.0.14.tar.gz -C /opt [[email protected] ~]# cd /opt [[email protected] opt]# mv apache-tomcat-9.0.14 tomcat
啟動tomcat
[[email protected] opt]# cd tomcat/bin/ [[email protected] bin]# ./startup.sh
埠是否啟動
[[email protected] bin]# netstat -ntlp | grep 8080 tcp6 0 0 :::8080 :::* LISTEN 2469/java
頁面響應碼訪問測試
[[email protected] bin]# curl -I http://127.0.0.1:8080 HTTP/1.1 200 Content-Type: text/html;charset=UTF-8 Transfer-Encoding: chunked Date: Mon, 07 Jan 2019 07:42:58 GMT
檢視日誌檔案是否正確生成
[[email protected] bin]# ls ../logs/ catalina.2019-01-07.log host-manager.2019-01-07.log manager.2019-01-07.log catalina.out localhost.2019-01-07.log
3、配置Tomcat
編輯server配置檔案
[[email protected] bin]# cd ../conf/ [[email protected] conf]# vim server.xml
修改日誌配置
註釋掉原日誌格式
<!-- Access log processes all example. Documentation at: /docs/config/valve.html Note: The pattern used is equivalent to using pattern="common" <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log" suffix=".txt" pattern="%h %l %u %t "%r" %s %b" /> -->
新增json格式引數
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" ##指定日誌存放目錄
prefix="tomcat_access_log" suffix=".log" ##指定日誌檔名稱及字尾
pattern="{"clientip":"%h","ClientUser":"%l","authenticated":" %u","AccessTime":"%t","method":"%r","status":"%s","SendBytes":" %b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}" />
測試日誌格式是否正確
停止並刪除啟動的測試檔案
[[email protected] conf]# cd ../bin [[email protected] bin]# ./shutdown.sh [[email protected] bin]# rm -rf ../logs/*
按上邊的步驟啟動tomcat測試是否會有報錯
[[email protected] bin]# ./startup.sh
訪問tomcat主頁
[[email protected] bin]# curl http://127.0.0.1:8080
檢視是否生成tomcat訪問日誌
[[email protected] bin]# ls ../logs/ catalina.2019-01-07.log host-manager.2019-01-07.log manager.2019-01-07.log catalina.out localhost.2019-01-07.log tomcat_access_log.2019-01-07.log
檢視日誌格式
[[email protected] logs]# cat tomcat_access_log.2019-01-07.log {"clientip":"127.0.0.1","ClientUser":"-","authenticated":" -","AccessTime":"[07/Jan/2019:16:10:59 +0800]","method":"GET / HTTP/1.1","status":"200","SendBytes":" 11286","Query?string":"","partner":"-","AgentVersion":"curl/7.29.0"}
4、釋出一個圖書管理系統
上傳bookmanage壓縮包並解壓
連結：https://pan.baidu.com/s/1c-T-U0hA0bvd0Q5Im43SNw
提取碼：8wp7
[[email protected] ~]# ll bookmanage.zip -rw-r--r-- 1 root root 6065967 Jan 1 18:41 bookmanage.zip [[email protected] ~]# unzip bookmanage.zip [[email protected] ~]# mv bookmanage /opt/tomcat
將web資料拷貝到根目錄
[[email protected] tomcat]# cp -r bookmanage/WebRoot webapps/book
安裝資料庫
[[email protected] tomcat]# yum -y install mariadb-server [[email protected] tomcat]# systemctl start mariadb [[email protected] tomcat]# mysql -uroot -p123456 MariaDB [(none)]> create database bookmanage;
將資料匯入資料庫
[[email protected] tomcat]# mysql -uroot -p123456 bookmanage < bookmanage/bookManage.sql [[email protected] tomcat]# mysql -uroot -p123456 MariaDB [(none)]> use bookmanage MariaDB [bookmanage]> show tables; +----------------------+ | Tables_in_bookmanage | +----------------------+ | admin | | books | | borrow_book | | history | | return_book | | student | +----------------------+ 6 rows in set (0.00 sec)
配置資料庫登陸方式
[[email protected] classes]# cat jdbc.properties driver=com.mysql.jdbc.Driver url=jdbc:mysql://localhost:3306/bookmanage?useUnicode=true&characterEncoding=UTF-8 username=root password=123456 [[email protected] classes]# pwd /opt/tomcat/webapps/book/WEB-INF/classes
web測試頁面是否可以訪問
5、安裝配置logstash
logstash安裝參考《使用logstash收集並json化MySQL慢日誌》
配置logstash
[[email protected] ~]# cd /etc/logstash/conf.d/ [[email protected] conf.d]# cp ../logstash-sample.conf ./logstash_tomcat.conf
修改配置檔案
[[email protected] conf.d]# cat logstash_tomcat.conf # Sample Logstash configuration for creating a simple # Beats -> Logstash -> Elasticsearch pipeline. input { file { type => "tomcat_log" path => "/opt/tomcat/logs/tomcat_access_log.*.log" start_position => "beginning" codec => json } } output { kafka { bootstrap_servers => "10.16.0.15:9092" topic_id => "tomcat" compression_type => "snappy" codec => json } }
logstash啟動測試
[[email protected] conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash_tomcat.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2019-01-08 17:13:31.024 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [INFO ] 2019-01-08 17:13:32.065 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
提示OK即可繼續配置分析主機
分析主機配置
elasticsearch+logstsh+kibana安裝參考《使用logstash收集並json化MySQL慢日誌》
下載安裝kafka
[[email protected] ~]# wget http://mirrors.hust.edu.cn/apache/kafka/2.1.0/kafka_2.12-2.1.0.tgz [[email protected] ~]# tar fx kafka_2.12-2.1.0.tgz [[email protected] ~]# mv kafka-2.1.0-src /usr/local/kafka
配置kafka
[[email protected] ~]# cd /usr/local/kafka [[email protected] kafka]# vim config/server.properties
修改：
listeners=PLAINTEXT://10.16.0.15:9092 zookeeper.connect=10.16.0.15:2181
啟動kafka
[[email protected] bin]# jobs [1]- Running nohup ./zookeeper-server-start.sh ../config/zookeeper.properties > zookeeper.log 2>&1 & [2]+ Running nohup ./kafka-server-start.sh ../config/server.properties > kafka.log 2>&1 &
[[email protected] bin]# netstat -ntlp ... tcp6 0 0 10.16.0.15:9092 :::* LISTEN 24212/java tcp6 0 0 :::2181 :::* LISTEN 23572/java ...
kafka測試
#定義producer寫入資料到topic test [[email protected] bin]# ./kafka-console-producer.sh --broker-list 10.16.0.15:9092 --topic test >hello #消費訊息 [[email protected] bin]# ./kafka-console-consumer.sh --bootstrap-server 10.16.0.15:9092 --topic test --from-beginning hello
topic管理
#建立topic [[email protected] bin]# ./kafka-topics.sh --create --zookeeper 10.16.0.15:2181 --replication-factor 1 --partitions 1 --topic test Created topic "test".
#檢視topic [[email protected] bin]# ./kafka-topics.sh --list --zookeeper 10.16.0.15:2181 test
#檢視某個topic得詳細資訊 [[email protected] bin]# ./kafka-topics.sh --describe --zookeeper 10.16.0.15:2181 --topic test Topic:testPartitionCount:1ReplicationFactor:1Configs: Topic: testPartition: 0Leader: 0Replicas: 0Isr: 0
partition：partition id
leader：當前負責讀寫得lead broker id
replicas：當前partition得所有replication broker list
lsr：relicas得子集，只包含處於活動狀態得broker
#刪除topic [[email protected] bin]# ./kafka-topics.sh --delete --zookeeper 10.16.0.15:2181 --topic test
配置logstash
[[email protected] ~]# cat /etc/logstash/conf.d/logstash_tomcat.conf # Sample Logstash configuration for creating a simple # Beats -> Logstash -> Elasticsearch pipeline. input { #beats { # port => 5044 #} kafka { bootstrap_servers => "10.16.0.15:9092" topics => "tomcat" codec => "json" consumer_threads => 5 decorate_events => true } } output { elasticsearch { hosts => ["http://10.16.0.15:9200"] index => "tomcat-%{+YYYY.MM.dd}" #user => "elastic" #password => "changeme" } #stdout { # codec => rubydebug #} }
螢幕輸出測試
{
"host" => "mode-01-0004.novalocal",
"method" => "GET /book/lib/ligerUI/skins/Aqua/images/win/taskbar-task.gif HTTP/1.1",
"Query?string" => "",
"status" => "200",
"AccessTime" => "[12/Jan/2019:23:02:23 +0800]",
"path" => "/opt/tomcat/logs/tomcat_access_log.2019-01-12.log",
"ClientUser" => "-",
"type" => "tomcat_log",
"authenticated" => " -",
"clientip" => "10.16.0.10",
"SendBytes" => " 277",
"@timestamp" => 2019-01-12T15:02:25.587Z,
"@version" => "1",
"partner" => "http://10.16.0.14:8080/book/lib/ligerUI/skins/Aqua/css/ligerui-dialog.css",
"AgentVersion" => "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0"
}
啟動elasticsearch+kibana配置相應的索引及圖表