logstash獲取nginx日誌 兩種方法
獲取nginx日誌要寫grok 還有很多正則來做
那麼很多像我一樣的新手不知道該如何操作
下面我們來個簡單的
第一種 :
重點是: 把nginx的access.log日誌格式改成json型別
更重要的是下面兩行
log_format json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"request":"$request",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"referer":"$http_referer",'
'"agent":"$http_user_agent",'
'"status":"$status"}';
access_log /data/nginx/logs/access_json.log json;
上面字型 顏色 一種顏色是一行
把這兩行加到nginx.conf的http裡面
如下程式碼:
http { include mime.types; default_type application/octet-stream; log_format json '{"@timestamp":"$time_iso8601",' '"host":"$server_addr",' '"clientip":"$remote_addr",' '"request":"$request",' '"size":$body_bytes_sent,' '"responsetime":$request_time,' '"upstreamtime":"$upstream_response_time",' '"upstreamhost":"$upstream_addr",' '"http_host":"$host",' '"url":"$uri",' '"referer":"$http_referer",' '"agent":"$http_user_agent",' '"status":"$status"}'; access_log /data/nginx/logs/access_json.log json; server_names_hash_bucket_size 128; client_header_buffer_size 32K; large_client_header_buffers 4 32k; --------------------------以下省略
重啟nginx 則在/data/nginx/logs/看到access_json.log的日誌檔案
下面我們寫logstash的配置
我們配置檔案是輸出到redis裡面,如果是直接寫到es裡面。需要改動
input { file { path => ['/data/nginx/logs/access_json.log'] start_position => "beginning" codec => "json" tags => ['user'] type => "nginx" } } output { if [type] == "nginx" { redis { host => "172.17.0.90" port => "6379" key => "nginx" db => "10" data_type => "list" } } }
上面的配置檔案就不做多解釋 其它文章裡面會介紹到
接下來就可以操作kibana了加索引了。會看到更多的列了
第二種 是後面發現的
https://grafana.com/dashboards/2292
參考grafana.com的
這個比第一種的全面
定義日誌型別
log_format main '{"@timestamp":"$time_iso8601",'
'"@source":"$server_addr",'
'"hostname":"$hostname",'
'"ip":"$http_x_forwarded_for",'
'"client":"$remote_addr",'
'"request_method":"$request_method",'
'"scheme":"$scheme",'
'"domain":"$server_name",'
'"referer":"$http_referer",'
'"request":"$request_uri",'
'"args":"$args",'
'"size":$body_bytes_sent,'
'"status": $status,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamaddr":"$upstream_addr",'
'"http_user_agent":"$http_user_agent",'
'"https":"$https"'
'}';
logstash的配置檔案
input {
file {
#這裡根據自己日誌命名使用正則匹配所有域名訪問日誌
#path => [ "/usr/local/nginx/logs/*_access.log" ]
path => ['/data/nginx/logs/access_json.log']
start_position => "beginning"
codec => "json"
tags => ['user']
type => "nginx"
}
}
filter {
mutate {
convert => [ "status","integer" ]
convert => [ "size","integer" ]
convert => [ "upstreatime","float" ]
remove_field => "message"
}
geoip {
source => "ip"
}
}
output {
if [type] == "nginx" {
redis {
host => "172.17.0.90"
port => "6379"
key => "nginx"
db => "10"
data_type => "list"
}
}
}