prometheus 監控 之 nginx 應用
用Prometheus進行nginx的監控可以自動的對相關server_name和upstream進行監控,你也可以自定義Prometheus的資料標籤,實現對不同機房和不同專案的nginx進行監控。
監控Nginx主要用到以下三個模組:
- nginx-module-vts:Nginx virtual host traffic status module,Nginx的監控模組,能夠提供JSON格式的資料產出。
- nginx-vts-exporter:Simple server that scrapes Nginx vts stats and exports them via HTTP for Prometheus consumption。主要用於收集Nginx的監控資料,並給Prometheus提供監控介面,預設埠號9913。
- Prometheus:監控Nginx-vts-exporter提供的Nginx資料,並存儲在時序資料庫中,可以使用PromQL對時序資料進行查詢和聚合。
1.nginx-module-vts模組的編譯
首先檢視nginx安裝了那些模組:
[[email protected] conf]#./sbin/nginx -V
下載nginx-module-vts模組,並新增模組到nginx:
[[email protected] conf]# cd /usr/local/src
[[email protected] conf]# git clone git://github.com/vozlt/nginx-module-vts.git
[ [email protected] conf]# ./configure --prefix=/usr/local/nginx --with-http_ssl_module --add-module=//usr/local/src/nginx-module-vts
[[email protected] conf]# make ##編譯 不要make install, 不然會覆蓋
替換nginx二進位制檔案:
[[email protected] conf]# mv /usr/local/nginx/sbin/nginx /usr/local/nginx/sbin/nginx.bak
[[email protected] conf]# cp ./objs/nginx /usr/local/nginx/sbin/
[[email protected] conf]# pkill -9 nginx ## 關閉nginx
[[email protected]x conf]# /usr/local/nginx/sbin/nginx ## 啟動nginx
修改nginx.conf配置,實驗安裝是否成功:
http {
...
vhost_traffic_status_zone;
vhost_traffic_status_filter_by_host on;
...
server {
...
location /status {
vhost_traffic_status_display;
vhost_traffic_status_display_format html;
}
}
配置解析: 1.1 開啟vhost過濾: vhost_traffic_status_filter_by_host on; 開啟此功能,在Nginx配置有多個server_name的情況下,會根據不同的server_name進行流量的統計,否則預設會把流量全部計算到第一個server_name上。
1.2.在不想統計流量的server區域禁用vhost_traffic_status,配置示例:
server {
...
vhost_traffic_status off;
...
}
假如nginx沒有規範配置server_name或者無需進行監控的server上,那麼建議在此vhost上禁用統計監控功能。否則會出現“127.0.0.1”,hostname等的域名監控資訊。
開啟瀏覽器訪問ip/status:
2.安裝nginx-vts-exporter
[[email protected] conf]# wget -O nginx-vts-exporter-0.5.zip https://github.com/hnlq715/nginx-vts-exporter/archive/v0.5.zip
[[email protected] conf]# unzip nginx-vts-exporter-0.5.zip
[[email protected] conf]# mv nginx-vts-exporter-0.5 /usr/local/prometheus/nginx-vts-exporter
[[email protected] conf]# chmod +x /usr/local/prometheus/nginx-vts-exporter/bin/nginx-vts-exporter
使用supervisor程序管理工具啟動nginx-vts-exporter(安裝supervisor請參考:CentOS7 supervisor安裝、配置、實戰):
[[email protected] conf]# yum install epel-release -y
[[email protected] conf]# yum install supervisor -y
[[email protected] conf]# tail -n 25 /etc/supervisord.conf
……
[include]
files = supervisord.d/*.ini
[program:nginx_exporter]
command=/usr/local/prometheus/nginx-vts-exporter/bin/nginx-vts-exporter -nginx.scrape_uri=http://10.100.xx.xxx/status/format/json
stdout_logfile=/tmp/prometheus/nginx-vts-exporter.log
autostart=true
autorestart=true
startsecs=5
priority=1
user=root
stopasgroup=true
killasgroup=true
[program:node_exporter]
command=/usr/local/bin/node_exporter
stdout_logfile=/tmp/prometheus/prometheus.log
autostart=true
autorestart=true
startsecs=5
priority=1
user=root
stopasgroup=true
killasgroup=true
……
[[email protected] conf]# chkconfig supervisord on
[[email protected] conf]# service supervisord start
[[email protected] conf]# supervisorctl status
nginx_exporter RUNNING pid 28008, uptime 0:05:57
node_exporter RUNNING pid 28007, uptime 0:05:57
防火牆開啟相應的埠:
[[email protected] conf]# vi /etc/sysconfig/iptables
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9913 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9100 -j ACCEPT
[[email protected] conf]# service iptables restart
開啟瀏覽器訪問http://10.100.xx.xxx/9913:
3.配置prometheus 新增nginx監控
修改prometheus配置(我的環境是使用consul動態的管理配置,請參考我的另一篇博文prometheus配置檔案動態管理)
[[email protected] conf]# vi prometheus.yml ##prometheus 主配置檔案
……
##新增
- job_name: 'nginx'
file_sd_configs:
- refresh_interval: 1m
files:
- ./conf.d/nginx*.json
……
[[email protected] conf]# vim nginx-discovery.ctmpl ## 建立nginx 自動發現模版檔案 nginx-discovery.ctmpl
[
{$ range tree "prometheus/nginx" $}
{
"targets": ["{$ .Value $}"],
"labels": {
"instance": "{$ .Key $}"
}
},
{$ end $}
{
"targets": ["10.100.xxx.xxx:9090"],
"labels": {
"instance": "prometheus01"
}
}
]
[[email protected] conf]# vim consul-template.conf ## consul 主配置檔案
……
## 新增
template {
source = "/usr/local/prometheus/templates/nginx-discovery.ctmpl"
destination = "/usr/local/prometheus/conf.d/nginx-discovery.json"
command = ""
backup = true
command_timeout = "60s"
left_delimiter = "{$"
right_delimiter = "$}"
wait {
min = "2s"
max = "20s"
}
}
……
[[email protected] conf]# curl -XPOST localhost:9090/-/reload ## 重新載入prometheus
[[email protected] conf]# supervisorctl restart consul-template ## 重新載入consul-template
配置consul 實現動態管理 監控nginx檔案:
來看看利用consul-template 動態生成的nginx 監控配置檔案:
[[email protected] conf]# cat nginx-discovery.json
[
{
"targets": ["10.100.xx.xxx:9913"],
"labels": {
"instance": "pre-nginx_10.100.xx.xxx"
}
},
{
"targets": ["10.100.xx.xxx:9913"],
"labels": {
"instance": "pre-nginx_10.100.xx.xxx"
}
},
{
"targets": ["10.100.xx.xxx:9090"],
"labels": {
"instance": "prometheus01"
}
}
]
prometheus監控介面:
大功告成!!