Prometheus Grafana實現對Kubernetes Nginx 的監控系統
用Prometheus配合Grafana實現多用途的服務監控系統。
一. Prometheus簡介
Prometheus是一個開源監控系統，它前身是SoundCloud的警告工具包。從2012年開始，許多公司和組織開始使用Prometheus。該專案的開發人員和使用者社群非常活躍，越來越多的開發人員和使用者參與到該專案中。
具體介紹及架構資訊參考Prometheus官網：https://prometheus.io
二. Grafana介紹
grafana是用於視覺化大型測量資料的開源程式，他提供了強大和優雅的方式去建立、共享、瀏覽資料。dashboard中顯示了你不同metric資料來源中的資料。
具體參考Grafana官網：https://grafana.com
三.實現例項
1.監控Openresty（Nginx）
1.1下載安裝Openresty及nginx-module-vts監控模組
#安裝編譯環境 yum -y install gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel #下載openresty和nginx-module-vts監控模組 cd /data/server wget https://openresty.org/download/openresty-1.13.6.2.tar.gz git clone https://github.com/vozlt/nginx-module-vts.git #解壓openresty tar -zxf openresty-1.13.6.2.tar.gz #新增模組編譯 ./configure --add-module=/data/server/nginx-module-vts make && make install
1.2 修改Openresty配置
不指定安裝路徑的話Openresty預設安裝在/usr/local/openresty
進入目錄/usr/local/openresty/nginx/conf 修改配置檔案
cd /usr/local/openresty/nginx/conf
#先備份
cp nginx.conf nginx.conf.bak
vim nginx.conf變更配置如下：
主要兩個配置，1.指定時間同步，輸出日誌格式。2.新增http訪問路由
user nobody; worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; vhost_traffic_status_zone; log_format main '{ "@timestamp": "$time_local", ' '"@fields": { ' '"uri":"$request_uri",' '"url":"$uri",' '"upstream_addr":"$upstream_addr",' '"remote_addr": "$remote_addr", ' '"remote_user": "$remote_user", ' '"body_bytes_sent": "$body_bytes_sent", ' '"host":"$host",' '"server_addr":"$server_addr",' '"request_time": "$request_time", ' '"request_time":"$request_time",' '"status":"$status",' '"request": "$request", ' '"request_method": "$request_method", ' '"size":$body_bytes_sent,' '"upstream_time":"$upstream_response_time"' '"http_referrer": "$http_referer", ' '"body_bytes_sent":"$body_bytes_sent", ' '"http_x_forwarded_for": "$http_x_forwarded_for", ' '"http_user_agent": "$http_user_agent" } }'; sendfile on; keepalive_timeout 65; gzip on; gzip_min_length 1k; gzip_buffers 4 16k; gzip_http_version 1.0; gzip_comp_level 3; gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript application/json application/javascript; gzip_vary on; proxy_http_version 1.1; proxy_set_header Connection ""; server { listen 80; server_name localhost; location / { root html; index index.html index.htm; } location /status { vhost_traffic_status_display; vhost_traffic_status_display_format html; } } }
測試配置檔案是否有誤
./sbin/nginx -t
通過測試後啟動Openresty
./sbin/nginx訪問地址http://[ip:port]/status
可以看到如下資訊：
1.3配置監控轉發模組
這裡先使用一個Github上已經開發好的第三方監控模組nginx-vts-exporter
下載地址：https://github.com/hnlq715/nginx-vts-exporter/releases
cd /data/server
wget https://github.com/hnlq715/nginx-vts-exporter/releases/download/v0.10.3/nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
tar -zxf nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
mv nginx-vts-exporter-0.10.3.linux-amd64 nginx-vts-exporter把nginx-vts-exporter 寫成服務(以Centos7.X為例)：
在/usr/lib/systemd/system/下建立啟動檔案
vim /usr/lib/systemd/system/nginx-vts-exporter.service配置啟動可執行檔案和引數
[Unit]
Description=nginx-vts-exporter
After=network.target
[Service]
Type=simple
ExecStart=/data/server/nginx-vts-exporter/nginx-vts-exporter -nginx.scrape_uri=http://127.0.0.1/status/format/json
Restart=on-failure
[Install]
WantedBy=multi-user.target生效配置檔案並啟動nginx-vts-exporter
#啟動服務，nginx-vts-exporter 預設監聽埠為 9913
systemctl enable nginx-vts-exporter
systemctl start nginx-vts-exporter
systemctl status nginx-vts-exporter1.4 配置Prometheus
配置檔案路徑：/data/server/prometheus/prometheus.yml，編輯配置檔案新增如下內容：
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
- job_name: 'openresty-master'
static_configs:
- targets: ['localhost:9913']
配置服務來啟動
vim /usr/lib/systemd/system/prometheus.service寫入如下內容：
[Unit]
Description=prometheus
After=network.target
[Service]
Type=simple
ExecStart=/data/server/prometheus/prometheus --config.file=/data/server/prometheus/prometheus.yml
Restart=on-failure
[Install]
WantedBy=multi-user.target啟動prometheus服務
#啟動prometheus服務
systemctl enable prometheus
systemctl start prometheus
systemctl status prometheus1.5 安裝配置Grafana
下載地址：https://grafana.com/grafana/download?platform=linux
按上述頁面說明安裝
wget https://dl.grafana.com/oss/release/grafana-6.1.3-1.x86_64.rpm
yum localinstall grafana-6.1.3-1.x86_64.rpm 我比較習慣先修改一下登入密碼（預設賬號密碼 admin admin）
先看一下安裝位置
whereis grafana發現安裝在/etc/grafana 去/etc/grafana下編輯grafana.ini檔案
修改[security]節點下的password為你期望的值
新版本 安裝後輸入admin預設密碼第一次進入會提示修改密碼
啟動Grafana服務並檢視狀態
systemctl start grafana-server
systemctl status grafana-server訪問頁面http://[ip]:3000
進入後點擊設定data sources add data sources
選擇Prometheus填寫如下資訊
使用Prometheus 2.0 Stats作為Dashboard效果如下：
2.監控Kubernetes（k8s）