Nginx + Keepalived實現應用高可用負載均衡功能
說明：此處僅介紹 Keepalived 實現nginx負載均衡器的高可用，關於nginx介紹和負載均衡實現可查看我的另兩篇博文 Nginx負載均衡 和 Nginx配置了解
應用背景：實現高可用，避免單點故障
技術實現：使用2臺虛擬機通過Keepalived工具來實現 nginx 的高可用（High Avaiability），達到一臺nginx入口服務器宕機，另一臺備機自動接管服務的效果。（nginx做反向代理，實現後端應用服務器的負載均衡）
環境準備
192.168.182.130：nginx + keepalived master
192.168.182.133：nginx + keepalived backup
192.168.182.131：tomcat
192.168.182.132：tomcat
虛擬ip(VIP)：192.168.182.100，對外提供服務的ip，也可稱作浮動ip
各個組件之間的關系圖如下：
在Nginx負載均衡的基礎上再配置一臺nginx服務器用於負載均衡器，也就是此處的192.168.182.133服務器。
安裝Keepalived
在兩臺nginx服務器分別安裝keepalived，步驟如下：
keepalived源碼包下載地址：http://www.keepalived.org/download.html
keepalived安裝： 1、下載最新版源碼包keepalived-1.4.0.tar.gz；tar -zvxf keepalived-1.4.0.tar.gz
cd keepalived-1.4.0
4、查看安裝環境是否適合,並指定安裝目錄；
./configure --prefix=/usr/local/keepalived
5、系統出現警告：*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.
yum -y install libnl libnl-devel
6、 安裝完以後重新查看安裝環境是否適合,並指定安裝目錄
./configure --prefix=/usr/local/keepalived
7、出現錯誤： configure: error: libnfnetlink headers missing
yum install -y libnfnetlink-devel
8、安裝完以後重新查看安裝環境是否適合,並指定安裝目錄
./configure --prefix=/usr/local/keepalived
9、顯示環境符合，可進行keepalived安裝
make && make install
10、進入指定安裝目錄可看到如下目錄結構：
11、把keepalived配置到系統服務
mkdir /etc/keepalived/ cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/ cp /usr/local/keepalived/etc/init/keepalived.conf /etc/init.d/keepalived cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
12、自定義編輯keepalived執行腳本
vi /etc/init.d/keepalived
keepalived腳本文件如下：
#!/bin/sh # # keepalived High Availability monitor built upon LVS and VRRP # # chkconfig: - 86 14 # description: Robust keepalive facility to the Linux Virtual Server project # with multilayer TCP/IP stack checks. ### BEGIN INIT INFO # Provides: keepalived # Required-Start: $local_fs $network $named $syslog # Required-Stop: $local_fs $network $named $syslog # Should-Start: smtpdaemon httpd # Should-Stop: smtpdaemon httpd # Default-Start: # Default-Stop: 0 1 2 3 4 5 6 # Short-Description: High Availability monitor built upon LVS and VRRP # Description: Robust keepalive facility to the Linux Virtual Server # project with multilayer TCP/IP stack checks. ### END INIT INFO # Source function library. . /etc/rc.d/init.d/functions exec="/usr/local/keepalived/sbin/keepalived" prog="keepalived" config="/etc/keepalived/keepalived.conf" [ -e /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog lockfile=/var/lock/subsys/keepalived start() { [ -x $exec ] || exit 5 [ -e $config ] || exit 6 echo -n $"Starting $prog: " daemon $exec $KEEPALIVED_OPTIONS retval=$? echo [ $retval -eq 0 ] && touch $lock-file return $retval } stop() { echo -n $"Stopping $prog: " killproc $prog retval=$? echo [ $retval -eq 0 ] && rm -f $lockfile return $retval } restart() { stop start } reload() { echo -n $"Reloading $prog: " killproc $prog -1 retval=$? echo return $retval } force_reload() { restart } rh_status() { status $prog } rh_status_q() { rh_status &>/dev/null } case "$1" in start) rh_status_q && exit 0 $1 ;; stop) rh_status_q || exit 0 $1 ;; restart) $1 ;; reload) rh_status_q || exit 7 $1 ;; force-reload) force_reload ;; status) rh_status ;; condrestart|try-restart) rh_status_q || exit 0 restart ;; *) echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}" exit 2 esac exit $?註意： 修改以上腳本兩個地方：
- exec="/usr/sbin/keepalived" 修改成keepalived執行程序的路徑。
- config="/etc/keepalived/keepalived.conf" 修改成keepalived核心配置文件的路徑。
13、由於/etc/init.d/keepalived權限不夠（白色字體），授權
chmod 755 /etc/init.d/keepalived
14、將keepalived添加到開機自啟
vi /etc/rc.local
在編輯區將keepalived執行程序的路徑添加到最後一行，如下：
註意：由於/etc/rc.local權限不夠，需授權
chmod 755 /etc/rc.local
15、keepalived到此安裝完成！
配置兩臺nginx服務器的keepalived，實現兩臺nginx負載均衡器高可用
1、先配置nginx-master，也就是192.168.182.130這臺服務器
vi /etc/keepalived/keepalived.conf
keepalived.conf
global_defs { router_id 30 # 設置nginx-master服務器的ID，這個值在整個keepalived高可用架構中是唯一的，此處用IP末尾數標識 } vrrp_script check_nginx { script "/etc/keepalived/check_nginx.sh" # 配置keepalived監控nginx負載均衡服務器狀態的腳本，看nginx是否掛掉等情況，然後做相應處理 interval 5 # 檢測腳本執行的時間間隔，單位是秒 } vrrp_instance VI_1 { state MASTER # 指定keepalived的角色，MASTER為主，BACKUP為輔 interface eth0 # 指定當前進行VRRP通訊的接口卡（此處是centos的網卡） virtual_router_id 51 # 虛擬路由編號，主從要一致，以表明主從服務器屬於同一VRRP通訊協議小組 mcast_src_ip 192.168.182.130 # 本機IP地址 priority 100 # 優先級，數值越大，處理請求的優先級越高 advert_int 1 # 檢查間隔，默認1s（vrrp組播周期秒數） authentication { auth_type PASS auth_pass 1111 } track_script { # 調用檢測腳本 check_nginx } virtual_ipaddress { # 定義虛擬IP（VIP）,可設置多個，每行一個，兩個節點設置的虛擬IP必須一致 192.168.182.100 } }
2、將keepalived.conf配置中的監控腳本check_nginx.sh，放在配置中書寫的路徑下
#!/bin/bash counter=$(ps -C nginx --no-heading|wc -l) if [ "${counter}" = "0" ]; then /usr/local/nginx/sbin/nginx sleep 2 counter=$(ps -C nginx --no-heading|wc -l) if [ "${counter}" = "0" ]; then /etc/init.d/keepalived stop fi fi
3、端口調整
因為nginx服務器配置中顯示，nginx默認監控端口80，所以為了80端口為我所用，先將nginx配置中默認的80端口修改為8888（只要不被占用即可），然後，在我們配置的server模塊中，監聽80端口，主機名改為localhost，這樣，我們訪問虛擬IP192.168.182.100（以為是80端口，所以不需要帶端口號）時，才能映射到nginx服務器IP192.168.182.130，然後，nginx再去進行請求轉發給兩臺tomcat真實服務器192.168.182.131和192.168.182.132中的一臺
nginx-master配置nginx.conf修改為：
#user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; #log_format main ‘$remote_addr - $remote_user [$time_local] "$request" ‘ # ‘$status $body_bytes_sent "$http_referer" ‘ # ‘"$http_user_agent" "$http_x_forwarded_for"‘; #access_log logs/access.log main; sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; #gzip on; upstream serverCluster{ server 192.168.182.131:8080 weight=1; server 192.168.182.132:8080 weight=1; } server { listen 8888; # 此處默認80修改為8888 server_name localhost; #charset koi8-r; #access_log logs/host.access.log main; location / { root html; index index.html index.htm; } #error_page 404 /404.html; # redirect server error pages to the static page /50x.html # error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } # proxy the PHP scripts to Apache listening on 127.0.0.1:80 # #location ~ \.php$ { # proxy_pass http://127.0.0.1; #} # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000 # #location ~ \.php$ { # root html; # fastcgi_pass 127.0.0.1:9000; # fastcgi_index index.php; # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name; # include fastcgi_params; #} # deny access to .htaccess files, if Apache‘s document root # concurs with nginx‘s one # #location ~ /\.ht { # deny all; #} } # another virtual host using mix of IP-, name-, and port-based configuration # #server { # listen 8000; # listen somename:8080; # server_name somename alias another.alias; # location / { # root html; # index index.html index.htm; # } #} server{ listen 80; server_name localhost; location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass http://serverCluster; expires 3d; } } # HTTPS server # #server { # listen 443 ssl; # server_name localhost; # ssl_certificate cert.pem; # ssl_certificate_key cert.key; # ssl_session_cache shared:SSL:1m; # ssl_session_timeout 5m; # ssl_ciphers HIGH:!aNULL:!MD5; # ssl_prefer_server_ciphers on; # location / { # root html; # index index.html index.htm; # } #} }
4、端口調整結束後，重啟keepalived，查看keepalived配置是否起作用
出現上圖中eth0（我們配置的VRRP通信網卡就是eth0）紅色標記中的內容，則表示虛擬IP成功映射真實服務器IP，keepalived配置成功！
5、同上1-4操作對nginx-slave，也就是IP為192.168.182.133的nginx負載均衡器從屬進行配置
實驗：
此處瀏覽器演示過程不再介紹，只把結論和演示過程中需要註意的地方進行陳述：
結論：
1、多次訪問虛擬IP192.168.182.100，瀏覽器會輪流展示兩臺tomcat的頁面，說明keepalived將虛擬IP映射到了nginx服務真實IP，並且nginx對請求進行了負載均衡。
2、將nginx-master手動停止nginx服務，幾秒過後，再去檢查nginx運行狀態，會發現，nginx正在運行中，這說明，keepalived的配置中的監控nginx腳本文件成功運行。
3、將nginx-master手動停止keepalived服務，會發現，虛擬IP被轉移到nginx-slave服務器，並且和nginx-slave真實IP進行映射，此時，用戶繼續訪問虛擬IP192.168.182.100，並不會感知發生什麽變化，但是真實情況是，nginx-master這臺機器已經不再工作，擔任系統負載均衡的變成了nginx-slave；如果你再把nginx-master的keepalived服務手動開啟，則情況會初始化為原先的狀態，即：nginx-master繼續擔任系統負載均衡角色，而nginx-slave則重新閑置，這種情況說明keepalived實現了nginx的高可用（HA）。
註意：
1、在實驗測試過程中，有可能因為瀏覽器緩存的緣故導致結果出乎意料，所以在每次測試前，先清一下緩存
2、請求走向：訪問虛擬IP(VIP)，keepalived將請求映射到本地nginx，nginx將請求轉發至tomcat，例如：http://192.168.182.100，被映射成http://192.168.182.130，端口是80，而130上nginx的端口正好是80；映射到nginx上後，nginx再進行請求的轉發。
3、VIP總會在keepalived服務器中的某一臺上，也只會在其中的某一臺上；VIP綁定的服務器上的nginx就是master，當VIP所在的服務器宕機了，keepalived會將VIP轉移到backup上，並將backup提升為master。
4、VIP也稱浮動ip，是公網ip，與域名進行映射，對外提供服務； 其他ip一般而言都是內網ip， 外部是直接訪問不了的
Nginx + Keepalived實現應用高可用負載均衡功能