keepalived+nginx實現HA高可用的web負載均衡
阿新 • • 發佈：2018-12-18
Keepalived 是一種高效能的伺服器高可用或熱備解決方案， Keepalived 可以用來防止伺服器單點故障的發生，通過配合 Nginx 可以實現 web 前端服務的高可用。
Keepalived 以 VRRP 協議為實現基礎，用 VRRP 協議來實現高可用性(HA)。 VRRP(Virtual RouterRedundancy Protocol)協議是用於實現路由器冗餘的協議， VRRP 協議將兩臺或多臺路由器裝置虛擬成一個裝置，對外提供虛擬路由器 IP(一個或多個)，而在路由器組內部，如果實際擁有這個對外 IP 的路由器如果工作正常的話就是 MASTER，或者是通過演算法選舉產生， MASTER 實現針對虛擬路由器 IP 的各種網路功能，如 ARP 請求， ICMP，以及資料的轉發等；其他裝置不擁有該虛擬 IP，狀態是 BACKUP，除了接收 MASTER 的VRRP 狀態通告資訊外，不執行對外的網路功能。當主機失效時， BACKUP 將接管原先 MASTER 的網路功能。VRRP 協議使用多播資料來傳輸 VRRP 資料， VRRP 資料使用特殊的虛擬源 MAC 地址傳送資料而不是自身網絡卡的 MAC 地址， VRRP 執行時只有 MASTER 路由器定時傳送 VRRP 通告資訊，表示 MASTER 工作正常以及虛擬路由器 IP(組)， BACKUP 只接收 VRRP 資料，不傳送資料，如果一定時間內沒有接收到 MASTER 的通告資訊，各 BACKUP 將宣告自己成為 MASTER，傳送通告資訊，重新進行 MASTER 選舉狀態。
ip規劃如下:定義VIP為:172.16.23.132
nginx1:172.16.23.129 keepalived:172.16.23.129
nginx2:172.16.23.130 keepalived:172.16.23.130
httpd1:172.16.23.128
httpd2:172.16.23.131
上面規劃中nginx只提供負載均衡作用,並不實現web訪問功能:
[[email protected] ~]# cat /etc/ansible/hosts|grep "^\[nodes" -A 2 [nodes] 172.16.23.129 172.16.23.130
檢視nginx服務狀態:
[[email protected] ~]# ansible nodes -m shell -a "systemctl status nginx"|grep running Active: active (running) since 二 2018-12-18 16:33:04 CST; 12min ago Active: active (running) since 二 2018-12-18 16:35:51 CST; 10min ago
首先nginx服務正常開啟,然後檢視後端服務httpd:
[[email protected] ~]# cat /etc/ansible/hosts|grep "^\[backend_nodes" -A 2 [backend_nodes] 172.16.23.128 172.16.23.131
檢視httpd服務狀態:
[[email protected] ~]# ansible backend_nodes -m shell -a "systemctl status httpd"|grep running Active: active (running) since 二 2018-12-18 16:29:36 CST; 22min ago Active: active (running) since 二 2018-12-18 16:30:03 CST; 21min ago
然後在nginx兩臺伺服器上分別測試負載均衡效果:
[[email protected] ~]# ansible 172.16.23.129 -m get_url -a "url=http://172.16.23.129/index.html dest=/tmp"|grep status_code "status_code": 200, [[email protected] ~]# ansible 172.16.23.129 -m shell -a "cat /tmp/index.html" 172.16.23.129 | CHANGED | rc=0 >> 172.16.23.128 [[email protected] ~]# ansible 172.16.23.129 -m get_url -a "url=http://172.16.23.129/index.html dest=/tmp"|grep status_code "status_code": 200, [[email protected] ~]# ansible 172.16.23.129 -m shell -a "cat /tmp/index.html" 172.16.23.129 | CHANGED | rc=0 >> 172.16.23.131
由上面可以看出nginx1:172.16.23.129上進行測試返回後端httpd服務的web頁面:172.16.23.128以及172.16.23.131,測試訪問沒有問題,負載均衡沒有問題
[[email protected] ~]# ansible 172.16.23.130 -m get_url -a "url=http://172.16.23.130/index.html dest=/tmp"|grep status_code "status_code": 200, [[email protected] ~]# ansible 172.16.23.130 -m shell -a "cat /tmp/index.html" 172.16.23.130 | CHANGED | rc=0 >> 172.16.23.128 [[email protected] ~]# ansible 172.16.23.130 -m get_url -a "url=http://172.16.23.130/index.html dest=/tmp"|grep status_code "status_code": 200, [[email protected] ~]# ansible 172.16.23.130 -m shell -a "cat /tmp/index.html" 172.16.23.130 | CHANGED | rc=0 >> 172.16.23.131
由上面可以看見nginx2服務訪問後端httpd服務也是完全OK的,於是nginx兩臺服務負載均衡效果達到,現在在nginx兩臺伺服器上安裝keepalived服務:
[[email protected] ~]# ansible nodes -m shell -a "systemctl status keepalived"|grep running Active: active (running) since 二 2018-12-18 16:06:38 CST; 52min ago Active: active (running) since 二 2018-12-18 16:05:04 CST; 54min ago
檢視VIP資訊:發現vip在node1節點上
[[email protected] ~]# ansible nodes -m shell -a "hostname;ip a|grep ens33|grep -Po '(?<=inet ).*(?=\/)'" 172.16.23.129 | CHANGED | rc=0 >> node1 172.16.23.129 172.16.23.132 172.16.23.130 | CHANGED | rc=0 >> node2 172.16.23.130
可以看出VIP落在了nginx1也就是node1節點上,然後通過訪問vip看看負載均衡效果:
[[email protected] ~]# curl http://172.16.23.132 172.16.23.131 [[email protected] ~]# curl http://172.16.23.132 172.16.23.128
由上面返回結果看,沒有任何問題,現在摘掉一臺nginx伺服器,看看keepalived情況,以及訪問vip的情況:
[[email protected] ~]# ansible 172.16.23.130 -m shell -a "systemctl stop nginx" 172.16.23.130 | CHANGED | rc=0 >>
檢視keepalived服務狀態,檢視vip資訊:
[[email protected] ~]# ansible nodes -m shell -a "systemctl status keepalived"|grep running Active: active (running) since 二 2018-12-18 16:05:04 CST; 1h 4min ago Active: active (running) since 二 2018-12-18 16:06:38 CST; 1h 3min ago [[email protected] ~]# ansible nodes -m shell -a "hostname;ip a|grep ens33|grep -Po '(?<=inet ).*(?=\/)'" 172.16.23.130 | CHANGED | rc=0 >> node2 172.16.23.130 172.16.23.129 | CHANGED | rc=0 >> node1 172.16.23.129 172.16.23.132
vip資訊沒有漂移,keepalived服務狀態正常,現在訪問vip:
[[email protected] ~]# curl http://172.16.23.132 172.16.23.128 [[email protected] ~]# curl http://172.16.23.132 172.16.23.131
通過vip訪問web服務沒有問題
現在將nginx服務開啟,端掉一個節點的keepalived服務:
[[email protected] ~]# ansible 172.16.23.130 -m shell -a "systemctl start nginx" 172.16.23.130 | CHANGED | rc=0 >> [[email protected] ~]# ansible nodes -m shell -a "systemctl status nginx"|grep running Active: active (running) since 二 2018-12-18 17:15:48 CST; 18s ago Active: active (running) since 二 2018-12-18 16:33:04 CST; 43min ago
[[email protected] ~]# ansible 172.16.23.130 -m shell -a "systemctl stop keepalived" 172.16.23.130 | CHANGED | rc=0 >>
然後在該節點日誌檢視如下:tail -f /var/log/message
Dec 18 17:16:50 node2 systemd: Stopping LVS and VRRP High Availability Monitor... Dec 18 17:16:50 node2 Keepalived[12981]: Stopping Dec 18 17:16:50 node2 Keepalived_healthcheckers[12982]: Stopped Dec 18 17:16:51 node2 Keepalived_vrrp[12983]: Stopped Dec 18 17:16:51 node2 Keepalived[12981]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2 Dec 18 17:16:52 node2 systemd: Stopped LVS and VRRP High Availability Monitor.
[[email protected] ~]# ansible nodes -m shell -a "systemctl status keepalived"|grep running Active: active (running) since 二 2018-12-18 16:06:38 CST; 1h 10min ago [[email protected] ~]# ansible nodes -m shell -a "hostname;ip a|grep ens33|grep -Po '(?<=inet ).*(?=\/)'" 172.16.23.130 | CHANGED | rc=0 >> node2 172.16.23.130 172.16.23.129 | CHANGED | rc=0 >> node1 172.16.23.129 172.16.23.132
由於斷掉的是nginx2也就是node2節點的keepalived服務,所以vip還是在node1上,並沒有漂移在node2,檢視node1和node2節點上keepalived服務的配置檔案:
[[email protected] ~]# ansible nodes -m shell -a "cat /etc/keepalived/keepalived.conf" 172.16.23.129 | CHANGED | rc=0 >> ! Configuration File for keepalived global_defs { notification_email { [email protected] } notification_email_from [email protected] smtp_server smtp.163.com smtp_connect_timeout 30 router_id test } vrrp_instance VI_1 { state BACKUP interface ens33 virtual_router_id 51 priority 100 nopreempt # 非搶佔模式 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.16.23.132/24 dev ens33 } } 172.16.23.130 | CHANGED | rc=0 >> ! Configuration File for keepalived global_defs { notification_email { [email protected] } notification_email_from [email protected] smtp_server smtp.163.com smtp_connect_timeout 30 router_id test } vrrp_instance VI_1 { state BACKUP interface ens33 virtual_router_id 51 priority 99 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.16.23.132/24 dev ens33 } }
可以由配置看出,只有優先順序不一樣以及node1節點設定了nopreempt # 非搶佔模式,現在將node2節點的keepalived服務開啟,然後將node1節點的keepalived服務關掉,看看vip資訊:
[[email protected] ~]# ansible 172.16.23.130 -m shell -a "systemctl start keepalived" 172.16.23.130 | CHANGED | rc=0 >>
檢視node2日誌:
Dec 18 17:23:14 node2 systemd: Starting LVS and VRRP High Availability Monitor... Dec 18 17:23:14 node2 Keepalived[15994]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2 Dec 18 17:23:14 node2 Keepalived[15994]: Opening file '/etc/keepalived/keepalived.conf'. Dec 18 17:23:14 node2 Keepalived[15995]: Starting Healthcheck child process, pid=15996 Dec 18 17:23:14 node2 Keepalived_healthcheckers[15996]: Opening file '/etc/keepalived/keepalived.conf'. Dec 18 17:23:14 node2 Keepalived[15995]: Starting VRRP child process, pid=15997 Dec 18 17:23:14 node2 systemd: Started LVS and VRRP High Availability Monitor. Dec 18 17:23:14 node2 Keepalived_vrrp[15997]: Registering Kernel netlink reflector Dec 18 17:23:14 node2 Keepalived_vrrp[15997]: Registering Kernel netlink command channel Dec 18 17:23:14 node2 Keepalived_vrrp[15997]: Registering gratuitous ARP shared channel Dec 18 17:23:14 node2 Keepalived_vrrp[15997]: Opening file '/etc/keepalived/keepalived.conf'. Dec 18 17:23:24 node2 Keepalived_vrrp[15997]: VRRP_Instance(VI_1) removing protocol VIPs. Dec 18 17:23:24 node2 Keepalived_vrrp[15997]: Using LinkWatch kernel netlink reflector... Dec 18 17:23:24 node2 Keepalived_vrrp[15997]: VRRP_Instance(VI_1) Entering BACKUP STATE Dec 18 17:23:24 node2 Keepalived_vrrp[15997]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
兩節點keepalived服務狀態,以及vip資訊:
[[email protected] ~]# ansible nodes -m shell -a "systemctl status keepalived"|grep running Active: active (running) since 二 2018-12-18 17:23:14 CST; 56s ago Active: active (running) since 二 2018-12-18 16:06:38 CST; 1h 17min ago [[email protected] ~]# ansible nodes -m shell -a "hostname;ip a|grep ens33|grep -Po '(?<=inet ).*(?=\/)'" 172.16.23.129 | CHANGED | rc=0 >> node1 172.16.23.129 172.16.23.132 172.16.23.130 | CHANGED | rc=0 >> node2 172.16.23.130