【Web 叢集實戰】20_Keepalived 高可用叢集
阿新 • • 發佈：2018-11-01
【Web 叢集實戰】20_Keepalived 高可用叢集
標籤（空格分隔）： Web叢集實戰
文章目錄
- 【Web 叢集實戰】20_Keepalived 高可用叢集
1. Keepalived 高可用故障切換原理
-
Keepalived 高可用服務隊之間的故障切換轉移，是通過 VRRP（Virtual Router Redundancy Protocol，虛擬路由器冗餘協議）來實現的。
-
VRRP 通過競選機制來實現虛擬路由器的功能，所有的協議報文都是通過 IP 多播（Muiticast)包（預設的多播地址 224.0.0.18）形式傳送的。虛擬路由器由 VRID（範圍 0-255）和一組 IP 地址組成，對外表現為一個周知的 MAC 地址：00-00-5E-00-01-{VRID}。所以，在一個虛擬路由器中，不管誰是 Master，對外都是相同的 MAC 和 IP（稱之為 VIP）。客戶端主機並不需要因 Master 的改變而修改自己的路由配置。對它們來說，這種切換是透明的。
2. Keepalived 高可用服務搭建準備
- 硬體準備
| HOSTNAME | IP | 說明 |
|---|---|---|
| lb001 | 192.168.2.129 | Keepalived 主伺服器（Nginx 主負載均衡器） |
| lb002 | 192.168.2.130 | Keepalived 輔伺服器（Nginx 輔負載均衡器） |
| web001 | 192.168.2.146 | web001 伺服器 |
| web002 | 192.168.2.131 | web002 伺服器 |
- 開始安裝 keepalived 軟體
[[email protected] ~]# yum install keepalived -y
[[email protected] ~]# yum install keepalived -y
3. 配置 Keepalived 實現單例項單 IP 自動漂移接管
3.1 配置 Keepalived 主伺服器 lb001 MASTER
- 關閉防火牆
[[email protected] ~]# systemctl stop firewalld
- 配置 lb001 MASTER 的 keepalived.conf 配置檔案
[[email protected] ~]# cd /etc/keepalived/
[[email protected] keepalived]# vim keepalived.conf
[[email protected] keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id lb001 # <-- 區域網內應唯一
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 55 # <-- 虛擬路由 ID 標識，在一個 keepalived.conf 中是唯一的。MASTER 和 BACKUP 配置中相同例項又必須是一致的。
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.2.188/24 dev ens33 label ens33:1
}
}
- 啟動 keepalived 服務
[[email protected] keepalived]# systemctl start keepalived.service
[[email protected] keepalived]# ps -ef|grep keepalived
root 1258 1 0 16:30 ? 00:00:00 /usr/sbin/keepalived -D
root 1259 1258 0 16:30 ? 00:00:00 /usr/sbin/keepalived -D
root 1260 1258 0 16:30 ? 00:00:00 /usr/sbin/keepalived -D
root 1298 1165 0 16:33 pts/0 00:00:00 grep --color=auto keepalived
[[email protected] keepalived]# ip addr|grep 192.168.2.188
inet 192.168.2.188/24 scope global secondary ens33:1
###3.1 配置 Keepalived 主伺服器 lb0012 BACKUP
- 關閉防火牆
[[email protected] ~]# systemctl stop firewalld
- 配置 lb002 BACKUP 的 keepalived.conf 配置檔案
[[email protected] ~]# cd /etc/keepalived/
[[email protected] keepalived]# vim keepalived.conf
[[email protected] keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id lb002 # <-- 區域網內應唯一
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 55 # <-- 虛擬路由 ID 標識，在一個 keepalived.conf 中是唯一的。MASTER 和 BACKUP 配置中相同例項又必須是一致的。
priority 100 # <-- 優先順序需低於主節點至少50
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.2.188/24 dev ens33 label ens33:1
}
}
- 啟動 keepalived 服務
[[email protected] keepalived]# systemctl start keepalived.service
[[email protected] keepalived]# ps -ef|grep keepalived
root 1298 1 0 16:43 ? 00:00:00 /usr/sbin/keepalived -D
root 1299 1298 0 16:43 ? 00:00:00 /usr/sbin/keepalived -D
root 1300 1298 0 16:43 ? 00:00:00 /usr/sbin/keepalived -D
root 1306 1192 0 16:43 pts/0 00:00:00 grep --color=auto keepalived
[[email protected] keepalived]# ip addr|grep 192.168.2.188
[[email protected] keepalived]#
# 此時應無返回
3.3 高可用主備伺服器切換實驗
[[email protected] ~]# systemctl stop keepalived.service
[[email protected] keepalived]# ip addr|grep 192.168.2.188
inet 192.168.2.188/24 scope global ens33:1
[[email protected] ~]# systemctl start keepalived.service
[[email protected] ~]# ip addr|grep 192.168.2.188
inet 192.168.2.188/24 scope global eth33:1
[[email protected] keepalived]# ip addr|grep 192.168.2.188
[[email protected] keepalived]#
4. Keepalived 雙例項雙主模式配置
Keepalived 雙例項雙主模式的 IP 及 VIP 規劃表
| HOSTNAME | IP | 說明 |
|---|---|---|
| lb001 | 192.168.2.129 | VIP：192.168.2.188（用於繫結 A 服務 www.yangyangyang.org 域名） |
| lb002 | 192.168.2.130 | VIP：192.168.2.189（用於繫結 B 服務 bbs.yangyangyang.org 域名） |
4.1 配置伺服器 lb001
- 配置 lb001 的 keepalived.conf
[[email protected] keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id lb001
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 55
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.2.188/24 dev ens33 label ens33:1
}
}
vrrp_instance VI_2 {
state BACKUP
interface ens33
virtual_router_id 56
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.2.189/24 dev ens33 label ens33:2
}
}
4.2 配置伺服器 lb002
- 配置 lb002 的 keepalived.conf
[[email protected] keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id lb002
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 55
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.2.188/24 dev ens33 label ens33:1
}
}
vrrp_instance VI_2 {
state MASTER
interface ens33
virtual_router_id 56
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.2.189/24 dev ens33 label ens33:2
}
}
4.3 高可用切換測試
- 在 lb001 和 lb002 分別啟動 Keepalived 服務
[[email protected] keepalived]# systemctl restart keepalived.service
[[email protected] keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
inet 192.168.2.188/24 scope global ens33:1
[[email protected] keepalived]# systemctl restart keepalived.service
[[email protected] keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
inet 192.168.2.189/24 scope global ens33:2
- 停掉 lb002 Keepalived 服務：
[[email protected] keepalived]# systemctl stop keepalived.service
[[email protected] keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
inet 192.168.2.188/24 scope global ens33:1
inet 192.168.2.189/24 scope global secondary ens33:2
[[email protected] keepalived]# systemctl start keepalived.service
[[email protected] keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
inet 192.168.2.189/24 scope global ens33:2
[[email protected] keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
inet 192.168.2.188/24 scope global ens33:1
- 停掉 lb001 Keepalived 服務：
[[email protected] keepalived]# systemctl stop keepalived.service
[[email protected] keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
inet 192.168.2.189/24 scope global ens33:2
inet 192.168.2.188/24 scope global secondary ens33:1
[[email protected] keepalived]# systemctl start keepalived.service
[[email protected] keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
inet 192.168.2.188/24 scope global ens33:1
[[email protected] keepalived]# ip addr|egrep "192.168.2.188|192.168.2.189"
inet 192.168.2.189/24 scope global ens33:2
5. Nginx 負載均衡配合 Keepalived 服務
5.1 在 lb001 和 lb002 上配置 Nginx 負載均衡
[[email protected] keepalived]# cat /application/nginx/conf/nginx.conf
worker_processes 1;
error_log logs/error.log;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
upstream www_server_pools {
server 192.168.2.131:80 weight=1;
server 192.168.2.146:80 weight=1;
}
server {
listen 192.168.2.188:80;
server_name www.yangyangyang.org;
location / {
proxy_pass http://www_server_pools;
include proxy.conf;
}
}
}
[[email protected] keepalived]# cat /application/nginx/conf/nginx.conf
worker_processes 1;
error_log logs/error.log;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
upstream www_server_pools {
server 192.168.2.131:80 weight=1;
server 192.168.2.146:80 weight=1;
}
server {
listen 192.168.2.188:80;
server_name www.yangyangyang.org;
location / {
proxy_pass http://www_server_pools;
include proxy.conf;
}
}
}
5.2 在 lb001 和 lb002 上配置 Keepalived 服務
[[email protected] keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id lb001
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 55
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.2.188/24 dev ens33 label ens33:1
}
}
[[email protected] keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id lb002
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 55
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.2.188/24 dev ens33 label ens33:1
}
}
5.3 解決服務監聽的網絡卡上不存在 IP 地址問題
[[email protected] keepalived]# echo 'net.ipv4.ip_nonlocal_bind = 1' >> /etc/sysctl.conf
[[email protected] keepalived]# tail -1 /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind = 1
[[email protected] keepalived]# sysctl -p
net.ipv4.ip_nonlocal_bind = 1
[[email protected] keepalived]# echo 'net.ipv4.ip_nonlocal_bind = 1' >> /etc/sysctl.conf
[[email protected] keepalived]# tail -1 /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind = 1
[[email protected] keepalived]# sysctl -p
net.ipv4.ip_nonlocal_bind = 1
p.s. 上述部分均由橋接模式完成，由於使用者模擬訪問時橋接模式無法訪問外網，於是改成NAT模式的DHCP方式上網，以下測試均為NAT模式。篇首的IP地址為NAT模式的地址。
5.4 使用者訪問準備及模擬實際訪問
（1）在客戶端 hosts 檔案裡把 www.yangyangyang.org 域名解析到 VIP 192.168.2.188 上，正式場景需要通過 DNS 解析。
192.168.2.188 www.yangyangyang.org
（2）兩臺 web 伺服器開啟 Nginx 服務，並配置首頁檔案
[[email protected] ~]# /application/nginx/sbin/nginx
[[email protected] ~]# cat /application/nginx/html/www/index.html
192.168.2.146 www.yangyangyang.org
[[email protected] ~]# /application/nginx/sbin/nginx
[[email protected] ~]# cat /application/nginx/html/www/index.html
192.168.2.131 www.yangyangyang.org
（3）兩臺負載均衡伺服器配好 Nginx 服務，並確保後面代理的 Web 節點可以測試訪問
[[email protected] keepalived]# /application/nginx/sbin/nginx
[[email protected] keepalived]# lsof -i:80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 1151 root 6u IPv4 20341 0t0 TCP www.yangyangyang.org:http (LISTEN)
nginx 1215 nginx 6u IPv4 20341 0t0 TCP www.yangyangyang.org:http (LISTEN)
[[email protected] keepalived]# ip addr|grep 192.168.2.188
inet 192.168.2.188/24 scope global secondary ens33:1
（4）模擬訪問
- 在客戶端瀏覽器輸入 www.yangyangyang.org 測試訪問，重新整理幾次
- 此時停止 lb001 伺服器或者停掉 Keepalived 服務，觀察業務是否正常
[[email protected] keepalived]# systemctl stop keepalived.service
- 觀察 lb002 備節點是否接管 VIP 192.168.2.188
[[email protected] keepalived]# ip addr|grep 192.168.2.188
inet 192.168.2.188/24 scope global secondary ens33:1
-
再次在客戶端瀏覽器輸入 www.yangyangyang.org 測試訪問，重新整理幾次，出現和切換 lb002 前相同的訪問結果
-
開啟 lb001 的 Keepalived 服務，VIP 又接管回來了。
6. 配置指定檔案接收 Keepalived 服務日誌
（1）編輯配置檔案 /etc/sysconfig/keepalived ，將 14 行的 KEEPALIVED_OPTIONS="-D" 修改為 KEEPALIVED_OPTIONS="-D -d -S 0"
[[email protected] ~]# sed -i '14 s#KEEPALIVED_OPTIONS="-D"#KEEPALIVED_OPTIONS="-D -d -S 0"#g' /etc/sysconfig/keepalived
[[email protected] ~]# sed -n '14p' /etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-D -d -S 0"
（2）修改 rsyslog 的配置檔案 vim /etc/rsyslog.conf ，在結尾處加上如下兩行內容
[[email protected] ~]# tail -2 /etc/rsyslog.conf
#keepalived
local0.* /var/log/keepalived.log
上述配置表示來自 local0 裝置的所有日誌資訊都記錄到 /var/log/keepalived.log 檔案。
（3）修改 rsyslog 的配置檔案 vim /etc/rsyslog.conf，將 54 行改成
*.info;mail.none;authpriv.none;cron.none;local0.none /var/log/messages
（4）配置完成後，重啟 rsyslog 服務
[[email protected] ~]# systemctl restart rsyslog.service
（5）關閉 keepalived 服務，測試 Keepalived 日誌記錄結果。
[[email protected] ~]# tail /var/log/keepalived.log
Oct 10 20:07:15 localhost Keepalived_vrrp[1252]: Sending gratuitous ARP on ens33 for 192.168.2.188
Oct 10 20:07:15 localhost Keepalived_vrrp[1252]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.2.188
Oct 10 20:07:15 localhost Keepalived_vrrp[1252]: Sending gratuitous ARP on ens33 for 192.168.2.188
Oct 10 20:07:15 localhost Keepalived_vrrp[1252]: Sending gratuitous ARP on ens33 for 192.168.2.188
Oct 10 20:07:15 localhost Keepalived_vrrp[1252]: Sending gratuitous ARP on ens33 for 192.168.2.188
Oct 10 20:07:15 localhost Keepalived_vrrp[1252]: Sending gratuitous ARP on ens33 for 192.168.2.188
Oct 10 20:07:33 localhost Keepalived[1250]: Stopping
Oct 10 20:07:33 localhost Keepalived_vrrp[1252]: VRRP_Instance(VI_1) sent 0 priority
Oct 10 20:07:33 localhost Keepalived_vrrp[1252]: VRRP_Instance(VI_1) removing protocol VIPs.
Oct 10 20:07:33 localhost Keepalived_healthcheckers[1251]: Stopped