https://www.linuxzen.com/lvskeepalivedshi-xian-gao-ke-yong-qun-ji-pei-zhi-xiang-jie.html

lvs是一個開源的軟體，由畢業於國防科技大學的章文嵩博士於1998年5月創立(中國人的專案)，可以實現LINUX平臺下的簡單負載均衡。LVS是Linux Virtual Server的縮寫，意思是Linux虛擬伺服器。本文將介紹lvs結合keepalived實現一個高科用的Linux群集系統.

lvs有三種工作模式NAT(地址轉換),IP Tunneling(IP隧道)、Direct Routing(直接路由)。 工作效率最低的是NAT模式,但NAT模式可以用於各種系統,各種環境的負載均衡,只需要一個公網ip即可實現 IP Tunneling模式排程器將連線分發到不同的後端real server,然後由real server處理請求直接相應給使用者,大大提高了排程器的排程效率,後端real server沒有物理位置和邏輯關係的限制,後端real server可以在Lan/Wlan,但是後端real server必須支援IP隧道協議. DR(Direct Routing)是效率最高的,與IP Tunneling類似,都是處理一般連線,將請求給後端real server,然後由real server處理請求直接相應給使用者,Direct Routing與IP Tunneling相比，沒有IP封裝的開銷，但由於採用物理層,所以DR模式的排程器和後端real server必須在一個物理網段裡,中間不能過路由器(也就是一個交換機相連).

lvs支援8種不同的排程演算法輪叫(rr)、加權輪叫(wrr)、最小連線(lc)、加權最小連線(wlc)、基於區域性性最小連線(lblc)、帶複製的基於區域性性最少連結(lblcr)、目標地址雜湊(dh)和源地址雜湊(sh).

下面就介紹如何來安裝和配置lvs+keepalived

本文使用環境: 作業系統:CentOS 5.5 32bit

主排程器:192.168.3.101/24

備排程器:192.168.3.102/24

後端real server: 192.168.3.3/24 |192.168.3.102/24(我們這裡使用備用lvs作為一個測試

vip(virtual ip):192.168.3.100/24

lvs在2.6的核心中是預設支援的,所以我們就不需要在來安裝,但是我們需要安裝使用者配置工具ipvsadm

yum -y install ipvsadm           # 分別在主從lvs上執行安裝ipvsadm

我們檢視lvs是否支援:

lsmod ¦ grep ip_vs          #
ip_vs                  78081  1
modprobe -l¦ grep ip_vs
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_dh.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_ftp.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_lblc.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_lblcr.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_lc.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_nq.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_rr.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_sed.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_sh.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_wlc.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_wrr.ko

本文介紹lvs的DR模式,首先部署keepalived.本博前面已經介紹如何來安裝keepalived.這裡就不在只簡單的貼一下步驟:

在主備伺服器上部署keepalived(因為前面已經rpm包安裝了ipvsadm,所以就不需要重複安裝):

vi /etc/sysctl.conf
net.ipv4.ip_forward = 1 # 此引數改為1
sysctl -p # 使修改生效

安裝依賴:

yum  -y install openssl-devel
# 下載並安裝keepalived
wget http://www.keepalived.org/software/keepalived-1.1.19.tar.gz
tar -zxvf keepalived-1.1.19.tar.gz
cd keepalived-1.1.19
./configure --prefix=/ \            # 安裝在預設位置(配置檔案,二進位制檔案,啟動指令碼放到預設位置)
--mandir=/usr/local/share/man/ \
--with-kernel-dir=/usr/src/kernels/2.6.18-194.el5-i686/    # 需要核心的標頭檔案
make && make install

在主備lvs上安裝keepalived完畢後我們先來配置主lvs上的keepalived: 編輯配置檔案/etc/keepalived/keepalived.conf:

! Configuration File for keepalived

global_defs {
   notification_email {
    [email protected]                                   # 發生故障時傳送的郵箱
   }
   notification_email_from [email protected]             # 使用哪個郵箱傳送
   smtp_server linuxzen.com                                  # 發件伺服器
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state MASTER             # 標示為主lvs
    interface eth0           # HA檢測埠
    virtual_router_id 51     # 主備的virtual_router_id 必須相同
    priority 100             # 優先順序,備lvs要比主lvs稍小
    advert_int 1             # VRRP Multicast 廣播週期秒數
    authentication {         # 定義認證
        auth_type PASS       # 認證方式為口令認證
        auth_pass 1111       # 定義口令
    }
    virtual_ipaddress {      # 定義vip
        192.168.3.100        # 多個vip可換行新增
    }
}

virtual_server 192.168.3.100 80 {
    delay_loop 6       # 每隔6秒檢視realserver狀態
    lb_algo wlc        # 排程演算法為加權最小連線數
    lb_kind DR         # lvs工作模式為DR(直接路由)模式
    nat_mask 255.255.255.0
    persistence_timeout 50  # 同一IP 的連線50秒內被分配到同一臺realserver(測試時建議改為0)
    protocol TCP            # 用TCP監測realserver的狀態

    real_server 192.168.3.3 80 {       # 定義realserver
        weight 3                       # 定義權重
        TCP_CHECK {  # 注意TCP_CHECK和{之間的空格,如果沒有的話只會新增第一個realserver
            connect_timeout 3          # 三秒無響應超時
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
    real_server 192.168.3.102 80 {
        weight 3
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}

配置備用lvs的keepalived,只需要將state MASTER 改為state BACKUP,降低priority 100 的值:

! Configuration File for keepalived

global_defs {
   notification_email {
    [email protected]                               # 發生故障時傳送的郵箱
   }
   notification_email_from [email protected]         # 使用哪個郵箱傳送
   smtp_server linuxzen.com                              # 發件伺服器
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state BACKUP              # 標示為備lvs
    interface eth0            # HA檢測埠
    virtual_router_id 51      # 主備的virtual_router_id 必須相同
    priority 99               # 優先順序,備lvs要比主lvs稍小
    advert_int 1              # VRRP Multicast 廣播週期秒數
    authentication {          # 定義認證
        auth_type PASS        # 認證方式為口令認證
        auth_pass 1111        # 定義口令
    }
    virtual_ipaddress {       # 定義vip
        192.168.3.100         # 多個vip可換行新增
    }
}

virtual_server 192.168.3.100 80 {
    delay_loop 6      # 每隔6秒檢視realserver狀態
    lb_algo wlc       # 排程演算法為加權最小連線數
    lb_kind DR        # lvs工作模式為DR(直接路由)模式
    nat_mask 255.255.255.0
    persistence_timeout 50  # 同一IP 的連線50秒內被分配到同一臺realserver
    protocol TCP            # 用TCP監測realserver的狀態

    real_server 192.168.3.3 80 {       # 定義realserver
        weight 3                       # 定義權重
        TCP_CHECK {    # 注意TCP_CHECK和{之間的空格,如果沒有的話只會新增第一個realserver
            connect_timeout 3          # 三秒無響應超時
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
    real_server 192.168.3.102 80 {
        weight 3
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}

由於使用keepalived就不需要使用指令碼來配置lvs排程器,但是這裡我們還是會給出一個指令碼內容,但我們不會用到這個指令碼:lvs已經內置於核心,配置命令是ipvsadm,所以lvs的一些操作是通過ipvsadm來控制.下面我們就編寫指令碼來實現lvs的DR模式:

編寫指令碼lvsdr:

我們把lvsvi /etc/init.d/lvsdr新增如下內容

#!/bin/sh
# 定義虛擬ip
VIP=192.168.3.100
# 定義realserver,並已逗號分開
RIPS=192.168.3.3,192.168.3.102 #,192.168.3.5,192.168.3.6

# 定義提供服務的埠
SERVICE=80

# 呼叫init.d指令碼的標準庫
. /etc/rc.d/init.d/functions
case $1 in
        start)
        echo "Start LVS of DR Mode"
        # lvs dr模式不需要路由轉發,但是keepalived需要
        #echo "0" > /proc/sys/net/ipv4/ip_forward
        # 開啟icmp包重定向
        echo "1" > /proc/sys/net/ipv4/conf/all/send_redirects
        echo "1" > /proc/sys/net/ipv4/conf/default/send_redirects
        echo "1" > /proc/sys/net/ipv4/conf/eth0/send_redirects
        # 繫結虛擬ip
        ifconfig eth0:0 $VIP broadcast $VIP netmask 255.255.255.255 up
        route add -host $VIP dev eth0:0
        # 清除lvs規則
        ipvsadm -C
        # 新增一條虛擬伺服器記錄
    # -p指定一定的時間內將相同的客戶端分配到同一臺後端伺服器
    # 用於解決session的問題,測試時或有別的解決方案時建議去掉
        ipvsadm -A -t $VIP:$SERVICE -s wlc -p

        # 新增真實伺服器記錄
        for RIP in `echo $RIPS ¦sed  -e 's/,/\n/g'`
        do
                ipvsadm -a -t $VIP:$SERVICE -r $RIP:$SERVICE -g -w 1
        done
        # 設定tcp tcpfin  udp的超時連線值
        ipvsadm --set 30 120 300
        ipvsadm
        ;;

        stop)
        echo "Stop LVS DR"
        ifconfig eth0:0 down
        ipvsadm -C
        ;;
        *)
        echo "Usage:$0 {start ¦ stop}"
        exit 1
esac

編輯完畢儲存退出,然後給這個指令碼執行許可權:

chmod +x /etc/init.d/lvsdr

然後就可以通過service命令來啟動lvs dr模式

service lvsdr start

將這個指令碼分別放到主備lvs的/etc/init.d/下,賦予執行許可權. 我們真正需要的是realserver的指令碼,下面我們來編寫realserver指令碼,同樣放在/etc/init.d/下,編輯rs指令碼:

vi /etc/init.d/lvsrs
#!/bin/sh
VIP=192.168.3.100
. /etc/rc.d/init.d/functions
case $1 in
        start)
        echo "lo:0 port starting"
        # 為了相應lvs排程器轉發過來的包,需在本地lo介面上繫結vip
        ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
        # 限制arp請求
        echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore
        echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce
        echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce
        ;;
        stop)
        echo "lo:0 port closing"
        ifconfig lo:0 down
        echo "0" > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "0" > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce
        ;;
        *)
        echo "Usage: $0 {start ¦ stop}"
        exit 1
esac

給指令碼賦予執行許可權

chmod +x /etc/init.d/lvsrs

並將這個指令碼放到所有的realserver的/etc/init.d/下.下面開始測試:

先來確認下我們做的變動:主從lvs分別安裝keepalived,並且在/etc/init.d/下添加了lvsdr指令碼(不使用).

後端realserver分別在/etc/init.d/下添加了lvsrs指令碼.我們先測試keepalived:

首先在主排程器上啟動keepalived:

service keepalived start

檢視日誌檔案:

tail -50 /var/log/message
Mar 21 22:29:10 master kernel: device eth0 left promiscuous mode
Mar 21 22:29:10 master kernel: type=1700 audit(1332340150.598:12): dev=eth0 prom=0 old_prom=256 auid=4294967295 ses=4294967295
Apr 16 13:31:32 master Keepalived: Starting Keepalived v1.1.19 (04/16,2012)
Apr 16 13:31:32 master Keepalived_healthcheckers: Netlink reflector reports IP 192.168.3.101 added
Apr 16 13:31:32 master Keepalived_healthcheckers: Registering Kernel netlink reflector
Apr 16 13:31:32 master Keepalived_healthcheckers: Registering Kernel netlink command channel
Apr 16 13:31:32 master Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
Apr 16 13:31:32 master Keepalived_healthcheckers: Configuration is using : 8897 Bytes
Apr 16 13:31:32 master Keepalived_healthcheckers: Using LinkWatch kernel netlink reflector...
Apr 16 13:31:32 master Keepalived: Starting Healthcheck child process, pid=5369
Apr 16 13:31:32 master Keepalived: Starting VRRP child process, pid=5370
Apr 16 13:31:32 master Keepalived_vrrp: Netlink reflector reports IP 192.168.3.101 added
Apr 16 13:31:32 master Keepalived_vrrp: Registering Kernel netlink reflector
Apr 16 13:31:32 master Keepalived_vrrp: Registering Kernel netlink command channel
Apr 16 13:31:32 master Keepalived_vrrp: Registering gratutious ARP shared channel
Apr 16 13:31:32 master Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Apr 16 13:31:32 master Keepalived_vrrp: Configuration is using : 36512 Bytes
Apr 16 13:31:32 master Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Apr 16 13:31:32 master Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Apr 16 13:31:33 master Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Apr 16 13:31:34 master Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Apr 16 13:31:34 master Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Apr 16 13:31:34 master Keepalived_healthcheckers: Netlink reflector reports IP 192.168.3.100 added
Apr 16 13:31:34 master Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.3.100
Apr 16 13:31:34 master Keepalived_vrrp: Netlink reflector reports IP 192.168.3.100 added
Apr 16 13:31:39 master Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.3.100

然後在備用排程器上啟動keepalived然後檢視日誌:

Apr 16 13:33:35 slave Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Apr 16 13:33:35 slave Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]
Apr 16 13:33:35 slave Keepalived_healthcheckers: Netlink reflector reports IP 192.168.3.102 added
Apr 16 13:33:35 slave Keepalived_healthcheckers: Registering Kernel netlink reflector
Apr 16 13:33:35 slave Keepalived_healthcheckers: Registering Kernel netlink command channel
Apr 16 13:33:35 slave Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
Apr 16 13:33:35 slave Keepalived_healthcheckers: Configuration is using : 8895 Bytes
Apr 16 13:33:35 slave kernel: IPVS: [wlc] scheduler registered.
Apr 16 13:33:35 slave Keepalived_healthcheckers: Using LinkWatch kernel netlink reflector...

在主排程器上執行

service keepalived stop

檢視備用排程器日誌:

tail -20 /var/log/message
Apr 16 13:39:44 slave Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Apr 16 13:39:45 slave Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Apr 16 13:39:45 slave Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Apr 16 13:39:45 slave Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.3.100
Apr 16 13:39:45 slave Keepalived_vrrp: Netlink reflector reports IP 192.168.3.100 added
Apr 16 13:39:45 slave Keepalived_healthcheckers: Netlink reflector reports IP 192.168.3.100 added

我們看到keepalived已經成功切換.

然後我們使用ipvsadm命令檢視(在此之前要確認後端realserver已經啟動了web服務):

ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.3.100:http wlc
  -> 192.168.3.3:http             Route   3      0          0
  -> 192.168.3.102:http           Route   3      0          0

然後分別啟動後端realserver的lvsrs服務:

servie lvsrs start

然後瀏覽器訪問192.168.3.100,如果keepalived的persistence_timeout引數值為0,而且兩個後端realserver是不同的內容,重新整理就可以看到兩個不同的頁面交替.