前言

對於訪問量較大的網站來說，隨著流量的增加單臺伺服器已經無法處理所有的請求，這時候需要多臺伺服器對大量的請求進行分流處理，即負載均衡。而如果實現負載均衡，必須在網站的入口部署伺服器（不只是一臺）對這些請求進行分發，這臺伺服器即反向代理。由於反向代理伺服器是網站的入口，其負載壓力大且易遭到攻擊，存在單點故障的風險，所以我們需要一個高可用的方案來實現當一臺反向代理伺服器宕機的時候，另一臺伺服器會自動接管服務。基於以上要求，我們使用HAProxy，KeepAlived來構建高可用的反向代理系統。

介紹

HAProxy是高效能的代理伺服器，其可以提供7層和4層代理，具有healthcheck，負載均衡等多種特性，效能卓越，包括Twitter，Reddit，StackOverflow，GitHub在內的多家知名網際網路公司在

KeepAlived是一個高可用方案，通過VIP(即虛擬IP)和心跳檢測來實現高可用。其原理是存在一組（兩臺）伺服器，分別賦予Master,Backup兩個角色，預設情況下Master會繫結VIP到自己的網絡卡上，對外提供服務。Master,Backup會在一定的時間間隔向對方傳送心跳資料包來檢測對方的狀態，這個時間間隔一般為2秒鐘，如果Backup發現Master宕機，那麼Backup會發送ARP包到閘道器，把VIP繫結到自己的網絡卡，此時Backup對外提供服務，實現自動化的故障轉移，當Master恢復的時候會重新接管服務。

環境

OS: CentOS Linux release 6.0 (Final) 2.6.32-71.29.1.el6.x86_64 
HAProxy: 1.4.18 
KeepAlived: 1.2.2 
VIP: 192.168.1.99 
M: 192.168.1.222 
S: 192.168.1.189

架構

                    192.168.1.99
             +-----------VIP----------+   
             |                        |
             |                        |
           Master                   Backup
        192.168.1.189            192.168.1.222
        +----------+             +----------+
        | HAProxy  |             | HAProxy  |
        |keepalived|             |keepalived|
        +----------+             +----------+
             |  
             v  
    +--------+---------+ 
    |        |         |
    |        |         |
    v        v         v
+------+  +------+  +------+
| WEB1 |  | WEB2 |  | WEB3 |
+------+  +------+  +------+
 

安裝HAProxy

安裝pcre

$ yum install pcre
$ wget http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.18.tar.gz
$ tar -zxvf haproxy-1.4.18.tar.gz
$ cd haproxy-1.4.18

注意編譯引數： 
TARGET是指自己系統的核心版本 ARCH指定系統是32位還是64位 
CPU=native: use the build machine's specific processor optimizations 
更多編譯引數內容見原始碼中的README 
$ make TARGET=linux26 ARCH=x86_64 USE_PCRE=1 CPU=native $ make install

配置檔案 /etc/haproxy.cfg

global
    log 127.0.0.1   local3
    maxconn 20000   
    uid 535  #uid和gid按照實際情況進行配置
    gid 520  
    chroot /var/chroot/haproxy
    daemon 
    nbproc 1 

defaults
   log     127.0.0.1       local3
   mode    http            
   option  httplog
   option  httpclose
   option  dontlognull
   option  forwardfor
   retries 2
   balance roundrobin 
   stats   uri     /haproxy-stats
   contimeout      5000
   clitimeout      50000
   srvtimeout      50000

frontend http-in
        bind *:80 
        default_backend pool1

backend pool1
        option httpchk HEAD / HTTP/1.0
        stats refresh 2
        server WEB1 192.168.1.189:81 weight 3 maxconn 10000 check 
        server WEB2 192.168.1.222:81 weight 3 maxconn 10000 check

檢視HAProxy的狀態：http://192.168.1.99/haproxy-stats，這個頁面會顯示HAProxy本身以及後端伺服器的狀態。

日誌

haproxy會把日誌記錄傳送到syslog server(CentOS6下是rsyslogd，UDP514埠)， 編輯/etc/rsyslog.conf檔案，新增如下內容：

$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 127.0.0.1
local3.*                /var/log/haproxy.log

重啟rsyslog

$ /etc/init.d/rsyslog restart

自動輪轉日誌，編輯/etc/logrotate.d/haproxy.cfg，新增如下內容：

/var/log/haproxy.log
{
    rotate 4
    daily
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
    reload rsyslog > /dev/null 2>&1 || true
    endscript
}

啟動指令碼

$ wget -O haproxy https://raw.github.com/gist/3665034/4125bd5b81977a72e5eec30650fb21f3034782a0/haproxy-init.d 
$ cp haproxy /etc/init.d/haproxy
$ chmod +x /etc/init.d/haproxy
#使用方式
$ /etc/init.d/haproxy start|stop|restart

安裝KeepAlived

安裝依賴庫

$ yum install popt popt-devel

安裝KeepAlived

$ wget http://www.keepalived.org/software/keepalived-1.2.2.tar.gz
$ tar -zxvf keepalived-1.2.2.tar.gz
$ cd keepalived-1.2.2
$ ./configure --prefix=/usr/local/keepalived
$ make && make install

$ cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived
$ cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
$ cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
$ mkdir -p /etc/keepalived/
$ cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf 
$ chmod +x /etc/init.d/keepalived

使用方式

$ /etc/init.d/keepalived start|stop|restart

Master伺服器上的配置 /etc/keepalived/keepalived.conf

global_defs {

   notification_email {
       [email protected]
   }

   notification_email_from [email protected]
   smtp_server 192.168.x.x
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

#監測haproxy程序狀態，每2秒執行一次
vrrp_script chk_haproxy {
    script "/usr/local/keepalived/chk_haproxy.sh"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    state MASTER #標示狀態為MASTER
    interface eth0
    virtual_router_id 51
    priority 101   #MASTER權重要高於BACKUP
    advert_int 1
    mcast_src_ip 192.168.1.189 #Master伺服器IP

    authentication {
        auth_type PASS #主從伺服器驗證方式
        auth_pass 1111
    }

    track_script {
        chk_haproxy #監測haproxy程序狀態
    }

    #VIP
    virtual_ipaddress {
        192.168.1.99 #虛擬IP
    }
}

Bakcup伺服器上的配置 /etc/keepalived/keepalived.conf

global_defs {
   notification_email {
   [email protected]
   }

   notification_email_from [email protected]
   smtp_server 192.168.x.x
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

#監測haproxy程序狀態，每2秒執行一次
vrrp_script chk_haproxy {
    script "/usr/local/keepalived/chk_haproxy.sh"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    state BACKUP #狀態為BACKUP
    interface eth0
    virtual_router_id 51
    priority 100  #權重要低於MASTER
    advert_int 1
    mcast_src_ip 192.168.1.222 #Backup伺服器的IP

    authentication {
        auth_type PASS
        auth_pass 1111
    }

    track_script {
        chk_haproxy #監測haproxy程序狀態
    }

    #VIP
    virtual_ipaddress {
        192.168.1.99 #虛擬IP
    }
}

chk_haproxy.sh內容

#!/bin/bash
#
# author: weizhifeng
# description: 
# 定時檢視haproxy是否存在，如果不存在則啟動haproxy，
# 如果啟動失敗，則停止keepalived
# 
status=$(ps aux|grep haproxy | grep -v grep | grep -v bash | wc -l)
if [ "${status}" = "0" ]; then
    /etc/init.d/haproxy start

    status2=$(ps aux|grep haproxy | grep -v grep | grep -v bash |wc -l)

    if [ "${status2}" = "0"  ]; then
            /etc/init.d/keepalived stop
    fi
fi

高可用測試

1. 在Master上停止keepalived，檢視系統日誌，發現MASTER釋放了VIP

    $ /etc/init.d/keepalived stop
    $ tail -f /var/log/message
    Keepalived: Terminating on signal Keepalived: Stopping Keepalived v1.2.2 (11/03,2011) 
    Keepalived_vrrp: Terminating VRRP child process on signal 
    Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
   

2. 在Backup上檢視系統日誌，發現Backup已經進入MASTER角色，並且綁定了VIP 192.168.1.99

    $ tail -f /var/log/message
    Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
    Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs
    Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.99 #在Backup上檢視VIP是否已經繫結
   

3. 在Master上重新啟動keepalived，檢視系統日誌，發現重新獲得MASTER角色，並且繫結VIP 192.168.1.99

    $ /etc/init.d/keepalived start
    $ tail -f /var/log/message
    Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
    Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
    Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
    Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.99
   

4. 在Backup上檢視系統日誌，發現其重新回到BACKUP角色，並且釋放VIP

    $ tail -f /var/log/message
    Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
    Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
    Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
   

併發測試

我們使用webbench來對HAProxy進行併發測試

$ yum install ctags
$ wget http://home.tiscali.cz/~cz210552/distfiles/webbench-1.5.tar.gz
$ tar -zxvf webbench-1.5.tar.gz
$ cd webbench-1.5
$ make 
$ mkdir -p /usr/local/man && make install

測試環境： 
CPU：Intel 雙核 x86_64 主頻3191MHZ 
Mem：2G
修改php-fpm.conf，設定PHP-FPM spawn的程序數量為100：

pm.start_servers = 100
pm.max_spare_servers = 100

測試方法：

$ webbench -c 100 -t 3000 http://192.168.1.99/check.txt
$ webbench -c 100 -t 3000 http://192.168.1.99/test.php

測試結果：

併發訪問txt檔案，HAProxy的session數量為10000左右，這說明HAProxy能夠hold住10000個併發連線；併發訪問php檔案，HAProxy的session峰值為200左右，接近於後端PHP的併發處理能力(100x2)。

參考：