Linux系統中採用Atlas+Keepalived實現MySQL讀寫分離、負載均衡
==========================================================================================
一、基礎介紹
==========================================================================================
1、背景描述
目前我們的高可用DB的代理層採用的是360開源的Atlas，從上線以來，已穩定執行2個多月。無論是從效能上，還是穩定性上，相比其他開源元件（amoeba、cobar、MaxScale、MySQL-Proxy等），還是很出色的。
當初我們之所以選擇Atlas，主要看中它有以下優點：
(1)、基於mysql-proxy-0.8.2進行修改，程式碼完全開源；
(2)、比較輕量級，部署配置也比較簡單；
(3)、支援DB讀寫分離；
(4)、支援從DB讀負載均衡，並自動剔除故障從DB；
(5)、支援平滑上下線DB；
(6)、具備較好的安全機制（IP過濾、賬號認證）；
(7)、版本更新、問題跟進、交流圈子都比較活躍。
在測試期間以及線上問題排查過程中，得到了360 Atlas作者朱超的熱心解答，在此表示感謝。有關更多Atlas的介紹，我就不一一例舉，可以參考以下連結：
2、總體架構圖
3、系統環境
CentOS 6.3 x86_64
==========================================================================================
二、安裝部署
==========================================================================================
1、需注意的地方
(1)、本次安裝不使用系統預設的glib庫，之前的yum安裝只是為了先解決依賴庫的問題；
(2)、LUA庫的版本不能太高，為5.1.x即可；
(3)、glib庫的版本也不能太高，為glib-2.32.x
(4)、對於編譯不成功的情況，注意檢視下面的說明。
2、GLIB依賴的基礎庫安裝
# yum -y install *glib*
3、LUA庫安裝
# tar xvzf ncurses-5.9.tar.gz
# cd ncurses-5.9
# ./configure --prefix=/usr/local
# make && make install
# tar xvzf readline-6.2.tar.gz
# cd readline-6.2
# ./configure --prefix=/usr/local
# make && make install
# tar xvzf lua-5.1.5.tar.gz
# cd lua-5.1.5
# make linux install
注意：
修改當前目錄下的“Makefile”中的INSTALL_TOP= /usr/local為INSTALL_TOP= /usr/local/lua
主要是為了避免與系統自帶的lua庫發生衝突的可能
在“src/Makefile”檔案中加入“-lncurses”，完整內容如下：
linux:
$(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_LINUX" SYSLIBS="-Wl,-E -ldl-lncurses -lreadline"
4、GLIB庫安裝
# tar xvzf libffi-3.0.13.tar.gz
# cd libffi-3.0.13
# ./configure --prefix=/usr/local
# make && make install
# tar xvzf libiconv-1.14.tar.gz
# cd libiconv-1.14
# ./configure --prefix=/usr/local
# make && make install
# tar xvzf xz-5.0.5.tar.gz
# cd xz-5.0.5
# ./configure --prefix=/usr/local
# make && make install
# /sbin/ldconfig
# xz -d glib-2.32.4.tar.xz
# tar -xvf glib-2.32.4.tar
# cd glib-2.32.4
# ./configure --prefix=/usr/local/glib-2.32.4 \
--with-libiconv=/usr/local \
LIBFFI_CFLAGS="-I/usr/local/include" \
LIBFFI_LIBS="-L/usr/local/lib -lffi"
# make && make install
注意：編譯報錯處理
(1)、configure階段
# vim ./glib/gconvert.c
註釋掉第26、28行的內容
註釋掉從61行到67行的內容
# vim ./configure
在7880行之上新增如下內容：
found_iconv=yes
(2)、make階段
# ln -s /usr/local/lib/libffi-3.0.13/include/ffi.h /usr/local/include
# ln -s /usr/local/lib/libffi-3.0.13/include/ffitarget.h /usr/local/include
glib庫需要安裝在單獨的目錄“/usr/local/glib-2.32.4”，也是為了避免與系統自帶的glib庫發生衝突的可能
5、Atlas安裝
(1)、其他基礎元件安裝
# tar xvzf libevent-2.0.21-stable.tar.gz
# cd libevent-2.0.21-stable
# ./configure --prefix=/usr/local
# make && make install
# tar xvzf openssl-1.0.1h.tar.gz
# cd openssl-1.0.1h
# ./config shared --prefix=/usr/local
# make && make install
(2)、MySQL安裝（無需啟動）
# tar -xvzf cmake-2.8.10.2.tar.gz
# cd cmake-2.8.10.2
# ./bootstrap --prefix=/usr/local
# gmake --jobs=`grep processor /proc/cpuinfo | wc -l`
# gmake install
# tar -xvzf mysql-5.5.24.tar.gz
# cd mysql-5.5.24
# rm-f CMakeCache.txt
# cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \
-DMYSQL_UNIX_ADDR=/var/run/mysql/mysql.sock \
-DDEFAULT_CHARSET=utf8 \
-DDEFAULT_COLLATION=utf8_general_ci \
-DEXTRA_CHARSETS=all \
-DWITH_MYISAM_STORAGE_ENGINE=1 \
-DWITH_INNOBASE_STORAGE_ENGINE=1 \
-DWITH_READLINE=1 \
-DENABLED_LOCAL_INFILE=1 \
-DWITH_EMBEDDED_SERVER=1 \
-DMYSQL_DATADIR=/data/dbdata/data \
-DMYSQL_TCP_PORT=3306
# make --jobs=`grep processor /proc/cpuinfo | wc -l`
# make install
(3)、DB中介軟體安裝
# tar xvzf Atlas-2.2.1.tar.gz
# cd Atlas-2.2.1
# ./configure --prefix=/usr/local/mysql-proxy \
--with-lua=/usr/local/lua \
--with-mysql=/usr/local/mysql \
GLIB_CFLAGS="-I/usr/local/glib-2.32.4/include/glib-2.0" \
GLIB_LIBS="-L/usr/local/glib-2.32.4/lib/glib-2.0 -lglib-2.0" \
GMODULE_CFLAGS="-I/usr/local/glib-2.32.4/include" \
GMODULE_LIBS="-L/usr/local/glib-2.32.4/lib -lgmodule-2.0" \
GTHREAD_CFLAGS="-I/usr/local/glib-2.32.4/include" \
GTHREAD_LIBS="-L/usr/local/glib-2.32.4/lib -lgthread-2.0" \
LUA_CFLAGS="-I/usr/local/lua/include" \
LUA_LIBS="-L/usr/local/lua/lib -llua-5.1" \
CFLAGS="-DHAVE_LUA_H -O2" \
LDFLAGS="-L/usr/local/lib -L/usr/local/lib64 -lm -ldl -lcrypto"
# make && make install
注意：
編譯報錯處理
# ln -s /usr/local/glib-2.32.4/lib/glib-2.0/include/glibconfig.h /usr/local/glib-2.32.4/include/glib-2.0
# cd /usr/local
# mv mysql-proxy atlas-2.2.1 && ln -s atlas-2.2.1 mysql-proxy
6、DB中間層配置
(1)、主配置
# vim /usr/local/mysql-proxy/conf/mysql-proxy.cnf
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
[mysql-proxy]
admin-username = sysadmin
admin-password = [email protected]()
proxy-backend-addresses = 10.222.5.224:3306
proxy-read-only-backend-addresses = 10.240.95.107:3306,10.240.95.108:3306
pwds = health_check1:/iZxz+0GRoA=,health_check2:/iZxz+0GRoA=
daemon = true
keepalive = true
event-threads = 16
log-level = message
log-path = /usr/local/mysql-proxy/log
sql-log = ON
proxy-address = 0.0.0.0:3306
admin-address = 10.209.6.101:3307
charset = utf8
|
(2)、啟動指令碼
# vim /etc/init.d/mysql-proxy
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 |
#!/bin/sh
#
# mysql-proxy This script starts and stops the mysql-proxy daemon
#
# chkconfig: - 78 30
# processname: mysql-proxy
# description: mysql-proxy is a proxy daemon to mysql
# config: /usr/local/mysql-proxy/conf/mysql-proxy.cnf
# pidfile: /usr/local/mysql-proxy/log/mysql-proxy.pid
#
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON="/usr/local/mysql-proxy/bin/mysql-proxy"
CONFIGFILE="/usr/local/mysql-proxy/conf/mysql-proxy.cnf"
PIDFILE="/usr/local/mysql-proxy/log/mysql-proxy.pid"
LOCKFILE="/var/lock/subsys/mysql-proxy"
PROG=`basename $DAEMON`
RETVAL=0
start() {
echo -n $"Starting ${PROG}......"
[ -x $DAEMON ] || exit 5
[ -f $CONFIGFILE ] || exit 6
${DAEMON} --defaults-file=${CONFIGFILE} || echo -n "${PROG} already running"
RETVAL=$?
echo
[[ $RETVAL -eq 0 ]] && touch $LOCKFILE
return $RETVAL
}
stop() {
echo -n $"Stopping ${PROG}......"
if [[ `ps aux | grep bin/mysql-proxy | grep -v grep | wc -l` -gt 0 ]]; then
kill -TERM `ps -A -oppid,pid,cmd | grep bin/mysql-proxy | grep -v grep | awk '{print $2}'`
fi
RETVAL=$?
echo
[[ $RETVAL -eq 0 ]] && rm -f $LOCKFILE $PIDFILE
return $RETVAL
}
restart() {
stop
sleep 1
start
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
condrestart)
[[ -e $LOCKFILE ]] && restart
;;
*)
echo "Usage: $0 {start|stop|restart|condrestart}"
RETVAL=1
;;
esac
exit $RETVAL
|
# chmod +x /etc/init.d/mysql-proxy
# chmod 0660 /usr/local/mysql-proxy/conf/mysql-proxy.cnf
# service mysql-proxy start
# ps aux | grep mysql-prox[y]
7、Atlas高可用【Keepalived】環境安裝
# tar xvzf popt-1.14.tar.gz
# cd popt-1.14
# ./configure --prefix=/usr/local
# make && make install
# tar xvzf libnl-3.2.24.tar.gz
# cd libnl-3.2.24
# ./configure --prefix=/usr/local
# make && make install
# ln -s /usr/local/include/libnl3/netlink /usr/local/include
# /sbin/ldconfig
# tar xvzf keepalived-1.2.10.tar.gz
# cd keepalived-1.2.10
# ./configure --prefix=/usr/local/keepalived
# make && make install
# ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin
# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig
# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d
# chkconfig --add keepalived
8、Atlas高可用【Keepalived】配置
# mkdir –p /etc/keepalived /data/scripts
(1)、主節點配置
# vim /etc/keepalived/keepalived.conf
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id dbproxy1
}
vrrp_script chk_mysql_proxy_health {
script "/data/scripts/keepalived_check_mysql_proxy.sh"
interval 1
weight -2
}
vrrp_instance VI_1 {
state MASTER
interface eth1
virtual_router_id 51
priority 100
advert_int 1
smtp_alert
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.209.6.115
}
track_script {
chk_mysql_proxy_health
}
notify_master "/data/scripts/notify.sh master"
notify_bakcup "/data/scripts/notify.sh backup"
notify_fault "/data/scripts/notify.sh fault"
}
|
(2)、備用節點配置
# vim /etc/keepalived/keepalived.conf
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id dbproxy2
}
vrrp_script chk_mysql_proxy_health {
script "/data/scripts/keepalived_check_mysql_proxy.sh"
interval 1
weight -2
}
vrrp_instance VI_1 {
state BACKUP
interface eth1
virtual_router_id 51
priority 90
advert_int 1
smtp_alert
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.209.6.115
}
track_script {
chk_mysql_proxy_health
}
notify_master "/data/scripts/notify.sh master"
notify_bakcup "/data/scripts/notify.sh backup"
notify_fault "/data/scripts/notify.sh fault"
}
|
(3)、VIP切換通知指令碼
# vim /data/scripts/notify.sh
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
#!/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin
KEEPALIVE_CONF="/etc/keepalived/keepalived.conf"
VIP=`grep -A 1 virtual_ipaddress ${KEEPALIVE_CONF} | tail -1 | sed 's/\t//g; s/ //g'`
ETH1_ADDR=`/sbin/ifconfig eth1 | awk '/inet addr:/{print $2}' | awk -F: '{print $2}'`
MONITOR="/usr/local/oms/agent/alarm/BusMonitorAgent"
TOKEN="ha_monitor"
function notify() {
TITLE="$ETH1_ADDR to be $1: $VIP floating"
CONTENT="vrrp transition, $ETH1_ADDR changed to be $1"
${MONITOR} -c 2 -f ${TOKEN} -t "${TITLE}" -i "${CONTENT}"
}
case "$1" in
master)
notify master
exit 0
;;
backup)
notify backup
exit 0
;;
fault)
notify fault
exit 0
;;
*)
echo 'Usage: `basename $0` {master|backup|fault}'
exit 1
;;
esac
|
(4)、DB中間層程序檢查指令碼
# vim /data/scripts/keepalived_check_mysql_proxy.sh
| 1 2 3 4 5 6 7 |
#!/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin
if [[ `pgrep mysql-proxy | wc -l` -eq 0 ]]; then
/sbin/service mysql-proxy start && sleep 5
[[ -z `pgrep mysql-proxy` ]] && /sbin/service keepalived stop
fi
|
# chmod +x /data/scripts/*.sh
# service keepalived start
# ip addr show eth1
# ps aux | grep keepalive[d]
==========================================================================================
三、其他設定
==========================================================================================
1、Atlas服務監控
# vim /usr/local/mysql-proxy/bin/check_service.sh
|
1
2
3
|