corosync+pacemaker+drbd 實現mysql的高可用性
一、環境準備
1.操作系統
centos 6.4 （32位）系統要是雙網卡
2.配置各節點互相解析
node1:
[[email protected] ~]# uname -n
node1.test.com
[[email protected] ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.31 node1.test.com node1
192.168.1.32 node2.test.com node2
node2:
[[email protected] ~]# uname -n
node2.test.com
[[email protected] ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.31 node1.test.com node1
192.168.1.32 node2.test.com node2
3.配置各節點ssh互信
node1:
[[email protected] ~]# ssh-keygen -t rsa
[[email protected] ~]# ssh-copy-id -i .ssh/id_rsa.pub [email protected]
node2:
[[email protected] ~]# ssh-keygen -t rsa
[[email protected] ~]# ssh-copy-id -i .ssh/id_rsa.pub [email protected]
4.配置各節點時間同步
node1:
[[email protected] ~]#hwclock -s
node2:
[[email protected] ~]# hwclock -s
5.各節點關閉防火墻與SELinux
node1:
[[email protected] ~]# service iptables stop
[[email protected] ~]# chkconfig iptables off
[[email protected] ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
node2:
[[email protected] ~]# service iptables stop
[[email protected] ~]# chkconfig iptables off
[[email protected] ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
三、Corosync 安裝與配置
1.安裝Corosync
node1:
[[email protected] ~]# rpm -ivh corosync-1.4.1-17.el6_5.1.i686.rpm corosynclib-1.4.1-17.el6_5.1.i686.rpm libibverbs-1.1.7-1.el6.i686.rpm
librdmacm-1.0.17-1.el6.i686.rpm lm_sensors-libs-3.1.1-17.el6.i686.rpm net-snmp-libs-5.5-49.el6_5.4.i686.rpm openssl-1.0.1e-16.el6_5.15.i686.rpm
node2:
[[email protected] ~]# rpm -ivh corosync-1.4.1-17.el6_5.1.i686.rpm corosynclib-1.4.1-17.el6_5.1.i686.rpm libibverbs-1.1.7-1.el6.i686.rpm
librdmacm-1.0.17-1.el6.i686.rpm lm_sensors-libs-3.1.1-17.el6.i686.rpm net-snmp-libs-5.5-49.el6_5.4.i686.rpm openssl-1.0.1e-16.el6_5.15.i686.rpm
2.配置Corosync
[[email protected] ~]# cd /etc/corosync/
[[email protected] corosync]# ll
總用量 16
-rw-r--r-- 1 root root 445 5月 15 05:09 corosync.conf.example
-rw-r--r-- 1 root root 1084 5月 15 05:09 corosync.conf.example.udpu
drwxr-xr-x 2 root root 4096 5月 15 05:09 service.d
drwxr-xr-x 2 root root 4096 5月 15 05:09 uidgid.d
[[email protected] corosync]# cp corosync.conf.example corosync.conf
[[email protected] corosync]# vim corosync.conf
[[email protected] corosync]# cat corosync.conf
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
version: 2
secauth: on
threads: 0
interface {
ringnumber: 0
bindnetaddr: 192.168.188.0
mcastaddr: 226.94.10.10
mcastport: 5405
ttl: 1
}
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
to_syslog: no
logfile: /var/log/cluster/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
amf {
mode: disabled
}
service {
ver: 0
name: pacemaker
}
aisexec {
user: root
group: root
}
3.生成密鑰文件
[[email protected] corosync]# mv /dev/{random,random.bak}
[[email protected] corosync]# ln -s /dev/urandom /dev/random
[[email protected] corosync]# corosync-keygen
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Writing corosync key to /etc/corosync/authkey.
[[email protected] corosync]# ll
總用量 24
-r-------- 1 root root 128 8月 17 17:17 authkey
-rw-r--r-- 1 root root 544 8月 17 17:14 corosync.conf
-rw-r--r-- 1 root root 445 5月 15 05:09 corosync.conf.example
-rw-r--r-- 1 root root 1084 5月 15 05:09 corosync.conf.example.udpu
drwxr-xr-x 2 root root 4096 5月 15 05:09 service.d
drwxr-xr-x 2 root root 4096 5月 15 05:09 uidgid.d
4.將node1上配置文件復制到node2上
[[email protected] corosync]# scp -p authkey corosync.conf node2:/etc/corosync/
authkey 100% 128 0.1KB/s 00:00
corosync.conf 100% 542 0.5KB/s 00:00
好了，到這裏corosync配置完成，下面我們配置pacemaker
四、Pacemaker 安裝與配置
1.安裝pacemaker
node1:
[[email protected] ~]# rpm -ivh clusterlib-3.0.12.1-59.el6_5.3.i686.rpm libqb-0.16.0-2.el6.i686.rpm pacemaker-1.1.10-14.el6_5.3.i686.rpm pacemaker-cli-1.1.10-14.el6_5.3.i686.rpm pacemaker-cluster-libs-1.1.10-14.el6_5.3.i686.rpm pacemaker-libs-1.1.10-14.el6_5.3.i686.rpm perl-TimeDate-1.16-11.1.el6.noarch.rpm resource-agents-3.9.2-40.el6_5.10.i686.rpm
node2:
[[email protected] ~]# rpm -ivh clusterlib-3.0.12.1-59.el6_5.3.i686.rpm libqb-0.16.0-2.el6.i686.rpm pacemaker-1.1.10-14.el6_5.3.i686.rpm pacemaker-cli-1.1.10-14.el6_5.3.i686.rpm pacemaker-cluster-libs-1.1.10-14.el6_5.3.i686.rpm pacemaker-libs-1.1.10-14.el6_5.3.i686.rpm perl-TimeDate-1.16-11.1.el6.noarch.rpm resource-agents-3.9.2-40.el6_5.10.i686.rpm
2.安裝crmsh
node1:
[[email protected] ~]# rpm -ivh python-pssh-2.3.1-4.1.i686.rpm pssh-2.3.1-4.1.i686.rpm redhat-rpm-config-9.0.3-42.el6.noarch.rpm
crmsh-2.1-1.6.i686.rpm
[[email protected] ~]# crm
Cannot change active directory to /var/lib/pacemaker/cores/root: No such file or directory
crm(live)# help
This is crm shell, a Pacemaker command line interface.
Available commands:
cib manage shadow CIBs
resource resources management
configure CRM cluster configuration
node nodes management
options user preferences
history CRM cluster history
site Geo-cluster support
ra resource agents information center
status show cluster status
help,? show help (help topics for list of topics)
end,cd,up go back one level
quit,bye,exit exit the program
node2上和node1上的操作一樣。
[[email protected] ~]# crm
Cannot change active directory to /var/lib/pacemaker/cores/root: No such file or directory
crm(live)# help
This is crm shell, a Pacemaker command line interface.
Available commands:
cib manage shadow CIBs
resource resources management
configure CRM cluster configuration
node nodes management
options user preferences
history CRM cluster history
site Geo-cluster support
ra resource agents information center
status show cluster status
help,? show help (help topics for list of topics)
end,cd,up go back one level
quit,bye,exit exit the program
3.啟動corosync（註，在配置corosync時，將pacemaker整合進corosync中，corosync啟動的同時也會啟動
pacemaker）
[[email protected] ~]# ssh node2 "service corosync start"
Starting Corosync Cluster Engine (corosync): [確定]
[[email protected] ~]# service corosync start
Starting Corosync Cluster Engine (corosync): [確定]
4.查看啟動信息
(1).查看corosync引擎是否正常啟動
[[email protected] ~]# grep -e "Corosync Cluster Engine" -e "configuration file"
(2).查看初始化成員節點通知是否正常發出
(3).檢查啟動過程中是否有錯誤產生
(4).查看pacemaker是否正常啟動
5.查看集群狀態
五、DRBD 安裝與配置
1.安裝DRBD
node1:
編譯獲得的RPM包
drbd-8.4.3-2.el6.i686.rpm
drbd-bash-completion-8.4.3-2.el6.i686.rpm
drbd-heartbeat-8.4.3-2.el6.i686.rpm
drbd-km-2.6.32_358.el6.i686-8.4.3-2.el6.i686.rpm
drbd-pacemaker-8.4.3-2.el6.i686.rpm
drbd-udev-8.4.3-2.el6.i686.rpm
drbd-utils-8.4.3-2.el6.i686.rpm
drbd-xen-8.4.3-2.el6.i686.rpm
4）安裝drbd rpm包
[[email protected] ~]#rpm -ivh drbd以上開頭的所有包
查看模塊是否加載成功
[[email protected] ~]# lsmod | grep drbd
drbd 292307 0
libcrc32c 841 1 drbd
[[email protected] ~]#rpm -ivh drbd開頭的所有包
查看模塊是否加載成功
[[email protected] ~]# lsmod | grep drbd
drbd 292307 0
libcrc32c 841 1 drbd
2.配置DRBD
[[email protected] ~]# cat /etc/drbd.d/global_common.conf
global {
usage-count no; #讓linbit公司收集目前drbd的使用情況，yes為參加，我們這裏不參加設置為no
#minor-count dialog-refresh disable-ip-verification
}
common {
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-
reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-
reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh;
echo o > /proc/sysrq-trigger ; halt -f";
}
startup {
}
options {
}
disk {
on-io-error detach; #同步錯誤的做法是分離
}
net {
cram-hmac-alg "sha1"; #設置加密算法sha1
shared-secret "mydrbdlab"; #設置加密key
}
syncer{
rate 100M;
}
}
3.在node1和node2上新增磁盤如下圖
如上圖分一個5G大小的linux分區
4.增加資源
[[email protected] drbd.d]# cat web.res
resource web {
on node1.test.com {
device /dev/drbd0;
disk /dev/sda4;
address 192.168.188.31:7789;
meta-disk internal;
}
on node2.test.com {
device /dev/drbd0;
disk /dev/sda4;
address 192.168.188.32:7789;
meta-disk internal;
}
}
5.同步配置文件到node2’
[[email protected] drbd.d]# scp global_common.conf web.res node2:/etc/drbd.d/
6.node1與node2上初始化資源
node1:
[[email protected] ~]# drbdadm create-md web
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
node2:
[[email protected] ~]# drbdadm create-md web
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
如果有其他錯誤信息執行dd if=/dev/zero bs=1M count=1 of=/dev/sda4，在執行即可。
7.啟動DRBD
node1:
[[email protected] ~]# service drbd start
node2:
[[email protected] ~]# service drbd start
8.查看一下狀態
node1:
[[email protected] ~]# drbd-overview
0:web/0Connected Secondary/SecondaryInconsistent/InconsistentC r-----
node2:
[[email protected] ~]# drbd-overview
0:web/0Connected Secondary/SecondaryInconsistent/InconsistentC r-----
9.設置node1為主節點
[[email protected] ~]# drbdadm -- --overwrite-data-of-peer primary web
[[email protected] ~]# drbd-overview
0:web/0 Connected Primary/Secondary UpToDate/UpToDate C r-----
10.格式化並掛載
[[email protected] ~]# mkfs.ext4 /dev/drbd0
[[email protected] ~]# mkdir /mydata
[[email protected] ~]# mount /dev/drbd0 /mydata/
[[email protected] ~]# mount
/dev/sda2on / typeext4 (rw)
proc on /proctypeproc (rw)
sysfs on /systypesysfs (rw)
devpts on /dev/ptstypedevpts (rw,gid=5,mode=620)
tmpfs on /dev/shmtypetmpfs (rw)
/dev/sda1on /boottypeext4 (rw)
/dev/sda4on /mydatatypeext4 (rw)
none on /proc/sys/fs/binfmt_misctypebinfmt_misc (rw)
/dev/drbd0on /drbdtypeext4 (rw)
[[email protected] ~]# cd /mydata/
[[email protected] mydata]# cp /etc/inittab /mydata/
[[email protected] mydata]# ll
總用量 20
-rw-r--r-- 1 root root 884 8月 17 13:50 inittab
drwx------ 2 root root 16384 8月 17 13:49 lost+found
[[email protected] mydata]# mkdir data
[[email protected] ~]# ll /mydata/
總用量 20
drwxr-xr-x 4 mysql mysql 4096 10鏈10 21:31 data
-rw-r--r-- 1 root root 884 8月 17 13:50 inittab
drwx------ 2 root root 16384 8月 17 13:49 lost+found
11.設置node2為主節點
[[email protected] ~]# umount /mydata/
[[email protected] ~]# drbdadm secondary web
[[email protected] ~]# drbd-overview
[[email protected] ~]# drbdadm primary web
[[email protected] ~]# drbd-overview
0:web/0Connected Primary/SecondaryUpToDate/UpToDateC r-----
[[email protected] ~]# mkdir /mydata
[[email protected] ~]# mount /dev/drbd0 /mydata/
[[email protected] ~]# ll /mydata/
總用量 20
drwxr-xr-x 4 mysql mysql 4096 10鏈10 21:31 data
-rw-r--r-- 1 root root 884 8月 17 13:50 inittab
drwx------ 2 root root 16384 8月 17 13:49 lost+found
五、數據庫配置
1.把一設為主節點：
[[email protected] ~]# umount /mydata/
[[email protected] ~]# drbdadm secondary web
[[email protected] ~]# drbd-overview
[[email protected] ~]# drbdadm primary web
[[email protected] ~]# drbd-overview
[[email protected] ~]# mount /dev/drbd0 /mydata/
0:web/0Connected Primary/SecondaryUpToDate/UpToDateC r-----
安裝數據庫：yum install -y mysql mysql-server
2.data目錄為mysql的數據存放目錄，故改變其屬主屬組：
# chown -R mysql.mysql /mydata/data/
3.mysql配置文件
# vim /etc/my.cnf
datadir = /mydata/data //數據存放路徑
# scp /etc/my.cnf node2:/etc/
4.啟動服務測試
# service mysqld start
#查看啟動數據目錄文件
[[email protected] data]# pwd
/mydata/data
[[email protected] data]# ls
ibdata1 ib_logfile0 ib_logfile1 mysql test
5.關閉mysqld服務，並保證開機不自動啟動
# service mysqld stop && chkconfig mysqld off
6.node2上不用初始化了，直接掛載
[[email protected] ~]# umount /mydata/
[[email protected] ~]# drbdadm secondary web
[[email protected] ~]# drbd-overview
[[email protected] ~]# drbdadm primary web
[[email protected] ~]# drbd-overview
0:web/0Connected Primary/SecondaryUpToDate/UpToDateC r-----
[[email protected] ~]# mkdir /mydata
[[email protected] ~]# mount /dev/drbd0 /mydata/
[[email protected] data]# pwd
/mydata/data
[[email protected] data]# ls
ibdata1 ib_logfile0 ib_logfile1 mysql test
7.關閉mysqld服務，並保證開機不自動啟動
# service mysqld stop && chkconfig mysqld off
# umount /mydata
六、集群配置
1.corosync默認啟用了stonith，而當前集群並沒有相應的stonith設備，因此此默認配置目前尚不可用，這
可以通過如下命令先禁用stonith
crm configure property stonith-enabled=false
2.對於雙節點的集群來說，我們要配置此選項來忽略quorum，即這時候票數不起作用，一個節點也能正常運
行
crm configure property no-quorum-policy=ignore
3.關閉drbd設為開機不啟動
[[email protected] ~]# service drbd stop && chkconfig drbd off
[[email protected] ~]# service drbd stop && chkconfig drbd off
4.drbd需要同時運行在兩個節點上，但只能有一個節點（primary/secondary模型）是Master，而另一個節
點為Slave；因此，它是一種比較特殊的集群資源，其資源類型為多狀態（Multi-state）clone類型，即主
機節點有Master和Slave之分，且要求服務剛啟動時兩個節點都處於slave狀態
primitive mysqldrbd ocf:linbit:drbd params drbd_resource="mysql" op monitor role="Master" interval="30s" op monitor role="Slave" interval="31s" op start timeout="240s" op stop timeout="100s"
5.創建master類型的資源，將mydrbd加入
ms MS_mysqldrbd mysqldrbd meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify="true"
6.我們實現將drbd設置自動掛載至/mydata目錄。此外，此自動掛載的集群資源需要運行於drbd服務的
Master節點上，並且只能在drbd服務將某節點設置為Primary以後方可啟動
primitive MysqlFS ocf:heartbeat:Filesystem params device="/dev/drbd0" directory="/mydata" fstype="ext4" op start timeout="60s" op stop timeout="60s"
7.mysql資源的定義
primitive myip ocf:heartbeat:IPaddr params ip=192.168.1.100
primitive mysqlserver lsb:mysqld
8.設置約束次序
colocation MysqlFS_with_mysqldrbd inf: MysqlFS MS_mysqldrbd:Master myip mysqlserver
order MysqlFS_after_mysqldrbd inf: MS_mysqldrbd:promote MysqlFS:start
order myip_after_MysqlFS mandatory: MysqlFS myip
order mysqlserver_after_myip mandatory: myip mysqlserver
提交查看狀態：
七、常見腦列問題
1.查看主服務器
[[email protected] ~]# service drbd status
drbd driver loaded OK; device status:version: 8.4.3 (api:1/proto:86-101)GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by [email protected], 2013-11-03 00:03:40m:res cs ro ds p mounted fstype1:web StandAlone Primary/Unknown UpToDate/DUnknown r-----
2.查看備服務器
[[email protected] ~]# service drbd status
drbd driver loaded OK; device status:version: 8.4.3 (api:1/proto:86-101)GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by [email protected], 2013-11-03 00:03:40m:res cs ro ds p mounted fstype1:web StandAlone Secondary/Unknown UpToDate/DUnknown r-----
解決方法：
在備服務器操作：其中web是資源名
[[email protected] ~]# drbdadm secondary web
[[email protected] ~]# drbdadm --discard-my-data connect web
在主服務器操作
[[email protected] ~]# drbdadm connect web
[[email protected] ~]# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by [email protected],2013-11-03 00:03:40 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----- ns:6852 nr:0 dw:264460 dr:8393508al:39 bm:512 lo:0 pe:2 ua:0 ap:0 ep:1 wo:d oos:257728[>....................] sync‘ed: 4.7% (257728/264412)K finish: 0:03:47speed: 1,112 (1,112) K/sec
備機上查看：DRBD恢復正常
[[email protected] ~]#service drbd status
drbd driver loaded OK; device status:version: 8.4.3 (api:1/proto:86-101)GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by [email protected], 2013-11-03 00:03:40m:res cs ro ds p mounted fstype1:web Connected Secondary/Primary UpToDate/UpToDate C
本文出自 “Linux_woniu” 博客，請務必保留此出處http://llu1314.blog.51cto.com/5925801/1965319
corosync+pacemaker+drbd 實現mysql的高可用性