MySQL高可用介紹4-Mysql主從同步+DRBD+heartbeat
阿新 • • 發佈：2019-02-19
介紹
MySQL一主多從，可以保證資料庫叢集高可用，由於是多從，所以掛掉一個從庫對叢集沒影響；如果掛了一個主庫我們可以把從庫轉換成主庫，但是由於網路延遲問題主從角色切換會造成資料丟失，為了解決這個問題，解決主機單節點執行使用heartbeat做主節點高可用，主為了保證2個主節點資料同步可以使用drbd做主節點資料同步。
多主可以使用lvs來實現，如果是多主多從需要Mysql+DRBD+heartbeat來實現，架構如下（拷貝老男孩的一張架構圖）：
注意事項：
1.db-1-2熱備主，跟db-1-1通過heartbeat做高可用，brdb做資料同步;
2.正常情況下，db-1-1對外提供寫，從對外提供讀；
3.mysql slave可以做一個lb，通過vip跟db-1-1做資料同步；
4.web server提供讀寫分離機制，讀分發到mysql slave，寫分發到db-1-1;
5.db-1-1、db-1-2資料同步使用專有鏈路，heartbeat也必須使用專有鏈路，也可以使用串列埠線來完成，不建議使用同一鏈路。
db-1-1故障之後，有db-1-2接管vip及資料庫資源；
mysql slave由於是使用vip跟主庫同步，所以故障發生之後可以自動跟db-1-2做資料同步；
部署
Mysql支援多例項，drbd也支援多分割槽同步。如果節點流量比較大可以使用網絡卡繫結。
我們可以把mysql資料庫和資料檔案建立在/data目錄，這樣兩個主節點都可以同步共享。
網絡卡及地址規劃：
主庫
node1:
eth0 10.37.2.100 //管理地址
eth1 192.168.52.106//心跳線
vip 10.37.2.240 //vip
node2：
eth0 10.37 主庫每臺機器至少兩塊硬碟，容量大小一致。
兩臺主庫安裝drbd和heartbeat，安裝步驟及配置請參考前三個部落格。
heartbeat管理drbd
安裝完成之後，heartbeat我們還需要做一個配置，由heartbeat管理drbd。
兩個主節點建立/data目錄
mkdir -p /data
主節點格式化drbd0
mkfs -t ext3 /dev/drbd0
主節點掛載
mount /dev/drbd0 /data
heartbeat禁止系統自啟動
[root@node1 data]# chkconfig heartbeat offvi /etc/ha.d/haresources (注意不要換行，兩個主節點都需要配置)
##heartbeat drbd配置
node1 IPaddr::10.37.2.240/24/eth0 drbddisk::data Filesystem::/dev/drbd0::/data::ext3
##把drbd0裝置以ext3方式掛載到/data 上。
##heartbeat nfs 配置
node1 IPaddr::10.37.2.240/24/eth0 drbddisk::data Filesystem::/dev/drbd0::/data::ext3 rsdata啟動heartbeat，檢視vip和drbd是否正常啟動.
[root@node1 /]# /etc/init.d/heartbeat start
Starting High-Availability services: INFO: Resource is stopped
INFO: Resource is stopped
Done.
[root@node1 data]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:0c:29:50:f0:77 brd ff:ff:ff:ff:ff:ff
inet 10.37.2.100/24 brd 10.37.2.255 scope global eth0
inet 10.37.2.240/24 brd 10.37.2.255 scope global secondary eth0
inet6 fe80::20c:29ff:fe50:f077/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:0c:29:50:f0:81 brd ff:ff:ff:ff:ff:ff
inet 192.168.52.100/24 brd 192.168.52.255 scope global eth1
inet6 fe80::20c:29ff:fe50:f081/64 scope link
valid_lft forever preferred_lft forever
[root@node1 data]# cat /proc/drbd
version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by root@node1, 2017-08-30 11:34:17
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:12 nr:8 dw:172 dr:8549339 al:3 bm:526 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@node1 data]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_ams-lv_root
36G 5.1G 29G 15% /
tmpfs 1.6G 224K 1.6G 1% /dev/shm
/dev/sda1 477M 32M 420M 8% /boot
/dev/sr0 4.4G 4.4G 0 100% /media/CentOS_6.6_Final
/dev/sr0 4.4G 4.4G 0 100% /mnt
/dev/drbd0 15G 166M 14G 2% /data
heartbeat啟動日誌:
ep 01 11:31:27 node1 heartbeat: [24937]: info: Pacemaker support: false
Sep 01 11:31:27 node1 heartbeat: [24937]: WARN: Logging daemon is disabled --enabling logging daemon is recommended
Sep 01 11:31:27 node1 heartbeat: [24937]: info: **************************
Sep 01 11:31:27 node1 heartbeat: [24937]: info: Configuration validated. Starting heartbeat 3.0.4
Sep 01 11:31:27 node1 heartbeat: [24938]: info: heartbeat: version 3.0.4
Sep 01 11:31:27 node1 heartbeat: [24938]: info: Heartbeat generation: 1503900466
Sep 01 11:31:27 node1 heartbeat: [24938]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1
Sep 01 11:31:27 node1 heartbeat: [24938]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1
Sep 01 11:31:27 node1 heartbeat: [24938]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0
Sep 01 11:31:27 node1 heartbeat: [24938]: info: glib: ucast: bound send socket to device: eth0
Sep 01 11:31:27 node1 heartbeat: [24938]: info: glib: ucast: set SO_REUSEPORT(w)
Sep 01 11:31:27 node1 heartbeat: [24938]: info: glib: ucast: bound receive socket to device: eth0
Sep 01 11:31:27 node1 heartbeat: [24938]: info: glib: ucast: set SO_REUSEPORT(w)
Sep 01 11:31:27 node1 heartbeat: [24938]: info: glib: ucast: started on port 694 interface eth0 to 192.168.52.103
Sep 01 11:31:27 node1 heartbeat: [24938]: info: glib: ping heartbeat started.
Sep 01 11:31:28 node1 heartbeat: [24938]: info: G_main_add_TriggerHandler: Added signal manual handler
Sep 01 11:31:28 node1 heartbeat: [24938]: info: G_main_add_TriggerHandler: Added signal manual handler
Sep 01 11:31:28 node1 heartbeat: [24938]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Sep 01 11:31:28 node1 heartbeat: [24938]: info: Local status now set to: 'up'
Sep 01 11:31:28 node1 heartbeat: [24938]: info: Link node1:eth1 up.
Sep 01 11:32:26 node1 heartbeat: [24938]: info: Link node2:eth1 up.
Sep 01 11:32:26 node1 heartbeat: [24938]: info: Status update for node node2: status up
harc(default)[24951]: 2017/09/01_11:32:26 info: Running /etc/ha.d//rc.d/status status
Sep 01 11:33:28 node1 heartbeat: [24938]: WARN: node 192.168.52.103: is dead
harc(default)[24971]: 2017/09/01_11:33:28 info: Running /etc/ha.d//rc.d/status status
Sep 01 11:33:29 node1 heartbeat: [24938]: info: Comm_now_up(): updating status to active
Sep 01 11:33:29 node1 heartbeat: [24938]: info: Local status now set to: 'active'
Sep 01 11:34:27 node1 heartbeat: [24938]: info: Status update for node node2: status active
harc(default)[25001]: 2017/09/01_11:34:27 info: Running /etc/ha.d//rc.d/status status
Sep 01 11:34:37 node1 heartbeat: [24938]: info: remote resource transition completed.
Sep 01 11:34:37 node1 heartbeat: [24938]: info: remote resource transition completed.
Sep 01 11:34:37 node1 heartbeat: [24938]: info: Initial resource acquisition complete (T_RESOURCES(us))
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.37.2.240)[25055]: 2017/09/01_11:34:37 INFO: Resource is stopped
Sep 01 11:34:37 node1 heartbeat: [25019]: info: Local Resource acquisition completed.
harc(default)[25136]: 2017/09/01_11:34:37 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
ip-request-resp(default)[25136]: 2017/09/01_11:34:37 received ip-request-resp IPaddr::10.37.2.240/24/eth0 OK yes
ResourceManager(default)[25159]: 2017/09/01_11:34:37 info: Acquiring resource group: node1 IPaddr::10.37.2.240/24/eth0 drbddisk::data Filesystem::/dev/drbd0::/data::ext3
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.37.2.240)[25187]: 2017/09/01_11:34:38 INFO: Resource is stopped
ResourceManager(default)[25159]: 2017/09/01_11:34:38 info: Running /etc/ha.d/resource.d/IPaddr 10.37.2.240/24/eth0 start
IPaddr(IPaddr_10.37.2.240)[25310]: 2017/09/01_11:34:38 INFO: Adding inet address 10.37.2.240/24 with broadcast address 10.37.2.255 to device eth0
IPaddr(IPaddr_10.37.2.240)[25310]: 2017/09/01_11:34:38 INFO: Bringing device eth0 up
IPaddr(IPaddr_10.37.2.240)[25310]: 2017/09/01_11:34:38 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.37.2.240 eth0 10.37.2.240 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.37.2.240)[25284]: 2017/09/01_11:34:38 INFO: Success
ResourceManager(default)[25159]: 2017/09/01_11:34:38 info: Running /etc/ha.d/resource.d/drbddisk data start
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0)[25438]: 2017/09/01_11:34:38 INFO: Resource is stopped
ResourceManager(default)[25159]: 2017/09/01_11:34:38 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /data ext3 start
Filesystem(Filesystem_/dev/drbd0)[25528]: 2017/09/01_11:34:38 INFO: Running start for /dev/drbd0 on /data
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0)[25520]: 2017/09/01_11:34:38 INFO: Success
錯誤1：
ResourceManager(default)[21772]: 2017/09/02_12:32:34 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddisk
ResourceManager(default)[21772]: 2017/09/02_12:32:35 info: Retrying failed stop operation [drbddisk::data]
ResourceManager(default)[21772]: 2017/09/02_12:32:35 info: Running /etc/ha.d/resource.d/drbddisk data stop
ResourceManager(default)[21772]: 2017/09/02_12:32:37 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddisk
ResourceManager(default)[21772]: 2017/09/02_12:32:37 ERROR: Resource script for drbddisk::data probably not LSB-compliant.
ResourceManager(default)[21772]: 2017/09/02_12:32:37 WARN: it (drbddisk::data) MUST succeed on a stop when already stopped
ResourceManager(default)[21772]: 2017/09/02_12:32:37 WARN: Machine reboot narrowly avoided!
ResourceManager(default)[21772]: 2017/09/02_12:32:37 info: Running /etc/ha.d/resource.d/IPaddr 10.37.2.240/24/eth0 stop解決方案：檢視 ha-log日誌，發現heartbeat管理drbd資源一直報錯，可以先把drbd手動啟動起來。
這樣雖然可以正常啟動，但是沒有被heartbeat接管，可能會造成某一個節點宕機，另外一個節點的drbd會起不起來，最好的方法是讓heartbeat接管drbd。
以上可以看到，heartbeat正常啟動並正常接管資源（drbd,vip），drbd0正常掛載。
以上drbd跟heartbeat整合完成，下面開始整合mysql。
4臺伺服器安裝mysql，採用編譯安裝mysql-5.5.32
安裝cmake
安裝依賴包
yum -y install ncurses-devel bison gcc gcc-c++進到cmake解壓包目錄，編譯安裝
./configure
make
make install
編譯安裝mysql-5.5.32
cmake . -DCMAKE_INSTALL_PREFIX=/data/app/mysql \
-DMYSQL_DATADIR=/data/app/mysql/data \
-DMYSQL_UNIX_ADDR=/data/app/mysql/mysql.sock \
-DDEFAULT_CHARSET=utf8 \
-DDEFAULT_COLLATION=utf8_general_ci \
-DEXTRA_CHARSETS=gbk,gb2312,utf8,ascii \
-DENABLED_LOCAL_INFILE=on \
-DWITH_INNOBASE_STORAGE_ENGINE=1 \
-DWITH_FEDERATED_STORAGE_ENGINE=1 \
-DWITH_BLACKHOLE_STORAGE_ENGINE=1 \
-DWITHOUT_EXAMPLE_STORAGE_ENGINE=1 \
-DWITHOUT_PARTITION_STORAGE_ENGINE=1 \
-DWITH_FAST_MUTEXES=1 \
-DWITH_ZLIB=BUNDLED \
-DENABLED_LOCAL_INFILE=1 \
-DWITH_READLINE=1 \
-DWITH_EMBEDDED_SERVER=1 \
-DWITH_DEBUG=0錯誤：
CMake Error: The current CMakeCache.txt directory /data/app/mysql/CMakeCache.txt is different than the directory /home/mysql/mysql-5.5.32 where CMakeCache.txt was created. This may result in binaries being created in the wrong place. If you are not sure, reedit the CMakeCache.txt解決：
rm -rf CMakeCache.txt接下來開始編譯安裝
make
make install建立mysql賬戶
groupadd -u mysql
useradd -u 5001 -g mysql -s /sbin/nologin -d /home/ mysql mysql 注意：“-s /sbin/nologin” 禁止該使用者登入。
配置
cp support-files/my-small.cnf /etc/my.cnf
配置環境變數：export PATH=/home/mysql/app/bin:$PATH
許可權：chown -R mysql.mysql ./mysql/app/data/
chmod -R 1777
初始化：
[root@web app]# /home/mysql/app/scripts/mysql_install_db --basedir=/home/mysql/app
--datadir=/home/mysql/app/data
--user=mysql啟動：
[root@web app]# mysqld_safe --defaults-file=/home/mysql/app/data/my.conf 錯誤1：
InnoDB: Error: space header page consists of zero bytes in data file ./ibdata1
170903 12:43:39 InnoDB: Could not open or create data files.
170903 12:43:39 InnoDB: If you tried to add new data files, and it failed here,
170903 12:43:39 InnoDB: you should now edit innodb_data_file_path in my.cnf back
170903 12:43:39 InnoDB: to what it was, and remove the new ibdata files InnoDB created
170903 12:43:39 InnoDB: in this failed attempt. InnoDB only wrote those files full of
170903 12:43:39 InnoDB: zeros, but did not yet use them in any way. But be careful: do not
170903 12:43:39 InnoDB: remove old data files which contain your precious data!
170903 12:43:39 [ERROR] Plugin 'InnoDB' init function returned error.
170903 12:43:39 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
170903 12:43:39 [ERROR] Unknown/unsupported storage engine: InnoDB
170903 12:43:39 [ERROR] Aborting解決：
看如上提示，mysql在安裝資料檔案ibdata1失敗，最簡單的方法就是重新初始化mysql。
其餘3臺伺服器類似如上步驟安裝mysql。
兩臺從庫開啟binlog，用於同步。
到目前為止，兩個主節點安裝了heartbeat，drbd和mysql；
heartbeat接管drbd服務，drbd資料盤掛載到/data目錄用於同步兩個主節點資料，我們可以把mysql資料檔案掛在/data目錄。
手工切換在drbd管理的node1節點，在node1節點執行
[root@node1 data]# /usr/share/heartbeat/hb_standby
Going standby [all].
[root@node1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_ams-lv_root
36G 7.6G 27G 23% /
tmpfs 1.6G 0 1.6G 0% /dev/shm
/dev/sda1 477M 32M 420M 8% /boot檢視node2節點：
[root@node2 /]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_web-lv_root
35G 5.7G 28G 18% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
/dev/sda1 477M 32M 420M 8% /boot
/dev/drbd0 8.1G 1.7G 6.0G 23% /data發現node2節點已經自動掛載了/data目錄。
在node2安裝mysql資料庫，把資料檔案，my.cnf和mysql.sock檔案對映到/data目錄。(注意許可權)
在node2節點啟動mysql ：
[root@node2 /]# mysqld_safe --defaults-file=/home/mysql/app/data/my.conf
Warning: World-writable config file '/home/mysql/app/data/my.conf' is ignored
Warning: World-writable config file '/home/mysql/app/data/my.conf' is ignored
170903 16:29:36 mysqld_safe Logging to '/home/mysql/app/data/node2.err'.
170903 16:29:36 mysqld_safe Starting mysqld daemon with databases from /home/mysql/app/data可知node2節點mysql啟動正常。
由於heartbeat暫時沒有接管mysql，所以先在node2把mysql殺掉，在node1節點接管資源，在啟動mysql服務.如下：
node2節點殺掉mysql服務：
[root@node2 /]# mysqladmin -uroot -p'ckf77856' shutdown node1節點接管drbd資源
[root@node1 ~]# /usr/share/heartbeat/hb_takeover local
[root@node1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_ams-lv_root
36G 6.4G 28G 19% /
tmpfs 1.6G 0 1.6G 0% /dev/shm
/dev/sda1 477M 32M 420M 8% /boot
/dev/drbd0 8.1G 176M 7.5G 3% /data在drbd主節點啟動mysql服務
mysqld_safe --defaults-file=/home/mysql/app/data/my.conf
可知主節點mysql服務正常。
heartbeat管理mysql服務
以上都是手工管理mysql服務，下面使用heartbeat來實現mysql資源接管。
vi /etc/ha.d/haresources，把mysql新增如下配置(drbd兩個節點都需執行)
#node-name resource1 resource2 ... resourceN
node2 IPaddr::10.37.2.241/24/eth0
node1 IPaddr::10.37.2.240/24/eth0 drbddisk::data Filesystem::/dev/drbd0::/data::ext3 mysql把mysql啟動指令碼放在/etc/ha.d/resource.d/下面並新增可執行許可權。(drbd兩個節點都需執行)
[root@node2 ~]# cp /home/mysql/app/support-files/mysql.server /etc/ha.d/resource.d/mysql
[root@node2 ~]# chmod +x /etc/ha.d/resource.d/mysql禁止mysql自啟動
chkconfig mysqld off啟動heartbeat服務，在node1,node2節點檢視檢視vip和drbd角色啟動狀態。
在node1節點檢視drbd管理drbd0裝置掛載情況和mysql服務狀態。
測試故障轉移：
在node1節點把heartbeat停止掉，在node2節點檢視vip，drbd角色狀態，drbd0裝置掛載和mysql服務狀態。
node2節點：
啟動heartbeat
[root@node2 ~]# /etc/init.d/heartbeat start
Starting High-Availability services: INFO: Resource is stopped
INFO: Resource is stopped
Done.檢視heartbeat日誌
tail -f /var/log/ha-log
Sep 08 18:01:54 node2 heartbeat: [9343]: info: **************************
Sep 08 18:01:54 node2 heartbeat: [9343]: info: Configuration validated. Starting heartbeat 3.0.4
Sep 08 18:01:54 node2 heartbeat: [9344]: info: heartbeat: version 3.0.4
Sep 08 18:01:54 node2 heartbeat: [9344]: info: Heartbeat generation: 1503897788
Sep 08 18:01:54 node2 heartbeat: [9344]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1
Sep 08 18:01:54 node2 heartbeat: [9344]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1
Sep 08 18:01:54 node2 heartbeat: [9344]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth1
Sep 08 18:01:54 node2 heartbeat: [9344]: info: glib: ucast: bound send socket to device: eth1
Sep 08 18:01:54 node2 heartbeat: [9344]: info: glib: ucast: set SO_REUSEPORT(w)
Sep 08 18:01:54 node2 heartbeat: [9344]: info: glib: ucast: bound receive socket to device: eth1
Sep 08 18:01:54 node2 heartbeat: [9344]: info: glib: ucast: set SO_REUSEPORT(w)
Sep 08 18:01:54 node2 heartbeat: [9344]: info: glib: ucast: started on port 694 interface eth1 to 192.168.52.100
Sep 08 18:01:54 node2 heartbeat: [9344]: info: glib: ping heartbeat started.
Sep 08 18:01:54 node2 heartbeat: [9344]: info: G_main_add_TriggerHandler: Added signal manual handler
Sep 08 18:01:54 node2 heartbeat: [9344]: info: G_main_add_TriggerHandler: Added signal manual handler
Sep 08 18:01:54 node2 heartbeat: [9344]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Sep 08 18:01:54 node2 heartbeat: [9344]: info: Local status now set to: 'up'
Sep 08 18:01:54 node2 heartbeat: [9344]: info: Link 192.168.52.100:192.168.52.100 up.
Sep 08 18:01:54 node2 heartbeat: [9344]: info: Status update for node 192.168.52.100: status ping
Sep 08 18:01:54 node2 heartbeat: [9344]: info: Link node2:eth1 up.
Sep 08 18:03:55 node2 heartbeat: [9344]: WARN: node node1: is dead
Sep 08 18:03:55 node2 heartbeat: [9344]: info: Comm_now_up(): updating status to active
Sep 08 18:03:55 node2 heartbeat: [9344]: info: Local status now set to: 'active'
Sep 08 18:03:55 node2 heartbeat: [9344]: WARN: No STONITH device configured.
Sep 08 18:03:55 node2 heartbeat: [9344]: WARN: Shared disks are not protected.
Sep 08 18:03:55 node2 heartbeat: [9344]: info: Resources being acquired from node1.
harc(default)[9357]: 2017/09/08_18:03:55 info: Running /etc/ha.d//rc.d/status status
mach_down(default)[9393]: 2017/09/08_18:03:55 info: Taking over resource group IPaddr::10.37.2.240/24/eth0
ResourceManager(default)[9444]: 2017/09/08_18:03:55 info: Acquiring resource group: node1 IPaddr::10.37.2.240/24/eth0 drbddisk::data Filesystem::/dev/drbd0::/data::ext3 mysql
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.37.2.241)[9425]: 2017/09/08_18:03:55 INFO: Resource is stopped
Sep 08 18:03:55 node2 heartbeat: [9358]: info: Local Resource acquisition completed.
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.37.2.240)[9497]: 2017/09/08_18:03:55 INFO: Resource is stopped
ResourceManager(default)[9444]: 2017/09/08_18:03:55 info: Running /etc/ha.d/resource.d/IPaddr 10.37.2.240/24/eth0 start
IPaddr(IPaddr_10.37.2.240)[9669]: 2017/09/08_18:03:55 INFO: Adding inet address 10.37.2.240/24 with broadcast address 10.37.2.255 to device eth0
IPaddr(IPaddr_10.37.2.240)[9669]: 2017/09/08_18:03:55 INFO: Bringing device eth0 up
IPaddr(IPaddr_10.37.2.240)[9669]: 2017/09/08_18:03:55 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.37.2.240 eth0 10.37.2.240 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.37.2.240)[9643]: 2017/09/08_18:03:55 INFO: Success
ResourceManager(default)[9444]: 2017/09/08_18:03:55 info: Running /etc/ha.d/resource.d/drbddisk data start
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0)[9797]: 2017/09/08_18:03:55 INFO: Resource is stopped
ResourceManager(default)[9444]: 2017/09/08_18:03:55 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /data ext3 start
Filesystem(Filesystem_/dev/drbd0)[9887]: 2017/09/08_18:03:55 INFO: Running start for /dev/drbd0 on /data
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0)[9879]: 2017/09/08_18:03:55 INFO: Success
ResourceManager(default)[9444]: 2017/09/08_18:03:55 info: Running /etc/ha.d/resource.d/mysql start
mach_down(default)[9393]: 2017/09/08_18:03:56 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down(default)[9393]: 2017/09/08_18:03:56 info: mach_down takeover complete for node node1.
Sep 08 18:03:56 node2 heartbeat: [9344]: info: mach_down takeover complete.
Sep 08 18:03:56 node2 heartbeat: [9344]: info: Initial resource acquisition complete (mach_down)
harc(default)[10120]: 2017/09/08_18:03:56 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
ip-request-resp(default)[10120]: 2017/09/08_18:03:56 received ip-request-resp IPaddr::10.37.2.241/24/eth0 OK yes
ResourceManager(default)[10143]: 2017/09/08_18:03:56 info: Acquiring resource group: node2 IPaddr::10.37.2.241/24/eth0
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.37.2.241)[10171]: 2017/09/08_18:03:56 INFO: Resource is stopped
ResourceManager(default)[10143]: 2017/09/08_18:03:57 info: Running /etc/ha.d/resource.d/IPaddr 10.37.2.241/24/eth0 start
IPaddr(IPaddr_10.37.2.241)[10294]: 2017/09/08_18:03:57 INFO: Adding inet address 10.37.2.241/24 with broadcast address 10.37.2.255 to device eth0
IPaddr(IPaddr_10.37.2.241)[10294]: 2017/09/08_18:03:57 INFO: Bringing device eth0 up
IPaddr(IPaddr_10.37.2.241)[10294]: 2017/09/08_18:03:57 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.37.2.241 eth0 10.37.2.241 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.37.2.241)[10268]: 2017/09/08_18:03:57 INFO: Success
Sep 08 18:04:05 node2 heartbeat: [9344]: info: Local Resource acquisition completed. (none)
Sep 08 18:04:05 node2 heartbeat: [9344]: info: local resource transition completed.vip分配情況
[root@node2 ~]