1. 程式人生 > >MySQL高可用方案之DRBD+MySQL+RHCS(下)

MySQL高可用方案之DRBD+MySQL+RHCS(下)

續:MySQL高可用方案之DRBD+MySQL+RHCS(上)

五、MySQL5.6.42安裝

安裝步驟(兩臺機器都要安裝)

[[email protected] ~]# cd /opt/

[[email protected] opt]# ls
mysql-5.6.42-linux-glibc2.12-x86_64.tar.gz

[[email protected] opt]# tar -xvf mysql-5.6.42-linux-glibc2.12-x86_64.tar.gz

[[email protected] opt]# ln -s /opt/mysql-5.6.42-linux-glibc2.12-x86_64 /opt/mysql

[
[email protected]
opt]# mkdir /data/mysql/{data,log,tmp,run,undo} -p [[email protected] opt]# groupadd mysql [[email protected] opt]# useradd mysql -r -g mysql [[email protected] opt]# chown -R mysql:mysql /data/mysql/ [[email protected] opt]# cd /opt/mysql/ [
[email protected]
mysql]# ./scripts/mysql_install_db --user=mysql [[email protected] mysql]# echo 'PATH=$PATH:/opt/mysql/bin' >> /etc/profile [[email protected] mysql]# source /etc/profile [[email protected] mysql]# cp support-files/mysql.server /etc/init.d/mysqld [[email protected]
mysql]# /etc/init.d/mysqld start

六、RHCS介紹

1)什麼是RHCS

RHCS是Red Hat Cluster Suite的縮寫,也就是紅帽叢集套件,RHCS是一個能夠提供高可用性、高可靠性、負載均衡、儲存共享且經濟廉價的叢集工具集合,它將集群系統中三大叢集架構融合一體,可以給web應用、資料庫應用等提供安全、穩定的執行環境。
更確切的說,RHCS是一個功能完備的叢集應用解決方案,它從應用的前端訪問到後端的資料儲存都提供了一個行之有效的叢集架構實現,通過RHCS提供的這種解決方案,不但能保證前端應用持久、穩定的提供服務,同時也保證了後端資料儲存的安全。
RHCS提供了集群系統中三種叢集構架,分別是高可用性叢集、負載均衡叢集、儲存叢集。

2)RHCS提供的三個核心功能

高可用叢集是RHCS的核心功能。當應用程式出現故障,或者系統硬體、網路出現故障時,應用可以通過RHCS提供的高可用管理元件自動、快速從一個節點切換到另一個節點,節點故障轉移功能對客戶端來說是透明的,從而保證應用持續、不間斷的對外提供服務,這就是RHCS高可用叢集實現的功能。
RHCS通過LVS(Linux Virtual Server)來提供負載均衡叢集,而LVS是一個開源的、功能強大的基於IP的負載均衡技術,LVS由負載排程器和服務訪問節點組成,通過LVS的負載排程功能,可以將客戶端請求平均的分配到各個服務節點,同時,還可以定義多種負載分配策略,當一個請求進來時,集群系統根據排程演算法來判斷應該將請求分配到哪個服務節點,然後,由分配到的節點響應客戶端請求,同時,LVS還提供了服務節點故障轉移功能,也就是當某個服務節點不能提供服務時,LVS會自動遮蔽這個故障節點,接著將失敗節點從叢集中剔除,同時將新來此節點的請求平滑的轉移到其它正常節點上來;而當此故障節點恢復正常後,LVS又會自動將此節點加入到叢集中去。而這一系列切換動作,對使用者來說,都是透明的,通過故障轉移功能,保證了服務的不間斷、穩定執行。
RHCS通過GFS檔案系統來提供儲存叢集功能,GFS是Global File System的縮寫,它允許多個服務同時去讀寫一個單一的共享檔案系統,儲存叢集通過將共享資料放到一個共享檔案系統中從而消除了在應用程式間同步資料的麻煩,GFS是一個分散式檔案系統,它通過鎖管理機制,來協調和管理多個服務節點對同一個檔案系統的讀寫操作。

3)RHCS叢集的組成

RHCS是一個叢集工具的集合,主要有下面幾大部分組成:

叢集構架管理器

這是RHCS叢集的一個基礎套件,提供一個叢集的基本功能,使各個節點組成叢集在一起工作,具體包含分散式叢集管理器(CMAN)、成員關係管理、鎖管理(DLM)、配置檔案管理(CCS)、柵裝置(FENCE)。

高可用服務管理器

提供節點服務監控和服務故障轉移功能,當一個節點服務出現故障時,將服務轉移到另一個健康節點。

叢集配置管理工具

RHCS最新版本通過LUCI來配置和管理RHCS叢集,LUCI是一個基於web的叢集配置方式,通過luci可以輕鬆的搭建一個功能強大的集群系統。

LVS(Linux Virtual Server)

LVS是一個開源的負載均衡軟體,利用LVS可以將客戶端的請求根據指定的負載策略和演算法合理的分配到各個服務節點,實現動態、智慧的負載分擔。
RHCS除了上面的幾個核心構成,還可以通過下面一些元件來補充RHCS叢集功能。

GFS(Global File System)

GFS是Redhat公司開發的一款叢集檔案系統,目前的最新版本是GFS2,GFS檔案系統允許多個服務同時讀寫一個磁碟分割槽,通過GFS可以實現資料的集中管理,免去了資料同步和拷貝的麻煩,但GFS並不能孤立的存在,安裝GFS需要RHCS的底層元件支援。

CLVM(Cluster Logical Volume Manager)

Cluster邏輯卷管理,即CLVM,是LVM的擴充套件,這種擴充套件允許cluster中的機器使用LVM來管理共享儲存。

iSCSI

iSCSI是一種在Internet協議上,特別是乙太網上進行資料塊傳輸的標準,它是一種基於IP Storage理論的新型儲存技術,RHCS可以通過ISCSI技術來匯出和分配共享儲存的使用。

GNBD(Global Network Block Device)

全域性網路模組,簡稱GNBD,是GFS的一個補充元件,用於RHCS分配和管理共享儲存,GNBD分為客戶端和服務端,在服務端GNBD允許匯出多個塊裝置或者GNBD檔案,而GNBD客戶端通過匯入這些匯出的塊裝置或者檔案,就可以把它們當作本地塊裝置使用。由於現在GNBD已經停止了開發,所以使用GNBD的越來越少。

4)RHCS叢集結構

RHCS叢集從整體上分為三大部分,負載均衡叢集、 高可用性叢集、儲存叢集,如圖所示:

wKiom1ZxP26AhyanAAEA5ekWqGo409.png

上圖是典型的RHCS叢集拓撲結構:整個拓撲結構分為三個層面:
最上層是LVS負載均衡層,中間一層是Real Server層,就是服務節點部分,最後一層是共享儲存層,主要用於給GFS檔案系統提供共享儲存空間。

5)RHCS叢集執行原理及功能介紹

wKiom1ZyFXqxyYaMAAEOL9MHcVU990.jpg

5.1、分散式叢集管理器(CMAN)
Cluster Manager,簡稱CMAN,是一個分散式叢集管理工具,它執行在叢集的各個節點上,為RHCS提供叢集管理任務。
CMAN用於管理叢集成員、訊息和通知。它通過監控每個節點的執行狀態來了解節點成員之間的關係,當叢集中某個節點出現故障,節點成員關係將發生改變,CMAN及時將這種改變通知底層,進而做出相應的調整。
5.2、鎖管理(DLM)
Distributed Lock Manager,簡稱DLM,表示一個分散式鎖管理器,它是RHCS的一個底層基礎構件,同時也為叢集提供了一個公用的鎖執行機制,在RHCS集群系統中,DLM執行在叢集的每個節點上,GFS通過鎖管理器的鎖機制來同步訪問檔案系統元資料。CLVM通過鎖管理器來同步更新資料到LVM卷和卷組。
DLM不需要設定鎖管理伺服器,它採用對等的鎖管理方式,大大的提高了處理效能。同時,DLM避免了當單個節點失敗需要整體恢復的效能瓶頸,另外,DLM的請求都是本地的,不需要網路請求,因而請求會立即生效。最後,DLM通過分層機制,可以實現多個鎖空間的並行鎖模式。
5.3、配置檔案管理(CCS)
Cluster Configuration System,簡稱CCS,主要用於叢集配置檔案管理和配置檔案在節點之間的同步。CCS執行在叢集的每個節點上,監控每個叢集節點上的單一配置檔案/etc/cluster/cluster.conf的狀態,當這個檔案發生任何變化時,都將此變化更新到叢集中的每個節點,時刻保持每個節點的配置檔案同步。例如,管理員在節點A上更新了叢集配置檔案,CCS發現A節點的配置檔案發生變化後,馬上將此變化傳播到其它節點上去。
rhcs的配置檔案是cluster.conf,它是一個xml檔案,具體包含叢集名稱、叢集節點資訊、叢集資源和服務資訊、fence裝置等,這個會在後面講述。
5.4、柵裝置(FENCE)
FENCE裝置是RHCS叢集中必不可少的一個組成部分,通過FENCE裝置可以避免因出現不可預知的情況而造成的“腦裂”現象,FENCE裝置的出現,就是為了解決類似這些問題,Fence裝置主要就是通過伺服器或儲存本身的硬體管理介面,或者外部電源管理裝置,來對伺服器或儲存直接發出硬體管理指令,將伺服器重啟或關機,或者與網路斷開連線。
FENCE的工作原理是:當意外原因導致主機異常或者宕機時,備機會首先呼叫FENCE裝置,然後通過FENCE裝置將異常主機重啟或者從網路隔離,當FENCE操作成功執行後,返回資訊給備機,備機在接到FENCE成功的資訊後,開始接管主機的服務和資源。這樣通過FENCE裝置,將異常節點佔據的資源進行了釋放,保證了資源和服務始終執行在一個節點上。
RHCS的FENCE裝置可以分為兩種:內部FENCE和外部FENCE,常用的內部FENCE有IBM RSAII卡,HP的iLO卡,還有IPMI的裝置等,外部fence裝置有UPS、SAN SWITCH、NETWORK SWITCH等
5.5、高可用服務管理器
高可用性服務管理主要用來監督、啟動和停止叢集的應用、服務和資源。它提供了一種對叢集服務的管理能力,當一個節點的服務失敗時,高可用性叢集服務管理程序可以將服務從這個失敗節點轉移到其它健康節點上來,並且這種服務轉移能力是自動、透明的。
RHCS通過rgmanager來管理叢集服務,rgmanager執行在每個叢集節點上,在伺服器上對應的程序為clurgmgrd。
在一個RHCS叢集中,高可用**包含叢集服務和叢集資源兩個方面,叢集服務其實就是應用服務,例如apache、mysql等,叢集資源有很多種,例如一個IP地址、一個執行指令碼、ext3/GFS檔案系統等。
在RHCS叢集中,高可用**是和一個失敗轉移域結合在一起的,所謂失敗轉移域是一個執行特定服務的叢集節點的集合。在失敗轉移域中,可以給每個節點設定相應的優先順序,通過優先順序的高低來決定節點失敗時服務轉移的先後順序,如果沒有給節點指定優先順序,那麼叢集高可用服務將在任意節點間轉移。因此,通過建立失敗轉移域不但可以設定服務在節點間轉移的順序,而且可以限制某個服務僅在失敗轉移域指定的節點內進行切換。
5.6、叢集配置管理工具
RHCS提供了多種叢集配置和管理工具,常用的有基於GUI的system-config-cluster、Conga等,也提供了基於命令列的管理工具。
system-config-cluster是一個用於建立叢集和配置叢集節點的圖形化管理工具,它有叢集節點配置和叢集管理兩個部分組成,分別用於建立叢集節點配置檔案和維護節點執行狀態。一般用在RHCS早期的版本中。
Conga是一種新的基於網路的叢集配置工具,與system-config-cluster不同的是,Conga是通過web方式來配置和管理叢集節點的。Conga有兩部分組成,分別是luci和ricci,luci安裝在一**立的計算機上,用於配置和管理叢集,ricci安裝在每個叢集節點上,Luci通過ricci和叢集中的每個節點進行通訊。
RHCS也提供了一些功能強大的叢集命令列管理工具,常用的有clustat、cman_tool、ccs_tool、fence_tool、clusvcadm等,這些命令的用法將在下面講述。
5.7、Redhat GFS
GFS是RHCS為集群系統提供的一個儲存解決方案,它允許叢集多個節點在塊級別上共享儲存,每個節點通過共享一個儲存空間,保證了訪問資料的一致性,更切實的說,GFS是RHCS提供的一個叢集檔案系統,多個節點同時掛載一個檔案系統分割槽,而檔案系統資料不受破壞,這是單一的檔案系統,例如EXT3、EXT2所不能做到的。
為了實現多個節點對於一個檔案系統同時讀寫操作,GFS使用鎖管理器來管理I/O操作,當一個寫程序操作一個檔案時,這個檔案就被鎖定,此時不允許其它程序進行讀寫操作,直到這個寫程序正常完成才釋放鎖,只有當鎖被釋放後,其它讀寫程序才能對這個檔案進行操作,另外,當一個節點在GFS檔案系統上修改資料後,這種修改操作會通過RHCS底層通訊機制立即在其它節點上可見。
在搭建RHCS叢集時,GFS一般作為共享儲存,執行在每個節點上,並且可以通過RHCS管理工具對GFS進行配置和管理。這些需要說明的是RHCS和GFS之間的關係,一般初學者很容易混淆這個概念:執行RHCS,GFS不是必須的,只有在需要共享儲存時,才需要GFS支援,而搭建GFS叢集檔案系統,必須要有RHCS的底層支援,所以安裝GFS檔案系統的節點,必須安裝RHCS元件。

七、RHCS安裝

1、安裝步驟(DRBD1和DRBD2都要安裝)

# yum install fence-virtd-multicast fence-virtd fence-virtd-libvirt -y

# yum install cman rgmanager -y

2、配置檔案(DRBD1和DRBD2完全相同)

# vim /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="2" name="mysql_cluster">
        <fence_daemon post_fail_delay="0" post_join_delay="3"/>
        <clusternodes>
                <clusternode name="DRBD1" nodeid="1" votes="1">
                        <fence>
                                <method name="1">
                                        <device lanplus="" name="mysql1_fence" nodename="DRBD1"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="DRBD2" nodeid="2" votes="1">
                        <fence>
                                <method name="1">
                                        <device lanplus="" name="mysql2_fence" nodename="DRBD2"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>
        <cman expected_votes="1" two_node="1"/>
        <fencedevices>
                <fencedevice agent="fence_ilo" hostname="10.10.110.231" login="root" name="mysql1_fence" passwd="Yuelei66"/>
                <fencedevice agent="fence_ilo" hostname="10.10.110.230" login="root" name="mysql2_fence" passwd="Yuelei66"/>
        </fencedevices>
        <rm>
                <failoverdomains>
                        <failoverdomain name="mysql_faildomain" ordered="1" restricted="0">
                                <failoverdomainnode name="DRBD1" priority="1"/>
                                <failoverdomainnode name="DRBD2" priority="2"/>
                        </failoverdomain>
                </failoverdomains>
                <resources>
                        <script file="/etc/init.d/mysqld" name="mysql_script"/>
                        <fs device="/dev/drbd0" force_fsck="0" force_unmount="1" fstype="ext4" mountpoint="/data" name="drbd_filesystem" options="noatime,nodiratime" self_fence="1"/>
                        <ip address="10.10.110.229" monitor_link="1"/>
                        <drbd name="res_drbd0" resource="r0"/>
                </resources>
                <service autostart="1" domain="mysql_faildomain" name="mysql_service" recovery="relocate">
                        <ip ref="10.10.110.229"/>
                        <drbd ref="res_drbd0">
                                <fs ref="drbd_filesystem"/>
                                <script ref="mysql_script"/>
                        </drbd>
                </service>
        </rm>
</cluster>

3、啟動(DRBD1和DRBD2都要啟動)

# service cman start

# service rgmanager start

4、校驗資訊

[[email protected] cluster]# clustat
Cluster Status for mysql_cluster @ Wed Nov 21 17:09:53 2018
Member Status: Quorate

 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 DRBD1                                                               1 Online, Local, rgmanager
 DRBD2                                                               2 Online, rgmanager

 Service Name                                                     Owner (Last)                                                     State        
 ------- ----                                                     ----- ------                                                     -----        
 service:mysql_service                                            DRBD1                                                            started      

[[email protected] cluster]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:3a:b2:c8 brd ff:ff:ff:ff:ff:ff
    inet 10.10.110.231/24 brd 10.10.110.255 scope global eth0
    inet 10.10.110.229/24 scope global secondary eth0
    inet6 fe80::a00:27ff:fe3a:b2c8/64 scope link
       valid_lft forever preferred_lft forever


[[email protected] cluster]# mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.42-log MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> grant all on *.* to [email protected]'%' identified by 'abc123';
Query OK, 0 rows affected (0.12 sec)


[[email protected] ~]# mysql -upowdba -h10.10.110.229 -pabc123
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.6.42-log MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

5、測試驗證

7.5.1 DRBD1造數
[[email protected] ~]# mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 7
Server version: 5.6.42-log MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use test
Database changed

mysql> create table test5 (id int,name char(2));
Query OK, 0 rows affected (0.16 sec)

mysql> select count(*) from test5;
+----------+
| count(*) |
+----------+
|      100 |
+----------+
1 row in set (0.00 sec)
7.5.2 DRBD1停機(模擬宕機環境)
[[email protected] ~]# clustat
Cluster Status for mysql_cluster @ Wed Nov 21 17:18:04 2018
Member Status: Quorate

 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 DRBD1                                                               1 Online, Local, rgmanager
 DRBD2                                                               2 Online, rgmanager

 Service Name                                                     Owner (Last)                                                     State        
 ------- ----                                                     ----- ------                                                     -----        
 service:mysql_service                                            DRBD1                                                            started      
[[email protected] ~]# reboot


Broadcast message from [email protected]
        (/dev/pts/0) at 17:19 ...

The system is going down for reboot NOW!
7.5.3 DRBD2接管服務
[[email protected] ~]# clustat
Cluster Status for mysql_cluster @ Wed Nov 21 17:19:24 2018
Member Status: Quorate

 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 DRBD1                                                               1 Online, rgmanager
 DRBD2                                                               2 Online, Local, rgmanager

 Service Name                                                     Owner (Last)                                                     State        
 ------- ----                                                     ----- ------                                                     -----        
 service:mysql_service                                            DRBD1                                                            stopping     
[[email protected] ~]# clustat
Cluster Status for mysql_cluster @ Wed Nov 21 17:19:31 2018
Member Status: Quorate

 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 DRBD1                                                               1 Online
 DRBD2                                                               2 Online, Local, rgmanager

 Service Name                                                     Owner (Last)                                                     State        
 ------- ----                                                     ----- ------                                                     -----        
 service:mysql_service                                            DRBD2                                                            starting

[[email protected] data]# clustat     
Cluster Status for mysql_cluster @ Wed Nov 21 19:55:16 2018
Member Status: Quorate

 Member Name ID Status
 ------ ---- ---- ------
 DRBD1 1 Offline
 DRBD2 2 Online, Local, rgmanager

 Service Name Owner (Last) State         
 ------- ---- ----- ------ -----         
 service:mysql_service DRBD2 started       

[[email protected] data]# drbd-overview 
NOTE: drbd-overview will be deprecated soon.
Please consider using drbdtop.

 0:r0/0 WFConnection Primary/Unknown UpToDate/DUnknown /data ext4 32G 4.4G 26G 15% 

[[email protected] data]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:f4:d4:1d brd ff:ff:ff:ff:ff:ff
    inet 10.10.110.230/24 brd 10.10.110.255 scope global eth0
    inet 10.10.110.229/24 scope global secondary eth0
    inet6 fe80::a00:27ff:fef4:d41d/64 scope link
       valid_lft forever preferred_lft forever

[[email protected] ~]# mysql -upowdba -h10.10.110.229 -pabc123
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.42-log MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use test
Database changed

mysql> select count(*) from test5;
+----------+
| count(*) |
+----------+
|      100 |
+----------+
1 row in set (0.18 sec)
7.5.4 DRBD1重新啟動
[[email protected] ~]# drbd-overview
NOTE: drbd-overview will be deprecated soon.
Please consider using drbdtop.

 0:r0/0  Unconfigured . .

[[email protected] ~]# service drbd start
Starting DRBD resources: [
     create res: r0
   prepare disk: r0
    adjust disk: r0
     adjust net: r0
]
.
[[email protected] ~]# drbd-overview    
NOTE: drbd-overview will be deprecated soon.
Please consider using drbdtop.

 0:r0/0  Connected Secondary/Primary UpToDate/UpToDate

[[email protected] ~]# service cman start
Starting cluster:
   Checking if cluster has been disabled at boot...        [  OK  ]
   Checking Network Manager...                             [  OK  ]
   Global setup...                                         [  OK  ]
   Loading kernel modules...                               [  OK  ]
   Mounting configfs...                                    [  OK  ]
   Starting cman... tempfile:8: element device: Relax-NG validity error : Invalid attribute nodename for element device
Relax-NG validity error : Extra element fence in interleave
tempfile:4: element clusternodes: Relax-NG validity error : Element clusternode failed to validate content
tempfile:5: element clusternode: Relax-NG validity error : Element clusternodes has extra content: clusternode
Relax-NG validity error : Extra element fencedevices in interleave
tempfile:21: element fencedevices: Relax-NG validity error : Element cluster failed to validate content
Configuration fails to validate
                                                           [  OK  ]
   Waiting for quorum...                                   [  OK  ]
   Starting fenced...                                      [  OK  ]
   Starting dlm_controld...                                [  OK  ]
   Tuning DLM kernel config...                             [  OK  ]
   Starting gfs_controld...                                [  OK  ]
   Unfencing self...                                       [  OK  ]
   Joining fence domain...                                 [  OK  ]

[[email protected] ~]# service rgmanager start
Starting Cluster Service Manager:                          [  OK  ]

[[email protected] ~]# clustat
Cluster Status for mysql_cluster @ Wed Nov 21 17:49:21 2018
Member Status: Quorate

 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 DRBD1                                                               1 Online, Local, rgmanager
 DRBD2                                                               2 Online, rgmanager

 Service Name                                                     Owner (Last)                                                     State        
 ------- ----                                                     ----- ------                                                     -----        
 service:mysql_service                                            DRBD2                                                            stopping     

[[email protected] ~]# clustat
Cluster Status for mysql_cluster @ Wed Nov 21 17:51:14 2018
Member Status: Quorate

 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 DRBD1                                                               1 Online, Local, rgmanager
 DRBD2                                                               2 Online, rgmanager

 Service Name                                                     Owner (Last)                                                     State        
 ------- ----                                                     ----- ------                                                     -----        
 service:mysql_service                                            DRBD1                                                            started      

[[email protected] ~]# drbd-overview
NOTE: drbd-overview will be deprecated soon.
Please consider using drbdtop.

 0:r0/0  Connected Primary/Secondary UpToDate/UpToDate /data ext4 32G 4.4G 26G 15%

[[email protected] ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:3a:b2:c8 brd ff:ff:ff:ff:ff:ff
    inet 10.10.110.231/24 brd 10.10.110.255 scope global eth0
    inet 10.10.110.229/24 scope global secondary eth0
    inet6 fe80::a00:27ff:fe3a:b2c8/64 scope link
       valid_lft forever preferred_lft forever

[[email protected] mysql]# ps -ef|grep mysqld
root      3515     1  0 17:49 ?        00:00:00 /bin/sh /opt/mysql/bin/mysqld_safe --datadir=/data/mysql/data --pid-file=/data/mysql/data/mysql.pid
mysql     4968  3515  0 17:49 ?        00:00:02 /opt/mysql/bin/mysqld --basedir=/opt/mysql --datadir=/data/mysql/data --plugin-dir=/opt/mysql/lib/plugin --user=mysql --log-error=/data/mysql/log/error.log --open-files-limit=65535 --pid-file=/data/mysql/data/mysql.pid --socket=/data/mysql/run/mysql.sock --port=3306
root      7812  1844  0 17:53 pts/0    00:00:00 grep mysqld
7.5.5 DRBD1中pkill mysql
[[email protected] data]# clustat 
Cluster Status for mysql_cluster @ Wed Nov 21 19:41:41 2018
Member Status: Quorate

 Member Name ID Status
 ------ ---- ---- ------
 DRBD1 1 Online, Local, rgmanager
 DRBD2 2 Online, rgmanager

 Service Name Owner (Last) State         
 ------- ---- ----- ------ -----         
 service:mysql_service DRBD1 started       

[[email protected] data]# drbd-overview 
NOTE: drbd-overview will be deprecated soon.
Please consider using drbdtop.

 0:r0/0 Connected Primary/Secondary UpToDate/UpToDate /data ext4 32G 4.4G 26G 15% 

[[email protected] data]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:3a:b2:c8 brd ff:ff:ff:ff:ff:ff
    inet 10.10.110.231/24 brd 10.10.110.255 scope global eth0
    inet 10.10.110.229/24 scope global secondary eth0
    inet6 fe80::a00:27ff:fe3a:b2c8/64 scope link 
       valid_lft forever preferred_lft forever

[[email protected] ~]# clustat 
Cluster Status for mysql_cluster @ Wed Nov 21 19:41:57 2018
Member Status: Quorate

 Member Name ID Status
 ------ ---- ---- ------
 DRBD1 1 Online, rgmanager
 DRBD2 2 Online, Local, rgmanager

 Service Name Owner (Last) State         
 ------- ---- ----- ------ -----         
 service:mysql_service DRBD1 started       

[[email protected] ~]# drbd-overview 
NOTE: drbd-overview will be deprecated soon.
Please consider using drbdtop.

 0:r0/0 Connected Secondary/Primary UpToDate/UpToDate 

[[email protected] data]# pkill mysqld

[[email protected] data]# ps -ef|grep mysqld
root 18098 1844 0 19:42 pts/0 00:00:00 grep mysqld

[[email protected] data]# drbd-overview     
NOTE: drbd-overview will be deprecated soon.
Please consider using drbdtop.

 0:r0/0 Connected Secondary/Secondary UpToDate/UpToDate 

[[email protected] data]# clustat           
Cluster Status for mysql_cluster @ Wed Nov 21 19:42:52 2018
Member Status: Quorate

 Member Name ID Status
 ------ ---- ---- ------
 DRBD1 1 Online, Local, rgmanager
 DRBD2 2 Online, rgmanager

 Service Name Owner (Last) State         
 ------- ---- ----- ------ -----         
 service:mysql_service (DRBD2) recoverable   

[[email protected] data]# clustat 
Cluster Status for mysql_cluster @ Wed Nov 21 19:43:17 2018
Member Status: Quorate

 Member Name ID Status
 ------ ---- ---- ------
 DRBD1 1 Online, Local, rgmanager
 DRBD2 2 Online, rgmanager

 Service Name Owner (Last) State         
 ------- ---- ----- ------ -----         
 service:mysql_service DRBD2 starting      
   
[[email protected] data]# clustat 
Cluster Status for mysql_cluster @ Wed Nov 21 19:43:36 2018
Member Status: Quorate

 Member Name ID Status
 ------ ---- ---- ------
 DRBD1 1 Online, Local, rgmanager
 DRBD2 2 Online, rgmanager

 Service Name Owner (Last) State         
 ------- ---- ----- ------ -----         
 service:mysql_service DRBD2 started       

[[email protected] ~]# ps -ef|grep mysqld
root 30799 1 0 19:42 ? 00:00:00 /bin/sh /opt/mysql/bin/mysqld_safe --datadir=/data/mysql/data --pid-file=/data/mysql/data/mysql.pid
mysql 32239 30799 7 19:42 ? 00:00:02 /opt/mysql/bin/mysqld --basedir=/opt/mysql --datadir=/data/mysql/data --plugin-dir=/opt/mysql/lib/plugin --user=mysql --log-error=/data/mysql/log/error.log --open-files-limit=65535 --pid-file=/data/mysql/data/mysql.pid --socket=/data/mysql/run/mysql.sock --port=3306
root 32333 17473 0 19:43 pts/0 00:00:00 grep mysqld

[[email protected] ~]# clustat 
Cluster Status for mysql_cluster @ Wed Nov 21 19:43:49 2018
Member Status: Quorate

 Member Name ID Status
 ------ ---- ---- ------
 DRBD1 1 Online, rgmanager
 DRBD2 2 Online, Local, rgmanager

 Service Name Owner (Last) State         
 ------- ---- ----- ------ -----         
 service:mysql_service DRBD2 started       

[[email protected] ~]# drbd-overview 
NOTE: drbd-overview will be deprecated soon.
Please consider using drbdtop.

 0:r0/0 Connected Primary/Secondary UpToDate/UpToDate /data ext4 32G 4.4G 26G 15% 

[[email protected] ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:f4:d4:1d brd ff:ff:ff:ff:ff:ff
    inet 10.10.110.230/24 brd 10.10.110.255 scope global eth0
    inet 10.10.110.229/24 scope global secondary eth0
    inet6 fe80::a00:27ff:fef4:d41d/64 scope link 
       valid_lft forever preferred_lft forever