一：軟體簡介

Heartbeat介紹

官方站點：http://linux-ha.org/wiki/Main_Page
heartbeat可以資源(VIP地址及程式服務)從一臺有故障的伺服器快速的轉移到另一臺正常的伺服器提供服務，heartbeat和keepalived相似，heartbeat可以實現failover功能，但不能實現對後端的健康檢查

heartbeat和keepalived應用場景及區別

很多網友說為什麼不使用keepalived而使用長期不更新的heartbeat，下面說一下它們之間的應用場景及區別
1、對於web，db，負載均衡(lvs,haproxy,nginx)等，heartbeat和keepalived都可以實現
2、lvs最好和keepalived結合，因為keepalived最初就是為lvs產生的，(heartbeat沒有對RS的健康檢查功能，heartbeat可以通過ldircetord來進行健康檢查的功能)
3、mysql雙主多從，NFS/MFS儲存，他們的特點是需要資料同步，這樣的業務最好使用heartbeat，因為heartbeat有自帶的

drbd指令碼
總結：無資料同步的應用程式高可用可選擇keepalived，有資料同步的應用程式高可用可選擇heartbeat

DRBD介紹

官方站點：http://www.drbd.org/
DRBD(DistributedReplicatedBlockDevice)是一個基於塊裝置級別在遠端伺服器直接同步和映象資料的軟體，用軟體實現的、無共享的、伺服器之間映象塊裝置內容的儲存複製解決方案。它可以實現在網路中兩臺伺服器之間基於塊裝置級別的實時映象或同步複製(兩臺伺服器都寫入成功)/非同步複製(本地伺服器寫入成功)，相當於網路的RAID1，由於是基於塊裝置(磁碟，LVM邏輯卷)，在檔案系統的底層，所以資料複製要比cp命令更快
DRBD已經被MySQL官方寫入文件手冊作為推薦的高可用的方案之一

nfs介紹

NFS（Network File System）即網路檔案系統，是FreeBSD支援的檔案系統中的一種，它允許網路中的計算機之間通過TCP/IP網路共享資源。在NFS的應用中，本地NFS的客戶端應用可以透明地讀寫位於遠端NFS伺服器上的檔案，就像訪問本地檔案一樣。

二：架構基本環境準備

1.本次實驗架構圖如下：

nfs高可用架構

2.實驗基礎環境：

[[email protected] ~]# cat /etc/hosts 10.0.0.72  master 10.0.0.71  backup [[email protected] ~]# uname -a Linux backup 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [[email protected] ~]# uname -a Linux master 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [[email protected] ~]# /etc/init.d/iptables stop [[email protected] ~]# /etc/init.d/iptables stop [[email protected] ~]# setenforce 0 [[email protected] ~]#  setenforce 0 or sed -i 's/SELINUX=enforcing/SELINUX=disabled' /etc/selinux/config

時間同步：

[[email protected] ~]# ntpdate 10.0.2.2 29 Aug 09:26:22 ntpdate[2288]: step time server 10.0.2.2 offset -28524.389257 sec [[email protected] ~]#  ntpdate 10.0.2.2 29 Aug 09:26:55 ntpdate[2255]: step time server 10.0.2.2 offset 13332955.402534 sec

新增路由心跳線：

master: route add -host 10.20.23.111  dev eth2 echo "/sbin/route add -host 10.20.23.111  dev eth2" >>/ect/rc.local backup route add -host 10.20.23.115  dev eth2 echo "/sbin/route add -host 10.20.23.115  dev eth2" >>/ect/rc.local

三.配置heartbeat

1.安裝heartbeat

主備操作一致

wget http://mirrors.sohu.com/fedora-epel/6/i386/epel-release-6-8.noarch.rpm rpm -ivh epel-release-6-8.noarch.rpm yum install heartbeat* 檢視安裝版本： [[email protected] ~]# rpm -qa heartbeat heartbeat-3.0.4-2.el6.x86_64 [[email protected] ~]# rpm -qa heartbeat heartbeat-3.0.4-2.el6.x86_64

2.配置檔案

ha.cf heartbeat引數配置檔案
authkey heartbeat認證檔案
haresource 資源配置檔案
模板配置檔案：

[[email protected] ha.d]# ll /usr/share/doc/heartbeat-3.0.4/ -rw-r--r--. 1 root root  1873 Dec  2  2013 apphbd.cf -rw-r--r--. 1 root root   645 Dec  2  2013 authkeys -rw-r--r--. 1 root root  3701 Dec  2  2013 AUTHORS -rw-r--r--. 1 root root 58752 Dec  2  2013 ChangeLog -rw-r--r--. 1 root root 17989 Dec  2  2013 COPYING -rw-r--r--. 1 root root 26532 Dec  2  2013 COPYING.LGPL -rw-r--r--. 1 root root 10502 Dec  2  2013 ha.cf -rw-r--r--. 1 root root  5905 Dec  2  2013 haresources -rw-r--r--. 1 root root  2935 Dec  2  2013 README cd /etc/ha.d [[email protected] ha.d]# cp /usr/share/doc/heartbeat-3.0.4/ha.cf ./ [[email protected] ha.d]# cp /usr/share/doc/heartbeat-3.0.4/authkeys ./ [[email protected] ha.d]# cp /usr/share/doc/heartbeat-3.0.4/haresources ./

賦許可權：

chmod 600 /etc/ha.d/authkeys

修改配置檔案：

[[email protected] ha.d]# cp ha.cf{,.bak} [[email protected] ha.d]# cp authkeys{,.bak}  [[email protected] ha.d]# cp haresources{,.bak}  cat >/etc/ha.d/ha.cf<<EOF debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 keepalive 2 deadtime 30 warntime 10 initdead 60 mcast eth1 225.0.0.1 694 1 0 auto_failback on node master node backup EOF [[email protected] ha.d]# grep -Ev '#|^$' /etc/ha.d/ha.cf cat >/etc/ha.d/authkeys <<EOF auth 1 1 sha1 c4f9375f9834b4e7f0a528cc65c055702bf5f24a EOF chmod 600 /etc/ha.d/authkeys ll /etc/ha.d/authkeys grep -Ev '#|^$' /etc/ha.d/authkeys cat >/etc/ha.d/haresources<<EOF master    IPaddr::10.0.0.73/25/eth0   EOF

3.啟動主的heartbeat

[[email protected] network-scripts]# /etc/init.d/heartbeat start

檢視vip開啟情況：
開始的時候沒發現VIP 60s後vip繫結原因是initdead 60；

[[email protected] network-scripts]# ip add|grep 10.0.0. inet 10.0.0.72/25 brd 10.0.0.127 scope global eth0 inet 10.0.0.73/25 brd 10.0.0.127 scope global secondary eth0 [[email protected] ~]# ip addr |grep 10.0.0 inet 10.0.0.71/25 brd 10.0.0.127 scope global eth0

4.測試heartbeat

停掉masterheartbeat，檢視是否接管。

[[email protected] ha.d]# /etc/init.d/heartbeat stop Stopping High-Availability services: Done.

檢視backup是否接管；基本很快接管。

5.測試專用：

釋放heartbeat到另一臺
/usr/share/heartbeat/hb_standby
接管回來：
/usr/share/heartbeat/hb_takeover

四：配置drbd

1.主備各新增一塊磁碟，2G,3G。

2.對新磁碟進行分割槽

[[email protected] ~]# fdisk -l Disk /dev/sdb: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 [[email protected] ~]# fdisk -l Disk /dev/sdb: 3221 MB, 3221225472 bytes 255 heads, 63 sectors/track, 391 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 備註：16T以上可以使用e2fsprogs進行格式化，或者改成xfs格式 大於2T分割槽格式化：parted master,backup都要操作： master: [[email protected] ~]# yum install -y  parted [[email protected] ~]# parted /dev/sdb mklabel gpt yes ##調整分割槽表 Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue? Information: You may need to update /etc/fstab.       [[email protected] ~]# parted /dev/sdb mkpart primary ext4 0 1000 ignore  #增加一個分割槽 Warning: The resulting partition is not properly aligned for best performance. Information: You may need to update /etc/fstab. [[email protected] ~]# parted /dev/sdb mkpart primary ext4 1001 2000  Yes ignore #增加一個分割槽 Warning: You requested a partition from 1001MB to 2000MB.                 The closest location we can manage is 1001MB to 1001MB. Is this still acceptable to you? Warning: The resulting partition is not properly aligned for best performance. Information: You may need to update /etc/fstab.    [[email protected] ~]# parted /dev/sdb p   #檢視分割槽結果 Model: VMware, VMware Virtual S (scsi) Disk /dev/sdb: 2147MB Sector size (logical/physical): 512B/512B Partition Table: gpt Number  Start   End     Size    File system  Name     Flags 1      17.4kB  1000MB  1000MB               primary 2      1001MB  2000MB  998MB                primary [[email protected] ~]#  ls /dev/sdb* /dev/sdb  /dev/sdb1  /dev/sdb2

備上同樣操作

3.格式化分割槽：

meta data分割槽無需格式化

[[email protected] ~]# mkfs.ext4 /dev/sdb1 [[email protected] ~]# tune2fs -c -1 /dev/sdb1  #取消磁碟檢查 tune2fs 1.41.12 (17-May-2010) Setting maximal mount count to -1 [[email protected] ~]# mkfs.ext4 /dev/sdb1 [[email protected] ~]# tune2fs -c -1 /dev/sdb1 tune2fs 1.41.12 (17-May-2010) Setting maximal mount count to -1

4.安裝DRBD軟體：

master，backup都安裝drbd

rpm -Uvh http://www.elrepo.org/elrepo-release-6-8.el6.elrepo.noarch.rpm yum install drbd  kmod-drbd84 -y [[email protected] src]#  rpm -qa drbd84-utils kmod-drbd84 drbd84-utils-8.9.8-1.el6.elrepo.x86_64 kmod-drbd84-8.4.9-1.el6.elrepo.x86_64 [[email protected] ~]#  rpm -qa drbd84-utils kmod-drbd84 drbd84-utils-8.9.8-1.el6.elrepo.x86_64 kmod-drbd84-8.4.9-1.el6.elrepo.x86_64

載入到核心中：

master: [[email protected] ~]# lsmod | grep drbd [[email protected] ~]# modprobe drbd [[email protected] ~]# lsmod | grep drbd drbd                  374888  0 libcrc32c               1246  1 drbd backup： [[email protected] ~]# lsmod | grep drbd [[email protected] ~]# modprobe drbd [[email protected] ~]# lsmod | grep drbd drbd                  374888  0 libcrc32c               1246  1 drbd echo "modprobe drbd" >>/etc/rc.local or echo "modprobe drbd" >/etc/sysconfig/modules/drbd.modules #載入到核心模組中 chmod 755  /etc/sysconfig/modules/drbd.modules

5：編輯drbd.conf

global { usage-count no;  ##是否參加DRBD使用者統計，預設是yes } common {   ##通用配置 syncer { rate 100M;  ##設定主備節點同步時的網路速率最大值 verify-alg crc32c; } } # primary for drbd1 resource data {     ##data是資源名字 protocol C;    ##協議 disk {         ##磁碟錯誤控制 on-io-error   detach;  ##分離 } on master {          ##節點hostname device    /dev/drbd0; disk      /dev/sdb1;    ##drbd0對應的磁碟 address   10.0.0.82:7788;  ##通訊監聽地址，心跳IP meta-disk /dev/sdb2[0];  ##存放meta資訊 } on backup {   device     /dev/drbd0; disk       /dev/sdb1; address    10.0.0.83:7788; meta-disk  /dev/sdb2[0]; } }

主備配置一致.

6.master,backup一起執行，初始化drbd

[[email protected] etc]# drbdadm create-md data initializing activity log NOT initializing bitmap Writing meta data... New drbd meta data block successfully created. [[email protected] ~]# drbdadm create-md data 啟動drbd: drbdadm up all or /etc/init.d/drbd start 執行結果輸出： Starting DRBD resources: [ create res: data prepare disk: data adjust disk: data adjust net: data ]

檢視啟動狀態：

cat /proc/drbd or /etc/init.d/drbd status [[email protected] etc]# /etc/init.d/drbd status drbd driver loaded OK; device status: version: 8.4.9-1 (api:1/proto:86-101) GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by [email protected], 2016-12-13 18:38:15 m:res   cs         ro                   ds                         p  mounted  fstype 0:data  Connected  Secondary/Secondary  Inconsistent/Inconsistent  C [[email protected] ~]# cat /proc/drbd version: 8.4.9-1 (api:1/proto:86-101) GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by [email protected], 2016-12-13 18:38:15 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----- ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:976548

7.master提升為主：

在master執行

drbdadm -- --overwrite-data-of-peer primary data

檢視備節點：

8.掛載drbd

[email protected] etc]# mkdir /data -p
[[email protected] etc]# mount /dev/drbd0 /data
檢視掛載情況

五.配置nfs

1.安裝nfs

[[email protected] ~]# yum install -y rpcbind nfs-utils [[email protected] data]# yum install -y rpcbind nfs-utils

2.配置nfs共享目錄

[[email protected] ~]# cat  /etc/exports ####nfs+drbd+heartbeat /data  10.0.0.0/24(rw,sync,all_squash) [[email protected] data]# cat /etc/exports ####nfs+drbd+heartbeat /data  10.0.0.0/24(rw,sync,all_squash) 參考標準： [[email protected] drbd]# cat /etc/exports    /data 192.168.0.0/24(rw,sync,no_root_squash,anonuid=0,anongid=0)   [[email protected] ~]# cat /etc/exports    /data 192.168.0.0/24(rw,sync,no_root_squash,anonuid=0,anongid=0)

3.啟動rpcbind和nfs服務

[[email protected] ~]# /etc/init.d/rpcbind start;chkconfig rpcbind off [[email protected] ~]#  /etc/init.d/nfs start;chkconfig nfs off  Starting NFS services:                                     [  OK  ] Starting NFS quotas:                                       [  OK  ] Starting NFS mountd:                                       [  OK  ] Starting NFS daemon:                                       [  OK  ] Starting RPC idmapd:                                       [  OK  ] [[email protected] data]#  /etc/init.d/rpcbind start;chkconfig rpcbind off [[email protected] data]#  /etc/init.d/nfs start;chkconfig nfs off Starting NFS services:                                     [  OK  ] Starting NFS quotas:                                       [  OK  ] Starting NFS mountd:                                       [  OK  ] Starting NFS daemon:                                       [  OK  ] Starting RPC idmapd:                                       [  OK  ] [[email protected] data]# rpcinfo -p 127.0.0.1 program vers proto   port  service 100000    4   tcp    111  portmapper 100000    3   tcp    111  portmapper 100000    2   tcp    111  portmapper 100000    4   udp    111  portmapper

4.測試nfs

[[email protected] ~]# showmount -e 10.0.0.72 Export list for 10.0.0.72: /data 10.0.0.0/24 [[email protected] data]# showmount -e 10.0.0.71 Export list for 10.0.0.71: /data 10.0.0.0/24 說明nfs已經搭建成功。 掛載到其他伺服器測試： [[email protected] ~]# mount -t nfs 10.0.0.72:/data /data/data1 [[email protected] ~]# mount -t nfs 10.0.0.71:/data /data/data2 [[email protected] ~]# df -h Filesystem               Size  Used Avail Use% Mounted on /dev/mapper/centos-root  8.5G  2.5G  6.0G  30% / devtmpfs                 481M     0  481M   0% /dev tmpfs                    490M     0  490M   0% /dev/shm tmpfs                    490M  6.7M  484M   2% /run tmpfs                    490M     0  490M   0% /sys/fs/cgroup /dev/sda1                497M  120M  378M  25% /boot tmpfs                     98M     0   98M   0% /run/user/0 10.0.0.72:/data          923M  1.2M  874M   1% /data/data1 10.0.0.71:/data          8.4G  2.2G  5.8G  28% /data/data2

六、整合Heartbeat、DRBD和NFS服務

關於heartbeat的修改兩臺都要修改，保持一致。

1.修改heartbeat配置檔案haresources

master  drbddisk::data Filesystem::/dev/drbd0::/data::ext4 nfsd IPaddr::10.0.0.73/25/eth0

2.heartbeat預設沒有關於nfs的指令碼，我們要自己寫一個

#!/bin/bash   case $1 in start)   /etc/init.d/nfs restart   ;;   stop)   for proc in rpc.mountd rpc.rquotad nfsd nfsd   do  killall -9 $proc   done  ;;   esac  新增執行許可權： chmod +x /etc/ha.d/resource.d/nfsd

3.測試nfs高可用

[[email protected] ha.d]# /etc/init.d/heartbeat stop [[email protected] ha.d]# /etc/init.d/heartbeat stop [[email protected] ha.d]# /etc/init.d/heartbeat start Starting High-Availability services: INFO:  Resource is stopped Done. [[email protected] ha.d]# /etc/init.d/heartbeat start Starting High-Availability services: INFO:  Resource is stopped Done.

master各種狀態：


backup各種狀態：


檢視客戶端掛載情況


正常可用。
現在停掉主上的heartbeat，模擬down機狀態：

[[email protected] ha.d]# /etc/init.d/heartbeat stop Stopping High-Availability services: Done.

檢視備上各種狀態：


狀態是正常的。
檢視client能否正常寫入：


啟動主上heartbeat：

[[email protected] ha.d]# /etc/init.d/heartbeat start Starting High-Availability services: INFO:  Resource is stopped Done.

本次實驗到此結束。

目測，NFS必須需要2分鐘的延遲。可以通過編寫指令碼，通過服務端控制客戶端的umount和mount，減少nfs切換帶來的影響