實驗主機環境(redhat 6.5 x86_64bit)

實驗步驟

1.cobbler1，cobbler2安裝測試mfs-master 和 cgi-server
2.cobbler3，cobbler4安裝測試chunkserver
3.cobbler5安裝測試mfs-client
4.1.cobbler1，cobbler2安裝測試keepalived
5.cobbler1，cobbler2配置mfs高可用

1.cobbler1，cobbler2安裝測試mfs-master 和 cgi-server(兩臺主機操作相同)

安裝包本人已經上傳到百度雲盤，用到的rpm包如下：

yum install moosefs-cgi-3.0.80-1.x86_64.rpm moosefs-cgiserv-3.0.80-1.x86_64.rpm moosefs-master-3.0.80-1.x86_64.rpm -y
 

啟動服務：

mfsmaster start
mfscgiserv start

做好本地解析
/etc/hosts#cobbler1

192.168.1.8 mfsmaster

/etc/hosts#cobbler2

192.168.1.9 mfsmaster

此時進入/var/lib/mfs 可以看到 moosefs 所產生的資料: 
.mfsmaster.lock 檔案記錄正在執行的 mfsmaster 的主程序 
metadata.mfs, metadata.mfs.back MooseFS 檔案系統的元資料 metadata 的映象 
changelog.*.mfs 是 MooseFS 檔案系統元資料的改變日誌(每一個小時合併到 metadata.mfs中一次) 
Metadata 檔案的大小是取決於檔案數的多少(而不是他們的大小)。changelog 日誌的大小是取決於每小時操作的數目,但是這個時間長度(預設是按小時)是可配置的
 

2.cobbler3，cobbler4安裝測試chunkserver（兩臺主機同步）

yum install moosefs-chunkserver-3.0.80-1.x86_64.rpm -y#軟體安裝
mkdir /mnt/chunk123#建立儲存目錄，兩臺主機目錄要區分開
chown mfs.mfs /mnt/chunk123/ -R#修改目錄所有者
編輯配置檔案
[[email protected] ~]# sed -n '/#/!p' /etc/mfs/mfshdd.cfg
/mnt/chunk123

mfschunkserver start#啟動服務

做好解析之後訪問測試：
/etc/hosts#兩臺都做，可以指定mfsmaster為192.168.1.8或者192.168.1.9都可以，我這裡都指向了192.168.1.8

192.168.1.8 mfsmaster cobbler1

頁面訪問測試是否加入

3.cobbler5安裝測試mfs-client

yum install moosefs-client-3.0.80-1.x86_64.rpm -y

vim /etc/mfs/mfsmount.cfg#定義客戶端預設掛載
/mnt/mfs

vim /etc/hosts#做解析
192.168.1.8 mfsmaster cobbler1


mfsmount

cd /mnt/mfs/

[[email protected] mfs]# ls#以下列出的為手動建立的目錄，用來做資料測試的
test1  test2

[[email protected] mfs]# pwd
/mnt/mfs

mfssetgoal -r 1 test1#設定此目錄下資料的備份數為1
mfssetgoal -r 2 test2#設定此目錄下資料的備份數為2

cp /etc/passwd test1
cp /etc/passwd test2

根據上圖的結果可知cobbler4上存了兩份資料，分別來自test1和test2，我們去看一下：

4.1.cobbler1，cobbler2安裝測試keepalived

[[email protected] ~]# wget http://www.keepalived.org/software/keepalived-1.3.5.tar.gz
[[email protected] ~]# tar -zxf keepalived-1.3.5.tar.gz
[[email protected] ~]# cd keepalived-1.3.5
[[email protected] ~]# ./configure --prefix=/usr/local/keepalived --with-init=SYSV
[[email protected] ~]# make && make install
[[email protected] ~]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
[[email protected] ~]# chmod +x /etc/init.d/keepalived
[[email protected] ~]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[[email protected] ~]# ln -s /usr/local/keepalived/etc/keepalived/ /etc/
[[email protected] ~]# ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin/
[[email protected] ~]# scp -r /usr/local/keepalived/ cobbler2:/usr/local/

因為安裝目錄都scp到cobbler2上了，所以cobbler2上只需要執行以下操作即可：

    [[email protected] ~]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
    [[email protected] ~]# chmod +x /etc/init.d/keepalived
    [[email protected] ~]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
    [[email protected] ~]# ln -s /usr/local/keepalived/etc/keepalived/ /etc/
    [[email protected] ~]# ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin/

啟動服務看是否正常：#最好1和2都測試一下

[[email protected] ~]# /etc/init.d/keepalived start
Starting keepalived: 
[[email protected] ~]# ps -aux | grep keep
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
root      15261  0.0  0.0  37344   988 ?        Ss   23:19   0:00 keepalived -D
root      15263  0.0  0.1  37344  1952 ?        S    23:19   0:00 keepalived -D
root      15264  0.0  0.1  37344  1388 ?        S    23:19   0:01 keepalived -D
root      20677  0.0  0.0 103272   880 pts/0    S+   23:55   0:00 grep keep

5.cobbler1，cobbler2配置mfs高可用

cobbler1上

[[email protected] ~]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
  notification_email {
    [email protected]
    }

notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id MFS_HA_MASTER#作為mfs高可用的MASTER
}

vrrp_script chk_mfs {                          
  script "/etc/keepalived/ifalived_mfsmaster.sh"#不斷執行的服務檢測指令碼
  interval 2
  weight 2
}

vrrp_instance VI_1 {
  state MASTER#MASTER身份
  interface eth1
  virtual_router_id 51
  priority 100#優先順序，初始一定要比BACKUP高
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass 1111
}
  track_script {
    chk_mfs#上面定義的指令碼呼叫
}
virtual_ipaddress {
    192.168.1.99/24
}
notify_master "/etc/keepalived/mfs-data-sync.sh "#當它成為master之後要執行的指令碼，自己可以定義
}

cobbler2和cobbler1的大體配置一致，需要修改三處

服務監控指令碼（cobbler1和cobbler2一樣，要有執行許可權）：
[[email protected] ~]# cat /etc/keepalived/ifalived_mfsmaster.sh

#!/bin/bash
STATUS=`ps -C mfsmaster --no-header | wc -l`
if [ $STATUS -eq 0 ]
then
mfsmaster start
sleep 3
   if [ `ps -C mfsmaster --no-header | wc -l ` -eq 0 ]
    then
        killall -9 mfscgiserv
        killall -9 keepalived
   fi
fi

配置結束，因為到這裡高可用算是搭建起來了，後面將通過VIP來進行master切換，所以修改以下解析（五臺主機都做）

/etc/hosts
192.168.1.99 mfsmater

啟動所有服務之後會看到VIP在keepalived 的 master主機上

訪問頁面測試一下：

因為在mfsmaster切換的時候可能會出現資料不同步的問題，所以我們需要寫個資料同步的指令碼來做這件事情：

[[email protected] ~]# cat /etc/keepalived/mfs-data-sync.sh

#!/bin/bash
STATUS=`ip addr|grep 192.168.1.99|awk -F" " '{print $2}'|cut -d"/" -f1`
if [ $STATUS == 192.168.1.99 ];then
   mfsmaster stop
   /bin/rm -f /var/lib/mfs/*
   /usr/bin/rsync -e "ssh -p22" -avpgolr 192.168.1.8:/var/lib/mfs/* /var/lib/mfs/
   /usr/sbin/mfsmetarestore -m
   mfsmaster -ai
   sleep 3
   echo "this server has become the master of MFS"
   if [ $STATUS != 192.168.1.99 ];then
   echo "this server is still MFS's slave"
   fi
fi

然後將這個指令碼路徑加到剛才keepalived配置的notify_master引數後面，這樣每次進行master切換之後都會執行資料同步，如：

notify_master "/etc/keepalived/mfs-data-sync.sh "#當它成為master之後要執行的資料同步指令碼

後面就可以通過在一邊終止掉keepalived程序來進行故障模擬然後觀察vip的漂移以及這個過程的檔案系統可訪問性測試高可用。