###########叢集+高可用+後端儲存########
一.知識梳理
1.什麼是叢集？
叢集是一組相互獨立的、通過高速網路互聯的計算機，它們構成了一個組，並以單一系統的模式加以管理。一個客戶與叢集相互作用時，叢集像是一個獨立的伺服器。叢集配置是用於提高可用性和可縮放性。
2.分類
高可用性(High Available Cluster), 例:Linux-HA
負載均衡(Load balancing Cluster), 例:LVS、MOSIX，在企業中經常用於處理網路流量負載。
高效能運算(High Performance Computing), 例:Beowulf

高可用叢集：高可用性叢集使伺服器系統的執行速度和響應速度儘可能快。它們經常利用在多臺機器上執行的冗餘節點和服務，用來相互跟蹤。如果某個節點失敗，它的替補者將在幾秒鐘或更短時間內接管它的職責。因此，對於使用者而言,叢集永遠不會停機。

3.工作模式

HA叢集維護節點的高可用，只有一個節點是真正工作，另一個節點處於熱備狀態，兩個叢集節點會有一根網路直連，還有心跳線，corosyc（心跳引擎,用於檢測主機的可用性）， 每個主機都有自己的ip，但是對外使用的是vip，vip、servers、filesystem構成了一個資源組，當一個節點down掉時，整個資源組就會遷移到另一各節點，由於vip不變，在客戶對於伺服器故障時沒有感覺的，這裡需要使用ARP協議，mac地址發生了變化。

4.通過fence實現對兩個節點的控制管理

在叢集中為了防止伺服器出現“腦裂”的現象，叢集中一般會新增Fence裝置，如果兩個節點之間的網路斷了，或者正在工作的節點由於負載過高down掉了，corosync檢測不到心跳資訊就會通知另一個節點接管工作，此時如果第一個節點恢復了，而由於網路或者其他原因，對方對沒有檢測到，這時就會發生資源爭搶，兩個節點同時對檔案系統進行操作，就會導致檔案系統出錯，因此需要fence進行控制，當一個節點出現問題時，fence就會直接切斷電源（不是重啟，因為重啟還是會將記憶體的資訊寫入磁碟，而斷電就直接將記憶體清空），這樣就可避免資源爭搶。

二.建立叢集
配置環境：
selinux     Enforcing           vim /etc/sysconfig/selinux
date        時間同步             ntpdate  
iptables    關閉火牆             iptables -F
NetwortManger 關閉              

1.建立節點
在test1和test2上配置好yum源,實驗中test1即是管理主機，又是叢集節點，test2是另外一個叢集節點，需要在tets1中做好解析
test1的ip為172.25.66.10 test2的ip為172.25.66.11

[

[email protected] ~]# vim /etc/yum.repos.d/rhel-source.repo
[rhel-source]
name=Red Hat Enterprise Linux $releasever - $basearch - Source
baseurl=http://172.25.66.250/rhel6.5
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[HighAvailability]
name=HighAvailability
baseurl=http://172.25.66.250/rhel6.5/HighAvailability
gpgcheck=0
[LoadBalancer]
name=LoadBalancer
baseurl=http://172.25.66.250/rhel6.5/LoadBalancer
gpgcheck=0
[ResilientStorage]
name=ResilientStorage
baseurl=http://172.25.66.250/rhel6.5/ResilientStorage
gpgcheck=0
[ScalableFileSystem]
name=ScalableFileSystem
baseurl=http://172.25.66.250/rhel6.5/ScalableFileSystem
gpgcheck=0
[[email protected] ~]# vim /etc/yum.repos.d/rhel-source.repo
[rhel-source]
name=Red Hat Enterprise Linux $releasever - $basearch - Source
baseurl=http://172.25.66.250/rhel6.5
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[HighAvailability]
name=HighAvailability
baseurl=http://172.25.66.250/rhel6.5/HighAvailability
gpgcheck=0
[LoadBalancer]
name=LoadBalancer
baseurl=http://172.25.66.250/rhel6.5/LoadBalancer
gpgcheck=0
[ResilientStorage]
name=ResilientStorage
baseurl=http://172.25.66.250/rhel6.5/ResilientStorage
gpgcheck=0
[ScalableFileSystem]
name=ScalableFileSystem
baseurl=http://172.25.66.250/rhel6.5/ScalableFileSystem
gpgcheck=0
[[email protected] ~]# yum install -y  ricci luci    ##安裝luci，使用web方式配置叢集
[[email protected] ~]# passwd ricci            ##置ricci密碼
[[email protected] ~]# /etc/init.d/ricci start        ##啟動ricci，監聽埠埠為11111
[[email protected] ~]# /etc/init.d/luci    start    ##啟動luci，監聽埠埠為8084
[[email protected] ~]# chkconfig luci on        ##服務開機自啟
[[email protected] ~]# chkconfig ricci on

[[email protected] ~]# yum install -y ricci
[[email protected] ~]# passwd ricci
[[email protected] ~]# /etc/init.d/ricci start
[[email protected] ~]# chkconfig ricci on
[[email protected] ~]# clustat             ##測試命令
Cluster Status for test @ Mon Jul 24 05:15:48 2017
Member Status: Quorate

 Member Name                                 ID   Status
 ------ ----                                 ---- ------
 test1                                         1 Online, Local
 test2                                         2 Online
測試：

在網頁訪問https：//172.25.66.10：8084  建立節點

將key檔案傳到節點的/etc/cluster目錄下就可以通過luci給叢集新增fence了 點選Fence Devices新增fence，有多種fence裝置可供選擇這裡選的是Fence virt（MulticastMode ），然後點選Nodes裡的節點為叢集新增fence

（在建立節點時，如果出現錯誤，根據頁面提示進行排錯，出錯原因可能有地址解析，服務開啟，主機名設定）
節點建立成功：

三.安裝fence_virtd、建立fence裝置

1）在真實的物理機上有一個libvirtd服務，是管理虛擬機器和其他虛擬化功能，他包含一個API庫，一個守護程序（libvirtd）和一個命令列工具(virsh)，如果將這個服務關閉了，那麼使用者就不能通過virsh管理虛擬機器。
[[email protected] iso]# yum installfence-virtd-multicast.x86_64 fence-virtd-libvirt.x86_64 fence-virtd.x86_64 -y
[[email protected] Desktop]# fence_virtd -c        ##建立fence服務
Module search path [/usr/lib64/fence-virt]:

Available backends:
    libvirt 0.1
Available listeners:
    multicast 1.2

Listener modules are responsible for accepting requests
from fencing clients.

Listener module [multicast]:                     ##多播監聽

The multicast listener module is designed for use environments
where the guests and hosts may communicate over a network using
multicast.

The multicast address is the address that a client will use to
send fencing requests to fence_virtd.

Multicast IP Address [225.0.0.12]:                 ##使用預設的多播地址

Using ipv4 as family.

Multicast IP Port [1229]:                     ##預設埠

Setting a preferred interface causes fence_virtd to listen only
on that interface.  Normally, it listens on all interfaces.
In environments where the virtual machines are using the host
machine as a gateway, this *must* be set (typically to virbr0).
Set to 'none' for no interface.

Interface [br0]:                         ##fence介面

The key file is the shared key information which is used to
authenticate fencing requests.  The contents of this file must
be distributed to each physical host and virtual machine within
a cluster.

Key File [/etc/cluster/fence_xvm.key]:                 ##虛擬機器會通過key進行校驗

Backend modules are responsible for routing requests to
the appropriate hypervisor or management layer.

Backend module [libvirt]:

Configuration complete.                        ##配置完成，顯示資訊

=== Begin Configuration ===
fence_virtd {
    listener = "multicast";
    backend = "libvirt";
    module_path = "/usr/lib64/fence-virt";
}

listeners {
    multicast {
        key_file = "/etc/cluster/fence_xvm.key";
        address = "225.0.0.12";
        interface = "br0";
        family = "ipv4";
        port = "1229";
    }

}

backends {
    libvirt {
        uri = "qemu:///system";
    }

}

=== End Configuration ===
Replace /etc/fence_virt.conf with the above [y/N]? y        ##生成新的配置檔案
[[email protected] Desktop]# mkdir   /etc/cluster
[[email protected] Desktop]#dd if=/dev/urandom of=/etc/cluster/fence_xvm.key bs=128 count=1
[[email protected] Desktop]# systemctl restart fence_virtd.service    
##/dev/urandom會產生隨機數，利用此來產生key
[[email protected] Desktop]# scp /etc/cluster/fence_xvm.key [email protected]:/etc/cluster/
[[email protected] Desktop]# scp /etc/cluster/fence_xvm.key [email protected]:/etc/cluster/


2）在網頁訪問https：//172.25.66.10：8084  建立fence devices ，新增節點，這裡選擇vmfence（xvm Virtual Machine Fencing）其中Domain填寫虛擬機器的名字或UUID
叢集內部只知道主機名，而fance所在的物理機知道的是虛擬機器的名字，因此要將它們做一個對映，寫入UUID
在網頁寫入的內容會被直接記錄在/etc/cluster/cluster.conf檔案中

測試節點建立成功：    

[[email protected] cluster]# fence_node test2        ##test2重啟
fence test2 success

   

[[email protected] ~]# ip link set eth0 down        ##將test2的eth0的down掉,會直接斷電，test2重啟,應用fence的作用


新增資源組：新增Failover Domain ——————> Resource（加入IP Address,Script)————————>Service Groups(apache)

將vip和apache服務加入sources，必須先加ip在加服務，然後在連線節點主機上安裝apache服務並開啟服務，此時在瀏覽器就可以訪問到172.25.66.100的apache的預設網頁

Prioritized設定優先順序

Restricted設定只在勾選節點中執行

No Failback設定當優先順序高的節點恢復時不會接管正在執行的服務，不回切

然後在Resource中新增資源這裡新增虛擬ip和httpd的指令碼（叢集要安裝httpd）：

Automatically Start This Service設定自動開啟服務

Run Exclusive 設定當執行此服務時不再執行其他服務,獨佔

點選 Add Resource 增加Resource中的資源這裡新增的是上一步配置的虛擬IP和httpd指令碼，先加ip後加指令碼

[[email protected] cluster]# /etc/init.d/httpd start
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 172.25.66.10 for ServerName
                                                           [  OK  ]

[[email protected] ~]# /etc/init.d/httpd start
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 172.25.66.11 for ServerName

                                                           [  OK  ]

為了實驗效果更加明顯，這裡我們將兩個節點apache預設網頁改為不同內容
[[email protected] cluster]# vim /var/www/html/index.html
[[email protected] cluster]# cat /var/www/html/index.html

<h1>test1-www.westos.org</h1>

當test1和test2httpd服務正常時，apache在test1時工作，可用clustat檢視狀態

當test1節點的apache服務關閉時，test2接替工作，同時接管vip

[[email protected] cluster]# /etc/init.d/httpd stop

Stopping httpd:                                            [  OK  ]
[[email protected] cluster]# clustat
Cluster Status for test @ Wed Jul 26 16:58:11 2017
Member Status: Quorate

 Member Name                         ID   Status
 ------ ----                         ---- ------
 test1                                   1 Online, Local, rgmanager
 test2                                   2 Online, rgmanager

 Service Name               Owner (Last)               State         
 ------- ----               ----- ------               -----         

 service:apache             test2                      started

 

[[email protected] ~]#  echo c > /proc/sysrq-trigger  Write failed: Broken pipe

#######################################################################

二.建立共享裝置iscsi+節點同步
開啟一個虛擬機器(server3)作為共享儲存，加上8G的硬碟，客戶端(test1和test2）安裝iscsi，服務端（server3）安裝scsi，通過issic和scsi實現網路儲存
[[email protected] ~]# yum install -y scsi-*
[[email protected] ~]# yum install -y iscsi-*
[[email protected] ~]# yum install -y iscsi-*

[[email protected] ~]# vim /etc/tgt/targets.conf         
 38 <target iqn.2017-07.com.example:server.target1>
 39     backing-store /dev/vdb
 40     initiator-address 172.25.66.10
 41     initiator-address 172.25.66.11
 42 </target>
[[email protected] ~]# iscsiadm -m discovery -t st -p 172.25.66.3    ##發現服務端iscsi共享儲存
Starting iscsid:                                           [  OK  ]
172.25.66.3:3260,1 iqn.2017-07.com.example:server.target1    ##登入服務端iscsi共享儲存
[[email protected] ~]# iscsiadm -m node -l
Logging in to [iface: default, target: iqn.2017-07.com.example:server.target1, portal[[email protected] ~]: 172.25.66.3,3260] (multiple)
Login to [iface: default, target: iqn.2017-07.com.example:server.target1, portal: 172.25.66.3,3260] successful.



[[email protected] ~]# iscsiadm -m discovery -t st -p 172.25.66.3        
Starting iscsid:                                           [  OK  ]
172.25.66.3:3260,1 iqn.2017-07.com.example:server.target1        
[[email protected] ~]# iscsiadm -m node -l                    ##test2和tset3同時登陸服務端共享儲存
Logging in to [iface: default, target: iqn.2017-07.com.example:server.target1, portal: 172.25.66.2,3260] (multiple)
Login to [iface: default, target: iqn.2017-07.com.example:server.target1, portal: 172.25.66.3,3260] successful.

[[email protected] ~]# fdisk -cu /dev/sda                ##磁碟分割槽，在tets1上對共享儲存進行操作

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4):
Value out of range.
Partition number (1-4): 1
First sector (2048-16777215, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-16777215, default 16777215):
Using default value 16777215

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): wq
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

[[email protected] ~]# fdisk -l                    ##在test2上可以檢視到儲存裝置分割槽同步

Disk /dev/vda: 9663 MB, 9663676416 bytes
16 heads, 63 sectors/track, 18724 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a7cda

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *           3        1018      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/vda2            1018       18725     8924160   8e  Linux LVM
Partition 2 does not end on cylinder boundary.

Disk /dev/mapper/VolGroup-lv_root: 8170 MB, 8170504192 bytes
255 heads, 63 sectors/track, 993 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/VolGroup-lv_swap: 964 MB, 964689920 bytes
255 heads, 63 sectors/track, 117 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sda: 8589 MB, 8589934592 bytes
64 heads, 32 sectors/track, 8192 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xbab2f040

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               2        8192     8387584   8e  Linux LVM
[[email protected] ~]#pvcreate /dev/sda1
[[email protected] ~]# vgs                        ##一個節點建立物理分割槽，邏輯卷組，邏輯卷，另一個節點同步更新
  VG        #PV #LV #SN Attr   VSize VFree
  VolGroup    1   2   0 wz--n- 8.51g    0
  clustervg   1   0   0 wz--nc 8.00g 8.00g
[[email protected] ~]# lvcreate -L +2G -n demo clustervg
  Error locking on node test2: Unable to resume clustervg-demo (253:2)
  Failed to activate new LV.
注意檢視兩個節點的分割槽資訊必須一致，當兩個結點的資訊不一致時，要及時排錯，可能會遇到下面兩種情況
[[email protected] ~]# cat /proc/partitions        ##分割槽大小不一致
major minor  #blocks  name

 252        0    9437184 vda
 252        1     512000 vda1
 252        2    8924160 vda2
 253        0    7979008 dm-0
 253        1     942080 dm-1
   8        0    8388608 sda
   8        1    8387584 sda1


[[email protected] ~]# cat /proc/partitions
major minor  #blocks  name

 252        0    9437184 vda
 252        1     512000 vda1
 252        2    8924160 vda2
 253        0    7979008 dm-0
 253        1     942080 dm-1
   8        0    8388608 sda
   8        1    2097152 sda1
[[email protected] ~]# partprobe             ##用partprobe命令重新整理下
Warning: WARNING: the kernel failed to re-read the partition table on /dev/vda (Device or resource busy).  As a result, it may not reflect all of your changes until after reboot.
[[email protected] ~]# cat /proc/partitions         
major minor  #blocks  name

 252        0    9437184 vda
 252        1     512000 vda1
 252        2    8924160 vda2
 253        0    7979008 dm-0
 253        1     942080 dm-1
   8        0    8388608 sda
   8        1    8387584 sda1
[[email protected] ~]# lvcreate -L +2G -n demo clustervg    ##重新整理後資訊同步，但還是無法建立邏輯卷
  Error locking on node test2: device-mapper: create ioctl on clustervg-demo failed: Device or resource busy
  Failed to activate new LV.


[[email protected] ~]# fence_node test2            ##考慮節點資訊不同步，直接將test2斷電重啟
fence test2 success

[[email protected] ~]# lvcreate -L +2G -n demo clustervg    ##建立邏輯捲成功
  Logical volume "demo" created

[[email protected] ~]# mkfs.ext4 /dev/clustervg/demo        ##格式化為ext4格式，ext4只能進行本地共享，無法實現網路共享
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
131072 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=536870912
16 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912

Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

磁碟可以格式化為本地系統或者叢集檔案系統

1.磁碟格式化為本地檔案系統

節點執行pvs，vgs，lvs。命令進行同步

最後在網頁進行配置：

點選Resource新增資源：選擇File System

在之前的Resource Group中的apache組新增資源順序為虛擬IP -> File System ->httpd指令碼

因為stest2主機的優先順序高所以先掛載在test主機，apache在test1是工作

[[email protected] ~]# lvextend -L +2G /dev/clustervg/demo         ##擴充套件邏輯卷
  Extending logical volume demo to 4.00 GiB
  Logical volume demo successfully resized
[[email protected] ~]# lvs
  LV      VG        Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_root VolGroup  -wi-ao----   7.61g                                             
  lv_swap VolGroup  -wi-ao---- 920.00m                                             
  demo    clustervg -wi-a-----   4.00g 

2.磁碟格式化為叢集檔案系統

[[email protected] ~]# mkfs.gfs2 -j 3 -p lock_dlm -t test:mygfs2 /dev/clustervg/demo     ##格式化為gfs2格式，gfs2格式可以做為網路共享，並且可以實現同時掛在，同時寫入
This will destroy any data on /dev/clustervg/demo.
It appears to contain: symbolic link to `../dm-2'

Are you sure you want to proceed? [y/n] y

Device:                    /dev/clustervg/demo
Blocksize:                 4096
Device Size                4.00 GB (1048576 blocks)
Filesystem Size:           4.00 GB (1048575 blocks)
Journals:                  3
Resource Groups:           16
Locking Protocol:          "lock_dlm"
Lock Table:                "test:mygfs2"
UUID:                      0085b03e-a622-0aec-f936-f5dca4333740

在/etc/fstab中寫入（UUID可以通過blkid檢視,所用節點都做）

[[email protected] html]# vim /etc/fstab        ##寫在配置檔案中，開機自啟，由於gfs2是網路共享儲存。必須先啟動網路，在掛載
UUID="0085b03e-a622-0aec-f936-f5dca4333740" /var/www/html gfs2  _netdev,defaults 0 0

[[email protected] html]# gfs2_tool journals /dev/clustervg/demo     ##結點日誌
journal2 - 128MB
journal1 - 128MB
journal0 - 128MB
3 journal(s) found.
[[email protected] html]# gfs2_jadd -j 3 /dev/clustervg/demo         ##增加3個結點
Filesystem:            /var/www/html
Old Journals           3
New Journals           6
[[email protected] html]# gfs2_tool journals /dev/clustervg/demo
journal2 - 128MB
journal3 - 128MB
journal1 - 128MB
journal5 - 128MB
journal4 - 128MB
journal0 - 128MB
6 journal(s) found.
[[email protected] html]# df -h
Filesystem                    Size  Used Avail Use% Mounted on   
/dev/mapper/VolGroup-lv_root  7.5G  1.1G  6.1G  15% /
tmpfs                         246M   32M  215M  13% /dev/shm
/dev/vda1                     485M   33M  427M   8% /boot
/dev/mapper/clustervg-demo    4.0G  776M  3.3G  19% /var/www/html
[[email protected] html]# lvextend -L +4G /dev/clustervg/demo
  Extending logical volume demo to 8.00 GiB
  Insufficient free space: 1024 extents needed, but only 1023 available
[[email protected] html]# lvextend -l +1023 /dev/clustervg/demo
  Extending logical volume demo to 8.00 GiB
  Logical volume demo successfully resized
[[email protected] html]# lvs
  LV      VG        Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_root VolGroup  -wi-ao----   7.61g                                             
  lv_swap VolGroup  -wi-ao---- 920.00m                                             
  demo    clustervg -wi-ao----   8.00g                                             

[[email protected] html]# vgs
  VG        #PV #LV #SN Attr   VSize VFree
  VolGroup    1   2   0 wz--n- 8.51g    0
  clustervg   1   1   0 wz--nc 8.00g    0
[[email protected] html]# gfs2_grow /dev/clustervg/demo         ##擴充套件
FS: Mount Point: /var/www/html
FS: Device:      /dev/dm-2
FS: Size:        1048575 (0xfffff)
FS: RG size:     65533 (0xfffd)
DEV: Size:       2096128 (0x1ffc00)
The file system grew by 4092MB.
gfs2_grow complete.
[[email protected] html]# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root  7.5G  1.1G  6.1G  15% /
tmpfs                         246M   32M  215M  13% /dev/shm
/dev/vda1                     485M   33M  427M   8% /boot
/dev/mapper/clustervg-demo    7.8G  776M  7.0G  10% /var/www/html