CentOS7實戰Kubernetes部署
CentOS 7 搭建GlusterFS
實驗需求：
4臺機器安裝GlusterFS組成一個集群
客戶端把docker registry存儲到文件系統裏
4個節點的硬盤空間不整合成一個硬盤空間，要求每個節點都存儲一份，保證數據安全
環境規劃
-----------------------------------------------------
server
-----------------------------------------------------
node1:192.168.56.11 主機名:glusterfs1
-----------------------------------------------------
node2:192.168.56.12 主機名:glusterfs2
-----------------------------------------------------
node3:192.168.56.13 主機名:glusterfs3
-----------------------------------------------------
node4:192.168.56.14 主機名:glusterfs4
-----------------------------------------------------
client
-----------------------------------------------------
192.168.56.20 主機名:client
-----------------------------------------------------
OS: BCLinux release 7.2 (Mars)
實驗前準備：
所有主機關閉防火墻，SElinux修改hosts文件，能夠互相解析
192.168.56.11 glusterfs1
192.168.56.12 glusterfs2
192.168.56.13 glusterfs3
192.168.56.14 glusterfs4
192.168.56.20 client
。SSH互信（最好選擇主機名）
-->ssh-keygen -t rsa
-->ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
-->ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
-->ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
-->ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
-->ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
一、安裝
服務端
1.在glusterfs {1-4}節點上安裝GlusrerFS軟件包
。Glusterfs下載地址：
http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-api-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-cli-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-client-xlators-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-debuginfo-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-fuse-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-geo-replication-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-libs-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-rdma-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-server-3.7.17-1.el7.x86_64.rpm
安裝：
-->yum install libibverbs librdmacm -y
-->wget http://dl.fedoraproject.org/pub/epel/6/x86_64//userspace-rcu-0.7.7-1.el6.x86_64.rpm
-->rpm -ivh userspace-rcu-0.7.7-1.el6.x86_64.rpm
-->yum localinstall *.rpm -y
----------------------------------------------------------------------------------------------
報錯：
需要：liburcu-bp.so.1()(64bit)
錯誤：依賴檢測失敗：
libibverbs.so.1()(64bit) 被 glusterfs-rdma-3.7.17-1.el7.x86_64 需要
libibverbs.so.1(IBVERBS_1.0)(64bit) 被 glusterfs-rdma-3.7.17-1.el7.x86_64 需要
libibverbs.so.1(IBVERBS_1.1)(64bit) 被 glusterfs-rdma-3.7.17-1.el7.x86_64 需要
librdmacm.so.1()(64bit) 被 glusterfs-rdma-3.7.17-1.el7.x86_64 需要
librdmacm.so.1(RDMACM_1.0)(64bit) 被 glusterfs-rdma-3.7.17-1.el7.x86_64 需要
liburcu-bp.so.1()(64bit) 被 glusterfs-server-3.7.17-1.el7.x86_64 需要
liburcu-cds.so.1()(64bit) 被 glusterfs-server-3.7.17-1.el7.x86_64 需要
解決方法：
-->yum install libibverbs librdmacm -y
-->wget http://dl.fedoraproject.org/pub/epel/6/x86_64//userspace-rcu-0.7.7-1.el6.x86_64.rpm
-->rpm -ivh userspace-rcu-0.7.7-1.el6.x86_64.rpm
----------------------------------------------------------------------------------------------
3.userspace-rcu下載地址：
-->wget http://dl.fedoraproject.org/pub/epel/7/x86_64/u/userspace-rcu-0.7.16-1.el7.x86_64.rpm
-->rpm -ivh serspace-rcu-0.7.16-1.el7.x86_64.rpm
安裝：
-->yum localinstall *.rpm
。查看安裝的版本
-->glusterfs -V
glusterfs 3.7.17 built on Nov 3 2016 14:29:48
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
。啟動glusterfs服務
-->systemctl enable glusterd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/glusterd.service to /usr/lib/systemd/system/glusterd.service.
-->systemctl start gluterd.service
-->systemctl status glusterd.service
● glusterd.service - GlusterFS, a clustered file-system server
Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2017-07-09 16:20:51 CST; 30s ago
Process: 4684 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 4685 (glusterd)
CGroup: /system.slice/glusterd.service
└─4685 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
Jul 09 16:20:48 glusterfs1 systemd[1]: Starting GlusterFS, a clustered file-system server...
Jul 09 16:20:51 glusterfs1 systemd[1]: Started GlusterFS, a clustered file-system server.
2.在glusterfs1節點上配置整個GlusterFS集群，把各個節點加入到集群
-->gluster peer probe glusterfs1
peer probe: success. Probe on localhost not needed
-->gluster peer probe glusterfs2
peer probe: success.
-->gluster peer probe glusterfs3
peer probe: success.
-->gluster peer probe glusterfs4
peer probe: success.
3.查看節點狀態
-->gluster peer status
Number of Peers: 3
Hostname: glusterfs2
Uuid: 9149c992-e86c-436d-8da9-b2a7245289cb
State: Peer in Cluster (Connected)
Hostname: glusterfs3
Uuid: dcbe562d-5bbd-4ee8-8855-c48428bba7ef
State: Peer in Cluster (Connected)
Hostname: glusterfs4
Uuid: 677514f3-7c3c-4b8e-b93d-db3941485166
State: Peer in Cluster (Connected)
4.在glusterfs{1-4}上創建數據存儲目錄
-->mkdir -p /usr/local/share/models
5.在glusterfs1上創建GlusterFS磁盤
註意：
加上replica 4就是4個節點中，每個節點都要把數據存儲一次，就是一個數據存儲4份，每個節點一份
如果不加replica 4，就是4個節點的磁盤空間整合成一個硬盤，
-->gluster volume create models replica 4 glusterfs1:/usr/local/share/models glusterfs2:/usr/local/share/models glusterfs3:/usr/local/share/models glusterfs4:/usr/local/share/models force
volume create: models: success: please start the volume to access data
6.啟動
-->gluster volume start models
volume start: models: success
----------------------------------------------------------------------------
客戶端
1.部署GlusterFS客戶端並mount GlusterFS文件系統
-->mkdir /usr/local/src/rpm
方法同服務器端（下載rpm安裝)
-->yum install libibverbs librdmacm -y
-->wget http://dl.fedoraproject.org/pub/epel/6/x86_64//userspace-rcu-0.7.7-1.el6.x86_64.rpm
-->rpm -ivh userspace-rcu-0.7.7-1.el6.x86_64.rpm
-->yum localinstall *.rpm -y
最好按下面方法安裝客戶端包
-->rpm -ivh glusterfs-client-xlators-3.7.17-1.el7.x86_64.rpm glusterfs-fuse-3.7.17-1.el7.x86_64.rpm
-->mkdir -p /mnt/models
-->mount -t glusterfs -o ro glusterfs1:models /mnt/models/
2.查看效果
-->df -h
df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda2 ext4 50G 1.5G 46G 4% /
devtmpfs devtmpfs 904M 0 904M 0% /dev
tmpfs tmpfs 913M 0 913M 0% /dev/shm
tmpfs tmpfs 913M 8.8M 904M 1% /run
tmpfs tmpfs 913M 0 913M 0% /sys/fs/cgroup
/dev/sda6 ext4 35G 4.7G 29G 15% /var
/dev/sda3 ext4 9.8G 37M 9.2G 1% /home
/dev/sda1 ext4 477M 94M 354M 21% /boot
tmpfs tmpfs 183M 0 183M 0% /run/user/0
glusterfs1:models fuse.glusterfs 50G 1.5G 46G 4% /mnt/models
其他操作命令
。刪除GlusterFS磁盤
-->gluster volume stop models 先停止
-->gluster volume delete models 再刪除
。卸載GlusterFS磁盤
-->gluster peer detach glusterfs4
。ACL訪問控制
-->gluster volume set models auth.allow 192.168.56.*,10.0.1.*
volume set: success
。添加GlusterFS節點
-->gluster peer probe sc2-log5
-->gluster peer probe sc2-log6
-->gluster volume add-brick models sc2-log5:/data/gluster sc2-log6:/data/gluster
。遷移GlusterFS數據
-->gluster volume remove-brick models sc2-log1:/usr/local/share/models sc2-log5:/usr/local/share/models start
-->gluster volume remove-brick models sc2-log1:/usr/local/share/models sc2-log5:/usr/local/share/models status
-->gluster volume remove-brick models sc2-log1:/usr/local/share/models sc2-log5:/usr/local/share/models commit
。修復GlusterFS數據(在節點1宕機的情況下)
-->gluster volume replace-brick models sc2-log1:/usr/local/share/models sc2-log5:/usr/local/share/models commit -force
-->gluster volume heal models full
CentOS7實戰Kubernetes部署