基於docker的ceph叢集
阿新 • • 發佈：2019-01-04
ceph部署環境
| 主機名稱 | 內網IP |
|---|---|
| node01 | 192.168.0.1 |
| node02 | 192.168.0.2 |
| node03 | 192.168.0.3 |
虛機開啟selinux：
vi /etc/selinux/config設定：
SELINUX=enforcing設定目錄許可權：
mkdir -p /etc/ceph
mkdir -p /var/lib/ceph
chcon -Rt svirt_sandbox_file_t /etc/ceph
chcon -Rt svirt_sandbox_file_t /var/lib/ceph如果系統只有一個數據盤，並且資料盤已經分割槽，需要對資料盤進行分割槽刪除，並進行格式化處理。
格式化命令如下：
mkfs.ext4 /dev/vdb下面我們採用docker進行安裝，ceph/daemon的版本為tag-build-master-jewel-centos-7。
node01啟動monitor
啟動monitor：
docker run -d --name=mon --net=host \
--restart=always \
-v /etc/ceph:/etc/ceph \
-v /var/lib/ceph:/var/lib/ceph \
-e MON_IP=192.168.0.1 \
-e CEPH_PUBLIC_NETWORK=192.168.0.0/24 \
ceph/daemon:tag-build-master-jewel-centos-7 檢視ceph狀態：
docker exec b3cc55582498 ceph -s執行結果：
# docker exec b3cc55582498 ceph -s
cluster 05527e2d-5d80-4c85-8d35-7dcddafa197e
health HEALTH_ERR
no osds
monmap e1: 1 mons at {iZbp1isotv99f45cg37lxgZ=192.168.0.1:6789/0}
election epoch 3, quorum 0 iZbp1isotv99f45cg37lxgZ
osdmap e1: 0 拷貝node01配置檔案到其他機器
scp -r /etc/ceph/ceph* [email protected]192.168.0.2:/etc/ceph/
scp -r /var/lib/ceph/bootstrap-* [email protected]192.168.0.2:/var/lib/ceph/
scp -r /etc/ceph/ceph* [email protected]192.168.0.3:/etc/ceph/
scp -r /var/lib/ceph/bootstrap-* [email protected]192.168.0.3:/var/lib/ceph/其他節點啟動monitor
在node02上執行：
docker run -d --name=mon --net=host \
--restart=always \
-v /etc/ceph:/etc/ceph \
-v /var/lib/ceph:/var/lib/ceph \
-e MON_IP=192.168.0.2 \
-e CEPH_PUBLIC_NETWORK=192.168.0.0/24 \
ceph/daemon:tag-build-master-jewel-centos-7 mon在node03上執行：
docker run -d --name=mon --net=host \
--restart=always \
-v /etc/ceph:/etc/ceph \
-v /var/lib/ceph:/var/lib/ceph \
-e MON_IP=192.168.0.3 \
-e CEPH_PUBLIC_NETWORK=192.168.0.0/24 \
ceph/daemon:tag-build-master-jewel-centos-7 mon檢視ceph狀態：
docker exec b3cc55582498 ceph -s執行結果：
#docker exec b3cc55582498 ceph -s
cluster 230a93b6-3876-4d69-bffe-c0c054c49653
health HEALTH_ERR
no osds
monmap e3: 3 mons at {iZbp1isotv99f45cg37lxfZ=192.168.0.1:6789/0,iZbp1isotv99f45cg37lxiZ=192.168.0.2:6789/0,iZbp1isotv99f45cg37lxjZ=192.168.0.3:6789/0}
election epoch 6, quorum 0,1,2 iZbp1isotv99f45cg37lxjZ,iZbp1isotv99f45cg37lxfZ,iZbp1isotv99f45cg37lxiZ
osdmap e1: 0 osds: 0 up, 0 in
flags sortbitwise,require_jewel_osds
pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
0 kB used, 0 kB / 0 kB avail
64 creating在所有節點部署osd
docker run -d --name=osd1 --net=host \
--restart=always \
-v /etc/ceph:/etc/ceph \
-v /var/lib/ceph/:/var/lib/ceph/ \
-v /dev/:/dev/ --privileged=true \
-e OSD_FORCE_ZAP=1 \
-e OSD_DEVICE=/dev/vdb \
ceph/daemon:tag-build-master-jewel-centos-7 osd_ceph_disk檢視ceph狀態：
docker exec b3cc55582498 ceph -s執行結果：
# docker exec b3cc55582498 ceph -s
cluster 230a93b6-3876-4d69-bffe-c0c054c49653
health HEALTH_ERR
37 pgs are stuck inactive for more than 300 seconds
27 pgs degraded
3 pgs peering
37 pgs stuck inactive
64 pgs stuck unclean
27 pgs undersized
monmap e3: 3 mons at {iZbp1isotv99f45cg37lxfZ=192.168.0.1:6789/0,iZbp1isotv99f45cg37lxiZ=192.168.0.2:6789/0,iZbp1isotv99f45cg37lxjZ=192.168.0.3:6789/0}
election epoch 6, quorum 0,1,2 iZbp1isotv99f45cg37lxjZ,iZbp1isotv99f45cg37lxfZ,iZbp1isotv99f45cg37lxiZ
osdmap e10: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v11: 64 pgs, 1 pools, 0 bytes data, 0 objects
35320 kB used, 699 GB / 699 GB avail
34 creating
27 active+undersized+degraded
3 creating+peering在node01部署mds
docker run -d --name=mds --net=host \
--restart=always \
-v /etc/ceph:/etc/ceph \
-v /var/lib/ceph/:/var/lib/ceph/ \
-e CEPHFS_CREATE=1 \
ceph/daemon:tag-build-master-jewel-centos-7 mds
檢視ceph狀態：
docker exec b3cc55582498 ceph -s執行結果：
# docker exec b3cc55582498 ceph -s
cluster 230a93b6-3876-4d69-bffe-c0c054c49653
health HEALTH_WARN
4 pgs peering
monmap e3: 3 mons at {iZbp1isotv99f45cg37lxfZ=10.135.204.114:6789/0,iZbp1isotv99f45cg37lxiZ=10.135.204.132:6789/0,iZbp1isotv99f45cg37lxjZ=10.135.204.107:6789/0}
election epoch 6, quorum 0,1,2 iZbp1isotv99f45cg37lxjZ,iZbp1isotv99f45cg37lxfZ,iZbp1isotv99f45cg37lxiZ
fsmap e4: 1/1/1 up {0=iZbp1isotv99f45cg37lxjZ=up:creating}
osdmap e13: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v20: 80 pgs, 3 pools, 0 bytes data, 0 objects
100 MB used, 2098 GB / 2098 GB avail
64 active+clean
12 creating
4 creating+peering在node01部署gateway
docker run -d --name=rgw -p 80:80 \
--restart=always \
-v /etc/ceph:/etc/ceph -v \
/var/lib/ceph/:/var/lib/ceph/ \
ceph/daemon:tag-build-master-jewel-centos-7 rgw
檢視ceph狀態：
docker exec b3cc55582498 ceph -s執行結果：
# docker exec b3cc55582498 ceph -s
cluster 230a93b6-3876-4d69-bffe-c0c054c49653
health HEALTH_OK
monmap e3: 3 mons at {iZbp1isotv99f45cg37lxfZ=10.135.204.114:6789/0,iZbp1isotv99f45cg37lxiZ=10.135.204.132:6789/0,iZbp1isotv99f45cg37lxjZ=10.135.204.107:6789/0}
election epoch 6, quorum 0,1,2 iZbp1isotv99f45cg37lxjZ,iZbp1isotv99f45cg37lxfZ,iZbp1isotv99f45cg37lxiZ
fsmap e5: 1/1/1 up {0=iZbp1isotv99f45cg37lxjZ=up:active}
osdmap e14: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v26: 88 pgs, 4 pools, 2068 bytes data, 20 objects
101 MB used, 2098 GB / 2098 GB avail
80 active+clean
8 creating