Openstack 之 ceph存儲池分級
上面左邊是我的個人微信，如需進一步溝通，請加微信。 右邊是我的公眾號“Openstack私有雲”，如有興趣，請關註。
上篇《Openstack 之使用外部ceph存儲》中提到，為了達到不同的pool可以使用不同類型的硬盤組成的存儲池，對於只是需要解決存儲分級的小規模集群場景來說，先使用kolla進行統一部署ceph，在部署完成之後，然後通過手動調整ceph crush存儲規則，新建rule，將相應的osd劃入對應的rule，最後將相應的pool調整到相應的rule。這裏我們就這個想法進行實際驗證。
Openstack集群環境：
系統centos7.3
Openstack版本 Ocata 4.0.2.1 ，使用kolla部署
Openstack集群服務器分配如下：
| 節點 | 服務 | IP |
| control01 | control+compute+network | 192.168.1.130 |
| control02 | control+compute+network | 192.168.1.131 |
| control03 | control+compute+network | 192.168.1.132 |
已經成功部署了3個節點的控制+存儲+計算+網絡的融合節點，ceph的osd初始分布如下：
ceph> osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-5 0 root ssd
-6 0 chassis ssd-hosts
-1 9.00000 root default
-2 3.00000 host 192.168.1.132
0 1.00000 osd.0 up 1.00000 1.00000
3 1.00000 osd.3 up 1.00000 1.00000
6 1.00000 osd.6 up 1.00000 1.00000
-3 3.00000 host 192.168.1.130
1 1.00000 osd.1 up 1.00000 1.00000
5 1.00000 osd.5 up 1.00000 1.00000
7 1.00000 osd.7 up 1.00000 1.00000
-4 3.00000 host 192.168.1.131
2 1.00000 osd.2 up 1.00000 1.00000
4 1.00000 osd.4 up 1.00000 1.00000
8 1.00000 osd.8 up 1.00000 1.00000
我們的目標是，將ssd硬盤 osd.0/osd.1/osd.2 等3個ssd類型硬盤osd劃入一個新的root根ssd，新的host組ssd-hosts ,新的host，將這3個osd分別移入新的host中，同時新建一個rule規則ssd-rule ，使這個根ssd的所有設備使用這個ssd-rule規則。最後，將openstack存放虛擬機的pool池vms設置使用這個ssd-rule規則，這樣，就達到了虛擬機池vms使用ssd硬盤池的目的。同樣的道理，對於glance和cinder-backup可以使用慢速硬盤SATA盤組成的慢速存儲池，也可以用同樣的方法進行調整。
操作記錄：
創建ssd root
docker exec -it ceph_mon ceph osd crush add-bucket ssd root
//創建一個新的桶叫ssd ，級別是root最高級
創建chassis
docker exec -it ceph_mon ceph osd crush add-bucket ssd-hosts chassis
//創建一個新的桶叫ssd-hosts ，級別是機框
docker exec -it ceph_mon ceph osd crush move ssd-hosts root=ssd
//將ssd-hosts歸入ssd
創建host
docker exec -it ceph_mon ceph osd crush add-bucket ssd-132 host
docker exec -it ceph_mon ceph osd crush add-bucket ssd-130 host
docker exec -it ceph_mon ceph osd crush add-bucket ssd-131 host
//創建3個新的桶叫ssd-132/ssd-130/ssd-131 ，級別是host
docker exec -it ceph_mon ceph osd crush move ssd-132 chassis=ssd-hosts
docker exec -it ceph_mon ceph osd crush move ssd-130 chassis=ssd-hosts
docker exec -it ceph_mon ceph osd crush move ssd-131 chassis=ssd-hosts
//將ssd-132/ssd-130/ssd-131歸入ssd-hosts
移動osd
docker exec -it ceph_mon ceph osd crush set osd.0 1.0 host=ssd-132
docker exec -it ceph_mon ceph osd crush set osd.1 1.0 host=ssd-130
docker exec -it ceph_mon ceph osd crush set osd.2 1.0 host=ssd-131
//將osd.0/osd.1 /osd.2移動到對應的host 中
創建crush rule
docker exec -it ceph_mon ceph osd crush rule create-simple ssd-rule ssd host firstn
//創建crush rule，rule名稱是ssd-rule，root=ssd，tpye=host，mode=firstn
經過調整之後osd的分布如下：
ceph> osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-5 3.00000 root ssd
-6 3.00000 chassis ssd-hosts
-7 1.00000 host ssd-132
0 1.00000 osd.0 up 1.00000 1.00000
-8 1.00000 host ssd-130
1 1.00000 osd.1 up 1.00000 1.00000
-9 1.00000 host ssd-131
2 1.00000 osd.2 up 1.00000 1.00000
-1 6.00000 root default
-2 2.00000 host 192.168.1.132
3 1.00000 osd.3 up 1.00000 1.00000
6 1.00000 osd.6 up 1.00000 1.00000
-3 2.00000 host 192.168.1.130
5 1.00000 osd.5 up 1.00000 1.00000
7 1.00000 osd.7 up 1.00000 1.00000
-4 2.00000 host 192.168.1.131
4 1.00000 osd.4 up 1.00000 1.00000
8 1.00000 osd.8 up 1.00000 1.00000
crush rule規則如下：
# docker exec -it ceph_mon ceph osd crush rule dump
[
{
"rule_id": 0,
"rule_name": "replicated_ruleset",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 1,
"rule_name": "disks",
"ruleset": 1,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 2,
"rule_name": "ssd-rule",
"ruleset": 2,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -5,
"item_name": "ssd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]
可以看到新增的ssd-rule的ruleid是2。
更改vms pool的rule：
設置openstack的虛擬機存儲池vms使用的rule為ssd-rule：
docker exec -it ceph_mon ceph ceph osd pool set vms crush_ruleset 2
驗證：
1. 列出vms中的images：
[root@control01 neutron]# docker exec -it ceph_mon rbd -p vms ls
4ba3e273-b729-4a24-8040-629b9b58fe43_disk
2. 查看images是不是分布在osd.0/osd.1/osd.2這3塊ssd盤中：
[root@control01 ceph-mon]# docker exec -it ceph_mon bash
(ceph-mon)[root@control01 ceph]# vi rbd_loc.sh
#!/bin/bash
# USAGE:./rbd-loc <pool> <image>
if [ -z ${1} ] || [ -z ${2} ];
then
echo "USAGE: ./rbd-loc <pool> <image>"
exit 1
fi
rbd_prefix=$(rbd -p ${1} info ${2} | grep block_name_prefix | awk '{print $2}')
for i in $(rados -p ${1} ls | grep ${rbd_prefix})
do
ceph osd map ${1} ${i}
done
(ceph-mon)[root@control01 ceph]# chmod +x rbd_loc.sh
(ceph-mon)[root@control01 ceph]# ./rbd_loc.sh vms 4ba3e273-b729-4a24-8040-629b9b58fe43_disk
osdmap e685 pool 'vms' (9) object 'rbd_data.2744fe204970c8.0000000000000004' -> pg 9.eec101a0 (9.20) -> up ([2,1], p2) acting ([2,1], p2)
osdmap e685 pool 'vms' (9) object 'rbd_data.2744fe204970c8.0000000000000006' -> pg 9.9f03d335 (9.35) -> up ([1,2], p1) acting ([1,2], p1)
osdmap e685 pool 'vms' (9) object 'rbd_data.2744fe204970c8.0000000000000001' -> pg 9.67b18ab5 (9.35) -> up ([1,2], p1) acting ([1,2], p1)
osdmap e685 pool 'vms' (9) object 'rbd_data.2744fe204970c8.0000000000000003' -> pg 9.5897ddc8 (9.48) -> up ([0,2], p0) acting ([0,2], p0)
osdmap e685 pool 'vms' (9) object 'rbd_data.2744fe204970c8.0000000000000007' -> pg 9.dbace74b (9.4b) -> up ([2,0], p2) acting ([2,0], p2)
osdmap e685 pool 'vms' (9) object 'rbd_data.2744fe204970c8.0000000000000000' -> pg 9.f1ab8f4d (9.4d) -> up ([0,2], p0) acting ([0,2], p0)
osdmap e685 pool 'vms' (9) object 'rbd_data.2744fe204970c8.0000000000000005' -> pg 9.20dc9e6c (9.6c) -> up ([1,2], p1) acting ([1,2], p1)
osdmap e685 pool 'vms' (9) object 'rbd_data.2744fe204970c8.0000000000000002' -> pg 9.7f6d55f7 (9.77) -> up ([2,0], p2) acting ([2,0], p2)
從上面的結果可以看到，所有的數據副本object所在的pg，都放在0，1，2上 。
驗證成功！
Openstack 之 ceph存儲池分級