1. 程式人生 > >OpenStack實踐系列⑨雲硬盤服務Cinder

OpenStack實踐系列⑨雲硬盤服務Cinder

ecs 刪掉 lag 文件系統 primary nas 不同 attach driver

OpenStack實踐系列⑨雲硬盤服務Cinder
八、cinder
8.1存儲的三大分類

塊存儲:硬盤,磁盤陣列DAS,SAN存儲
文件存儲:nfs,GluserFS,Ceph(PB級分布式文件系統),MooserFS(缺點Metadata數據丟失,虛擬機就毀了)

11.2網絡類型選擇

對象存儲:swift,S3

8.2 cinder控制節點的部署

安裝cinder

[[email protected] ~]# yum install openstack-cinder python-cinderclient -y

修改cinder配置文件

[default]
rpc_backend = rabbit
glance_host = 192.168.3.199
auth_strategy = keystone

[oslo_concurrency]
lock_path = /var/lib/cinder/tmp

[oslo_messaging_rabbit]
rabbit_host = 192.168.3.199 # rabbitmq的主機
rabbit_port = 5672 # rabbitmq的端口
rabbit_userid = openstack # rabbitmq的用戶
rabbit_password = openstack # rabbitmq的密碼

[database]
connection = mysql://cinder:[email protected]

/* *//cinder # 配置mysql地址

[keystone_authtoken]
auth_uri = http://192.168.3.199:5000
auth_url = http://192.168.3.199:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = cinder

修改後結果如下

[[email protected] cinder]# grep
-n ^[a-Z] /etc/cinder/cinder.conf 421:glance_host = 192.168.3.199 536:auth_strategy = keystone 2294:rpc_backend = rabbit 2516:connection = mysql://cinder:[email protected]/cinder 2641:auth_uri = http://192.168.3.199:5000 2642:auth_url = http://192.168.3.199:35357 2643:auth_plugin = password 2644:project_domain_id = default 2645:user_domain_id = default 2646:project_name = service 2647:username = cinder 2648:password = cinder 2874:lock_path = /var/lib/cinder/tmp 3173:rabbit_host = 192.168.3.199 3177:rabbit_port = 5672 3189:rabbit_userid = openstack 3193:rabbit_password = openstack

修改nova的配置文件

[[email protected] ~]# vim /etc/nova/nova.conf 
os_region_name = RegionOne # 通知nova使用cinder [cinder]部分

執行同步數據庫操作

[[email protected] ~]# su -s /bin/sh -c "cinder-manage db sync" cinder

檢查導入數據庫結果
MariaDB [(none)]> use cinder
Database changed
MariaDB [cinder]> show tables;
+----------------------------+
| Tables_in_cinder |
+----------------------------+
| backups |
| cgsnapshots |
| consistencygroups |
| driver_initiator_data |
| encryption |
| image_volume_cache_entries |
| iscsi_targets |
| migrate_version |
| quality_of_service_specs |
| quota_classes |
| quota_usages |
| quotas |
| reservations |
| services |
| snapshot_metadata |
| snapshots |
| transfers |
| volume_admin_metadata |
| volume_attachment |
| volume_glance_metadata |
| volume_metadata |
| volume_type_extra_specs |
| volume_type_projects |
| volume_types |
| volumes |
+----------------------------+
25 rows in set (0.00 sec)

創建一個cinder用戶,加入service項目,給予admin角色

[[email protected] ~]# source admin-openrc.sh
[[email protected] ~]# openstack user create --domain default --password-prompt cinder
User Password:
Repeat User Password: # (密碼需要配置成cinder就是/etc/cinder/cinder.conf配置文件中配置的2648行)
+-----------+----------------------------------+
| Field | Value |
+-----------+----------------------------------+
| domain_id | default |
| enabled | True |
| id | 420d7573e9fc43b3b263f31bb6dd76e2 |
| name | cinder |
+-----------+----------------------------------+
[[email protected] ~]# openstack role add --project service --user cinder admin

重啟nova-api服務和啟動cinder服務

[[email protected] ~]# systemctl restart openstack-nova-api.service
[[email protected] ~]# systemctl enable openstack-cinder-api.service openstack-cinder-scheduler.service

[[email protected] ~]# systemctl start openstack-cinder-api.service openstack-cinder-scheduler.service

創建服務(包含V1和V2)

[[email protected] ~]# openstack service create --name cinder --description "OpenStack Block Storage" volume
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | OpenStack Block Storage |
| enabled | True |
| id | 6e3b2c3940d14300ab28aed272ade1d3 |
| name | cinder |
| type | volume |
+-------------+----------------------------------+

[[email protected] ~]# openstack service create --name cinderv2 --description "OpenStack Block Storage" volumev2
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | OpenStack Block Storage |
| enabled | True |
| id | 2108489d055e4fcb8f9c88fa9d5e4e3d |
| name | cinderv2 |
| type | volumev2 |
+-------------+----------------------------------+

分別對V1和V2創建三個環境(admin,internal,public)的endpoint

[[email protected] ~]# openstack endpoint create --region RegionOne volume public http://192.168.3.199:8776/v1/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 007497468db7456d81157962f8740540 |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 6e3b2c3940d14300ab28aed272ade1d3 |
| service_name | cinder |
| service_type | volume |
| url | http://192.168.3.199:8776/v1/%(tenant_id)s |
+--------------+--------------------------------------------+

[[email protected] ~]# openstack endpoint create --region RegionOne volume internal http://192.168.3.199:8776/v1/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | e7543b96b69342bcabead7ad8a583860 |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 6e3b2c3940d14300ab28aed272ade1d3 |
| service_name | cinder |
| service_type | volume |
| url | http://192.168.3.199:8776/v1/%(tenant_id)s |
+--------------+--------------------------------------------+

[[email protected] ~]# openstack endpoint create --region RegionOne volume admin http://192.168.3.199:8776/v1/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 12e4bea586384d43b16e4de5a00afb1b |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 6e3b2c3940d14300ab28aed272ade1d3 |
| service_name | cinder |
| service_type | volume |
| url | http://192.168.3.199:8776/v1/%(tenant_id)s |
+--------------+--------------------------------------------+

[[email protected] ~]# openstack endpoint create --region RegionOne volumev2 public http://192.168.3.199:8776/v2/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 07c56b0033454fbda201f3cc58ce0a1b |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | http://192.168.3.199:8776/v2/%(tenant_id)s |
+--------------+--------------------------------------------+

[[email protected] ~]# openstack endpoint create --region RegionOne volumev2 internal http://192.168.3.199:8776/v2/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 66b34a18d4de456ab32ebba24831b959 |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | http://192.168.3.199:8776/v2/%(tenant_id)s |
+--------------+--------------------------------------------+

[[email protected] ~]# openstack endpoint create --region RegionOne volumev2 admin http://192.168.3.199:8776/v2/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | fc0bed271a7048a5aff7e63aebd0199a |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | http://192.168.3.199:8776/v2/%(tenant_id)s |
+--------------+--------------------------------------------+

8.3 cinder存儲節點的部署(此處使用nova的計算節點即node2.chinasoft.com)

  本文中cinder後端存儲使用ISCSI(類似於nova-computer使用的kvm),ISCSI使用LVM,在定義好的VG中,每創建一個雲硬盤,就會增加一個LV,使用ISCSI發布。
在存儲節點上加一個硬盤

技術分享

查看磁盤添加情況

[[email protected] ~]# fdisk -l

Disk /dev/sdb: 85.9 GB, 85899345920 bytes, 167772160 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 128.8 GB, 128849018880 bytes, 251658240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0004c2a9

Device Boot Start End Blocks Id System
/dev/sda1 * 2048 616447 307200 83 Linux
/dev/sda2 616448 155811839 77597696 8e Linux LVM

Disk /dev/mapper/centos-root: 32.2 GB, 32212254720 bytes, 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-swap: 4294 MB, 4294967296 bytes, 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-data: 42.9 GB, 42945478656 bytes, 83877888 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

創建一個pv和vg(名為cinder-volumes)

[[email protected] ~]# pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created
[[email protected] ~]# vgcreate cinder-volumes /dev/sdb
Volume group "cinder-volumes" successfully created

修改lvm的配置文件中添加filter,只有instance可以訪問

[[email protected] ~]# vim /etc/lvm/lvm.conf
131 filter = [ "a/sdb/", "r/.*/"]

存儲節點安裝

[[email protected] ~]# yum install openstack-cinder targetcli python-oslo-policy -y

修改存儲節點的配置文件,在這裏直接拷貝控制節點的文件
[[email protected] ~]# scp /etc/cinder/cinder.conf 192.168.3.200:/etc/cinder/

修改存儲(即計算節點node2.chinasoft.com)上的/etc/cinder/cinder.conf文件

添加如下配置:
默認沒有[lvm]需要自己創建
[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver # 使用lvm後端存儲
volume_group = cinder-volumes # vg的名稱:剛才創建的
iscsi_protocol = iscsi # 使用iscsi協議
iscsi_helper = lioadm

[[email protected] cinder]# grep -n ‘^[a-Z]‘ /etc/cinder/cinder.conf
421:glance_host = 192.168.3.199
536:auth_strategy = keystone
540:enabled_backends = lvm # lvm 使用的後端是lvm,要對應添加的[lvm],自定義也可以
2294:rpc_backend = rabbit
2516:connection = mysql://cinder:[email protected]/cinder
2641:auth_uri = http://192.168.3.199:5000
2642:auth_url = http://192.168.3.199:35357
2643:auth_plugin = password
2644:project_domain_id = default
2645:user_domain_id = default
2646:project_name = service
2647:username = cinder
2648:password = cinder
2874:lock_path = /var/lib/cinder/tmp
3173:rabbit_host = 192.168.3.199
3177:rabbit_port = 5672
3189:rabbit_userid = openstack
3193:rabbit_password = openstack
[lvm] # 此行不是grep過濾出來的,因為是在配置文件最後添加上的,其對應的是540行的lvm
3416:volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver # 使用lvm後端存儲
3417:volume_group = cinder-volumes # vg的名稱:剛才創建的
3418:iscsi_protocol = iscsi # 使用iscsi協議
3419:iscsi_helper = lioadm

啟動存儲節點的cinder(這裏是node2)

[[email protected] cinder]# systemctl enable openstack-cinder-volume.service target.service
Created symlink from /etc/systemd/system/multi-user.target.wants/openstack-cinder-volume.service to /usr/lib/systemd/system/openstack-cinder-volume.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.
[[email protected] cinder]# systemctl start openstack-cinder-volume.service target.service

查看雲硬盤服務狀態(如果是虛擬機作為宿主機,時間不同步,會產生問題)

[[email protected] ~]# source admin-openrc.sh 
[[email protected] ~]# cinder service-list
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | node1.chinasoft.com | nova | enabled | up | 2017-04-28T09:59:54.000000 | - |
| cinder-volume | [email protected] | nova | enabled | up | 2017-04-28T09:59:58.000000 | - |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+

創建一個雲硬盤,將雲硬盤掛載到虛擬機上,啟動在虛擬機實例詳情可以查看到,雲硬盤掛載的情況
具體步驟:
1.創建雲硬盤

技術分享

技術分享
2.點擊 動作選項下的 管理已連接雲硬盤,選擇需要的虛擬機 將雲硬盤掛載到指定的虛擬機上

技術分享

技術分享

技術分享

技術分享

在虛擬機中對掛載的硬盤進行分區格式化,如果有時不想掛載這個雲硬盤了,一定不要刪掉,生產環境一定要註意,否則虛擬機會出現error,應該使用umont確定卸載了,再使用dashboard進行刪除雲硬盤

# ssh cirros@192.168.3.103

$ sudo fdisk -l

Disk /dev/vda: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/vda1 * 16065 2088449 1036192+ 83 Linux

Disk /dev/vdb: 1073 MB, 1073741824 bytes
16 heads, 63 sectors/track, 2080 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/vdb doesnt contain a valid partition table

# 分區
$ sudo fdisk /dev/vdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x3fecc8a5.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content wont be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-2097151, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151): 
Using default value 2097151

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

查看分區,已經生成了新的分區/dev/vdb
$ sudo fdisk -l

Disk /dev/vda: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/vda1 * 16065 2088449 1036192+ 83 Linux

Disk /dev/vdb: 1073 MB, 1073741824 bytes
9 heads, 8 sectors/track, 29127 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xfaacdc93

Device Boot Start End Blocks Id System
/dev/vdb1 2048 2097151 1047552 83 Linux


# 格式化
$ sudo mkfs.ext4 /dev/vdb1
mke2fs 1.42.2 (27-Mar-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 261888 blocks
13094 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
32768, 98304, 163840, 229376

Allocating group tables: done 
Writing inode tables: done 
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

# 掛載
$ sudo mkdir /data
$ sudo mount /dev/vdb1 /data

$ df -h
Filesystem Size Used Available Use% Mounted on
/dev 242.3M 0 242.3M 0% /dev
/dev/vda1 23.2M 18.0M 4.0M 82% /
tmpfs 245.8M 0 245.8M 0% /dev/shm
tmpfs 200.0K 72.0K 128.0K 36% /run
/dev/vdb1 1006.9M 17.3M 938.5M 2% /data

方法二:從雲硬盤啟動虛擬機

①先創建一個demo2的雲硬盤

技術分享

②動作-作為實例啟動

技術分享

九、虛擬機創建流程:

技術分享

第一階段:用戶操作
1)用戶使用Dashboard或者CLI連接keystone,發送用戶名和密碼,待keystone驗證通過,keystone會返回給dashboard一個authtoken
2)Dashboard會帶著上述的authtoken訪問nova-api進行創建虛擬機請求
3)nova-api會通過keytoken確認dashboard的authtoken認證消息。

第二階段:nova內組件交互階段
4)nova-api把用戶要創建的虛擬機的信息記錄到數據庫中.
5)nova-api使用rpc-call的方式發送請求給消息隊列
6)nova-scheduler獲取消息隊列中的消息
7)nova-scheduler和查看數據庫中要創建的虛擬機信息和計算節點的信息,進行調度
8)nova-scheduler把調度後的信息發送給消息隊列
9)nova-computer獲取nova-schedur發送給queue的消息
10)nova-computer通過消息隊列發送消息給nova-conudctor,想要獲取數據庫中的要創建虛擬機信息
11)nova-conductor獲取消息隊列的消息
12)nova-conductor讀取數據庫中要創建虛擬機的信息
13)nova-conductor把從數據庫獲取的消息返回給消息隊列
14)nova-computer獲取nova-conducter返回給消息隊列的信息

第三階段:nova和其他組件進行交互
15)nova-computer通過authtoken和數據庫返回的鏡像id請求glance服務
16)glance會通過keystone進行認證
17)glance驗證通過後把鏡像返回給nova-computer
18)nova-computer通過authtoken和數據庫返回的網絡id請求neutron服務
19)neutron會通過keystone進行認證
20)neutron驗證通過後把網絡分配情況返回給nova-computer
21)nova-computer通過authtoken和數據庫返回的雲硬盤請求cinder服務
22)cinder會通過keystone進行認證
23)cinder驗證通過後把雲硬盤分配情況返回給nova-computer

第四階段:nova創建虛擬機
24)nova-compute通過libvirt調用kvm根據已有的信息創建虛擬機,動態生成xml
25)nova-api會不斷的在數據庫中查詢信息並在dashboard顯示虛擬機的狀態

生產場景註意事項:
1、新加的一個計算節點,創建虛擬機時間會很長,因為第一次使用計算節點,沒有鏡像,計算節點要把glance的鏡像放在後端文件(/var/lib/nova/instance/_base)下,
鏡像如果很大,自然會需要很長時間,然後才會在後端文件的基礎上創建虛擬機(寫時復制copy on write)。
2、創建虛擬機失敗的原因之一:創建網橋失敗。要保證eth0網卡配置文件的BOOTPROTE是static而不是dhcp狀態。

OpenStack實踐系列⑨雲硬盤服務Cinder