Openstack 多後端配置--詳解NFS作為Cinder的後端儲存

阿新 • • 發佈：2019-01-31

材料：
1.現有一外接NFS儲存，大小為300G，路徑為：192.168.8.213:/test3/cinder
2.mitaka版本，已配置lvm後端儲存，大小為20G

NFS後端儲存本地掛載(不掛載也不影響)
掛載到本地目錄

cd /root
mkdir nfsshare
mount -t nfs 192.168.8.213:/test3/cinder /root/nfsshare/

檢視掛載結果

    [root@controllernode~(keystone_admin)]# df -h  
    Filesystem                           Size  Used 
 Avail Use% Mounted on  
    /dev/sda1                             97G  4.8G  87G   6% /  
    tmpfs                                3.9G  4.0K 3.9G   1% /dev/shm  
    /srv/loopback-device/swift_loopback  1.9G  67M  1.8G   4% /srv/node/swift_loopback  
    192.168.8.213:/test3/cinder           300G 1.4G  298G   1% /root/nfsshare

需要注意的是，如果此時NFS伺服器出現故障，或者是客戶端不能連線到服務端，由於該命令要等待檔案系統查詢超時後才返回結果，所以該操作會變的很慢，該原理是用於所有針對檔案系統的命令，例如 df,ls,cp等

Cinder節點NFS後端儲存配置
①建立/etc/cinder/nfsshares檔案（該檔案的作用是列出所有可用的nfs儲存），並編輯內容如下

192.168.8.213:/test3/cinder

設定配置檔案的許可權

[root@controllernode~]# chown root:cinder /etc/cinder/nfsshares
[root@controllernode 
~]# chmod 0640 /etc/cinder/nfsshares

②/etc/cinder/cinder.conf中配置cinder的volume 服務新增NFS：
這裡寫圖片描述
即：enabled_backends = lvm,nfs

③並新增如下內容
這裡寫圖片描述
即：

[nfs]
volume_driver=cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config = /etc/cinder/nfs_shares
nfs_mount_point_base = /var/lib/cinder/mnt
volume_backend_name = nfs

上述配置條目詳細作用介紹：
這裡寫圖片描述

【備註：新增③內容，可以用命令列執行新增–
修改/etc/cinder/cinder.conf中的值為/etc/cinder/nfsshares,可執行下面的命令
openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_shares_config /etc/cinder/nfsshares
配置cinder的volume使用的驅動，可使用以下命令
openstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.nfs.NfsDriver
】

④重新啟動服務
service openstack-cinder-volume restart

此時檢視客戶端上磁碟的情況
這裡寫圖片描述
添加了最後一條記錄

⑤建立一個虛擬機器，網路硬碟，並將網路硬碟掛載到主機上
控制檯：
這裡寫圖片描述

虛擬機器：
這裡寫圖片描述

在虛擬機器中掛載硬碟
先使用下面的命令格式化
mkfs.ext4 /dev/vdb

問題1：
掛載的過程中，nova/compute.log中出現以下異常

    2014-10-2312:23:28.193 1747 INFO urllib3.connectionpool [-] Starting new HTTP connection(1): 192.168.40.248  
    2014-10-2312:23:28.395 1747 WARNING nova.virt.libvirt.utils[req-5bf92b88-6d15-4c41-8ed7-3325fdea0dcf 5832a2295dc14de79522ee8b42e7daac9207105ae2ac4ef3bdf5dfe40d99fd8d] systool is not installed  
    2014-10-2312:23:28.449 1747 WARNING nova.virt.libvirt.utils[req-5bf92b88-6d15-4c41-8ed7-3325fdea0dcf 5832a2295dc14de79522ee8b42e7daac9207105ae2ac4ef3bdf5dfe40d99fd8d] systool is not installed  
    2014-10-2312:23:28.451 1747 INFO urllib3.connectionpool [-] Starting new HTTP connection(1): 192.168.40.248  
    2014-10-2312:23:28.960 1747 ERROR nova.virt.block_device[req-5bf92b88-6d15-4c41-8ed7-3325fdea0dcf 5832a2295dc14de79522ee8b42e7daac9207105ae2ac4ef3bdf5dfe40d99fd8d] [instance:eb1742c6-1e73-4656-b646-ca8442519e7a] Driver failed to attach volume a1862c54-0671-4cc5-9fce-5e5f8485c21fat /dev/vdb  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a] Traceback (most recent call last):  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File"/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line239, in attach  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    device_type=self['device_type'], encryption=encryption)  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line1267, in attach_volume  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    disk_dev)  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",line 68, in __exit__  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    six.reraise(self.type_, self.value, self.tb)  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line1254, in attach_volume  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    virt_dom.attachDeviceFlags(conf.to_xml(), flags)  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File "/usr/lib/python2.6/site-packages/eventlet/tpool.py",line 183, in doit  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    result = proxy_call(self._autowrap, f, *args, **kwargs)  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File "/usr/lib/python2.6/site-packages/eventlet/tpool.py",line 141, in proxy_call  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    rv = execute(f, *args, **kwargs)  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File 
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    six.reraise(c, e, tb)  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File "/usr/lib/python2.6/site-packages/eventlet/tpool.py",line 80, in tworker  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    rv = meth(*args, **kwargs)  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File "/usr/lib64/python2.6/site-packages/libvirt.py", line419, in attachDeviceFlags  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed',dom=self)  
    2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a] libvirtError: internal error unable toexecute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk1'could not be initialized

解決方法
這個錯來自libvirt，做以下設定即可，先察看virt_use_nfs是off還是on
/usr/sbin/getsebool virt_use_nfs

如果是off，做以下設定
/usr/sbin/setsebool -P virt_use_nfs on

問題2：
雲硬碟掛載過程中，nova/compute.log中出現異常：

2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4756, in _attach_volume
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     do_check_attach=False, do_driver_attach=True)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 54, in wrapped
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     ret_val = method(obj, context, *args, **kwargs)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 288, in attach
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     connector)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 279, in attach
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     device_type=self['device_type'], encryption=encryption)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1358, in attach_volume
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     self._disconnect_volume(connection_info, disk_dev)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1347, in attach_volume
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     guest.attach_device(conf, persistent=True, live=live)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 250, in attach_device
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     self._domain.attachDeviceFlags(conf.to_xml(), flags=flags)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     rv = execute(f, *args, **kwargs)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     six.reraise(c, e, tb)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     rv = meth(*args, **kwargs)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher libvirtError: Cannot access storage file '/var/lib/nova/mnt/82b29b840347978f288c9b93a4d5e849/volume-9b99cf0e-9381-48fb-8bdd-07a40efa80fa' (as uid:107, gid:107): No such file or directory
2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher

解決方法
這個異常來自於/etc/nova/nova.conf配置的nfs_mount_point_base預設路徑
修改或新增以下內容：
nfs_mount_point_base=/var/lib/cinder/mnt

==========NFS後端配置到此為止==========

Cinder節點多後端儲存配置

下面是包含三個配置組的多後端示例（/etc/cinder/cinder.conf ）：

enable_backends =sata,sas,ssd
[sata]
volume_group=cinder-volumes-1
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name=sata

[sas]
volume_group=cinder-volumes-2
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name=sas

[ssd]
volume_group=cinder-volumes-3
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name=ssd

注意配置組名和後端名（volume_backend_name）沒有關聯，你可以取任何合法的字串作為配置組名和後端名，而不用保持二者一致。

Cinder設定卷型別
可以為每個後端關聯一個卷型別，建立卷的時候，排程器就能根據卷型別選擇合適的後端來處理請求
下面的例子中建立一個名為sata的卷型別並與名為sata的後端關聯

#> cinder type-create sata
#> cinder type-key sata set volume_backend_name=sata

如果volume_backend_name指定的後端不存在，在建立卷的時候，過濾排程器將返回無法找到合適的後端的錯誤

Cinder建立卷
完成後端與卷型別的關聯後，就可以建立捲了。當然你也可以不指定卷型別，排程器將使用預設的卷型別（由default_volume_type指定）.
下面的例子建立了一個大小為10G型別為sata的卷

#> cinder create-volume --volume-type sata --display_name sata1 10

整個配置過程還是比較簡單的，希望本文能給剛入門的你一點收穫。

Cinder支援的儲存後端

    * Blockbridge EPS
    * Ceph RADOS Block Device (RBD)
    * CloudByte volume driver
    * Coho Data volume driver
    * Dell EqualLogic volume driver
    * Dell Storage Center Fibre Channel and iSCSI drivers
    * Dot Hill AssuredSAN Fibre Channel and iSCSI drivers
    * EMC ScaleIO Block Storage driver configuration
    * EMC VMAX iSCSI and FC drivers
    * EMC VNX driver
    * EMC XtremIO Block Storage driver configuration
    * Fujitsu ETERNUS DX driver
    * GlusterFS driver
    * HDS HNAS iSCSI and NFS driver
    * Hitachi storage volume driver
    * HPE 3PAR Fibre Channel and iSCSI drivers
    * HPE LeftHand/StoreVirtual driver
    * HP MSA Fibre Channel and iSCSI drivers
    * Huawei volume driver
    * IBM GPFS volume driver
    * IBM Storwize family and SVC volume driver
    * IBM XIV and DS8000 volume driver
    * IBM FlashSystem volume driver
    * ITRI DISCO volume driver
    * Lenovo Fibre Channel and iSCSI drivers
    * LVM
    * NetApp unified driver
    * Nimble Storage volume driver
    * NexentaStor 4.x NFS and iSCSI drivers
    * NexentaStor 5.x NFS and iSCSI drivers
    * NexentaEdge iSCSI driver
    * NFS driver
    * ProphetStor Fibre Channel and iSCSI drivers
    * Pure Storage iSCSI and Fibre Channel volume drivers
    * Quobyte driver
    * Scality SOFS driver
    * Sheepdog driver
    * SambaFS driver
    * SolidFire
    * Tintri
    * Violin Memory 7000 Series FSP volume driver
    * VMware VMDK driver
    * Windows iSCSI volume driver
    * X-IO volume driver
    * Oracle ZFS Storage Appliance iSCSI driver
    * Oracle ZFS Storage Appliance NFS driver

Openstack 多後端配置--詳解NFS作為Cinder的後端儲存

Openstack 多後端配置--詳解NFS作為Cinder的後端儲存

筆記：Spring Cloud Ribbon 客戶端配置詳解

MySQL 5.7多源複製配置詳解

vsftpd服務端配置詳解

kafka1.0Server端配置詳解

Apache為本地主機配置多個網站根目錄詳解

NFS服務器原理和安裝配置詳解附案例演練

NFS配置詳解

Android FileProvider 屬性配置詳解及FileProvider多節點問題

cookie和session在頁面和後端的詳解

華為PAT端口地址映射配置詳解（一）

華為PAT端口地址映射配置詳解（二）

華為PAT端口地址映射配置詳解（三）

HikariCP配置詳解+多資料來源

一臺電腦上配置多個tomcat，詳解

Mybatis關聯查詢之一對多XML配置詳解

Windows RabbitMQ服務端安裝與配置詳解

Mybatis關聯查詢之一對多和多對一XML配置詳解

NFS服務配置詳解

Centos6.4下OpenVPN在windows和centos下客戶端安裝與配置詳解

Openstack 多後端配置--詳解NFS作為Cinder的後端儲存

相關推薦