1. 程式人生 > >Openstack虛擬機器關機後無法啟動(start).md

Openstack虛擬機器關機後無法啟動(start).md

Openstack虛擬機器關機後無法啟動(start)

#現象 執行啟動虛擬機器(nova start)之後,虛擬機器並未啟動,到該計算節點母機檢視/var/log/nova/nova-compute.log日誌發現如下報錯
2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2044, in reboot
2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher     block_device_info)
2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2156, in _hard_reboot 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher vifs_already_plugged=True) 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py"
, line 4429, in _create_domain_and_network 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher power_on=power_on) 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4360, in _create_domain 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher LOG.error(
err) 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__ 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4350, in _create_domain 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher domain.createWithFlags(launch_flags) 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs) 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher rv = execute(f, *args, **kwargs) 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher six.reraise(c, e, tb) 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher rv = meth(*args, **kwargs) 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/libvirt.py", line 996, in createWithFlags 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher libvirtError: unsupported configuration: Unable to find security driver for label selinux 2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher 2018-09-30 02:32:14.535 4537 INFO nova.compute.resource_tracker [req-efb15755-2bc0-45d2-946c-ba01bf451836 - - - - -] Auditing locally available compute resources for node openstack-com04

分析解決

通過日誌直接分析問題

2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher libvirtError: unsupported configuration: Unable to find security driver for label selinux

這兩行獲取到倆個資訊

  • withflags failed說明flag可能有問題
  • 第二行不支援的安全驅動配置

排查flag,先看他是什麼東西

根據上面日誌資訊

vim /usr/lib64/python2.7/site-packages/libvirt.py +996
"""Launch a defined domain. If the call succeeds the domain moves from the
        defined to the running domains pools.
        
        If the VIR_DOMAIN_START_PAUSED flag is set, or if the guest domain
        has a managed save image that requested paused state (see
        virDomainManagedSave()) the guest domain will be started, but its
        CPUs will remain paused. The CPUs can later be manually started
        using virDomainResume().  In all other cases, the guest domain will
        be running.
        
        If the VIR_DOMAIN_START_AUTODESTROY flag is set, the guest
        domain will be automatically destroyed when the virConnectPtr
        object is finally released. This will also happen if the
        client application crashes / loses its connection to the
        libvirtd daemon. Any domains marked for auto destroy will
        block attempts at migration, save-to-file, or snapshots.
        
        If the VIR_DOMAIN_START_BYPASS_CACHE flag is set, and there is a
        managed save file for this domain (created by virDomainManagedSave()),
        then libvirt will attempt to bypass the file system cache while restoring
        the file, or fail if it cannot do so for the given system; this can allow
        less pressure on file system cache, but also risks slowing loads from NFS.
        
        If the VIR_DOMAIN_START_FORCE_BOOT flag is set, then any managed save
        file for this domain is discarded, and the domain boots from scratch. """
        ret = libvirtmod.virDomainCreateWithFlags(self._o, flags)
        if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)

通過註釋猜測到flag是一個虛擬機器狀態識別符號,如果是-1則擲出異常,-1一般是無效值,但這個虛擬機器原來狀態是好的,由此推測狀態被破壞

再看異常

2018-09-30 02:31:27.083 4537 TRACE oslo_messaging.rpc.dispatcher libvirtError: unsupported configuration: Unable to find security driver for label selinux

先看下libvirt中關於security driver配置

grep security_driver /etc/libvirt/qemu.conf
#       security_driver = [ "selinux", "apparmor" ]
# value of security_driver cannot contain "dac".  The value "none" is
# a special value; security_driver can be set to that value in
#security_driver = "selinux"

可以看到都是註釋的,再看下虛擬機器xml檔案裡面有沒有包含sec(安全)與lable的欄位,虛擬機器配置檔案在

平臺 虛擬機器配置檔案路徑
kvm /etc/libvirt/qemu/
openstack /var/lib/nova/instances/

檢視並對比配置檔案之後未發現與其他有正常伺服器有不同之處,沒有發現

檢視系統日誌

Sep 30 02:28:38 openstack-com04 journal: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so not accessible
Sep 30 02:28:38 openstack-com04 systemd: Started Virtualization daemon.
Sep 30 02:29:18 openstack-com04 journal: unsupported configuration: Unable to find security driver for label selinux
Sep 30 02:29:18 openstack-com04 journal: Unable to restore from managed state /var/lib/libvirt/qemu/save/instance-0000001f.save. Maybe the file is corrupted?
Sep 30 02:31:26 openstack-com04 journal: unsupported configuration: Unable to find security driver for label selinux
Sep 30 02:31:26 openstack-com04 journal: Unable to restore from managed state /var/lib/libvirt/qemu/save/instance-0000001f.save. Maybe the file is corrupted?
Sep 30 02:32:53 openstack-com04 kernel: device tapb4172400-9b entered promiscuous mode

kvm中.save字尾的檔案是儲存虛擬機器狀態的檔案,從日誌可以看出,無法從這個狀態檔案恢復資訊,可以移除掉這個檔案直接重啟虛擬機器即可

mv /var/lib/libvirt/qemu/save/instance-0000001f.save  /tmp/201809