Mitaka版Openstack虛擬機器啟動流程

阿新 • • 發佈：2019-01-18

最近工作的主要內容是在dashboard中啟動虛擬機器的時候加一個cache靜態劃分的功能，是在隔離區還是在共享區，這就需要對整個虛擬機器的啟動流程程式碼非常熟悉，並且還要適當的改前端。但是在看原始碼包括實踐的過程發現了一件很神奇的時候，網上的教程大多是IceHose版本的，其啟動虛擬機器走的是正常的workflow中的handle函式，但是也不知道是我們裝的Mitaka版本OpenStack有問題還是怎麼樣，起虛擬機器偏偏不走那裡，第一天遇到這個問題的時候我就給hcch師兄說我見鬼了。。但好在，用了三天時間調出來了，同時也記錄一下Mitaka版本的虛擬機器啟動的流程。

在查資料的時候找到了這麼一個圖，侵刪。。

這裡寫圖片描述

1. Dashboard中

1. /usr/share/openstack-dashboard/openstack_dashboard/dashboards/project/instances/workflows/create_instance.py

是關於建立虛擬機器的workflow，但是不走LaunchInstance中的handle進行處理

class LaunchInstance(workflows.Workflow):
    slug = "launch_instance"
    name = _("Launch Instance")
    finalize_button_name = _("Launch" 
)
    success_message = _('Launched %(count)s named "%(name)s".')
    failure_message = _('Unable to launch %(count)s named "%(name)s".')
    success_url = "horizon:project:instances:index"
    multipart = True
    default_steps = (SelectProjectUser,    
                     SetInstanceDetails,
                     SetAccessControls,
                     SetNetwork,
                     SetNetworkPorts,
                     PostCreationStep,
                     SetAdvanced)

這裡面定義了7個Steps，每個Steps中都有對應的表單顯示。其中每個標籤卡顯示的具體內容和相應的html和js相對應。

具體的js和html的檔案放在以下目錄

/usr/share/openstack-dashboard/openstack_dashboard/dashboards/project/static/dashboard/project/workflow/launch-instance

2. /usr/share/openstack-dashboard/openstack_dashboard/dashboards/project/static/dashboard/project/workflow/launch-instance/launch-instance-model.service.js

當在dashboard中把建立虛擬機器的表單填寫好後，所有的資料都通過這個js檔案中的 createInstance() 函式進行請求的進一步傳送

function createInstance() {
      var finalSpec = angular.copy(model.newInstanceSpec);

      cleanNullProperties(finalSpec);

      setFinalSpecBootsource(finalSpec);
      setFinalSpecFlavor(finalSpec);
      setFinalSpecNetworks(finalSpec);
      setFinalSpecPorts(finalSpec);
      setFinalSpecKeyPairs(finalSpec);
      setFinalSpecSecurityGroups(finalSpec);
      setFinalSpecMetadata(finalSpec);

      return novaAPI.createServer(finalSpec).then(successMessage);     
    }

3. /usr/share/openstack-dashboard/openstack_dashboard/static/app/core/openstack-service-api/nova.service.js

第二步中的請求傳送到這個js檔案中的 createServer(newServer) 函式中，其中newServer引數中放了建立虛擬機器的引數

    function createServer(newServer) {
      return apiService.post('/api/nova/servers/', newServer)    # 向api/rest/nova.py 發請求
        .error(function () {
          toastService.add('error', gettext('Unable to create the server.'));
        });
    }

4. /usr/share/openstack-dashboard/openstack_dashboard/api/rest/nova.py

請求傳送到了 Servers 類下的 Post 函式中，其中所有的引數在request中

@rest_utils.ajax(data_required=True)
    def post(self, request):
        try:
            args = (
                request,
                request.DATA['name'],
                request.DATA['source_id'],
                request.DATA['flavor_id'],
                request.DATA['key_name'],
                request.DATA['user_data'],
                request.DATA['security_groups'],
            )
        except KeyError as e:
            raise rest_utils.AjaxError(400, 'missing required parameter '
                                            "'%s'" % e.args[0])
        kw = {}
        for name in self._optional_create:
            if name in request.DATA:
                kw[name] = request.DATA[name]

        new = api.nova.server_create(*args, **kw)
        return rest_utils.CreatedResponse(
            '/api/nova/servers/%s' % utils_http.urlquote(new.id),
            new.to_dict()
        )

5. /usr/share/openstack-dashboard/openstack_dashboard/api/nova.py

通過 server_create 函式呼叫novaclient

def server_create(request, name, image, flavor, key_name, user_data,
                  security_groups, block_device_mapping=None,
                  block_device_mapping_v2=None, nics=None,
                  availability_zone=None, instance_count=1, admin_pass=None,
                  disk_config=None, config_drive=None, meta=None):
    return Server(novaclient(request).servers.create(
        name, image, flavor, userdata=user_data,
        security_groups=security_groups,
        key_name=key_name, block_device_mapping=block_device_mapping,
        block_device_mapping_v2=block_device_mapping_v2,
        nics=nics, availability_zone=availability_zone,
        min_count=instance_count, admin_pass=admin_pass,
        disk_config=disk_config, config_drive=config_drive,
        meta=meta), request)

2. novaclient中

1. /usr/lib/python2.7/site-packages/novaclient/v2/servers.py

ServerManager下的create函式接受來自dashboard的請求

def create(self, name, image, flavor, meta=None, files=None,
               reservation_id=None, min_count=None,
               max_count=None, security_groups=None, userdata=None,
               key_name=None, availability_zone=None,
               block_device_mapping=None, block_device_mapping_v2=None,
               nics=None, scheduler_hints=None,
               config_drive=None, disk_config=None, admin_pass=None,
               access_ip_v4=None, access_ip_v6=None, **kwargs):

        # 中間省略掉。。

        return self._boot(resource_url, response_key, *boot_args,
                          **boot_kwargs)

接下來在這個檔案中呼叫 _boot 函式

    def _boot(self, resource_url, response_key, name, image, flavor,
              meta=None, files=None, userdata=None,
              reservation_id=None, return_raw=False, min_count=None,
              max_count=None, security_groups=None, key_name=None,
              availability_zone=None, block_device_mapping=None,
              block_device_mapping_v2=None, nics=None, scheduler_hints=None,
              config_drive=None, admin_pass=None, disk_config=None,
              access_ip_v4=None, access_ip_v6=None, description=None,
              **kwargs):
        """
        Create (boot) a new server.
        """

        # 中間是各種引數初始化，略過。。

        return self._create(resource_url, body, response_key,
                            return_raw=return_raw, **kwargs)

2. /usr/lib/python2.7/site-packages/novaclient/base.py

從這以下到novaclient結束，都不會設計程式碼的修改，因此只寫到這裡，具體呼叫的檔案會列出來，但不再貼程式碼了

    def _create(self, url, body, response_key, return_raw=False, **kwargs):
        self.run_hooks('modify_body_for_create', body, **kwargs)
        resp, body = self.api.client.post(url, body=body)     
        # 在這裡繼續往下走，呼叫下一步的函式

        if return_raw:
            return self.convert_into_with_meta(body[response_key], resp)

3. /usr/lib/python2.7/site-packages/novaclient/client.py

好吧，本來是不想繼續往下寫關於novaclient的了，但是在nova那邊發現我從novaclient傳到nova的引數居然丟了！丟了！肯定就是在下面我沒接觸到的步驟丟了，只能繼續往下看程式碼了。

順序是這樣的 HTTPClient.post -> HTTPClient._cs_request -> HTTPClient._time_request -> HTTPClient.request

    def post(self, url, **kwargs):
        return self._cs_request(url, 'POST', **kwargs)

    def _cs_request(self, url, method, **kwargs):
        # 前面的省略
        try:
            kwargs.setdefault('headers', {})['X-Auth-Token'] = self.auth_token
            if self.projectid:
                kwargs['headers']['X-Auth-Project-Id'] = self.projectid

            resp, body = self._time_request(url, method, **kwargs)   # 在這裡繼續發請求
            return resp, body
        except exceptions.Unauthorized as e:
            # 略過，。。

    def _time_request(self, url, method, **kwargs):
        with utils.record_time(self.times, self.timings, method, url):
            resp, body = self.request(url, method, **kwargs)
        return resp, body

好吧，很無奈，最後發現，引數傳到nova當中，其引數並不在body裡，而是在req的body裡面，當時怎麼那麼傻沒打出來看看。。。

3. Nova中

1. /usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py

    def create(self, req, body):
        """Creates a new server for a given user."""

        #其他的地方都省略

            #建立虛擬機器
            (instances, resv_id) = self.compute_api.create(context,
                            inst_type,
                            image_uuid,
                            display_name=name,
                            display_description=description,
                            availability_zone=availability_zone,
                            forced_host=host, forced_node=node,
                            metadata=server_dict.get('metadata', {}),
                            admin_password=password,
                            requested_networks=requested_networks,
                            check_server_group_quota=True,
                            **create_kwargs)

2. /usr/lib/python2.7/site-packages/nova/compute/api.py

    @hooks.add_hook("create_instance")
    def create(self, context, instance_type, 
               image_href, kernel_id=None, ramdisk_id=None,
               min_count=None, max_count=None,
               display_name=None, display_description=None,
               key_name=None, key_data=None, security_group=None,
               availability_zone=None, forced_host=None, forced_node=None,
               user_data=None, metadata=None, injected_files=None,
               admin_password=None, block_device_mapping=None,
               access_ip_v4=None, access_ip_v6=None, requested_networks=None,
               config_drive=None, auto_disk_config=None, scheduler_hints=None,
               legacy_bdm=True, shutdown_terminate=False,
               check_server_group_quota=False):

        # 中間略過。。。

        # 跳到這個檔案的_create_instance函式
        return self._create_instance(
                       context, instance_type, 
                       image_href, kernel_id, ramdisk_id,
                       min_count, max_count,
                       display_name, display_description,
                       key_name, key_data, security_group,
                       availability_zone, user_data, metadata,
                       injected_files, admin_password,
                       access_ip_v4, access_ip_v6,
                       requested_networks, config_drive,
                       block_device_mapping, auto_disk_config,
                       filter_properties=filter_properties,
                       legacy_bdm=legacy_bdm,
                       shutdown_terminate=shutdown_terminate,
                       check_server_group_quota=check_server_group_quota)

    def _create_instance(self, context, instance_type, 
               image_href, kernel_id, ramdisk_id,
               min_count, max_count,
               display_name, display_description,
               key_name, key_data, security_groups,
               availability_zone, user_data, metadata, injected_files,
               admin_password, access_ip_v4, access_ip_v6,
               requested_networks, config_drive,
               block_device_mapping, auto_disk_config, filter_properties,
               reservation_id=None, legacy_bdm=True, shutdown_terminate=False,
               check_server_group_quota=False):
        """Verify all the input parameters regardless of the provisioning
        strategy being performed and schedule the instance(s) for
        creation.
        """

        # 中間略去。。

        # 跳轉到 nova/conductor/api.py
        self.compute_task_api.build_instances(context,
                instances=instances, image=boot_meta,
                filter_properties=filter_properties,
                admin_password=admin_password,
                injected_files=injected_files,
                requested_networks=requested_networks,
                security_groups=security_groups,
                block_device_mapping=block_device_mapping,
                legacy_bdm=False)

        return (instances, reservation_id)

3. /usr/lib/python2.7/site-packages/nova/conductor/api.py

ComputeTaskAPI.build_instances

def build_instances(self, context, instances, image, filter_properties,
            admin_password, injected_files, requested_networks,
            security_groups, block_device_mapping, legacy_bdm=True):
        self.conductor_compute_rpcapi.build_instances(context,
                instances=instances, image=image,
                filter_properties=filter_properties,
                admin_password=admin_password, injected_files=injected_files,
                requested_networks=requested_networks,
                security_groups=security_groups,
                block_device_mapping=block_device_mapping,
                legacy_bdm=legacy_bdm)

4. /usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py

ComputeTaskAPI.build_instances

在這裡就要解釋一下了，Nova中各個服務之間的通訊使用了基於AMQP實現的RPC機制，其中 nova-compute、nova-conductor 和 nova-scheduler 在啟動時都會註冊一個RPC Server。

def build_instances(self, context, instances, image, filter_properties,
            admin_password, injected_files, requested_networks,
            security_groups, block_device_mapping, legacy_bdm=True):

        # 中間略過。。

        # 獲得目標機的RPC Client
        cctxt = self.client.prepare(version=version)

        # 將跳轉至，/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py
        # RPC cast 主要用於非同步形式，比如建立虛擬機器，在建立過程中可能需要很長時間，
        # 如果使用RPC call顯然對效能有很大的影響。cast()的第二個引數是RPC呼叫的函式名，
        # 後面的引數將作為引數被傳入該函式
        cctxt.cast(context, 'build_instances', **kw)

類ComputeAPI只是暴露給其他服務的RPC呼叫介面，Compute服務的RPC Server接受到RPC請求後，真正完成任務的是nova.compute.manager 模組。從ComputeAPI 到ComputeManager的過程即是RPC呼叫過程。

5. /usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py

RPCClient.cast() —> _CallContext.cast()

    def cast(self, ctxt, method, **kwargs):

        self.prepare().cast(ctxt, method, **kwargs)

    def cast(self, ctxt, method, **kwargs):
        """Invoke a method and return immediately. See RPCClient.cast()."""
        msg = self._make_message(ctxt, method, kwargs)
        ctxt = self.serializer.serialize_context(ctxt)

        if self.version_cap:
            self._check_version_cap(msg.get('version'))
        try:
            self.transport._send(self.target, ctxt, msg, retry=self.retry)
        except driver_base.TransportDriverError as ex:
            raise ClientSendError(self.target, ex)

6. 中間各種過程不貼程式碼

_CallContext.cast() —> site-packages/oslo_messaging/transport.py Transport._send()

走到這個transport._send()的之後實在找不到走到哪裡了，知道的小夥伴幫忙說一下吧

7. /usr/lib/python2.7/site-packages/nova/conductor/manager.py

ComputeTaskManager.build_instances

之所以來到了這裡，在nova第4步的註釋中已有解釋，這裡要補充的一件事情就是我遇到的坑。。

在4中，我在kw引數中加了cache_type這個引數，其實它是和這裡的 build_instances 函式相對應的，我在kw裡面加了那個引數但是在這個函式的引數列表中忘記加了（其實是當時根本就不知道那邊最後會走到哪個函式。。。），導致起虛擬機器的時候一直處於scheduling當中，如果也是需要在原始碼基礎上修改的話，一定要記住加！！

    def build_instances(self, context, instances, image, filter_properties,
            admin_password, injected_files, requested_networks,
            security_groups, block_device_mapping=None, legacy_bdm=True):

        # 中間的省略。。
        # 此處的日誌是在 /var/log/nova/nova-conductor.log

            # 此處呼叫的是 nova/compute/rpcapi.py  --->  build_and_run_instance()
            self.compute_rpcapi.build_and_run_instance(context,
                    instance=instance, host=host['host'], image=image,
                    request_spec=request_spec,
                    filter_properties=local_filter_props,
                    admin_password=admin_password,
                    injected_files=injected_files,
                    requested_networks=requested_networks,
                    security_groups=security_groups,
                    block_device_mapping=bdms, node=host['nodename'],
                    limits=host['limits'])

8. /usr/lib/python2.7/site-packages/nova/computer/rpcapi.py

ComputeAPI.build_and_run_instance()

    def build_and_run_instance(self, ctxt, instance, host, image, request_spec,
            filter_properties, admin_password=None, injected_files=None,
            requested_networks=None, security_groups=None,
            block_device_mapping=None, node=None, limits=None):

        version = '4.0'
        cctxt = self.client.prepare(server=host, version=version)

        # 此處又重複了第5、6步的內容，下面這點不再重複的寫。
        # 注意，這裡對應的函式是 build_and_run_instance
        # 其位置在 nova/compute/manager.py   ComputeManager.build_and_run_instance()

        cctxt.cast(ctxt, 'build_and_run_instance', instance=instance,
                image=image, request_spec=request_spec,
                filter_properties=filter_properties,
                admin_password=admin_password,
                injected_files=injected_files,
                requested_networks=requested_networks,
                security_groups=security_groups,
                block_device_mapping=block_device_mapping, node=node,
                limits=limits)

9. /usr/lib/python2.7/site-packages/nova/compute/manager.py

ComputeManager.build_and_run_instance() —> ComputeManager._locked_do_build_and_run_instance() —> ComputeManager._do_build_and_run_instance() —> ComputeManager._build_and_run_instance()

build_and_run_instance

    def build_and_run_instance(self, context, instance, image, request_spec,
                     filter_properties, admin_password=None,
                     injected_files=None, requested_networks=None,
                     security_groups=None, block_device_mapping=None,
                     node=None, limits=None):

        @utils.synchronized(instance.uuid)
        def _locked_do_build_and_run_instance(*args, **kwargs):
            # NOTE(danms): We grab the semaphore with the instance uuid
            # locked because we could wait in line to build this instance
            # for a while and we want to make sure that nothing else tries
            # to do anything with this instance while we wait.
            with self._build_semaphore:
                self._do_build_and_run_instance(*args, **kwargs)

        # NOTE(danms): We spawn here to return the RPC worker thread back to
        # the pool. Since what follows could take a really long time, we don't
        # want to tie up RPC workers.
        # 這個spawn_n 就呼叫 _locked_do_build_and_run_instance 這個函式
        utils.spawn_n(_locked_do_build_and_run_instance,
                      context, instance, image, request_spec,
                      filter_properties, admin_password, injected_files,
                      requested_networks, security_groups,
                      block_device_mapping, node, limits)

_do_build_and_run_instance

    def _do_build_and_run_instance(self, context, instance, image,
            request_spec, filter_properties, admin_password, injected_files,
            requested_networks, security_groups, block_device_mapping,
            node=None, limits=None):

        # 中間的省略。。

        try:
            with timeutils.StopWatch() as timer:
                self._build_and_run_instance(context, instance, image,
                        decoded_files, admin_password, requested_networks,
                        security_groups, block_device_mapping, node, limits,
                        filter_properties)

_build_and_run_instance

    def _build_and_run_instance(self, context, instance, image, injected_files,
            admin_password, requested_networks, security_groups,
            block_device_mapping, node, limits, filter_properties):

        image_name = image.get('name')
        self._notify_about_instance_usage(context, instance, 'create.start',
                extra_usage_info={'image_name': image_name})
        try:

            # 中間省略。。

                        # 從這裡呼叫 /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py 裡面的 spawn()
                        self.driver.spawn(context, instance, cache_type, image_meta,
                                          injected_files, admin_password,
                                          network_info=network_info,
                                          block_device_info=block_device_info)

10. /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py

在這裡進行建立虛擬機器的動作

    def spawn(self, context, instance, image_meta, injected_files,
              admin_password, network_info=None, block_device_info=None):
        disk_info = blockinfo.get_disk_info(CONF.libvirt.virt_type,
                                            instance,
                                            image_meta,
                                            block_device_info)
        self._create_image(context, instance,
                           disk_info['mapping'],
                           network_info=network_info,
                           block_device_info=block_device_info,
                           files=injected_files,
                           admin_pass=admin_password)
        xml = self._get_guest_xml(context, instance, network_info,
                                  disk_info, image_meta,
                                  block_device_info=block_device_info,
                                  write_to_disk=True)
        self._create_domain_and_network(context, xml, instance, network_info,
                                        disk_info,
                                        block_device_info=block_device_info)
        LOG.debug("Instance is running", instance=instance)

        def _wait_for_boot():
            """Called at an interval until the VM is running."""
            state = self.get_info(instance).state

            if state == power_state.RUNNING:
                LOG.info(_LI("Instance spawned successfully."),
                         instance=instance)
                raise loopingcall.LoopingCallDone()

        timer = loopingcall.FixedIntervalLoopingCall(_wait_for_boot)
        timer.start(interval=0.5).wait()

未完待續。。。