OpenStack實踐SR-IOV計算節點

OpenStack MacOS · 發表 2018-10-15 11:36:47

摘要： SR-IOV 使一個單一的功能單元（比如，一個乙太網埠）能看起來像多個獨立的物理裝置，即支援SR-IOV 功能的物理裝置能被配置為多個功能單元。 SR-IOV 兩種功能（function）：物理功能（Physical Functions，PF）：這是完整的帶有 SR-IOV ...

SR-IOV 使一個單一的功能單元（比如，一個乙太網埠）能看起來像多個獨立的物理裝置，即支援SR-IOV 功能的物理裝置能被配置為多個功能單元。

SR-IOV 兩種功能（function）：

物理功能（Physical Functions，PF）：這是完整的帶有 SR-IOV 能力的PCIe 裝置。PF 能像普通 PCI 裝置那樣被發現、管理和配置。

虛擬功能（Virtual Functions，VF）：簡單的 PCIe 功能，它只能處理I/O。每個 VF 都是從 PF 中分離出來的。每個物理硬體都有一個 VF 數目的限制。一個 PF，能被虛擬成多個 VF 用於分配給多個虛擬機器。

Hypervisor 能將一個或者多個 VF 分配給一個虛機。在某一時刻，一個 VF 只能被分配給一個虛機。一個虛機可以擁有多個 VF。在虛機的作業系統看來，一個 VF 網絡卡看起來和一個普通網絡卡沒有區別。SR-IOV 驅動是在核心中實現的。

網絡卡 SR-IOV 的例子：

將sriov計算節點的PF虛擬化成多個VF

sriov計算節點選擇光口1（例如 enp5s0f1）作為sriov網絡卡

1) 在計算節點上，設定BIOS，對於Intel使支援VT-d，可通過cat /proc/cpuinfo | grep vmx驗證

2) 配置計算節點的/etc/default/grub檔案，在GRUB_CMDLINE_LINUX中新增intel_iommu=on來啟用VT-d功能，重啟物理機（本環境採用intel 82599系列網絡卡）

$ cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=bclinux/root rd.lvm.lv=bclinux/swap intel_iommu=on rhgb quiet"
GRUB_DISABLE_RECOVERY="true"

$ grub2-mkconfig --output=/boot/grub2/grub.cfg
$ reboot

3) 本環境中82599網絡卡最多支援64個vf，擬在每個sriov計算節點虛擬出16個vf供使用命令如下：

# echo '16' > /sys/class/net/enp5s0f1/device/sriov_numvfs
###檢視是否生效
# lspci | grep Ethernet

05:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
05:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
05:10.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:10.3 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:10.5 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:10.7 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:11.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:11.3 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:11.5 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:11.7 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:12.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:12.3 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:12.5 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:12.7 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:13.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:13.3 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:13.5 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
05:13.7 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
08:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
08:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)

# ip link show enp5s0f1

5: enp5s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether 6c:92:bf:04:66:01 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off

4) 保證重啟後vfs仍然存在，需要將設定vf的數目命令寫入/etc/rc.local，命令如下

# echo "echo '16' > /sys/class/net/enp5s0f1/device/sriov_numvfs" >> /etc/rc.local
# chmod +x /etc/rc.local /etc/rc.d/rc.local

5) 修改控制節點nova.conf檔案，讓nova-schedule支援對pci passthrough的過濾

# vim /etc/nova/nova.conf
enable_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter
available_filters = nova.scheduler.filters.all_filters

# systemctl restart openstack-nova-scheduler

配置SRIOV虛機計算節點

1) 安裝neutron-sriov-agent

# yum install openstack-neutron-sriov-nic-agent openstack-neutron

2) 配置neutron.conf

可以將控制節點neutron.conf配置拷貝過來，修改相應引數即可，主要是修改本地管理網ip引數

3) 配置sriov_agent.ini

# vim /etc/neutron/plugins/ml2/sriov_agent.ini

[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

[sriov_nic]
physical_device_mappings = default: enp5s0f1
exclude_devices =

4) 啟動neutron-sriov-agent服務

# systemctl enable neutron-sriov-nic-agent
# systemctl start neutron-sriov-nic-agent

5) 修改該節點nova.conf檔案，重啟nova-compute

# vim /etc/nova/nova.conf
passthrough_whitelist = {"devname": "enp5s0f1", "physical_network": "physnet1"}

# systemctl restart openstack-nova-compute

.. note:: 配置多SRIOV網絡卡

[root@controller ~]# vim /etc/nova/nova.conf
[pci]
passthrough_whitelist = {"devname": "enp129s0f0", "physical_network": "physnet1"}
passthrough_whitelist = {"devname": "enp130s0f0", "physical_network": "physnet1"}
[root@SRIOV03 ~]# vim /etc/neutron/plugins/ml2/sriov_agent.ini 
[sriov_nic]
physical_device_mappings = physnet1:enp129s0f0,physnet1:enp130s0f0

測試srivo非親和功能

即：設定vm的兩個vnic在同一個非親和組，生成的sriov虛擬機器的兩個vnic會分佈在不同的兩個物理SRIOV網絡卡上

[root@controller ~]# openstack port create --vnic-type direct --binding-profile anti_affinity_group=gwgh729 --network 40fa3655-1dc6-4fa1-8a21-507ae2a92cc5 sriov-port-1
+-----------------------+-----------------------------------------------------------------------------+
| Field| Value|
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up| UP|
| allowed_address_pairs ||
| binding_host_id||
| binding_profile| anti_affinity_group='gwgh729'|
| binding_vif_details||
| binding_vif_type| unbound|
| binding_vnic_type| direct|
| created_at| 2018-10-11T07:11:06Z|
| data_plane_status| None|
| description||
| device_id||
| device_owner||
| dns_assignment| None|
| dns_name| None|
| extra_dhcp_opts||
| fixed_ips| ip_address='192.168.5.48', subnet_id='8e026845-10d0-486b-9e34-45f2f2f01ada' |
| id| cb96ed5a-9bbc-46f7-9a44-fdc45e11b25b|
| ip_address| None|
| mac_address| fa:16:3e:c8:f7:9f|
| name| sriov-port-1|
| network_id| 40fa3655-1dc6-4fa1-8a21-507ae2a92cc5|
| option_name| None|
| option_value| None|
| port_security_enabled | True|
| project_id| 39db8ee1aced4109aa7ea45c9b45aa97|
| qos_policy_id| None|
| revision_number| 3|
| security_group_ids| 46a06d50-4ed7-495f-9458-eab6843495a8|
| status| DOWN|
| subnet_id| None|
| tags||
| trunk_details| None|
| updated_at| 2018-10-11T07:11:07Z|
+-----------------------+-----------------------------------------------------------------------------+
[root@controller ~]# openstack port create --vnic-type direct --binding-profile anti_affinity_group=gwgh729 --network 40fa3655-1dc6-4fa1-8a21-507ae2a92cc5 sriov-port-2
+-----------------------+-----------------------------------------------------------------------------+
| Field| Value|
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up| UP|
| allowed_address_pairs ||
| binding_host_id||
| binding_profile| anti_affinity_group='gwgh729'|
| binding_vif_details||
| binding_vif_type| unbound|
| binding_vnic_type| direct|
| created_at| 2018-10-11T07:11:17Z|
| data_plane_status| None|
| description||
| device_id||
| device_owner||
| dns_assignment| None|
| dns_name| None|
| extra_dhcp_opts||
| fixed_ips| ip_address='192.168.5.60', subnet_id='8e026845-10d0-486b-9e34-45f2f2f01ada' |
| id| d8ae3337-b98a-4523-9665-ed7773c6da90|
| ip_address| None|
| mac_address| fa:16:3e:23:0e:45|
| name| sriov-port-2|
| network_id| 40fa3655-1dc6-4fa1-8a21-507ae2a92cc5|
| option_name| None|
| option_value| None|
| port_security_enabled | True|
| project_id| 39db8ee1aced4109aa7ea45c9b45aa97|
| qos_policy_id| None|
| revision_number| 3|
| security_group_ids| 46a06d50-4ed7-495f-9458-eab6843495a8|
| status| DOWN|
| subnet_id| None|
| tags||
| trunk_details| None|
| updated_at| 2018-10-11T07:11:17Z|
+-----------------------+-----------------------------------------------------------------------------+
[root@controller ~]# nova boot han-sriov-antiaffinity --image 4ce53c0e-b3f5-4463-bdcd-20ee0f74fec6 --flavor 071b6288-8087-4d2e-959a-0b3bbb7fc565 --nic port-id=cb96ed5a-9bbc-46f7-9a44-fdc45e11b25b --nic port-id=d8ae3337-b98a-4523-9665-ed7773c6da90 --availability-zone sriov:SRIOV03
+--------------------------------------+------------------------------------------------------+
| Property| Value|
+--------------------------------------+------------------------------------------------------+
| OS-DCF:diskConfig| MANUAL|
| OS-EXT-AZ:availability_zone| sriov|
| OS-EXT-SRV-ATTR:host| -|
| OS-EXT-SRV-ATTR:hostname| han-sriov-antiaffinity|
| OS-EXT-SRV-ATTR:hypervisor_hostname| -|
| OS-EXT-SRV-ATTR:instance_name||
| OS-EXT-SRV-ATTR:kernel_id||
| OS-EXT-SRV-ATTR:launch_index| 0|
| OS-EXT-SRV-ATTR:ramdisk_id||
| OS-EXT-SRV-ATTR:reservation_id| r-3vjlmls4|
| OS-EXT-SRV-ATTR:root_device_name| -|
| OS-EXT-SRV-ATTR:user_data| -|
| OS-EXT-STS:power_state| 0|
| OS-EXT-STS:task_state| scheduling|
| OS-EXT-STS:vm_state| building|
| OS-SRV-USG:launched_at| -|
| OS-SRV-USG:terminated_at| -|
| accessIPv4||
| accessIPv6||
| adminPass| Z255abC53sLN|
| config_drive||
| created| 2018-10-11T07:12:43Z|
| description| -|
| flavor:disk| 10|
| flavor:ephemeral| 0|
| flavor:extra_specs| {}|
| flavor:original_name| 2u4g10g|
| flavor:ram| 4096|
| flavor:swap| 0|
| flavor:vcpus| 2|
| hostId||
| host_status||
| id| ea80e14b-2b9a-4a31-ad27-94366fa600bf|
| image| centos7-10-08 (4ce53c0e-b3f5-4463-bdcd-20ee0f74fec6) |
| key_name| -|
| locked| False|
| metadata| {}|
| name| han-sriov-antiaffinity|
| os-extended-volumes:volumes_attached | []|
| progress| 0|
| security_groups| default|
| status| BUILD|
| tags| []|
| tenant_id| 39db8ee1aced4109aa7ea45c9b45aa97|
| updated| 2018-10-11T07:12:43Z|
| user_id| f6b12716402343e6beacdaa265e4e6c5|
+--------------------------------------+------------------------------------------------------+
[root@controller ~]# nova show han-sriov-antiaffinity
+--------------------------------------+----------------------------------------------------------+
| Property| Value|
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig| MANUAL|
| OS-EXT-AZ:availability_zone| sriov|
| OS-EXT-SRV-ATTR:host| SRIOV03|
| OS-EXT-SRV-ATTR:hostname| han-sriov-antiaffinity|
| OS-EXT-SRV-ATTR:hypervisor_hostname| SRIOV03|
| OS-EXT-SRV-ATTR:instance_name| instance-00000cfd|
| OS-EXT-SRV-ATTR:kernel_id||
| OS-EXT-SRV-ATTR:launch_index| 0|
| OS-EXT-SRV-ATTR:ramdisk_id||
| OS-EXT-SRV-ATTR:reservation_id| r-3vjlmls4|
| OS-EXT-SRV-ATTR:root_device_name| /dev/vda|
| OS-EXT-SRV-ATTR:user_data| -|
| OS-EXT-STS:power_state| 1|
| OS-EXT-STS:task_state| -|
| OS-EXT-STS:vm_state| active|
| OS-SRV-USG:launched_at| 2018-10-11T07:14:49.000000|
| OS-SRV-USG:terminated_at| -|
| accessIPv4||
| accessIPv6||
| config_drive||
| created| 2018-10-11T07:12:42Z|
| description| -|
| flavor:disk| 10|
| flavor:ephemeral| 0|
| flavor:extra_specs| {}|
| flavor:original_name| 2u4g10g|
| flavor:ram| 4096|
| flavor:swap| 0|
| flavor:vcpus| 2|
| hostId| c9ae967a850ab4bf64f157ac3c05ef0adb86743d3a1f86115cda7f04 |
| host_status| UP|
| id| ea80e14b-2b9a-4a31-ad27-94366fa600bf|
| image| centos7-10-08 (4ce53c0e-b3f5-4463-bdcd-20ee0f74fec6)|
| key_name| -|
| locked| False|
| metadata| {}|
| name| han-sriov-antiaffinity|
| os-extended-volumes:volumes_attached | []|
| progress| 0|
| security_groups| default|
| status| ACTIVE|
| tags| []|
| tenant_id| 39db8ee1aced4109aa7ea45c9b45aa97|
| updated| 2018-10-11T07:13:37Z|
| user_id| f6b12716402343e6beacdaa265e4e6c5|
| vlan811-common network| 192.168.5.48, 192.168.5.60|
+--------------------------------------+----------------------------------------------------------+

[root@SRIOV03 ~]# ip link show enp130s0f0|grep fa:16:3e:c8:f7:9f 
[root@SRIOV03 ~]# ip link show enp129s0f0|grep fa:16:3e:c8:f7:9f 
vf 0 MAC fa:16:3e:c8:f7:9f, vlan 811, spoof checking on, link-state auto, trust off, query_rss off
[root@SRIOV03 ~]# ip link show enp130s0f0|grep fa:16:3e:23:0e:45
vf 0 MAC fa:16:3e:23:0e:45, vlan 811, spoof checking on, link-state auto, trust off, query_rss off
[root@SRIOV03 ~]# ip link show enp129s0f0|grep fa:16:3e:23:0e:45
[root@SRIOV03 ~]#

mac地址為fa:16:3e:c8:f7:9f的vnic落在enp129s0f0，mac地址為fa:16:3e:23:0e:45的vnic落在enp130s0f0

OpenStack實踐SR-IOV計算節點

將sriov計算節點的PF虛擬化成多個VF

配置SRIOV虛機計算節點

測試srivo非親和功能

您可能也會喜歡…