場景

虛機只有一張網絡卡能通業務網，但是需要有另一張網絡卡能通管理網。

以下手動給虛機新增一張網絡卡。

VM有nic-1通過br-int連線到物理網絡卡eth0上，需要增加nic-2通過br-bond0連線到物理interface bond0.706上。

原理很簡單：建立Linux bridge br-bond0，把物理介面bond0.706加到br-bond0上，再用virsh attach-interface給VM新增網絡卡nic-2

在CentOS 7上嘗試兩種配置：

配置一

直接修改、新增網絡卡配置檔案。
1. 去掉bond0.706上配置的IP，並增加BRIDGE屬性


[

br-bond0就是網橋名，現在還沒有這個網橋，下一步中配置。

1. 新增br-bond0配置檔案

   [[email protected] ~]# cat /etc/sysconfig/network-scripts/ifcfg-br-bond0
   NAME=br-bond0
   BOOTPROTO=static
   DEVICE=br-bond0
   ONBOOT=yes
   TYPE=Bridge
   NM_CONTROLLER=no
   PREFIX=24
   IPADDR=172.16.231.27
   GATEWAY=172.16.231.1
   VLAN=yes

   注意，

   • 原來bond0.706上的IP地址放到了這個bridge上

   • TYPE=Bridge，區分大小寫！我一開始寫成了TYPE=bridge，重啟之後死活報錯：

   May 17 07:15:01 NFJD-TESTN-COMPUTE-3 network: Bringing up interface br-bond0:  ERROR    : [/etc/sysconfig/network-scripts/ifup-eth] Device br-bond0 does not seem to be present, delaying initialization.
   May 17 07:15:01 NFJD-TESTN-COMPUTE-3 /etc/sysconfig/network-scripts/ifup-eth: Device br-bond0 does not seem to be present, delaying initialization.
   May 17 07:15:01 NFJD-TESTN-COMPUTE-3 network: [FAILED]

   檢視/etc/sysconfig/network-scripts/ifup-eth指令碼後發現TYPE一定要寫成Bridge。（實際上這個指令碼也是用brctl等命令操作的）

2. 給VM新增網絡卡

   [root@NFJD-TESTN-COMPUTE-3 ~]# virsh attach-interface instance-000005ac --type bridge --source br-bond0
   Interface attached successfully
   
   [root@NFJD-TESTN-COMPUTE-3 ~]# virsh domiflist instance-000005ac
   Interface  Type       Source     Model       MAC
   <hr />
   
   tape6ffc79e-b4 bridge     alubr0     virtio      fa:16:3e:83:d7:0b
   vnet3      bridge     br-bond0   rtl8139     52:54:00:69:b5:d2

3. 在VM中配置靜態IP

   [root@bigdata04 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens7
   BOOTPROTO=static
   NAME=ens7
   DEVICE=ens7
   ONBOOT=yes
   IPADDR=172.16.231.120
   PREFIX=25
   [root@bigdata04 ~]# ifconfig
   ens7: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
       inet 172.16.231.120  netmask 255.255.255.128  broadcast 172.16.231.127
       inet6 fe80::5054:ff:fee2:6be0  prefixlen 64  scopeid 0x20<link>
       ether 52:54:00:e2:6b:e0  txqueuelen 1000  (Ethernet)
       RX packets 373  bytes 23929 (23.3 KiB)
       RX errors 0  dropped 0  overruns 0  frame 0
       TX packets 100  bytes 12967 (12.6 KiB)
       TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
   
   eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
       inet 10.10.10.5  netmask 255.255.255.0  broadcast 10.10.10.255
       inet6 fe80::f816:3eff:fe83:d70b  prefixlen 64  scopeid 0x20<link>
       ether fa:16:3e:83:d7:0b  txqueuelen 1000  (Ethernet)
       RX packets 9  bytes 976 (976.0 B)
       RX errors 0  dropped 0  overruns 0  frame 0
       TX packets 8  bytes 920 (920.0 B)
       TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
   
   lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
       inet 127.0.0.1  netmask 255.0.0.0
       inet6 ::1  prefixlen 128  scopeid 0x10<host>
       loop  txqueuelen 0  (Local Loopback)
       RX packets 0  bytes 0 (0.0 B)
       RX errors 0  dropped 0  overruns 0  frame 0
       TX packets 0  bytes 0 (0.0 B)
       TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

4. 修改nova-compute啟動配置
   物理機重啟時，經常看到nova-compute連不上libvirtd而啟動失敗，所以修改nova-compute啟動配置：

   [[email protected] ~]# cat /usr/lib/systemd/system/openstack-nova-compute.service
   [Unit]
   Description=OpenStack Nova Compute Server
   After=syslog.target network.target libvirtd.service     # 把libvirtd.service加上
   
   #After=syslog.target network.target
   
   
   [Service]
   Environment=LIBGUESTFS_ATTACH_METHOD=appliance
   Type=notify
   NotifyAccess=all
   TimeoutStartSec=0
   Restart=always
   User=nova
   ExecStart=/usr/bin/nova-compute
   
   [Install]
   WantedBy=multi-user.target

注意：
物理機重啟時，nova會從資料庫讀取配置並重新生成VM的libvirt.xml檔案，所以新增網絡卡會消失，需要再次執行virsh attach-interface命令。如果是用virsh啟動的虛機，可以用virsh dumpxml instance-000005ac > /etc/libvirt/qemu/instance-000005ac.xml儲存新增加的網絡卡配置。

配置二

用virsh、brctl配置（另一個計算節點）

1. 和配置一中的第一步一樣，去掉bond0.706上配置的IP ，但不增加BRIDGE屬性。

2. 建立nic.xml

   [[email protected] ~]# cat nic.xml
   <network>
       <name>virsh-net-0</name>
       <bridge name="br-bond0"/>
       <forward mode="route" dev="bond0"/>
       <model type='virtio'/>
       <ip address="172.16.171.2" netmask="255.255.255.0">
       </ip>
   </network>

   建立一個Linux bridge名叫br-bond0，給橋配置上IP地址。
   <forward mode="route" dev="bond0"/>這個沒太懂，可以看libvirt官方解釋1、官方解釋2:

   forward - This element is optional. When not defined, the virtual network will work in isolated mode. However, inclusion of this element indicates that the virtual network is to be connected to the physical network. The element can have two attributes, ‘mode’ and ‘dev‘. The first one specifies the mode in which will the virtual bridge operates. Allowed values are ‘nat’ and ‘route‘. The second attribute is used whenever one wants to restrict forwarding to the named device only. If no attributes are set, NAT forwarding will be used for connectivity. Firewall rules will allow forwarding to any other network device.

   簡單來說，如果沒有forward引數，則這個net是isolated network。如果要配，則有兩種模式：1) nat：虛機IP外部不可見：出去時做IP轉換；2) route：虛機IP外部可見

3. 建立libvirt net

   [root@NFJD-PSC-IBMN-SV502 ~]# virsh net-define nic.xml
   Network virsh-net-0 defined from nic.xml
   
   [root@NFJD-PSC-IBMN-SV502 ~]# virsh net-list --all
   Name                 State      Autostart     Persistent
   <hr />
   
   default              active     yes           yes
   virsh-net-0          inactive   no            yes

4. 啟動這個net，並設定為自啟動

   [[email protected] ~]# virsh net-start virsh-net-0
   Network virsh-net-0 started
   [[email protected] ~]# virsh net-autostart virsh-net-0
   Network virsh-net-0 marked as autostarted

   autostart表示libvirtd重啟時，會自動start這個net

   [[email protected] ~]# virsh net-info virsh-net-0
   Name:           virsh-net-0
   UUID:           19442a5d-45fd-4fe7-bded-caced3a137c1
   Active:         yes
   Persistent:     yes
   Autostart:      yes
   Bridge:         br-bond0
   
   [[email protected] ~]# virsh net-dumpxml virsh-net-0
   <network>
   <name>virsh-net-0</name>
   <uuid>19442a5d-45fd-4fe7-bded-caced3a137c1</uuid>
   <forward dev='bond0' mode='route'>
   <interface dev='bond0'/>
   </forward>
   <bridge name='br-bond0' stp='on' delay='0'/>
   <mac address='52:54:00:e9:7e:39'/>
   <ip address='172.16.171.2' netmask='255.255.255.0'>
   </ip>
   </network>

   此時可以看到bridge br-bond0

   [root@NFJD-PSC-IBMN-SV502 ~]# brctl show
   bridge name    bridge id        STP enabled    interfaces
   br-bond0        8000.525400e97e39    yes        br-bond0-nic
   virbr0        8000.5254003ea676    yes        virbr0-nic

5. 把bond0加到br-bond0上

   [root@NFJD-PSC-IBMN-SV502 ~]# brctl addif br-bond0 bond0
   [root@NFJD-PSC-IBMN-SV502 ~]# brctl show
   bridge name    bridge id        STP enabled    interfaces
   br-bond0        8000.525400e97e39    yes        bond0   <=== 多了bond0
                           br-bond0-nic
   virbr0        8000.5254003ea676    yes        virbr0-nic

6. 給VM新增網絡卡

   [[email protected]-PSC-IBMN-SV502 ~]# virsh attach-interface instance-000a039d --type bridge --source br-bond0
   Interface attached successfully
   
   [[email protected]-PSC-IBMN-SV502 ~]# virsh domiflist instance-000a039d
   Interface  Type       Source     Model       MAC
   <hr />
   
   tap6d788a74-ea bridge     alubr0     virtio      fa:16:3e:ea:02:2c
   vnet0      bridge     br-bond0   rtl8139     52:54:00:22:33:c7

7. 設定/etc/rc.local

   [[email protected] ~]# tail /etc/rc.local
   touch /var/lock/subsys/local
   
   
   #Added by eldon
   
   sleep 3
   
   #ifconfig bond0 down
   
   
   #virsh net-start virsh-net-0
   
   
   #ifconfig bond0 up
   
   brctl addif br-bond0 bond0
   
   route add -net 0.0.0.0 gateway 172.16.171.254

   如果不sleep 3：

   May 17 21:49:38 NFJD-PSC-IBMN-SV502 rc.local: bridge br-bond0 does not exist!

   還曾嘗試不使用brctl addif br-bond0 bond0，直接建立ifcfg-br-bond0，但是在物理機啟動時/var/log/messages顯示virsh net-start virsh-net-0時br-bond0已經存在了。

後續