OVS + dpdk 安裝與實驗環境配置
阿新 • • 發佈:2017-06-05
binding exp pidfile *** 掛載 rem 環境變量 rep new
$ grep Huge /proc/meminfo
$ echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
$ mkdir /mnt/huge
$ mount -t hugetlbfs nodev /mnt/huge
2:編譯DPDK,在DPDK源代碼目錄中
設置環境變量
DPDK_DIR = "DPDK源代碼目錄"
$ export DPDK_TARGET=x86_64-native-linuxapp-gcc
$ export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
編譯
$ make install T=$DPDK_TARGET DESTDIR=install
3:安裝OVS,在OVS源代碼目錄中
3.1:配置configure
$ ./boot.sh
$ ./configure --with-dpdk=$DPDK_BUILD
3.2:編譯OVS
$ make
3.3:安裝OVS
$ make install
3.4:創建OVSDB schema
$ mkdir -p /usr/local/etc/openvswitch
$ ovsdb-tool create /usr/local/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema
3.5:啟動OVSDB
$ mkdir -p /usr/local/var/run/openvswitch
$ ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
--remote=db:Open_vSwitch,Open_vSwitch,manager_options \
--private-key=db:Open_vSwitch,SSL,private_key \
--certificate=db:Open_vSwitch,SSL,certificate \
--bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \
--pidfile --detach --log-file=/root/log/ovsdb-server.log
4:掛載兼容的UIO驅動模塊
$ modprobe uio_pci_generic
或
$ modprobe vfio-pci
$ chmod a+x /dev/vfio
$ chmod 0666 /dev/vfio/*
5:把網卡綁定到兼容的UIO驅動上
6:啟動vswitchd
$ export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
$ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
$ ovs-vswitchd unix:$DB_SOCK --pidfile --detach --log-file=/root/log/ovs-vswitchd.log
7:創建datapath為netdev類型的bridge
$ ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
8:測試:添加dpdk類型的port,
$ ovs-vsctl add-port br0 dpdk0 \
-- set Interface dpdk0 type=dpdk options:dpdk-devargs=00:09.0
接下來就是實驗環境了 ------------------------------[host 1]--------------------------------------- | | | (br-int datapath_type=netdev) | | [vm0]---------[port:vm0 type=dpdkvhostuser] | | [port:vxlan0 type=vxlan remote_ip=10.0.0.2] | | | | | | (br-phy datapath_type=netdev other_config:hwaddr=aa:aa:aa:aa:aa:aa) | | [port:dpdk0 type=dpdk options:dpdk-devargs= NIC] | | | | | ------------------------- [ NIC binding to DPDK]----------------------------- | | | | [physical switch] | | | | ------------------------- [ NIC binding to DPDK]------------------------------ | | | | | [port:dpdk1 type=dpdk options:dpdk-devargs= NIC] | | (br-phy datapath_type=netdev other_config:hwaddr=bb:bb:bb:bb:bb:bb) | | | | | | [port:vxlan1 type=vxlan remote_ip=10.0.0.1] | | [port:vm1 type=dpdkvhostuser]------------- [vm1] | | (br-int datapath_type=netdev) | | | ------------------------------[host 2]---------------------------------------- 在host1與host2上安裝好OVS+DPDK,然後執行如下的動作,兩個host上執行的動作相同,以host1為例 1:創建br-int $ ovs-vsctl --may-exist add-br br-int \ -- set Bridge br-int datapath_type=netdev \ -- br-set-external-id br-int bridge-id br-int \ -- set bridge br-int fail-mode=standalone 2:向br-int中添加dpdkvhostuser型port vm0/vm1 對於host1,如下: $ ovs-vsctl add-port br-int vm0 \ -- set Interface vm0 type=dpdkvhostuser 對於host2,如下: $ ovs-vsctl add-port br-int vm1 \ -- set Interface vm1 type=dpdkvhostuser 3:創建虛擬機,以virsh配合XML配置文件的形式創建虛擬機,XML文件如下: vm0.xml 4:在虛擬機內,配置IP 對於host1上的vm0 $ ip addr add 192.168.1.1/24 對於host2上的vm1 $ ip addr add 192.168.1.2/24 5:向br-int中添加vxlan tunnel 對於host1,如下: $ ovs-vsctl add-port br-int vxlan0 \ -- set interface vxlan0 type=vxlan options:remote_ip=10.0.0.2 對於host2,如下: $ ovs-vsctl add-port br-int vxlan0 \ -- set interface vxlan0 type=vxlan options:remote_ip=10.0.0.1 6:創建br-phy 對於host1,如下: $ ovs-vsctl --may-exist add-br br-phy \ -- set Bridge br-phy datapath_type=netdev \ -- br-set-external-id br-phy bridge-id br-phy \ -- set bridge br-phy fail-mode=standalone \ other_config:hwaddr=aa:aa:aa:aa:aa:aa 對於host2,如下: $ ovs-vsctl --may-exist add-br br-phy \ -- set Bridge br-phy datapath_type=netdev \ -- br-set-external-id br-phy bridge-id br-phy \ -- set bridge br-phy fail-mode=standalone \ other_config:hwaddr=bb:bb:bb:bb:bb:bb 7:將物理網卡綁到DPDK兼容的UIO驅動上 略 8:將物理網卡捅到br-phy上 對於host1,如下: $ ovs-vsctl add-port br-phy dpdk0 \ -- set Interface dpdk0 type=dpdk options:dpdk-devargs=??????? 對於host2,如下: $ ovs-vsctl add-port br-phy dpdk1 \ -- set Interface dpdk0 type=dpdk options:dpdk-devargs=??????? 9:配置br-phy 對於host1,如下: $ ip addr add 10.0.0.1/24 dev br-phy $ ip link set br-phy up $ iptables -F 對於host2,如下: $ ip addr add 10.0.0.2/24 dev br-phy $ ip link set br-phy up $ iptables -F
參考文檔: http://docs.openvswitch.org/en/latest/howto/userspace-tunneling/ http://docs.openvswitch.org/en/latest/topics/dpdk/vhost-user/ https://libvirt.org/formatdomain.html#elementVhostuser http://www.sdnlab.com/16593.html
***DPDK datapath的OVS的安裝與實驗環境配置
首先肯定是DPDK的安裝 0:安裝必要的工具 make gcc libssl libcap-ng0 libtool autoconf python 2.7 python module six(可用PIP安裝) 1:設置hugepages,host最好有4G以上的內存可用, 共搞1024個hugepage,每個page 2M,所以光hugepages就占用2G內存接下來就是實驗環境了 ------------------------------[host 1]--------------------------------------- | | | (br-int datapath_type=netdev) | | [vm0]---------[port:vm0 type=dpdkvhostuser] | | [port:vxlan0 type=vxlan remote_ip=10.0.0.2] | | | | | | (br-phy datapath_type=netdev other_config:hwaddr=aa:aa:aa:aa:aa:aa) | | [port:dpdk0 type=dpdk options:dpdk-devargs= NIC] | | | | | ------------------------- [ NIC binding to DPDK]----------------------------- | | | | [physical switch] | | | | ------------------------- [ NIC binding to DPDK]------------------------------ | | | | | [port:dpdk1 type=dpdk options:dpdk-devargs= NIC] | | (br-phy datapath_type=netdev other_config:hwaddr=bb:bb:bb:bb:bb:bb) | | | | | | [port:vxlan1 type=vxlan remote_ip=10.0.0.1] | | [port:vm1 type=dpdkvhostuser]------------- [vm1] | | (br-int datapath_type=netdev) | | | ------------------------------[host 2]---------------------------------------- 在host1與host2上安裝好OVS+DPDK,然後執行如下的動作,兩個host上執行的動作相同,以host1為例 1:創建br-int $ ovs-vsctl --may-exist add-br br-int \ -- set Bridge br-int datapath_type=netdev \ -- br-set-external-id br-int bridge-id br-int \ -- set bridge br-int fail-mode=standalone 2:向br-int中添加dpdkvhostuser型port vm0/vm1 對於host1,如下: $ ovs-vsctl add-port br-int vm0 \ -- set Interface vm0 type=dpdkvhostuser 對於host2,如下: $ ovs-vsctl add-port br-int vm1 \ -- set Interface vm1 type=dpdkvhostuser 3:創建虛擬機,以virsh配合XML配置文件的形式創建虛擬機,XML文件如下: vm0.xml 4:在虛擬機內,配置IP 對於host1上的vm0 $ ip addr add 192.168.1.1/24 對於host2上的vm1 $ ip addr add 192.168.1.2/24 5:向br-int中添加vxlan tunnel 對於host1,如下: $ ovs-vsctl add-port br-int vxlan0 \ -- set interface vxlan0 type=vxlan options:remote_ip=10.0.0.2 對於host2,如下: $ ovs-vsctl add-port br-int vxlan0 \ -- set interface vxlan0 type=vxlan options:remote_ip=10.0.0.1 6:創建br-phy 對於host1,如下: $ ovs-vsctl --may-exist add-br br-phy \ -- set Bridge br-phy datapath_type=netdev \ -- br-set-external-id br-phy bridge-id br-phy \ -- set bridge br-phy fail-mode=standalone \ other_config:hwaddr=aa:aa:aa:aa:aa:aa 對於host2,如下: $ ovs-vsctl --may-exist add-br br-phy \ -- set Bridge br-phy datapath_type=netdev \ -- br-set-external-id br-phy bridge-id br-phy \ -- set bridge br-phy fail-mode=standalone \ other_config:hwaddr=bb:bb:bb:bb:bb:bb 7:將物理網卡綁到DPDK兼容的UIO驅動上 略 8:將物理網卡捅到br-phy上 對於host1,如下: $ ovs-vsctl add-port br-phy dpdk0 \ -- set Interface dpdk0 type=dpdk options:dpdk-devargs=??????? 對於host2,如下: $ ovs-vsctl add-port br-phy dpdk1 \ -- set Interface dpdk0 type=dpdk options:dpdk-devargs=??????? 9:配置br-phy 對於host1,如下: $ ip addr add 10.0.0.1/24 dev br-phy $ ip link set br-phy up $ iptables -F 對於host2,如下: $ ip addr add 10.0.0.2/24 dev br-phy $ ip link set br-phy up $ iptables -F
參考文檔: http://docs.openvswitch.org/en/latest/howto/userspace-tunneling/ http://docs.openvswitch.org/en/latest/topics/dpdk/vhost-user/ https://libvirt.org/formatdomain.html#elementVhostuser http://www.sdnlab.com/16593.html
OVS + dpdk 安裝與實驗環境配置