Docke 使用Open vSwitch實現跨主機通信
阿新 • • 發佈：2019-03-31
byte multicast del amp ces ets port sysctl warn
Node 172.16.17.2 docker 192.168.42.0/24
鑒於centos6系統的限制 這裏選擇openvswitch-2.3.0的版本 下面有關於centos6安裝時遇到的各種問題
具體的實現步驟如下：
--xxong親自驗證
目前市面上主流的解決方法有flannel, weave, Pipework, Open vSwitch等。
Open vSwitch實現比較簡單，成熟且功能強大，所以很適合作為解決docker底層網絡互聯互通的工具。
拓撲如下：
重點 我這裏兩個宿主機的系統分別是 ： Master ： centos7.2 node ：centos6.10
Node 172.16.17.2 docker 192.168.42.0/24
鑒於centos6系統的限制 這裏選擇openvswitch-2.3.0的版本 下面有關於centos6安裝時遇到的各種問題
具體的實現步驟如下：
1. 安裝docker, bridge-utils和openvswitch [root@master ~]# yum install docker bridge-utils -y [root@master ~]# yum install wget openssl-devel -y [root@master ~]# yum groupinstall "Development Tools" [root@master ~]# adduser ovswitch [root@master ~]# su - ovswitch [ovswitch@master ~]$ wget http://openvswitch.org/releases/openvswitch-2.3.0.tar.gz [ovswitch@master ~]$ tar -zxvpf openvswitch-2.3.0.tar.gz [ovswitch@master ~]$ mkdir -p ~/rpmbuild/SOURCES [ovswitch@master ~]$ sed ‘s/openvswitch-kmod, //g‘ openvswitch-2.3.0/rhel/openvswitch.spec > openvswitch-2.3.0/rhel/openvswitch_no_kmod.spec [ovswitch@master ~]$ cp openvswitch-2.3.0.tar.gz rpmbuild/SOURCES/ [ovswitch@master ~]$ rpmbuild -bb --without check ~/openvswitch-2.3.0/rhel/openvswitch_no_kmod.spec 這裏如果報錯 No build ID found in.... 需要在openvswitch_no_kmod.spec文件中加入 %define __debug_install_post %{_rpmconfigdir}/find-debuginfo.sh %{?_find_debuginfo_opts} "%{_builddir}/%{?buildsubdir}"%{nil} [ovswitch@master ~]$ exit [root@master ~]# yum localinstall /home/ovswitch/rpmbuild/RPMS/x86_64/openvswitch-2.3.0-1.x86_64.rpm -y 註意：相應的位置只有這一個rpm包 切記： 如果裝1.9.3的版本的話 這裏是有倆個包的 但是網上文檔基本都只顯示一個rpm包 也就是是說在這上一步 需要rpmbuild 一個openvswitch.spec否則，進行這一步時會報錯 缺依賴； 並且，這裏可能會提示你需要glibc2.14 ,表示glibc版本太低 需要升級glibc版本，我這邊的話直接rpm 下載glibc來進行升級的 這裏的話提供一個鏈接：https://www.cnblogs.com/dpf-learn/p/8763696.html [root@master ~]# mkdir /etc/openvswitch [root@master ~]# setenforce 0 2. docker master端配置如下 [root@master ~]# systemctl start openvswitch.service && systemctl enable openvswitch.service [root@master ~]# ovs-vsctl add-br br0 [root@master ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP qlen 1000 link/ether 00:0c:29:97:92:e8 brd ff:ff:ff:ff:ff:ff inet 10.16.170.10/24 brd 255.255.255.0 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe97:92e8/64 scope link valid_lft forever preferred_lft forever 3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether 02:42:45:b7:c2:fd brd ff:ff:ff:ff:ff:ff inet 192.168.18.1/24 scope global docker0 valid_lft forever preferred_lft forever inet6 fe80::42:45ff:feb7:c2fd/64 scope link valid_lft forever preferred_lft forever 5: vethcff8026@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP link/ether 32:4a:f5:b7:33:f7 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::304a:f5ff:feb7:33f7/64 scope link valid_lft forever preferred_lft forever 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether 8a:ac:8e:a1:68:2b brd ff:ff:ff:ff:ff:ff 7: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether 82:ae:47:8e:30:4d brd ff:ff:ff:ff:ff:ff [root@master ~]# ovs-vsctl add-port br0 gre0 -- set interface gre0 type=gre option:remote_ip=172.16.17.2 [root@master ~]# ovs-vsctl show 4fe9a5b3-46ec-432c-a990-bb8e8fee96fe Bridge "br0" Port "gre0" Interface "gre0" type: gre options: {remote_ip="172.16.17.2"} Port "br0" Interface "br0" type: internal ovs_version: "2.3.2" [root@master ~]# brctl addif docker0 br0 [root@master ~]# brctl show bridge name bridge id STP enabled interfaces docker0 8000.024245b7c2fd no br0 [root@master ~]# ip link set dev br0 up [root@master ~]# ip link set dev docker0 up [root@master ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP qlen 1000 link/ether 00:0c:29:97:92:e8 brd ff:ff:ff:ff:ff:ff inet 10.16.170.10/24 brd 255.255.255.0 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe97:92e8/64 scope link valid_lft forever preferred_lft forever 3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether 02:42:45:b7:c2:fd brd ff:ff:ff:ff:ff:ff inet 192.168.18.1/24 scope global docker0 valid_lft forever preferred_lft forever inet6 fe80::42:45ff:feb7:c2fd/64 scope link valid_lft forever preferred_lft forever 5: vethcff8026@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP link/ether 32:4a:f5:b7:33:f7 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::304a:f5ff:feb7:33f7/64 scope link valid_lft forever preferred_lft forever 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether 8a:ac:8e:a1:68:2b brd ff:ff:ff:ff:ff:ff 7: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UNKNOWN qlen 1000 link/ether 82:ae:47:8e:30:4d brd ff:ff:ff:ff:ff:ff inet6 fe80::80ae:47ff:fe8e:304d/64 scope link valid_lft forever preferred_lft forever 8: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN qlen 1000 link/gre 0.0.0.0 brd 0.0.0.0 9: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 10: gre_sys@NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65490 qdisc fq master ovs-system state UNKNOWN qlen 1000 link/ether aa:3a:19:78:48:89 brd ff:ff:ff:ff:ff:ff inet6 fe80::a83a:19ff:fe78:4889/64 scope link valid_lft forever preferred_lft forever [root@master ~]# ip route add 192.168.42.0/24 dev docker0 [root@master ~]# docker run -itd --name c1 docker.io/centos ‘/bin/bash‘ WARNING: IPv4 forwarding is disabled. Networking will not work. a326fb2eae1ecf1c0b1a26b4b947f20eb44864fc5232e253b582c8c7bb50522a [root@master ~]# vim /etc/sysctl.conf [root@master ~]# sysctl -p net.core.default_qdisc = fq net.ipv4.tcp_congestion_control = bbr net.ipv4.ip_forward = 1 [root@master ~]# (3) docker node端配置如下 [root@node1 ~]# service openvswitch start && chkconfig openvswitch on [root@node1 ~]# brctl show bridge name bridge id STP enabled interfaces docker0 8000.02429f5f947d no [root@node1 ~]# ovs-vsctl add-br br0 [root@node1 ~]# ovs-vsctl add-port br0 gre0 -- set interface gre0 type=gre option:remote_ip=10.16.170.10 這裏如果寫錯ip 修改的話需要刪除網橋: ovs-vsctl del-br br0 [root@node1 ~]# brctl addif docker0 br0 [root@node1 ~]# brctl show bridge name bridge id STP enabled interfaces docker0 8000.02429f5f947d no br0 [root@node1 ~]# [root@node1 ~]# ip link set dev br0 up [root@node1 ~]# ip link set dev docker0 up [root@node1 ~]# ip route add 192.168.18.0/24 dev docker0 [root@node1 ~]# brctl show bridge name bridge id STP enabled interfaces docker0 8000.02429f5f947d no br0 [root@node1 ~]# ovs-vsctl show f0be12f7-1aa7-4b93-8d4f-5511b56efec7 Bridge "br0" Port "gre0" Interface "gre0" type: gre options: {remote_ip="10.16.170.10"} Port "br0" Interface "br0" type: internal ovs_version: "2.3.2" [root@node1 ~]# echo ‘net.ipv4.ip_forward = 1‘ >> /etc/sysctl.conf [root@node1 ~]# sysctl -p [root@node1 ~]# docker run -itd --name c2 docker.io/centos ‘/bin/bash‘ c9414017f86e6c362b9481ceffc658275b3557cf0991e84853066d4eccb37b0f [root@node1 ~]# (4) 測試 [root@node1 ~]# docker attach c941 [root@c9414017f86e /]# ping -c1 192.168.18.1 PING 192.168.18.1 (192.168.18.1) 56(84) bytes of data. 64 bytes from 192.168.18.1: icmp_seq=1 ttl=64 time=5.19 ms --- 192.168.18.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.194/5.194/5.194/0.000 ms [root@c9414017f86e /]# ping -c1 192.168.18.2 PING 192.168.18.2 (192.168.18.2) 56(84) bytes of data. 64 bytes from 192.168.1.2: icmp_seq=1 ttl=63 time=2.74 ms --- 192.168.18.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 2.742/2.742/2.742/0.000 ms [root@master ~]#docker exec -it 34fb bash 選擇一臺容器進入 [root@34fb4017f86 /]# ping -c1 192.168.42.1 PING 192.168.42.1 (192.168.42.1) 56(84) bytes of data. 64 bytes from 192.168.42.1: icmp_seq=1 ttl=64 time=0.051 ms --- 192.168.42.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.051/0.051/0.051/0.000 ms
ok 搞定
Docke 使用Open vSwitch實現跨主機通信