關鍵字：Strict Interface ARP, Multi-link, Multipath, Multiple network cards on same subnet problem
公司那個提供音樂下載的域名流量直逼1Gbps，但是系統的連線卻不高，伺服器用的是DELL2850的，板載兩個Gbe的網口，還插了兩塊Intel的Gbe網絡卡，因此打算通過多網絡卡的負載均衡擴充一下系統的網路頻寬。但是不想用多網絡卡的邦定，感覺那樣不太靈活，因此決定採用Advanced Routing來解決這個問題。 查了一下 man ip，然後開始做了： 1) 給四個網口配置了四個地址 2) 在BIGIP上面把四個地址都加入到負載均衡 3) 用ip命令實現多網絡卡負載均衡 ip route replace default equalize scope global nexthop via 172.24.x.11 dev eth0 weight 1 nexthop via 172.24.x.11 dev eth1 weight 1 nexthop via 172.24.x.1 dev eth2 weight 1 nexthop via 172.24.x.11 dev eth3 weight 1 立即看到所有的流量被均衡到4個網口 但是出現了新的問題，大量的IP地址衝突資訊出現在BIGIP的日誌： Nov 23 19:18:56 tc04 kernel: arp info overwritten for 172.24.x.30 by 00:14:22:1b:94:dc
Nov 23 20:30:24 tc04 kernel: arp info overwritten for 172.24.x.30 by 00:04:23:c0:5c:88
Nov 23 20:30:28 tc04 kernel: arp info overwritten for 172.24.x.30 by 00:14:22:1b:94:dc
Nov 23 21:03:14 tc04 kernel: arp info overwritten for 172.24.x.30 by 00:14:22:1b:94:dd 發現出現的幾個產生IP衝突的MAC地址都是這臺機器上面其他的網口的MAC，這就奇怪了。 通過tcpdump抓包，分析ARP資訊發現是因為Linux響應ARP請求的時候，使用的IP地址並未被嚴格的限定，因此導致了這個問題。 發現沒有什麼可用的資訊，然後就硬著頭皮進入到 /proc/sys/net/ipv4/conf/all 目錄看看有啥線索是解決2.6的 看到三個檔案 arp_announce, arp_ignore, arp_filter 立即去 /usr/src/linux/Documentation/networking/ip-sysctl.txt 找到相關的說明： arp_filter - BOOLEAN
        1 - Allows you to have multiple network interfaces on the same
        subnet, and have the ARPs for each interface be answered
        based on whether or not the kernel would route a packet from
        the ARP'd IP out that interface (therefore you must use source
        based routing for this to work). In other words it allows control
        of which cards (usually 1) will respond to an arp request.         0 - (default) The kernel can respond to arp requests with addresses
        from other interfaces. This may seem wrong but it usually makes
        sense, because it increases the chance of successful communication.
        IP addresses are owned by the complete host on Linux, not by
        particular interfaces. Only for more complex setups like load-
        balancing, does this behaviour cause problems.         arp_filter for the interface will be enabled if at least one of
        conf/{all,interface}/arp_filter is set to TRUE,
        it will be disabled otherwise arp_announce - INTEGER
        Define different restriction levels for announcing the local
        source IP address from IP packets in ARP requests sent on
        interface:
        0 - (default) Use any local address, configured on any interface
        1 - Try to avoid local addresses that are not in the target's
        subnet for this interface. This mode is useful when target
        hosts reachable via this interface require the source IP
        address in ARP requests to be part of their logical network
        configured on the receiving interface. When we generate the
        request we will check all our subnets that include the
        target IP and will preserve the source address if it is from
        such subnet. If there is no such subnet we select source
        address according to the rules for level 2.
        2 - Always use the best local address for this target.
        In this mode we ignore the source address in the IP packet
        and try to select local address that we prefer for talks with
        the target host. Such local address is selected by looking
        for primary IP addresses on all our subnets on the outgoing
        interface that include the target IP address. If no suitable
        local address is found we select the first local address
        we have on the outgoing interface or on all other interfaces,
        with the hope we will receive reply for our request and
        even sometimes no matter the source IP address we announce.         The max value from conf/{all,interface}/arp_announce is used.         Increasing the restriction level gives more chance for
        receiving answer from the resolved target while decreasing
        the level announces more valid sender's information. arp_ignore - INTEGER
        Define different modes for sending replies in response to
        received ARP requests that resolve local target IP addresses:
        0 - (default): reply for any local target IP address, configured
        on any interface
        1 - reply only if the target IP address is local address
        configured on the incoming interface
        2 - reply only if the target IP address is local address
        configured on the incoming interface and both with the
        sender's IP address are part from same subnet on this interface
        3 - do not reply for local addresses configured with scope host,
        only resolutions for global and link addresses are replied
        4-7 - reserved
        8 - do not reply for all local addresses         The max value from conf/{all,interface}/arp_ignore is used
        when ARP request is received on the {interface} 嘿嘿，通過： echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore or echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter 立即搞定！ 繼續 Google 了一些資訊，發現前面兩個核心控制好像是在 2.6.5 實現的 :-) 推薦用前面的兩個，arp_filter應該也能解決，感覺前面的兩個是對arp_filter的更細節控制的實現。 然後還發現在inter pro/1000的網絡卡驅動的安裝指南里面也有一節： Multiple Interfaces on Same Ethernet Broadcast Network
------------------------------------------------------ Due to the default ARP behavior on Linux, it is not possible to have 
one system on two IP networks in the same Ethernet broadcast domain 
(non-partitioned switch) behave as expected. All Ethernet interfaces 
will respond to IP traffic for any IP address assigned to the system.
This results in unbalanced receive traffic. If you have multiple interfaces in a server, either turn on ARP 
filtering by entering: echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter
(this only works if your kernel's version is higher than 2.4.5), or, install the interfaces in separate broadcast domains (either
in different switches or in a switch partitioned to VLANs). 參考資訊：