我們知道TCP socket有傳送緩衝區和接收緩衝區，這二個緩衝區都可以透過setsockopt設定SO_SNDBUF，SO_RCVBUF來修改，但是這些值設多大呢？這些值和協議棧的記憶體控制相關的值什麼關係呢？
我們來解釋下：

$ sysctl net|grep mem
net.core.wmem_max = 131071
net.core.rmem_max = 131071
net.core.wmem_default = 124928
net.core.rmem_default = 124928
net.core.optmem_max = 20480
net.ipv4.igmp_max_memberships = 20
net.ipv4.tcp_mem = 4631520 6175360 9263040
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.udp_mem = 4631520 6175360 9263040
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096

下面的圖很好的解釋了上面的問題：

這裡要記住的是：TCP協議棧記憶體是不可交換實體記憶體，用一位元組少一位元組。
也正是由於這一點，作業系統出廠的時候上面的預設的記憶體設定都不算太大。對於一個不是網路密集型的伺服器問題不大，但是對於如承擔C1M連結的伺服器來講，問題就來了。我們在實踐中會發現tcp服務經常超時，有時候超過100ms. 那麼這個問題如何定位呢？

我們知道當協議棧缺少記憶體的時候會呼叫sk_stream_wait_memory等待其他程序釋放出記憶體，所以這個函式的等待時間就是我們的程序被阻塞的時間。
下面我們來驗證下：

$ cat /usr/share/doc/systemtap-1.6/examples/network/sk_stream_wait_memory.stp
#!/usr/bin/stap
# Simple probe to detect when a process is waiting for more socket send
# buffer memory. Usually means the process is doing writes larger than the
# socket send buffer size or there is a slow receiver at the other side.
# Increasing the socket's send buffer size might help decrease application
# latencies, but it might also make it worse, so buyer beware.
#
# Typical output: timestamp in microseconds: procname(pid) event
#
# 1218230114875167: python(17631) blocked on full send buffer
# 1218230114876196: python(17631) recovered from full send buffer
# 1218230114876271: python(17631) blocked on full send buffer
# 1218230114876479: python(17631) recovered from full send buffer

probe kernel.function("sk_stream_wait_memory")
{
        printf("%u: %s(%d) blocked on full send buffer\n",
                gettimeofday_us(), execname(), pid())
}

probe kernel.function("sk_stream_wait_memory").return
{
        printf("%u: %s(%d) recovered from full send buffer\n",
                gettimeofday_us(), execname(), pid())
}
$ sudo stap sk_stream_wait_memory.stp  
1218230114875167: python(17631) blocked on full send buffer
1218230114876196: python(17631) recovered from full send buffer
1218230114876271: python(17631) blocked on full send buffer
1218230114876479: python(17631) recovered from full send buffer

如果我們觀察到了程序由於缺少記憶體被阻塞，那麼是時候調整協議棧的記憶體限制了。

小結：網路很複雜，需要定量分析！

祝玩得開心。

Post Footer automatically generated by wp-posturl plugin for wordpress.