Linux常用效能工具功能、用法及原理(一)
Linux效能觀測工具按類別可分為系統級別和程序級別,系統級別對整個系統的效能做統計,而程序級別則具體到程序,為每個程序維護統計資訊。
按實現原理分,可分為基於計數器和跟蹤以及剖析。含義如下:
計數器:核心維護的統計資料,通常為無符號整型,用於對發生的事件計數,比如,網路包接收計數器,磁碟IO計數器等。
跟蹤:跟蹤會收集每一個事件的具體資料,由於跟蹤捕獲事件資料需要消耗CPU且需要較大的儲存空間儲存收集資料,預設不開啟。日誌就是一種低頻率的跟蹤,會記錄事件資料。
剖析:對目標取樣或快照來歸納目標特徵,如:CPU使用率,通過對程式計數器取樣(一種暫存器,用於指示下一條指令的地址),跟蹤棧找到消耗CPU週期的程式碼路徑。剖析也可以通過非計時的硬體事件,如CPU硬體快取未命中或匯流排活動,這類資訊可以幫助開發人員針對系統資源的使用來優化自己的程式碼。
一 系統級計數器工具
系統級別的工具有vmstat、mpstat、iostat、netstat、sar,這些工具有使用慣例,可選時間間隔和次數。程序級別的工具使用核心為每個程序維護的計數器,有ps、top、pmap。本節主要講基於計數器的系統級效能工具。
1.1 vmstat
vmstat是Virtual Meomory Statistics(虛擬記憶體統計),用來報告程序、記憶體、磁碟讀寫、CPU使用整體狀況。常用形式vmstat delay count,其中delay表示取樣間隔,count表示取樣次數,命令執行結果如下圖1.1所示,其中第一行表示系統啟動以來各指標的平均值。
圖1.1 vmstat命令
各個區域的含義如下:
- Procs
- r: The number of processes waiting for run time.執行態和就緒態程序數目
- b: The number of processes in uninterruptible sleep.不可中斷程序數目,程序進行系統呼叫且被阻塞,不可中斷和kill掉時的狀態,絕大多數不可中斷系統呼叫都比較快完成 [1] ,比如mkdir(2)呼叫過程不會返回EINTR(呼叫過程被中斷返回),不會被中斷 [2] 。
- Memory
- swpd: the amount of virtual memory used.使用虛擬記憶體的大小
- free: the amount of idle memory.空閒記憶體的大小(實體記憶體)
- buff: the amount of memory used as buffers。Buff用於快取磁碟塊資料,如檔案系統元資料,檔案許可權、位置等(metadata)
- cache: the amount of memory used as cache。Cache用於快取檔案內容 [3]
- inact: the amount of inactive memory. (-a option)
- active: the amount of active memory. (-a option)
- Swap, 交換區,當記憶體不夠時,記憶體中的頁置換到磁碟中 [4]
- si: Amount of memory swapped in from disk (/s).從磁碟交換到記憶體的大小
- so: Amount of memory swapped to disk (/s).從記憶體交換到磁碟的大小
- IO
- bi: Blocks received from a block device (blocks/s).每秒從塊裝置讀出的塊數(如從磁碟讀檔案會導致增加)
- bo: Blocks sent to a block device (blocks/s).每秒寫入塊裝置的塊數(如寫資料到磁碟會導致增加)
- System
- in: The number of interrupts per second, including the clock.每秒中斷次數(包括時鐘中斷)
- cs: The number of context switches per second.每秒上下文切換次數CPU
- us: Time spent running non-kernel code. (user time, including nice time),使用者態程式碼時間,執行計算密集型程序(如壓縮檔案),會導致使用者態CPU增加 [3] 。
- sy: Time spent running kernel code. (system time),核心態程式碼時間(如頻繁執行系統呼叫/dev/urandom生成隨機數,會導致sy增加)
- id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.空閒時間,包含了wa時間(cpu空閒的時候時間上會執行空閒執行緒)。
- wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.CPU空閒,但有由此CPU發起的IO在進行 [4]
- st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.Steal time is the percentage of time a virtual CPU waits for a real CPU while the hypervisor is servicing another virtual processor.在雲環境下,物理CPU由多臺虛擬機器共享,st表示本虛擬CPU等待分配真實CPU的時間,st過高表示可能有其他使用者的虛擬機器搶佔CPU [5] 。
vmstat中的統計值由核心互動檔案/proc/meminfo(記憶體使用資訊)、/proc/stat(系統整體程序狀態,如執行態、不可中斷阻塞態程序數,cpu使用情況)、/proc/*/stat(每個程序各自的狀態資訊)統計出。
/proc/meminfo檔案內容如下,stat檔案內容見1.2。
圖1.2 meminfo檔案內容
1.2 mpstat
mpstat是Multiprocessor Statistics的縮寫,可以檢視每個CPU核心的資訊,而vmstat只能檢視系統整體的CPU資訊。
用法: mpstat [-P {|ALL}] [internal [count]],-P表示CPU列表,interval表示間隔,count表示取樣次數。執行結果如下圖1.3所示,各欄位含義如下:
圖1.3 mpstat結果圖
- CPU:Processor number. The keyword all indicates that statistics are calculated as averages among all processors.具體的CPU,all值表示所有CPU的平均值統計資訊。
- %usr:Show the percentage of CPU utilization that occurred while executing at the user level (application).
- %nice:Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.nice值發生變動(即優先順序發生變化)的程序的CPU時間,nice值為負表示程序的優先順序變高。
- %sys:Show the percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this does not include time spent servicing hardware and software interrupts.核心程式碼執行時間,不包括中斷服務時間。
- %iowait:Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.含義見1.1.1節vmstat的wa欄位。
- %irq:Show the percentage of time spent by the CPU or CPUs to service hardware interrupts.CPU花在硬中斷服務時間
- %soft:Show the percentage of time spent by the CPU or CPUs to service software interrupts. CPU花在軟中斷服務時間
- %steal:Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.含義見1.1.1節vmstat的st欄位
- %guest:Show the percentage of time spent by the CPU or CPUs to run a virtual processor.CPU執行虛擬處理器所花費的時間
- %gnice:Show the percentage of time spent by the CPU or CPUs to run a niced guest.
- %idle:Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.CPU空閒時間,和vmstat有點不同,這裡不包含iowait時間,但vmstat包含。
vmstat資料來源/proc/stat,此檔案包含所有cpu的活動,是開機累積到現在的統計值。檔案內容如下圖1.4所示,cpu表示總的資訊,cpu0、cpu1…表示各cpu的資訊,從前到後的值含義分別為user、nice、system、idle、iowait、irq、softirq、stealstolen、guest,以時間片為單位,intr後的第一個值表示系統自啟動到現在的中斷次數,後面的每一個值表示某種中斷型別發生的次數,ctxt表示cpu上下文切換的次數,btime表示系統啟動到現在的時間(實際觀察前後兩次未發生變化), processes表示系統啟動到現在以來建立的任務個數, procs_running表示當前執行佇列任務個數, procs_blocked表示阻塞任務個數,softirq表示總軟中斷次數及各種型別軟中斷的次數 [6] 。
圖1.4 /proc/stat檔案內容
1.3 iostat
iostat是I/O statistics(輸入/輸出統計)的縮寫,用來動態監視系統的磁碟操作活動。
用法: iostat [ -p [ device [,...] | ALL ] ] [ device [...] | ALL ] [ interval [ count ] ], -p表示裝置列表,interval表示間隔,count表示取樣次數。執行結果如下圖1.5和1.6所示,其中選項-d表示顯示裝置統計(Display the device utilization report),-x表示顯示詳細資訊(Display extended statistics)。各欄位含義如下:
圖1.5 iostat裝置io統計資訊
圖1.6 iostat裝置io詳細統計資訊
- Device:This column gives the device (or partition) name as listed in the /dev directory.裝置名稱
- Tps:Indicate the number of transfers per second that were issued to the device. A transfer is an I/O request to the device. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size.每秒I/O次數(即每秒裝置請求次數)
- Blk_read/s (kB_read/s, MB_read/s),Indicate the amount of data read from the device expressed in a number of blocks (kilobytes, megabytes) per second. Blocks are equivalent to sectors and therefore have a size of 512 bytes.每秒讀資料多少
- Blk_wrtn/s (kB_wrtn/s, MB_wrtn/s),Indicate the amount of data written to the device expressed in a number of blocks (kilobytes, megabytes) per second.每秒寫資料多少
- Blk_read (kB_read, MB_read),The total number of blocks (kilobytes, megabytes) read.interval間隔內讀資料總量
- Blk_wrtn (kB_wrtn, MB_wrtn),The total number of blocks (kilobytes, megabytes) written. interval間隔內寫資料總量
- rrqm/s,The number of read requests merged per second that were queued to the device.每秒合併讀操作的數目,當兩個讀操作讀相鄰的資料塊時,會被合併為一個請求,提高效率, 合併的操作通常是I/O scheduler(也叫elevator)負責的。(順序讀和隨機讀對比,順序讀此值會比較大) [7]
- wrqm/s,The number of write requests merged per second that were queued to the device. .每秒合併寫操作的數目,當兩個寫操作寫相鄰的資料塊時,會被合併為一個請求,提高效率(順序寫和隨機寫對比,順序寫此值會比較大) [7]
- r/s,The number (after merges) of read requests completed per second for the device.每秒讀操作次數
- w/s,The number (after merges) of write requests completed per second for the device.每秒寫操作次數
- rsec/s (rkB/s, rMB/s),The number of sectors (kilobytes, megabytes) read from the device per second.每秒讀資料量大小,可以以sector和kB,MB為單位
- wsec/s (wkB/s, wMB/s),The number of sectors (kilobytes, megabytes) written to the device per second. 每秒寫資料量大小,可以以sector和kB,MB為單位
- avgrq-sz,The average size (in sectors) of the requests that were issued to the device. 每個IO的平均扇區數,即所有請求的平均大小,以扇區(512位元組)為單位
- avgqu-sz,The average queue length of the requests that were issued to the device.平均請求佇列長度
- await,The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.平均每個io的時間(包括了在等列中等待的時間和磁碟處理的時間) [8] 。如果I/O模式很隨機(磁碟服務時間)、I/O負載(佇列中等待時間)比較高,會導致磁頭亂跑,尋道時間長,那麼相應地await要估算得大一些;如果I/O模式是順序讀寫,只有單一程序產生I/O負載,那麼尋道時間和旋轉延遲都可以忽略不計,主要考慮傳輸時間,相應地await就應該很小,甚至不到1毫秒, 對磁碟陣列來說,因為有硬體快取,寫操作不等落盤就算完成,所以寫操作的service time大大加快了,如果磁碟陣列的寫操作不在一兩個毫秒以內就算慢的了;讀操作則未必,不在快取中的資料仍然需要讀取物理硬碟,單個小資料塊的讀取速度跟單盤差不多 [9] 。
- r_await,The average time (in milliseconds) for read requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them. 平均每個讀io的時間(包括了在等列中等待的時間和磁碟處理的時間)
- w_await,The average time (in milliseconds) for write requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them. 平均每個寫io的時間(包括了在等列中等待的時間和磁碟處理的時間)
- svctm,The average service time (in milliseconds) for I/O requests that were issued to the device. Warning! Do not trust this field any more. This field will be removed in a future sysstat version.(已被廢棄)
- %util,Percentage of elapsed time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.工作時間或者繁忙時間佔總時間的百分比,表示該裝置有I/O(即非空閒)的時間比率,不考慮I/O有多少,只考慮有沒有.現代硬碟都有並行處理的功能,當%util為100%時並不能說明磁碟處理已飽和,達到瓶頸,因為可能可以接收更多的並行處理請求,iostat沒有衡量磁碟是否飽和的指標 [9] 。
iostat的統計資料主要來源與檔案/proc/diskstats,檔案內容如下圖1.7所示:
圖1.7 diskstats檔案內容
每行各列的含義依序如下表1.1所示,各值為系統啟動到現在為止的累加值:
表1.1 diskstats含義表
Field |
Value |
Quoted |
解釋 |
F1 |
253 |
major number |
主裝置號 |
F2 |
0 |
minor mumber |
從裝置號 |
F3 |
vda |
device name |
裝置名 |
F4(rd_ios) |
1012759 |
reads completed |
讀完成次數 |
F5(rd_merges) |
3418 |
reads merged |
讀合併次數 |
F6(rd_sectors) |
48466695 |
sectors read |
讀取的扇區數 |
F7(rd_ticks) |
4027016 |
milliseconds spent reading |
所有讀操作時間的累加值(ms單位) |
F8(wr_ios) |
19073830 |
writes completed |
寫操作次數 |
F9(writes merged) |
33764041 |
writes merged |
寫合併次數 |
F10(wr_sectors) |
2195716912 |
sectors written |
寫入的扇區數 |
F11(wr_ticks) |
120769824 |
milliseconds spent writing |
所有寫操作消耗的時間累加(ms單位) |
F12(in_flight) |
0 |
I/Os currently in progress |
未完成io的數目 |
F13(io_ticks) |
10982660 |
milliseconds spent doing I/Os |
該裝置用於處理I/O的自然時間(wall-clock time) |
F14(time_in_queue) |
124778072 |
weighted # of milliseconds spent doing I/Os |
對欄位F13(io_ticks)的加權值 |
in_flight表示系統未完成io的任務數,當io進入請求佇列時加1(I/O請求進入佇列時,而不是提交給硬碟裝置時),當io任務完成時減1, in_flight包括在佇列中等待和正在進行io的任務。
rd_ticks和wr_ticks是把每一個IO消耗時間累加起來,但是硬碟裝置一般可以並行處理多個IO,因此,rd_ticks和wr_ticks之和一般會比自然時間(wall-clock time)要大。而io_ticks 不關心佇列中有多少個IO在排隊,它只關心裝置有IO的時間。即不考慮IO有多少,只考慮IO有沒有。在實際運算中,in_flight不是0的時候保持計時,而in_flight 等於0的時候,時間不累加到io_ticks。
其中,io_ticks這個欄位被iostat用來計算%util,而time_in_queue這個欄位被iostat用來計算avgqu-sz,即平均佇列長度。 [9]
1.4 netstat
netstat主要用來檢視網路連線、路由表、網絡卡統計資訊。
1.4.1 檢視網路連線 (等價 ss)
用法:netstat [address_family_options] [--tcp|-t] [--udp|-u] [--udplite|-U] [--sctp|-S] [--raw|-w] [--listening|-l] [--all|-a] [--numeric|-n] [--numeric-hosts] [--numeric-ports] [--numeric-users] [--symbolic|-N] [--extend|-e[--extend|-e]] [--timers|-o] [--program|-p] [--verbose|-v] [--continuous|-c] [--wide|-W] [delay]
其中-t,-u等表示網路活動的協議,-l表示socket處於listening狀態,-a表示顯示所有狀態(包括listening狀態和非listening狀態),-n表示主機、埠、使用者用數字表示,不用解析成名稱,-e表示詳細資訊,-p顯示socket的程序pid和名稱,-c會每秒連續輸出網路資訊。執行結果如下圖1.8所示:
圖1.8 netstat檢視網路連線
部分欄位含義如下:
- Recv-Q.
- Established: The count of bytes not copied by the user program connected to this socket. 指收到的資料還在快取中,還沒被程序讀取,這個值就是還沒被程序讀取的 byte數。
- Listening: Since Kernel 2.6.18 this column contains the current syn backlog. syn backlog表示處於半連線狀態的佇列,接收到客戶端的syn請求,則會進入此佇列。
- Send-Q.
- Established: The count of bytes not acknowledged by the remote host. 傳送佇列中沒有被遠端主機確認的 bytes 數。
- Listening: Since Kernel 2.6.18 this column contains the maximum size of the syn backlog.半連線佇列的最大大小
- State.The state of the socket. Since there are no states in raw mode and usually no states used in UDP and UDPLite, this column may be left blank. Normally this can be one of several values:
- ESTABLISHED.The socket has an established connection.三次握手,第三次握手傳送ack後的狀態
- SYN_SENT.The socket is actively attempting to establish a connection.傳送syn後的狀態
- SYN_RECV.A connection request has been received from the network.接收到syn+ack後的狀態
- FIN_WAIT1.The socket is closed, and the connection is shutting down.socket被關閉,正在關閉連線(四次揮手,第一次揮手,主動傳送FIN後的狀態)
- FIN_WAIT2.Connection is closed, and the socket is waiting for a shutdown from the remote end.連線關閉,等待遠端關閉訊號(四次揮手,第二次揮手,接收到FIN的響應ACK後的狀態)
- TIME_WAIT.The socket is waiting after close to handle packets still in the network. FIN_WAIT2狀態接收到遠端的FIN包,變為TIME_WAIT狀態(四次揮手,收到第三次揮手FIN訊號)
- CLOSE .The socket is not being used.
- CLOSE_WAIT.The remote end has shut down, waiting for the socket to close.對端已關閉,但本端向遠端仍可以傳送資料。(四次揮手,第二次揮手,向對端傳送FIN訊號的響應)
- LAST_ACK.The remote end has shut down, and the socket is closed. Waiting for acknowledgement. CLOSE_WAIT狀態下,向遠端傳送FIN訊號(四次揮手,第三次揮手,向對端傳送FIN)
- LISTEN.The socket is listening for incoming connections. Such sockets are not included in the output unless you specify the --listening (-l) or --all (-a) option.
- CLOSING.Both sockets are shut down but we still don't have all our data sent.
- UNKNOWN.The state of the socket is unknown.
Recv-Q和Send-Q一般情況下為0值,可接收短暫情況情況下為非0情況,假如長時間為非0,表明佇列有堆積,假如時Recv-Q為非0,應用程式接收不急,可能是拒絕服務攻擊,假如是Send-Q為非0,可能是應用程式傳送程序和服務端接收程序速度不匹配,傳送太快或接收太慢 [13] 。
netstat的資訊來源於/proc/net資料夾,關於tcp連線的資訊來源於/proc/net/tcp檔案,內容如下圖1.9所示:
圖1.9 /proc/net/tcp檔案
1.4.2 檢視路由表資訊 (等價 ip route)
用法:netstat {--route|-r} [address_family_options] [--extend|-e[--extend|-e]] [--verbose|-v] [--numeric|-n] [--numeric-hosts] [--numeric-ports] [--numeric-users] [--continuous|-c] [delay]
執行結果如下圖1.10所示,路由表資訊來源於/proc/net/route檔案:
圖1.10 核心路由表
1.4.3 檢視網絡卡統計資訊 (等價 ip -s link)
用法:netstat {--interfaces|-I|-i} [--all|-a] [--extend|-e] [--verbose|-v] [--program|-p] [--numeric|-n] [--numeric-hosts] [--numeric-ports] [--numeric-users] [--continuous|-c] [delay]
執行結果如下圖1.11:
圖1.11 網絡卡活動統計
1.5 sar
sar(System Activity Reporter)用於統計和收集系統活動,包括CPU、記憶體、網路、IO等多方面,是一個比較全面的工具。sar收集系統資料儲存在/var/log/sa/sadd(dd為日期)檔案。
1.5.1 檢視 CPU使用情況
懷疑CPU存在瓶頸,可用 sar -u和 sar -q 等來檢視。sar -u和mpstat相似,欄位含義參考1.1.2。sar -p用於檢視佇列長度和負載資訊,執行結果如下圖1.12:
圖1.12 sar -q執行結果
- runq-sz.Run queue length (number of tasks waiting for run time).執行態和就緒態任務數
- plist-sz.Number of tasks in the task list.處於任務佇列的任務總數
- ldavg-1.System load average for the last minute. The load aver‐age is calculated as the average number of runnable or running tasks (R state), and the number of tasks in unin‐terruptible sleep (D state) over the specified interval.過去一分鐘負載(處於執行態和就緒態及不可中斷狀態任務數)
- ldavg-5.System load average for the past 5 minutes.過去五分鐘負載
- ldavg-15.System load average for the past 15 minutes.過去十五分鐘負載
- Blocked.Number of tasks currently blocked, waiting for I/O to complete.等待io完成的處於阻塞態任務數。
1.5.2 檢視記憶體使用情況
懷疑記憶體存在瓶頸,可用sar -B、sar -r 和 sar -W 等來檢視,Sar -B,統計頁交換情況。執行結果如圖1.13:
圖1.13 sar -B執行結果
- pgpgin/s,Total number of kilobytes the system paged in from disk per second.每秒從磁碟讀入頁的大小。
- pgpgout/s,Total number of kilobytes the system paged out to disk per second.每秒寫入磁碟的頁大小。
- fault/s,Number of page faults (major + minor) made by the system per second. This is not a count of page faults that generate I/O, because some page faults can be resolved without I/O.包括major和minor,major缺頁需要進行磁碟io,但minor不需要,minor falut頁在記憶體中,只是還未指定到程序地址空間,比如多程序程式碼共享,此時不需要重新從磁碟讀入程式碼。
- majflt/s,Number of major faults the system has made per second,those which have required loading a memory page from disk. 每秒產生的要進行磁碟IO的缺頁次數 [16]
- pgfree/s,Number of pages placed on the free list by the system per second. 每秒放置在可用列表中的頁數。
- pgscank/s,Number of pages scanned by the kswapd daemon per second. 表示的是Linux當中的負責記憶體回收的執行緒kswapd掃描的頁的數量,
- pgscand/s,Number of pages scanned directly per second. 每秒直接掃描的頁數
- pgsteal/s,Number of pages the system has reclaimed from cache (pagecache and swapcache) per second to satisfy its memory demands. 系統滿足自身的記憶體需要,每秒從快取回收的頁數.
- %vmeff,Calculated as pgsteal / pgscan, this is a metric of the efficiency of page reclaim. If it is near 100% then almost every page coming off the tail of the inactive list is being reaped. If it gets too low (e.g. less than 30%) then the virtual memory is having some difficulty. This field is displayed as zero if no pages have been scanned during the interval of time. pgscan= pgscank/s+ pgscand/s,衡量分頁回收有效性的指標,理想情況應該是100%或0(未發生頁面掃描時), 表示被掃描的頁當中, 有多少頁的資料被踢出去了換成其他頁資料了 [17]
page fault和swap不同,當要訪問的程式碼或資料沒有虛擬地址和實體地址對應,會產生page fault;當記憶體空間不足,需要釋放部分記憶體空間載入其它資料時,會將記憶體中的部分頁swap到磁碟。 [18]
sar -r統計記憶體使用情況。執行結果如下圖1.14所示:
圖1.14 sar -r執行結果
- kbcommit.Amount of memory in kilobytes needed for current workload. This is an estimate of how much RAM/swap is needed to guarantee that there never is out of memory.已經申請的記憶體的大小(可能未分配),對應/proc/meminfo中的Committed_AS
- %commit.Percentage of memory needed for current workload in relation to the total amount of memory (RAM+swap). This number may be greater than 100% because the kernel usually overcommits memory. Kbcommit和RAM+swap的百分比,可能大於100%,核心允許申請的記憶體大小大於總記憶體大小。 [19]
- kbactive.Amount of active memory in kilobytes (memory that has been used more recently and usually not reclaimed unless absolutely necessary).活躍記憶體大小
- kbinact.Amount of inactive memory in kilobytes (memory which has been less recently used. It is more eligible to be reclaimed for other purposes).不活躍記憶體大小
- kbdirty.Amount of memory in kilobytes waiting to get written back to the disk. 等待寫回硬碟的記憶體數量
sar -W統計swap in和out資訊,執行結果如下圖1.15所示:
圖1.15 sar -W執行結果
- pswpin/s,Total number of swap pages the system brought in per second.每秒swap in數目
- pswpout/s,Total number of swap pages the system brought out per second.每秒swap out數目
1.5.3 檢視裝置 IO情況
懷疑I/O存在瓶頸,可用 sar -b和 sar -d 等來檢視,sar -b統計IO和傳輸速率資訊,執行結果如下圖1.16所示:
圖1.16 sar -b執行結果
- tps.Total number of transfers per second that were issued to physical devices. A transfer is an I/O request to a physical device. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size.每秒IO請求數
- rtps.Total number of read requests per second issued to physical devices.都請求數
- wtps.Total number of write requests per second issued to physical devices.寫請求數
- bread/s.Total amount of data read from the devices in blocks per second. Blocks are equivalent to sectors with 2.4 kernels and newer and therefore have a size of 512 bytes. With older kernels, a block is of indeterminate size.每秒讀取資料塊個數
- bwrtn/s.Total amount of data written to devices in blocks per second.每秒寫入磁碟的資料塊數.
sar -d統計磁碟塊活動,執行結果如下圖1.17所示,和iostat欄位含義類似。
圖1.17 sar -d執行結果
1.5.4 檢視網路流量情況
sar –n用於統計網路接收包資訊,用法如下sar -n { keyword [,...] | ALL },關鍵有DEV, EDEV, FC, ICMP, EICMP, ICMP6, EICMP6, IP, EIP, IP6, EIP6, NFS, NFSD, SOCK, SOCK6, SOFT, TCP, ETCP, UDP and UDP6,分別從不同的角度展示網路統計資訊,這裡主要說明下DEV關鍵字,sar –n DEV執行結果如下圖1.18所示:
圖1.18 DEV結果
- IFACE.Name of the network interface for which statistics are reported.網路介面名
- rxpck/s.Total number of packets received per second.每秒收包數
- txpck/s.Total number of packets transmitted per second.每秒發包數
- rxkB/s.Total number of kilobytes received per second.每秒收到的kB數
- txkB/s.Total number of kilobytes transmitted per second.每秒傳送的kB數
- rxcmp/s.Number of compressed packets received per second (for cslip etc.).每秒收到的壓縮資料包
- txcmp/s. Number of compressed packets transmitted per second.每秒傳送的壓縮資料包
- rxmcst/s. Number of multicast packets received per second.每秒收到的廣播資料包數
sar效能資料來源於/var/log/sa/saDD、/var/log/sa/saYYYYMMDD、以及/proc 和 /sys中的檔案。
參考文獻
[1]. Uninterruptible Sleep. https://eklitzke.org/uninterruptible-sleep
[2]. Checking if errno != EINTR: what does it mean? https://stackoverflow.com/questions/41474299/checking-if-errno-eintr-what-does-it-mean
[3]. Linux Performance Measurements using vmstat。 https://www.thomas-krenn.com/en/wiki/Linux_Performance_Measurements_using_vmstat
[4]. Why CPU spent time on IO(wa)? https://serverfault.com/questions/684339/why-cpu-spent-time-on-iowa
[5]. Understanding CPU Steal Time - when should you be worried? http://blog.scoutapp.com/articles/2013/07/25/understanding-cpu-steal-time-when-should-you-be-worried
[6]. mpstat命令和/proc/stat檔案. https://yq.aliyun.com/articles/53583/
[7]. Interpreting iostat Output. https://blog.serverfault.com/2010/07/06/777852755/
[8]. 容易被誤讀的IOSTAT. http://linuxperf.com/?p=156
[9]. 深入理解iostat. http://bean-li.github.io/dive-into-iostat/
[10]. 深入分析diskstats. http://ykrocku.github.io/blog/2014/04/11/diskstats/
[11]. I/O statistics fields. https://www.kernel.org/doc/Documentation/iostats.txt
[12]. 深入理解Linux TCP backlog. https://www.jianshu.com/p/7fde92785056
[13]. netstat Recv-Q和Send-Q. https://blog.csdn.net/sjin_1314/article/details/9853163
[14]. netstat用法及TCP state解析 . https://www.cnblogs.com/vigarbuaa/archive/2012/03/07/2383064.html
[15]. /proc/net/tcp中各項引數說明. https://blog.csdn.net/justlinux2010/article/details/21028797
[16]. What does the fields in sar -B output mean? https://serverfault.com/questions/270283/what-does-the-fields-in-sar-b-output-mean
[17]. 如何控制Linux清理cache機制. https://www.zhihu.com/question/59053036/answer/171176545
[18]. Understanding page faults and memory swap-in/outs: when should you worry? http://blog.scoutapp.com/articles/2015/04/10/understanding-page-faults-and-memory-swap-in-outs-when-should-you-worry
[19].理解LINUX的MEMORY OVERCOMMIT. http://linuxperf.com/?p=102