內核調試神器SystemTap — 簡單介紹與使用（一）
a linux trace/probe tool.
官網：https://sourceware.org/systemtap/
簡單介紹
SystemTap是我眼下所知的最強大的內核調試工具，有些家夥甚至說它無所不能：）
(1) 發展歷程
Debuted in 2005 in Red Hat Enterprise Linux 4 Update 2 as a technology preview.
After four years in development, System 1.0 was released in 2009.
As of 2011 SystemTap runs fully supported in all Linux distributions.
(2) 官方介紹
SystemTap provides free software(GPL) infrastructure to simplify the gathering of information about the
running Linux system. This assists diagnosis of a performance or functional problem. SystemTap eliminates
the need for the developer to go through the tedious and disruptive instrument, recompile, install, and reboot
sequence that may be otherwise required to collect data.
SystemTap provides a simple command line interface and scripting language for writing instrumentation for
a live running kernel plus user-space application. We are publishing samples, as well as enlarging the internal
"tapset" script library to aid reuse and abstraction.
Among other tracing/probing tools, SystemTap is the tool of choice for complex tasks that may require live analysis,
programmable on-line response, and whole-system symbolic access. SystemTap can also handle simple tracing
jobs.
Current project members include Red Hat, IBM, Hitachi, and Oracle.
(3) 獲取源代碼
git clone git://sourceware.org/git/systemtap.git
安裝
(1) Ubuntu發行版
1. 安裝systemtap包
apt-get install systemtap
2. 安裝依賴包
gcc：C語言編譯器
elfutils：提供分析調試信息的庫函數
linux-headers-generic：編譯內核模塊所需的內核頭文件以及模塊配置信息
3. 安裝內核調試信息(kernel-debuginfo)
kernel-debuginfo提供了調試內核所需的符號表，假設沒有安裝的話SystemTap的威力就會大打折扣，
僅僅能提供kprobes系列的功能。
下載地址：http://ddebs.ubuntu.com/pool/main/l/linux/
下載相應的內核版本號，我的是linux-image-3.11.0-12-generic-dbgsym_3.11.0-12.19_amd64.ddeb
下載後安裝：dpkg -i linux-image-3.11.0-12-generic-dbgsym_3.11.0-12.19_amd64.ddeb
4. 驗證
stap -ve ‘probe kernel.function("do_fork") { print("hello world\n") exit() }‘
假設沒有提示錯誤，就是成功安裝了。
(2) CentOS/RedHat發行版
使用yum安裝下列rpm包就可以：
systemtap：SystemTap包
gcc：C語言編譯器
elfutils：提供庫函數來分析調試信息
kernel-devel：編譯內核模塊所需的內核頭文件及模塊配置信息
kernel-debuginfo：提供所需的內核調試信息來定位內核函數和變量的位置
使用
一些樣例SystemTap的簡單樣例。
(1) stap
通常直接使用stap運行用SystemTap語法編寫的腳本就可以。
stap - systemtap script translator/driver
stap test.stp // .stp後綴的文件是用SystemTap語法編寫的腳本
腳本主要元素：probe point + probe handler
stap [options] FILE // Run script in file
stap [options] -e SCRIPT // Run given script.
stap [options] -l PROBE // List matching probes.
stap [options] -L PROBE // List matching probes and local variables.
經常使用選項
-h：幫助
-g：guru模式，嵌入式C代碼須要
-m：指定編譯成的模塊名稱
-v：add verbosity to all passes
-k：不刪除暫時文件夾
-p NUM：stop after pass NUM 1-5, instead of 5 (parse, elaborate, translate, compile, run)
-b：bulk (percpu file) mode, 使用RelayFS將數據從內核空間傳輸到用戶空間
-o FILE：輸出到指定文件，而不是stdout
-c CMD：start the probes, run CMD, and exit when it finishes
stap是SystemTap的前端。當出現下面情況時退出：
1. The user interrupts the script with a CTRL-C.
2. The script executes the exit() function.
3. The script encounters a sufficient number of soft errors.
4. The monitored command started with the stap program‘s -c option exits.
(2) staprun
假設我們的輸入不是.stp腳本，而是一個用stap生成的模塊。那麽就用staprun來運行。
staprun - systemtap runtime
staprun [OPTIONS] MODULE [MODULE-OPTIONS]
staprun的作用：
The staprun program is the back-end of the Systemtap tool. It expects a kernel module produced by
the front-end stap tool.
Splitting the systemtap tool into a front-end and a back-end allows a user to compile a systemtap script
on a development machine that has the kernel debugging information (need to compile the script) and
then transfer the resulting kernel module to a production machine that doesn‘t have any development
tools or kernel debugging information installed.
staprun is a part of the SystemTap package, dedicated to module loading and unloading and kernel-to-user
data transfer.
經常使用選項
-o FILE：Send output to FILE.
-D：Run in background. This requires ‘-o‘ option.
(3) 監測內核函數
一個簡單腳本，每當內核函數do_fork()被調用時。顯示調用它的進程名、進程ID、函數參數。
global proc_counter
probe begin {
print("Started monitoring creation of new processes...Press ^C to terminate\n")
printf("%-25s %-10s %-s\n", "Process Name", "Process ID", "Clone Flags")
}
probe kernel.function("do_fork") {
proc_counter++
printf("%-25s %-10d 0x%-x\n", execname(), pid(), $clone_flags)
}
probe end {
printf("\n%d processes forked during the observed period\n", proc_counter)
}
(4) 監測系統調用
一個簡單腳本。顯示4秒內open系統調用的信息：調用進程名、進程ID、函數參數。
probe syscall.open
{
printf("%s(%d) open(%s)\n", execname(), pid(), argstr)
}
probe timer.ms(4000) # after 4 seconds
{
exit()
}
(5) 監測源文件裏全部函數入口和出口
括號內的探測點描寫敘述包括三個部分：
function name part：函數名
@file name part：文件名稱
function line part：所在行號
比如：
probe kernel.function("[email protected]/socket.c") {}
probe kernel.function("[email protected]/socket.c").return {}
這裏指定函數名為隨意（用*表示），指定文件名稱為net/socket.c。探測函數的入口和返回。
還能夠用“：行號”來指定行號。
(6) 查找匹配的內核函數和變量
查找名字中包括nit的內核函數：
stap -l ‘kernel.function("*nit*")‘
查找名字中包括nit的內核函數和變量：
stap -L ‘kernel.function("*nit*")‘
(7) 自帶的用例集
/root/systemtap/testsuite/systemtap.examples/。包括了很多用例腳本。
主要有幾個方面：
network、io、interrupt、locks、memory、process、virtualization等
(8) 監控全部進程的收發包情況
global recv, xmit
probe begin {
printf("Starting network capture...Press ^C to terminate\n")
}
probe netdev.receive {
recv[dev_name, pid(), execname()] <<< length
}
probe netdev.transmit {
xmit[dev_name, pid(), execname()] <<< length
}
probe end {
printf("\nCapture terminated\n\n")
printf("%-5s %-15s %-10s %-10s %-10s\n",
"If", "Process", "Pid", "RcvPktCnt", "XmtPktCnt")
foreach([dev, pid, name] in recv) {
recvcnt = @count(recv[dev, pid, name])
xmtcnt = @count(xmit[dev, pid, name])
printf("%-5s %-15s %-10d %-10d %-10d\n", dev, name, pid, recvcnt, xmtcnt)
}
}
(9) Systemtap usage stories and interesting demos
https://sourceware.org/systemtap/wiki/WarStories
官網提供的非常多樣例。
內核調試神器SystemTap — 簡單介紹與使用（一）