remove rss table ble 需求 nes 決定 href pru

tomcat異常停，進程被killed

對應tomcat日誌如下：

/application/tomcat-service-8080/bin/catalina.sh: line 386: 4370 Killed "/usr/local/jdk/bin/java" "-Djava.util.logging.config.file=/application/tomcat-service-8080/conf/logg
ing.properties" -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -Xms4096m -Xmx4096m -XX:PermSize=128M -XX:MaxNewSize=512m -XX:MaxPermSize=256m -Djava.awt.headless=tr
ue -Djava.endorsed.dirs="/application/tomcat-service-8080/endorsed" -classpath

日誌中沒有報錯，沒有內存溢出OutOfMemoryError

Google之後了解到一個Linux OOM Killer

Linux 內核根據應用程序的要求分配內存，通常來說應用程序分配了內存但是並沒有實際全部使用，為了提高性能，這部分沒用的內存可以留作它用，這部分內存是屬於每個進程的，內核直接回收利用的話比較麻煩，所以內核采用一種過度分配內存（over-commit memory）的辦法來間接利用這部分 “空閑” 的內存，提高整體內存的使用效率。一般來說這樣做沒有問題，但當大多數應用程序都消耗完自己的內存的時候麻煩就來了，因為這些應用程序的內存需求加起來超出了物理內存（包括 swap）的容量，內核（OOM killer）必須殺掉一些進程才能騰出空間保障系統正常運行。用銀行的例子來講可能更容易懂一些，部分人取錢的時候銀行不怕，銀行有足夠的存款應付，當全國人民（或者絕大多數）都取錢而且每個人都想把自己錢取完的時候銀行的麻煩就來了，銀行實際上是沒有這麽多錢給大家取的。

內核檢測到系統內存不足、挑選並殺掉某個進程的過程可以參考內核源代碼 linux/mm/oom_kill.c，當系統內存不足的時候，out_of_memory() 被觸發，然後調用 select_bad_process() 選擇一個 “bad” 進程殺掉，如何判斷和選擇一個 “bad” 進程呢，總不能隨機選吧？挑選的過程由 oom_badness() 決定，挑選的算法和想法都很簡單很樸實：最 bad 的那個進程就是那個最占用內存的進程。

查看/var/log/messages

Aug 2 20:01:03 iZ25crr8mcwZ systemd: Started Session 65527 of user root.
Aug 2 20:10:14 iZ25crr8mcwZ systemd: Created slice user-0.slice.
Aug 2 20:10:25 iZ25crr8mcwZ systemd: Starting Session 65528 of user root.
Aug 2 20:10:53 iZ25crr8mcwZ systemd: Started Session 65528 of user root.
Aug 2 20:12:24 iZ25crr8mcwZ kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: java cpuset=/ mems_allowed=0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: CPU: 0 PID: 4385 Comm: java Tainted: GF O-------------- 3.10.0-123.9.3.el7.x86_64 #1
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
Aug 2 20:12:24 iZ25crr8mcwZ kernel: ffff880008bf0b60 00000000f4c1c64c ffff880230453938 ffffffff815e239b
Aug 2 20:12:24 iZ25crr8mcwZ kernel: ffff8802304539c8 ffffffff815dd372 ffffffff810b69b8 ffff880036a9a050
Aug 2 20:12:24 iZ25crr8mcwZ kernel: 0000000000000202 ffff880008bf0b60 ffff8802304539b0 ffffffff811030af
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Call Trace:
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff815e239b>] dump_stack+0x19/0x1b
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff815dd372>] dump_header+0x8e/0x214
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff810b69b8>] ? ktime_get_ts+0x48/0xe0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff811030af>] ? delayacct_end+0x8f/0xb0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff8114526e>] oom_kill_process+0x24e/0x3b0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff81144dd6>] ? find_lock_task_mm+0x56/0xc0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff81145a96>] out_of_memory+0x4b6/0x4f0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff8114b5d9>] __alloc_pages_nodemask+0xa09/0xb10
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff81188349>] alloc_pages_current+0xa9/0x170
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff81141a67>] __page_cache_alloc+0x87/0xb0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff81143da8>] filemap_fault+0x188/0x430
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff81167e9e>] __do_fault+0x7e/0x520
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff8116c1e5>] handle_mm_fault+0x3e5/0xd90
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff815edb06>] __do_page_fault+0x156/0x540
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff810c2be2>] ? do_futex+0x172/0x5b0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff8109b836>] ? __dequeue_entity+0x26/0x40
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff81011619>] ? __switch_to+0x179/0x490
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff815edf0a>] do_page_fault+0x1a/0x70
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff810b6769>] ? do_gettimeofday+0x29/0x70
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff815ed599>] do_async_page_fault+0x29/0xe0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [<ffffffff815ea178>] async_page_fault+0x28/0x30
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Mem-Info:
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Node 0 DMA per-cpu:
Aug 2 20:12:24 iZ25crr8mcwZ kernel: CPU 0: hi: 0, btch: 1 usd: 0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: CPU 1: hi: 0, btch: 1 usd: 0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: CPU 2: hi: 0, btch: 1 usd: 0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: CPU 3: hi: 0, btch: 1 usd: 0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Node 0 DMA32 per-cpu:
Aug 2 20:12:24 iZ25crr8mcwZ kernel: CPU 0: hi: 186, btch: 31 usd: 84
Aug 2 20:12:24 iZ25crr8mcwZ kernel: CPU 1: hi: 186, btch: 31 usd: 113
Aug 2 20:12:24 iZ25crr8mcwZ kernel: CPU 2: hi: 186, btch: 31 usd: 109
Aug 2 20:12:24 iZ25crr8mcwZ kernel: CPU 3: hi: 186, btch: 31 usd: 45
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Node 0 Normal per-cpu:
Aug 2 20:12:24 iZ25crr8mcwZ kernel: CPU 0: hi: 186, btch: 31 usd: 0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: CPU 1: hi: 186, btch: 31 usd: 0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: CPU 2: hi: 186, btch: 31 usd: 0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: CPU 3: hi: 186, btch: 31 usd: 0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: active_anon:1714079 inactive_anon:38316 isolated_anon:0
active_file:106 inactive_file:169 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
free:25831 slab_reclaimable:17057 slab_unreclaimable:6114
mapped:3441 shmem:96445 pagetables:4873 bounce:0
free_cma:0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Node 0 DMA free:15900kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):
0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB uns
table:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug 2 20:12:24 iZ25crr8mcwZ kernel: lowmem_reserve[]: 0 2801 7792 7792
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Node 0 DMA32 free:44996kB min:24252kB low:30312kB high:36376kB active_anon:2645684kB inactive_anon:86640kB active_file:352kB inactive_file:608kB unevictab
le:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2869240kB mlocked:0kB dirty:0kB writeback:0kB mapped:10716kB shmem:230460kB slab_reclaimable:35324kB slab_unreclaimable:
8996kB kernel_stack:2048kB pagetables:9864kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1747 all_unreclaimable? yes
Aug 2 20:12:24 iZ25crr8mcwZ kernel: lowmem_reserve[]: 0 0 4990 4990
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Node 0 Normal free:42428kB min:43192kB low:53988kB high:64788kB active_anon:4210632kB inactive_anon:66624kB active_file:72kB inactive_file:68kB unevictabl
e:0kB isolated(anon):0kB isolated(file):0kB present:5242880kB managed:5110124kB mlocked:0kB dirty:0kB writeback:0kB mapped:3048kB shmem:155320kB slab_reclaimable:32904kB slab_unreclaimable:15
452kB kernel_stack:3408kB pagetables:9628kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:347 all_unreclaimable? yes
Aug 2 20:12:24 iZ25crr8mcwZ kernel: lowmem_reserve[]: 0 0 0 0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Node 0 DMA32: 1204*4kB (UEMR) 950*8kB (UEM) 1208*16kB (UEM) 252*32kB (UEM) 72*64kB (UEM) 6*128kB (UE) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45184kB
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Node 0 Normal: 400*4kB (UEM) 326*8kB (UEM) 543*16kB (UEM) 237*32kB (EM) 145*64kB (UEM) 61*128kB (UEM) 15*256kB (UEM) 0*512kB 1*1024kB (M) 0*2048kB 0*4096k
B = 42432kB
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug 2 20:12:24 iZ25crr8mcwZ kernel: 96713 total pagecache pages
Aug 2 20:12:24 iZ25crr8mcwZ kernel: 0 pages in swap cache
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Swap cache stats: add 0, delete 0, find 0/0
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Free swap = 0kB
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Total swap = 0kB
Aug 2 20:12:24 iZ25crr8mcwZ kernel: 2097151 pages RAM
Aug 2 20:12:24 iZ25crr8mcwZ kernel: 94167 pages reserved
Aug 2 20:12:24 iZ25crr8mcwZ kernel: 274302 pages shared
Aug 2 20:12:24 iZ25crr8mcwZ kernel: 1961600 pages non-shared
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 338] 0 338 14844 3705 33 0 0 systemd-journal
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 354] 0 354 26113 61 20 0 0 lvmetad
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 357] 0 357 10509 149 22 0 -1000 systemd-udevd
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 510] 0 510 164576 2774 152 0 0 rsyslogd
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 513] 81 513 7158 101 20 0 -900 dbus-daemon
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 515] 0 515 31583 163 17 0 0 crond
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 519] 28 519 363742 743 61 0 0 nscd
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 609] 0 609 1621 25 8 0 0 iprupdate
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 612] 0 612 1621 25 9 0 0 iprinit
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 649] 0 649 9781 23 9 0 0 iprdump
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 850] 38 850 7399 155 19 0 0 ntpd
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 895] 0 895 386 45 4 0 0 aliyun-service
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [19586] 0 19586 20697 210 44 0 -1000 sshd
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [22773] 996 22773 21196 206 42 0 0 zabbix_agentd
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [22774] 996 22774 21196 285 42 0 0 zabbix_agentd
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [22775] 996 22775 21226 238 43 0 0 zabbix_agentd
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [22776] 996 22776 21226 238 43 0 0 zabbix_agentd
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [22777] 996 22777 21226 254 43 0 0 zabbix_agentd
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [22778] 996 22778 21196 237 44 0 0 zabbix_agentd
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [13392] 0 13392 59021 2804 68 0 0 salt-minion
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [13395] 0 13395 170977 5407 110 0 0 salt-minion
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 1901] 0 1901 30507 151 11 0 0 wrapper
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 1911] 0 1911 627491 17315 90 0 0 java
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 5580] 0 5580 6338 162 16 0 0 AliYunDunUpdate
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 7267] 0 7267 48303 1580 91 0 0 AliYunDun
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 4955] 0 4955 27501 34 10 0 0 agetty
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [18909] 1000 18909 28313 73 11 0 0 catalina.sh
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [18910] 1000 18910 1077 38 8 0 0 cronolog
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [18911] 1000 18911 1245006 596619 1349 0 0 java
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [19657] 0 19657 8671 80 22 0 0 systemd-logind
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 4368] 1000 4368 28313 73 10 0 0 catalina.sh
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 4369] 1000 4369 1077 38 7 0 0 cronolog
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [ 4370] 1000 4370 1796041 1028084 2224 0 0 java

Aug 2 20:12:24 iZ25crr8mcwZ kernel: [30951] 0 30951 33414 294 66 0 0 sshd
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [31046] 1000 31046 33447 311 65 0 0 sshd
Aug 2 20:12:24 iZ25crr8mcwZ kernel: [31047] 1000 31047 29136 402 13 0 0 bash
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Out of memory: Kill process 4370 (java) score 514 or sacrifice child
Aug 2 20:12:24 iZ25crr8mcwZ kernel: Killed process 4370 (java) total-vm:7184164kB, anon-rss:4112336kB, file-rss:0kB
Aug 2 20:12:25 iZ25crr8mcwZ systemd-logind: Removed session 65322.
Aug 2 20:20:01 iZ25crr8mcwZ systemd: Created slice user-0.slice.
Aug 2 20:20:01 iZ25crr8mcwZ systemd: Starting Session 65529 of user root.

服務殺死了大的內存進程 4370 ，所以tomcat進程異常消失

tomcat正常運行一段時間後，tomcat異常停，進程被killed