Linux程序與執行緒

概念就不提了，Richard Stevens的描述：

fork is expensive. Memory is copied from the parent to the child, all descriptors are duplicated in the child, and so on. Current implementations use a technique called copy-on-write, which avoids a copy of the parent’s data space to the child until the child needs its own copy. But, regardless of this optimization, fork is expensive. IPC is required to pass information between the parent and child after the fork. Passing information from the parent to the child before the fork is easy, since the child starts with a copy of the parent’s data space and with a copy of all the parent’s descriptors. But, returning information from the child to the parent takes more work. Threads help with both problems. Threads are sometimes called lightweight processes since a thread is “lighter weight” than a process. That is, thread creation can be 10–100 times faster than process creation. All threads within a process share the same global memory. This makes the sharing of information easy between the threads, but along with this simplicity comes the problem.

Linux中建立程序用fork操作，執行緒用clone操作。通過ps -ef看到的是程序列表，執行緒可以通過ps -eLf來檢視。 用top命令的話，通過H開關也可以切換到執行緒檢視。

具體到Java執行緒模型，規範是沒有規定Java執行緒和系統執行緒的對應關係的，不過目前常見的實現是一對一的。 參考http://openjdk.java.net/groups/hotspot/docs/RuntimeOverview.html#Thread%20Management|outline

問題排查思路

如果建立不了Java執行緒，報錯是

Exception in thread “main” java.lang.OutOfMemoryError: unable to create new native thread

下面是常見的問題原因：

記憶體太小

在Java中建立一個執行緒需要消耗一定的棧空間，預設的棧空間是1M(可以根據應用情況指定-Xss引數進行調整)，棧空間過小或遞迴呼叫過深，可能會出現StackOverflowError。

對於一個程序來說，假設一定量可使用的記憶體，分配給堆空間的越多，留給棧空間的就越少。這個限制常見於32位Java應用，程序空間4G，使用者空間2G(Linux下3G，所以通常堆可以設定更大一些)，減去堆空間大小(通過-Xms、-Xmx指定範圍)，減去非堆空間(其中永久代部分通過PermSize、MaxPermSize指定大小，在Java8換成了MetaSpace，預設不限制大小)，再減去虛擬機器自身消耗，剩下的就是棧空間，假設剩下300M，那麼理論上就限制了只能開300執行緒。不過對於64位應用，由於程序空間近乎無限大，所以可以不考慮這個問題。

ulimit限制

執行緒數還會受到系統限制，系統限制通過ulimit -a可以檢視到。

https://ss64.com/bash/ulimit.html

[email protected]:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7823
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7823
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

相關的限制有

max memory size - 最大記憶體限制，在64位系統上通常都設定成unlimited  
max user processes - 每使用者總的最大程序數(包括執行緒) 
virtual memory - 虛擬記憶體限制，在64位系統上通常都設定成unlimited 

這些引數可以通過ulimit命令(當前使用者臨時生效)或者配置檔案/etc/security/limits.conf(永久生效)進行修改。   檢查某個程序的限制是否生效，可以通過/proc/PID/limits檢視執行時狀態。

引數sys.kernel.threads-max限制

https://www.kernel.org/doc/Documentation/sysctl/kernel.txt

This value controls the maximum number of threads that can be created
using fork().

During initialization the kernel sets this value such that even if the
maximum number of threads is created, the thread structures occupy only
a part (1/8th) of the available RAM pages.

The minimum value that can be written to threads-max is 20.
The maximum value that can be written to threads-max is given by the
constant FUTEX_TID_MASK (0x3fffffff).
If a value outside of this range is written to threads-max an error
EINVAL occurs.

The value written is checked against the available RAM pages. If the
thread structures would occupy too much (more than 1/8th) of the
available RAM pages threads-max is reduced accordingly.

表示系統全域性的匯流排程數限制。設定方式有:

# 方式1 執行時限制,臨時生效
echo 999999 > /proc/sys/kernel/threads-max
# 方式2 修改/etc/sysctl.conf，永久生效
sys.kernel.threads-max = 999999

引數sys.kernel.pid_max限制

https://www.kernel.org/doc/Documentation/sysctl/kernel.txt

PID allocation wrap value.  When the kernel's next PID value
reaches this value, it wraps back to a minimum PID value.
PIDs of value pid_max or larger are not allocated.

表示系統全域性的PID號數值的限制。設定方式有:

# 方式1 執行時限制,臨時生效
echo 999999 > /proc/sys/kernel/pid_max
# 方式2 修改/etc/sysctl.conf，永久生效
sys.kernel.pid_max = 999999

引數sys.vm.max_map_count限制

https://www.kernel.org/doc/Documentation/sysctl/vm.txt

This file contains the maximum number of memory map areas a process
may have. Memory map areas are used as a side-effect of calling
malloc, directly by mmap, mprotect, and madvise, and also when loading
shared libraries.

While most applications need less than a thousand maps, certain
programs, particularly malloc debuggers, may consume lots of them,
e.g., up to one or two maps per allocation.

The default value is 65536.

表示單個程式所能使用記憶體對映空間的數量限制。設定方式有:

# 方式1 執行時限制,臨時生效
echo 999999 > /proc/sys/vm/max_map_count
# 方式2 修改/etc/sysctl.conf，永久生效
sys.vm.max_map_count = 999999

在其他資源可用的情況下，單個vm能開啟的最大執行緒數是這個值的一半，可以通過cat /proc/PID/maps | wc -l檢視目前使用的對映數量。

至於為什麼只有一半，結合一些材料和原始碼分析了一下:

常見的警告資訊是這樣的，見JavaThread::create_stack_guard_pages()

Attempt to protect stack guard pages failed.
Attempt to deallocate stack guard pages failed. 

見current_stack_region()的圖示，結合一下R大的相關解釋:http://hllvm.group.iteye.com/group/topic/37717

如下圖所示，通常的Java執行緒，會包括一個glibc的guard page和HotSpot的guard pages，其中JavaThread::create_stack_guard_pages()就是建立HotSpot Guard Pages用的，這裡正常應該會有2次VMA，所以最大值只能有一半，從/proc/PID/maps中也可以看到增加一個執行緒會增加2個地址相連的對映空間。

// Java thread:
//
//   Low memory addresses
//    +------------------------+
//    |                        |\  JavaThread created by VM does not have glibc
//    |    glibc guard page    | - guard, attached Java thread usually has
//    |                        |/  1 page glibc guard.
// P1 +------------------------+ Thread::stack_base() - Thread::stack_size()
//    |                        |\
//    |  HotSpot Guard Pages   | - red and yellow pages
//    |                        |/
//    +------------------------+ JavaThread::stack_yellow_zone_base()
//    |                        |\
//    |      Normal Stack      | -
//    |                        |/
// P2 +------------------------+ Thread::stack_base()
//
// Non-Java thread:
//
//   Low memory addresses
//    +------------------------+
//    |                        |\
//    |  glibc guard page      | - usually 1 page
//    |                        |/
// P1 +------------------------+ Thread::stack_base() - Thread::stack_size()
//    |                        |\
//    |      Normal Stack      | -
//    |                        |/
// P2 +------------------------+ Thread::stack_base()
//
// ** P1 (aka bottom) and size ( P2 = P1 - size) are the address and stack size returned from
//    pthread_attr_getstack()

cgroup限制

現在新點的作業系統採用systemd的init程式，支援cgroup控制特性。docker的資源隔離底層技術就是這個。

其中有個重要的限制就是最大任務數TasksMax,通過設定cgroup的pids.max來限制。例如suse sp2的發行說明，見https://www.suse.com/releasenotes/x86_64/SUSE-SLES/12-SP2/#fate-320358

If you notice regressions, you can change a number of TasksMax settings.

To control the default TasksMax= setting for services and scopes running on the system, use the system.conf setting DefaultTasksMax=. This setting defaults to 512, which means services that are not explicitly configured otherwise will only be able to create 512 processes or threads at maximum.

For thread- or process-heavy services, you may need to set a higher TasksMax value. In such cases, set TasksMax directly in the specific unit files. Either choose a numeric value or even infinity.

Similarly, you can limit the total number of processes or tasks each user can own concurrently. To do so, use the logind.conf setting UserTasksMax (the default is 12288).

nspawn containers now also have a TasksMax value set, with a default of 16384.

上面的描述，說明

對於登入會話，有個預設的限制UserTasksMax，配置在/etc/systemd/logind.conf，限制了某個使用者的預設的總任務數，例如上面限制了最大12288，修改這個配置檔案可以通過systemctl restart systemd-logind重新載入

對於服務來說，配置在/etc/systemd/system.conf的DefaultTasksMax引數，預設是512(不同的發行版很可能不一樣)，如果需要定製，需要根據服務獨立配置  

通過find /sys/fs/cgroup -name “pids.max” 可以看到各種細化的配置，例如./pids/user.slice/user-1000.slice/pids.max就是id為1000的使用者的限制，相當於覆蓋了上面logind.conf的預設設定，修改這個值會立即生效。

要檢視某個程序的具體限制，可以通過/proc/PID/cgroup檢視執行時狀態，其中裡邊有pids.max就是對應的限制情況。詳細點的可以看看這個案例:https://zhuanlan.zhihu.com/p/29192624