目錄

三、對策

一、Docker啟動異常表現：

1.狀態反覆restaring，用命令檢視

$docker ps -a
CONTAINER ID    IMAGE  COMMAND       CREATED      STATUS       PORTS    NAMES
21c09be88c11     docker.xxxx.cn:5000/xxx-tes/xxx_tes:1.0.6   "/usr/local/tomcat..."   9 days ago          Restarting (1) Less than a second ago                      xxx10
 

2.Docker日誌有明顯問題：

$docker logs [容器名/容器ID]

二、Docker啟動異常的可能原因：

2.1.記憶體不夠 

Docker 啟動至少需要2G記憶體，首先執行free -mh命令檢視剩餘記憶體是否足夠

直接檢視記憶體

$free -mh
            total        used        free      shared  buff/cache   available
Mem:         15G         14G          627M        195M        636M     726M
Swap:         0B          0B          0B
 

分析日誌

有時候一瞬間記憶體過載溢位，導致部分程序被殺死，看起來記憶體也是夠用的，事實上docker還是會反覆重啟，就需要通過docker日誌和系統日誌信的息來進一步分析：

分析docker日誌

檢視docker日誌看到記憶體溢位的資訊，要仔細翻閱才能找到資訊，並不是在最下面

$docker logs [容器名/容器ID]|less 
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# An error report file with more information is saved as:
# //hs_err_pid1.log
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# Can not save log file, dump to screen..
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2756), pid=1, tid=140325689620224
#
# JRE version:  (7.0_79-b15) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: //core or core.1
#
 

分析系統日誌

檢視系統日誌，發現有大量由於記憶體溢位，程序被殺死的記錄

$grep -i 'Out of Memory' /var/log/messages
Apr  7 10:04:02 centos106 kernel: Out of memory: Kill process 1192 (java) score 54 or sacrifice child
Apr  7 10:08:00 centos106 kernel: Out of memory: Kill process 2301 (java) score 54 or sacrifice child
Apr  7 10:09:59 centos106 kernel: Out of memory: Kill process 28145 (java) score 52 or sacrifice child
Apr  7 10:20:40 centos106 kernel: Out of memory: Kill process 2976 (java) score 54 or sacrifice child
Apr  7 10:21:08 centos106 kernel: Out of memory: Kill process 3577 (java) score 47 or sacrifice child
Apr  7 10:21:08 centos106 kernel: Out of memory: Kill process 3631 (java) score 47 or sacrifice child
Apr  7 10:21:08 centos106 kernel: Out of memory: Kill process 3634 (java) score 47 or sacrifice child
Apr  7 10:21:08 centos106 kernel: Out of memory: Kill process 3640 (java) score 47 or sacrifice child
Apr  7 10:21:08 centos106 kernel: Out of memory: Kill process 3654 (java) score 47 or sacrifice child
Apr  7 10:27:27 centos106 kernel: Out of memory: Kill process 6998 (java) score 51 or sacrifice child
Apr  7 10:27:28 centos106 kernel: Out of memory: Kill process 7027 (java) score 52 or sacrifice child
Apr  7 10:28:10 centos106 kernel: Out of memory: Kill process 7571 (java) score 42 or sacrifice child
Apr  7 10:28:10 centos106 kernel: Out of memory: Kill process 7586 (java) score 42 or sacrifice child

2.2.埠衝突

該docker監聽埠已經被其他程序佔用，一般此種問題容易出現在新部署的服務，或在原有機器上部署新的後臺服務，所以在部署之前應該執行命令檢查埠是否已經被佔用，如果上線後發現佔有則應改為可用埠再重啟之。

檢查命令：    $netstat -nltp|grep [規劃的埠號]

三、對策

3.1.記憶體不夠的對策：

對策1:

3.1.1 saltstack的minion在執行過久之後，可能佔用大量記憶體，需要將其重啟。重啟命令可能有時並不起作用。主要檢查執行狀態，如果未成功停止，則重新重啟；

對策2:

3.2.2 ELK日誌收集程式或者其他java程序佔用過高，用top和ps命令排查，謹慎確定程序的作用，在確保不影響業務的情況下，停止相關程序；

對策3:

釋放被佔用的記憶體（buff/cache）：

 $sync  #將記憶體資料寫入磁碟
 $echo 3 > /proc/sys/vm/drop_caches  #釋放被佔用的記憶體

對策4:

有時候並不是buff/cache過高導致記憶體不夠用，確實是被很多必要的程序消耗掉了記憶體，那就需要從機器資源分配使用的層面去考慮和解決了。

3.2 埠衝突的對策

對策1:

一般此種問題容易出現在新部署的服務，或在原有機器上部署新的後臺服務，所以在部署之前應該執行命令檢查埠是否已經被佔用，如果上線後發現佔有則應改為可用埠再重啟之。 

檢查命令：    $netstat -nltp|grep [規劃的埠號]