1. 程式人生 > >Kubernetes之kubectl常用命令使用指南:2:故障排查

Kubernetes之kubectl常用命令使用指南:2:故障排查

alpha eth message resources mount 權限 copyright count limit

kubectl是一個用於操作kubernetes集群的命令行接口,通過利用kubectl的各種命令可以實現各種功能,是在使用kubernetes中非常常用的工具。這裏我們會通過一些簡單的實例來展現其中一些高頻命令的使用方法。
更為重要的是這些命令使用的場景以及能夠解決什麽樣的問題。上篇文章我們介紹了創建和刪除相關的幾條命令,這篇文章我們來看一下出現問題時最常用的另外九條命令。

常用命令

kubectl故障排查相關,本文將會簡單介紹一下如下命令

項番命令說明
No.1 version 顯示客戶端和服務器側版本信息
No.2 api-versions 以group/version的格式顯示服務器側所支持的API版本
No.3 explain 顯示資源文檔信息
No.4 get 取得確認對象信息列表
No.5 describe 取得確認對象的詳細信息
No.6 logs 取得pod中容器的log信息
No.7 exec 在容器中執行一條命令
No.8 cp 從容器考出或向容器考入文件
No.9 attach Attach到一個運行中的容器上

事前準備

kubectl version

version命令用於確認客戶端和服務器側的版本信息,不同的版本的情況變化可能很大,所以故障排除時首先也需要確認的是現場環境的版本信息。
從下面可以清楚地看到,本文驗證時所使用的版本為1.5.2

[[email protected] tmp]# kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
[[email protected]
/* */ tmp]#
  • 1
  • 2
  • 3
  • 4

集群構成

一主三從的Kubernetes集群

項番類型HostnameIP
No.1 Master ku8-1 192.168.32.131
No.1 Node ku8-2 192.168.32.132
No.1 Node ku8-3 192.168.32.133
No.1 Node ku8-4 192.168.32.134
[root@ku8-1 tmp]# kubectl get nodes
NAME             STATUS    AGE
192.168.32.132   Ready     12m
192.168.32.133   Ready     11m
192.168.32.134   Ready     11m
[root@ku8-1 tmp]# 
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6

kubectl api-versions

使用api-versions命令可以列出當前版本的kubernetes的服務器端所支持的api版本信息。

[root@ku8-1 tmp]# kubectl api-versions
apps/v1beta1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1beta1
autoscaling/v1
batch/v1
certificates.k8s.io/v1alpha1
extensions/v1beta1
policy/v1beta1
rbac.authorization.k8s.io/v1alpha1
storage.k8s.io/v1beta1
v1
[root@ku8-1 tmp]#
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13

kubectl explain

使用kubectl explain可以和kubectl help一樣進行輔助的功能確認,使用它可以了解各個部分的說明和組成部分。比如如下可以看到對rc的說明,在故障排除時作用並不具有太大作用,到是可以多讀讀加深一下對各個部分的理解。

[root@ku8-1 ~]# kubectl explain rc
DESCRIPTION:
ReplicationController represents the configuration of a replication controller.

FIELDS:
   apiVersion   <string>
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#resources

   kind <string>
     Kind is a string value representing the REST resource this object
     represents. Servers may infer this from the endpoint the client submits
     requests to. Cannot be updated. In CamelCase. More info:
     http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#types-kinds

   metadata <Object>
     If the Labels of a ReplicationController are empty, they are defaulted to
     be the same as the Pod(s) that the replication controller manages. Standard
     object‘s metadata. More info:
     http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata

   spec <Object>
     Spec defines the specification of the desired behavior of the replication
     controller. More info:
     http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status

   status   <Object>
     Status is the most recently observed status of the replication controller.
     This data may be out of date by some window of time. Populated by the
     system. Read-only. More info:
     http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status


[root@ku8-1 ~]# 
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36

explain命令能夠確認的信息類別

其所能支持的類別如下:

類別
clusters (僅對federation apiservers有效)
componentstatuses (縮寫 cs)
configmaps (縮寫 cm)
daemonsets (縮寫 ds)
deployments (縮寫 deploy)
endpoints (縮寫 ep)
events (縮寫 ev)
horizontalpodautoscalers (縮寫 hpa)
ingresses (縮寫 ing)
jobs
limitranges (縮寫 limits)
namespaces (縮寫 ns)
networkpolicies
nodes (縮寫 no)
persistentvolumeclaims (縮寫 pvc)
persistentvolumes (縮寫 pv)
pods (縮寫 po)
podsecuritypolicies (縮寫 psp)
podtemplates
replicasets (縮寫 rs)
replicationcontrollers (縮寫 rc)
resourcequotas (縮寫 quota)
secrets
serviceaccounts (縮寫 sa)
services (縮寫 svc)
statefulsets
storageclasses
thirdpartyresources

事前準備

剩下的一些命令需要事前作一些準備,我們還是用上篇文章所用的yaml文件創建mysql和sonarqube的Deployment和pod。

yaml文件準備

[root@ku8-1 tmp]# ls yamls
mysql.yaml  sonar.yaml
[root@ku8-1 tmp]# cat yamls/mysql.yaml 
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: mysql
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: mysql
    spec:
      containers:
      - name: mysql
        image: 192.168.32.131:5000/mysql:5.7.16
        ports:
        - containerPort: 3306
          protocol: TCP
        env:
          - name: MYSQL_ROOT_PASSWORD
            value: "hello123"
[root@ku8-1 tmp]# cat yamls/sonar.yaml 
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: sonarqube
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: sonarqube
    spec:
      containers:
      - name: sonarqube
        image: 192.168.32.131:5000/sonarqube:5.6.5
        ports:
        - containerPort: 9000
          protocol: TCP
[root@ku8-1 tmp]# 
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44

啟動

[root@ku8-1 tmp]# kubectl create -f yamls/
deployment "mysql" created
deployment "sonarqube" created
[root@ku8-1 tmp]# 
  • 1
  • 2
  • 3
  • 4

kubectl get

使用get命令確認所創建出來的pod和deployment的信息

確認pod

可以看到創建出來的pod的所有信息,也可以使用Kubectl get po進行確認

[root@ku8-1 tmp]# kubectl get pods
NAME                         READY     STATUS    RESTARTS   AGE
mysql-478535978-1dnm2        1/1       Running   0          34s
sonarqube-3574384362-m7mdq   1/1       Running   0          34s
[root@ku8-1 tmp]# 
  • 1
  • 2
  • 3
  • 4
  • 5

確認deployment

可以看到創建出來的deployment的所有信息

[root@ku8-1 tmp]# kubectl get deployments
NAME        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
mysql       1         1         1            1           41s
sonarqube   1         1         1            1           41s
[root@ku8-1 tmp]#
  • 1
  • 2
  • 3
  • 4
  • 5

如果希望得到更加詳細一點的信息,可以加上-o wide參數,比如對pods可以看到此pod在哪個node上運行,此pod的集群IP是多少也被一並顯示了

[root@ku8-1 tmp]# kubectl get pods -o wide
NAME                         READY     STATUS    RESTARTS   AGE       IP             NODE
mysql-478535978-1dnm2        1/1       Running   0          2m        172.200.44.2   192.168.32.133
sonarqube-3574384362-m7mdq   1/1       Running   0          2m        172.200.59.2   192.168.32.134
[root@ku8-1 tmp]#
  • 1
  • 2
  • 3
  • 4
  • 5

確認node信息

顯示node的信息

[[email protected] tmp]# kubectl get nodes -o wide
NAME             STATUS    AGE       EXTERNAL-IP
192.168.32.132   Ready     6h        <none>
192.168.32.133   Ready     6h        <none>
192.168.32.134   Ready     6h        <none>
[[email protected] tmp]#
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6

確認namespace信息

列出所有的namespace

[root@ku8-1 tmp]# kubectl get namespaces
NAME          STATUS    AGE
default       Active    6h
kube-system   Active    6h
[root@ku8-1 tmp]# 
  • 1
  • 2
  • 3
  • 4
  • 5

get命令能夠確認的信息類別

使用node/pod/event/namespaces等結合起來,能夠獲取集群基本信息和狀況, 其所能支持的類別如下:

類別
clusters (僅對federation apiservers有效)
componentstatuses (縮寫 cs)
configmaps (縮寫 cm)
daemonsets (縮寫 ds)
deployments (縮寫 deploy)
endpoints (縮寫 ep)
events (縮寫 ev)
horizontalpodautoscalers (縮寫 hpa)
ingresses (縮寫 ing)
jobs
limitranges (縮寫 limits)
namespaces (縮寫 ns)
networkpolicies
nodes (縮寫 no)
persistentvolumeclaims (縮寫 pvc)
persistentvolumes (縮寫 pv)
pods (縮寫 po)
podsecuritypolicies (縮寫 psp)
podtemplates
replicasets (縮寫 rs)
replicationcontrollers (縮寫 rc)
resourcequotas (縮寫 quota)
secrets
serviceaccounts (縮寫 sa)
services (縮寫 svc)
statefulsets
storageclasses
thirdpartyresources

kubectl describe

確認node詳細信息

一般使用get命令取得node信息,然後使用describe確認詳細信息。

[root@ku8-1 tmp]# kubectl get nodes
NAME             STATUS    AGE
192.168.32.132   Ready     6h
192.168.32.133   Ready     6h
192.168.32.134   Ready     6h
[root@ku8-1 tmp]# kubectl describe node 192.168.32.132
Name:           192.168.32.132
Role:           
Labels:         beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/os=linux
            kubernetes.io/hostname=192.168.32.132
Taints:         <none>
CreationTimestamp:  Wed, 28 Jun 2017 23:06:22 -0400
Phase:          
Conditions:
  Type          Status  LastHeartbeatTime           LastTransitionTime          Reason              Message
  ----          ------  -----------------           ------------------          ------              -------
  OutOfDisk         False   Thu, 29 Jun 2017 05:52:07 -0400     Wed, 28 Jun 2017 23:06:22 -0400     KubeletHasSufficientDisk    kubelet has sufficient disk space available
  MemoryPressure    False   Thu, 29 Jun 2017 05:52:07 -0400     Wed, 28 Jun 2017 23:06:22 -0400     KubeletHasSufficientMemory  kubelet has sufficient memory available
  DiskPressure      False   Thu, 29 Jun 2017 05:52:07 -0400     Wed, 28 Jun 2017 23:06:22 -0400     KubeletHasNoDiskPressure    kubelet has no disk pressure
  Ready         True    Thu, 29 Jun 2017 05:52:07 -0400     Wed, 28 Jun 2017 23:06:34 -0400     KubeletReady            kubelet is posting ready status
Addresses:      192.168.32.132,192.168.32.132,192.168.32.132
Capacity:
 alpha.kubernetes.io/nvidia-gpu:    0
 cpu:                   1
 memory:                2032128Ki
 pods:                  110
Allocatable:
 alpha.kubernetes.io/nvidia-gpu:    0
 cpu:                   1
 memory:                2032128Ki
 pods:                  110
System Info:
 Machine ID:            22718f24279240be9fe0c469187f901a
 System UUID:           9F584D56-F5B3-FAB8-3985-938D67451312
 Boot ID:           fe3b2606-37ee-4b07-8de2-438fe29bf765
 Kernel Version:        3.10.0-514.el7.x86_64
 OS Image:          CentOS Linux 7 (Core)
 Operating System:      linux
 Architecture:          amd64
 Container Runtime Version: docker://1.13.1
 Kubelet Version:       v1.5.2
 Kube-Proxy Version:        v1.5.2
ExternalID:         192.168.32.132
Non-terminated Pods:        (0 in total)
  Namespace         Name        CPU Requests    CPU Limits  Memory Requests Memory Limits
  ---------         ----        ------------    ----------  --------------- -------------
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.
  CPU Requests  CPU Limits  Memory Requests Memory Limits
  ------------  ----------  --------------- -------------
  0 (0%)    0 (0%)      0 (0%)      0 (0%)
No events.
[root@ku8-1 tmp]#
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54

確認pod

確認某一pod詳細信息

[root@ku8-1 tmp]# kubectl describe pod mysql-478535978-1dnm2
Name:       mysql-478535978-1dnm2
Namespace:  default
Node:       192.168.32.133/192.168.32.133
Start Time: Thu, 29 Jun 2017 05:04:21 -0400
Labels:     name=mysql
        pod-template-hash=478535978
Status:     Running
IP:     172.200.44.2
Controllers:    ReplicaSet/mysql-478535978
Containers:
  mysql:
    Container ID:   docker://47ef1495e86f4b69414789e81081fa55b837dafe9e47944894e7cb3733700410
    Image:      192.168.32.131:5000/mysql:5.7.16
    Image ID:       docker-pullable://192.168.32.131:[email protected]:410b279f6827492da7a355135e6e9125849f62eeca76429974a534f021852b58
    Port:       3306/TCP
    State:      Running
      Started:      Thu, 29 Jun 2017 05:04:22 -0400
    Ready:      True
    Restart Count:  0
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dzs1w (ro)
    Environment Variables:
      MYSQL_ROOT_PASSWORD:  hello123
Conditions:
  Type      Status
  Initialized   True 
  Ready     True 
  PodScheduled  True 
Volumes:
  default-token-dzs1w:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-dzs1w
QoS Class:  BestEffort
Tolerations:    <none>
No events.
[root@ku8-1 tmp]# 
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37

確認deployment詳細信息

確認某一deployment的詳細信息

[root@ku8-1 tmp]# kubectl get deployment
NAME        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
mysql       1         1         1            1           1h
sonarqube   1         1         1            1           1h
[root@ku8-1 tmp]# kubectl describe deployment mysql
Name:           mysql
Namespace:      default
CreationTimestamp:  Thu, 29 Jun 2017 05:04:21 -0400
Labels:         name=mysql
Selector:       name=mysql
Replicas:       1 updated | 1 total | 1 available | 0 unavailable
StrategyType:       RollingUpdate
MinReadySeconds:    0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Conditions:
  Type      Status  Reason
  ----      ------  ------
  Available     True    MinimumReplicasAvailable
OldReplicaSets: <none>
NewReplicaSet:  mysql-478535978 (1/1 replicas created)
No events.
[root@ku8-1 tmp]# 
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22

describe命令能夠確認的信息

describe命令所能支持的類別如下:

類別
clusters (僅對federation apiservers有效)
componentstatuses (縮寫 cs)
configmaps (縮寫 cm)
daemonsets (縮寫 ds)
deployments (縮寫 deploy)
endpoints (縮寫 ep)
events (縮寫 ev)
horizontalpodautoscalers (縮寫 hpa)
ingresses (縮寫 ing)
jobs
limitranges (縮寫 limits)
namespaces (縮寫 ns)
networkpolicies
nodes (縮寫 no)
persistentvolumeclaims (縮寫 pvc)
persistentvolumes (縮寫 pv)
pods (縮寫 po)
podsecuritypolicies (縮寫 psp)
podtemplates
replicasets (縮寫 rs)
replicationcontrollers (縮寫 rc)
resourcequotas (縮寫 quota)
secrets
serviceaccounts (縮寫 sa)
services (縮寫 svc)
statefulsets
storageclasses
thirdpartyresources

kubectl logs

類似於docker logs,使用kubectl logs能夠取出pod中鏡像的log,也是故障排除時候的重要信息

[root@ku8-1 tmp]# kubectl get pods
NAME                         READY     STATUS    RESTARTS   AGE
mysql-478535978-1dnm2        1/1       Running   0          1h
sonarqube-3574384362-m7mdq   1/1       Running   0          1h
[root@ku8-1 tmp]# kubectl logs mysql-478535978-1dnm2
Initializing database
...
2017-06-29T09:04:37.081939Z 0 [Note] Event Scheduler: Loaded 0 events
2017-06-29T09:04:37.082097Z 0 [Note] mysqld: ready for connections.
Version: ‘5.7.16‘  socket: ‘/var/run/mysqld/mysqld.sock‘  port: 3306  MySQL Community Server (GPL)
[root@ku8-1 tmp]# 
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11

kubectl exec

exec命令用於到容器中執行一條命令,比如下述命令用於到mysql的鏡像中執行hostname命令

[root@ku8-1 tmp]# kubectl get pods
NAME                         READY     STATUS    RESTARTS   AGE
mysql-478535978-1dnm2        1/1       Running   0          1h
sonarqube-3574384362-m7mdq   1/1       Running   0          1h
[root@ku8-1 tmp]# kubectl exec mysql-478535978-1dnm2 hostname
mysql-478535978-1dnm2
[root@ku8-1 tmp]# 
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7

更為常用的方式則是登陸到pod中,在有條件的時候,進行故障發生時的現場確認,這種方式是最為直接有效和快速,但是對權限要求也較多。

[root@ku8-1 tmp]# kubectl exec -it mysql-478535978-1dnm2 sh
# hostname
mysql-478535978-1dnm2
# 
  • 1
  • 2
  • 3
  • 4

kubectl cp

用於pod和外部的文件交換,比如如下示例了如何在進行內外文件交換。

在pod中創建一個文件message.log

[root@ku8-1 tmp]# kubectl exec -it mysql-478535978-1dnm2 sh
# pwd
/
# cd /tmp
# echo "this is a message from `hostname`" >message.log
# cat message.log
this is a message from mysql-478535978-1dnm2
# exit
[root@ku8-1 tmp]#
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9

拷貝出來並確認

[[email protected] tmp]# kubectl cp mysql-478535978-1dnm2:/tmp/message.log message.log
tar: Removing leading `/‘ from member names
[[email protected] tmp]# cat message.log
this is a message from mysql-478535978-1dnm2
[[email protected] tmp]#
  • 1
  • 2
  • 3
  • 4
  • 5

更改message.log並拷貝回pod

[root@ku8-1 tmp]# echo "information added in `hostname`" >>message.log 
[root@ku8-1 tmp]# cat message.log 
this is a message from mysql-478535978-1dnm2
information added in ku8-1
[root@ku8-1 tmp]# kubectl cp message.log mysql-478535978-1dnm2:/tmp/message.log
[root@ku8-1 tmp]# 
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6

確認更改後的信息

[root@ku8-1 tmp]# kubectl exec mysql-478535978-1dnm2 cat /tmp/message.log
this is a message from mysql-478535978-1dnm2
information added in ku8-1
[root@ku8-1 tmp]#
  • 1
  • 2
  • 3
  • 4

kubectl attach

類似於docker attach的功能,用於取得實時的類似於kubectl logs的信息

[root@ku8-1 tmp]# kubectl get pods
NAME                         READY     STATUS    RESTARTS   AGE
mysql-478535978-1dnm2        1/1       Running   0          1h
sonarqube-3574384362-m7mdq   1/1       Running   0          1h
[root@ku8-1 tmp]# kubectl attach sonarqube-3574384362-m7mdq
If you don‘t see a command prompt, try pressing enter.
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7

kubectl cluster-info

使用cluster-info和cluster-info dump也能取出一些信息,尤其是你需要看整體的全部信息的時候一條命令一條命令的執行不如kubectl cluster-info dump來的快一些

[root@ku8-1 tmp]# kubectl cluster-info
Kubernetes master is running at http://localhost:8080

To further debug and diagnose cluster problems, use ‘kubectl cluster-info dump‘.
[root@ku8-1 tmp]# 
  • 1
  • 2
  • 3
  • 4
  • 5

總結

這篇文章中介紹了九個kubectl的常用命令,利用它們在故障確認和排查中非常有效。

http://www.woaipu.com/shops/zuzhuan/61406
http://www.znds.com/tv-967956-1-1.html
http://www.znds.com/tv-967958-1-1.html

http://www.woaipu.com/shops/zuzhuan/61406
http://www.znds.com/tv-967956-1-1.html
http://www.znds.com/tv-967958-1-1.html

Kubernetes之kubectl常用命令使用指南:2:故障排查