我是參考 Tony Bai 部落格進行k8s掛載ceph的儲存，但是發現最終pod的狀態一直是ContainerCreating

一、環境說明：

• Tony Bai 是把k8s 和 ceph都部署在那兩臺虛擬機器上
• 我的環境是k8s叢集和ceph儲存叢集分別部署在不同機器上的

ceph儲存叢集環境部署可以參考Tony Bai的，或者網上找很多教程，我這裡只是記錄k8s掛載到ceph塊儲存所遇到的問題。

二、配置檔案

# ceph-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
data:
  key: QVFEUGpCVlpnRWphREJBQUtMWFd5SVFsMzRaQ2JYMitFQW1wK2c9PQo=

##########################################
 

# ceph-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: ceph-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  rbd:
    monitors:
      - 192.168.100.81:6789
    pool: rbd
    image: ceph-image
    keyring: /etc/ceph/ceph.client.admin.keyring
    user: admin
    secretRef:
      name: ceph-secret
    fsType: ext4
    readOnly: false
  persistentVolumeReclaimPolicy: Recycle

##########################################
 

# ceph-pvc.yaml

kind: PersistentVolumeClaim

apiVersion: v1
metadata:
  name: ceph-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

##########################################
# ceph-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: ceph-pod1
spec:
  containers:
  - name: ceph-busybox1
    image: 192.168
 
.100.90:5000/duni/busybox:latest
    command: ["sleep", "600000"]
    volumeMounts:
    - name: ceph-vol1
      mountPath: /usr/share/busybox
      readOnly: false
  volumes:
  - name: ceph-vol1
    persistentVolumeClaim:
      claimName: ceph-claim

三、查詢掛載失敗的原因

檢視物件的狀態

$ kubectl get pv,pvc,pods
NAME         CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                REASON    AGE
pv/ceph-pv   1Gi        RWO           Recycle         Bound     default/ceph-claim             11s

NAME             STATUS    VOLUME    CAPACITY   ACCESSMODES   AGE
pvc/ceph-claim   Bound     ceph-pv   1Gi        RWO           10s

NAME                   READY     STATUS              RESTARTS   AGE
po/ceph-pod1           0/1       ContainerCreating   0          11s

發現ceph-pod1狀態一直是ContainerCreating

檢視pod的event

$ kubectl describe po/ceph-pod1

Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----            -------------   --------    ------      -------
  2m        2m      1   {default-scheduler }            Normal      Scheduled   Successfully assigned ceph-pod1 to duni-node1
  6s        6s      1   {kubelet duni-node1}            Warning     FailedMount Unable to mount volumes for pod "ceph-pod1_default(6656394a-37b6-11e7-b652-000c2932f92e)": timeout expired waiting for volumes to attach/mount for pod "ceph-pod1"/"default". list of unattached/unmounted volumes=[ceph-vol1]
  6s        6s      1   {kubelet duni-node1}            Warning     FailedSync  Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "ceph-pod1"/"default". list of unattached/unmounted volumes=[ceph-vol1]

又到ceph-pod1 所在k8s節點機上檢視kubelet日誌

$ journalctl -u  kubelet -f 

May 13 15:09:52 duni-node1 kubelet[5167]: I0513 15:09:52.650241    5167 operation_executor.go:802] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/e38290de-33a7-11e7-b6       52-000c2932f92e-default-token-91w6v" (spec.Name: "default-token-91w6v") pod "e38290de-33a7-11e7-b652-000c2932f92e" (UID: "e38290de-33a7-11e7-b652-000c2932f92e").
203299 May 13 15:10:15 duni-node1 kubelet[5167]: E0513 15:10:15.801855    5167 kubelet.go:1813] Unable to mount volumes for pod "ceph-pod1_default(ef4e99c4-37aa-11e7-b652-000c2932f92e)": t       imeout expired waiting for volumes to attach/mount for pod "ceph-pod1"/"default". list of unattached/unmounted volumes=[ceph-vol1]; skipping pod
203300 May 13 15:10:15 duni-node1 kubelet[5167]: E0513 15:10:15.801930    5167 pod_workers.go:184] Error syncing pod ef4e99c4-37aa-11e7-b652-000c2932f92e, skipping: timeout expired waiting        for volumes to attach/mount for pod "ceph-pod1"/"default". list of unattached/unmounted volumes=[ceph-vol1]
203301 May 13 15:10:17 duni-node1 kubelet[5167]: I0513 15:10:17.252663    5167 reconciler.go:299] MountVolume operation started for volume "kubernetes.io/secret/ddee5d45-3490-11e7-b652-000       c2932f92e-default-token-91w6v" (spec.Name: "default-token-91w6v") to pod "ddee5d45-3490-11e7-b652-000c2932f92e" (UID: "ddee5d45-3490-11e7-b652-000c2932f92e"). Volume is already moun       ted to pod, but remount was requested.

四、解決方法

在k8s節點機上安裝ceph common

yum install ceph-common

刪掉 ceph-pod1 重新執行，等一會就看到狀態Running了