2. 程式人生 > >kubernetes實戰(九)：k8s叢集動態儲存管理GlusterFS及容器化GlusterFS擴容

kubernetes實戰(九)：k8s叢集動態儲存管理GlusterFS及容器化GlusterFS擴容

阿新 • • 發佈：2018-12-21

1、準備工作

　　所有節點安裝GFS客戶端

yum install glusterfs glusterfs-fuse -y

　　如果不是所有節點要部署GFS管理服務，就在需要部署的節點上打上標籤

[[email protected] ~]# kubectl label node k8s-node01 storagenode=glusterfs
node/k8s-node01 labeled
[[email protected]-master01 ~]# kubectl label node k8s-node02 storagenode=glusterfs
node/k8s-node02 labeled
[
 
[email protected]-master01 ~]# kubectl label node k8s-master01 storagenode=glusterfs
node/k8s-master01 labeled

2、建立GFS管理服務容器叢集

　　本文采用容器化方式部署GFS，公司如有GFS叢集可直接使用。

　　GFS已Daemonset的方式進行部署，保證每臺需要部署GFS管理服務的Node上都執行一個GFS管理服務。

　　下載相關檔案：

wget https://github.com/heketi/heketi/releases/download/v7.0.0/heketi-client-v7.0.0.linux.amd64.tar.gz

　　建立叢集：

[[email protected]8s-master01 kubernetes]# kubectl create -f glusterfs-daemonset.json 
daemonset.extensions/glusterfs created
[[email protected]-master01 kubernetes]# pwd
/root/heketi-client/share/heketi/kubernetes

　　注意1：此時採用的為預設的掛載方式，可使用其他磁碟當做GFS的工作目錄。

　　注意2：此時建立的namespace為預設的default，按需更改

　　檢視pods

[[email protected] kubernetes]# kubectl get pods -l glusterfs-node=daemonset
NAME              READY     STATUS    RESTARTS   AGE
glusterfs-5npwn   1/1       Running   0          1m
glusterfs-bd5dx   1/1       Running   0          1m...

3、建立Heketi服務

　　Heketi是一個提供RESTful API管理GFS卷的框架，並能夠在K8S、OpenShift、OpenStack等雲平臺上實現動態儲存資源供應，支援GFS多叢集管理，便於管理員對GFS進行操作。

　　建立Heketi的ServiceAccount物件：

[[email protected] kubernetes]# cat heketi-service-account.json 
{
  "apiVersion": "v1",
  "kind": "ServiceAccount",
  "metadata": {
    "name": "heketi-service-account"
  }
}
[[email protected]-master01 kubernetes]# kubectl create -f heketi-service-account.json 
serviceaccount/heketi-service-account created
[[email protected]-master01 kubernetes]# pwd
/root/heketi-client/share/heketi/kubernetes
[[email protected]-master01 kubernetes]# kubectl get sa
NAME                     SECRETS   AGE
default                  1         13d
heketi-service-account   1         <invalid>

　　建立Heketi對應的許可權和secret

[[email protected] kubernetes]# kubectl create clusterrolebinding heketi-gluster-admin --clusterrole=edit --serviceaccount=default:heketi-service-account
clusterrolebinding.rbac.authorization.k8s.io/heketi-gluster-admin created
[[email protected]-master01 kubernetes]# kubectl create secret generic heketi-config-secret --from-file=./heketi.json
secret/heketi-config-secret created

　　初始化部署Heketi

[[email protected] kubernetes]# kubectl create -f heketi-bootstrap.json 
secret/heketi-db-backup created
service/heketi created
deployment.extensions/heketi created
[[email protected]-master01 kubernetes]# pwd
/root/heketi-client/share/heketi/kubernetes

4、設定GFS叢集

[[email protected] heketi-client]# cp bin/heketi-cli /usr/local/bin/
[[email protected]-master01 heketi-client]# pwd
/root/heketi-client

[[email protected]-master01 heketi-client]# heketi-cli -v
heketi-cli v7.0.0

　　修改topology-sample，manage為GFS管理服務的Node節點主機名，storage為Node節點IP，devices為Node節點上的裸裝置

[[email protected] kubernetes]# cat topology-sample.json 
{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "k8s-master01"
              ],
              "storage": [
                "192.168.20.20"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdc",
              "destroydata": false
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "k8s-node01"
              ],
              "storage": [
                "192.168.20.30"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "k8s-node02"
              ],
              "storage": [
                "192.168.20.31"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        }
      ]
    }
  ]
}

　　檢視當前pod的ClusterIP

[[email protected] kubernetes]# kubectl get svc | grep heketi
deploy-heketi   ClusterIP   10.110.217.153   <none>        8080/TCP   26m
[[email protected]-master01 kubernetes]# export HEKETI_CLI_SERVER=http://10.110.217.153:8080

　　建立GFS叢集

[[email protected]master01 kubernetes]# heketi-cli topology load --json=topology-sample.json 
Creating cluster ... ID: a058723afae149618337299c84a1eaed
    Allowing file volumes on cluster.
    Allowing block volumes on cluster.
    Creating node k8s-master01 ... ID: 929909065ceedb59c1b9c235fc3298ec
        Adding device /dev/sdc ... OK
    Creating node k8s-node01 ... ID: 37409d82b9ef27f73ccc847853eec429
        Adding device /dev/sdb ... OK
    Creating node k8s-node02 ... ID: e3ab676be27945749bba90efb34f2eb9
        Adding device /dev/sdb ... OK

　　建立heketi持久化卷

yum install device-mapper* -y
[[email protected] kubernetes]# heketi-cli setup-openshift-heketi-storage
Saving heketi-storage.json
[[email protected]-master01 kubernetes]# ls
glusterfs-daemonset.json  heketi.json                  heketi-storage.json
heketi-bootstrap.json     heketi-service-account.json  README.md
heketi-deployment.json    heketi-start.sh              topology-sample.json
[[email protected]-master01 kubernetes]# kubectl create -f heketi-storage.json
secret/heketi-storage-secret created
endpoints/heketi-storage-endpoints created
service/heketi-storage-endpoints created
job.batch/heketi-storage-copy-job created

　　如果出現如下報錯：

[[email protected] kubernetes]# heketi-cli setup-openshift-heketi-storage
Error: /usr/sbin/modprobe failed: 1
  thin: Required device-mapper target(s) not detected in your kernel.
  Run `lvcreate --help' for more information.

　　解決辦法：所有節點執行modprobe dm_thin_pool

　　刪除中間產物

[[email protected] kubernetes]# kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"
pod "deploy-heketi-59f8dbc97f-5rf6s" deleted
service "deploy-heketi" deleted
service "heketi" deleted
deployment.apps "deploy-heketi" deleted
replicaset.apps "deploy-heketi-59f8dbc97f" deleted
job.batch "heketi-storage-copy-job" deleted
secret "heketi-storage-secret" deleted

　　建立持久化Heketi，持久化方式也可以選用其他方法。

[[email protected] kubernetes]# kubectl create -f heketi-deployment.json
service/heketi created
deployment.extensions/heketi created

　　待pod起來後，部署完成

[[email protected]master01 kubernetes]# kubectl get po
NAME                               READY     STATUS      RESTARTS   AGE
glusterfs-5npwn                    1/1       Running     0          3h
glusterfs-8zfzq                    1/1       Running     0          3h
glusterfs-bd5dx                    1/1       Running     0          3h
heketi-5cb5f55d9f-5mtqt            1/1       Running     0          2m

 　　檢視最新部署的持久化Heketi的svc，並更改HEKETI_CLI_SERVER的值

[[email protected]master01 kubernetes]# kubectl get svc
NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
heketi                     ClusterIP   10.111.95.240   <none>        8080/TCP   12h
heketi-storage-endpoints   ClusterIP   10.99.28.153    <none>        1/TCP      12h
kubernetes                 ClusterIP   10.96.0.1       <none>        443/TCP    14d
[[email protected]-master01 kubernetes]# export HEKETI_CLI_SERVER=http://10.111.95.240:8080
[[email protected] kubernetes]# curl http://10.111.95.240:8080/hello
Hello from Heketi

　　檢視GFS資訊

Hello from Heketi[[email protected] kubernetes]# heketi-cli topology info

Cluster Id: 5dec5676c731498c2bdf996e110a3e5e

    File:  true
    Block: true

    Volumes:

    Name: heketidbstorage
    Size: 2
    Id: 828dc2dfaa00b7213e831b91c6213ae4
    Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
    Mount: 192.168.20.31:heketidbstorage
    Mount Options: backup-volfile-servers=192.168.20.30,192.168.20.20
    Durability Type: replicate
    Replica: 3
    Snapshot: Disabled

        Bricks:
            Id: 16b7270d7db1b3cfe9656b64c2a3916c
            Path: /var/lib/heketi/mounts/vg_04290ec786dc7752a469b66f5e94458f/brick_16b7270d7db1b3cfe9656b64c2a3916c/brick
            Size (GiB): 2
            Node: fb181b0cef571e9af7d84d2ecf534585
            Device: 04290ec786dc7752a469b66f5e94458f

            Id: 828da093d9d78a2b1c382b13cc4da4a1
            Path: /var/lib/heketi/mounts/vg_80b61df999fcac26ebca6e28c4da8e61/brick_828da093d9d78a2b1c382b13cc4da4a1/brick
            Size (GiB): 2
            Node: d38819746cab7d567ba5f5f4fea45d91
            Device: 80b61df999fcac26ebca6e28c4da8e61

            Id: e8ef0e68ccc3a0416f73bc111cffee61
            Path: /var/lib/heketi/mounts/vg_82af8e5f2fb2e1396f7c9e9f7698a178/brick_e8ef0e68ccc3a0416f73bc111cffee61/brick
            Size (GiB): 2
            Node: 0f00835397868d3591f45432e432ba38
            Device: 82af8e5f2fb2e1396f7c9e9f7698a178


    Nodes:

    Node Id: 0f00835397868d3591f45432e432ba38
    State: online
    Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
    Zone: 1
    Management Hostnames: k8s-node02
    Storage Hostnames: 192.168.20.31
    Devices:
        Id:82af8e5f2fb2e1396f7c9e9f7698a178   Name:/dev/sdb            State:online    Size (GiB):39      Used (GiB):22      Free (GiB):17      
            Bricks:
                Id:e8ef0e68ccc3a0416f73bc111cffee61   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_82af8e5f2fb2e1396f7c9e9f7698a178/brick_e8ef0e68ccc3a0416f73bc111cffee61/brick

    Node Id: d38819746cab7d567ba5f5f4fea45d91
    State: online
    Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
    Zone: 1
    Management Hostnames: k8s-node01
    Storage Hostnames: 192.168.20.30
    Devices:
        Id:80b61df999fcac26ebca6e28c4da8e61   Name:/dev/sdb            State:online    Size (GiB):39      Used (GiB):22      Free (GiB):17      
            Bricks:
                Id:828da093d9d78a2b1c382b13cc4da4a1   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_80b61df999fcac26ebca6e28c4da8e61/brick_828da093d9d78a2b1c382b13cc4da4a1/brick

    Node Id: fb181b0cef571e9af7d84d2ecf534585
    State: online
    Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
    Zone: 1
    Management Hostnames: k8s-master01
    Storage Hostnames: 192.168.20.20
    Devices:
        Id:04290ec786dc7752a469b66f5e94458f   Name:/dev/sdc            State:online    Size (GiB):39      Used (GiB):22      Free (GiB):17      
            Bricks:
                Id:16b7270d7db1b3cfe9656b64c2a3916c   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_04290ec786dc7752a469b66f5e94458f/brick_16b7270d7db1b3cfe9656b64c2a3916c/brick

5、定義StorageClass

[[email protected] gfs]# cat storageclass-gfs-heketi.yaml 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gluster-heketi
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://10.111.95.240:8080"
  restauthenabled: "false"
[[email protected]-master01 gfs]# ku
[[email protected]-master01 gfs]# kubectl create -f storageclass-gfs-heketi.yaml 
storageclass.storage.k8s.io/gluster-heketi created

　　Provisioner引數須設定為"kubernetes.io/glusterfs"

　　resturl地址為API Server所在主機可以訪問到的Heketi服務的某個地址

6、定義PVC及測試Pod

[[email protected] gfs]# kubectl create -f pod-use-pvc.yaml 
pod/pod-use-pvc created
persistentvolumeclaim/pvc-gluster-heketi created
[[email protected]-master01 gfs]# cat pod-use-pvc.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: pod-use-pvc
spec:
  containers:
  - name: pod-use-pvc
    image: busybox
    command:
    - sleep
    - "3600"
    volumeMounts:
    - name: gluster-volume
      mountPath: "/pv-data"
      readOnly: false
  volumes:
  - name: gluster-volume
    persistentVolumeClaim:
      claimName: pvc-gluster-heketi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-gluster-heketi
spec:
  accessModes: [ "ReadWriteOnce" ]
  storageClassName: "gluster-heketi"
  resources:
    requests:
      storage: 1Gi

　　PVC定義一旦生成，系統便觸發Heketi進行相應的操作，主要為在GlusterFS叢集上建立brick，再建立並啟動一個volume

　　建立的pv及pvc如下

[[email protected] gfs]# kubectl get pv,pvc  | grep gluster

persistentvolume/pvc-4a8033e8-e7f7-11e8-9a09-000c293bfe27   1Gi        RWO            Delete           Bound       default/pvc-gluster-heketi                                gluster-heketi                          5m
persistentvolumeclaim/pvc-gluster-heketi   Bound     pvc-4a8033e8-e7f7-11e8-9a09-000c293bfe27   1Gi        RWO            gluster-heketi   5m

7、測試資料

　　進入到pod並建立檔案

[[email protected] /]# kubectl exec -ti pod-use-pvc -- /bin/sh
/ # cd /pv-data/
/pv-data # mkdir {1..10}
/pv-data # ls
{1..10}

　　宿主機掛載測試

# 檢視volume
[[email protected]-master01 /]# heketi-cli topology info

Cluster Id: 5dec5676c731498c2bdf996e110a3e5e

    File:  true
    Block: true

    Volumes:

    Name: vol_56d636b452d31a9d4cb523d752ad0891
    Size: 1
    Id: 56d636b452d31a9d4cb523d752ad0891
    Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
    Mount: 192.168.20.31:vol_56d636b452d31a9d4cb523d752ad0891
    Mount Options: backup-volfile-servers=192.168.20.30,192.168.20.20
    Durability Type: replicate
    Replica: 3
    Snapshot: Enabled
...
...# 或者使用volume list檢視

　[[email protected] mnt]# heketi-cli volume list  Id:56d636b452d31a9d4cb523d752ad0891 Cluster:5dec5676c731498c2bdf996e110a3e5e Name:vol_56d636b452d31a9d4cb523d752ad0891  Id:828dc2dfaa00b7213e831b91c6213ae4 Cluster:5dec5676c731498c2bdf996e110a3e5e Name:heketidbstorage  [[email protected] mnt]#

　　vol_56d636b452d31a9d4cb523d752ad0891為volume Name，Mount: 192.168.20.31:vol_56d636b452d31a9d4cb523d752ad0891，掛載方式。

　　掛載檢視資料

[[email protected] /]# mount -t glusterfs 192.168.20.31:vol_56d636b452d31a9d4cb523d752ad0891  /mnt/
[[email protected]-master01 /]# cd /mnt/
[[email protected]-master01 mnt]# ls
{1..10}

8、測試Deployments

[[email protected] gfs]# cat nginx-gluster.yaml 
apiVersion: extensions/v1beta1 
kind: Deployment 
metadata: 
  name: nginx-gfs
spec: 
  replicas: 2
  template: 
    metadata: 
      labels: 
        name: nginx 
    spec: 
      containers: 
        - name: nginx 
          image: nginx 
          imagePullPolicy: IfNotPresent
          ports: 
            - containerPort: 80
          volumeMounts:
            - name: nginx-gfs-html
              mountPath: "/usr/share/nginx/html"
            - name: nginx-gfs-conf
              mountPath: "/etc/nginx/conf.d"
      volumes:
      - name: nginx-gfs-html
        persistentVolumeClaim:
          claimName: glusterfs-nginx-html
      - name: nginx-gfs-conf
        persistentVolumeClaim:
          claimName: glusterfs-nginx-conf
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: glusterfs-nginx-html
spec:
  accessModes: [ "ReadWriteMany" ]
  storageClassName: "gluster-heketi"
  resources:
    requests:
      storage: 0.5Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: glusterfs-nginx-conf
spec:
  accessModes: [ "ReadWriteMany" ]
  storageClassName: "gluster-heketi"
  resources:
    requests:
      storage: 0.1Gi
[[email protected]-master01 gfs]# kubectl get po,pvc,pv | grep nginx
pod/nginx-gfs-77c758ccc-2hwl6          1/1       Running             0          4m
pod/nginx-gfs-77c758ccc-kxzfz          0/1       ContainerCreating   0          3m

persistentvolumeclaim/glusterfs-nginx-conf   Bound     pvc-f40c5d4b-e800-11e8-8a89-000c293ad492   1Gi        RWX            gluster-heketi   2m
persistentvolumeclaim/glusterfs-nginx-html   Bound     pvc-f40914f8-e800-11e8-8a89-000c293ad492   1Gi        RWX            gluster-heketi   2m

persistentvolume/pvc-f40914f8-e800-11e8-8a89-000c293ad492   1Gi        RWX            Delete           Bound       default/glusterfs-nginx-html                              gluster-heketi                          4m
persistentvolume/pvc-f40c5d4b-e800-11e8-8a89-000c293ad492   1Gi        RWX            Delete           Bound       default/glusterfs-nginx-conf                              gluster-heketi                          4m

　　檢視掛載情況

[[email protected] gfs]# kubectl exec -ti nginx-gfs-77c758ccc-2hwl6 -- df -Th
Filesystem                                         Type            Size  Used Avail Use% Mounted on
overlay                                            overlay          86G  6.6G   80G   8% /
tmpfs                                              tmpfs           7.8G     0  7.8G   0% /dev
tmpfs                                              tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/centos-root                            xfs              86G  6.6G   80G   8% /etc/hosts
shm                                                tmpfs            64M     0   64M   0% /dev/shm
192.168.20.20:vol_b9c68075c6f20438b46db892d15ed45a fuse.glusterfs 1014M   43M  972M   5% /etc/nginx/conf.d
192.168.20.20:vol_32146a51be9f980c14bc86c34f67ebd5 fuse.glusterfs 1014M   43M  972M   5% /usr/share/nginx/html
tmpfs                                              tmpfs           7.8G   12K  7.8G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs

　　掛載並建立index.html

[[email protected] gfs]# mount -t glusterfs 192.168.20.20:vol_32146a51be9f980c14bc86c34f67ebd5 /mnt/
[[email protected]-master01 gfs]# cd /mnt/
[[email protected]-master01 mnt]# ls
[[email protected]-master01 mnt]# echo "test" > index.html
[[email protected]-master01 mnt]# kubectl exec -ti nginx-gfs-77c758ccc-2hwl6 -- cat /usr/share/nginx/html/index.html
test

　　擴容nginx

[[email protected] ~]# kubectl get deploy
NAME        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
heketi      1         1         1            1           14h
nginx-gfs   2         2         2            2           23m
[[email protected]-master01 ~]# kubectl scale deploy nginx-gfs --replicas 3
deployment.extensions/nginx-gfs scaled
[[email protected]-master01 ~]# kubectl get po
NAME                               READY     STATUS      RESTARTS   AGE
glusterfs-5npwn                    1/1       Running     0          18h
glusterfs-8zfzq                    1/1       Running     0          17h
glusterfs-bd5dx                    1/1       Running     0          18h
heketi-5cb5f55d9f-5mtqt            1/1       Running     0          14h
nginx-gfs-77c758ccc-2hwl6          1/1       Running     0          11m
nginx-gfs-77c758ccc-6fphl          1/1       Running     0          8m
nginx-gfs-77c758ccc-kxzfz          1/1       Running     0          10m

　　檢視檔案內容

　　[[email protected] ~]# kubectl exec -ti nginx-gfs-77c758ccc-6fphl -- cat /usr/share/nginx/html/index.html　　test

9、擴容GlusterFS

9.1新增磁碟至已存在的node節點

　　基於上述節點，假設在k8s-node02上增加磁碟

　　檢視k8s-node02部署的pod name及IP

[[email protected] ~]# kubectl get po -o wide -l glusterfs-node
NAME              READY     STATUS    RESTARTS   AGE       IP              NODE
glusterfs-5npwn   1/1       Running   0          20h       192.168.20.31   k8s-node02
glusterfs-8zfzq   1/1       Running   0          20h       192.168.20.20   k8s-master01
glusterfs-bd5dx   1/1       Running   0          20h       192.168.20.30   k8s-node01

　　在node02上確認新新增的碟符

Disk /dev/sdc: 42.9 GB, 42949672960 bytes, 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

　　使用heketi-cli檢視cluster ID和所有node ID

[[email protected] ~]# heketi-cli cluster info
Error: Cluster id missing
[[email protected]-master01 ~]# heketi-cli cluster list
Clusters:
Id:5dec5676c731498c2bdf996e110a3e5e [file][block]
[[email protected]-master01 ~]# heketi-cli cluster info 5dec5676c731498c2bdf996e110a3e5e
Cluster id: 5dec5676c731498c2bdf996e110a3e5e
Nodes:
0f00835397868d3591f45432e432ba38
d38819746cab7d567ba5f5f4fea45d91
fb181b0cef571e9af7d84d2ecf534585
Volumes:
32146a51be9f980c14bc86c34f67ebd5
56d636b452d31a9d4cb523d752ad0891
828dc2dfaa00b7213e831b91c6213ae4
b9c68075c6f20438b46db892d15ed45a
Block: true

File: true

　　找到對應的k8s-node02的node ID

[[email protected] ~]# heketi-cli node info 0f00835397868d3591f45432e432ba38
Node Id: 0f00835397868d3591f45432e432ba38
State: online
Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
Zone: 1
Management Hostname: k8s-node02
Storage Hostname: 192.168.20.31
Devices:
Id:82af8e5f2fb2e1396f7c9e9f7698a178   Name:/dev/sdb            State:online    Size (GiB):39      Used (GiB):25      Free (GiB):14      Bricks:4

　　新增磁碟至GFS叢集的node02

[[email protected] ~]# heketi-cli device add --name=/dev/sdc --node=0f00835397868d3591f45432e432ba38
Device added successfully

　　檢視結果

[[email protected] ~]# heketi-cli node info 0f00835397868d3591f45432e432ba38
Node Id: 0f00835397868d3591f45432e432ba38
State: online
Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
Zone: 1
Management Hostname: k8s-node02
Storage Hostname: 192.168.20.31
Devices:
Id:5539e74bc2955e7c70b3a20e72c04615   Name:/dev/sdc            State:online    Size (GiB):39      Used (GiB):0       Free (GiB):39      Bricks:0       
Id:82af8e5f2fb2e1396f7c9e9f7698a178   Name:/dev/sdb            State:online    Size (GiB):39      Used (GiB):25      Free (GiB):14      Bricks:4

9.2 新增新節點

　　假設將k8s-master03，IP為192.168.20.22的加入glusterfs叢集，並將該節點的/dev/sdc加入到叢集

　　加標籤，之後會自動建立pod

[[email protected] kubernetes]# kubectl label node k8s-master03 storagenode=glusterfs
node/k8s-master03 labeled
[[email protected]-master01 kubernetes]# kubectl  get pod -owide -l glusterfs-node
NAME              READY     STATUS              RESTARTS   AGE       IP              NODE
glusterfs-5npwn   1/1       Running             0          21h       192.168.20.31   k8s-node02
glusterfs-8zfzq   1/1       Running             0          21h       192.168.20.20   k8s-master01
glusterfs-96w74   0/1       ContainerCreating   0          2m        192.168.20.22   k8s-master03
glusterfs-bd5dx   1/1       Running             0          21h       192.168.20.30   k8s-node01

　　在任意節點執行peer probe

[[email protected] kubernetes]# kubectl exec -ti glusterfs-5npwn -- gluster peer probe 192.168.20.22
peer probe: success.

　　將新節點加入到glusterfs叢集中

[[email protected] kubernetes]# heketi-cli cluster list
Clusters:
Id:5dec5676c731498c2bdf996e110a3e5e [file][block]
[[email protected]-master01 kubernetes]# heketi-cli node add --zone=1 --cluster=5dec5676c731498c2bdf996e110a3e5e --management-host-name=k8s-master03 --storage-host-name=192.168.20.22
Node information:
Id: 150bc8c458a70310c6137e840619758c
State: online
Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
Zone: 1
Management Hostname k8s-master03
Storage Hostname 192.168.20.22

　　將新節點的磁碟加入到叢集中

[[email protected] kubernetes]# heketi-cli device add --name=/dev/sdc --node=150bc8c458a70310c6137e840619758c
Device added successfully

　　驗證

[[email protected] kubernetes]# heketi-cli node list
Id:0f00835397868d3591f45432e432ba38    Cluster:5dec5676c731498c2bdf996e110a3e5e
Id:150bc8c458a70310c6137e840619758c    Cluster:5dec5676c731498c2bdf996e110a3e5e
Id:d38819746cab7d567ba5f5f4fea45d91    Cluster:5dec5676c731498c2bdf996e110a3e5e
Id:fb181b0cef571e9af7d84d2ecf534585    Cluster:5dec5676c731498c2bdf996e110a3e5e
[[email protected]-master01 kubernetes]# heketi-cli node info 150bc8c458a70310c6137e840619758c
Node Id: 150bc8c458a70310c6137e840619758c
State: online
Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
Zone: 1
Management Hostname: k8s-master03
Storage Hostname: 192.168.20.22
Devices:
Id:2d5210c19858fb7ea3f805e6f582ecce   Name:/dev/sdc            State:online    Size (GiB):39      Used (GiB):0       Free (GiB):39      Bricks:0

相關推薦

kubernetes實戰(九)：k8s叢集動態儲存管理GlusterFS及容器化GlusterFS擴容

1、準備工作 　　所有節點安裝GFS客戶端 yum install glusterfs glusterfs-fuse -y 　　如果不是所有節點要部署GFS管理服務，就在需要部署的節點上打上標籤 [[email protected] ~]# kubectl label node k8s-nod

kubernetes實戰(八)：k8s叢集安全機制RBAC

1、基本概念 　　RBAC(Role-Based Access Control，基於角色的訪問控制)在k8s v1.5中引入，在v1.6版本時升級為Beta版本，併成為kubeadm安裝方式下的預設選項，相對於其他訪問控制方式，新的RBAC具有如下優勢： 　　- 對叢集中的資源和非資源許可權均有完整的覆蓋 　

kubernetes實戰(四)：k8s持久化安裝rabbitmq叢集

1、下載檔案 https://github.com/dotbalo/k8s/tree/master/rabbitmq-cluster   2、建立namespace kubectl create namespace public-service 　　如果不使用publi

Kubernetes實戰(二)：k8s v1.11.1 prometheus traefik元件安裝及叢集測試

1、traefik 　　traefik：HTTP層路由，官網：http://traefik.cn/，文件：https://docs.traefik.io/user-guide/kubernetes/ 　　功能和nginx ingress類似。 　　相對於nginx ingress，traefix能夠實時跟Ku

kubernetes實戰(五)：k8s持久化安裝Redis Sentinel

1、PV建立 　　在nfs或者其他型別後端儲存建立pv，首先建立共享目錄 [[email protected] ~]# cat /etc/exports /k8s/redis-sentinel/0 *(rw,sync,no_subtree_check,no_root_squash) /k8

Kubernetes實戰(一)：k8s高可用安裝

1、部署架構 　　 　　詳細架構： 　　 2、基本配置 主機名IP地址說明元件 k8s-master01 ~ 03 192.168.20.20 ~ 22 master節點 * 3 keepalived、nginx、etcd、kubelet、kube-apiserver k8s-master-lb

kubernetes實戰(十)：k8s使用Helm安裝harbor

1、基本概念 　　對於複雜的應用中介軟體，需要設定映象執行的需求、環境變數，並且需要定製儲存、網路等設定，最後設計和編寫Deployment、Configmap、Service及Ingress等相關yaml配置檔案，再提交給kubernetes進行部署。這些複雜的過程將逐步被Helm應用包管理工具實現。 　　

kubernetes實戰(十三)：k8s使用helm持久化部署harbor整合openLDAP登入

[root@k8s-master01 harbor-helm-ldap]# helm install --name hb-ldap . --set externalDomain=harbor.xxx.net --wait --timeout 1500 --debug --namespace harbor

flink開發實戰一：flink叢集的搭建與部署

flink實戰案例一：flink叢集的搭建與部署 1.下載Flink壓縮包 下載地址：http://flink.apache.org/downloads.html 根據叢集環境的情況下載相應的版本的flink壓縮包 hadoop2.6，Scala2.11，所以下載：flink-1

邊學邊敲邊記之爬蟲系列(九)：Item+Pipeline資料儲存

一、寫在前面 好久沒更新了，快半個月了，也少有讀者催著更新，於是乎自己就拖啊，為公眾號出路想方設法，著實是有點迷失自我，廢話不多說了。 今天是爬蟲系列第9篇，上一篇Scrapy系統爬取伯樂線上中我們已經利用Scrapy獲取到了伯樂線上網站所有文章的基本資料，但我們沒有做儲存操作

機器學習實戰系列：sklearn 中模型儲存的兩種方法

一、 sklearn中提供了高效的模型持久化模組joblib，將模型儲存至硬碟。 from sklearn.externals import joblib #lr是一個LogisticRegression模型 joblib.dump(lr, 'lr.model') lr =

動態儲存管理_free崩潰

一、 生存期 從變數值的存在時間（生存期）來觀察。有的變數在程式執行的整個過程都是存在的，而有的變數則是在呼叫其所在函式才臨時分配儲存單元，而在函式呼叫結束後該儲存單元就馬上釋放了，變數就不存在了。 1.1 靜態儲存方式 是指在程式執行期間由系統分配固定的儲存空間的方式。 1.2

資料結構隨筆—動態儲存管理

動態儲存管理的基本問題是如何應使用者請求分配記憶體，如何回收那些使用者不在使用而釋放的記憶體。 經過一段時間的執行，有些使用者執行結束，它所佔用的記憶體區變成空閒塊，只就使整個記憶體區呈現出佔用快和空閒塊交錯的狀態，假如此時又有新的使用者進入系統請求分配記憶體，這時通常有兩

C語言 動態儲存管理

為什麼需要動態儲存管理 程式中需要用變數（各種簡單型別變數、陣列變數等）儲存被處理的資料和各種狀態資訊，變數在使用之前必須安排好儲存：放在哪裡、佔據多少儲存單元，等等，這個工作被稱作儲存分配。用機器語言寫程式時，所有儲存分配問題都需要人處理，這個工作瑣碎繁雜、很容易出錯。在用

資料結構之動態儲存管理(C語言)

一、 概述 1. 佔用塊 佔用塊：已分配給使用者使用的地址連續的記憶體區 可利用空間塊：未曾分配的地址連續的記憶體區 2. 動態儲存分配過程的記憶體狀態 系統執行一段時間後，有些程式的記憶體被釋放，造成了上圖(b)中的狀態。假如此時又有新

作業系統：虛擬頁式儲存管理（缺頁中斷、頁面置換演算法）

1、基本工作原理 1、基本工作原理 在程序開始執行之前，不是全部裝入頁面，而是裝入一個或者零個頁面，之後根據程序執行的需要，動態裝入其他頁面；當記憶體已滿，而又需要裝入 新的頁面時，則根據某種演算法淘

Tensorflow實戰7：實現AlexNet卷積神經網路及運算時間評測

之前已經介紹過了AlexNet的網路構建了，這次主要不是為了訓練資料，而是為了對每個batch的前饋（Forward）和反饋（backward）的平均耗時進行計算。在設計網路的過程中，分類的結果很重要，但是運算速率也相當重要。尤其是在跟蹤（Tracking）

C語言記憶體的動態儲存管理2-空閒連結串列

空閒連結串列三種結構形式： （1）所有請求的記憶體大小相同。這是一種最簡單的動態儲存管理方式。 對此，系統通常的做法是： a）系統啟動時，將記憶體按大小均分成若干個塊，並形成一個連結串列。 b）分配時，只需將連結串列中第一個節點分配給使用者即可，無需掃描整個連結串列。 c）回

K8s 叢集節點線上率達到 99.9% 以上，擴容效率提升 50%，我們做了這 3 個深度改造

點選下載《不一樣的 雙11 技術：阿里巴巴經濟體雲原生實踐》 本文節選自《不一樣的 雙11 技術：阿里巴巴經濟體雲原生實踐》一書，點選上方圖片即可下載！ 作者 | 張振（守辰）阿里云云原生應用平臺高階技術專家 導讀：2019 年阿里巴巴核心系統 100% 以雲原生方式上雲，完美地支撐了 雙11 大促。這

spring eureka叢集+spring boot 微服務，容器化部署示例

1.搭建eureka docker叢集 （1）建立eureka容器 參考如下指令，在不同的伺服器上建立eureka容器。（提前在docker hup 上下載好java:8的映象） docker run -d --name registry1 -p 8762:808