前言：

考慮到公司持續整合與docker容器技術實施已有一段時間，取得了不錯的效果，但對於裝置運維、系統隔離、裝置利用率和擴充套件性還有待提升，綜合目前比較成熟的微服務技術，打算把現有業務遷移到K8S叢集。

由於公司所有業務均部署在阿里雲上，最開始就調研了阿里雲自己提供的Kubernetes叢集，但後來還是放棄了，主要考慮幾方面：

• 阿里雲K8S叢集尚不成熟，使用的版本也相對較老，不能及時更新版本
• 阿里雲K8S叢集目前只支援多主多從結構，同時限定Master節點只能是3個，不能增減，這對於小型業務或者巨型業務均不適用
• 自建原生K8S叢集更有利於拓展和理解整體結構

接下來會詳細介紹在阿里雲搭建原生Kubernetes叢集的過程。

一、K8S叢集搭建

下面的實戰操作基於阿里雲的VPC網路，在4臺ECS上搭建K8S單主多從叢集，部署Gitlab，Gitlab的資料儲存在阿里雲NAS上，服務通過SLB暴露至外網

• 阿里雲VPC * 1
  • EIP * 2
  • NAT閘道器 * 1
  • 共享流量包 * 1
• 阿里雲ECS（無外網IP） * 4
• 阿里雲SLB * 4
• 阿里雲NAS * 1

1.1 VPC組網

對於VPC，新建交換機，目標網段用192.168.0.0/24，4臺ECS的內網IP分別設定為192.168.0.1 ~ 192.168.0.4

1.2 NAT閘道器與EIP打通網路

由於VPC網路內，所有的ECS沒有配置外網IP，所以這裡要配置NAT閘道器和彈性IP來打通外網和VPC的通訊。

• 開通一個NAT閘道器，並加入到VPC內
• 開通兩個EIP，一個用於DNAT(VPC訪問外網)，另一個用於SNAT(外網訪問EIP)
• 繫結EIP到NAT閘道器
• 配置DNAT(外網訪問VPC)
  • 我們有4臺ECS，每臺機器的22埠分別對映到EIP的不同埠上，如23301~23304，該埠用於SSH訪問ECS
  • 同時對映192.168.0.1的6443埠到EIP上，如對映至23443埠，該埠用於訪問K8S叢集的API，見第二章內容
• 配置SNAT(VPC訪問外網)

配置完成後，便可以使用繫結DNAT的EIP的對映埠通過SSH訪問ECS

1.3 使用Kubeasz部署K8S叢集

搭建K8S叢集相對比較簡單，使用kubeasz的AllinOne部署即可

• 修改hosts檔案，根據實際環境配置master、node、etc的ip
• 這裡將192.168.0.1設定為master，使用單主多從的方式
• 配置完成後重啟所有ECS

二、部署Gitlab實戰

2.1 K8S Dashboard

部署好集群后，我們可以使用DNAT的EIP，通過對映埠23443訪問K8S API和Dashboard

https://EIP:Port/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

• 進入後會要求輸入API的賬號密碼，與1.3章節hosts檔案裡配置的賬號密碼一致
• 通過賬號密碼驗證後可看到K8S Dashboard登入介面
• 令牌可在Master節點通過以下命令獲取kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

2.2 PV與PVC

K8S中的PV和PVC的概念這裡不再多提，引用官方的一段解釋：

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.

A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (e.g., can be mounted once read/write or many times read-only).

在Gitlab for Docker中，我們看到Volumes 有三個，如下表所示

所以我們也需要給Gitlab for K8S分配3個PV和PVC，這裡我們用到了阿里雲NAS

• 給NAS添加掛載點，選擇VPC網路和VPC的交換機
• 檢視掛載地址
• SSH登入Master節點，掛載NAS，並建立資料夾（ 注意PV的path必須已存在才可以成功建立，所以需要先在NAS中建立資料夾）

  mkdir /nas sudo mount -t nfs -o vers=4.0 xxx.xxx.nas.aliyuncs.com:/ /nas mkdir -p /gitlab/data mkdir -p /gitlab/logs mkdir -p /gitlab/config

• 編寫PV和PVC的YAML，根據實際需求替換server節點的NAS掛載地址配置以及storage大小配置

  apiVersion: v1  kind: Namespace  metadata:   name: gitlab   labels:   name: gitlab --- apiVersion: v1 kind: PersistentVolume metadata:  name: gitlab-data  labels:  release: gitlab-data  namespace: gitlab spec:  capacity:  storage: 500Gi  accessModes:  - ReadWriteMany  persistentVolumeReclaimPolicy: Retain  nfs:  path: /gitlab/data  server: xxx.xxx.nas.aliyuncs.com --- apiVersion: v1 kind: PersistentVolume metadata:  name: gitlab-config  labels:  release: gitlab-config  namespace: gitlab spec:  capacity:  storage: 1Gi  accessModes:  - ReadWriteMany  persistentVolumeReclaimPolicy: Retain  nfs:  path: /gitlab/config  server: xxx.xxx.nas.aliyuncs.com --- apiVersion: v1 kind: PersistentVolume metadata:  name: gitlab-log  labels:  release: gitlab-log  namespace: gitlab spec:  capacity:  storage: 1Gi  accessModes:  - ReadWriteMany  persistentVolumeReclaimPolicy: Retain  nfs:  path: /gitlab/log  server: xxx.xxx.nas.aliyuncs.com --- apiVersion: v1 kind: PersistentVolumeClaim metadata:  name: gitlab-data-claim  namespace: gitlab spec:  accessModes:  - ReadWriteMany  resources:   requests:  storage: 500Gi  selector:  matchLabels:  release: gitlab-data --- apiVersion: v1 kind: PersistentVolumeClaim metadata:  name: gitlab-config-claim  namespace: gitlab spec:  accessModes:  - ReadWriteMany  resources:   requests:  storage: 1Gi  selector:  matchLabels:  release: gitlab-config --- apiVersion: v1 kind: PersistentVolumeClaim metadata:  name: gitlab-log-claim  namespace: gitlab spec:  accessModes:  - ReadWriteMany  resources:   requests:  storage: 1Gi  selector:  matchLabels:  release: gitlab-log

2.3 K8S部署Gitlab

接下來補全Gitlab的Deployment和Service

apiVersion: apps/v1 kind: Deployment metadata:  name: gitlab  namespace: gitlab spec:   selector:  matchLabels:  app: gitlab  replicas: 1  strategy:  type: Recreate  template:  metadata:  labels:  app: gitlab  spec:  containers:  - image: gitlab/gitlab-ce:latest  name: gitlab  ports:  - containerPort: 80  name: gitlab-http  - containerPort: 443  name: gitlab-https  - containerPort: 22  name: gitlab-ssh  volumeMounts:  - name: gitlab-config  mountPath: /etc/gitlab  - name: gitlab-log  mountPath: /var/log/gitlab  - name: gitlab-data  mountPath: /var/opt/gitlab  volumes:  - name: gitlab-data  persistentVolumeClaim:  claimName: gitlab-data-claim  - name: gitlab-config  persistentVolumeClaim:  claimName: gitlab-config-claim  - name: gitlab-log  persistentVolumeClaim:  claimName: gitlab-log-claim --- kind: Service apiVersion: v1 metadata:  name: gitlab-service  labels:  app: gitlab-service  namespace: gitlab spec:  selector:  app: gitlab  ports:  - protocol: TCP  name: gitlab-https  port: 443  targetPort: 443  - protocol: TCP  name: gitlab-http  port: 80  targetPort: 80 --- kind: Service apiVersion: v1 metadata:  name: gitlab-ssh-service  labels:  app: gitlab-ssh-service  namespace: gitlab spec:  type: NodePort  selector:  app: gitlab  ports:  - protocol: TCP  name: gitlab-ssh  port: 22  targetPort: 22  nodePort: 30000

• 注意在Deployment中，開放了Gitlab Pod的80、443和22埠，用於Gitlab的HTTP、HTTPS和SSH的訪問
• 建立了2個Service，第一個只將80和443埠開放到Cluster IP上，第二個Service通過NodePort將22埠對映到NodeIp的30000埠上
• 我們將2.2章節PV與PVC中的相關程式碼和上面的程式碼合併，並命名成gitlab.yaml，上傳到Master節點，執行命令

  kubectl apply -f gitlab.yaml

• 接下來進入Gitlab的Pod，修改gitlab的域名，並啟用https訪問

  kubectl get pod --namespace=gitlab # 獲得gitlab pod名稱後 kubectl exec -it gitlab-xxxx-xxxx --namespace=gitlab /bin/bash # 進入pod後 vi /etc/gitlab/gitlab.rb # 修改external_url 'https://xxx.xxx.com'，儲存後退出 gitlab-ctl reconfigure exit

到這裡，配置與部署基本完成了，但我們還不能從外網訪問Gitlab，不過至少可以在叢集內驗證配置是否正確。

• 在Master節點檢視Service

  kubectl get svc --namespace=gitlab

  

  可以看到443和80埠已經開發給Cluster IP，同時22埠對映到了30000的NodePort上

• 通過curl命令檢視訪問結果

  curl https://10.68.88.97 --insecure

  這時返回一串包含redirect的字元，如下

  <html><body>You are being <a href="https://10.68.88.97/users/sign_in">redirected</a>.</body></html>

  表示服務已部署成功

• 如果有telnet客戶端，還可以驗證30000埠，在任何一個節點上執行任意一條命令

  telnet 192.168.0.1:30000 telnet 192.168.0.2:30000 telnet 192.168.0.3:30000 telnet 192.168.0.4:30000

2.4 使用Ingress-Nginx和阿里雲SLB暴露服務

K8S暴露服務的方法有3種：

• ClusterIP：叢集內可訪問，但外部不可訪問
• NodePort：通過NodeIP:NodePort方式可以在叢集內訪問，結合EIP或者雲服務VPC負載均衡也可在叢集外訪問，但開放NodePort一方面不安全，另一方面隨著應用的增多不方便管理
• LoadBalancer：某些雲服務提供商會直接提供LoadBalancer模式，將服務對接到負載均衡，其原理是基於kubernetes的controller做二次開發，並整合到K8S叢集，使得叢集可以與雲服務SDK互動

由於我們的叢集搭建在阿里雲上，所以第一時間想到的是LoadBalancer方案，但很遺憾，沒辦法使用，原因如下：

• 阿里雲自己提供的Kubernetes叢集才可以使用LoadBalancer，但我們是自己在阿里雲上搭建的，不具備這個功能
• 阿里雲也有開源的alicloud-controller-manager，但長期沒有更新，同時網上現有的資料都過時很久，版本對應不上
• • https://github.com/AliyunContainerService/alicloud-controller-manager
  • https://blog.csdn.net/cleverfoxloving/article/details/79186574
  • https://yq.aliyun.com/articles/69580

迴歸到NodePort的方式，目前已有的解決方案是基於Ingress的幾款工具，如Ingress-Nginx、Traefik-Ingress，他們的對比如下（注意，目前的版本是IngressNginx 0.13.0、Traefik 1.6）：

• IngressNginx和Traefik都是通過hostname方式反向代理已解決埠暴露問題
• IngressNginx依賴於Nginx，功能更多；Traefik不依賴Nginx，所以更輕量
• IngressNginx支援4層和7層LB，但4層也不好用，Traefik只支援7層代理
• 目前網上關於IngressNginx的文章都是beta 0.9.X版本的資訊，而IngressNginx在Github的地址也變化了，直接由Kubernetes維護，所以網上的文章基本沒參考性，還需看官方文件，但是官方文件極其混亂和不完善！！！ 後面會有填坑指南。

最終我們還是選擇了Ingress-Nginx，結合阿里雲SLB，最終的拓撲圖如下所示：

其原理是：

• 通過Service的ClusterIP負載Pod
• 通過Ingress-Nginx監聽Ingress配置，動態生成Nginx，並將Nginx暴露到23456的NodePort
• 通過阿里雲SLB監聽所有節點的23456埠

接下來看詳細步驟。

2.4.1 部署Ingress-Nginx

主要參考https://kubernetes.github.io/ingress-nginx/deploy/，並做一些小調整

• 替換gcr.io的映象為阿里雲映象
• 暴露服務埠到NodePort 23456
• 整合成一個ingress-nginx.yaml

apiVersion: v1 kind: Namespace metadata:  name: ingress-nginx --- apiVersion: extensions/v1beta1 kind: Deployment metadata:  name: default-http-backend  labels:  app: default-http-backend  namespace: ingress-nginx spec:  replicas: 1  selector:  matchLabels:  app: default-http-backend  template:  metadata:  labels:  app: default-http-backend  spec:  terminationGracePeriodSeconds: 60  containers:  - name: default-http-backend  # Any image is permissible as long as:  # 1. It serves a 404 page at /  # 2. It serves 200 on a /healthz endpoint  image: registry.cn-shenzhen.aliyuncs.com/heygears/defaultbackend:1.4  livenessProbe:  httpGet:  path: /healthz  port: 8080  scheme: HTTP  initialDelaySeconds: 30  timeoutSeconds: 5  ports:  - containerPort: 8080  resources:  limits:  cpu: 10m  memory: 20Mi  requests:  cpu: 10m  memory: 20Mi --- ​ apiVersion: v1 kind: Service metadata:  name: default-http-backend  namespace: ingress-nginx  labels:  app: default-http-backend spec:  ports:  - port: 80  targetPort: 8080  selector:  app: default-http-backend --- kind: ConfigMap apiVersion: v1 metadata:  name: nginx-configuration  namespace: ingress-nginx  labels:  app: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata:  name: tcp-services  namespace: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata:  name: udp-services  namespace: ingress-nginx --- apiVersion: v1 kind: ServiceAccount metadata:  name: nginx-ingress-serviceaccount  namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata:  name: nginx-ingress-clusterrole rules:  - apiGroups:  - ""  resources:  - configmaps  - endpoints  - nodes  - pods  - secrets  verbs:  - list  - watch  - apiGroups:  - ""  resources:  - nodes  verbs:  - get  - apiGroups:  - ""  resources:  - services  verbs:  - get  - list  - watch  - apiGroups:  - "extensions"  resources:  - ingresses  verbs:  - get  - list  - watch  - apiGroups:  - ""  resources:  - events  verbs:  - create  - patch  - apiGroups:  - "extensions"  resources:  - ingresses/status  verbs:  - update --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata:  name: nginx-ingress-role  namespace: ingress-nginx rules:  - apiGroups:  - ""  resources:  - configmaps  - pods  - secrets  - namespaces  verbs:  - get  - apiGroups:  - ""  resources:  - configmaps  resourceNames:  # Defaults to "<election-id>-<ingress-class>"  # Here: "<ingress-controller-leader>-<nginx>"  # This has to be adapted if you change either parameter  # when launching the nginx-ingress-controller.  - "ingress-controller-leader-nginx"  verbs:  - get  - update  - apiGroups:  - ""  resources:  - configmaps  verbs:  - create  - apiGroups:  - ""  resources:  - endpoints  verbs:  - get ​ --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata:  name: nginx-ingress-role-nisa-binding  namespace: ingress-nginx roleRef:  apiGroup: rbac.authorization.k8s.io  kind: Role  name: nginx-ingress-role subjects:  - kind: ServiceAccount  name: nginx-ingress-serviceaccount  namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata:  name: nginx-ingress-clusterrole-nisa-binding roleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: nginx-ingress-clusterrole subjects:  - kind: ServiceAccount  name: nginx-ingress-serviceaccount  namespace: ingress-nginx --- apiVersion: extensions/v1beta1 kind: Deployment metadata:  name: nginx-ingress-controller  namespace: ingress-nginx  spec:  replicas: 1  selector:  matchLabels:  app: ingress-nginx  template:  metadata:  labels:  app: ingress-nginx  annotations:  prometheus.io/port: '10254'  prometheus.io/scrape: 'true'  spec:  serviceAccountName: nginx-ingress-serviceaccount  containers:  - name: nginx-ingress-controller  image: registry.cn-shenzhen.aliyuncs.com/heygears/nginx-ingress-controller:0.13.0  args:  - /nginx-ingress-controller  - --default-backend-service=$(POD_NAMESPACE)/default-http-backend  - --configmap=$(POD_NAMESPACE)/nginx-configuration  - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services  - --udp-services-configmap