ingress高可用--使用DaemonSet方式部署ingress-nginx
阿新 • • 發佈：2018-12-01
前言
為了配置kubernetes中的ingress的高可用，對於kubernetes叢集以外只暴露一個訪問入口，需要使用keepalived排除單點問題。需要使用daemonset方式將ingress-controller部署在邊緣節點上。
邊緣節點
首先解釋下什麼叫邊緣節點（Edge Node），所謂的邊緣節點即叢集內部用來向叢集外暴露服務能力的節點，叢集外部的服務通過該節點來呼叫叢集內部的服務，邊緣節點是叢集內外交流的一個Endpoint。
邊緣節點要考慮兩個問題
- 邊緣節點的高可用，不能有單點故障，否則整個kubernetes叢集將不可用
- 對外的一致暴露埠，即只能有一個外網訪問IP和埠
架構
為了滿足邊緣節點的以上需求，我們使用keepalived來實現。
在Kubernetes中添加了ingress後，在DNS中新增A記錄，域名為你的ingress中host的內容，IP為你的keepalived的VIP，這樣叢集外部就可以通過域名來訪問你的服務，也解決了單點故障。
選擇Kubernetes的三個node作為邊緣節點，並安裝keepalived。
安裝keepalived服務
yum install -y keepalived
node1的keealived配置
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id edgenode
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 88
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.40.0.109 dev eth0 label eth0:1
}
}
node2 keepalive配置
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id edgenode
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 88
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.40.0.109 dev eth0 label eth0:1
}
}
node3 keepalived配置
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id edgenode
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 88
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.40.0.109 dev eth0 label eth0:1
}
}
設定keepalived開機自啟動
systemctl enable keepalived.service
啟動keepalived服務
systemctl start keepalived.service
安裝ingress-nginx
給邊緣節點打標籤
kubectl label nodes 10.40.0.105 edgenode=true kubectl label nodes 10.40.0.106 edgenode=true kubectl label nodes 10.40.0.107 edgenode=true
檢視node標籤
[[email protected] ingress]# kubectl get node --show-labels NAME STATUS ROLES AGE VERSION LABELS 10.40.0.105 Ready <none> 25d v1.12.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,edgenode=true,kubernetes.io/hostname=10.40.0.105 10.40.0.106 Ready <none> 25d v1.12.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,edgenode=true,env_role=dev,kubernetes.io/hostname=10.40.0.106 10.40.0.107 Ready <none> 17d v1.12.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,edgenode=true,kubernetes.io/hostname=10.40.0.
將原來ingress的yaml檔案由deployment改為daemonset
cat ingress-daemonset.yaml
apiVersion: extensions/v1beta1 #kind: Deployment kind: DaemonSet metadata: name: nginx-ingress-controller namespace: ingress-nginx spec: #replicas: 3 #selector: #matchLabels: #app: ingress-nginx template: metadata: labels: app: ingress-nginx annotations: prometheus.io/port: '10254' prometheus.io/scrape: 'true' spec: serviceAccountName: nginx-ingress-serviceaccount hostNetwork: true nodeSelector: edgenode: 'true' containers: - name: nginx-ingress-controller image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.20.0 args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1
namespace部署檔案
cat namespace.yaml
apiVersion: v1 kind: Namespace metadata: name: ingress-nginx
configmap部署檔案
cat configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rbac部署檔案
cat rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
tcp-service-configmap
cat tcp-services-configmap.yaml kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: ingress-nginx
udp-servcie-configmap
cat udp-services-configmap.yaml kind: ConfigMap apiVersion: v1 metadata: name: udp-services namespace: ingress-nginx
default-backend
cat default-backend.yaml
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: default-http-backend labels: app: default-http-backend namespace: ingress-nginx spec: replicas: 1 template: metadata: labels: app: default-http-backend spec: terminationGracePeriodSeconds: 60 containers: - name: default-http-backend # Any image is permissable as long as: # 1. It serves a 404 page at / # 2. It serves 200 on a /healthz endpoint image: registry.cn-hangzhou.aliyuncs.com/google_containers/defaultbackend:1.4 livenessProbe: httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 ports: - containerPort: 8080 resources: limits: cpu: 10m memory: 20Mi requests: cpu: 10m memory: 20Mi --- apiVersion: v1 kind: Service metadata: name: default-http-backend namespace: ingress-nginx labels: app: default-http-backend spec: ports: - port: 80 targetPort: 8080 selector: app: default-http-backend
部署
kubectl create -f namespace.yaml kubectl create -f configmap.yaml kubectl create -f rbac.yaml kubectl create -f default-backend.yaml kubectl create -f tcp-services-configmap.yaml kubectl create -f udp-services-configmap.yaml kubectl create -f ingress-daemonset.yaml
檢視ingress-controller
[[email protected] ingress]# kubectl get ds -n ingress-nginx NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE nginx-ingress-controller 3 3 3 3 3 edgenode=true 57m
[[email protected] ingress]# kubectl get pods -n ingress-nginx -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE default-http-backend-86569b9d95-x4bsn 1/1 Running 12 24d 172.17.65.6 10.40.0.105 <none> nginx-ingress-controller-5b7xg 1/1 Running 0 58m 10.40.0.105 10.40.0.105 <none> nginx-ingress-controller-b5mxc 1/1 Running 0 58m 10.40.0.106 10.40.0.106 <none> nginx-ingress-controller-t5n5k 1/1 Running 0 58m 10.40.0.107 10.40.0.107 <none>