Kubernetes之服務發現ingress & ingress controller

阿新 • • 發佈：2018-12-13

Service雖然解決了服務發現和負載均衡的問題，但它在使用上還是有一些限制：

只支援4層負載均衡，沒有7層功能
對外訪問時，NodePort型別需要在外部搭建額外的負載均衡，而LoadBalancer要求kubernetes必須跑在支援的cloud provider上

一、ingress引入歷程

我們已經瞭解到Kubernetes暴露服務的方式目前只有三種：LoadBalancer Service、ExternalName、NodePort Service；那還有沒有更好的方式呢？我們先來了解一下以下幾個問題：

1、Pod漂移問題

我們知道Pod是有生命週期的，Pod宕機之後，其IP必然會改變，那我們如何把這個動態的Pod IP暴露出去呢？總所周知，我們可以藉助Kubernetes的Service機制，在Service中以標籤的形式選定一組帶有指定標籤的Pod，並監控和自動負載到它們的Pod IP，那麼我們向外只需要暴露Service IP就行了，這就是我們常使用的NodePort模式：即在每個節點上開啟一個埠，然後轉發到內部的Pod IP上。此時的訪問方式為：http: //nodeip:nodeport/ 在這裡插入圖片描述

2、埠管理問題

採用NodePort方式暴露服務面臨的問題是：服務一旦躲起來，NodePort在每個節點上開啟的埠會極其龐大，而且難以維護。想一想，鑑於nginx的能力，我們能夠使用一個nginx直接對內進行轉發呢？由於Pod與Pod之間是可以互相通訊的，而Pod是可以共享宿主機的網路名稱空間的，此時Pod上所監聽的就是Node上的埠。那這又該如何實現呢？簡單的實現就是使用DaemonSet在每個Node上監聽80埠，然後寫好規則，因為nginx外面綁定了宿主機80埠(就像NodePort)，本身又在叢集內，那麼向後直接轉發到相應Service IP就行了在這裡插入圖片描述

3. 域名分配及動態更新問題

用上面的方法，我們採用nginx Pod似乎已經解決了問題，但是當每次有新服務加入又該如何修改nginx配置呢？nginx可以通過虛擬主機域名區分不同的服務，而每個服務通過upstream進行定義不同的負載均衡池，再加上location進行負載均衡的反向代理，在日常使用中只需要修改nginx.conf即可實現，但是在k8s中又該如何實現這種方式排程呢？假設後端的服務開始只有ecshop，後面增加了bbs和member服務，那麼又該如何將這2個服務加入到nginx Pod進行排程呢？總不能每次手動改後者Rolling update前端nginx pod吧！所以k8s引入了ingress，ingress就是為了解決這些問題而生的，ingress包含兩大元件：ingress controller和ingress。

ingress 簡單理解就是你原來需要修改nginx配置，然後配置各種域名對應哪個Service，現在把這個動作抽象出來，變成一個Ingress物件，你可以用yaml建立，每次不要去修改nginx了，直接改yaml然後建立/更新就行了，那麼問題又來了：nginx該如何處理呢？
ingress controller 這個東西就是解決nginx如何處理這個問題的，ingress controller通過與kubernetes API互動，動態的去感知叢集中Ingress規則變化，然後讀取它，按照它自己的模板生成一段nginx配置，再寫到nginx Pod中，最後reload以下，工作流程如下圖

圖表說明：首先我們有一個外部的負載均衡器externalLB把請求排程到一個nodePort型別的Service(ingress-nginx)上，然後nodePort型別的Service(ingress-nginx)又把它排程到內部的叫做ingressController的Pod上，ingressCtroller根據ingress中的定義(虛擬主機還是URL)，每一組主機名或者URL都對應後端的Pod資源，並且用Service分組(圖中的<service>site1/<service>site2只是用來做分組用的)

二、Ingress Nginx部署

使用Ingress功能步驟：

安裝部署ingressCtroller Pod
部署ingress-nginx service
部署後端服務

部署ingress 從前面的描述我們知道，Ingress可以使用yaml的方式進行建立，從而得知Ingress也是標準的K8S資源，其定義的方式也可以使用kubectl explain進行檢視：

 [[email protected] manifests]# kubectl explain ingress
 KIND:     Ingress
 VERSION:  extensions/v1beta1

 DESCRIPTION:
 Ingress is a collection of rules that allow inbound connections to reach
 the endpoints defined by a backend. An Ingress can be configured to give
 services externally-reachable urls, load balance traffic, terminate SSL,
 offer name based virtual hosting etc.

 FIELDS:
 apiVersion	<string>
  APIVersion defines the versioned schema of this representation of an
  object. Servers should convert recognized schemas to the latest internal
  value, and may reject unrecognized values. More info:
  https://git.k8s.io/community/contributors/devel/api-conventions.md#resources

 kind	<string>
  Kind is a string value representing the REST resource this object
  represents. Servers may infer this from the endpoint the client submits
  requests to. Cannot be updated. In CamelCase. More info:
  https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds

 metadata	<Object>
  Standard object's metadata. More info:
  https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata

 spec	<Object>
  Spec is the desired state of the Ingress. More info:
  https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status

 status	<Object>
  Status is the current state of the Ingress. More info:
  https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status

2.1. 部署ingress controller

2.1.1. 下載ingress相關yaml檔案

[[email protected] ~]# mkdir ingress-nginx
[[email protected] ~]# cd ingress-nginx/
[[email protected] ingress-nginx]# for filr in namespace.yaml configmap.yaml rbac.yaml tcp-services-configmap.yaml with-rbac.yaml udp-services-configmap.yaml default-backend.yaml;do wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/$file;done
[[email protected] ingress-nginx]# ls -l | awk '{print $9}'

configmap.yaml						#用於為nginx從外部注入配置的
default-backend.yaml				#配置預設後端服務
namespace.yaml						#建立獨立的名稱空間
rbac.yaml							#用於叢集角色授權
tcp-services-configmap.yaml
udp-services-configmap.yaml
with-rbac.yaml

2.1.2. 建立ingress-nginx名稱空間

[[email protected] ingress-nginx]# cat namespace.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx

---
[[email protected] ingress-nginx]# kubectl apply -f namespace.yaml 
namespace/ingress-nginx created

2.1.3. 建立ingress controller的Pod

[[email protected] ingress-nginx]# kubectl apply -f .
configmap/nginx-configuration created
deployment.extensions/default-http-backend created
service/default-http-backend created
namespace/ingress-nginx configured
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole configured
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding configured
configmap/tcp-services created
configmap/udp-services created
deployment.extensions/nginx-ingress-controller created

此處遇到一個問題，新版本的Kubernetes在安裝部署中，需要從k8s.grc.io倉庫拉去所需映象檔案，但是由於國內防火牆問題導致無法正常拉去。辛虧docker.io倉庫對google的容器做了mirror，可以通過下列命令拉取相關映象：

[[email protected] ~]# docker pull mirrorgooglecontainers/defaultbackend-amd64:1.5
1.5: Pulling from mirrorgooglecontainers/defaultbackend-amd64
9ecb1e82bb4a: Pull complete 
Digest: sha256:d08e129315e2dd093abfc16283cee19eabc18ae6b7cb8c2e26cc26888c6fc56a
Status: Downloaded newer image for mirrorgooglecontainers/defaultbackend-amd64:1.5

[[email protected] ~]# docker tag mirrorgooglecontainers/defaultbackend-amd64:1.5 k8s.gcr.io/defaultbackend-amd64:1.5
[[email protected] ~]# docker image ls
REPOSITORY                                    TAG                 IMAGE ID            CREATED             SIZE
mirrorgooglecontainers/defaultbackend-amd64   1.5                 b5af743e5984        34 hours ago        5.13MB
k8s.gcr.io/defaultbackend-amd64               1.5                 b5af743e5984        34 hours ago        5.13MB

2.2. 部署ingress-nginx service

通過ingress-controller對外提供服務，現在還需要手動給ingress-controller建立一個servcie，接收叢集外部流量。

2.2.1 下載ingress-controller service的yaml檔案

[[email protected] ingress-nginx]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml
[[email protected] ingress-nginx]# cat service-nodeport.yaml 
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

2.2.2. 建立ingress-controller的service

[[email protected] ingress-nginx]# kubectl apply -f service-nodeport.yaml 
service/ingress-nginx created
[[email protected] ingress-nginx]# kubectl get svc -n ingress-nginx
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
default-http-backend   ClusterIP   10.107.114.219   <none>        80/TCP                       27m
ingress-nginx          NodePort    10.108.12.235    <none>        80:31875/TCP,443:32020/TCP   38s

2.3. 部署ingress

2.3.1. 檢視ingress的配置清單選項

[[email protected] ingress-nginx]# kubectl explain ingress.spec
KIND:     Ingress
VERSION:  extensions/v1beta1

RESOURCE: spec <Object>

DESCRIPTION:
     Spec is the desired state of the Ingress. More info:
     https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status

     IngressSpec describes the Ingress the user wishes to exist.

FIELDS:
   backend  <Object>     #定義後端有哪幾個主機
     A default backend capable of servicing requests that don't match any rule.
     At least one of 'backend' or 'rules' must be specified. This field is
     optional to allow the loadbalancer controller or defaulting logic to
     specify a global default.

   rules    <[]Object>    #定義規則
     A list of host rules used to configure the Ingress. If unspecified, or no
     rule matches, all traffic is sent to the default backend.

   tls  <[]Object>
     TLS configuration. Currently the Ingress only supports a single TLS port,
     443. If multiple members of this list specify different hosts, they will be
     multiplexed on the same port according to the hostname specified through
     the SNI TLS extension, if the ingress controller fulfilling the ingress
     supports SNI.

2.3.2. 編寫ingress的配置清單並建立ingress

[[email protected] ingress-nginx]# vim ingress-default-backend.yaml
apiVersion: extensions/v1beta1				#api版本
kind: Ingress								#清單型別
metadata:									#元資料
  name: ingress-default-backend				#ingress名稱
  namespace: ingress-nginx					#所屬名稱空間
  annotations:								#註解資訊
    kubernetes.io/ingress.class: "nginx"
spec:										#規格
  rules:									#定義後端轉發規則
  - host: default.backend.com				#通過域名進行轉發
    http:
      paths:
        - path:								#配置訪問路徑
          backend:							#配置後端服務
            serviceName: default-http-backend
            servicePort: 80 
[[email protected] ingress-nginx]# kubectl apply -f ingress-default-backend.yaml 
ingress.extensions/ingress-default-backend created

檢視ingress-default-backend的詳細資訊：

[[email protected] ingress-nginx]# kubectl describe ingress -n ingress-nginx ingress-default-backend
Name:             ingress-default-backend
Namespace:        ingress-nginx
Address:          
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  default.backend.com  
                          default-http-backend:80 (<none>)
Annotations:
  kubernetes.io/ingress.class:                       nginx
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-default-backend","namespace":"ingress-nginx"},"spec":{"rules":[{"host":"default.backend.com","http":{"paths":[{"backend":{"serviceName":"default-http-backend","servicePort":80},"path":null}]}}]}}

Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  9m    nginx-ingress-controller  Ingress ingress-nginx/ingress-default-backend

進入nginx-ingress-controller進行檢視是否注入了nginx配置：

[[email protected] ingress-nginx]# kubectl get pods -n ingress-nginx
NAME                                        READY     STATUS    RESTARTS   AGE
default-http-backend-7db7c45b69-znpxp       1/1       Running   0          1h
nginx-ingress-controller-6bd7c597cb-fgdz8   1/1       Running   0          1h
[[email protected] ingress-nginx]# kubectl exec -n ingress-nginx -ti nginx-ingress-controller-6bd7c597cb-fgdz8 -- /bin/sh
$ cat nginx.conf
....
....
## start server default.backend.com
server {
	server_name default.backend.com ;
	
	listen 80;
	
	set $proxy_upstream_name "-";
	
	location / {
		
		set $namespace      "ingress-nginx";
		set $ingress_name   "ingress-default-backend";
		set $service_name   "default-http-backend";
		set $service_port   "80";
		set $location_path  "/";
		
		rewrite_by_lua_block {
			
			balancer.rewrite()
			
		}
		
		log_by_lua_block {
			
			balancer.log()
			
			monitor.call()
		}
		
		port_in_redirect off;
		
		set $proxy_upstream_name "ingress-nginx-default-http-backend-80";
		
		client_max_body_size                    "1m";
		
		proxy_set_header Host                   $best_http_host;
		
		# Pass the extracted client certificate to the backend
		
		# Allow websocket connections
		proxy_set_header                        Upgrade           $http_upgrade;
		
		proxy_set_header                        Connection        $connection_upgrade;
		
		proxy_set_header X-Request-ID           $req_id;
		proxy_set_header X-Real-IP              $the_real_ip;
		
		proxy_set_header X-Forwarded-For        $the_real_ip;
		
		proxy_set_header X-Forwarded-Host       $best_http_host;
		proxy_set_header X-Forwarded-Port       $pass_port;
		proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
		
		proxy_set_header X-Original-URI         $request_uri;
		
		proxy_set_header X-Scheme               $pass_access_scheme;
		
		# Pass the original X-Forwarded-For
		proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
		
		# mitigate HTTPoxy Vulnerability
		# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
		proxy_set_header Proxy                  "";
		
		# Custom headers to proxied server
		
		proxy_connect_timeout                   5s;
		proxy_send_timeout                      60s;
		proxy_read_timeout                      60s;
		
		proxy_buffering                         "off";
		proxy_buffer_size                       "4k";
		proxy_buffers                           4 "4k";
		proxy_request_buffering                 "on";
		
		proxy_http_version                      1.1;
		
		proxy_cookie_domain                     off;
		proxy_cookie_path                       off;
		
		# In case of errors try the next upstream server before returning an error
		proxy_next_upstream                     error timeout;
		proxy_next_upstream_tries               3;
		
		proxy_pass http://upstream_balancer;
		
		proxy_redirect                          off;
		
	}
	
}
## end server default.backend.com
....
....

可以發現我們剛剛部署的ingress的的確確已經注入到nginx-ingress-controller的nginx配置中了

2.3.3. 通過ingress訪問服務

修改本地host檔案：

192.168.116.128 default.backend.com
192.168.116.129 default.backend.com

在這裡插入圖片描述

三、構建TLS站點

3.1. 新增tomcat服務

[[email protected] ingress-nginx]# vim tomcat-demo.yaml
apiVersion: v1
kind: Service
metadata:
  name: tomcat
  namespace: default
spec:
  selector:
    app: tomcat
    release: stable
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  - name: ajp
    port: 8009
    targetPort: 8009
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-demo
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tomcat
      release: stable
  template:
    metadata:
      labels:
        app: tomcat
        release: stable
    spec:
      containers:
      - name: tomcat
        image: tomcat:8.5.34-jre8-alpine
        ports:
        - name: http
       	  containerPort: 8080
        - name: ajp
      	  containerPort: 8009
[[email protected] ingress-nginx]# kubectl create -f tomcat-demo.yaml 
service/tomcat created
deployment.apps/tomcat-demo created
[[email protected] ingress-nginx]# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP             12d
tomcat       ClusterIP   10.108.64.162   <none>        8080/TCP,8009/TCP   8s
[[email protected] ingress-nginx]# kubectl get deployments
NAME          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
tomcat-demo   3         3         3            0           16s
[[email protected] ingress-nginx]# kubectl get pods
NAME                           READY     STATUS    RESTARTS   AGE
tomcat-demo-798cdf69b9-qk9pd   1/1       Running   0          7m
tomcat-demo-798cdf69b9-t642t   1/1       Running   0          7m
tomcat-demo-798cdf69b9-tlfq6   1/1       Running   0          7m

3.2. 準備證書並生成secret

準備證書：

[[email protected] ingress-nginx]# openssl genrsa -out tls.key 2048
Generating RSA private key, 2048 bit long modulus
............................+++
..................+++
e is 65537 (0x10001)
[[email protected] ingress-nginx]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.zwx.com

生成secret：

[[email protected] ingress-nginx]# kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key
secret/tomcat-ingress-secret created
[[email protected] ingress-nginx]# kubectl get secret
NAME                    TYPE                                  DATA      AGE
default-token-l9jws     kubernetes.io/service-account-token   3         12d
tomcat-ingress-secret   kubernetes.io/tls                     2         33s
[[email protected] ingress-nginx]# kubectl describe secret tomcat-ingress-secret
Name:         tomcat-ingress-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.crt:  1285 bytes
tls.key:  1679 bytes

3.3. 建立ingress

[[email protected] ingress-nginx]# vim ingress-tomcat-tls.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-tomcat-tls
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - tomcat.zwx.com
    secretName: tomcat-ingress-secret
  rules:
  - host: tomcat.zwx.com
    http:
      paths:
      - backend:
          serviceName: tomcat
          servicePort: 8080
[[email protected] ingress-nginx]# kubectl apply -f ingress-tomcat-tls.yaml 
ingress.extensions/ingress-tomcat-tls created

3.4. 訪問測試

檢視ingress-nginx 服務

[[email protected] ingress-nginx]# kubectl get svc -n ingress-nginx -o wide
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE       SELECTOR
default-http-backend   ClusterIP   10.107.114.219   <none>        80/TCP                       4h        app.kubernetes.io/name=default-http-backend,app.kubernetes.io/part-of=ingress-nginx
ingress-nginx          NodePort    10.108.12.235    <none>        80:31875/TCP,443:32020/TCP   3h        app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx

所以訪問https://tomcat.zwx.com:32020 在這裡插入圖片描述