istio sidecar自動注入過程分析
istio sidecar自動注入過程分析
istio通過mutating webhook admission controller機制實現sidecar的自動注入.istio sidecard在每個服務建立pod時都會被自動注入.
sidecar自動注入檢查
檢查kube-apiserver
webhook支援需要Kubernets1.9或者更高的版本,使用以下命令檢視
[[email protected] ~]# kubectl api-versions | grep admissionregistration admissionregistration.k8s.io/v1beta1
同時檢查kube-apiserver有沒加入引數MutatingAdmissionWebhook和ValidatingAdmissionWebhook
檢查sidecar-injector的configmap
在sidecar-injector的configmap中設定policy=enabled欄位來檢視是否啟用自動注入
[[email protected] ~]# kubectl describe cm istio-sidecar-injector -n istio-system Name: istio-sidecar-injector Namespace: istio-system Labels: app=istio chart=istio-1.0.3 heritage=Tiller istio=sidecar-injector release=istio ... Data ==== config: ---- policy: enabled
檢查namespace標籤
為需要自動注入的namespace打上標籤istio-injection: enabled
[[email protected] ~]# kubectl get namespace -L istio-injection NAME STATUS AGE ISTIO-INJECTION default Active 3d enabled istio-system Active 3d kube-public Active 3d kube-system Active 3d
kubectl label namespace default istio-injection=enabled
sidecar自動注入過程
webhook過程
檢視sidecar的webhook
[[email protected] ~]# kubectl get MutatingWebhookConfiguration -n istio-system
NAME CREATED AT
istio-sidecar-injector 2018-11-12T09:14:44Z
[[email protected]test1 ~]# kubectl describe MutatingWebhookConfiguration istio-sidecar-injector -n istio-system
Name: istio-sidecar-injector
Namespace:
Labels: app=istio-sidecar-injector
chart=sidecarInjectorWebhook-1.0.3
heritage=Tiller
release=istio
... ...
Webhooks:
Client Config:
... ...
Service:
Name: istio-sidecar-injector
Namespace: istio-system
Path: /inject
Failure Policy: Fail
Name: sidecar-injector.istio.io
Namespace Selector:
Match Labels:
Istio - Injection: enabled
Rules:
API Groups:
API Versions:
v1
Operations:
CREATE
Resources:
pods
由上面可以看出建立pod時會呼叫sidecar的webhook,接著向istio-sidecar-injector的服務傳送inject註冊(post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s).
檢視istio-sidecar-injector的日誌
[[email protected] ~]# kubectl get pods -n istio-system | grep istio-sidecar
istio-sidecar-injector-d96cd9459-lbf66 1/1 Running 0 13d
[[email protected] ~]# kubectl logs istio-sidecar-injector-d96cd9459-lbf66 -n istio-system
2018-11-09T06:40:53.895979Z info AdmissionReview for Kind=/v1, Kind=Pod Namespace=default Name= () UID=67d96021-e3ea-11e8-a721-00163e0c1d10 Rfc6902PatchOperation=CREATE UserInfo={system:unsecured [system:masters system:authenticated] map[]}
2018-11-09T06:40:53.897821Z info AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"docker.io/istio/proxy_init:1.0.0","args":["-p","15001","-u","1337","-m","REDIRECT","-i","10.0.0.1/24","-x","","-b","80,","-d",""] ... ...},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"docker.io/istio/proxyv2:1.0.0","args":["proxy","sidecar",... ...\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}}]hook傳送inject後,sidecar會返回兩個container,istio-init和istio-proxy.下面我們來具體分析下.
獲取pod具體資訊
[[email protected] ~]#kubectl describe pod nginx-dm-fff68d674-9tv9w
Name: nginx-dm-fff68d674-9tv9w
Namespace: default
Node: 10.0.3.126/10.0.3.126
Start Time: Fri, 09 Nov 2018 14:40:53 +0800
Labels: name=nginx
pod-template-hash=999248230
Annotations: sidecar.istio.io/status={"version":"5aa52d92ced8dab93e04a5a4701773b2f3d78968c04b05bb430f32e80a4d9be1","initContainers":["istio-init"],"containers":["istio-proxy"],...
Status: Running
IP: 172.30.2.21
Controlled By: ReplicaSet/nginx-dm-fff68d674
Init Containers:
istio-init:
Container ID: docker://43668b6cf4bb331542b8d98348a7670dad99b735aa0ef0ca572bf4ee1966538b
Image: docker.io/istio/proxy_init:1.0.0
Image ID: docker-pullable://istio/[email protected]:345c40053b53b7cc70d12fb94379e5aa0befd979a99db80833cde671bd1f9fad
Port: <none>
Host Port: <none>
Args:
-p
15001
... ...
Containers:
Containers:
nginx:
Container ID: docker://d917ffa9282bc4f82a0af1c8cbd6b51c0392fca6a85de6f8db6da128700db204
Image: nginx:alpine
Image ID:
Port: 80/TCP
Host Port: 0/TCP
istio-proxy:
Container ID: docker://932a8bc6b85f1106cde057bd55598337bf7f9963fc4e796d3d88907d717a8eff
Image: docker.io/istio/proxyv2:1.0.0
Image ID: docker-pullable://istio/[email protected]:77915a0b8c88cce11f04caf88c9ee30300d5ba1fe13146ad5ece9abf8826204c
Port: <none>
Host Port: <none>
Args:
proxy
sidecar
--configPath
/etc/istio/proxy
--binaryPath
/usr/local/bin/envoy
--serviceCluster
... ...由具體資訊可知,pod除了自身的容器外,還額外注入了兩個容器.這就是由istio-sidecar-injector完成的.
proxy_init
proxy_init是一個Init Containers.Init Containers用於pod中執行初始化的任務,執行完畢退出後,才會執行後面的containers.
[[email protected] ~]# docker inspect docker.io/istio/proxy_init:1.0.0
[
{
"RepoTags": [
"istio/proxy_init:1.0.0",
"gcr.io/istio-release/proxy_init:1.0.0"
],
"ContainerConfig": {
...
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"ENTRYPOINT [\"/usr/local/bin/istio-iptables.sh\"]"
],
...
},
]如上Cmd可以知道,這個容器主要執行的是istio-iptables.sh的指令碼.
檢視指令碼內容
...
while getopts ":p:u:g:m:b:d:i:x:h" opt; do
case ${opt} in
p)
PROXY_PORT=${OPTARG}
;;
u)
...該指令碼通過配置iptable來劫持pod中的流量.結合前面的-p 15001可知pod的資料流量被轉發向envoy的15001埠.
proxyv2
檢視pod內istio-proxy的程序
[[email protected] ~]# kubectl exec nginx-dm-fff68d674-9tv9w -c istio-proxy -- ps -ef
UID PID PPID C STIME TTY TIME CMD
istio-p+ 1 0 0 Nov09 ? 00:00:12 /usr/local/bin/pilot-agent proxy sidecar --configPath /etc/istio/proxy --binaryPath /usr/local/bin/envoy --serviceCluster istio-proxy --drainDuration 45s --parentShutdownDuration 1m0s --discoveryAddress istio-pilot.istio-system:15007 --discoveryRefreshDelay 1s --zipkinAddress zipkin.istio-system:9411 --connectTimeout 10s --statsdUdpAddress istio-statsd-prom-bridge.istio-system:9125 --proxyAdminPort 15000 --controlPlaneAuthPolicy NONE
istio-p+ 24 1 0 Nov09 ? 00:42:50 /usr/local/bin/envoy -c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-proxy --service-node sidecar~172.30.2.21~nginx-dm-fff68d674-9tv9w.default~default.svc.cluster.local --max-obj-name-len 189 -l warn --v2-config-only上面有兩個程序pilot-agent和envoy.
pilot-agent根據k8s api生成配置資訊,並負責管理(啟動,熱更新,關閉等)整個envoy.生成的配置資訊在 /etc/istio/proxy/envoy-rev0.json,具體內容可自己檢視.
envoy由pilot-agent程序啟動,Envoy讀取Pilot-agent為它生成的配置檔案(envoy-rev0.json),然後根據該檔案的配置獲取到Pilot的地址,通過資料面標準API的xDS介面從pilot拉取動態配置資訊.
參考文件:
1.https://istio.io/docs/setup/kubernetes/sidecar-injection/
2.https://zhaohuabing.com/post/2018-09-25-istio-traffic-management-impl-intro/