kubernetes 1.13.3版本升級至1.14.1版本
阿新 • • 發佈：2019-05-06
connect tro 謝謝 heal don sse 工作 交流 tmp 相關推薦
本文的kubernetes環境：https://blog.51cto.com/billy98/2350660
一、說明
本文章介紹如何將使用kubeadm創建的Kubernetes集群從版本1.13.x升級到版本1.14.x。
只能從一個MINOR版本升級到下一個MINOR版本，或者在同一個MINOR的PATCH版本之間升級。也就是說，升級時不能跳過MINOR版本。例如，可以從1.y升級到1.y + 1，但不能從1.y升級到1.y + 2。
升級工作流程如下：
- 升級主master節點。
- 升級其他master節點。
- 升級work節點。
當前kubernetes版本信息：
[[email protected] ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node-01 Ready master 99d v1.13.3 node-02 Ready master 99d v1.13.3 node-03 Ready master 99d v1.13.3 node-04 Ready <none> 99d v1.13.3 node-05 Ready <none> 99d v1.13.3 node-06 Ready <none> 99d v1.13.3
二、升級主master節
1. 在第一個master節點上，升級kubeadm
yum install kubeadm-1.14.1 -y2. 驗證下載是否有效並具有預期版本：
kubeadm version3. 運行kubeadm upgrade plan
此命令檢查您的群集是否可以升級，並獲取可以升級到的版本
,您應該看到與此類似的輸出：
[[email protected] ~]# kubeadm upgrade plan [preflight] Running pre-flight checks. [upgrade] Making sure the cluster is healthy: [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml‘ [upgrade] Fetching available versions to upgrade to [upgrade/versions] Cluster version: v1.13.3 [upgrade/versions] kubeadm version: v1.14.1 I0505 13:55:58.449783 12871 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get https://dl.k8s.io/release/stable.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I0505 13:55:58.449867 12871 version.go:97] falling back to the local client version: v1.14.1 [upgrade/versions] Latest stable version: v1.14.1 I0505 13:56:08.645796 12871 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.13.txt": Get https://dl.k8s.io/release/stable-1.13.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I0505 13:56:08.645861 12871 version.go:97] falling back to the local client version: v1.14.1 [upgrade/versions] Latest version in the v1.13 series: v1.14.1 Components that must be upgraded manually after you have upgraded the control plane with ‘kubeadm upgrade apply‘: COMPONENT CURRENT AVAILABLE Kubelet 6 x v1.13.3 v1.14.1 Upgrade to the latest version in the v1.13 series: COMPONENT CURRENT AVAILABLE API Server v1.13.3 v1.14.1 Controller Manager v1.13.3 v1.14.1 Scheduler v1.13.3 v1.14.1 Kube Proxy v1.13.3 v1.14.1 CoreDNS 1.2.6 1.3.1 Etcd 3.2.24 3.3.10 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.14.1 _____________________________________________________________________
4. 運行升級命令
kubeadm upgrade apply v1.14.1你應該看到與此類似的輸出
[[email protected] ~]# kubeadm upgrade apply v1.14.1 [preflight] Running pre-flight checks. [upgrade] Making sure the cluster is healthy: [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml‘ [upgrade/version] You have chosen to change the cluster version to "v1.14.1" [upgrade/versions] Cluster version: v1.13.3 [upgrade/versions] kubeadm version: v1.14.1 [upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y [upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd] [upgrade/prepull] Prepulling image for component etcd. [upgrade/prepull] Prepulling image for component kube-apiserver. [upgrade/prepull] Prepulling image for component kube-scheduler. [upgrade/prepull] Prepulling image for component kube-controller-manager. [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager [apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver [apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler [apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-etcd [apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager [upgrade/prepull] Prepulled image for component kube-scheduler. [upgrade/prepull] Prepulled image for component kube-apiserver. [upgrade/prepull] Prepulled image for component etcd. [upgrade/prepull] Prepulled image for component kube-controller-manager. [upgrade/prepull] Successfully prepulled the images for all the control plane components [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.14.1"... Static pod: kube-apiserver-node-01 hash: 26d86add2bfd0fd6825f5507fff1fb5e Static pod: kube-controller-manager-node-01 hash: 21ea3d3ccb8d8dc00056209ca3da698b Static pod: kube-scheduler-node-01 hash: a8d928943d47ec793a700ef95c4b6b4a [upgrade/etcd] Upgrading to TLS for etcd Static pod: etcd-node-01 hash: 35015766b0b1714f398bed77aae5be95 [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-00-02/etcd.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: etcd-node-01 hash: 35015766b0b1714f398bed77aae5be95 Static pod: etcd-node-01 hash: 35015766b0b1714f398bed77aae5be95 Static pod: etcd-node-01 hash: 17ddbcfb2ddf1d447ceec2b52c9faa96 [apiclient] Found 3 Pods for label selector component=etcd [upgrade/staticpods] Component "etcd" upgraded successfully! [upgrade/etcd] Waiting for etcd to become available [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests940835611" [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-00-02/kube-apiserver.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: kube-apiserver-node-01 hash: 26d86add2bfd0fd6825f5507fff1fb5e Static pod: kube-apiserver-node-01 hash: ff2267bcddb83b815efb49ff766ad897 [apiclient] Found 3 Pods for label selector component=kube-apiserver [upgrade/staticpods] Component "kube-apiserver" upgraded successfully! [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-00-02/kube-controller-manager.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: kube-controller-manager-node-01 hash: 21ea3d3ccb8d8dc00056209ca3da698b Static pod: kube-controller-manager-node-01 hash: ff8be061048a4660a1fbbf72db229d0d [apiclient] Found 3 Pods for label selector component=kube-controller-manager [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully! [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-00-02/kube-scheduler.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: kube-scheduler-node-01 hash: a8d928943d47ec793a700ef95c4b6b4a Static pod: kube-scheduler-node-01 hash: 959a5cdf1468825401daa8d35329351e [apiclient] Found 3 Pods for label selector component=kube-scheduler [upgrade/staticpods] Component "kube-scheduler" upgraded successfully! [upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [addons] Applied essential addon: CoreDNS [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [addons] Applied essential addon: kube-proxy [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.14.1". Enjoy! [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven‘t already done so.
5. 手動升級您的CNI提供程序插件。
你的容器網絡接口（CNI）提供商可能有自己的升級說明。檢查插件頁面以查找你的CNI提供商，並查看是否需要其他升級步驟。
6. 升級第一個master節點上的kubelet和kubectl
yum install kubectl-1.14.1 kebulet-1.14.1 -y重啟kubelet
systemctl daemon-reload
systemctl restart kubelet三、升級其他master節點
1. 升級kubeadm程序
yum install kubeadm-1.14.1 -y 2. 升級靜態pod
kubeadm upgrade node experimental-control-plane你可以看到類似信息：
[[email protected] ~]# kubeadm upgrade node experimental-control-plane
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml‘
[upgrade] Upgrading your Static Pod-hosted control plane instance to version "v1.14.1"...
Static pod: kube-apiserver-node-02 hash: 26d86add2bfd0fd6825f5507fff1fb5e
Static pod: kube-controller-manager-node-02 hash: 21ea3d3ccb8d8dc00056209ca3da698b
Static pod: kube-scheduler-node-02 hash: a8d928943d47ec793a700ef95c4b6b4a
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-node-02 hash: 1aa55e50528bd0621c5734515b3265fd
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-10-46/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: etcd-node-02 hash: 1aa55e50528bd0621c5734515b3265fd
Static pod: etcd-node-02 hash: 1aa55e50528bd0621c5734515b3265fd
Static pod: etcd-node-02 hash: 4710a34897e7838519a1bf8fe4dccf07
[apiclient] Found 3 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests483113569"
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-10-46/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-node-02 hash: 26d86add2bfd0fd6825f5507fff1fb5e
Static pod: kube-apiserver-node-02 hash: fe1005f40c3f390280358921c3073223
[apiclient] Found 3 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-10-46/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-node-02 hash: 21ea3d3ccb8d8dc00056209ca3da698b
Static pod: kube-controller-manager-node-02 hash: ff8be061048a4660a1fbbf72db229d0d
[apiclient] Found 3 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-10-46/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-node-02 hash: a8d928943d47ec793a700ef95c4b6b4a
Static pod: kube-scheduler-node-02 hash: 959a5cdf1468825401daa8d35329351e
[apiclient] Found 3 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade] The control plane instance for this node was successfully updated!3. 更新kubelet和kubectl
yum install kubectl-1.14.1 kebulet-1.14.1 -y4. 重啟kubelet
systemctl daemon-reload
systemctl restart kubelet四、升級work節點
1. 在work節點上升級kubeadm：
yum install -y kubeadm-1.14.12.調整調度策略
通過將節點標記為不可調度並逐出pod來準備節點以進行維護(此步驟在master上執行)。
kubectl drain $NODE --ignore-daemonsets[[email protected] ~]# kubectl drain node-04 --ignore-daemonsets
node/node-04 already cordoned
WARNING: ignoring DaemonSet-managed Pods: cattle-system/cattle-node-agent-h555m, default/glusterfs-vhdqv, kube-system/canal-mbwvf, kube-system/kube-flannel-ds-amd64-zdfn8, kube-system/kube-proxy-5d64l
evicting pod "coredns-55696d4b79-kfcrh"
evicting pod "cattle-cluster-agent-66bd75c65f-k7p6n"
pod/cattle-cluster-agent-66bd75c65f-k7p6n evicted
pod/coredns-55696d4b79-kfcrh evicted
node/node-04 evicted3. 更新node
[[email protected] ~]# kubeadm upgrade node config --kubelet-version v1.14.1
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.4. 更新kubelet和kubectl
yum install kubectl-1.14.1 kebulet-1.14.1 -y5.重啟kubelet
systemctl daemon-reload
systemctl restart kubelet6. 恢復調度策略（master執行）
kubectl uncordon $NODE其他work節點升級根據以上的步驟執行一遍即可。
四、驗證群集的狀態
在所有節點上升級kubelet之後，通過從kubectl可以訪問群集的任何位置運行以下命令來驗證所有節點是否可用：
[[email protected] ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
node-01 Ready master 99d v1.14.1
node-02 Ready master 99d v1.14.1
node-03 Ready master 99d v1.14.1
node-04 Ready <none> 99d v1.14.1
node-05 Ready <none> 99d v1.14.1
node-06 Ready <none> 99d v1.14.1該STATUS列所有節點應顯示Ready，並且應更新版本號。
五、工作原理
kubeadm upgrade apply 執行以下操作：
- 檢查您的群集是否處於可升級狀態：
- API服務器是可訪問的
- 所有節點都處於該Ready狀態
- master是否健康
- 執行版本控制策略。
- 確保master鏡像可用或可用於拉到機器。
- 升級master組件或回滾（如果其中任何一個組件無法啟動）。
- 應用新的kube-dns和kube-proxy清單，並確保創建所有必需的RBAC規則。
- 創建API服務器的新證書和密鑰文件，如果舊文件即將在180天後過期，則備份舊文件。
kubeadm upgrade node experimental-control-plane在其他控制平面節點上執行以下操作：
- ClusterConfiguration從群集中獲取kubeadm 。
- 可選擇備份kube-apiserver證書。
- 升級master組件的靜態Pod清單。
本次的分享就到此，如有問題歡迎在下面留言交流，希望大家多多關註和點贊，謝謝！
kubernetes 1.13.3版本升級至1.14.1版本