kubeadm部署kubernetes 1.13.1叢集
kubeadm是Kubernetes官方提供的用於快速部署Kubernetes叢集的工具,本次使用kubeadm搭建一個3節點的單叢集k8s。
#kubernetes部署dashboard視覺化外掛:
https://blog.csdn.net/networken/article/details/85607593
#kubernetes部署 Prometheus Operator監控系統:
https://blog.csdn.net/networken/article/details/85620793
#kubernetes部署 EFK日誌系統:
https://blog.csdn.net/networken/article/details/85775371
#kubernetes部署rook+ceph儲存系統:
https://blog.csdn.net/networken/article/details/85772418
#如果存在映象拉取失敗問題,參考這這篇文章:
https://blog.csdn.net/networken/article/details/84571373
1.準備基礎環境
我們將使用kubeadm部署3個節點的 Kubernetes Cluster,整體結構圖:
Kubernetes基本架構圖:
節點詳細資訊:
節點主機名 | 節點IP | 節點角色 | 作業系統 | 節點配置 |
---|---|---|---|---|
k8s-master | 192.168.92.56 | master | CentOS7.5 | 2C2G |
k8s-node1 | 192.168.92.57 | node | CentOS7.5 | 2C2G |
k8s-node2 | 192.168.92.58 | node | CentOS7.5 | 2C2G |
節點元件分佈:
Master 和 Node 節點由於分工不一樣,所以安裝的服務不同,最終安裝完畢,Master 和 Node 啟動的核心服務分別如下:
Master節點 | Node節點 |
---|---|
kube-apiserver | kube-proxy |
kube-controller-manager | kube-flannel |
kube-scheduler | other apps |
kube-proxy | |
etcd | |
coredns | |
kube-flannel |
無特殊說明以下操作在所有節點執行:
修改主機名:
#master節點:
hostnamectl set-hostname k8s-master
#node1節點:
hostnamectl set-hostname k8s-node1
#node2節點:
hostnamectl set-hostname k8s-node2
基本配置:
#修改/etc/hosts檔案
cat >> /etc/hosts << EOF
192.168.92.56 k8s-master
192.168.92.57 k8s-node1
192.168.92.58 k8s-node2
EOF
#關閉防火牆和selinux
systemctl stop firewalld && systemctl disable firewalld
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config && setenforce 0
#關閉swap
swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak |grep -v swap > /etc/fstab
配置時間同步
使用chrony同步時間,配置master節點與網路NTP伺服器同步時間,所有node節點與master節點同步時間。
配置master節點:
#安裝chrony:
yum install -y chrony
#註釋預設ntp伺服器
sed -i 's/^server/#&/' /etc/chrony.conf
#指定上游公共 ntp 伺服器,並允許其他節點同步時間
cat >> /etc/chrony.conf << EOF
server 0.asia.pool.ntp.org iburst
server 1.asia.pool.ntp.org iburst
server 2.asia.pool.ntp.org iburst
server 3.asia.pool.ntp.org iburst
allow all
EOF
#重啟chronyd服務並設為開機啟動:
systemctl enable chronyd && systemctl restart chronyd
#開啟網路時間同步功能
timedatectl set-ntp true
配置所有node節點:
(注意修改master IP地址)
#安裝chrony:
yum install -y chrony
#註釋預設伺服器
sed -i 's/^server/#&/' /etc/chrony.conf
#指定內網 master節點為上游NTP伺服器
echo server 192.168.92.56 iburst >> /etc/chrony.conf
#重啟服務並設為開機啟動:
systemctl enable chronyd && systemctl restart chronyd
所有節點執行chronyc sources命令,檢視存在以^*開頭的行,說明已經與伺服器時間同步
設定網橋包經過iptalbes
RHEL / CentOS 7上的一些使用者報告了由於iptables被繞過而導致流量路由不正確的問題。建立/etc/sysctl.d/k8s.conf檔案,新增如下內容:
cat <<EOF > /etc/sysctl.d/k8s.conf
vm.swappiness = 0
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
# 使配置生效
modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf
kube-proxy開啟ipvs的前提條件
由於ipvs已經加入到了核心的主幹,所以為kube-proxy開啟ipvs的前提需要載入以下的核心模組:
在所有的Kubernetes節點執行以下指令碼:
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
#執行指令碼
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
上面指令碼建立了/etc/sysconfig/modules/ipvs.modules檔案,保證在節點重啟後能自動載入所需模組。 使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4命令檢視是否已經正確載入所需的核心模組。
接下來還需要確保各個節點上已經安裝了ipset軟體包。 為了便於檢視ipvs的代理規則,最好安裝一下管理工具ipvsadm。
# yum install ipset ipvsadm -y
安裝Docker
Kubernetes預設的容器執行時仍然是Docker,使用的是kubelet中內建dockershim CRI實現。需要注意的是,Kubernetes 1.13最低支援的Docker版本是1.11.1,最高支援是18.06,而Docker最新版本已經是18.09了,故我們安裝時需要指定版本為18.06.1-ce。
#配置docker yum源
yum-config-manager \
--add-repo \
http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
#安裝指定版本,這裡安裝18.06
yum list docker-ce --showduplicates | sort -r
yum install -y docker-ce-18.06.1.ce-3.el7
systemctl start docker && systemctl enable docker
指令碼安裝docker-ce並配置daocloud映象加速(可選):
wget https://raw.githubusercontent.com/zhwill/LinuxShell/master/Install_docker-ce.sh && bash Install_docker-ce.sh
安裝kubeadm、kubelet、kubectl
官方安裝文件可以參考:
https://kubernetes.io/docs/setup/independent/install-kubeadm/
- kubelet 在群集中所有節點上執行的核心元件, 用來執行如啟動pods和containers等操作。
- kubeadm 引導啟動k8s叢集的命令列工具,用於初始化 Cluster。
- kubectl 是 Kubernetes 命令列工具。通過 kubectl 可以部署和管理應用,檢視各種資源,建立、刪除和更新各種元件。
#配置kubernetes.repo的源,由於官方源國內無法訪問,這裡使用阿里雲yum源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
#在所有節點上安裝指定版本 kubelet、kubeadm 和 kubectl
yum install -y kubelet-1.13.1 kubeadm-1.13.1 kubectl-1.13.1
#啟動kubelet服務
systemctl enable kubelet && systemctl start kubelet
2.初始化master節點
完整的官方文件可以參考:
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
Master節點執行初始化:
注意這裡執行初始化用到了- -image-repository選項,指定初始化需要的映象源從阿里雲映象倉庫拉取。
kubeadm init \
--apiserver-advertise-address=192.168.92.56 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.13.1 \
--pod-network-cidr=10.244.0.0/16
初始化命令說明:
--apiserver-advertise-address
指明用 Master 的哪個 interface 與 Cluster 的其他節點通訊。如果 Master 有多個 interface,建議明確指定,如果不指定,kubeadm 會自動選擇有預設閘道器的 interface。
--pod-network-cidr
指定 Pod 網路的範圍。Kubernetes 支援多種網路方案,而且不同網路方案對 --pod-network-cidr 有自己的要求,這裡設定為 10.244.0.0/16 是因為我們將使用 flannel 網路方案,必須設定成這個 CIDR。
--image-repository
Kubenetes預設Registries地址是 k8s.gcr.io,在國內並不能訪問 gcr.io,在1.13版本中我們可以增加–image-repository引數,預設值是 k8s.gcr.io,將其指定為阿里雲映象地址:registry.aliyuncs.com/google_containers。
--kubernetes-version=v1.13.1
關閉版本探測,因為它的預設值是stable-1,會導致從https://dl.k8s.io/release/stable-1.txt下載最新的版本號,我們可以將其指定為固定版本(最新版:v1.13.1)來跳過網路請求。
初始化過程如下:
[[email protected] ~]# kubeadm init \
> --image-repository registry.aliyuncs.com/google_containers \
> --kubernetes-version v1.13.1 \
> --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.13.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.92.56 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.92.56 127.0.0.1 ::1]
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.92.56]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 21.009858 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-master" as an annotation
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 60syk6.vnplamkn3zhwu3s3
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 192.168.92.56:6443 --token 60syk6.vnplamkn3zhwu3s3 --discovery-token-ca-cert-hash sha256:7d50e704bbfe69661e37c5f3ad13b1b88032b6b2b703ebd4899e259477b5be69
[[email protected] ~]#
(注意記錄下初始化結果中的kubeadm join命令,部署worker節點時會用到)
初始化過程說明:
- [preflight] kubeadm 執行初始化前的檢查。
- [kubelet-start] 生成kubelet的配置檔案”/var/lib/kubelet/config.yaml”
- [certificates] 生成相關的各種token和證書
- [kubeconfig] 生成 KubeConfig 檔案,kubelet 需要這個檔案與 Master 通訊
- [control-plane] 安裝 Master 元件,會從指定的 Registry 下載元件的 Docker 映象。
- [bootstraptoken] 生成token記錄下來,後邊使用kubeadm join往叢集中新增節點時會用到
- [addons] 安裝附加元件 kube-proxy 和 kube-dns。
- Kubernetes Master 初始化成功,提示如何配置常規使用者使用kubectl訪問叢集。
- 提示如何安裝 Pod 網路。
- 提示如何註冊其他節點到 Cluster。
配置 kubectl
kubectl 是管理 Kubernetes Cluster 的命令列工具,前面我們已經在所有的節點安裝了 kubectl。Master 初始化完成後需要做一些配置工作,然後 kubectl 就能使用了。
依照 kubeadm init 輸出的最後提示,推薦用 Linux 普通使用者執行 kubectl。
建立普通使用者centos
#建立普通使用者並設定密碼123456
useradd centos && echo "centos:123456" | chpasswd centos
#追加sudo許可權,並配置sudo免密
sed -i '/^root/a\centos ALL=(ALL) NOPASSWD:ALL' /etc/sudoers
#儲存叢集安全配置檔案到當前使用者.kube目錄
su - centos
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
#啟用 kubectl 命令自動補全功能(登出重新登入生效)
echo "source <(kubectl completion bash)" >> ~/.bashrc
需要這些配置命令的原因是:Kubernetes 叢集預設需要加密方式訪問。所以,這幾條命令,就是將剛剛部署生成的 Kubernetes 叢集的安全配置檔案,儲存到當前使用者的.kube 目錄下,kubectl 預設會使用這個目錄下的授權資訊訪問 Kubernetes 叢集。
如果不這麼做的話,我們每次都需要通過 export KUBECONFIG 環境變數告訴 kubectl 這個安全配置檔案的位置。
配置完成後centos使用者就可以使用 kubectl 命令管理叢集了。
檢視叢集狀態:
[[email protected] ~]$ kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
[[email protected] ~]$
確認各個元件都處於healthy狀態。
檢視節點狀態
[[email protected] ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master NotReady master 36m v1.13.1
[[email protected] ~]$
可以看到,當前只存在1個master節點,並且這個節點的狀態是 NotReady。
使用 kubectl describe 命令來檢視這個節點(Node)物件的詳細資訊、狀態和事件(Event):
[[email protected] ~]$ kubectl describe node k8s-master
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 33m kubelet, k8s-master Starting kubelet.
Normal NodeHasSufficientMemory 33m (x8 over 33m) kubelet, k8s-master Node k8s-master status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 33m (x8 over 33m) kubelet, k8s-master Node k8s-master status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 33m (x7 over 33m) kubelet, k8s-master Node k8s-master status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 33m kubelet, k8s-master Updated Node Allocatable limit across pods
Normal Starting 33m kube-proxy, k8s-master Starting kube-proxy.
通過 kubectl describe 指令的輸出,我們可以看到 NodeNotReady 的原因在於,我們尚未部署任何網路外掛,kube-proxy等元件還處於starting狀態。
另外,我們還可以通過 kubectl 檢查這個節點上各個系統 Pod 的狀態,其中,kube-system 是 Kubernetes 專案預留的系統 Pod 的工作空間(Namepsace,注意它並不是 Linux Namespace,它只是 Kubernetes 劃分不同工作空間的單位):
[[email protected] ~]$ kubectl get pod -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-78d4cf999f-7jdx7 0/1 Pending 0 29m <none> <none> <none> <none>
coredns-78d4cf999f-s6mhk 0/1 Pending 0 29m <none> <none> <none> <none>
etcd-k8s-master 1/1 Running 0 34m 192.168.92.56 k8s-master <none> <none>
kube-apiserver-k8s-master 1/1 Running 0 34m 192.168.92.56 k8s-master <none> <none>
kube-controller-manager-k8s-master 1/1 Running 0 34m 192.168.92.56 k8s-master <none> <none>
kube-proxy-przwf 1/1 Running 0 34m 192.168.92.56 k8s-master <none> <none>
kube-scheduler-k8s-master 1/1 Running 0 34m 192.168.92.56 k8s-master <none> <none>
[[email protected] ~]$
可以看到,CoreDNS依賴於網路的 Pod 都處於 Pending 狀態,即排程失敗。這當然是符合預期的:因為這個 Master 節點的網路尚未就緒。
叢集初始化如果遇到問題,可以使用kubeadm reset命令進行清理然後重新執行初始化。
部署網路外掛
要讓 Kubernetes Cluster 能夠工作,必須安裝 Pod 網路,否則 Pod 之間無法通訊。
Kubernetes 支援多種網路方案,這裡我們使用 flannel
執行如下命令部署 flannel:
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
[[email protected] ~]$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created
[[email protected] ~]$
部署完成後,我們可以通過 kubectl get 重新檢查 Pod 的狀態:
[[email protected] ~]$ kubectl get pod -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-78d4cf999f-7jdx7 1/1 Running 0 11h 10.244.0.3 k8s-master <none> <none>
coredns-78d4cf999f-s6mhk 1/1 Running 0 11h 10.244.0.2 k8s-master <none> <none>
etcd-k8s-master 1/1 Running 1 11h 192.168.92.56 k8s-master <none> <none>
kube-apiserver-k8s-master 1/1 Running 1 11h 192.168.92.56 k8s-master <none> <none>
kube-controller-manager-k8s-master 1/1 Running 1 11h 192.168.92.56 k8s-master <none> <none>
kube-flannel-ds-amd64-lkf2f 1/1 Running 0 10h 192.168.92.56 k8s-master <none> <none>
kube-proxy-przwf 1/1 Running 1 11h 192.168.92.56 k8s-master <none> <none>
kube-scheduler-k8s-master 1/1 Running 1 11h 192.168.92.56 k8s-master <none> <none>
[[email protected] ~]$
可以看到,所有的系統 Pod 都成功啟動了,而剛剛部署的flannel網路外掛則在 kube-system 下面新建了一個名叫kube-flannel-ds-amd64-lkf2f的 Pod,一般來說,這些 Pod 就是容器網路外掛在每個節點上的控制組件。
Kubernetes 支援容器網路外掛,使用的是一個名叫 CNI 的通用介面,它也是當前容器網路的事實標準,市面上的所有容器網路開源專案都可以通過 CNI 接入 Kubernetes,比如 Flannel、Calico、Canal、Romana 等等,它們的部署方式也都是類似的“一鍵部署”。
再次檢視master節點狀態已經為ready狀態:
[[email protected] ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 11h v1.13.1
[[email protected] ~]$
至此,Kubernetes 的 Master 節點就部署完成了。如果你只需要一個單節點的 Kubernetes,現在你就可以使用了。不過,在預設情況下,Kubernetes 的 Master 節點是不能執行使用者 Pod 的。
3.部署worker節點
Kubernetes 的 Worker 節點跟 Master 節點幾乎是相同的,它們執行著的都是一個 kubelet 元件。唯一的區別在於,在 kubeadm init 的過程中,kubelet 啟動後,Master 節點上還會自動執行 kube-apiserver、kube-scheduler、kube-controller-manger 這三個系統 Pod。
在 k8s-node1 和 k8s-node2 上分別執行如下命令,將其註冊到 Cluster 中:
#執行以下命令將節點接入叢集
kubeadm join 192.168.92.56:6443 --token 67kq55.8hxoga556caxty7s --discovery-token-ca-cert-hash sha256:7d50e704bbfe69661e37c5f3ad13b1b88032b6b2b703ebd4899e259477b5be69
#如果執行kubeadm init時沒有記錄下加入叢集的命令,可以通過以下命令重新建立
kubeadm token create --print-join-command
在k8s-node1上執行kubeadm join :
[[email protected] ~]# kubeadm join 192.168.92.56:6443 --token 67kq55.8hxoga556caxty7s --discovery-token-ca-cert-hash sha256:7d50e704bbfe69661e37c5f3ad13b1b88032b6b2b703ebd4899e259477b5be69
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "192.168.92.56:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.92.56:6443"
[discovery] Requesting info from "https://192.168.92.56:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.92.56:6443"
[discovery] Successfully established connection with API Server "192.168.92.56:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-node1" as an annotation
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the master to see this node join the cluster.
[[email protected] ~]#
重複執行以上操作將k8s-node2也加進去(注意重新執行kubeadm token create --print-join-command)。
然後根據提示,我們可以通過 kubectl get nodes 檢視節點的狀態:
[[email protected] ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 11h v1.13.1
k8s-node1 Ready <none> 24m v1.13.1
k8s-node2 Ready <none> 4m9s v1.13.1
[[email protected] ~]$
nodes狀態全部為ready,由於每個節點都需要啟動若干元件,如果node節點的狀態是 NotReady,可以檢視所有節點pod狀態,確保所有pod成功拉取到映象並處於running狀態:
[[email protected] ~]$ kubectl get pod --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-78d4cf999f-7jdx7 1/1 Running 0 11h 10.244.0.3 k8s-master <none> <none>
kube-system coredns-78d4cf999f-s6mhk 1/1 Running 0 11h 10.244.0.2 k8s-master <none> <none>
kube-system etcd-k8s-master 1/1 Running 1 12h 192.168.92.56 k8s-master <none> <none>
kube-system kube-apiserver-k8s-master 1/1 Running 1 12h 192.168.92.56 k8s-master <none> <none>
kube-system kube-controller-manager-k8s-master 1/1 Running 1 12h 192.168.92.56 k8s-master <none> <none>
kube-system kube-flannel-ds-amd64-d2r8p 1/1 Running 0 6m43s 192.168.92.58 k8s-node2 <none> <none>
kube-system kube-flannel-ds-amd64-d85c6 1/1 Running 0 27m 192.168.92.57 k8s-node1 <none> <none>
kube-system kube-flannel-ds-amd64-lkf2f 1/1 Running 0 11h 192.168.92.56 k8s-master <none> <none>
kube-system kube-proxy-k8jx8 1/1 Running 0 6m43s 192.168.92.58 k8s-node2 <none> <none>
kube-system kube-proxy-n95ck 1/1 Running 0 27m 192.168.92.57 k8s-node1 <none> <none>
kube-system kube-proxy-przwf 1/1 Running 1 12h 192.168.92.56 k8s-master <none> <none>
kube-system kube-scheduler-k8s-master 1/1 Running 1 12h 192.168.92.56 k8s-master <none> <none>
[[email protected] ~]$
這時,所有的節點都已經 Ready,Kubernetes Cluster 建立成功,一切準備就緒。
如果pod狀態為Pending、ContainerCreating、ImagePullBackOff 都表明 Pod 沒有就緒,Running 才是就緒狀態。
如果有pod提示Init:ImagePullBackOff,說明這個pod的映象在對應節點上拉取失敗,我們可以通過 kubectl describe pod 檢視 Pod 具體情況,以確認拉取失敗的映象:
[[email protected] ~]$ kubectl describe pod kube-flannel-ds-amd64-d2r8p --namespace=kube-system
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m14s default-scheduler Successfully assigned kube-system/kube-flannel-ds-amd64-lzx5v to k8s-node2
Warning Failed 109s kubelet, k8s-node2 Failed to pull image "quay.io/coreos/flannel:v0.10.0-amd64": rpc error: code = Unknown desc = Error response from daemon: Get https://quay.io/v2/: net/http: TLS handshake timeout
Warning Failed 109s kubelet, k8s-node2 Error: ErrImagePull
Normal BackOff 108s kubelet, k8s-node2 Back-off pulling image "quay.io/coreos/flannel:v0.10.0-amd64"
Warning Failed 108s kubelet, k8s-node2 Error: ImagePullBackOff
Normal Pulling 94s (x2 over 2m6s) kubelet, k8s-node2 pulling image "quay.io/coreos/flannel:v0.10.0-amd64"
這裡看最後events輸出內容,可以看到在下載 image 時失敗,如果網路質量不好,這種情況是很常見的。我們可以耐心等待,因為 Kubernetes 會重試,我們也可以自己手工執行 docker pull 去下載這個映象。
[[email protected] ~]# docker pull quay.io/coreos/flannel:v0.10.0-amd64
v0.10.0-amd64: Pulling from coreos/flannel
ff3a5c916c92: Already exists
8a8433d1d437: Already exists
306dc0ee491a: Already exists
856cbd0b7b9c: Already exists
af6d1e4decc6: Already exists
Digest: sha256:88f2b4d96fae34bfff3d46293f7f18d1f9f3ca026b4a4d288f28347fcb6580ac
Status: Image is up to date for quay.io/coreos/flannel:v0.10.0-amd64
[[email protected] ~]#
如果無法從 quay.io/coreos/flannel:v0.10.0-amd64 下載映象,可以從阿里雲或者dockerhub映象倉庫下載,然後改回原來的tag即可:
docker pull registry.cn-hangzhou.aliyuncs.com/kubernetes_containers/flannel:v0.10.0-amd64
docker tag registry.cn-hangzhou.aliyuncs.com/kubernetes_containers/flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.10.0-amd64
docker rmi registry.cn-hangzhou.aliyuncs.com/kubernetes_containers/flannel:v0.10.0-amd64
檢視master節點下載了哪些映象:
[[email protected] ~]$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-proxy v1.13.1 fdb321fd30a0 2 weeks ago 80.2MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.13.1 40a63db91ef8 2 weeks ago 181MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.13.1 ab81d7360408 2 weeks ago 79.6MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.13.1 26e6f1db2a52 2 weeks ago 146MB
registry.aliyuncs.com/google_containers/coredns 1.2.6 f59dcacceff4 8 weeks ago 40MB
registry.aliyuncs.com/google_containers/etcd 3.2.24 3cab8e1b9802 3 months ago 220MB
quay.io/coreos/flannel v0.10.0-amd64 f0fad859c909 11 months ago 44.6MB
registry.aliyuncs.com/google_containers/pause 3.1 da86e6ba6ca1 12 months ago 742kB
[[email protected] ~]$
檢視node節點下載了哪些映象:
[[email protected] ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-proxy v1.13.1 fdb321fd30a0 2 weeks ago 80.2MB
quay.io/coreos/flannel v0.10.0-amd64 f0fad859c909 11 months ago 44.6MB
registry.aliyuncs.com/google_containers/pause 3.1 da86e6ba6ca1 12 months ago 742kB
[[email protected] ~]#