k8s 二進制安裝 1.11.0
阿新 • • 發佈：2019-01-23
sse 復制 ann 分發 perm art nds ace 1.3
本篇安裝單個etcd，然後進行擴容etcd節點至2個、3個 二進制安裝k8s 1.11.0 實驗架構 master: 192.168.0.91 etcd node2: 192.168.0.92 node3: 192.168.0.93 1、環境配置 如下操作在所有節點操作 配置hosts解析 [root@host-10-1-1-8 k8s]# hostnamectl set-hostname master [root@host-10-1-1-68 ~]# hostnamectl set-hostname node2 [root@host-10-1-1-111 ~]# hostnamectl set-hostnamenode3 cat >>/etc/hosts<<EOF 192.168.0.91 master 192.168.0.92 node2 192.168.0.93 node3 EOF 禁用selinux sed -i ‘s/SELINUX=permissive/SELINUX=disabled/‘ /etc/sysconfig/selinux 關閉swap 註釋/etc/fstab文件裏swap相關的行 所有節點都重啟 開啟forward iptables -P FORWARD ACCEPT 配置轉發相關參數 cat >> /etc/sysctl.d/k8s.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 vm.swappiness=0 EOF sysctl --system 加載ipvs相關內核模塊 如果重新開機，需要重新加載 modprobe ip_vs modprobe ip_vs_rr modprobe ip_vs_wrr modprobe ip_vs_sh modprobe nf_conntrack_ipv4 lsmod | grep ip_vs 3、安裝CFSSL證書生成工具 只在master節點操作 mkdir -pv /server/software/k8s cd/server/software/k8s wget下載cfssl工具 wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 安裝cfssl工具 只要把安裝包改下名字，移動到usr/local/bin/下，加上授權即可 mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson chmod +x /usr/local/bin/cfssl* 4、創建CA配置文件：生成其他組件ca證書時需要用到(除了根證書) 只在master節點操作 mkdir -p $HOME/ssl && cd $HOME/ssl cat >ca-config.json<<EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } } EOF 5、生成 ca 根證書和私鑰: 生成其他組件ca證書時需要用到 只在master節點操作 cd $HOME/ssl cat >ca-csr.json<<EOF { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ], "ca": { "expiry": "87600h" } } EOF cfssl gencert -initca ca-csr.json | cfssljson -bare ca 查看生成的證書和私鑰 ca-key.pem ca.pem 把根證書和私鑰復制到一個目錄裏面 mkdir -p /etc/kubernetes/cert/ cp ca*.pem /etc/kubernetes/cert/ 6、安裝、配置、啟動etcd 只在master節點上操作 6.1、生成etcd的ca證書和私鑰 cd $HOME/ssl cat >etcd-csr.json<<EOF { "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.0.91", "192.168.0.92", "192.168.0.93" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "etcd", "OU": "Etcd Security" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd 查看生成的證書和私鑰 etcd-key.pem etcd.pem 把etcd證書復制到一個目錄裏面 mkdir -p /etc/etcd/cert/ cp etcd*.pem /etc/etcd/cert/ 6.2、安裝etcd mkdir -p /server/software/k8s mkdir -p /opt/k8s/bin cd /server/software/k8s wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz tar -xf etcd-v3.2.18-linux-amd64.tar.gz mv etcd-v3.2.18-linux-amd64/etcd* /opt/k8s/bin chmod +x /opt/k8s/bin/* ln -s /opt/k8s/bin/etcd /usr/bin/etcd etcd --version 6.3 配置etcd啟動腳本 註意：經過多次嘗試，沒有 --force-new-claster 就無法添加成功,這句話意思是強制生成新的節點 cat >> /etc/profile << EOF export ETCD_NAME=$(hostname) export INTERNAL_IP=$(hostname -i | awk ‘{print $NF}‘) export ECTD_CLUSTER=‘master=https://192.168.0.91:2380‘ EOF source /etc/profile mkdir -p /data/etcd cat > /etc/systemd/system/etcd.service <<EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target Documentation=https://github.com/coreos [Service] Type=notify WorkingDirectory=/data/etcd EnvironmentFile=-/etc/etcd/etcd.conf ExecStart=/opt/k8s/bin/etcd \ --name $ETCD_NAME \ --cert-file=/etc/etcd/cert/etcd.pem \ --key-file=/etc/etcd/cert/etcd-key.pem \ --peer-cert-file=/etc/etcd/cert/etcd.pem \ --peer-key-file=/etc/etcd/cert/etcd-key.pem \ --trusted-ca-file=/etc/kubernetes/cert/ca.pem \ --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \ --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \ --listen-peer-urls https://${INTERNAL_IP}:2380 \ --listen-client-urls https://${INTERNAL_IP}:2379,http://127.0.0.1:2379 \ --advertise-client-urls https://${INTERNAL_IP}:2379 \ --initial-cluster-token my-etcd-token \ --initial-cluster $ECTD_CLUSTER \ --initial-cluster-state new \ --force-new-claster \ --data-dir=/data/etcd Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF 6.4、啟動etctd、設置開機啟動 systemctl daemon-reload #一定要執行，否則報錯 systemctl start etcd systemctl status etcd systemctl enable etcd systemctl stop etcd 6.5、查看單個etcd集群狀態 [root@master ~]# etcdctl cluster-health member 42f7141ed6110de1 is healthy: got healthy result from https://192.168.0.91:2379 cluster is healthy 提前分發k8s所有組件二進制文件、順便安裝kubectl工具 # 安裝包解壓後包括 kubectl 工具，所以不需要單獨使用kubernetes-server-client-amd64.tar.gz 安裝包分發 kubectl 工具 下載、解壓安裝包 cd /server/software/k8s wget https://dl.k8s.io/v1.11.0/kubernetes-server-linux-amd64.tar.gz tar -xf kubernetes-server-linux-amd64.tar.gz # 分發所有組件二進制文件， # 後面配置 kube-apiserver 等組件啟動文件裏面需要帶上二進制文件路徑 mkdir -p /usr/local/kubernetes/bin cd /server/software/k8s/kubernetes/server/bin cp kube-apiserver kube-controller-manager kube-scheduler kube-proxy kubelet kubectl /usr/local/kubernetes/bin #這一步很關鍵 # 安裝kubectl工具 # 後面創建 admin kubeconfig等配置文件時候需要用到； # admin.conf = ~/.kube/config，因為是復制過來的 ； # kubectl作用：當kubelet組件通過bootstrap token 認證後，kubectl默認從 ~/.kube/config 文件讀取 kube-apiserver 地址、證書、用戶名等信息； cp /usr/local/kubernetes/bin/kubectl /usr/local/bin/kubectl # 查看 kubectl 版本， # 出現下面的情況就是正確的。did you specify the right host or port? 這個報錯忽略，因為還沒有安裝kubelet服務 kubectl version [root@master bin]# kubectl version Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"} The connection to the server localhost:8080 was refused - did you specify the right host or port? cd $HOME 6、生成admin的ca證書和私鑰 （kubectl 作為集群的管理工具，需要被授予最高權限。這裏創建具有最高權限的 admin 證書、admin kubeconfig） 註意：後面只有apiserver和kubelet這兩個服務啟動參數會用到admin的ca證書；kubectl工具和kubelet服務不是一回事 cd $HOME/ssl cat >admin-csr.json<<EOF { "CN": "admin", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:masters", "OU": "System" } ] } EOF # 生成 admin ca cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin # 查看生成的admin ca ls admin*.pem 7、配置 kube-apiserver ca # 10.96.0.1 是 kube-apiserver 指定的 service-cluster-ip-range 網段的第一個IP cd $HOME/ssl cat >kube-apiserver-csr.json<<EOF { "CN": "kube-apiserver", "hosts": [ "127.0.0.1", "192.168.0.91", "10.96.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF # 生成 kube-apiserver ca cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver # 查看生成的kube-apiserver ca ls kube-apiserver*.pem 8、配置 kube-controller-manager ca cd $HOME/ssl cat >kube-controller-manager-csr.json<<EOF { "CN": "system:kube-controller-manager", "hosts": [ "127.0.0.1", "192.168.0.91" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:kube-controller-manager", "OU": "System" } ] } EOF # 生成 kube-controller-manager ca cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager # 查看生成的kube-controller-manager ca ls kube-controller-manager*.pem 9、配置 kube-scheduler ca cd $HOME/ssl cat >kube-scheduler-csr.json<<EOF { "CN": "system:kube-scheduler", "hosts": [ "127.0.0.1", "192.168.0.91" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:kube-scheduler", "OU": "System" } ] } EOF # 生成 kube-scheduler ca cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler # 查看生成的kube-scheduler ca ls kube-scheduler*.pem 10、配置 kube-proxy ca # 只是node節點需要用到 cd $HOME/ssl cat >kube-proxy-csr.json<<EOF { "CN": "system:kube-proxy", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:kube-proxy", "OU": "System" } ] } EOF # 生成 kube-proxy ca cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy # 查看生成的kube-proxy ca ls kube-proxy*.pem 11、復制所有的ca 到一個目錄裏面，方便管理 cd $HOME/ssl mkdir -p /etc/kubernetes/pki cp ca*.pem admin*.pem kube-proxy*.pem kube-scheduler*.pem kube-controller-manager*.pem kube-apiserver*.pem /etc/kubernetes/pki cp etcd.pem etcd-key.pem /etc/kubernetes/pki/etcd/ 開啟 bootstrap token 認證 ，kubelet TLS Boostrap機制 # kube-apiserver、kubelet啟動文件需要用到token， # token中包含kubelet-bootstrap用戶 # 後面kubelet組件啟動參數中需要使用 kubelet-bootstrap.conf 文件向 kube-apiserver 發送 CSR 請求， # 請求通過後，kubectl才會從 ~/.kube/config 文件讀取 kube-apiserver 地址、證書、用戶名等信息 # 靜態獲取token # 還有一種是用 kubeadm 動態獲取token，kubeadm token create,這樣可以使用 TLS bootstrap 機制自動生成 client 和 server 證書，過期後自動輪轉。 # 本實驗采用靜態獲取，一天後過期 export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ‘ ‘) # 創建token.csv文件 # 只有kube-apiserver 啟動文件中需要用到 cat > /etc/kubernetes/token.csv <<EOF ${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap" EOF # 創建kubelet-bootstrap.conf # 只有kubelet啟動文件需要用到 cd /etc/kubernetes export KUBE_APISERVER="https://192.168.0.91:6443" kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=kubelet-bootstrap.conf kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap.conf kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.conf kubectl config use-context default --kubeconfig=kubelet-bootstrap.conf # 給kubelet-bootstrap用戶授權 # 創建一個 clusterrolebinding，將 bootstrap token 文件中的 kubelet-bootstrap 用戶賦予 system:node-bootstrapper cluster 角色 # 默認情況下，bootstrap這個 user 和 group 沒有創建 CSR 的權限，kubelet 會啟動失敗，所以要給kubelet-bootstrap角色授權 kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap # 如果沒有授權會出現下面錯誤： [root@node2 kubernetes]# journalctl -u kubelet |tail failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "kubelet-bootstrap" cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope 12、創建 admin kubeconfig # 只有kubelet服務啟動參數需要用到admin kubeconfig， # admin.conf = ~/.kube/config，因為復制過來的 ；kubectl 默認從 ~/.kube/config 文件讀取 kube-apiserver 地址、證書、用戶名等信息， cd /etc/kubernetes export KUBE_APISERVER="https://192.168.0.91:6443" #設置集群參數 kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=admin.conf #設置客戶端認證參數 kubectl config set-credentials admin --client-certificate=/etc/kubernetes/pki/admin.pem --client-key=/etc/kubernetes/pki/admin-key.pem --embed-certs=true --kubeconfig=admin.conf #設置上下文參數 kubectl config set-context default --cluster=kubernetes --user=admin --kubeconfig=admin.conf #設置默認上下文 kubectl config use-context default --kubeconfig=admin.conf 13、創建 kube-controller-manager kubeconfig cd /etc/kubernetes export KUBE_APISERVER="https://192.168.0.91:6443" kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=kube-controller-manager.conf kubectl config set-credentials kube-controller-manager --client-certificate=/etc/kubernetes/pki/kube-controller-manager.pem --client-key=/etc/kubernetes/pki/kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.conf kubectl config set-context default --cluster=kubernetes --user=kube-controller-manager --kubeconfig=kube-controller-manager.conf kubectl config use-context default --kubeconfig=kube-controller-manager.conf 14、創建 kube-scheduler kubeconfig cd /etc/kubernetes export KUBE_APISERVER="https://192.168.0.91:6443" kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=kube-scheduler.conf kubectl config set-credentials kube-scheduler --client-certificate=/etc/kubernetes/pki/kube-scheduler.pem --client-key=/etc/kubernetes/pki/kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.conf kubectl config set-context default --cluster=kubernetes --user=kube-scheduler --kubeconfig=kube-scheduler.conf kubectl config use-context default --kubeconfig=kube-scheduler.conf 15、創建 kube-proxy kubeconfig # 只是node節點需要用到kube-proxy kubeconfig cd /etc/kubernetes export KUBE_APISERVER="https://192.168.0.91:6443" kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=kube-proxy.conf kubectl config set-credentials kube-proxy --client-certificate=/etc/kubernetes/pki/kube-proxy.pem --client-key=/etc/kubernetes/pki/kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.conf kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.conf kubectl config use-context default --kubeconfig=kube-proxy.conf cd $HOME 17、配置啟動kube-apiserver # 復制 etcd ca mkdir -pv /etc/kubernetes/pki/etcd cd /etc/etcd/ssl cp etcd.pem ca-key.pem ca.pem /etc/kubernetes/pki/etcd # 生成 service account key cd /etc/kubernetes/pki/ openssl genrsa -out /etc/kubernetes/pki/sa.key 2048 openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub ls /etc/kubernetes/pki/sa.* cd $HOME # 啟動文件 cat >/etc/systemd/system/kube-apiserver.service<<EOF [Unit] Description=Kubernetes API Service Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] EnvironmentFile=-/etc/kubernetes/config EnvironmentFile=-/etc/kubernetes/apiserver ExecStart=/usr/local/kubernetes/bin/kube-apiserver \ \$KUBE_LOGTOSTDERR \ \$KUBE_LOG_LEVEL \ \$KUBE_ETCD_ARGS \ \$KUBE_API_ADDRESS \ \$KUBE_SERVICE_ADDRESSES \ \$KUBE_ADMISSION_CONTROL \ \$KUBE_APISERVER_ARGS Restart=on-failure Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF # 配置參數變量文件 # 下面 kube-apiserver、kube-controller-manager、kube-scheduler、kube-proxy這些服務 都需要用到，這裏只配置一次，以後重復利用，後面的也寫了變量文件，只是為了知道怎麽回事 cat >/etc/kubernetes/config<<EOF KUBE_LOGTOSTDERR="--logtostderr=true" KUBE_LOG_LEVEL="--v=2" EOF # 配置apiserver # 註意參數--token-auth-file=/etc/kubernetes/token.csv 表示在 apiserver 中靜態配置bootstrap token，和後面開啟 bootstrap token 認證步驟相呼應，不是動態的，所以有過期時間， # 後面kubelet組件啟動參數中需要使用 kubelet-bootstrap.conf 文件向 kube-apiserver 發送 CSR 請求，--bootstrap-kubeconfig 文件裏面包含token和apiserver裏面的token是一樣的， cat >/etc/kubernetes/apiserver<<EOF KUBE_API_ADDRESS="--advertise-address=192.168.0.91" KUBE_ETCD_ARGS="--etcd-servers=https://192.168.0.91:2379 --etcd-cafile=/etc/kubernetes/pki/ca.pem --etcd-certfile=/etc/kubernetes/pki/etcd/etcd.pem --etcd-keyfile=/etc/kubernetes/pki/etcd/etcd-key.pem" KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.96.0.0/12" KUBE_ADMISSION_CONTROL="--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota" KUBE_APISERVER_ARGS="--allow-privileged=true --authorization-mode=Node,RBAC --enable-bootstrap-token-auth=true --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/pki/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/pki/kube-apiserver-key.pem --client-ca-file=/etc/kubernetes/pki/ca.pem --service-account-key-file=/etc/kubernetes/pki/sa.pub --enable-swagger-ui=true --secure-port=6443 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --anonymous-auth=false --kubelet-client-certificate=/etc/kubernetes/pki/admin.pem --kubelet-client-key=/etc/kubernetes/pki/admin-key.pem" EOF # 啟動 systemctl daemon-reload systemctl enable kube-apiserver systemctl start kube-apiserver systemctl status kube-apiserver systemctl stop kube-apiserver # 通過瀏覽器訪問測試 # 報錯正常，不影響，以後解決 curl https://192.168.0.91:6443/swaggerapi 18、配置啟動kube-controller-manager # 配置啟動文件 cat >/etc/systemd/system/kube-controller-manager.service<<EOF Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] EnvironmentFile=-/etc/kubernetes/config EnvironmentFile=-/etc/kubernetes/controller-manager ExecStart=/usr/local/kubernetes/bin/kube-controller-manager \ \$KUBE_LOGTOSTDERR \ \$KUBE_LOG_LEVEL \ \$KUBECONFIG \ \$KUBE_CONTROLLER_MANAGER_ARGS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF # 配置參數變量文件 # 配置kube-apiserver啟動文件時已經配置過參數變量文件，這裏就不需要再做，寫在這裏只是為了知道下面配置文件裏的參數怎麽回事 cat >/etc/kubernetes/config<<EOF KUBE_LOGTOSTDERR="--logtostderr=true" KUBE_LOG_LEVEL="--v=2" EOF # 配置controller-manager文件 cat >/etc/kubernetes/controller-manager<<EOF KUBECONFIG="--kubeconfig=/etc/kubernetes/kube-controller-manager.conf" KUBE_CONTROLLER_MANAGER_ARGS="--address=127.0.0.1 --cluster-cidr=10.244.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem --service-account-private-key-file=/etc/kubernetes/pki/sa.key --root-ca-file=/etc/kubernetes/pki/ca.pem --leader-elect=true --use-service-account-credentials=true --node-monitor-grace-period=10s --pod-eviction-timeout=10s --allocate-node-cidrs=true --controllers=*,bootstrapsigner,tokencleaner" EOF 啟動 systemctl daemon-reload systemctl enable kube-controller-manager systemctl start kube-controller-manager systemctl status kube-controller-manager 19、配置啟動kube-scheduler # 配置啟動文件 cat >/etc/systemd/system/kube-scheduler.service<<EOF [Unit] Description=Kubernetes Scheduler Plugin Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/etc/kubernetes/config EnvironmentFile=-/etc/kubernetes/scheduler ExecStart=/usr/local/kubernetes/bin/kube-scheduler \ \$KUBE_LOGTOSTDERR \ \$KUBE_LOG_LEVEL \ \$KUBECONFIG \ \$KUBE_SCHEDULER_ARGS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF # 配置參數變量文件 # 配置kube-apiserver啟動文件時已經配置過參數變量文件，這裏就不需要再做，寫在這裏只是為了知道下面配置文件裏的參數怎麽回事 cat >/etc/kubernetes/config<<EOF KUBE_LOGTOSTDERR="--logtostderr=true" KUBE_LOG_LEVEL="--v=2" EOF # 配置scheduler文件 cat >/etc/kubernetes/scheduler<<EOF KUBECONFIG="--kubeconfig=/etc/kubernetes/kube-scheduler.conf" KUBE_SCHEDULER_ARGS="--leader-elect=true --address=127.0.0.1" EOF 啟動 systemctl daemon-reload systemctl enable kube-scheduler systemctl start kube-scheduler systemctl status kube-scheduler 21、單獨配置node2 相關組件 21.1、安裝docker 註意：docker和flannel是一體的，哪個節點上需要安裝flannel,哪個節點上就需要安裝docker v1.11.0版本推薦使用docker v17.03, v1.11,v1.12,v1.13, 也可以使用，再高版本的docker可能無法正常使用。 測試發現17.09無法正常使用，不能使用資源限制(內存CPU) 卸載自帶docker yum remove -y docker-ce docker-ce-selinux container-selinux 配置Docker倉庫鏡像 wget https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-17.03.2.ce-1.el7.centos.x86_64.rpm wget https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpm 安裝 Docker 和依賴包 yum install -y docker-ce-*.rpm 開機啟動 systemctl enable docker 啟動 docker 服務 systemctl start docker 21.2、部署 kubelet組件 # 下載、解壓安裝包 mkdir -p /server/software/k8s cd /server/software/k8s wget https://dl.k8s.io/v1.11.0/kubernetes-server-linux-amd64.tar.gz tar -xf kubernetes-server-linux-amd64.tar.gz 分發kubelet二進制文件 # 後面kubelet啟動文件需要用到kubelet二進制文件路徑 mkdir -p /usr/local/kubernetes/bin cp /server/software/k8s/kubernetes/server/bin/kubelet /usr/local/kubernetes/bin #這一步很關鍵 # 安裝kubectl工具 # kubecctl工具和kubelet服務不是一回事。安裝包解壓後包括 kubectl 工具，所以不需要單獨使用kubernetes-server-client-amd64.tar.gz 安裝包分發 kubectl 工具 # kubectl作用：當kubelet組件通過bootstrap token 認證後，kubectl 默認會從 ~/.kube/config 文件讀取 kube-apiserver 地址、證書、用戶名等信息； # admin.conf = ~/.kube/config，因為復制過來的 ； cp /server/software/k8s/kubernetes/server/bin/kubectl /usr/local/bin/kubectl # 查看 kubectl 版本， # 出現下面的情況就是正確的。did you specify the right host or port? 這個報錯忽略，因為還沒有安裝kubelet服務 kubectl version [root@master bin]# kubectl version Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"} The connection to the server localhost:8080 was refused - did you specify the right host or port? cd $HOME # 從master上復制admin.conf到node節點 scp /etc/kubernetes/admin.conf [email protected]:/etc/kubernetes/ # 復制 admin.conf 到 /.kube/config # kubectl 默認從 ~/.kube/config 文件讀取 kube-apiserver 地址、證書、用戶名等信息，如果沒有配置，執行 kubectl 命令時可能會出錯： rm -rf $HOME/.kube mkdir -p $HOME/.kube cp /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config 或者： export KUBECONFIG=/etc/kubernetes/admin.conf # 從masters上復制 bootstrap.conf 到node節點 # 後面kubelet服務啟動參數中需要使用 kubelet-bootstrap.conf 向 kube-apiserver 發送 CSR 請求， # 當kubelet服務通過bootstrap token 認證後，kubectl 默認會從 ~/.kube/config 文件讀取 kube-apiserver 地址、證書、用戶名等信息； scp /etc/kubernetes/kubelet-bootstrap.conf [email protected]:/etc/kubernetes/ # 查看組件狀態 kubectl get componentstatuses [root@node2 ~]# kubectl get componentstatuses Unable to connect to the server: x509: certificate signed by unknown authority # 安裝cni # kubelet 的啟動參數需要用到 cd /server/software/k8s wget https://github.com/containernetworking/plugins/releases/download/v0.7.1/cni-plugins-amd64-v0.7.1.tgz mkdir -p /opt/cni/bin tar -xf cni-plugins-amd64-v0.7.1.tgz -C /opt/cni/bin ls -l /opt/cni/bin cd $HOME 報錯：network plugin is not ready: cni config uninitialized 原因是因為kubelet配置了network-plugin=cni，但是還沒安裝，所以狀態會是NotReady,會報上面的錯誤，不想看這個報錯或者不需要網絡，就修改kubelet配置文件，去掉network-plugin=cni 就可以了 # 配置啟動kubelet # 創建數據目錄 mkdir -p /data/kubelet # 配置kubelet啟動文件 cat >/etc/systemd/system/kubelet.service<<EOF [Unit] Description=Kubernetes Kubelet Server Documentation=https://github.com/kubernetes/kubernetes After=docker.service Requires=docker.service [Service] WorkingDirectory=/data/kubelet EnvironmentFile=-/etc/kubernetes/config EnvironmentFile=-/etc/kubernetes/kubelet ExecStart=/usr/local/kubernetes/bin/kubelet \ \$KUBE_LOGTOSTDERR \ \$KUBE_LOG_LEVEL \ \$KUBELET_CONFIG \ \$KUBELET_HOSTNAME \ \$KUBELET_POD_INFRA_CONTAINER \ \$KUBELET_ARGS Restart=on-failure [Install] WantedBy=multi-user.target EOF # 配置參數變量文件 cat >/etc/kubernetes/config<<EOF KUBE_LOGTOSTDERR="--logtostderr=true" KUBE_LOG_LEVEL="--v=2" EOF # 配置kubelet.conf 文件 # 從master節點復制admin.conf過來，改成kubelet.conf即可；kubelet組件啟動參數需要用到kubelet.conf scp /etc/kubernetes/admin.conf 192.168.0.92:/etc/kubernetes/kubelet.conf # 配置kubelet文件 # 註意修改相關ip node節點也配置，node節點改成對應的nodeip cat >/etc/kubernetes/kubelet<<EOF KUBELET_HOSTNAME="--hostname-override=192.168.0.92" KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1" KUBELET_CONFIG="--config=/etc/kubernetes/kubelet-config.yml" KUBELET_ARGS="--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.conf --kubeconfig=/etc/kubernetes/kubelet.conf --cert-dir=/etc/kubernetes/pki --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d" EOF # 復制ca證書 # 從master節點復制 ca 證書，kubelet-config.yml 文件參數需要用到 scp $HOME/ssl/ca.pem 192.168.0.92:/etc/kubernetes/pki/ # 配置kubelet-config.yml文件 # 註意修改kubelet-config.yml相關ip，master node2 node3 使用各自ip cat >/etc/kubernetes/kubelet-config.yml<<EOF kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: 192.168.0.92 port: 10250 cgroupDriver: cgroupfs clusterDNS: - 10.96.0.10 clusterDomain: cluster.local. hairpinMode: promiscuous-bridge serializeImagePulls: false authentication: x509: clientCAFile: /etc/kubernetes/pki/ca.pem EOF # 啟動 systemctl daemon-reload systemctl enable kubelet systemctl start kubelet systemctl status kubelet 21.3、通過證書請求 # 在配置了kubectl的節點上執行如下操作 # 查看 kubectl get csr # 通過，下面的長字符串填寫在所有節點執行上一步的結果，包括master kubectl certificate approve node-csr-Yiiv675wUCvQl3HH11jDr0cC9p3kbrXWrxvG3EjWGoE # 查看節點 # 此時節點狀態為 NotReady，需要操作完後面才可以ready kubectl get nodes # 在node節點查看生成的文件 ls -l /etc/kubernetes/kubelet.conf ls -l /etc/kubernetes/pki/kubelet* 21.4、配置啟動kube-proxy # 安裝 yum install -y conntrack-tools # 復制kube-proxy.conf 把master節點上的kube-proxy.conf復制到node節點/etc/kubernetes/下 scp /etc/kubernetes/pki/kube-proxy*.pem 192.168.0.92:/etc/kubernetes/pki # 復制ca證書 mkdir -p /etc/kubernetes/pki 把master節點上的kube-proxy的ca證書復制到node節點/etc/kubernetes/pki 下 # 配置啟動文件 cat >/etc/systemd/system/kube-proxy.service<<EOF [Unit] Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] EnvironmentFile=-/etc/kubernetes/config EnvironmentFile=-/etc/kubernetes/proxy ExecStart=/usr/local/kubernetes/bin/kube-proxy \ \$KUBE_LOGTOSTDERR \ \$KUBE_LOG_LEVEL \ \$KUBECONFIG \ \$KUBE_PROXY_ARGS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF # 配置參數變量文件： cat >/etc/kubernetes/config<<EOF KUBE_LOGTOSTDERR="--logtostderr=true" KUBE_LOG_LEVEL="--v=2" EOF # 配置proxy文件 # 註意修改相關ip，master node2 node3 使用各自ip # 看下面proxy文件‘--proxy-mode=iptables‘,由於采用iptables模式，因為 ipvs 模式在centos7上有bug無法正常使用，1.11.0 以後的版本就可使用 ipvs 模式了， # 本實驗采用centos7.5、kubernetes 1.11.0 版本，所以使用iptables模式， cat >/etc/kubernetes/proxy<<EOF KUBECONFIG="--kubeconfig=/etc/kubernetes/kube-proxy.conf" KUBE_PROXY_ARGS="--bind-address=192.168.0.92 --proxy-mode=iptables --hostname-override=192.168.0.92 --cluster-cidr=10.244.0.0/16" EOF # 啟動 systemctl daemon-reload systemctl enable kube-proxy systemctl start kube-proxy systemctl status kube-proxy 21.5、設置集群角色 # 設置 master 為 master kubectl label nodes 192.168.0.91 node-role.kubernetes.io/master= # 設置 node2 node3 為 node kubectl label nodes 192.168.0.92 node-role.kubernetes.io/node= # 設置 master 一般情況下不接受負載 kubectl taint nodes 192.168.0.91 node-role.kubernetes.io/master=true:NoSchedule master運行pod kubectl taint nodes master.k8s node-role.kubernetes.io/master- master不運行pod kubectl taint nodes master.k8s node-role.kubernetes.io/master=:NoSchedule # 查看節點，此時節點狀態為 NotReady kubectl get no 21.6、配置使用flannel網絡 # 只有在安裝了docker的節點上才可以安裝flannel # 註意下面的網卡名稱要填寫對應的網卡名稱 # 下載配置 mkdir flannel && cd flannel wget https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml # 修改配置 # 此處的ip配置要與上面kubeadm的pod-network一致 net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan" } } # 如果Node有多個網卡的話，參考flannel issues 39701，https://github.com/kubernetes/kubernetes/issues/39701 # 目前需要在kube-flannel.yml中使用--iface參數指定集群主機內網網卡的名稱，否則可能會出現dns無法解析。容器無法通信的情況， # 修改鏡像 image: registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64 # flanneld啟動參數加上--iface=<iface-name> containers: - name: kube-flannel image: registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr - --iface=ens33 # 啟動 kubectl apply -f kube-flannel.yml # 查看 kubectl get pods -n kube-system kubectl get svc -n kube-system # 查看節點狀態 # 當 flannel pod 全部啟動之後，節點狀態為 Ready kubectl get no 配置使用coredns # 在master操作，註意下面用的 1.2.0 # 10.96.0.10 是 kubelet中配置的dns cd $HOME && mkdir coredns && cd coredns wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh chmod +x deploy.sh ./deploy.sh -i 10.96.0.10 > coredns.yaml kubectl apply -f coredns.yml 註意： 查看10.96.0.10 是否添加到文檔裏面 # 查看 kubectl get pods -n kube-system kubectl get svc -n kube-system 測試 啟動 kubectl run nginx --replicas=2 --image=nginx:alpine --port=80 kubectl expose deployment nginx --type=NodePort --name=example-service-nodeport kubectl expose deployment nginx --name=example-service kubectl scale --replicas=3 deployment/nginx 查看狀態 kubectl get deploy -o wide kubectl get pods -o wide kubectl get svc -o wide kubectl describe svc example-service DNS解析 kubectl run -it --rm --image=infoblox/dnstools dns-client nslookup kubernetes nslookup example-service curl example-service 訪問測試 # 10.96.59.56 為查看svc時獲取到的clusterip curl "10.107.91.153:80" # 32223 為查看svc時獲取到的 nodeport http://192.168.0.91:32223/ http://192.168.0.92:32223/ http://192.168.0.93:32223/ 清理 kubectl delete svc example-service example-service-nodeport kubectl delete deploy nginx curl
k8s 二進制安裝 1.11.0