# 二進位制部署 k8s 1.18.3 插播一條：ansible 一鍵部署：https://github.com/liyongjian5179/k8s-ansible ## 1、相關前置資訊 ### 1.1 版本資訊 kube_version: v1.18.3 etcd_version: v3.4.9 flannel: v0.12.0 coredns: v1.6.7 cni-plugins: v0.8.6 pod 網段：10.244.0.0/16 service 網段：10.96.0.0/12 kubernetes 內部地址：10.96.0.1 coredns 地址: 10.96.0.10 apiserver 域名：lb.5179.top ### 1.2 機器安排 | 主機名 | IP | 角色及元件 | k8s 相關元件 | | ------------- | ------------ | :----------------------: | :----------------------------------------------------------: | | centos7-nginx | 10.10.10.127 | nginx 四層代理 | nginx | | centos7-a | 10.10.10.128 | master,node,etcd,flannel | kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy | | centos7-b | 10.10.10.129 | master,node,etcd,flannel | kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy | | centos7-c | 10.10.10.130 | master,node,etcd,flannel | kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy | | centos7-d | 10.10.10.131 | node,flannel | kubelet kube-proxy | | centos7-e | 10.10.10.132 | node,flannel | kubelet kube-proxy | ## 2、部署前環境準備 以 `centos7-nginx` 當主控機對其他機器做免密 ### 2.1、 安裝`ansible`用於批量操作 安裝過程略 ```bash [root@centos7-nginx ~]# cat /etc/ansible/hosts [masters] 10.10.10.128 10.10.10.129 10.10.10.130 [nodes] 10.10.10.131 10.10.10.132 [k8s] 10.10.10.[128:132] ``` 推送宿主機 hosts 檔案 ```bash cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.10.10.127 centos7-nginx lb.5179.top 10.10.10.128 centos7-a 10.10.10.129 centos7-b 10.10.10.130 centos7-c 10.10.10.131 centos7-d 10.10.10.132 centos7-e ansible k8s -m shell -a "mv /etc/hosts /etc/hosts.bak" ansible k8s -m copy -a "src=/etc/hosts dest=/etc/hosts" ``` ### 2.2 關閉防火牆及SELINUX ```bash # 關閉防火牆 ansible k8s -m shell -a "systemctl stop firewalld && systemctl disable firewalld" # 關閉 selinux ansible k8s -m shell -a "setenforce 0 && sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux && sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config " ``` ### 2.3 關閉 swap 分割槽 ```bash ansible k8s -m shell -a "swapoff -a && sed -i 's/.*swap.*/#&/' /etc/fstab" ``` ### 2.4 安裝 docker及加速器 ```bash vim ./install_docker.sh #!/bin/bash # yum install -y yum-utils device-mapper-persistent-data lvm2 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yum -y install docker-ce-19.03.11-19.03.11 systemctl enable docker systemctl start docker docker version # 安裝加速器 tee /etc/docker/daemon.json <<-'EOF' { "registry-mirrors": ["https://ajpb7tdn.mirror.aliyuncs.com"], "log-opts": {"max-size":"100m", "max-file":"5"} } EOF systemctl daemon-reload systemctl restart docker ``` 然後使用 ansible 批量執行 ```bash ansible k8s -m script -a "./install_docker.sh" ``` ### 2.5 修改核心引數 ```bash vim 99-k8s.conf #sysctls for k8s node config net.ipv4.ip_forward=1 net.ipv4.tcp_slow_start_after_idle=0 net.core.rmem_max=16777216 fs.inotify.max_user_watches=524288 kernel.softlockup_all_cpu_backtrace=1 kernel.softlockup_panic=1 fs.file-max=2097152 fs.inotify.max_user_instances=8192 fs.inotify.max_queued_events=16384 vm.max_map_count=262144 vm.swappiness=0 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.may_detach_mounts=1 net.core.netdev_max_backlog=16384 net.ipv4.tcp_wmem=4096 12582912 16777216 net.core.wmem_max=16777216 net.core.somaxconn=32768 net.ipv4.ip_forward=1 net.ipv4.tcp_max_syn_backlog=8096 net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.tcp_rmem=4096 12582912 16777216 ``` 拷貝至遠端 ```bash ansible k8s -m copy -a "src=./99-k8s.conf dest=/etc/sysctl.d/" ansible k8s -m shell -a "cd /etc/sysctl.d/ && sysctl --system" ``` ### 2.6 建立對應的目錄 master 用 ```bash vim mkdir_k8s_master.sh #!/bin/bash mkdir /opt/etcd/{bin,data,cfg,ssl} -p mkdir /opt/kubernetes/{bin,cfg,ssl,logs} -p mkdir /opt/kubernetes/logs/{kubelet,kube-proxy,kube-scheduler,kube-apiserver,kube-controller-manager} -p echo 'export PATH=$PATH:/opt/kubernetes/bin' >> /etc/profile echo 'export PATH=$PATH:/opt/etcd/bin' >> /etc/profile source /etc/profile ``` node 用 ```bash vim mkdir_k8s_node.sh #!/bin/bash mkdir /opt/kubernetes/{bin,cfg,ssl,logs} -p mkdir /opt/kubernetes/logs/{kubelet,kube-proxy} -p echo 'export PATH=$PATH:/opt/kubernetes/bin' >> /etc/profile source /etc/profile ``` 呼叫 ansible 執行 ```bash ansible masters -m script -a "./mkdir_k8s_master.sh" ansible nodes -m script -a "./mkdir_k8s_node.sh" ``` ### 2.7 準備 LB 為三臺`master`提供高可用，可以選用雲廠商的 slb，也可以用 兩臺 nginx + keepalived 實現。 此處，為實驗環境，用單臺 nginx 坐四層代理實現 ```bash # 安裝 nginx [root@centos7-nginx ~]# yum install -y nginx # 建立子配置檔案 [root@centos7-nginx ~]# cd /etc/nginx/conf.d/ [root@centos7-nginx conf.d]# vim lb.tcp stream { upstream master { hash $remote_addr consistent; server 10.10.10.128:6443 max_fails=3 fail_timeout=30; server 10.10.10.129:6443 max_fails=3 fail_timeout=30; server 10.10.10.130:6443 max_fails=3 fail_timeout=30; } server { listen 6443; proxy_pass master; } } # 在主配置檔案中引入該檔案 [root@centos7-nginx ~]# cd /etc/nginx/ [root@centos7-nginx nginx]# vim nginx.conf ... include /etc/nginx/conf.d/*.tcp; ... # 加入開機自啟，並啟動 nginx [root@centos7-nginx nginx]# systemctl enable nginx Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service. [root@centos7-nginx nginx]# systemctl start nginx ``` ## 3、部署 ### 3.1 生成證書 執行指令碼 ```bash [root@centos7-nginx ~]# mkdir ssl && cd ssl [root@centos7-nginx ssl]# vim ./k8s-certificate.sh [root@centos7-nginx ssl]# ./k8s-certificate.sh 10.10.10.127,10.10.10.128,10.10.10.129,10.10.10.130,lb.5179.top,10.96.0.1 ``` **IP 說明：** * 10.10.10.127|lb.5179.top: nginx * 10.10.10.128|129|130: masters * 10.96.0.1: kubernetes(service 網段的第一個 IP) 指令碼內容如下 ```bash #!/bin/bash # 二進位制部署，生成 k8s 證書檔案 if [ $# -ne 1 ];then echo "please user in: `basename $0` MASTERS[10.10.10.127,10.10.10.128,10.10.10.129,10.10.10.130,lb.5179.top,10.96.0.1]" exit 1 fi MASTERS=$1 KUBERNETES_HOSTNAMES=kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.svc.cluster.local for i in `echo $MASTERS | tr ',' ' '`;do if [ -z $IPS ];then IPS=\"$i\", else IPS=$IPS\"$i\", fi done command_exists() { command -v "$@" > /dev/null 2>&1 } if command_exists cfssl; then echo "命令已存在" else # 下載生成證書命令 wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 # 新增執行許可權 chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64 # 移動到 /usr/local/bin 目錄下 mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo fi # 預設籤 10 年 cat > ca-config.j