k8s1.13.0二進制部署-master節點（三）
阿新 • • 發佈：2018-12-19
int scrip n-k inet6 als code jin rsa space 
部署apiserver
創建生成CSR的JSON配置文件
[root@k8s-master1 ssl]# vim kubernetes-csr.json { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.0.123", "192.168.0.124", "192.168.0.130", "10.0.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
生成kubernetes證書和私鑰
cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem -ca-key=/opt/kubernetes/ssl/ca-key.pem-config=/opt/kubernetes/ssl/ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
分發證書
[root@k8s-master1 ssl]# cp kubernetes*.pem /opt/kubernetes/ssl/
準備軟件包
下載二進制包：https://github.com/kubernetes/kubernetes
cd /usr/local/src/
wget https://dl.k8s.io/v1.13.0/kubernetes-server-linux-amd64.tar.gz
tar xf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin/
cp kube-apiserver kube-scheduler kube-controller-manager kubectl /opt/kubernetes/bin/
創建kube-apiserver使用的客戶端token文件
export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ‘ ‘) cat > /opt/kubernetes/cfg/token.csv <<EOF ${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap" EOF
創建kube-apiserver配置文件
[root@k8s-master1 ~]# vim /opt/kubernetes/cfg/kube-apiserver KUBE_APISERVER_OPTS="--logtostderr=false \ --v=4 --log-dir=/opt/kubernetes/log --etcd-servers=https://192.168.0.123:2379,https://192.168.0.125:2379,https://192.168.0.126:2379 \ --bind-address=0.0.0.0 --secure-port=6443 --advertise-address=192.168.0.123 --allow-privileged=true --service-cluster-ip-range=10.0.0.0/24 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --enable-bootstrap-token-auth --token-auth-file=/opt/kubernetes/cfg/token.csv --service-node-port-range=30000-50000 --tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --etcd-cafile=/opt/kubernetes/ssl/ca.pem --etcd-certfile=/opt/kubernetes/ssl/etcd.pem --etcd-keyfile=/opt/kubernetes/ssl/etcd-key.pem"
參數說明：
--logtostderr 啟用日誌
--v 日誌等級
--etcd-servers etcd集群地址
--bind-address 監聽地址
--secure-port https安全端口
--advertise-address 集群通告地址
--allow-privileged 啟用授權
--service-cluster-ip-range Service虛擬IP地址段
--enable-admission-plugins 準入控制模塊
--authorization-mode 認證授權，啟用RBAC授權和節點自管理
--enable-bootstrap-token-auth 啟用TLS bootstrap功能，後面會講到
--token-auth-file token文件
--service-node-port-range Service Node類型默認分配端口範圍
創建kube-apiserver系統服務
[root@k8s-master1 ~]# vim /usr/lib/systemd/system/kube-apiserver.service [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS Restart=on-failure RestartSec=5 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target
啟動apiserver
systemctl daemon-reload systemctl enable kube-apiserver systemctl restart kube-apiserver systemctl status kube-apiserver
通過url訪問api接口
[root@k8s-master1 ~]# curl -L --cacert /opt/kubernetes/ssl/ca.pem https://192.168.0.123:6443/api { "kind": "APIVersions", "versions": [ "v1" ], "serverAddressByClientCIDRs": [ { "clientCIDR": "0.0.0.0/0", "serverAddress": "192.168.0.123:6443" } ] } [root@k8s-master1 ~]# curl -L http://127.0.0.1:8080/api { "kind": "APIVersions", "versions": [ "v1" ], "serverAddressByClientCIDRs": [ { "clientCIDR": "0.0.0.0/0", "serverAddress": "192.168.0.123:6443" } ] }
部署Controller Manager
創建配置文件
[root@k8s-master1 ~]# vim /opt/kubernetes/cfg/kube-controller-manager KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \ --v=4 --log-dir=/opt/kubernetes/log --master=127.0.0.1:8080 --leader-elect=true --address=127.0.0.1 --service-cluster-ip-range=10.0.0.0/24 --cluster-name=kubernetes --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem --root-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem --experimental-cluster-signing-duration=87600h0m0s"
創建服務文件
[root@k8s-master1 ~]# vim /usr/lib/systemd/system/kube-controller-manager.service [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target
啟動服務
systemctl daemon-reload systemctl enable kube-controller-manager systemctl restart kube-controller-manager systemctl status kube-controller-manager
部署scheduller
創建配置文件
[root@k8s-master1 ~]# vim /opt/kubernetes/cfg/kube-scheduler KUBE_SCHEDULER_OPTS="--logtostderr=false \ --v=4 --log-dir=/opt/kubernetes/log --master=127.0.0.1:8080 --leader-elect"
--master 連接本地apiserver
--leader-elect 當該組件啟動多個時，自動選舉（HA）
創建服務文件
[root@k8s-master1 ~]# vim /usr/lib/systemd/system/kube-scheduler.service [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target
啟動服務
systemctl daemon-reload systemctl enable kube-scheduler systemctl restart kube-scheduler systemctl status kube-scheduler
master2部署
將master1的配置復制到master2，修改對應ip啟動服務即可
scp -r /opt/kubernetes 192.168.0.124:/opt/ scp /usr/lib/systemd/system/kube-* 192.168.0.124:/usr/lib/systemd/system/
訪問master2 API接口
[root@k8s-master2 ~]# curl -L --cacert /opt/kubernetes/ssl/ca.pem https://192.168.0.124:6443/api { "kind": "APIVersions", "versions": [ "v1" ], "serverAddressByClientCIDRs": [ { "clientCIDR": "0.0.0.0/0", "serverAddress": "192.168.0.124:6443" } ] } [root@k8s-master2 ~]# curl -L http://127.0.0.1:8080/api { "kind": "APIVersions", "versions": [ "v1" ], "serverAddressByClientCIDRs": [ { "clientCIDR": "0.0.0.0/0", "serverAddress": "192.168.0.124:6443" } ] }
配置apiserver的高可用
安裝keepalived
yum -y install keepalived
master1 的keepalived配置文件
[root@k8s-master1 ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { # 接收郵件地址 notification_email { [email protected] [email protected] [email protected] } # 郵件發送地址 notification_email_from [email protected] smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id API_MASTER } vrrp_script check_api { script "/etc/keepalived/check_api.sh" } vrrp_instance VI_1 { state MASTER interface ens32 virtual_router_id 51 # VRRP 路由 ID實例，每個實例是唯一的 priority 100 # 優先級，備服務器設置 90 advert_int 1 # 指定VRRP 心跳包通告間隔時間，默認1秒 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.0.130/24 } track_script { check_api } }View Code
master2的keepalived配置文件
[root@k8s-master2 ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { # 接收郵件地址 notification_email { [email protected] [email protected] [email protected] } # 郵件發送地址 notification_email_from [email protected] smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id API_MASTER } vrrp_script check_api { script "/etc/keepalived/check_api.sh" } vrrp_instance VI_1 { state BACKUP interface ens32 virtual_router_id 51 # VRRP 路由 ID實例，每個實例是唯一的 priority 90 # 優先級，備服務器設置 90 advert_int 1 # 指定VRRP 心跳包通告間隔時間，默認1秒 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.0.130/24 } track_script { check_api } }View Code
準備檢查apiserver的腳本
[root@k8s-master1 ~]# vim /etc/keepalived/check_api.sh count=$(ps -ef |grep kube-apiserver |egrep -cv "grep|$$") if [ "$count" -eq 0 ];then systemctl stop keepalived fi [root@k8s-master1 ~]# chmod +x /etc/keepalived/check_api.sh
啟動keepalived
systemctl start keepalived
systemctl enable keepalived
systemctl status keepalived
高可用測試
查看ip信息
[root@k8s-master1 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:8a:2b:5f brd ff:ff:ff:ff:ff:ff inet 192.168.0.123/24 brd 192.168.0.255 scope global ens32 valid_lft forever preferred_lft forever inet 192.168.0.130/24 scope global secondary ens32 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe8a:2b5f/64 scope link valid_lft forever preferred_lft forever [root@k8s-master2 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:77:dc:9c brd ff:ff:ff:ff:ff:ff inet 192.168.0.124/24 brd 192.168.0.255 scope global ens32 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe77:dc9c/64 scope link valid_lft forever preferred_lft forever
vip綁定在master1，訪問vip
[root@k8s-master1 ~]# curl -L --cacert /opt/kubernetes/ssl/ca.pem https://192.168.0.130:6443/api { "kind": "APIVersions", "versions": [ "v1" ], "serverAddressByClientCIDRs": [ { "clientCIDR": "0.0.0.0/0", "serverAddress": "192.168.0.123:6443" } ] }
停止master1的apiserver，再次訪問vip
[root@k8s-master1 ~]# curl -L --cacert /opt/kubernetes/ssl/ca.pem https://192.168.0.130:6443/api { "kind": "APIVersions", "versions": [ "v1" ], "serverAddressByClientCIDRs": [ { "clientCIDR": "0.0.0.0/0", "serverAddress": "192.168.0.124:6443" } ] }
查看ip信息，vip綁定在master2
[root@k8s-master1 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:8a:2b:5f brd ff:ff:ff:ff:ff:ff inet 192.168.0.123/24 brd 192.168.0.255 scope global ens32 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe8a:2b5f/64 scope link valid_lft forever preferred_lft forever [root@k8s-master2 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:77:dc:9c brd ff:ff:ff:ff:ff:ff inet 192.168.0.124/24 brd 192.168.0.255 scope global ens32 valid_lft forever preferred_lft forever inet 192.168.0.130/24 scope global secondary ens32 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe77:dc9c/64 scope link valid_lft forever preferred_lft forever
配置kubectl命令行工具
創建admin證書簽名請求
[root@k8s-master1 ssl]# vim admin-csr.json { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:masters", "OU": "System" } ] }
生成admin證書和密鑰
[root@k8s-master1 ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem -ca-key=/opt/kubernetes/ssl/ca-key.pem -config=/opt/kubernetes/ssl/ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
[root@k8s-master1 ssl]# cp admin*.pem /opt/kubernetes/ssl/
設置集群參數
kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true --server=https://192.168.0.130:6443
設置客戶端認證參數
kubectl config set-credentials admin --client-certificate=/opt/kubernetes/ssl/admin.pem --embed-certs=true --client-key=/opt/kubernetes/ssl/admin-key.pem
設置上下文參數
kubectl config set-context kubernetes --cluster=kubernetes --user=admin
設置默認上下文
kubectl config use-context kubernetes
查看集群信息
[root@k8s-master1 ~]# kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-1 Healthy {"health":"true"} etcd-2 Healthy {"health":"true"} etcd-0 Healthy {"health":"true"}
k8s1.13.0二進制部署-master節點（三）