centos7使用kubeadm安裝kubernetes 1.11版本多主高可用

阿新 • • 發佈：2018-08-01

ips refresh www ember issues abi dash iyu let

[TOC]

kubernetes介紹
要學習一個新的東西，先了解它是什麽，熟悉基本概念會有很大幫助。以下是我學習時看過的一篇核心概念介紹。
http://dockone.io/article/932

搭建Kubernetes集群環境有以下3種方式：

minikube
Minikube是一個工具，可以在本地快速運行一個單點的Kubernetes，嘗試Kubernetes或日常開發的用戶使用。不能用於生產環境。
官方地址：https://kubernetes.io/docs/setup/minikube/

以下是符合企業生產環境標準的Kubernetes集群環境方式：

kubeadm
Kubeadm也是一個工具，提供kubeadm init和kubeadm join，用於快速部署Kubernetes集群。

官方地址：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/

二進制包
從官方下載發行版的二進制包，手動部署每個組件，組成Kubernetes集群。
官方也提供了一個互動測試環境供大家玩耍：https://kubernetes.io/cn/docs/tutorials/kubernetes-basics/cluster-interactive/

1. 實驗環境說明

lab1: etcd master haproxy keepalived 192.168.105.92
lab2: etcd master haproxy keepalived 192.168.105.93
lab3: etcd master haproxy keepalived 192.168.105.94
lab4: node  192.168.105.95
lab4: node  192.168.105.96

vip(loadblancer ip): 192.168.105.99

virtualbox實驗使用的Vagrantfile：

# -*- mode: ruby -*-
# vi: set ft=ruby :

ENV["LC_ALL"] = "en_US.UTF-8"

Vagrant.configure("2") do |config|
    (2..6).each do |i|
      config.vm.define "lab#{i}" do |node|
        node.vm.box = "centos-7.4-docker-17"
        node.ssh.insert_key = false
        node.vm.hostname = "lab#{i}"
        node.vm.network "private_network", ip: "192.168.105.9#{i}"
        node.vm.provision "shell",
          inline: "echo hello from node #{i}"
        node.vm.provider "virtualbox" do |v|
          v.cpus = 2
          v.customize ["modifyvm", :id, "--name", "lab#{i}", "--memory", "2048"]
        end
      end
    end
end

2. 準備yum源

使用阿裏yum源，並將默認yum源文件都移走。

cd /etc/yum.repos.d
mkdir bak
mv *.repo bak/

vim CentOS-Base.repo

# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the 
# remarked out baseurl= line instead.
#
#

[base]
name=CentOS-$releasever - Base - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/os/$basearch/
        http://mirrors.aliyuncs.com/centos/$releasever/os/$basearch/
        http://mirrors.cloud.aliyuncs.com/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7

#released updates 
[updates]
name=CentOS-$releasever - Updates - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/updates/$basearch/
        http://mirrors.aliyuncs.com/centos/$releasever/updates/$basearch/
        http://mirrors.cloud.aliyuncs.com/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/extras/$basearch/
        http://mirrors.aliyuncs.com/centos/$releasever/extras/$basearch/
        http://mirrors.cloud.aliyuncs.com/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/centosplus/$basearch/
        http://mirrors.aliyuncs.com/centos/$releasever/centosplus/$basearch/
        http://mirrors.cloud.aliyuncs.com/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7

#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/contrib/$basearch/
        http://mirrors.aliyuncs.com/centos/$releasever/contrib/$basearch/
        http://mirrors.cloud.aliyuncs.com/centos/$releasever/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7

vim epel-7.repo

[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=http://mirrors.aliyun.com/epel/7/$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0

[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=http://mirrors.aliyun.com/epel/7/SRPMS
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0

vim docker-ce.repo

[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-stable-debuginfo]
name=Docker CE Stable - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/debug-$basearch/stable
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-stable-source]
name=Docker CE Stable - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/source/stable
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-edge]
name=Docker CE Edge - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/$basearch/edge
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-edge-debuginfo]
name=Docker CE Edge - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/debug-$basearch/edge
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-edge-source]
name=Docker CE Edge - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/source/edge
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-test]
name=Docker CE Test - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/$basearch/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-test-debuginfo]
name=Docker CE Test - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/debug-$basearch/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-test-source]
name=Docker CE Test - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/source/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-nightly]
name=Docker CE Nightly - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/$basearch/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-nightly-debuginfo]
name=Docker CE Nightly - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/debug-$basearch/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-nightly-source]
name=Docker CE Nightly - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/source/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

vim kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

2. 安裝配置docker

v1.11.1版本推薦使用docker v17.03,v1.11,v1.12,v1.13, 也可以使用，再高版本的docker可能無法正常使用。

這裏安裝v1.13版本。

yum -y install docker
systemctl enable docker && systemctl restart docker

docker啟動錯誤解決：

Error starting daemon: SELinux is not supported with the overlay2 graph driver on this kernel. Either boot into a newer kernel or disable selinux in docke...-enabled=false)

修改/etc/sysconfig/docker中的--selinux-enabled=false

3. 安裝 kubeadm, kubelet 和 kubectl

如下操作在所有節點操作

yum install -y kubelet kubeadm kubectl ipvsadm
systemctl enable kubelet && systemctl start kubelet

4. 配置系統相關參數

如下操作在所有節點操作

# 設置時區
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

# 臨時禁用selinux
# 永久關閉 修改/etc/sysconfig/selinux文件設置
sed -i ‘s/SELINUX=.*/SELINUX=disabled/‘ /etc/sysconfig/selinux
setenforce 0  # 需要重啟生效

# 開啟forward
# Docker從1.13版本開始調整了默認的防火墻規則
# 禁用了iptables filter表中FOWARD鏈
# 這樣會引起Kubernetes集群中跨Node的Pod無法通信
iptables -P FORWARD ACCEPT

# 臨時關閉swap
# 永久關閉 註釋/etc/fstab文件裏swap相關的行
swapoff -a

# 開啟防火墻允許集群機器間通信（為了方便測試或者直接關閉防火墻）
firewall-cmd --add-rich-rule ‘rule family=ipv4 source address=192.168.105.0/24 accept‘ # # 指定源IP（段），即時生效
firewall-cmd --add-rich-rule ‘rule family=ipv4 source address=192.168.105.0/24 accept‘ --permanent # 指定源IP（段），永久生效

如下操作在所有master節點操作

# 配置轉發相關參數，否則可能會出錯
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF
sysctl --system

# 加載ipvs相關內核模塊
# 如果重新開機，需要重新加載
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack_ipv4
lsmod | grep ip_vs

5. 配置hosts解析

如下操作在所有節點操作

cat >>/etc/hosts<<EOF
192.168.105.92 lab1
192.168.105.93 lab2
192.168.105.94 lab3
192.168.105.95 lab4
192.168.105.96 lab5
EOF

6. 配置haproxy代理和keepalived

如下操作在節點lab1,lab2,lab3操作

# 拉取haproxy鏡像
docker pull haproxy:1.7.8-alpine
mkdir /etc/haproxy
cat >/etc/haproxy/haproxy.cfg<<EOF
global
  log 127.0.0.1 local0 err
  maxconn 50000
  uid 99
  gid 99
  #daemon
  nbproc 1
  pidfile haproxy.pid

defaults
  mode http
  log 127.0.0.1 local0 err
  maxconn 50000
  retries 3
  timeout connect 5s
  timeout client 30s
  timeout server 30s
  timeout check 2s

listen admin_stats
  mode http
  bind 0.0.0.0:1080
  log 127.0.0.1 local0 err
  stats refresh 30s
  stats uri     /haproxy-status
  stats realm   Haproxy\ Statistics
  stats auth    will:will
  stats hide-version
  stats admin if TRUE

frontend k8s-https
  bind 0.0.0.0:8443
  mode tcp
  #maxconn 50000
  default_backend k8s-https

backend k8s-https
  mode tcp
  balance roundrobin
  server lab1 192.168.105.92:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
  server lab2 192.168.105.93:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
  server lab3 192.168.105.94:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
EOF

# 啟動haproxy
docker run -d --name my-haproxy -v /etc/haproxy:/usr/local/etc/haproxy:ro -p 8443:8443 -p 1080:1080 --restart always haproxy:1.7.8-alpine

# 查看日誌
docker logs my-haproxy

# 瀏覽器查看狀態
http://192.168.105.92:1080/haproxy-status
http://192.168.105.93:1080/haproxy-status
http://192.168.105.94:1080/haproxy-status

# 拉取keepalived鏡像
docker pull osixia/keepalived:1.4.4

# 啟動
# 載入內核相關模塊
lsmod | grep ip_vs
modprobe ip_vs

# 啟動keepalived
# ens32為本次實驗192.168.105.0/24網段的所在網卡
docker run --net=host --cap-add=NET_ADMIN -e KEEPALIVED_INTERFACE=ens32 -e KEEPALIVED_VIRTUAL_IPS="#PYTHON2BASH:[‘192.168.105.99‘]" -e KEEPALIVED_UNICAST_PEERS="#PYTHON2BASH:[‘192.168.105.92‘,‘192.168.105.93‘,‘192.168.105.94‘]" -e KEEPALIVED_PASSWORD=hello --name k8s-keepalived --restart always -d osixia/keepalived:1.4.4

# 查看日誌
# 會看到兩個成為backup 一個成為master
docker logs k8s-keepalived

# 此時會配置 192.168.105.99 到其中一臺機器
# ping測試
ping -c4 192.168.105.99

# 如果失敗後清理後，重新實驗
#docker rm -f k8s-keepalived
#ip a del 192.168.105.99/32 dev ens32

7. 配置啟動kubelet

如下操作在所有節點操作

# 配置kubelet使用國內pause鏡像
# 配置kubelet的cgroups
# 獲取docker的cgroups
DOCKER_CGROUPS=$(docker info | grep ‘Cgroup‘ | cut -d‘ ‘ -f3)
echo $DOCKER_CGROUPS
cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=$DOCKER_CGROUPS --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"
EOF

# 啟動
systemctl daemon-reload
systemctl enable kubelet && systemctl restart kubelet

8. 配置master

8.1 配置第一個master

如下操作在lab1節點操作

# 1.11 版本 centos 下使用 ipvs 模式會出問題
# 參考 https://github.com/kubernetes/kubernetes/issues/65461

cd /etc/kubernetes
# 生成配置文件
CP0_IP="192.168.105.92"
CP0_HOSTNAME="lab1"
cat >kubeadm-master.config<<EOF
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers

apiServerCertSANs:
- "lab1"
- "lab2"
- "lab3"
- "192.168.105.92"
- "192.168.105.93"
- "192.168.105.94"
- "192.168.105.99"
- "127.0.0.1"

api:
  advertiseAddress: $CP0_IP
  controlPlaneEndpoint: 192.168.105.99:8443

etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://$CP0_IP:2379"
      advertise-client-urls: "https://$CP0_IP:2379"
      listen-peer-urls: "https://$CP0_IP:2380"
      initial-advertise-peer-urls: "https://$CP0_IP:2380"
      initial-cluster: "$CP0_HOSTNAME=https://$CP0_IP:2380"
    serverCertSANs:
      - $CP0_HOSTNAME
      - $CP0_IP
    peerCertSANs:
      - $CP0_HOSTNAME
      - $CP0_IP

controllerManagerExtraArgs:
  node-monitor-grace-period: 10s
  pod-eviction-timeout: 10s

networking:
  podSubnet: 10.244.0.0/16

kubeProxy:
  config:
    # mode: ipvs
    mode: iptables
EOF

# 提前拉取鏡像
# 如果執行失敗 可以多次執行
kubeadm config images pull --config kubeadm-master.config

# 初始化
# 註意保存返回的 join 命令
kubeadm init --config kubeadm-master.config

# 初始化失敗時使用
#kubeadm reset

# 將ca相關文件傳至其他master節點
USER=root # customizable
CONTROL_PLANE_IPS="lab2 lab3"
for host in ${CONTROL_PLANE_IPS}; do
    scp /etc/kubernetes/pki/ca.crt "${USER}"@$host:/etc/kubernetes/pki/ca.crt
    scp /etc/kubernetes/pki/ca.key "${USER}"@$host:/etc/kubernetes/pki/ca.key
    scp /etc/kubernetes/pki/sa.key "${USER}"@$host:/etc/kubernetes/pki/sa.key
    scp /etc/kubernetes/pki/sa.pub "${USER}"@$host:/etc/kubernetes/pki/sa.pub
    scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host:/etc/kubernetes/pki/front-proxy-ca.crt
    scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host:/etc/kubernetes/pki/front-proxy-ca.key
    ssh "${USER}"@$host "mkdir -p /etc/kubernetes/pki/etcd"
    scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:/etc/kubernetes/pki/etcd/ca.crt
    scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:/etc/kubernetes/pki/etcd/ca.key
    scp /etc/kubernetes/admin.conf "${USER}"@$host:/etc/kubernetes/admin.conf
done

kubeadm init失敗解決：
將阿裏雲image tag成官方的image，即可解決init失敗問題。(v1.11.0有此問題)

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:v1.11.1 k8s.gcr.io/kube-apiserver-amd64:v1.11.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:v1.11.1 k8s.gcr.io/kube-proxy-amd64:v1.11.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:3.2.18 k8s.gcr.io/etcd-amd64:3.2.18
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:v1.11.1 k8s.gcr.io/kube-scheduler-amd64:v1.11.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:v1.11.1 k8s.gcr.io/kube-controller-manager-amd64:v1.11.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.1.3 k8s.gcr.io/coredns:1.1.3
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1 k8s.gcr.io/pause-amd64:3.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1

docker images # 結果如下

docker images
REPOSITORY                                                                          TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-apiserver-amd64                                                     v1.11.1             214c48e87f58        3 weeks ago         187 MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64            v1.11.1             214c48e87f58        3 weeks ago         187 MB
k8s.gcr.io/kube-proxy-amd64                                                         v1.11.1             1d3d7afd77d1        3 weeks ago         97.8 MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64                v1.11.1             1d3d7afd77d1        3 weeks ago         97.8 MB
k8s.gcr.io/kube-controller-manager-amd64                                            v1.11.1             55b70b420785        3 weeks ago         155 MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64   v1.11.1             55b70b420785        3 weeks ago         155 MB
k8s.gcr.io/kube-scheduler-amd64                                                     v1.11.1             0e4a34a3b0e6        3 weeks ago         56.8 MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64            v1.11.1             0e4a34a3b0e6        3 weeks ago         56.8 MB
k8s.gcr.io/coredns                                                                  1.1.3               b3b94275d97c        2 months ago        45.6 MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                         1.1.3               b3b94275d97c        2 months ago        45.6 MB
docker.io/osixia/keepalived                                                         1.4.4               d83816204582        2 months ago        53.7 MB
registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel                                   v0.10.0-amd64       b949a39093d6        2 months ago        44.6 MB
k8s.gcr.io/etcd-amd64                                                               3.2.18              b8df3b177be2        3 months ago        219 MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64                      3.2.18              b8df3b177be2        3 months ago        219 MB
quay.io/coreos/flannel                                                              v0.10.0-amd64       f0fad859c909        6 months ago        44.6 MB
k8s.gcr.io/pause-amd64                                                              3.1                 da86e6ba6ca1        7 months ago        742 kB
k8s.gcr.io/pause                                                                    3.1                 da86e6ba6ca1        7 months ago        742 kB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64                     3.1                 da86e6ba6ca1        7 months ago        742 kB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                           3.1                 da86e6ba6ca1        7 months ago        742 kB
docker.io/haproxy                                                                   1.7.8-alpine        297a495c0e70        12 months ago       14.7 MB

8.2 配置第二個master

如下操作在lab2節點操作

# 1.11 版本 centos 下使用 ipvs 模式會出問題
# 參考 https://github.com/kubernetes/kubernetes/issues/65461

cd /etc/kubernetes
# 生成配置文件
CP0_IP="192.168.105.92"
CP0_HOSTNAME="lab1"
CP1_IP="192.168.105.93"
CP1_HOSTNAME="lab2"
cat >kubeadm-master.config<<EOF
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers

apiServerCertSANs:
- "lab1"
- "lab2"
- "lab3"
- "192.168.105.92"
- "192.168.105.93"
- "192.168.105.94"
- "192.168.105.99"
- "127.0.0.1"

api:
  advertiseAddress: $CP1_IP
  controlPlaneEndpoint: 192.168.105.99:8443

etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://$CP1_IP:2379"
      advertise-client-urls: "https://$CP1_IP:2379"
      listen-peer-urls: "https://$CP1_IP:2380"
      initial-advertise-peer-urls: "https://$CP1_IP:2380"
      initial-cluster: "$CP0_HOSTNAME=https://$CP0_IP:2380,$CP1_HOSTNAME=https://$CP1_IP:2380"
      initial-cluster-state: existing
    serverCertSANs:
      - $CP1_HOSTNAME
      - $CP1_IP
    peerCertSANs:
      - $CP1_HOSTNAME
      - $CP1_IP

controllerManagerExtraArgs:
  node-monitor-grace-period: 10s
  pod-eviction-timeout: 10s

networking:
  podSubnet: 10.244.0.0/16

kubeProxy:
  config:
    # mode: ipvs
    mode: iptables
EOF

# 配置kubelet
kubeadm alpha phase certs all --config kubeadm-master.config
kubeadm alpha phase kubelet config write-to-disk --config kubeadm-master.config
kubeadm alpha phase kubelet write-env-file --config kubeadm-master.config
kubeadm alpha phase kubeconfig kubelet --config kubeadm-master.config
systemctl restart kubelet

# 添加etcd到集群中
CP0_IP="192.168.105.92"
CP0_HOSTNAME="lab1"
CP1_IP="192.168.105.93"
CP1_HOSTNAME="lab2"
export KUBECONFIG=/etc/kubernetes/admin.conf 
kubectl exec -n kube-system etcd-${CP0_HOSTNAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP0_IP}:2379 member add ${CP1_HOSTNAME} https://${CP1_IP}:2380
kubeadm alpha phase etcd local --config kubeadm-master.config

# 提前拉取鏡像
# 如果執行失敗 可以多次執行
kubeadm config images pull --config kubeadm-master.config

# 部署
kubeadm alpha phase kubeconfig all --config kubeadm-master.config
kubeadm alpha phase controlplane all --config kubeadm-master.config
kubeadm alpha phase mark-master --config kubeadm-master.config

8.3 配置第三個master

如下操作在lab3節點操作

# 1.11 版本 centos 下使用 ipvs 模式會出問題
# 參考 https://github.com/kubernetes/kubernetes/issues/65461

cd /etc/kubernetes
# 生成配置文件
CP0_IP="192.168.105.92"
CP0_HOSTNAME="lab1"
CP1_IP="192.168.105.93"
CP1_HOSTNAME="lab2"
CP2_IP="192.168.105.94"
CP2_HOSTNAME="lab3"
cat >kubeadm-master.config<<EOF
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers

apiServerCertSANs:
- "lab1"
- "lab2"
- "lab3"
- "192.168.105.92"
- "192.168.105.93"
- "192.168.105.94"
- "192.168.105.99"
- "127.0.0.1"

api:
  advertiseAddress: $CP2_IP
  controlPlaneEndpoint: 192.168.105.99:8443

etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://$CP2_IP:2379"
      advertise-client-urls: "https://$CP2_IP:2379"
      listen-peer-urls: "https://$CP2_IP:2380"
      initial-advertise-peer-urls: "https://$CP2_IP:2380"
      initial-cluster: "$CP0_HOSTNAME=https://$CP0_IP:2380,$CP1_HOSTNAME=https://$CP1_IP:2380,$CP2_HOSTNAME=https://$CP2_IP:2380"
      initial-cluster-state: existing
    serverCertSANs:
      - $CP2_HOSTNAME
      - $CP2_IP
    peerCertSANs:
      - $CP2_HOSTNAME
      - $CP2_IP

controllerManagerExtraArgs:
  node-monitor-grace-period: 10s
  pod-eviction-timeout: 10s

networking:
  podSubnet: 10.244.0.0/16

kubeProxy:
  config:
    # mode: ipvs
    mode: iptables
EOF

# 配置kubelet
kubeadm alpha phase certs all --config kubeadm-master.config
kubeadm alpha phase kubelet config write-to-disk --config kubeadm-master.config
kubeadm alpha phase kubelet write-env-file --config kubeadm-master.config
kubeadm alpha phase kubeconfig kubelet --config kubeadm-master.config
systemctl restart kubelet

# 添加etcd到集群中
CP0_IP="192.168.105.92"
CP0_HOSTNAME="lab1"
CP2_IP="192.168.105.94"
CP2_HOSTNAME="lab3"
KUBECONFIG=/etc/kubernetes/admin.conf
kubectl exec -n kube-system etcd-${CP0_HOSTNAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP0_IP}:2379 member add ${CP2_HOSTNAME} https://${CP2_IP}:2380
kubeadm alpha phase etcd local --config kubeadm-master.config

# 提前拉取鏡像
# 如果執行失敗 可以多次執行
kubeadm config images pull --config kubeadm-master.config

# 部署
kubeadm alpha phase kubeconfig all --config kubeadm-master.config
kubeadm alpha phase controlplane all --config kubeadm-master.config
kubeadm alpha phase mark-master --config kubeadm-master.config

9. 配置使用kubectl

如下操作在任意master節點操作

rm -rf $HOME/.kube
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 查看node節點
kubectl get nodes

# 只有網絡插件也安裝配置完成之後，才能會顯示為ready狀態
# 設置master允許部署應用pod，參與工作負載，現在可以部署其他系統組件
# 如 dashboard, heapster, efk等
kubectl taint nodes --all node-role.kubernetes.io/master-

10. 配置使用網絡插件

如下操作在任意master節點操作

# 下載配置
cd /etc/kubernetes
mkdir flannel && cd flannel
wget https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml

# 修改配置
# 此處的ip配置要與上面kubeadm的pod-network一致
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }

# 修改鏡像
image: registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64

# 如果Node有多個網卡的話，參考flannel issues 39701，
# https://github.com/kubernetes/kubernetes/issues/39701
# 目前需要在kube-flannel.yml中使用--iface參數指定集群主機內網網卡的名稱，
# 否則可能會出現dns無法解析。容器無法通信的情況，需要將kube-flannel.yml下載到本地，
# flanneld啟動參數加上--iface=<iface-name>
    containers:
      - name: kube-flannel
        image: registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface=ens32

# 啟動
kubectl apply -f kube-flannel.yml

# 查看
kubectl get pods --namespace kube-system
kubectl get svc --namespace kube-system

11. 配置node節點加入集群

如下操作在所有node節點操作

# 此命令為初始化master成功後返回的結果
kubeadm join 192.168.105.99:8443 --token j6zjtl.tgptijigkhhnuc23 --discovery-token-ca-cert-hash sha256:f3e9ae0841084185649b6c111b7e992465b81f2442d42871c6a15731a17dabba

node節點報錯處理辦法：

tail -f /var/log/message

Jul 26 07:52:21 localhost kubelet: E0726 07:52:21.336281   10018 summary.go:102] Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"

在kubelet配置文件追加以下配置
/etc/sysconfig/kubelet

# Append configuration in Kubelet
--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice

12. 配置dashboard

默認是沒web界面的，可以在master機器上安裝一個dashboard插件，實現通過web來管理。

12.1 安裝Dashboard插件

如下操作在任意master節點操作

cd /etc/kubernetes
wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml  # 下載

編輯kubernetes-dashboard.yaml文件`：

      - name: kubernetes-dashboard
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.8.3

執行命令

# 安裝Dashboard插件
kubectl create -f kubernetes-dashboard.yaml
kubectl get svc,pod --all-namespaces | grep dashboard

可以看到kubernetes-dashboard已正常運行。

kube-system   service/kubernetes-dashboard   NodePort    10.108.96.71   <none>        443:30356/TCP   1m
kube-system   pod/kubernetes-dashboard-754f4d5f69-nfvrk   0/1       CrashLoopBackOff   3          1m

12.2 授予Dashboard賬戶集群管理權限

需要一個管理集群admin的權限，新建kubernetes-dashboard-admin.rbac.yaml文件，內容如下

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
---
# Create ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system

執行命令

kubectl create -f kubernetes-dashboard-admin.rbac.yaml

找到kubernete-dashboard-admin的token，用戶登錄使用

執行命令並查看結果

[root@lab1 kubernetes]# kubectl -n kube-system get secret | grep admin-user
admin-user-token-b9mpt                           kubernetes.io/service-account-token   3         29s

可以看到名稱是kubernetes-dashboard-admin-token-ddskx，使用該名稱執行如下命令

[root@lab1 kubernetes]# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk ‘{print $1}‘)
Name:         admin-user-token-b9mpt
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name=admin-user
              kubernetes.io/service-account.uid=f1247ca8-9173-11e8-bbc3-000c29ea3e30

Type:  kubernetes.io/service-account-token

Data
====
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLWI5bXB0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmMTI0N2NhOC05MTczLTExZTgtYmJjMy0wMDBjMjllYTNlMzAiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.g9F6vn84ds6iVi1TJViWaK1oHMsY0vQoV5Xq8n0nF5WF4uJkkToHxs0nHO4G4u927ZWsMuF0JiTD4rKB0uJcnHdc4GwBN6L2XMceD69YCpunLlgoFOFbu6z9IsZfyvHvFAYbm0Tv4JWBYxkCqmeTKtL1GOtobIs24dvfXk6inn51ZhTAUW_urWvIn8yqckOBJkq7B_wf6EZA0QeNEhhbt_GHPpwq0CMhk4cWHXh_a27y-qKkpu5Cbo_Ux2kUA44o1wmiHgbw4Lh-__KJY3LZmIu9PZzyWyIPaUWlSQ76GXHyOZQ8dw3WRANi9zaEpiwi4e4XXXXXXXXXXXXXXXXXXXXXXX
ca.crt:     1025 bytes
namespace:  11 bytes

記下這串token，等下登錄使用，這個token默認是永久的。

12.3 dashboard訪問方式

此處推薦API Server方式訪問。（谷歌內核瀏覽器）

12.3.1 `kubectl proxy`方式訪問

如下操作在lab1上操作

kubectl proxy --address=0.0.0.0 --disable-filter=true

即可通過瀏覽器訪問: http://192.168.105.92:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login

註意
儀表盤使用kubectl代理命令不應暴露公開,因為它只允許HTTP連接。域以外的localhost和127.0.0.1將不能登錄。在登錄頁面點擊登錄按鈕什麽都不會發生後，跳過登錄後，沒有任何權限。

此方式只允許開發測試使用。為了便於開發測試，以下配置用於提升默認權限為超級用戶權限。

vim kubernetes-dashboard-test.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
---
# Create ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system

12.3.2 NodePort方式訪問

這種訪問方式儀表板只建議在單個節點上設置開發環境。

編輯kubernetes-dashboard.yaml文件，添加type: NodePort和nodePort: 30001，暴露Dashboard服務為30001端口，參考如下。

# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  type: NodePort  # NodePort登錄方式
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001  # NodePort登錄暴露端口
  selector:
    k8s-app: kubernetes-dashboard

註意
儀表盤可以在master節點上訪問，如果是多節點集群，官方文檔說應該是使用節點IP和NodePort來訪問，但是經過測試，https://<master-ip>:<nodePort> 和 https://<node-ip>:<nodePort>都可以訪問。

12.3.3 API Server方式訪問

https://<master-ip>:<apiserver-port>/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

註意
這種方式訪問儀表盤的僅僅可能在安裝了你的用戶證書的瀏覽器上。與API Server通信可以使用示例所使用的證書kubeconfig文件。

瀏覽器訪問問題：

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "services \"https:kubernetes-dashboard:\" is forbidden: User \"system:anonymous\" cannot get services/proxy in the namespace \"kube-system\"",
  "reason": "Forbidden",
  "details": {
    "name": "https:kubernetes-dashboard:",
    "kind": "services"
  },
  "code": 403
}

這是因為最新版的k8s默認啟用了RBAC，並為未認證用戶賦予了一個默認的身份：anonymous。

對於API Server來說，它是使用證書進行認證的，我們需要先創建一個證書：

首先找到kubectl命令的配置文件，默認情況下為/etc/kubernetes/admin.conf，在上文中，我們已經復制到了$HOME/.kube/config中。
然後我們使用client-certificate-data和client-key-data生成一個p12文件，可使用下列命令：

# 生成client-certificate-data
grep ‘client-certificate-data‘ ~/.kube/config | head -n 1 | awk ‘{print $2}‘ | base64 -d >> kubecfg.crt

# 生成client-key-data
grep ‘client-key-data‘ ~/.kube/config | head -n 1 | awk ‘{print $2}‘ | base64 -d >> kubecfg.key

# 生成p12
openssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12 -name "kubernetes-client"

最後導入上面生成的p12文件，重新打開瀏覽器，顯示出現選擇證書選項，選OK，然後就可以看到熟悉的登錄界面了。我們可以使用一開始創建的admin-user用戶的token進行登錄，一切OK。

註意
對於生產系統，我們應該為每個用戶應該生成自己的證書，因為不同的用戶會有不同的命名空間訪問權限。

12.3.4 nginx ingress方式訪問

可以動態的更新Nginx配置等，是比較靈活，更為推薦的暴露服務的方式，但也相對比較復雜，業務環境推薦使用。

13. 基礎測試

測試容器間的通信和DNS
配置好網絡之後，kubeadm會自動部署coredns

如下測試可以在配置kubectl的節點上操作

# 啟動名為nginx的容器
kubectl run nginx --replicas=2 --image=nginx:alpine --port=80
# 暴露nginx容器為服務（--type=NodePort/ClusterIP/LoadBalancer，3種類型訪問方式不同）
kubectl expose deployment nginx --type=NodePort --name=example-service-nodeport
kubectl expose deployment nginx --name=example-service
# 查看狀態
kubectl get deploy
kubectl get pods
kubectl get svc
kubectl describe svc example-service
# DNS解析
kubectl run curl --image=radial/busyboxplus:curl -i --tty
nslookup kubernetes
nslookup example-service
curl example-service
# 訪問測試
# 10.103.184.0 為查看svc時獲取到的clusterip
curl "10.103.184.0:80"

# 32223 為查看svc時獲取到的 nodeport
http://192.168.105.93:32223/
http://192.168.105.94:32223/
# 清理刪除
kubectl delete svc example-service example-service-nodeport
kubectl delete deploy nginx curl
# 高可用測試
# 關閉任一master節點測試集群是能否正常執行上一步的基礎測試，查看相關信息，不能同時關閉兩個節點，因為3個節點組成的etcd集群，最多只能有一個當機。

# 查看組件狀態
kubectl get pod --all-namespaces -o wide
kubectl get pod --all-namespaces -o wide | grep lab1
kubectl get pod --all-namespaces -o wide | grep lab2
kubectl get pod --all-namespaces -o wide | grep lab3
kubectl get nodes -o wide
kubectl get deploy
kubectl get pods
kubectl get svc

# 訪問測試
CURL_POD=$(kubectl get pods | grep curl | grep Running | cut -d ‘ ‘ -f1)
kubectl exec -it $CURL_POD -- sh --tty
nslookup kubernetes
nslookup example-service
curl example-service

13. 小技巧

忘記初始master節點時的node節點加入集群命令怎麽辦

# 簡單方法
kubeadm token create --print-join-command

# 第二種方法
token=$(kubeadm token generate)
kubeadm token create $token --print-join-command --ttl=0

14. 相關命令

# 查看集群結點狀態
kubectl get nodes
# 查看詳細結點信息
kubectl describe nodes
# 查看所有pod
kubectl get pods --all-namespaces
# 查看集群服務狀態
kubectl get svc --all-namespaces
# 查看集群運行在那些ip上
kubectl cluster-info
# 查看master的各種token
kubectl get secret -n kube-system
# 查看某一個特定的token
kubectl describe secret/[token name] -n kube-system

參考文檔：
[1] https://kubernetes.io/docs/setup/independent/install-kubeadm/
[2] https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
[3] https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
[4] https://kubernetes.io/docs/setup/independent/high-availability/
[5] https://sealyun.com/post/k8s-ipvs/
[6] http://www.maogx.win/posts/33/
[7] https://github.com/opsnull/follow-me-install-kubernetes-cluster
[8] https://github.com/xizhibei/blog/issues/64
[9] https://www.cnblogs.com/RainingNight/p/deploying-k8s-dashboard-ui.html

centos7使用kubeadm安裝kubernetes 1.11版本多主高可用

ips refresh www ember issues abi dash iyu let centos7使用kubeadm安裝kubernetes 1.11版本多主高可用 [TOC] kubernetes介紹要學習一個新的東西，先了解它是什麽，熟悉基本概念會有很大幫助。以

centos7使用kubeadm安裝kubernetes 1.11版本多主高可用

1. 實驗環境說明

2. 準備yum源

2. 安裝配置docker

3. 安裝 kubeadm, kubelet 和 kubectl

4. 配置系統相關參數

5. 配置hosts解析

6. 配置haproxy代理和keepalived

7. 配置啟動kubelet

8. 配置master

8.1 配置第一個master

8.2 配置第二個master

8.3 配置第三個master

9. 配置使用kubectl

10. 配置使用網絡插件

11. 配置node節點加入集群

12. 配置dashboard

12.1 安裝Dashboard插件

12.2 授予Dashboard賬戶集群管理權限

12.3 dashboard訪問方式

12.3.1 kubectl proxy方式訪問

12.3.2 NodePort方式訪問

12.3.3 API Server方式訪問

12.3.4 nginx ingress方式訪問

13. 基礎測試

13. 小技巧

14. 相關命令

相關推薦

12.3.1 `kubectl proxy`方式訪問