1. 程式人生 > >15分鐘在阿裏雲Kubernetes服務上快速建立Jenkins X Platform並運用GitOps管理應用發布

15分鐘在阿裏雲Kubernetes服務上快速建立Jenkins X Platform並運用GitOps管理應用發布

cati rip static which complete ccf trying dmi all

本文主要介紹如何在阿裏雲容器服務Kubernetes上快速安裝部署Jenkins X Platform並結合demo實踐演示GitOps的操作流程。

註意:
本文中使用的jx工具、cloud-environments等做過改造用以適配阿裏雲Kubernetes容器服務,並未在自建Kubernetes集群中做過驗證。

先決條件:
首先,需要在 阿裏雲容器服務控制臺 創建一個Kubernetes集群,本次實踐使用的環境信息如下:
master1 192.168.0.119
master2 192.168.0.120
master3 192.168.0.121
worker1 192.168.0.122
jx-node 192.168.0.123

一、 部署Jenkins X Platform並創建Staging Production Env

1. kubectl 、helm、git和jx的安裝以及初始化

$ scp 192.168.0.119:/usr/bin/kubectl /usr/bin/kubectl
$ scp 192.168.0.119:/usr/local/bin/helm /usr/local/bin/helm
$ mkdir -p $HOME/.kube

將容器服務控制臺上顯示的kubeconfig文件內容復制到計算機 $HOME/.kube/config

初始化Helm Client:

$ helm init --client-only --stable-repo-url https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts-incubator/
$ helm repo update

安裝git

$ yum install git -y

安裝jx

$ wget http://acs-jenkins-x.oss-cn-beijing.aliyuncs.com/v1.3.699/jx && mv jx /usr/local/bin/ && chmod +x /usr/local/bin/jx

驗證jx:(註意不要選擇升級,jx做了定制化修改,目前版本1.3.699)

$ jx version
Error loading team settings. the server could not find the requested resource (get environments.jenkins.io)
Error loading team settings. the server could not find the requested resource (get environments.jenkins.io)
Failed to get git version: failed to run ‘git version‘ command in directory ‘‘, output: ‘‘: exec: "git": executable file not found in $PATH
NAME               VERSION
jx                 1.3.699
Kubernetes cluster v1.11.5
kubectl            v1.11.5
helm client        v2.11.0+g2e55dbe
helm server        v2.11.0+g2e55dbe
Operating System   "CentOS Linux release 7.4.1708 (Core) "

A new jx version is available: 1.3.736
? Would you like to upgrade to the new jx version? No

2. 安裝jenkins-x-platform

$ mkdir -p ~/.jx
$ cd ~/.jx
$ jx install --provider=kubernetes              --domain aliyunk8s-bj.com              --ingress-deployment=nginx-ingress-controller              --ingress-service=nginx-ingress-lb               --cloud-environment-repo=https://github.com/AliyunContainerService/cloud-environments.git              --no-tiller=true              --no-default-environments=true

交互及日誌輸出如下:

Namespace jx created
 Using helmBinary helm with feature flag: none
Context "kubernetes-admin-ccf3ebc497d9e4083aab8c89bb1b217bc" modified.
Storing the kubernetes provider kubernetes in the TeamSettings
Enabling helm template mode in the TeamSettings
? Please enter the name you wish to use with git:  jenkins-x-bot
? Please enter the email address you wish to use with git:  [email protected]
Git configured for user: jenkins-x-bot and email [email protected]
Trying to create ClusterRoleBinding kubernetes-admin-cluster-admin-binding for role: cluster-admin for user kubernetes-admin
 clusterrolebindings.rbac.authorization.k8s.io "kubernetes-admin-cluster-admin-binding" not found
Created ClusterRoleBinding kubernetes-admin-cluster-admin-binding
Using helm2
Skipping tiller
helm installed and configured
existing ingress controller found, no need to install a new one
Waiting for external loadbalancer to be created and update the nginx-ingress-controller service in kube-system namespace
External loadbalancer created
Waiting to find the external host name of the ingress controller Service in namespace kube-system with name nginx-ingress-lb
If you are installing Jenkins X on premise you may want to use the ‘--on-premise‘ flag or specify the ‘--external-ip‘ flags. See: https://jenkins-x.io/getting-started/install-on-cluster/#installing-jenkins-x-on-premise
You can now configure your wildcard DNS aliyunk8s-bj.com to point to 39.97.21.40
nginx ingress controller installed and configured
Lets set up a Git user name and API token to be able to perform CI/CD

Creating a local Git user for GitHub server
? GitHub user name: jenkins-x-bot
To be able to create a repository on GitHub we need an API Token
Please click this URL https://github.com/settings/tokens/new?scopes=repo,read:user,read:org,user:email,write:repo_hook,delete_repo

Then COPY the token and enter in into the form below:

? API Token: ****************************************
Select the CI/CD pipelines Git server and user
? Do you wish to use GitHub as the pipelines Git server: Yes
? Do you wish to use jenkins-x-bot as the pipelines Git user for GitHub server: Yes
Setting the pipelines Git server https://github.com and user name jenkins-x-bot.
Saving the Git authentication configurationCurrent configuration dir: /root/.jx
options.Flags.CloudEnvRepository: https://github.com/AliyunContainerService/cloud-environments.git
options.Flags.LocalCloudEnvironment: false
Cloning the Jenkins X cloud environments repo to /root/.jx/cloud-environments
Enumerating objects: 1317, done.
Counting objects: 100% (1317/1317), done.
Compressing objects: 100% (618/618), done.
Total 1317 (delta 673), reused 1317 (delta 673), pack-reused 0
No default password set, generating a random one
Generated helm values /root/.jx/extraValues.yaml
Creating Secret jx-install-config in namespace jx
Installing Jenkins X platform helm chart from: /root/.jx/cloud-environments/env-kubernetes
? Select Jenkins installation type: Static Master Jenkins
? Pick workload build pack:  Kubernetes Workloads: Automated CI+CD with GitOps Promotion
Setting the team build pack to kubernetes-workloads repo: https://github.com/AliyunContainerService/jenkins-x-kubernetes.git ref: master
Installing jx into namespace jx
Adding values file /root/.jx/cloud-environments/env-kubernetes/myvalues.yaml
Adding values file /root/.jx/adminSecrets.yaml
Adding values file /root/.jx/extraValues.yaml
Adding values file /root/.jx/cloud-environments/env-kubernetes/secrets.yaml
Fetched chart jenkins-x/jenkins-x-platform to dir /tmp/helm-template-workdir-860490114/jenkins-x/chartFiles/jenkins-x-platform
Generating Chart Template ‘template --name jenkins-x --namespace jx /tmp/helm-template-workdir-860490114/jenkins-x/chartFiles/jenkins-x-platform --output-dir /tmp/helm-template-workdir-860490114/jenkins-x/output --debug --values /root/.jx/cloud-environments/env-kubernetes/myvalues.yaml --values /root/.jx/adminSecrets.yaml --values /root/.jx/extraValues.yaml --values /root/.jx/cloud-environments/env-kubernetes/secrets.yaml‘
Applying generated chart jenkins-x/jenkins-x-platform YAML via kubectl in dir: /tmp/helm-template-workdir-860490114/jenkins-x/output
deployment.extensions/jenkins-x-chartmuseum created
persistentvolumeclaim/jenkins-x-chartmuseum created
secret/jenkins-x-chartmuseum created
service/jenkins-x-chartmuseum created
role.rbac.authorization.k8s.io/cleanup created
rolebinding.rbac.authorization.k8s.io/cleanup created
serviceaccount/cleanup created
clusterrole.rbac.authorization.k8s.io/controllercommitstatus-jx created
clusterrolebinding.rbac.authorization.k8s.io/controllercommitstatus-jx created
deployment.apps/jenkins-x-controllercommitstatus created
role.rbac.authorization.k8s.io/controllercommitstatus created
rolebinding.rbac.authorization.k8s.io/controllercommitstatus created
serviceaccount/jenkins-x-controllercommitstatus created
clusterrole.rbac.authorization.k8s.io/controllerrole-jx created
clusterrolebinding.rbac.authorization.k8s.io/controllerrole-jx created
deployment.apps/jenkins-x-controllerrole created
role.rbac.authorization.k8s.io/controllerrole created
rolebinding.rbac.authorization.k8s.io/controllerrole created
serviceaccount/jenkins-x-controllerrole created
clusterrole.rbac.authorization.k8s.io/controllerteam-jx created
clusterrolebinding.rbac.authorization.k8s.io/controllerteam-jx created
deployment.apps/jenkins-x-controllerteam created
role.rbac.authorization.k8s.io/controllerteam created
rolebinding.rbac.authorization.k8s.io/controllerteam created
serviceaccount/jenkins-x-controllerteam created
clusterrole.rbac.authorization.k8s.io/controllerworkflow-jx created
clusterrolebinding.rbac.authorization.k8s.io/controllerworkflow-jx created
deployment.apps/jenkins-x-controllerworkflow created
role.rbac.authorization.k8s.io/controllerworkflow created
rolebinding.rbac.authorization.k8s.io/controllerworkflow created
serviceaccount/jenkins-x-controllerworkflow created
configmap/exposecontroller created
role.rbac.authorization.k8s.io/expose created
rolebinding.rbac.authorization.k8s.io/expose created
serviceaccount/expose created
clusterrole.rbac.authorization.k8s.io/gcactivities-jx created
clusterrolebinding.rbac.authorization.k8s.io/gcactivities-jx created
cronjob.batch/jenkins-x-gcactivities created
role.rbac.authorization.k8s.io/gcactivities created
rolebinding.rbac.authorization.k8s.io/gcactivities created
serviceaccount/jenkins-x-gcactivities created
cronjob.batch/jenkins-x-gcpods created
role.rbac.authorization.k8s.io/gcpods created
rolebinding.rbac.authorization.k8s.io/gcpods created
serviceaccount/jenkins-x-gcpods created
clusterrole.rbac.authorization.k8s.io/gcpreviews-jx created
clusterrolebinding.rbac.authorization.k8s.io/gcpreviews-jx created
cronjob.batch/jenkins-x-gcpreviews created
role.rbac.authorization.k8s.io/gcpreviews created
rolebinding.rbac.authorization.k8s.io/gcpreviews created
serviceaccount/jenkins-x-gcpreviews created
deployment.extensions/jenkins-x-heapster created
service/heapster created
configmap/jenkins created
persistentvolumeclaim/jenkins created
service/jenkins-agent created
deployment.extensions/jenkins created
service/jenkins created
configmap/jenkins-x-git-kinds created
clusterrolebinding.rbac.authorization.k8s.io/jenkins-jx-role-binding created
secret/jenkins created
serviceaccount/jenkins created
configmap/jenkins-tests created
configmap/nexus created
deployment.extensions/jenkins-x-nexus created
persistentvolumeclaim/jenkins-x-nexus created
secret/nexus created
service/nexus created
role.rbac.authorization.k8s.io/committer created
clusterrolebinding.rbac.authorization.k8s.io/jenkins-x-team-controller created
configmap/jenkins-x-team-controller created
secret/jenkins-docker-cfg created
configmap/jenkins-x-devpod-config created
configmap/jenkins-x-docker-registry created
configmap/jenkins-x-extensions created
configmap/jenkins-x-pod-templates created
secret/jx-basic-auth created
role.rbac.authorization.k8s.io/jx-view created
secret/jenkins-maven-settings created
secret/jenkins-npm-token created
role.rbac.authorization.k8s.io/owner created
secret/jenkins-release-gpg created
secret/jenkins-ssh-config created
role.rbac.authorization.k8s.io/viewer created
Applying Helm hook post-upgrade YAML via kubectl in file: /tmp/helm-template-workdir-860490114/jenkins-x/helmHooks/jenkins-x-platform/charts/expose/templates/job.yaml
job.batch/expose created
Waiting for helm post-upgrade hook Job expose to complete before removing it

如果是首次部署jenkins x platfrom的話,此處需等待幾分鐘進行鏡像的拉取和容器的啟動:

$ kubectl -n jx get po
NAME                                                READY     STATUS              RESTARTS   AGE
expose-2bhmv                                        0/1       ContainerCreating   0          1m
jenkins-568884c766-sxc7d                            0/1       Init:0/1            0          1m
jenkins-x-chartmuseum-6cf566bfb-jm7cv               0/1       ContainerCreating   0          1m
jenkins-x-controllercommitstatus-6f57d857d8-kzllt   1/1       Running             0          1m
jenkins-x-controllerrole-57d864c96f-4lrrk           1/1       Running             0          1m
jenkins-x-controllerteam-c48fc44f-tsrkh             0/1       ContainerCreating   0          1m
jenkins-x-controllerworkflow-c758649d6-2kpfj        0/1       ContainerCreating   0          1m
jenkins-x-heapster-7fbdb867d9-wt6sl                 0/2       ContainerCreating   0          1m
jenkins-x-nexus-5d5455cfd7-4xsck                    0/1       ContainerCreating   0          1m

接下來的日誌交互及輸出如下:

^@^@^@Deleting helm hook sources from file: /tmp/helm-template-workdir-860490114/jenkins-x/helmHooks/jenkins-x-platform/charts/expose/templates/job.yaml
job.batch "expose" deleted
Removing Kubernetes resources from older releases using selector: jenkins.io/chart-release=jenkins-x,jenkins.io/version!=0.0.3193
waiting for install to be ready, if this is the first time then it will take a while to download images
^@Jenkins X deployments ready in namespace jx


    ********************************************************

         NOTE: Your admin password is: qub6n#mKkh0oN!S59nPp

    ********************************************************


Getting Jenkins API Token
Using url http://jenkins.jx.aliyunk8s-bj.com/me/configure
Unable to automatically find API token with chromedp using URL http://jenkins.jx.aliyunk8s-bj.com/me/configure
Error: creating the chrome client: fork/exec /usr/bin/google-chrome: no such file or directory
Please go to http://jenkins.jx.aliyunk8s-bj.com/me/configure and click Show API Token to get your API Token
Then COPY the token and enter in into the form below:

? API Token:

此處如果沒有DNS service解析域名jenkins.jx.aliyunk8s-bj.com的話, 需要先在jx-node上手動綁定hosts:

$ kubectl -n jx get ing
NAME          HOSTS                             ADDRESS       PORTS     AGE
chartmuseum   chartmuseum.jx.aliyunk8s-bj.com   xx.xx.xx.xx   80        5m
jenkins       jenkins.jx.aliyunk8s-bj.com       xx.xx.xx.xx   80        5m
nexus         nexus.jx.aliyunk8s-bj.com         xx.xx.xx.xx   80        5m
$ echo "xx.xx.xx.xx jenkins.jx.aliyunk8s-bj.com" >> /etc/hosts

根據日誌提示生成jenkins api token:
技術分享圖片
完成安裝:

Created user admin API Token for Jenkins server jenkins.jx.aliyunk8s-bj.com at http://jenkins.jx.aliyunk8s-bj.com
Updating Jenkins with new external URL details http://jenkins.jx.aliyunk8s-bj.com

Jenkins X installation completed successfully


    ********************************************************

         NOTE: Your admin password is: qub6n#mKkh0oN!S59nPp

    ********************************************************



Your Kubernetes context is now set to the namespace: jx
To switch back to your original namespace use: jx namespace default
For help on switching contexts see: https://jenkins-x.io/developing/kube-context/

To import existing projects into Jenkins:       jx import
To create a new Spring Boot microservice:       jx create spring -d web -d actuator
To create a new microservice from a quickstart: jx create quickstart

3. 創建staging env

$ jx create env -n staging -l Staging --namespace staging --fork-git-repo=‘https://github.com/haoshuwei/default-environment-charts.git‘ --domain=aliyunk8s-bj.com --promotion=Auto --prefix=‘bj‘

....
Creating GitHub webhook for haoshuwei/environment-bj-staging for url http://jenkins.jx.aliyunk8s-bj.com/github-webhook/

由於我們默認使用Ingress暴露服務, 所以需要修改environment-bj-staging的webhook:

$ kubectl -n jx get svc |grep LoadBalancer
kubectl -n jx get svc |grep LoadBalancer
jenkins                 LoadBalancer   172.19.11.179   xx.xx.xx.xx   8080:30456/TCP   13h

技術分享圖片

jenkins會自動創建environment-bj-staging job並進行掃描和構建:
技術分享圖片

4. 創建production env:

$ jx create env -n production -l Production --namespace production --fork-git-repo=‘https://github.com/haoshuwei/default-environment-charts.git‘ --domain=aliyunk8s-bj.com --promotion=Manual --prefix=‘bj‘

...
Creating GitHub webhook for haoshuwei/environment-bj-production for url http://jenkins.jx.aliyunk8s-bj.com/github-webhook/

修改environment-hz-production的webhook。

jenkins會自動創建environment-bj-production job並進行掃描和構建:
技術分享圖片

5. 配置訪問阿裏雲容器鏡像服務的docker registry secret:

$ kubectl -n jx delete secrets jenkins-docker-cfg
secret "jenkins-docker-cfg" deleted
$ docker login -u xxx -p xxx registry.cn-beijing.aliyuncs.com
$ kubectl create secret generic jenkins-docker-cfg -n jx --from-file=/root/.docker/config.json
secret/jenkins-docker-cfg created

二、 運用GitOps管理應用發布

1. 安裝openjdk

$ yum install -y java-1.8.0-openjdk

2. 創建一個Spring示例應用

$ jx create spring -d web -d actuator
? Language: java
? Group: com.example
? Artifact: jenkins-x-demo
Created Spring Boot project at /root/.jx/jenkins-x-demo
No username defined for the current Git server!
? Do you wish to use jenkins-x-bot as the Git user name: Yes
The directory /root/.jx/jenkins-x-demo is not yet using git
? Would you like to initialise git now? Yes
? Commit message:  Initial import

Git repository created
selected pack: /root/.jx/draft/packs/github.com/AliyunContainerService/jenkins-x-kubernetes/packs/maven
? Which organisation do you want to use? jenkins-x-bot
replacing placeholders in directory /root/.jx/jenkins-x-demo
app name: jenkins-x-demo, git server: github.com, org: jenkins-x-bot, Docker registry org: jenkins-x-bot
skipping directory "/root/.jx/jenkins-x-demo/.git"
Using Git provider GitHub at https://github.com


About to create repository jenkins-x-demo on server https://github.com with user jenkins-x-bot
? Enter the new repository name:  jenkins-x-demo


Creating repository jenkins-x-bot/jenkins-x-demo
Pushed Git repository to https://github.com/haoshuwei/jenkins-x-demo

Created Jenkins Project: http://jenkins.jx.aliyunk8s-bj.com/job/haoshuwei/job/jenkins-x-demo/

Watch pipeline activity via:    jx get activity -f jenkins-x-demo -w
Browse the pipeline log via:    jx get build logs haoshuwei/jenkins-x-demo/master
Open the Jenkins console via    jx console
You can list the pipelines via: jx get pipelines
When the pipeline is complete:  jx get applications

For more help on available commands see: https://jenkins-x.io/developing/browsing/

Note that your first pipeline may take a few minutes to start while the necessary images get downloaded!

Creating GitHub webhook for haoshuwei/jenkins-x-demo for url http://jenkins.jx.aliyunk8s-bj.com/github-webhook/

修改 jenkins-x-demo的webhook。

jenkins-x-demo項目成功創建後,jenkins job jenkins-x-demo會自動掃描並構建jenkins-x-demo-0.0.1.tgz發布到chartmuseum並自動promote到staging環境:
技術分享圖片
查看environment-bj-staging項目的PR:
技術分享圖片
PR的Check通過後會自動merge:
技術分享圖片
主要修改內容為:
技術分享圖片
然後environment-bj-staging項目自動構建並部署jenkins-x-demo應用到Staging Env。

3. 訪問Staging環境中的jenkins-x-demo應用服務

$ jx get apps
APPLICATION       PRODUCTION PODS URL                                            STAGING PODS URL
jx-jenkins-x-demo 0.0.1      1/1  http://jenkins-x-demo.staging.aliyunk8s-bj.com

訪問http://jenkins-x-demo.staging.aliyunk8s-bj.com:
技術分享圖片

4. 開發jenkins-x-demo應用並發布新版本到Staging環境

步驟3中, 我們訪問了一個並沒有準備好進入生產環境的jenkins-x-demo應用, 下面我們添加一個新的feature:

$ cd jenkins-x-demo
$ git checkout -b feature/add-index

新建src/main/resources/static/index.html並寫入如下內容:

<html>
  <head>
    <title> Jenkins X Spring Demo </title>
  </head>
  <body bgcolor=white>

    <table border="0" cellpadding="10">
      <tr>
        <td>
          <img height="300" width="300" src="https://github.com/jenkins-x/jenkins-x-website/raw/e5aae999166a67d6220aa469eed1f23e0996c1f7/images/logo.png">
        </td>
        <td>
          <h1>Jenkins X Spring Demo</h1>
        </td>
      </tr>
    </table>
  </body>

</html>

提交和推送後我們在GitHub上創建一個feature/add-index到master分支的PR:
技術分享圖片
jenkins會自動對PR進行構建和部署,這個過程中jenkins-x-demo的部署並不是部署到Staging環境,而是會部署為Preview環境提供預覽:
技術分享圖片
技術分享圖片
預覽jenkins-x-demo應用http://jenkins-x-demo.jx-haoshuwei-jenkins-x-demo-pr-1.aliyunk8s-bj.com/:
技術分享圖片

5. 發布新版本jenkins-x-demo應用到Staging環境

確認提交的PR沒問題後,即可merge到master分支:
技術分享圖片

jenkins會重新構建jenkins-x-demo v0.0.2並發布到Staging環境:

$ jx get apps
APPLICATION       PRODUCTION PODS URL                                            STAGING PODS URL
jx-jenkins-x-demo 0.0.2      1/1  http://jenkins-x-demo.staging.aliyunk8s-bj.com

6. jenkins-x-demo應用推送到Production環境

一旦應用通過測試驗證並準備好進入生產後, 我們就可以手動發布應用到Production環境:

$ jx get apps
APPLICATION       PRODUCTION PODS URL                                               STAGING PODS URL
jx-jenkins-x-demo 0.0.2      1/1  http://jenkins-x-demo.production.aliyunk8s-bj.com 0.0.2   1/1  http://jenkins-x-demo.staging.aliyunk8s-bj.com

訪問Production環境中的jenkins-x-demo:
技術分享圖片

三、 Production環境中應用的回滾

回滾到版本v0.x.x

$ jx promote --version 0.x.x --env production --timeout 20m

四、 刪除或重裝Jenkins X Platform

$ kubectl config current-context
$ jx uninstall
$ cd ~/.jx 
$ rm -rf *

五、 結束語

目前Jenkins X Platform還在不斷完善和改進中, 比如支持多集群等問題https://github.com/jenkins-x/jx/issues/479, 我們也會持續優化Jenkins X 在阿裏雲Kubernetes容器服務上的最佳實踐並更新博客。

原文鏈接
更多技術幹貨 請關註阿裏雲雲棲社區微信號 :yunqiinsight

15分鐘在阿裏雲Kubernetes服務上快速建立Jenkins X Platform並運用GitOps管理應用發布