歡迎訪問我的GitHub

https://github.com/zq2599/blog_demos

內容：所有原創文章分類彙總及配套原始碼，涉及Java、Docker、Kubernetes、DevOPS等；

系列文章連結

  1. kubebuilder實戰之一：準備工作
  2. kubebuilder實戰之二：初次體驗kubebuilder
  3. kubebuilder實戰之三：基礎知識速覽
  4. kubebuilder實戰之四：operator需求說明和設計
  5. kubebuilder實戰之五：operator編碼
  6. kubebuilder實戰之六：構建部署執行
  7. kubebuilder實戰之七：webhook
  8. kubebuilder實戰之八：知識點小記

本篇概覽

本文是《kubebuilder實戰》系列的第二篇，前文將kubebuilder環境準備完畢，今天咱們在此環境建立CRD和Controller，再部署到kubernetes環境並且驗證是否生效，整篇文章由以下內容組成：

  1. 建立API(CRD和Controller)
  2. 構建和部署CRD
  3. 編譯和執行controller
  4. 建立CRD對應的例項
  5. 刪除例項並停止controller
  6. 將controller製作成docker映象
  7. 解除安裝和清理

建立helloworld專案

  1. 執行以下命令，建立helloworld專案：
mkdir -p $GOPATH/src/helloworld

cd $GOPATH/src/helloworld

kubebuilder init --domain com.bolingcavalry
  1. 控制檯輸出類似以下內容：
[root@kubebuilder helloworld]# kubebuilder init --domain com.bolingcavalry

Writing scaffold for you to edit...

Get controller runtime:

$ go get sigs.k8s.io/[email protected]

Update go.mod:

$ go mod tidy

Running make:

$ make

/root/gopath/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."

go fmt ./...

go vet ./...

go build -o bin/manager main.go

Next: define a resource with:

$ kubebuilder create api
  1. 等待數分鐘後建立完成，在$GOPATH/src/helloworld目錄下新增以下內容，可見這是個標準的go module工程：
[root@kubebuilder ~]# tree $GOPATH/src/helloworld

/root/gopath/src/helloworld

├── bin

│   └── manager

├── config

│   ├── certmanager

│   │   ├── certificate.yaml

│   │   ├── kustomization.yaml

│   │   └── kustomizeconfig.yaml

│   ├── default

│   │   ├── kustomization.yaml

│   │   ├── manager_auth_proxy_patch.yaml

│   │   ├── manager_webhook_patch.yaml

│   │   └── webhookcainjection_patch.yaml

│   ├── manager

│   │   ├── kustomization.yaml

│   │   └── manager.yaml

│   ├── prometheus

│   │   ├── kustomization.yaml

│   │   └── monitor.yaml

│   ├── rbac

│   │   ├── auth_proxy_client_clusterrole.yaml

│   │   ├── auth_proxy_role_binding.yaml

│   │   ├── auth_proxy_role.yaml

│   │   ├── auth_proxy_service.yaml

│   │   ├── kustomization.yaml

│   │   ├── leader_election_role_binding.yaml

│   │   ├── leader_election_role.yaml

│   │   └── role_binding.yaml

│   └── webhook

│       ├── kustomization.yaml

│       ├── kustomizeconfig.yaml

│       └── service.yaml

├── Dockerfile

├── go.mod

├── go.sum

├── hack

│   └── boilerplate.go.txt

├── main.go

├── Makefile

└── PROJECT

9 directories, 30 files

建立API(CRD和Controller)

  1. 接下來要要建立資源相關的內容了，group/version/kind這三部分可以確定資源的唯一身份，命令如下：
cd $GOPATH/src/helloworld

kubebuilder create api \

--group webapp \

--version v1 \

--kind Guestbook
  1. 控制檯會提醒是否建立資源(Create Resource [y/n])，輸入y
  2. 接下來控制檯會提醒是否建立控制器(Create Controller [y/n])，輸入y
  3. kubebuilder會根據上述命令新增多個檔案，如下圖紅框所示：

構建和部署CRD

  1. kubebuilder提供的Makefile將構建和部署工作大幅度簡化，執行以下命令會將最新構建的CRD部署在kubernetes上：
cd $GOPATH/src/helloworld

make install
  1. 控制檯輸出如下內容，提示部署成功：
[root@kubebuilder helloworld]# make install

/root/gopath/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases

kustomize build config/crd | kubectl apply -f -

Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition

customresourcedefinition.apiextensions.k8s.io/guestbooks.webapp.com.bolingcavalry created

編譯和執行controller

  1. kubebuilder自動生成的controller原始碼地址是：$GOPATH/src/helloworld/controllers/guestbook_controller.go ， 內容如下：
package controllers

import (

	"context"

	"github.com/go-logr/logr"

	"k8s.io/apimachinery/pkg/runtime"

	ctrl "sigs.k8s.io/controller-runtime"

	"sigs.k8s.io/controller-runtime/pkg/client"

	webappv1 "helloworld/api/v1"

)

// GuestbookReconciler reconciles a Guestbook object

type GuestbookReconciler struct {

	client.Client

	Log    logr.Logger

	Scheme *runtime.Scheme

}

// +kubebuilder:rbac:groups=webapp.com.bolingcavalry,resources=guestbooks,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups=webapp.com.bolingcavalry,resources=guestbooks/status,verbs=get;update;patch

func (r *GuestbookReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {

	_ = context.Background()

	_ = r.Log.WithValues("guestbook", req.NamespacedName)

	// your logic here

	return ctrl.Result{}, nil

}

func (r *GuestbookReconciler) SetupWithManager(mgr ctrl.Manager) error {

	return ctrl.NewControllerManagedBy(mgr).

		For(&webappv1.Guestbook{}).

		Complete(r)

}
  1. 本文以體驗基本流程為主，不深入研究原始碼，所以對上面的程式碼僅做少量修改，用於驗證是否能生效，改動如下圖紅框所示：

  1. 執行以下命令，會編譯並啟動剛才修改的controller：
cd $GOPATH/src/helloworld

make run
  1. 此時控制檯輸出以下內容，這裡要注意，controller是在kubebuilder電腦上執行的，一旦使用Ctrl+c中斷控制檯，就會導致controller停止：
[root@kubebuilder helloworld]# make run

/root/gopath/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."

go fmt ./...

go vet ./...

/root/gopath/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases

go run ./main.go

2021-01-23T20:58:35.107+0800	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}

2021-01-23T20:58:35.108+0800	INFO	setup	starting manager

2021-01-23T20:58:35.108+0800	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}

2021-01-23T20:58:35.108+0800	INFO	controller-runtime.controller	Starting EventSource	{"controller": "guestbook", "source": "kind source: /, Kind="}

2021-01-23T20:58:35.208+0800	INFO	controller-runtime.controller	Starting Controller	{"controller": "guestbook"}

2021-01-23T20:58:35.209+0800	INFO	controller-runtime.controller	Starting workers	{"controller": "guestbook", "worker count": 1}

建立Guestbook資源的例項

  1. 現在kubernetes已經部署了Guestbook型別的CRD，而且對應的controller也已正在執行中，可以嘗試建立Guestbook型別的例項了(相當於有了pod的定義後，才可以建立pod)；
  2. kubebuilder已經自動建立了一個型別的部署檔案：$GOPATH/src/helloworld/config/samples/webapp_v1_guestbook.yaml ，內容如下，很簡單，接下來咱們就用這個檔案來建立Guestbook例項：
apiVersion: webapp.com.bolingcavalry/v1

kind: Guestbook

metadata:

  name: guestbook-sample

spec:

  # Add fields here

  foo: bar
  1. 重新開啟一個控制檯，登入kubebuilder電腦，執行以下命令即可建立Guestbook型別的例項：
cd $GOPATH/src/helloworld

kubectl apply -f config/samples/
  1. 如下所示，控制檯提示資源建立成功：
[root@kubebuilder helloworld]# kubectl apply -f config/samples/

guestbook.webapp.com.bolingcavalry/guestbook-sample created
  1. 用kubectl get命令可以看到例項已經建立：
[root@kubebuilder helloworld]# kubectl get Guestbook

NAME               AGE

guestbook-sample   112s
  1. 用命令kubectl edit Guestbook guestbook-sample編輯該例項，修改的內容如下圖紅框所示：



7. 此時去controller所在控制檯，可以看到新增和修改的操作都有日誌輸出，咱們新增的日誌都在裡面，程式碼呼叫棧一目瞭然：

2021-01-24T09:51:50.418+0800	INFO	controllers.Guestbook	1. default/guestbook-sample

2021-01-24T09:51:50.418+0800	INFO	controllers.Guestbook	2. goroutine 188 [running]:

runtime/debug.Stack(0xc0002a1808, 0xc0002fc600, 0x1b)

	/root/go/src/runtime/debug/stack.go:24 +0x9f

helloworld/controllers.(*GuestbookReconciler).Reconcile(0xc0003c9dd0, 0xc0002d02f9, 0x7, 0xc0002d02e0, 0x10, 0x12f449647b, 0xc000456f30, 0xc000456ea8, 0xc000456ea0)

	/root/gopath/src/helloworld/controllers/guestbook_controller.go:49 +0x1a9

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00022a480, 0x1430e00, 0xc0003e7560, 0x0)

	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:256 +0x166

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00022a480, 0xc000469600)

	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232 +0xb0

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc00022a480)

	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211 +0x2b

k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000292980)

	/root/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5f

k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000292980, 0x3b9aca00, 0x0, 0x1609101, 0xc000102480)

	/root/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0x105

k8s.io/apimachinery/pkg/util/wait.Until(0xc000292980, 0x3b9aca00, 0xc000102480)

	/root/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d

created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1

	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:193 +0x32d

2021-01-24T09:51:50.418+0800	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "guestbook", "request": "default/guestbook-sample"}

2021-01-24T09:52:33.632+0800	INFO	controllers.Guestbook	1. default/guestbook-sample

2021-01-24T09:52:33.633+0800	INFO	controllers.Guestbook	2. goroutine 188 [running]:

runtime/debug.Stack(0xc0002a1808, 0xc0003fa5e0, 0x1b)

	/root/go/src/runtime/debug/stack.go:24 +0x9f

helloworld/controllers.(*GuestbookReconciler).Reconcile(0xc0003c9dd0, 0xc0002d02f9, 0x7, 0xc0002d02e0, 0x10, 0x1d0410fe42, 0xc000456f30, 0xc000456ea8, 0xc000456ea0)

	/root/gopath/src/helloworld/controllers/guestbook_controller.go:49 +0x1a9

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00022a480, 0x1430e00, 0xc0003d24c0, 0x0)

	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:256 +0x166

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00022a480, 0xc000469600)

	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232 +0xb0

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc00022a480)

	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211 +0x2b

k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000292980)

	/root/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5f

k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000292980, 0x3b9aca00, 0x0, 0x1609101, 0xc000102480)

	/root/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0x105

k8s.io/apimachinery/pkg/util/wait.Until(0xc000292980, 0x3b9aca00, 0xc000102480)

	/root/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d

created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1

	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:193 +0x32d

2021-01-24T09:52:33.633+0800	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "guestbook", "request": "default/guestbook-sample"}

刪除例項並停止controller

  1. 不再需要Guestbook例項的時候，執行以下命令即可刪除：
cd $GOPATH/src/helloworld

kubectl delete -f config/samples/
  1. 不再需要controller的時候，去它的控制檯使用Ctrl+c中斷即可；

將controller製作成docker映象

  1. 至此，咱們已經體驗過了kubebuilder的基本功能，不過實際生產環境中controller一般都會執行在kubernetes環境內，像上面這種執行在kubernetes之外的方式就不合適了，咱們來試試將其做成docker映象然後在kubernetes環境執行；
  2. 這裡有個要求，就是您要有個kubernetes可以訪問的映象倉庫，例如區域網內的Harbor，或者公共的hub.docker.com，我這為了操作方便選擇了hub.docker.com，使用它的前提是擁有hub.docker.com的註冊帳號；
  3. 在kubebuilder電腦上，開啟一個控制檯，執行docker login命令登入，根據提示輸入hub.docker.com的帳號和密碼，這樣就可以在當前控制檯上執行docker push命令將映象推送到hub.docker.com上了（這個網站的網路很差，可能要登入好幾次才能成功）；
  4. 執行以下命令構建docker映象並推送到hub.docker.com，映象名為bolingcavalry/guestbook:002：
cd $GOPATH/src/helloworld

make docker-build docker-push IMG=bolingcavalry/guestbook:002
  1. hub.docker.com的網路狀況不是一般的差，kubebuilder電腦上的docker一定要設定映象加速，上述命令如果遭遇超時失敗，請重試幾次，此外，構建過程中還會下載諸多go模組的依賴，也需要您耐心等待，也很容易遇到網路問題，需要多次重試，所以，最好是使用區域網內搭建的Habor服務；
  2. 最終，命令執行成功後輸出如下：
[root@kubebuilder helloworld]# make docker-build docker-push IMG=bolingcavalry/guestbook:002

/root/gopath/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."

go fmt ./...

go vet ./...

/root/gopath/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases

go test ./... -coverprofile cover.out

?   	helloworld	[no test files]

?   	helloworld/api/v1	[no test files]

ok  	helloworld/controllers	8.604s	coverage: 0.0% of statements

docker build . -t bolingcavalry/guestbook:002

Sending build context to Docker daemon  40.27MB

Step 1/14 : FROM golang:1.13 as builder

 ---> d6f3656320fe

Step 2/14 : WORKDIR /workspace

 ---> Using cache

 ---> 83d05ead1041

Step 3/14 : COPY go.mod go.mod

 ---> Using cache

 ---> ae3e15a529f4

Step 4/14 : COPY go.sum go.sum

 ---> Using cache

 ---> 082223532ccc

Step 5/14 : RUN go mod download

 ---> Using cache

 ---> bcdcfa1d65ca

Step 6/14 : COPY main.go main.go

 ---> Using cache

 ---> 81d6a629ca98

Step 7/14 : COPY api/ api/

 ---> Using cache

 ---> 75f99b174e97

Step 8/14 : COPY controllers/ controllers/

 ---> b130d9f47903

Step 9/14 : RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager main.go

 ---> Running in 768880aca19f

Removing intermediate container 768880aca19f

 ---> bb4a494d3b43

Step 10/14 : FROM gcr.io/distroless/static:nonroot

 ---> 947e6f3ed7c1

Step 11/14 : WORKDIR /

 ---> Using cache

 ---> 22cc43cef8fb

Step 12/14 : COPY --from=builder /workspace/manager .

 ---> 2137778f22c0

Step 13/14 : USER nonroot:nonroot

 ---> Running in 18295673073d

Removing intermediate container 18295673073d

 ---> f7545379ab1f

Step 14/14 : ENTRYPOINT ["/manager"]

 ---> Running in 550c47dd61dc

Removing intermediate container 550c47dd61dc

 ---> 31cb31a6b03f

Successfully built 31cb31a6b03f

Successfully tagged bolingcavalry/guestbook:002

docker push bolingcavalry/guestbook:002

The push refers to repository [docker.io/bolingcavalry/guestbook]

99035107a955: Pushed

728501c5607d: Layer already exists

002: digest: sha256:54f8ec88511cce5b04c5d65cc15e0f7a7b4a8afb6b235904a638bff79e3c5784 size: 739
  1. 去hub.docker.com網站看看，如下圖，新映象已經上傳，這樣只要任何機器只要能上網就能pull此映象到本地使用了：



8. 映象準備好之後，執行以下命令即可在kubernetes環境部署controller：

cd $GOPATH/src/helloworld

make deploy IMG=bolingcavalry/guestbook:002
  1. 控制檯會提示各類資源被建立（rbac居多）：
[root@kubebuilder ~]# cd $GOPATH/src/helloworld

[root@kubebuilder helloworld]# make deploy IMG=bolingcavalry/guestbook:002

/root/gopath/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases

cd config/manager && kustomize edit set image controller=bolingcavalry/guestbook:002

kustomize build config/default | kubectl apply -f -

namespace/helloworld-system created

Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition

customresourcedefinition.apiextensions.k8s.io/guestbooks.webapp.com.bolingcavalry configured

role.rbac.authorization.k8s.io/helloworld-leader-election-role created

clusterrole.rbac.authorization.k8s.io/helloworld-manager-role created

clusterrole.rbac.authorization.k8s.io/helloworld-proxy-role created

Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole

clusterrole.rbac.authorization.k8s.io/helloworld-metrics-reader created

rolebinding.rbac.authorization.k8s.io/helloworld-leader-election-rolebinding created

clusterrolebinding.rbac.authorization.k8s.io/helloworld-manager-rolebinding created

clusterrolebinding.rbac.authorization.k8s.io/helloworld-proxy-rolebinding created

service/helloworld-controller-manager-metrics-service created

deployment.apps/helloworld-controller-manager created
  1. 此時去看kubernetes環境的pod，發現確實已經新增了controller，如下圖紅框：

11.細心的您應該會發現上圖黃框中顯示這個pod實際上有兩個容器，用kubectl describe命令細看，分別是kube-rbac-proxy和manager，如下圖：



11. 由於有兩個容器，那麼檢視日誌時就要指定其中一個了，咱們的controller對應的是manager容器，因此檢視日誌的命令是：

kubectl logs -f \

helloworld-controller-manager-689d4b6f5b-h9pzg \

-n helloworld-system \

-c manager
  1. 再次建立Guestbook資源的例項，依舊是kubectl apply -f config/samples/命令，再去看manager容器的日誌，可見咱們修改的內容已經打印出來了：

解除安裝和清理

  • 體驗完畢後，如果想把前面建立的資源和CRD全部清理掉，可以執行以下命令：
cd $GOPATH/src/helloworld

make uninstall
  • 至此，通過kubebuilder建立Operator相關資源的基本流程，咱們已經體驗過一遍了，本篇以熟悉工具和流程為主，並未體驗到Operator實質性的強大功能，這些都留待後面的章節吧，咱們逐步深入學習實踐；

你不孤單，欣宸原創一路相伴

  1. Java系列
  2. Spring系列
  3. Docker系列
  4. kubernetes系列
  5. 資料庫+中介軟體系列
  6. DevOps系列

歡迎關注公眾號：程式設計師欣宸

微信搜尋「程式設計師欣宸」，我是欣宸，期待與您一同暢遊Java世界...

https://github.com/zq2599/blog_demos