helm的使用和redis、mq等公共組件的容器化實踐

阿新 • • 發佈：2019-05-03

分支 open 持久包管理 erp 工具負責初始 ani

Helm的使用

1. Helm簡介:

1. Helm的概念

Helm是Kubernetes的一個包管理工具，用來簡化Kubernetes應用的部署和管理。可以把Helm比作CentOS的yum工具.
Helm有如下幾個基本概念：

Chart: 是Helm管理的安裝包，裏面包含需要部署的安裝包資源。可以把Chart比作CentOS yum使用的rpm文件。每個Chart包含下面兩部分：
1. 包的基本描述文件Chart.yaml
2. 放在templates目錄中的一個或多個Kubernetes manifest文件模板
Release：是chart的部署實例，一個chart在一個Kubernetes集群上可以有多個release，即這個chart可以被安裝多次
Repository：chart的倉庫，用於發布和存儲chart

使用Helm可以完成以下事情：

管理Kubernetes manifest files
管理Helm安裝包charts
基於chart的Kubernetes應用分發

2. Helm的組成

Helm由兩部分組成，客戶端helm和服務端tiller。

tiller運行在Kubernetes集群上，管理chart安裝的release
helm是一個命令行工具，可在本地運行，一般運行在CI/CD Server上。

3. Helm的安裝

下載Helm二進制文件：https://github.com/helm/helm/releases/tag/v2.13.1

[[email protected] software]# tar xzvf helm-v2.13.1-linux-amd64.tar.gz
[[email protected] software]# cd linux-amd64/
[[email protected] linux-amd64]# cp -r helm /usr/local/bin/
[[email protected] linux-amd64]# helm version
Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
Error: Get http://localhost:8080/api/v1/namespaces/kube-system/pods?labelSelector=app%3Dhelm%2Cname%3Dtiller: dial tcp [::1]:8080: connect: connection refused
[[email protected] linux-amd64]#

此時運行helm version可以打印出客戶端helm的版本，同時會提示無法連接到服務端Tiller。
為了安裝服務端tiller，還需要在這臺機器上配置好kubectl工具和kubeconfig文件，確保kubectl工具可以在這臺機器上訪問apiserver且正常使用。
從kubemaster的服務器上面拷貝兩個文件到安裝Helm客戶端的機器，分別拷貝kubectl命令和/root/.kube/config文件

[[email protected] ~]# which kubectl
/usr/bin/kubectl
[[email protected] ~]# scp -r /usr/bin/kubectl [email protected]:/usr/bin/
kubectl                                                                                                                                                      100%   37MB  12.5MB/s   00:03
[[email protected] ~]# scp -r /root/.kube/config [email protected]:/root/.kube/
config                                                                                                                                                       100% 5448   328.6KB/s   00:00
[[email protected] ~]#

然後在安裝Helm客戶端的任何一臺機器上面，輸入命令檢查是否可以連接集群

[[email protected] linux-amd64]# kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health": "true"}

接下來是安裝服務端Tiller

# 配置 service account tiller 和 ClusterRoleBinding tiller
vim sa.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system

vim ClusterRoleBinding.yaml

  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: ClusterRoleBinding
  metadata:
    name: tiller
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: cluster-admin
  subjects:
    - kind: ServiceAccount
      name: tiller
      namespace: kube-system
kubectl apply -f  sa.yaml
kubectl apply -f  ClusterRoleBinding.yaml

初始化tiller服務端

helm init --service-account tiller --upgrade --tiller-image=registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.13.1
#註意這裏的tiller服務器鏡像版本，一定要和helm client的版本一致。這裏的版本都為最新的版本  v2.13.1
[[email protected] ~]# helm version
Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.13.1", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}

設置charts倉庫為微軟azure倉庫，這個倉庫同步的是最新的helm包，阿裏雲倉庫的helm包好久不更新了

[[email protected] ~]# cat /root/.helm/repository/repositories.yaml
apiVersion: v1
generated: 2018-11-10T20:32:27.187074383+08:00
repositories:
- caFile: ""
  cache: /root/.helm/repository/cache/stable-index.yaml
  certFile: ""
  keyFile: ""
  name: stable
  password: "password"
  url: http://mirror.azure.cn/kubernetes/charts/
  username: "username"
- caFile: ""
  cache: /root/.helm/repository/cache/local-index.yaml
  certFile: ""
  keyFile: ""
  name: local
  password: ""
  url: http://127.0.0.1:8879/charts
  username: ""
[[email protected] ~]#

#更新chart repo:
helm repo update

Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ? Happy Helming!?

4. 管理chart倉庫

[[email protected] software]# helm repo list
NAME    URL
stable  https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
local   http://127.0.0.1:8879/charts
[[email protected] software]#
#查看目前的chart倉庫

[[email protected] software]# helm repo remove stable
"stable" has been removed from your repositories
#刪除目前的倉庫

[[email protected] software]# helm repo add stable https://cnych.github.io/kube-charts-mirror/
"stable" has been added to your repositories
#添加新的遠程倉庫

[[email protected] software]# helm repo list
NAME    URL
local   http://127.0.0.1:8879/charts
stable  https://cnych.github.io/kube-charts-mirror/
#查看目前的chart倉庫

[[email protected] software]# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ? Happy Helming!?
[[email protected] software]#
#更新repo

5. Helm安裝hello-svc應用

使用helm create chart名稱安裝一個應用。同時會下載hello-svc項目的目錄文件到本地

helm create hello-svc
[[email protected] ~]# tree hello-svc/
hello-svc/
├── charts
├── Chart.yaml
├── templates
│?? ├── deployment.yaml
│?? ├── _helpers.tpl
│?? ├── ingress.yaml
│?? ├── NOTES.txt
│?? ├── service.yaml
│?? └── tests
│??     └── test-connection.yaml
└── values.yaml

我們來熟悉一下hello-svc這個chart中目錄和文件內容：

charts目錄中是本chart依賴的chart，當前是空的
Chart.yaml這個yaml文件用於描述Chart的基本信息，如名稱版本等
templates目錄是Kubernetes manifest文件模板目錄，模板使用chart配置的值生成Kubernetes manifest文件。模板文件使用的Go語言模板語法
templates/NOTES.txt 純文本文件，可在其中填寫chart的使用說明
value.yaml 是chart配置的默認值
通過查看deployment.yaml和value.yaml，大概知道這個chart默認安裝就是在Kubernetes部署一個nginx服務。

cd /root/hello-svc
helm install --dry-run --debug ./  #對chart的模板和配置進行測試
helm install ./  #在Kubernetes上安裝chart
helm list    #查看release
[[email protected] hello-svc]# helm list
NAME                REVISION    UPDATED                     STATUS      CHART               APP VERSION NAMESPACE
silly-gecko         1           Wed Mar 27 11:25:11 2019    DEPLOYED    hello-svc-0.1.0     1.0         default
stultified-prawn    1           Mon Mar  4 15:21:21 2019    DEPLOYED    nginx-ingress-0.9.5 0.10.2      default
[[email protected] hello-svc]#

6. 使用Helm安裝Redis-HA應用

在我以前的博文中介紹過在傳統虛擬機上面實現Redis的安裝步驟。本文重點介紹在kubernetes容器集群中如何部署Redis應用。在部署之前首先介紹一下Redis集群的兩種常用方案：

1. redis-HA方案：

Redis-HA方案應該叫做主從復制方案，不能稱為集群方案。主從復制方案的主要原理是，如果主節點宕機，從節點作為主節點的備份會隨時頂替主節點工作，也可以將讀請求分配到從節點，類似於Mysql的讀寫分離案節例一樣。但是如果主節宕機，從節升級為主節點，但是應用程序連接的redis服務器IP仍然為主節點IP，這個時候還需要人工
幹預應用程序的連結Redis服務器的IP，所以需要把這個IP配置成VIP地址，實現自動切換。

Redis Sentinel是一個分布式架構，包含若幹個Sentinel節點和Redis數據節點，每個Sentinel節點會對數據節點和其余Sentinel節點進行監控，當發現節點不可達時，會對節點做下線標識。
如果被標識的是主節點，他還會選擇和其他Sentinel節點進行“協商”，當大多數的Sentinel節點都認為主節點不可達時，他們會選舉出一個Sentinel節點來完成自動故障轉移工作，同時將這個變化通知給Redis應用方。

每個Sentinel節點都要定期PING命令來判斷Redis數據節點和其余Sentinel節點是否可達，如果超過30000毫秒且沒有回復，則判定不可達。當Sentinel節點集合對主節點故障判定達成一致時，Sentinel領導者節點會做故障轉移操作，選出新的主節點，原來的從節點會向新的主節點發起復制操作。

2. redis-Cluster方案：

Redis官方在3.0版本之後才開始支持集群方案。Redis 集群是一個可以在多個 Redis 節點之間進行數據共享的設施（installation）。Redis 集群通過分區（partition）來提供一定程度的可用性（availability）：即使集群中有一部分節點失效或者無法進行通訊，集群也可以繼續處理命令請求。
Redis集群提供了兩個好處，第一是將數據自動切分（split)到多個節點，第二是當集群中有一部分節點失效或者無法進行通訊時，仍然可以繼續處理命令請求；
Redis 集群使用數據分片（sharding）而非一致性哈希（consistency hashing）來實現。一個 Redis 集群包含 16384 個哈希槽（hash slot），數據庫中的每個鍵都屬於這 16384 個哈希槽的其中一個，集群使用公式 CRC16(key) % 16384 來計算鍵 key 屬於哪個槽，其中 CRC16(key) 語句用於計算鍵 key 的 CRC16 校驗和。
集群中的每個節點負責處理一部分哈希槽。舉個例子，一個集群可以有三個哈希槽，其中：

節點 A 負責處理 0 號至 5500 號哈希槽
節點 B 負責處理 5501 號至 11000 號哈希槽
節點 C 負責處理 11001 號至 16384 號哈希槽。

這種將哈希槽分布到不同節點的做法使得用戶可以很容易地向集群中添加或者刪除節點。比如說：

如果用戶將新節點 D 添加到集群中，那麽集群只需要將節點 A 、B 、 C 中的某些槽移動到節點 D 就可以了。
與此類似，如果用戶要從集群中移除節點 A ，那麽集群只需要將節點 A 中的所有哈希槽移動到節點 B 和節點 C ，然後再移除空白（不包含任何哈希槽）的節點 A 就可以了。

為了使得集群在一部分節點下線或者無法與集群的大多數（majority）節點進行通訊的情況下，仍然可以正常運作， Redis 集群對節點使用了主從復制功能：集群中的每個節點都有 1 個至 N 個復制品（replica），其中一個復制品為主節點（master），而其余的 N-1 個復制品為從節點（slave）。

在之前列舉的節點 A 、B 、C 的例子中，如果節點 B 下線了，那麽集群將無法正常運行，因為集群找不到節點來處理 5501 號至 11000號的哈希槽。
另一方面，假如在創建集群的時候（或者至少在節點 B 下線之前），我們為主節點 B 添加了從節點 B1 ，那麽當主節點 B 下線的時候，集群就會將 B1 設置為新的主節點，並讓它代替下線的主節點 B ，繼續處理 5501 號至 11000 號的哈希槽，這樣集群就不會因為主節點 B 的下線而無法正常運作了。

對於Redis集群方案，國內的豌豆莢公司開發的一個分布式的基於代理的高性能Redis 集群解決方案，名字叫做Codis.（國內還有類似於Twemproxy的軟件也能實現同樣的功能）用Go語言開發的。對於上層的應用來說，連接到 Codis Proxy 和連接原生的 Redis Server 沒有明顯的區別（不支持的命令列表），Codis 底層會處理請求的轉發，不停機的數據遷移等工作。所有後邊的一切事情，對於前面的客戶端來說是透明的，可以簡單的認為後邊連接的是一個內存無限大的 Redis 服務。具體的方案可以參考https://github.com/CodisLabs/codis

Codis 由四部分組成:

Codis Proxy (codis-proxy)，處理客戶端請求，支持Redis協議，因此客戶端訪問Codis Proxy跟訪問原生Redis沒有什麽區別；
Codis Dashboard (codis-config)，Codis 的管理工具，支持添加/刪除 Redis 節點、添加/刪除 Proxy 節點，發起數據遷移等操作。codis-config 本身還自帶了一個 http server，會啟動一個 dashboard，用戶可以直接在瀏覽器上觀察 Codis 集群的運行狀態；
Codis Redis (codis-server)，Codis 項目維護的一個 Redis 分支，基於 2.8.21 開發，加入了 slot 的支持和原子的數據遷移指令；
ZooKeeper/Etcd，Codis 依賴 ZooKeeper 來存放數據路由表和 codis-proxy 節點的元信息，codis-config 發起的命令都會通過 ZooKeeper 同步到各個存活的 codis-proxy

3. 使用Helm安裝redis-sentinel主從：

所有的官方Helm包都可以在https://github.com/helm/charts/tree/master/stable/ 網址下載。包括我本次實驗的Redis-HA，Mysql、RabbitMQ、Jenkins等。

配置好helm倉庫之後，最好微軟的那個helm倉庫，是同步更新github。然後可以使用命令:

[[email protected] helm_jenkins]#  helm repo list
NAME        URL
local       http://127.0.0.1:8879/charts
stable      http://mirror.azure.cn/kubernetes/charts/
kiwigrid    https://kiwigrid.github.io
[[email protected] helm_jenkins]# helm search redis
NAME                                CHART VERSION   APP VERSION DESCRIPTION
stable/prometheus-redis-exporter    1.0.2           0.28.0      Prometheus exporter for Redis metrics
stable/redis                        7.0.0           4.0.14      Open source, advanced key-value store. It is often referr...
stable/redis-ha                     3.4.0           5.0.3       Highly available Kubernetes implementation of Redis
stable/sensu                        0.2.3           0.28        Sensu monitoring framework backed by the Redis transport
[[email protected] helm_jenkins]#

然後下載redis-ha的helm包到本地.先創建一個本地的目錄，然後直接把helm包下載到本地的目錄

[[email protected] tmp]# mkdir -p redis-ha
[[email protected] tmp]# helm fetch stable/redis-ha --untar --untardir ./redis-ha/

第二步就是修改value.yaml文件。這個文件的修改方法一般是參考官方的helm說明。https://github.com/helm/charts/tree/master/stable/redis-ha 這個文件的修改方法一般是參考官方的helm說明。官網一般都會詳細描述每一個key的用途，value值的默認參數等。

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
image:
  repository: k8s.harbor.test.site/system/redis
  tag: 5.0.3-alpine
  pullPolicy: IfNotPresent
## replicas number for each component
replicas: 3

## Custom labels for the redis pod
labels: {}

## Pods Service Account
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
serviceAccount:
  ## Specifies whether a ServiceAccount should be created
  ##
  create: true
  ## The name of the ServiceAccount to use.
  ## If not set and create is true, a name is generated using the redis-ha.fullname template
  # name:

## Role Based Access
## Ref: https://kubernetes.io/docs/admin/authorization/rbac/
##

rbac:
  create: true

sysctlImage:
  enabled: false
  command: []
  registry: docker.io
  repository: k8s.harbor.test.site/system/minideb:latest
  tag: latest
  pullPolicy: Always
  mountHostSys: false

## Redis specific configuration options
redis:
  port: 6379
  masterGroupName: mymaster
  config:
    ## Additional redis conf options can be added below
    ## For all available options see http://download.redis.io/redis-stable/redis.conf
    min-slaves-to-write: 1
    min-slaves-max-lag: 5   # Value in seconds
    maxmemory: "0"       # Max memory to use for each redis instance. Default is unlimited.
    maxmemory-policy: "volatile-lru"  # Max memory policy to use for each redis instance. Default is volatile-lru.
    # Determines if scheduled RDB backups are created. Default is false.
    # Please note that local (on-disk) RDBs will still be created when re-syncing with a new slave. The only way to prevent this is to enable diskless replication.
    save: "900 1"
    # When enabled, directly sends the RDB over the wire to slaves, without using the disk as intermediate storage. Default is false.
    repl-diskless-sync: "yes"
    rdbcompression: "yes"
    rdbchecksum: "yes"

  ## Custom redis.conf files used to override default settings. If this file is
  ## specified then the redis.config above will be ignored.
  # customConfig: |-
      # Define configuration here

  resources: {}
  #  requests:
  #    memory: 200Mi
  #    cpu: 100m
  #  limits:
  #    memory: 700Mi

## Sentinel specific configuration options
sentinel:
  port: 26379
  quorum: 2
  config:
    ## Additional sentinel conf options can be added below. Only options that
    ## are expressed in the format simialar to ‘sentinel xxx mymaster xxx‘ will
    ## be properly templated.
    ## For available options see http://download.redis.io/redis-stable/sentinel.conf
    down-after-milliseconds: 10000
    ## Failover timeout value in milliseconds
    failover-timeout: 180000
    parallel-syncs: 5

  ## Custom sentinel.conf files used to override default settings. If this file is
  ## specified then the sentinel.config above will be ignored.
  # customConfig: |-
      # Define configuration here

  resources: {}
  #  requests:
  #    memory: 200Mi
  #    cpu: 100m
  #  limits:
  #    memory: 200Mi

securityContext:
  runAsUser: 1000
  fsGroup: 1000
  runAsNonRoot: true

## Node labels, affinity, and tolerations for pod assignment
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
affinity: |
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: {{ template "redis-ha.name" . }}
            release: {{ .Release.Name }}
        topologyKey: kubernetes.io/hostname
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app:  {{ template "redis-ha.name" . }}
              release: {{ .Release.Name }}
          topologyKey: failure-domain.beta.kubernetes.io/zone

# Prometheus exporter specific configuration options
exporter:
  enabled: true
  image: k8s.harbor.test.site/system/redis_exporter
  tag: v0.31.0
  pullPolicy: IfNotPresent

  # prometheus port & scrape path
  port: 9121
  scrapePath: /metrics

  # cpu/memory resource limits/requests
  resources: {}

  # Additional args for redis exporter
  extraArgs: {}

podDisruptionBudget: {}
  # maxUnavailable: 1
  # minAvailable: 1

## Configures redis with AUTH (requirepass & masterauth conf params)
auth: true
redisPassword: aaaaaa

## Use existing secret containing "auth" key (ignores redisPassword)
# existingSecret:

persistentVolume:
  enabled: true
  ## redis-ha data Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  storageClass: "dynamic"
  accessModes:
    - ReadWriteOnce
  size: 10Gi
  annotations: {}
init:
  resources: {}

# To use a hostPath for data, set persistentVolume.enabled to false
# and define hostPath.path.
# Warning: this might overwrite existing folders on the host system!
hostPath:
  ## path is evaluated as template so placeholders are replaced
  # path: "/data/{{ .Release.Name }}"

  # if chown is true, an init-container with root permissions is launched to
  # change the owner of the hostPath folder to the user defined in the
  # security context
  chown: true

在修改value.yaml之前，建議先看一下官方的說明。並且使用helm安裝redis-ha等由狀態應用之前，需要熟悉redis-ha有狀態應用的原理，
並且最好是自己搭建過環境，理解裏面的配置。

我一般的個人習慣是，先把value.yaml裏面用到的鏡像下載到本地然後上傳到公司的harbor私有倉庫，然後把value.yaml裏面的鏡像地址都換成
公司私有倉庫的鏡像地址，這樣部署起來會比較快。因為redis-ha應用需要用到持久化存儲，這個reids-ha應用主要是kubernetes裏面的statefulset資源類型，
也就是每一個Pod重啟之後都需要有固定的名稱，一般為pod名稱+無頭服務的名稱+namespace+svc.cluster.local. 並且每一個pod重新啟動之後需要找到原來的
pvc.

所以我們還要存儲這塊的配置，增加我們以前配置的storageClassName的名字為dynamic。這個是我們這個環境的ceph RBD存儲的sc的名字。這個heml包裏面還自帶安裝了redis-exporter。專門用於將redis的metrics監控數據上傳到prometheus的用途；

配置文件中這一段配置podAntiAffinity，主要的用途是調度反親和性，屬於運行時調度策略.podAntiAffinity主要解決POD不能和哪些POD部署在同一個拓撲域中的問題。它們處理的是Kubernetes集群內部POD和POD之間的關系。目的就是讓redis Master不要和redis Slave兩個pod不要啟動在一個物理節點上面；

同時還定義了redis的配置文件，sentinel的配置文件。指定了redis的副本數為3,指定了連接redis的密碼。

關於測試redis-ha配置是否正常，可以通過命令helm status redis-ha來查看

[[email protected] redis-ha]# helm status my-redis
LAST DEPLOYED: Tue Apr 23 15:57:27 2019
NAMESPACE: kube-system
STATUS: DEPLOYED

RESOURCES:
==> v1/ServiceAccount
NAME               SECRETS  AGE
my-redis-redis-ha  1        8d

==> v1/Role
NAME               AGE
my-redis-redis-ha  8d

==> v1/RoleBinding
NAME               AGE
my-redis-redis-ha  8d

==> v1/Service
NAME                          TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)                      AGE
my-redis-redis-ha-announce-2  ClusterIP  10.200.111.254  <none>       6379/TCP,26379/TCP           8d
my-redis-redis-ha-announce-0  ClusterIP  10.200.192.130  <none>       6379/TCP,26379/TCP           8d
my-redis-redis-ha-announce-1  ClusterIP  10.200.55.75    <none>       6379/TCP,26379/TCP           8d
my-redis-redis-ha             ClusterIP  None            <none>       6379/TCP,26379/TCP,9121/TCP  8d

==> v1/StatefulSet
NAME                      DESIRED  CURRENT  AGE
my-redis-redis-ha-server  3        3        8d

==> v1/Pod(related)
NAME                        READY  STATUS   RESTARTS  AGE
my-redis-redis-ha-server-0  3/3    Running  3         8d
my-redis-redis-ha-server-1  3/3    Running  4         5d18h
my-redis-redis-ha-server-2  3/3    Running  0         2d19h

==> v1/Secret
NAME               TYPE    DATA  AGE
my-redis-redis-ha  Opaque  1     8d

==> v1/ConfigMap
NAME                         DATA  AGE
my-redis-redis-ha-configmap  3     8d
my-redis-redis-ha-probes     2     8d

NOTES:
Redis can be accessed via port 6379 and Sentinel can be accessed via port 26379 on the following DNS name from within your cluster:
my-redis-redis-ha.kube-system.svc.cluster.local

To connect to your Redis server:
1. To retrieve the redis password:
   echo $(kubectl get secret my-redis-redis-ha -o "jsonpath={.data[‘auth‘]}" | base64 --decode)

2. Connect to the Redis master pod that you can use as a client. By default the my-redis-redis-ha-server-0 pod is configured as the master:

   kubectl exec -it my-redis-redis-ha-server-0 sh -n kube-system

3. Connect using the Redis CLI (inside container):

   redis-cli -a <REDIS-PASS-FROM-SECRET>

如果其他的應用需要連接這個redis-ha，就需要使用命令 redis-cli -h my-redis-redis-ha -p 6379 -a aaaaaa 使用這個命令只有在k8s集群內部的應用可以。如果是外部的應用，就需要把redis-ha的端口通過NodePort的類型暴露出去才可以；

4. 使用Helm安裝RabbitMQ集群：

前面的博文中我也已經介紹過如何在虛擬機裏面安裝RabbitMQ集群。通過練習大家應該知道了RabbitMQ集群的一些基礎概念和配置。包括生產者、消費者、Exchange、Queue、BinDings、Connection、Channel、交換機的類型單播和廣播、消息的持久化定義等。

最簡單的RabbitMQ集群需要有3個節點，兩個內存節點，一個磁盤節點。顧名思義內存節點就是將所有數據放在內存，磁盤節點將數據放在磁盤。不過，如果在投遞消息時，打開了消息的持久化，那麽即使是內存節點，數據還是安全的放在磁盤。

由於RabbitMQ是用erlang開發的，RabbitMQ完全依賴Erlang的Cluster，因為erlang天生就是一門分布式語言,集群非常方便，但其本身並不支持負載均衡。Erlang的集群中各節點是經由過程一個magic cookie來實現的，這個cookie存放在 $home/.erlang.cookie 中(像我的root用戶安裝的就是放在我的root/.erlang.cookie中)，文件是400的權限。所以必須包管各節點cookie對峙一致，不然節點之間就無法通信。

Rabbitmq集群大概分為二種方式：

普通模式：默認的集群模式。
鏡像模式：把需要的隊列做成鏡像隊列。（我們一般都采用此模式）
```
rabbitmqctl set_policy ha-all "^" ‘{"ha-mode":"all"}‘
```
這行命令創建了一個策略，策略名稱為ha-all,策略模式為 all 即復制到所有節點，包含新增節點，策略正則表達式為 “^” 表示所有匹配所有隊列名稱。

良好的設計架構可以如下：在一個集群裏，有3臺以上機器，其中1臺使用磁盤模式，其它使用內存模式。其它幾臺為內存模式的節點，無疑速度更快，因此客戶端（consumer、producer）連接訪問它們。而磁盤模式的節點，由於磁盤IO相對較慢，因此僅作數據備份使用。

和前面的redis-ha的helm安裝一樣，先是找到官方網站 https://github.com/helm/charts/tree/master/stable/rabbitmq-ha 仔細閱讀以下官方的說明，官方文檔裏面有針對配置value.yaml的詳細說明，並且指導你可以安裝chart，卸載chart等；顯示搜索rabbitmq，然後下載rabbitmq.這裏就不在介紹了。主要有個問題說明以下，rabbitmq有幾個版本的chart,有個rabbitmq的chart,我始終都沒有研究成功。每次配置好之只要已啟動就會報錯

NAME                                CHART VERSION   APP VERSION DESCRIPTION
stable/prometheus-rabbitmq-exporter 0.4.0           v0.29.0     Rabbitmq metrics exporter for prometheus
stable/rabbitmq                     5.5.0           3.7.14      Open source message broker software that implements the A...
stable/rabbitmq-ha                  1.25.0          3.7.12      Highly available RabbitMQ cluster, the open source messag...
[[email protected] redis-ha]#

我剛開始的時候一直使用的是stable/rabbitmq這個chart,然後研究了很久都不成功，安裝helm包之後啟動應用總是報錯.

kubectl logs -f -n kube-system my-rabbitmq-0 rabbitmq
Error from server (BadRequest): a container name must be specified for pod my-rabbitmq-0, choose one of: [rabbitmq metrics]

後面找了很久也沒有找到問題原因，網上都是說因為找不到正確的節點名字，所以集群無法創建成功。因為在k8s的rabbitmq方案中，rabbitmq有一個專門的k8s插件，這個插件可以實現通過k8s集群的api接口通過pod名字自動發現和創建集群。最終采用了另外一個chart stable/rabbitmq-ha就好了。如果有時間我還要再深究一下兩個之間的區別。我的value.yaml文件配置如下：

## RabbitMQ application credentials
## Ref: http://rabbitmq.com/access-control.html
##
rabbitmqUsername: guest
rabbitmqPassword: guest123

## RabbitMQ Management user used for health checks
managementUsername: management
managementPassword: 123456

## Place any additional key/value configuration to add to rabbitmq.conf
## Ref: https://www.rabbitmq.com/configure.html#config-items
extraConfig: |
#  queue_master_locator = min-masters

## Place advanced.config file in /etc/rabbitmq/advanced.config
## Ref: https://www.rabbitmq.com/configure.html#advanced-config-file
advancedConfig: |

## Definitions specification within the secret, will always be mounted
## at /etc/definitions/defintions.json
definitionsSource: definitions.json

## Place any additional plugins to enable in /etc/rabbitmq/enabled_plugins
## Ref: https://www.rabbitmq.com/plugins.html
extraPlugins: |
  rabbitmq_shovel,
  rabbitmq_shovel_management,
  rabbitmq_federation,
  rabbitmq_federation_management,

definitions:
  users: |-
   {
     "name": "test_producer01",
     "password": "111111",
     "tags": "administrator"
   }
  vhosts: |-
   {
     "name": "/rabbit"
   }
  parameters: |-
   {
     "value": {
       "src-uri": "amqp://localhost",
       "src-queue": "source",
       "dest-uri": "amqp://localhost",
       "dest-queue": "destination",
       "add-forward-headers": false,
       "ack-mode": "on-confirm",
       "delete-after": "never"
     },
     "vhost": "/",
     "component": "shovel",
     "name": "test"
   }
  permissions: |-
   {
     "user": "test_producer01",
     "vhost": "/rabbit",
     "configure": ".*",
     "write": ".*",
     "read": ".*"
   }
  queues: |-
    {
       "name":"myName",
       "vhost":"/rabbit",
       "durable":true,
       "auto_delete":false,
       "arguments":{}
    }
  exchanges: |-
    {
       "name":"myName",
       "vhost":"/rabbit",
       "type":"direct",
       "durable":true,
       "auto_delete":false,
       "internal":false,
       "arguments":{}
    }
  bindings: |-
    {
       "source":"myName",
       "vhost":"/rabbit",
       "destination":"myName",
       "destination_type":"queue",
       "routing_key":"myKey",
       "arguments":{}
    }
## Sets the policies in definitions.json. This can be used to control the high
## availability of queues by mirroring them to multiple nodes.
## Ref: https://www.rabbitmq.com/ha.html
  policies: |-
    {
      "name": "ha-all",
      "pattern": ".*",
      "vhost": "/",
      "definition": {
        "ha-mode": "all",
        "ha-sync-mode": "automatic",
        "ha-sync-batch-size": 1
      }
    }

## RabbitMQ default VirtualHost
## Ref: https://www.rabbitmq.com/vhosts.html
##
rabbitmqVhost: "/"

## Erlang cookie to determine whether different nodes are allowed to communicate with each other
## Ref: https://www.rabbitmq.com/clustering.html
##
# rabbitmqErlangCookie:

## RabbitMQ Memory high watermark
## Ref: http://www.rabbitmq.com/memory.html
##
rabbitmqMemoryHighWatermark: 256MB
rabbitmqMemoryHighWatermarkType: absolute

## EPMD port for peer discovery service used by RabbitMQ nodes and CLI tools
## Ref: https://www.rabbitmq.com/clustering.html
##
rabbitmqEpmdPort: 4369

## Node port
rabbitmqNodePort: 5672

## Manager port
rabbitmqManagerPort: 15672

## Set to true to precompile parts of RabbitMQ with HiPE, a just-in-time
## compiler for Erlang. This will increase server throughput at the cost of
## increased startup time. You might see 20-50% better performance at the cost
## of a few minutes delay at startup.
rabbitmqHipeCompile: false

## SSL certificates
## Red: http://www.rabbitmq.com/ssl.html
rabbitmqCert:
  enabled: false

  # Specifies an existing secret to be used for SSL Certs
  existingSecret: ""

  ## Create a new secret using these values
  cacertfile: |
  certfile: |
  keyfile: |

## Extra volumes for statefulset
extraVolumes: []

## Extra volume mounts for statefulset
extraVolumeMounts: []

## Authentication mechanism
## Ref: http://www.rabbitmq.com/authentication.html
rabbitmqAuth:
  enabled: false

  config: |
    # auth_mechanisms.1 = PLAIN
    # auth_mechanisms.2 = AMQPLAIN
    # auth_mechanisms.3 = EXTERNAL

## Automatic Partition Handling Strategy (split brain handling)
## Ref: https://www.rabbitmq.com/partitions.html#automatic-handling
## Note: pause-if-all-down is not supported without using a custom configmap since it requires extra
## configuration.

rabbitmqClusterPartitionHandling: autoheal

## Authentication backend
## Ref: https://github.com/rabbitmq/rabbitmq-auth-backend-http
rabbitmqAuthHTTP:
  enabled: false

  config: |
    # auth_backends.1 = http
    # auth_http.user_path     = http://some-server/auth/user
    # auth_http.vhost_path    = http://some-server/auth/vhost
    # auth_http.resource_path = http://some-server/auth/resource
    # auth_http.topic_path    = http://some-server/auth/topic

## LDAP Plugin
## Ref: http://www.rabbitmq.com/ldap.html
rabbitmqLDAPPlugin:
  enabled: false

  ## LDAP configuration:
  config: |
    # auth_backends.1 = ldap
    # auth_ldap.servers.1  = my-ldap-server
    # auth_ldap.user_dn_pattern = cn=${username},ou=People,dc=example,dc=com
    # auth_ldap.use_ssl    = false
    # auth_ldap.port       = 389
    # auth_ldap.log        = false

## MQTT Plugin
## Ref: http://www.rabbitmq.com/mqtt.html
rabbitmqMQTTPlugin:
  enabled: false

  ## MQTT configuration:
  config: |
    # mqtt.default_user     = guest
    # mqtt.default_pass     = guest
    # mqtt.allow_anonymous  = true

## Web MQTT Plugin
## Ref: http://www.rabbitmq.com/web-mqtt.html
rabbitmqWebMQTTPlugin:
  enabled: false

  ## Web MQTT configuration:
  config: |
    # web_mqtt.ssl.port       = 12345
    # web_mqtt.ssl.backlog    = 1024
    # web_mqtt.ssl.certfile   = /etc/cert/cacert.pem
    # web_mqtt.ssl.keyfile    = /etc/cert/cert.pem
    # web_mqtt.ssl.cacertfile = /etc/cert/key.pem
    # web_mqtt.ssl.password   = changeme

## STOMP Plugin
## Ref: http://www.rabbitmq.com/stomp.html
rabbitmqSTOMPPlugin:
  enabled: false

  ## STOMP configuration:
  config: |
    # stomp.default_user = guest
    # stomp.default_pass = guest

## Web STOMP Plugin
## Ref: http://www.rabbitmq.com/web-stomp.html
rabbitmqWebSTOMPPlugin:
  enabled: false

  ## Web STOMP configuration:
  config: |
    # web_stomp.ws_frame = binary
    # web_stomp.cowboy_opts.max_keepalive = 10

## AMQPS support
## Ref: http://www.rabbitmq.com/ssl.html
rabbitmqAmqpsSupport:
  enabled: false

  # NodePort
  amqpsNodePort: 5671

  # SSL configuration
  config: |
    # listeners.ssl.default             = 5671
    # ssl_options.cacertfile            = /etc/cert/cacert.pem
    # ssl_options.certfile              = /etc/cert/cert.pem
    # ssl_options.keyfile               = /etc/cert/key.pem
    # ssl_options.verify                = verify_peer
    # ssl_options.fail_if_no_peer_cert  = false

## Number of replicas
replicaCount: 3

image:
  repository: k8s.harbor.test.site/system/rabbitmq
  tag: 3.7.12-alpine
  pullPolicy: IfNotPresent
  ## Optionally specify an array of imagePullSecrets.
  ## Secrets must be manually created in the namespace.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  # pullSecrets:
  #   - myRegistrKeySecretName

busyboxImage:
  repository: k8s.harbor.test.site/system/busybox
  tag: latest
  pullPolicy: Always

## Duration in seconds the pod needs to terminate gracefully
terminationGracePeriodSeconds: 10

service:
  annotations: {}
  clusterIP: None

  ## List of IP addresses at which the service is available
  ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
  ##
  externalIPs: []

  loadBalancerIP: ""
  loadBalancerSourceRanges: []
  type: ClusterIP

  ## Customize nodePort number when the service type is NodePort
  ### Ref: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
  ###
  epmdNodePort: null
  amqpNodePort: null
  managerNodePort: null

podManagementPolicy: OrderedReady

## Statefulsets rolling update update strategy
## Ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#rolling-update
##
updateStrategy: OnDelete

## Statefulsets Pod Priority
## Ref: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
## priorityClassName: ""

## We usually recommend not to specify default resources and to leave this as
## a conscious choice for the user. This also increases chances charts run on
## environments with little resources, such as Minikube. If you do want to
## specify resources, uncomment the following lines, adjust them as necessary,
## and remove the curly braces after ‘resources:‘.
## If you decide to set the memory limit, make sure to also change the
## rabbitmqMemoryHighWatermark following the formula:
##   rabbitmqMemoryHighWatermark = 0.4 * resources.limits.memory
##
resources: {}
  # limits:
  #   cpu: 100m
  #   memory: 1Gi
  # requests:
  #   cpu: 100m
  #   memory: 1Gi
initContainer:
  resources: {}
  #   limits:
  #     cpu: 100m
  #     memory: 128Mi
  #   requests:
  #     cpu: 100m
  #     memory: 128Mi

## Use an alternate scheduler, e.g. "stork".
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
# schedulerName:

## Data Persistency
persistentVolume:
  enabled: true
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  storageClass: "dynamic"
  name: data
  accessModes:
    - ReadWriteOnce
  size: 8Gi
  annotations: {}

## Node labels for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
##
nodeSelector: {}

## Node tolerations for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
##
tolerations: []

## Extra Annotations to be added to pod
podAnnotations: {}

## Pod affinity
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
podAntiAffinity: soft

## Create default configMap
##
existingConfigMap: false

## Add additional labels to all resources
##
extraLabels: {}

## Role Based Access
## Ref: https://kubernetes.io/docs/admin/authorization/rbac/
##
rbac:
  create: true

## Service Account
## Ref: https://kubernetes.io/docs/admin/service-accounts-admin/
##
serviceAccount:
  create: true

  ## The name of the ServiceAccount to use.
  ## If not set and create is true, a name is generated using the fullname template
  # name:

ingress:
  ## Set to true to enable ingress record generation
  enabled: true

  path: /

  ## The list of hostnames to be covered with this ingress record.
  ## Most likely this will be just one host, but in the event more hosts are needed, this is an array
  hostName: k8s.rabbitmq.test.site

  ## Set this to true in order to enable TLS on the ingress record
  tls: false

  ## If TLS is set to true, you must declare what secret will store the key/certificate for TLS
  tlsSecret: myTlsSecret

  ## Ingress annotations done as key:value pairs
  annotations:
    kubernetes.io/ingress.class: traefik

livenessProbe:
  initialDelaySeconds: 120
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 6

readinessProbe:
  failureThreshold: 6
  initialDelaySeconds: 20
  timeoutSeconds: 3
  periodSeconds: 5

# Specifies an existing secret to be used for RMQ password and Erlang Cookie
existingSecret: ""

## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
##
securityContext:
  fsGroup: 101
  runAsGroup: 101
  runAsNonRoot: true
  runAsUser: 100

prometheus:
  ## Configures Prometheus Exporter to expose and scrape stats.
  exporter:
    enabled: true
    env: {}
    image:
      repository: kbudde/rabbitmq-exporter
      tag: v0.29.0
      pullPolicy: IfNotPresent

    ## Port Prometheus scrapes for metrics
    port: 9090
    ## Comma-separated list of extended scraping capabilities supported by the target RabbitMQ server
    capabilities: "bert,no_sort"

    ## Allow overriding of container resources
    resources: {}
     # limits:
     #   cpu: 200m
     #   memory: 1Gi
     # requests:
     #   cpu: 100m
     #   memory: 100Mi

  ## Prometheus is using Operator.  Setting to true will create Operator specific resources like ServiceMonitors and Alerts
  operator:
    ## Are you using Prometheus Operator? [Blog Post](https://coreos.com/blog/the-prometheus-operator.html)
    enabled: true

    ## Configures Alerts, which will be setup via Prometheus Operator / ConfigMaps.
    alerts:
      ## Prometheus exporter must be enabled as well
      enabled: true

      ## Selector must be configured to match Prometheus Install, defaulting to whats done by Prometheus Operator
      ## See [CoreOS Prometheus Chart](https://github.com/coreos/prometheus-operator/tree/master/helm)
      selector:
        role: alert-rules
      labels: {}

    serviceMonitor:
      ## Interval at which Prometheus scrapes RabbitMQ Exporter
      interval: 10s

      # Namespace Prometheus is installed in
      namespace: kube-system

      ## Defaults to whats used if you follow CoreOS [Prometheus Install Instructions](https://github.com/coreos/prometheus-operator/tree/master/helm#tldr)
      ## [Prometheus Selector Label](https://github.com/coreos/prometheus-operator/blob/master/helm/prometheus/templates/prometheus.yaml#L65)
      ## [Kube Prometheus Selector Label](https://github.com/coreos/prometheus-operator/blob/master/helm/kube-prometheus/values.yaml#L298)
      selector:
        prometheus: kube-prometheus

## Kubernetes Cluster Domain
clusterDomain: cluster.local

## Pod Disruption Budget
podDisruptionBudget: {}
  # maxUnavailable: 1
  # minAvailable: 1

根據官方文檔的參考，我主要修改了如下的配置

rabbitmq的一些初始賬戶和密碼信息；
默認安裝了一些pulgins，主要是隊列消息同步的組件；
定義了一個初始化的vhost、生產者用戶、用戶權限、隊列名、Exchange、Bindings、同步策略等；
定義了集群的參數，包括ErlangCookie、集群同步的端口、web管理端口15672、服務端口5672；
定義Pods的副本數為3，也就是創建一個3節點的mq集群；定義鏡像全部為私有倉庫鏡像，鏡像包括mq鏡像，用於初始化的busybox鏡像，用於prometheus監控的rabbitmq-exporter鏡像
還定義了Pod的啟動策略 OrderedReady，因為mq集群也是有狀態應用，啟動和停止都需要有順序；
定義了持久化存儲，使用ceph sc；
定義了一個ingress,主要是用於訪問web管理頁面；

使用helm status my-rabbitmq-ha 來查看如何訪問rabbitmq

[[email protected] rabbitmq-ha]# helm status my-rabbitmq-ha
LAST DEPLOYED: Thu Apr 25 19:43:02 2019
NAMESPACE: kube-system
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME            TYPE    DATA  AGE
my-rabbitmq-ha  Opaque  4     6d16h

==> v1beta1/Role
NAME            AGE
my-rabbitmq-ha  6d16h

==> v1beta1/Ingress
NAME            HOSTS                       ADDRESS  PORTS  AGE
my-rabbitmq-ha  k8s.rabbitmq.maimaiti.site  80       6d16h

==> v1/PrometheusRule
NAME                            AGE
my-rabbitmq-ha-rabbitmq-alerts  6d16h

==> v1/ConfigMap
NAME            DATA  AGE
my-rabbitmq-ha  2     6d16h

==> v1/ServiceAccount
NAME            SECRETS  AGE
my-rabbitmq-ha  1        6d16h

==> v1beta1/RoleBinding
NAME            AGE
my-rabbitmq-ha  6d16h

==> v1/Service
NAME                      TYPE       CLUSTER-IP  EXTERNAL-IP  PORT(S)                               AGE
my-rabbitmq-ha-discovery  ClusterIP  None        <none>       15672/TCP,5672/TCP,4369/TCP           6d16h
my-rabbitmq-ha            ClusterIP  None        <none>       15672/TCP,5672/TCP,4369/TCP,9090/TCP  6d16h

==> v1beta1/StatefulSet
NAME            DESIRED  CURRENT  AGE
my-rabbitmq-ha  3        3        6d16h

==> v1/ServiceMonitor
NAME            AGE
my-rabbitmq-ha  6d16h

==> v1/Pod(related)
NAME              READY  STATUS   RESTARTS  AGE
my-rabbitmq-ha-0  2/2    Running  2         5d20h
my-rabbitmq-ha-1  2/2    Running  2         6d15h
my-rabbitmq-ha-2  2/2    Running  0         2d20h

NOTES:
** Please be patient while the chart is being deployed **

  Credentials:

    Username      : guest
    Password      : $(kubectl get secret --namespace kube-system my-rabbitmq-ha -o jsonpath="{.data.rabbitmq-password}" | base64 --decode)
    ErLang Cookie : $(kubectl get secret --namespace kube-system my-rabbitmq-ha -o jsonpath="{.data.rabbitmq-erlang-cookie}" | base64 --decode)

  RabbitMQ can be accessed within the cluster on port 5672 at my-rabbitmq-ha.kube-system.svc.cluster.local

  To access the cluster externally execute the following commands:

    export POD_NAME=$(kubectl get pods --namespace kube-system -l "app=rabbitmq-ha" -o jsonpath="{.items[0].metadata.name}")
    kubectl port-forward $POD_NAME --namespace kube-system 5672:5672 15672:15672
    kubectl port-forward $POD_NAME --namespace kube-system 9090:9090

  To Access the RabbitMQ AMQP port:

    amqp://127.0.0.1:5672/
  To Access the RabbitMQ Exporter metrics port:

    http://127.0.0.1:9090/ 

  To Access the RabbitMQ Management interface:

    URL : http://127.0.0.1:15672

[[email protected] rabbitmq-ha]#

好的，今天的博文就寫到這裏。由於51放假這幾天的大多時間都用來陪伴家人了，趁著今天值班就把這幾天的k8s實驗總結一下。最近這幾個月一直都在忙k8s的項目，後續還會補充一些helm安裝mysql、helm安裝jenkins,重點還會介紹一下k8s容器裏面的CI/CD是如何實現的。希望大家持續關註一下我的技術博客和公眾號。

helm的使用和redis、mq等公共組件的容器化實踐

1. Helm簡介:

1. Helm的概念

2. Helm的組成

3. Helm的安裝

4. 管理chart倉庫

5. Helm安裝hello-svc應用

6. 使用Helm安裝Redis-HA應用

推薦關註我的個人微信公眾號 “雲時代IT運維”,周期性更新最新的應用運維類技術文檔。關註虛擬化和容器技術、CI/CD、自動化運維等最新前沿運維技術和趨勢；

helm的使用和redis、mq等公共組件的容器化實踐

.NET Core RSA密鑰的xml、pkcs1、pkcs8格式轉換和JavaScript、Java等語言進行對接

linux伺服器意外關機重啟mariaDB、redis、ngnix等服務

小程式例項：用js方法splict()、indexOf()、push()、replace()等運算元組Array的增刪改查

memcache和redis、Mongodb優缺點及應用場景

利用 jodconverter 和openoffice 結合把doc、ppt 等一系列文件轉換成pdf

Get和Post、冪等、淨荷

五分鐘學後端技術：如何學習Redis、memcache等常用快取技術

項目實戰之FORM、復選框組件的實現

五、vue常用UI組件

weex 項目開發（五）自定義過濾函數和混合及自定義 Header 組件

小程序公共組件之地址選擇五級聯動

xadmin後臺頁面定制和添加服務器監控組件

基於vue以及vuex的dialog彈出框公共組件

django之權限管理公共組件

Python實現整蠱、木馬等黑客軟件!

react全局的公共組件-------彈框

react 全局公共組件-----動態彈窗 (dialog)

使用dynamic和MEF實現輕量級的AOP組件 ---- 系列文章

Android開啟doc、xlsx、ppt等office文件解決方案

helm的使用和redis、mq等公共組件的容器化實踐

1. Helm簡介:

1. Helm的概念

2. Helm的組成

3. Helm的安裝

4. 管理chart倉庫

5. Helm安裝hello-svc應用

6. 使用Helm安裝Redis-HA應用

推薦關註我的個人微信公眾號 “雲時代IT運維”,周期性更新最新的應用運維類技術文檔。關註虛擬化和容器技術、CI/CD、自動化運維等最新前沿運維技術和趨勢；

相關推薦