聽說Redis 3以後安裝叢集比較簡單，幾個命令就可以完全搞定，在自己的電腦上面安裝試了下，果然比較爽。

今天準備把Redis 3的叢集安裝在K8s上面，通常的做法都是在github上面查詢是不是有人已經做過，果然很多。下面的程式碼基本上來自於github上面。不過就算用這個面的程式碼，在自己部署的時候也存在不少的問題。因為github上面的都是大神，很比比較小的問題不會寫的很清楚。

所以： 如果這個文件（blog）如何解決不了問題，就到github上面去查詢別的方法。

1>>>>>>>>>>>映象說明

  使用的映象名:    ausov/redis:3.2-alpine

  先使用下面命令下載映象: 

docker pull ausov/redis:3.2-alpine

   這個映象如果無法下載，則需要自己手動build一下.

   這個是編譯需要使用的檔案

[[email protected] alpine]# pwd
/root/k8s-redis/3.2/alpine
[[email protected] alpine]# ls
docker-entrypoint.sh  Dockerfile  redis.conf

cat docker-entrypoint.sh 

#!/bin/sh

set -eox pipefail
#shopt -s nullglob

REDIS_CONF=${REDIS_CONF:-"/opt/k8s-redis/redis.conf"}

if [ "${1#-}" != "$1" ] || [ "${1%.conf}" != "$1" ]; then
  set -- redis-server "
 
[email protected]"
fi

if [ "$1" = 'redis-server' ] && [ -n "$SLAVEOF" ] && [ -z "$SENTINEL" ]; then
  echo "Starting Redis replica"
  set -- [email protected] "$REDIS_CONF" --slaveof "$SLAVEOF" 6379

elif [ "$1" = 'redis-server' ] && [ -n "$SENTINEL" ]; then
  echo "Starting Redis sentinel"

  while true; do
    redis-cli -h $SENTINEL INFO
    if [[ "$?" == "0" ]]; then
      break
    fi
    echo "Connecting to master failed.  Waiting..."
    sleep 10
  done

  echo "sentinel monitor primary $SENTINEL 6379 2" >> "$REDIS_CONF"
  echo "sentinel down-after-milliseconds primary 5000" >> "$REDIS_CONF"
  echo "sentinel failover-timeout primary 10000" >> "$REDIS_CONF"
  echo "sentinel parallel-syncs primary 1" >> "$REDIS_CONF"

  set -- 
 
[email protected] "$REDIS_CONF" --port 26379 --sentinel --protected-mode no

elif [ "$1" = 'redis-server' ]; then  
  echo "Starting Redis master"
  set -- [email protected] "$REDIS_CONF"
fi

exec "[email protected]"

cat redis.conf | grep -Ev '(^#|^$)' 

protected-mode no
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize no
supervised no
pidfile /var/run/redis_6379.pid
loglevel notice
logfile ""
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir ./
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

cat Dockerfile 

FROM redis:3.2-alpine

COPY ["redis.conf", "/opt/k8s-redis/"]
COPY ["docker-entrypoint.sh", "/usr/local/bin/"]

docker build -t "ausov/redis:3.2-alpine" .

2>>>>>>>>>>>部署安裝

部署安裝我們需要使用三個檔案，分別是:

primary.yml  secondary.yml  sentinel.yml

下面是檔案內容:

[[email protected] example]# cat primary.yml 
apiVersion: v1
kind: Service
metadata:
  name: redis-primary
  labels:
    app: redis-primary
spec:
  ports:
  - port: 6379
    name: redis-primary
  clusterIP: None
  selector:
    app: redis-primary
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: redis-primary
spec:
  serviceName: redis-primary
  replicas: 1
  template:
    metadata:
      labels:
        app: redis-primary
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: redis-primary
        image: ausov/redis:3.2-alpine
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 6379
          name: redis-primary
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
      annotations:
        volume.alpha.kubernetes.io/storage-class: anything
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

[[email protected] example]# cat secondary.yml 
apiVersion: v1
kind: Service
metadata:
  name: redis-secondary
  labels:
    app: redis-secondary
spec:
  ports:
  - port: 6379
    name: redis-secondary
  clusterIP: None
  selector:
    app: redis-secondary
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: redis-secondary
spec:
  serviceName: redis-secondary
  replicas: 2
  template:
    metadata:
      labels:
        app: redis-secondary
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: redis-secondary
        image: ausov/redis:3.2-alpine
        imagePullPolicy: IfNotPresent
        env:
        - name: SLAVEOF
          value: redis-primary-0.redis-primary
        ports:
        - containerPort: 6379
          name: redis-secondary
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
      annotations:
        volume.alpha.kubernetes.io/storage-class: anything
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

[[email protected] example]# cat sentinel.yml 
apiVersion: v1
kind: Service
metadata:
  name: redis-sentinel
  labels:
    app: redis-sentinel
spec:
  ports:
  - port: 26379
    name: redis-sentinel
  clusterIP: None
  selector:
    app: redis-sentinel
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: redis-sentinel
spec:
  serviceName: redis-sentinel
  replicas: 3
  template:
    metadata:
      labels:
        app: redis-sentinel
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: redis-sentinel
        image: ausov/redis:3.2-alpine
        imagePullPolicy: IfNotPresent
        env:
        - name: SENTINEL
          value: redis-primary-0.redis-primary
        ports:
        - containerPort: 26379
          name: redis-sentinel

 kubectl create -f primary.yml  -f secondary.yml  -f  sentinel.yml

建立完成後，檢視一下pod

kubectl get pod 

會得到以下資訊:

NAME                                READY     STATUS    RESTARTS   AGE

redis-primary-0                     1/1       Running   0          2d
redis-secondary-0                   1/1       Running   0          2d
redis-secondary-1                   1/1       Running   0          2d
redis-sentinel-0                    1/1       Running   0          2d
redis-sentinel-1                    1/1       Running   0          2d
redis-sentinel-2                    1/1       Running   0          2d

[[email protected] example]# kubectl get statefulset
NAME              DESIRED   CURRENT   AGE
redis-primary     1         1         2d
redis-secondary   2         2         2d
redis-sentinel    3         3         2d

檢視持久儲存的資訊:

[[email protected]er example]# kubectl get pvc

NAME                                       STATUS    VOLUME                                     CAPACITY   ACCESSMODES   AGE

data-redis-primary-0                       Bound     pvc-94534255-d614-11e6-955d-1866daed1b04   5Gi        RWO           2d
data-redis-secondary-0                     Bound     pvc-9a8f2080-d614-11e6-955d-1866daed1b04   5Gi        RWO           2d
data-redis-secondary-1                     Bound     pvc-9a9029b4-d614-11e6-955d-1866daed1b04   5Gi        RWO           2d

如果不是，用命令

describe pvc data-redis-primary-0  

cannot find volume plugin for alpha provisioning

則需要修改檔案:

/etc/kubernetes/manifests/kube-controller-manager.json 

增加引數: --enable-hostpath-provisioner=true

以下是截取了這個檔案的一部分:

        "name": "kube-controller-manager",
        "image": "gcr.io/google_containers/kube-controller-manager-amd64:v1.5.1",
        "command": [
          "kube-controller-manager",
          "--address=127.0.0.1",
          "--leader-elect",
          "--master=127.0.0.1:8080",
          "--cluster-name=kubernetes",
          "--root-ca-file=/etc/kubernetes/pki/ca.pem",
          "--service-account-private-key-file=/etc/kubernetes/pki/apiserver-key.pem",
          "--cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem",
          "--cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem",
          "--insecure-experimental-approve-all-kubelet-csrs-for-group=system:kubelet-bootstrap",
          "--allocate-node-cidrs=true",
          "--enable-hostpath-provisioner=true",
          "--cluster-cidr=10.245.0.0/16"
        ],

通過上面的配置，Redis基本就配置完成: 

這兩個WARNING暫不影響redis使用，解決方法過兩天再寫上。

11:M 04 Jan 09:45:23.515 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
11:M 04 Jan 09:45:23.515 # Server started, Redis version 3.2.1
11:M 04 Jan 09:45:23.515 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
11:M 04 Jan 09:45:23.516 * The server is now ready to accept connections on port 6379
OK