1. 程式人生 > >redis cluster 4.0.9 之四: redis-trib.rb add-node

redis cluster 4.0.9 之四: redis-trib.rb add-node

os: ubuntu 16.04
redis:4.0.9
ruby:2.4.4

ip規劃

192.168.0.101 7000,7001,7002
192.168.0.102 7000,7001,7002
192.168.0.103 7000,7001,7002

現在加三個redis例項
192.168.0.101 7003
192.168.0.102 7003
192.168.0.103 7003

參考之前的blog,在 ubuntu 16.04 安裝了 redis cluster 4.0.9,詳細過程略。

下面簡單記錄下新增 redis 例項的過程。

redis-trib.rb check

# /usr/redis/redis-4.0.9/src/redis-trib.rb check 192.168.0.101:7000
>>> Performing Cluster Check (using node 192.168.0.101:7000)
M: bae1665a3a55f5f2f2c224e1533118a1168b6a1d 192.168.0.101:7000
   slots:0-5460 (5461 slots) master
   2 additional replica(s)
M: fbf3b7827137251d03638436d28a2a9b82cea44c 192.168.0.102:7000
   slots:5461-10922 (5462 slots) master
   2 additional replica(s)
S: f5a34cea79ae1614241eb489808b9309514fa8eb 192.168.0.101:7002
   slots: (0 slots) slave
   replicates fbf3b7827137251d03638436d28a2a9b82cea44c
S: 550f9347512ee841cf558ef254dbdf3b4379f6a9 192.168.0.103:7001
   slots: (0 slots) slave
   replicates bae1665a3a55f5f2f2c224e1533118a1168b6a1d
M: ca76bd2110000f6285360e1d0979ac14fec7f86a 192.168.0.103:7000
   slots:10923-16383 (5461 slots) master
   2 additional replica(s)
S: d93a37d9fb2e526d01a5f4c6b6b9743674656286 192.168.0.101:7001
   slots: (0 slots) slave
   replicates ca76bd2110000f6285360e1d0979ac14fec7f86a
S: 464b9cab8863085b4bff880eca339a9e7983898c 192.168.0.103:7002
   slots: (0 slots) slave
   replicates fbf3b7827137251d03638436d28a2a9b82cea44c
S: bcdecbdb2528f2953145aa08c82d8c4a9a89e0a0 192.168.0.102:7002
   slots: (0 slots) slave
   replicates ca76bd2110000f6285360e1d0979ac14fec7f86a
S: 141c2cf5b07da8c0d5b4b7c6dc42335d7638946f 192.168.0.102:7001
   slots: (0 slots) slave
   replicates bae1665a3a55f5f2f2c224e1533118a1168b6a1d
[OK] All nodes agree about slots configurat./7003/redis.confion.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

基本情況如下:
192.168.0.101:7000 > 192.168.0.102:7001,192.168.0.103:7001
192.168.0.102:7000 > 192.168.0.101:7002,192.168.0.103:7002
192.168.0.103:7000 > 192.168.0.101:7001,192.168.0.102:7002

redis-trib.rb add-node

node1、node2、node3 節點建立7003

# cd /var/lib/redis/4.0.9/cluster-data
# cp -r ./7000 ./7003
# sed -i "s/7000/7003/g" ./7003/redis.conf
# rm ./7003/appendonly.aof ./7003/cluster_7000.conf ./7003/dump.rdb
# /usr/local/bin/redis-server /var/lib/redis/4.0.9/cluster-data/7003/redis.conf

node1 節點上操作

# /usr/redis/redis-4.0.9/src/redis-trib.rb
  add-node        new_host:new_port existing_host:existing_port
                  --slave
                  --master-id <arg>
# /usr/redis/redis-4.0.9/src/redis-trib.rb add-node 192.168.0.101:7003 192.168.0.101:7000

其中 existing_host:existing_port 可以為任何一個redis例項
new_host:new_port = 192.168.0.101:7003
existing_host:existing_port = 192.168.0.101:7000

# /usr/redis/redis-4.0.9/src/redis-trib.rb check 192.168.0.101:7000
>>> Performing Cluster Check (using node 192.168.0.101:7000)
M: bae1665a3a55f5f2f2c224e1533118a1168b6a1d 192.168.0.101:7000
   slots:0-5460 (5461 slots) master
   2 additional replica(s)
M: fbf3b7827137251d03638436d28a2a9b82cea44c 192.168.0.102:7000
   slots:5461-10922 (5462 slots) master
   2 additional replica(s)
S: f5a34cea79ae1614241eb489808b9309514fa8eb 192.168.0.101:7002
   slots: (0 slots) slave
   replicates fbf3b7827137251d03638436d28a2a9b82cea44c
S: 550f9347512ee841cf558ef254dbdf3b4379f6a9 192.168.0.103:7001
   slots: (0 slots) slave
   replicates bae1665a3a55f5f2f2c224e1533118a1168b6a1d
M: ca76bd2110000f6285360e1d0979ac14fec7f86a 192.168.0.103:7000
   slots:10923-16383 (5461 slots) master
   2 additional replica(s)
S: d93a37d9fb2e526d01a5f4c6b6b9743674656286 192.168.0.101:7001
   slots: (0 slots) slave
   replicates ca76bd2110000f6285360e1d0979ac14fec7f86a
S: 464b9cab8863085b4bff880eca339a9e7983898c 192.168.0.103:7002
   slots: (0 slots) slave
   replicates fbf3b7827137251d03638436d28a2a9b82cea44c
M: 4a77a0134068e2d9f2734213ead14bf9bce2e072 192.168.0.101:7003
   slots: (0 slots) master
   0 additional replica(s)
S: bcdecbdb2528f2953145aa08c82d8c4a9a89e0a0 192.168.0.102:7002
   slots: (0 slots) slave
   replicates ca76bd2110000f6285360e1d0979ac14fec7f86a
S: 141c2cf5b07da8c0d5b4b7c6dc42335d7638946f 192.168.0.102:7001
   slots: (0 slots) slave
   replicates bae1665a3a55f5f2f2c224e1533118a1168b6a1d
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

# /usr/redis/redis-4.0.9/src/redis-trib.rb info 192.168.0.101:7000
192.168.0.101:7000 (bae1665a...) -> 0 keys | 5461 slots | 2 slaves.
192.168.0.102:7000 (fbf3b782...) -> 1 keys | 5462 slots | 2 slaves.
192.168.0.103:7000 (ca76bd21...) -> 0 keys | 5461 slots | 2 slaves.
192.168.0.101:7003 (4a77a013...) -> 0 keys | 0 slots | 0 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.

可以看到剛才新加的節點為 master,沒有slave

M: 4a77a0134068e2d9f2734213ead14bf9bce2e072 192.168.0.101:7003
   slots: (0 slots) master
   0 additional replica(s)

給 192.168.0.101:7003 加兩個slave


# /usr/redis/redis-4.0.9/src/redis-trib.rb add-node --slave --master-id 4a77a0134068e2d9f2734213ead14bf9bce2e072 192.168.0.102:7003 192.168.0.101:7003
# /usr/redis/redis-4.0.9/src/redis-trib.rb add-node --slave --master-id 4a77a0134068e2d9f2734213ead14bf9bce2e072 192.168.0.103:7003 192.168.0.101:7003

# /usr/redis/redis-4.0.9/src/redis-trib.rb check 192.168.0.101:7000
>>> Performing Cluster Check (using node 192.168.0.101:7000)
M: bae1665a3a55f5f2f2c224e1533118a1168b6a1d 192.168.0.101:7000
   slots:0-5460 (5461 slots) master
   2 additional replica(s)
M: fbf3b7827137251d03638436d28a2a9b82cea44c 192.168.0.102:7000
   slots:5461-10922 (5462 slots) master
   2 additional replica(s)
S: f5a34cea79ae1614241eb489808b9309514fa8eb 192.168.0.101:7002
   slots: (0 slots) slave
   replicates fbf3b7827137251d03638436d28a2a9b82cea44c
S: b7ee4163c0d14e7c763a2c4072f10042c84e4ca6 192.168.0.103:7003
   slots: (0 slots) slave
   replicates 4a77a0134068e2d9f2734213ead14bf9bce2e072
S: 550f9347512ee841cf558ef254dbdf3b4379f6a9 192.168.0.103:7001
   slots: (0 slots) slave
   replicates bae1665a3a55f5f2f2c224e1533118a1168b6a1d
M: ca76bd2110000f6285360e1d0979ac14fec7f86a 192.168.0.103:7000
   slots:10923-16383 (5461 slots) master
   2 additional replica(s)
S: 179d2cc1d16d59c479df8536e7720977772e75c8 192.168.0.102:7003
   slots: (0 slots) slave
   replicates 4a77a0134068e2d9f2734213ead14bf9bce2e072
S: d93a37d9fb2e526d01a5f4c6b6b9743674656286 192.168.0.101:7001
   slots: (0 slots) slave
   replicates ca76bd2110000f6285360e1d0979ac14fec7f86a
S: 464b9cab8863085b4bff880eca339a9e7983898c 192.168.0.103:7002
   slots: (0 slots) slave
   replicates fbf3b7827137251d03638436d28a2a9b82cea44c
M: 4a77a0134068e2d9f2734213ead14bf9bce2e072 192.168.0.101:7003
   slots: (0 slots) master
   2 additional replica(s)
S: bcdecbdb2528f2953145aa08c82d8c4a9a89e0a0 192.168.0.102:7002
   slots: (0 slots) slave
   replicates ca76bd2110000f6285360e1d0979ac14fec7f86a
S: 141c2cf5b07da8c0d5b4b7c6dc42335d7638946f 192.168.0.102:7001
   slots: (0 slots) slave
   replicates bae1665a3a55f5f2f2c224e1533118a1168b6a1d
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

# /usr/redis/redis-4.0.9/src/redis-trib.rb info 192.168.0.101:7000
192.168.0.101:7000 (bae1665a...) -> 0 keys | 5461 slots | 2 slaves.
192.168.0.102:7000 (fbf3b782...) -> 1 keys | 5462 slots | 2 slaves.
192.168.0.103:7000 (ca76bd21...) -> 0 keys | 5461 slots | 2 slaves.
192.168.0.101:7003 (4a77a013...) -> 0 keys | 0 slots | 2 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.

現在基本情況如下:
192.168.0.101:7000 > 192.168.0.102:7001,192.168.0.103:7001
192.168.0.102:7000 > 192.168.0.101:7002,192.168.0.103:7002
192.168.0.103:7000 > 192.168.0.101:7001,192.168.0.102:7002

192.168.0.101:7003 > 192.168.0.102:7003,192.168.0.103:7003

注意觀察,新master 192.168.0.101:7003 並沒有分配 slots

M: 4a77a0134068e2d9f2734213ead14bf9bce2e072 192.168.0.101:7003
   slots: (0 slots) master
   2 additional replica(s)

此時的狀況是雖然成功添加了1個master,2個salve,但是並沒有分配slots。

redis-trib.rb reshard

現在手動重新分配 slots

# /usr/redis/redis-4.0.9/src/redis-trib.rb
  reshard         host:port
                  --from <arg>
                  --to <arg>
                  --slots <arg>
                  --yes
                  --timeout <arg>
                  --pipeline <arg>
# /usr/redis/redis-4.0.9/src/redis-trib.rb reshard 192.168.0.101:7003

其中 host:port 為叢集中的任何一個例項即可。
下面為 redis-trib.rb reshard 的一些輸入簡介,可以參考下。

How many slots do you want to move (from 1 to 16384)? 

表示移動多少個 slots,建議使用 16384/4=4096,這裡輸入4000

What is the receiving node ID? 

表示新master 192.168.0.101:7003 的 id,這裡輸入 4a77a0134068e2d9f2734213ead14bf9bce2e072

Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:

建議輸入 all,從所有節點上移動部分 slots 到新的master上。

    Moving slot 12253 from ca76bd2110000f6285360e1d0979ac14fec7f86a
    Moving slot 12254 from ca76bd2110000f6285360e1d0979ac14fec7f86a
    Moving slot 12255 from ca76bd2110000f6285360e1d0979ac14fec7f86a
Do you want to proceed with the proposed reshard plan (yes/no)?

輸入yes,確認用。

在遷移中碰到了如下錯誤

Moving slot 5798 from 192.168.0.102:7000 to 192.168.0.101:7003: 
[ERR] Calling MIGRATE: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)

check 時有 WARNING 提示

# /usr/redis/redis-4.0.9/src/redis-trib.rb check 192.168.0.101:7000

>>> Check for open slots...
[WARNING] Node 192.168.0.102:7000 has slots in migrating state (5798).
[WARNING] Node 192.168.0.101:7003 has slots in importing state (5798).
[WARNING] The following slots are open: 5798

參考 http://www.cnblogs.com/chenmh/p/9221672.html 使用解決辦法2。

# redis-cli -h 192.168.0.101 -p 7003 -c
192.168.0.101:7003> cluster setslot 5798 stable
OK

# redis-cli -h 192.168.0.102 -p 7000 -c
192.168.0.102:7000> cluster setslot 5798 stable
OK
# /usr/redis/redis-4.0.9/src/redis-trib.rb fix 192.168.0.101:7003
# /usr/redis/redis-4.0.9/src/redis-trib.rb fix 192.168.0.102:7000

檢查

# /usr/redis/redis-4.0.9/src/redis-trib.rb check 192.168.0.101:7000
M: 4a77a0134068e2d9f2734213ead14bf9bce2e072 192.168.0.101:7003
   slots:5461-5797 (337 slots) master
   2 additional replica(s)
   
# /usr/redis/redis-4.0.9/src/redis-trib.rb info 192.168.0.101:7000
192.168.0.101:7000 (bae1665a...) -> 0 keys | 5461 slots | 2 slaves.
192.168.0.102:7000 (fbf3b782...) -> 1 keys | 5125 slots | 2 slaves.
192.168.0.103:7000 (ca76bd21...) -> 0 keys | 5461 slots | 2 slaves.
192.168.0.101:7003 (4a77a013...) -> 0 keys | 337 slots | 2 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.

可以看到 192.168.0.101:7003 的slots遷移成功了一部分,再手動繼續遷移時依舊報錯。

Moving slot 5798 from 192.168.0.102:7000 to 192.168.0.101:7003: 
[ERR] Calling MIGRATE: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)

後經過分析後發現是是使用gem 安裝的 redis 介面版本過高,隨後降級到版本 3.3.3

# ruby --version
ruby 2.4.4p296 (2018-03-28 revision 63013) [x86_64-linux]
# gem list |grep -i redis
redis (4.0.2)
# gem uninstall redis --version 4.0.2
# gem install redis -v 3.3.3

再次reshard 成功執行。


# /usr/redis/redis-4.0.9/src/redis-trib.rb check 192.168.0.101:7000
>>> Performing Cluster Check (using node 192.168.0.101:7000)
M: 4a77a0134068e2d9f2734213ead14bf9bce2e072 192.168.0.101:7003
   slots:0-1971,5461-6895,10923-12893 (5378 slots) master
   2 additional replica(s)

參考:
http://www.cnblogs.com/chenmh/p/9221672.html