1. 程式人生 > >redis群集節點刪除和添加

redis群集節點刪除和添加

ror fail ould cati example 提供服務 tool constant bdb

redis集群介紹

多個redis節點實現網絡互連,數據共享
所有節點都是一主一從(或一主多從),其中從不提供服務,僅作為備份備用,保證redis數據的安全性
不支持同時處理多個鍵,redis集群寫入數據是向集群裏隨機寫入的(比如raid5磁盤陣列格式),因為redis集群需要把鍵平均分布在各個節點上,並發量很高的情況下同時創建鍵值會降低性能並會導致不可預估的行為
redis集群支持在線增加、刪除節點,在擴充redis的情況下,是非常容易擴容集群服務數量的
客戶端可以連接任意一個主節點進行讀寫

技術分享圖片

redis集群配置

redis群集場景設定
使用兩臺機器,分別開啟三個redis服務(基於不同端口)
服務器ip為192.168.1.223和192.168.1.234

192.168.1.234上開啟7000、7002、7002三個端口,設置角色全部為主
192.168.1.223上開啟7001、7003、7005三個端口,設置角色全部為從
兩臺機器都要編譯安裝redis,編輯三個不同的redis.conf,分別設置不同端口號、dir等參數,還需要增加cluster相關參數,啟動為六個redis服務
首先在192.168.1.234機器上拷貝三個不同的redis配置文件,修改配置文件的端口、存儲數據文件的路徑等,並創建配置文件中指定存儲數據文件的目錄
在192.168.1.234機器上創建三個基於不同端口的配置文件,如:

[root@nfs1 /]# cat /etc/redis_7000.conf 
port 7000
bind 192.168.1.234
daemonize yes
pidfile /var/run/redis_7000.pid
dir /data/redis_data/7000
cluster-enabled yes
cluster-config-file nodes_7000.conf
cluster-node-timeout 10100
appendonly yes
[root@nfs1 /]# cat /etc/redis_7002.conf 
port 7002
bind 192.168.1.234
daemonize yes
pidfile /var/run/redis_7002.pid
dir /data/redis_data/7002
cluster-enabled yes
cluster-config-file nodes_7002.conf
cluster-node-timeout 10100
appendonly yes
[root@nfs1 /]# cat /etc/redis_7004.conf 
port 7004
bind 192.168.1.234
daemonize yes
pidfile /var/run/redis_7004.pid
dir /data/redis_data/7004
cluster-enabled yes
cluster-config-file nodes_7004.conf
cluster-node-timeout 10100
appendonly yes

啟動這三個基於不同端口的redis服務,並查看其啟動端口信息

[root@nfs1 /]# /usr/local/bin/redis-server /etc/redis_7000.conf 
91981:C 14 Nov 2018 18:31:38.224 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
91981:C 14 Nov 2018 18:31:38.224 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=91981, just started
91981:C 14 Nov 2018 18:31:38.224 # Configuration loaded
[root@nfs1 /]# /usr/local/bin/redis-server /etc/redis_7002.conf 
91986:C 14 Nov 2018 18:31:46.752 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
91986:C 14 Nov 2018 18:31:46.752 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=91986, just started
91986:C 14 Nov 2018 18:31:46.752 # Configuration loaded
[root@nfs1 /]# /usr/local/bin/redis-server /etc/redis_7004.conf 
91991:C 14 Nov 2018 18:31:50.830 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
91991:C 14 Nov 2018 18:31:50.830 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=91991, just started
91991:C 14 Nov 2018 18:31:50.830 # Configuration loaded
[root@nfs1 /]# netstat -ntlp |grep redis
tcp ? ? ? ?0 ? ? ?0 192.168.1.234:7000 ? ? ?0.0.0.0:* ? ? ? ? ? ? ? LISTEN ? ? ?91982/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.234:7002 ? ? ?0.0.0.0:* ? ? ? ? ? ? ? LISTEN ? ? ?91987/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.234:7004 ? ? ?0.0.0.0:* ? ? ? ? ? ? ? LISTEN ? ? ?91992/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.234:17000 ? ? 0.0.0.0:* ? ? ? ? ? ? ? LISTEN ? ? ?91982/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.234:17002 ? ? 0.0.0.0:* ? ? ? ? ? ? ? LISTEN ? ? ?91987/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.234:17004 ? ? 0.0.0.0:* ? ? ? ? ? ? ? LISTEN ? ? ?91992/redis-server 
192.168.1.223的redis也是按照這種方式啟動多個redis服務的
[root@Huaching /]# /usr/local/bin/redis-server /etc/redis_7001.conf 
26608:C 15 Nov 2018 00:00:10.297 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
26608:C 15 Nov 2018 00:00:10.297 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=26608, just started
26608:C 15 Nov 2018 00:00:10.297 # Configuration loaded
[root@Huaching /]# /usr/local/bin/redis-server /etc/redis_7003.conf 
26622:C 15 Nov 2018 00:00:22.984 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
26622:C 15 Nov 2018 00:00:22.985 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=26622, just started
26622:C 15 Nov 2018 00:00:22.985 # Configuration loaded
[root@Huaching /]# /usr/local/bin/redis-server /etc/redis_7005.conf 
26627:C 15 Nov 2018 00:00:27.128 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
26627:C 15 Nov 2018 00:00:27.128 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=26627, just started
26627:C 15 Nov 2018 00:00:27.128 # Configuration loaded
[root@Huaching /]# netstat -antp |grep redis
tcp ? ? ? ?0 ? ? ?0 192.168.1.200:7003 ? ? ? ? ?0.0.0.0:* ? ? ? ? ? ? ? ? ? LISTEN ? ? ?26623/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.200:7005 ? ? ? ? ?0.0.0.0:* ? ? ? ? ? ? ? ? ? LISTEN ? ? ?26628/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.200:17001 ? ? ? ? 0.0.0.0:* ? ? ? ? ? ? ? ? ? LISTEN ? ? ?26609/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.200:17003 ? ? ? ? 0.0.0.0:* ? ? ? ? ? ? ? ? ? LISTEN ? ? ?26623/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.200:17005 ? ? ? ? 0.0.0.0:* ? ? ? ? ? ? ? ? ? LISTEN ? ? ?26628/redis-server ?
tcp ? ? ? ?0 ? ? ?0 192.168.1.200:7001 ? ? ? ? ?0.0.0.0:* ? ? ? ? ? ? ? ? ? LISTEN ? ? ?26609/redis-server ?

啟動多個redis服務後接下來安裝ruby中間件實現群集功能

redis群集安裝ruby

安裝ruby只需要在一臺服務器上進行安裝運行
安裝Development的軟件組,安裝過程省略

[root@nfs1 redis_data]# yum -y groupinstall "Development Tools"

安裝需要支持的庫,安裝過程省略

[root@nfs1 redis_data]# yum install gdbm-devel linbdb4-devel libffi-devel libyaml libyaml-devel ncurses-devel openssl_devel readline-devel tcl-deve

創建編譯rpm包的目錄,下載ruby的源碼包文件,將源碼包下載只剛剛創建的目錄下
如果下載有rpm編譯器,則執行編譯器來生成rpm包進行安裝,如果沒有就使用./configure編譯安裝

[root@nfs1 ~]# mkdir -p /root/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
[root@nfs1 ~]# rpmbuild -bb /root/rpmbuild/SPECS/ruby22x.spec
[root@nfs1 ~]# yum -y localinstall rpmbuild/RPMS/x86_64/ruby-2.4.5-1.e17.centos.x86_64.rpm

由於這裏我無法找到ruby的rpm編譯器,所以無法編譯成rpm包。這裏依舊使用解壓源碼包,使用./configure來編譯安裝

[root@nfs1 ruby-2.4.5]# ./configure
[root@nfs1 ruby-2.4.5]# make 
-----------------------------編譯內容
  Files: 890
  Classes: 1312 ( 549 undocumented)
  Modules: 275 ( 110 undocumented)
  Constants: 2143 ( 538 undocumented)
  Attributes: 1053 ( 247 undocumented)
  Methods: 9837 (2106 undocumented)
  Total: 14620 (3550 undocumented)
 ? 75.72% documented
  Elapsed: 86.3s
[root@nfs1 ruby-2.4.5]# make install
-----------------------------編譯內容
installing bundle gems: ? ? ? /usr/local/lib/ruby/gems/2.4.0 (build_info, cache, doc, extensions, gems, specifications)
 ? ? ? ? ? ? ? ? ? ? ? ? ? ?  minitest 5.10.1
 ? ? ? ? ? ? ? ? ? ? ? ? ? ?  xmlrpc 0.2.1
 ? ? ? ? ? ? ? ? ? ? ? ? ? ?  test-unit 3.2.3
 ? ? ? ? ? ? ? ? ? ? ? ? ? ?  net-telnet 0.1.1
 ? ? ? ? ? ? ? ? ? ? ? ? ? ?  rake 12.0.0
 ? ? ? ? ? ? ? ? ? ? ? ? ? ?  did_you_mean 1.1.0
 ? ? ? ? ? ? ? ? ? ? ? ? ? ?  power_assert 0.4.1
installing rdoc: ? ? ? ? ? ?  /usr/local/share/ri/2.4.0/system
installing capi-docs: ? ? ? ? /usr/local/share/doc/ruby

編譯安裝完成後可以查看下ruby的版本號,驗證是否編譯安裝正確

[root@nfs1 ruby-2.4.5]# ruby -v
ruby 2.4.5p335 (2018-10-18 revision 65137) [x86_64-linux]
安裝redis的ruby集群的相關命令
[root@nfs1 ruby-2.4.5]# gem install redis
Fetching: redis-4.0.3.gem (100%)
Successfully installed redis-4.0.3
Parsing documentation for redis-4.0.3
Installing ri documentation for redis-4.0.3
Done installing documentation for redis after 1 seconds
1 gem installed

拷貝redis的ruby命令到/usr/local/bin目錄下,新版本舍棄了redis-trib.rb的這個命令,如果安裝新版本,可以不執行此步操作

[root@nfs1 ruby-2.4.5]# cp /usr/local/src/redis-5.0.0/src/redis-trib.rb /usr/local/bin/

在新版本的redis中,使用redis-trib.rb來創建redis的群集會提示版本命令不可用,因為redis-trib.rb命令被舍棄不再使用。具體信息如下:

WARNING: redis-trib.rb is not longer available!
You should use redis-cli instead.
All commands and features belonging to redis-trib.rb have been moved
to redis-cli.
In order to use them you should call redis-cli with the --cluster
option followed by the subcommand name, arguments and options.
Use the following syntax:
redis-cli --cluster SUBCOMMAND [ARGUMENTS] [OPTIONS]
Example:
redis-cli --cluster create 192.168.1.234:700 192.168.1.234:7002 192.168.1.234:7004 192.168.1.200:7001 192.168.1.200:7003 192.168.1.200:7005 --cluster-replicas 1
To get help about all subcommands, type:
redis-cli --cluster help

從上面提示可以了解到redis-trub.rb命令已經被redis-cli代替使用
那麽我們現在使用redis-cli來創建群集
格式為redis-cli --cluster create ?ip:prot ?ip:port ...... --cluster-replicas 1
這裏我創建的群集如下,在確認好群集信息後,會提示你是否按照這個主從指定的方式保存

[root@nfs1 ruby-2.4.5]# redis-cli --cluster create 192.168.1.234:7000 192.168.1.234:7002 192.168.1.234:7004 192.168.1.200:7001 192.168.1.200:7003 192.168.1.200:7005 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.1.200:7003 to 192.168.1.234:7000 ? ? ? ? ? ? ? ? 這裏指定了主從關系:從redis-->主redis
Adding replica 192.168.1.234:7004 to 192.168.1.200:7001
Adding replica 192.168.1.200:7005 to 192.168.1.234:7002
M: 3af55accd31b6a057945c4b9e9378c7932d5f1f8 192.168.1.234:7000 ? ? ? ?  輸出信息中的M為master ? S則是slave
 ? slots:[0-5460] (5461 slots) master
M: d8418ab100f8f32539fc60a7c8ead5927591f52a 192.168.1.234:7002 ? ? ? ? 
 ? slots:[10923-16383] (5461 slots) master
S: 2a642a2e14d15ae8cb77b3ef04f58d32cfb09933 192.168.1.234:7004
 ? replicates 3e8c289a64638da55b0e812f924aaa242efbc315
M: 3e8c289a64638da55b0e812f924aaa242efbc315 192.168.1.200:7001
 ? slots:[5461-10922] (5462 slots) master
S: 9882ec7d971ec8b159ff126ac8e1ce480a6ba9e0 192.168.1.200:7003
 ? replicates 3af55accd31b6a057945c4b9e9378c7932d5f1f8
S: a3c7b8a393f2b4d3b9838753fb6496c3861bbb69 192.168.1.200:7005
 ? replicates d8418ab100f8f32539fc60a7c8ead5927591f52a
Can I set the above configuration? (type ‘yes‘ to accept): yes

在上面確認交互的地方確認保存後會輸出兩個ok信息,可以判斷redis群集創建成功

Can I set the above configuration? (type ‘yes‘ to accept): yes
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

redis創建集群

redis集群創建好後,登錄redis時需要指定登入ip和端口,創建幾個key值查看redis存儲的redis信息
-c ?登入redis集群模式?
-h 指定集群中的一個redis

[root@nfs1 ruby-2.4.5]# redis-cli -c -h 192.168.1.234 -p 7000
192.168.1.234:7000> set key1 123
-> Redirected to slot [9189] located at 192.168.1.200:7001
OK
192.168.1.200:7001> set key3 abc
-> Redirected to slot [935] located at 192.168.1.234:7000
OK
192.168.1.234:7000> set key2 ABC
OK
192.168.1.234:7000> set key4 ABC
-> Redirected to slot [13120] located at 192.168.1.234:7002
OK
192.168.1.234:7002> set key5 ABC
-> Redirected to slot [9057] located at 192.168.1.200:7001
OK
192.168.1.200:7001> set key6 ABC
-> Redirected to slot [4866] located at 192.168.1.234:7000
OK

查看群集狀態信息,redis-cli代替了redis-trib.rb,所以這裏使用新的命令格式。可以看出群集的redis信息狀態

[root@nfs1 ruby-2.4.5]# redis-cli --cluster check 192.168.1.234:7000
192.168.1.234:7000 (3af55acc...) -> 4 keys | 5461 slots | 1 slaves.
192.168.1.234:7002 (d8418ab1...) -> 1 keys | 5461 slots | 1 slaves.
192.168.1.200:7001 (3e8c289a...) -> 2 keys | 5462 slots | 1 slaves.
[OK] 7 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.234:7000)
M: 3af55accd31b6a057945c4b9e9378c7932d5f1f8 192.168.1.234:7000
 ? slots:[0-5460] (5461 slots) master
 ? 1 additional replica(s)
S: a3c7b8a393f2b4d3b9838753fb6496c3861bbb69 192.168.1.200:7005
 ? slots: (0 slots) slave
 ? replicates d8418ab100f8f32539fc60a7c8ead5927591f52a
M: d8418ab100f8f32539fc60a7c8ead5927591f52a 192.168.1.234:7002
 ? slots:[10923-16383] (5461 slots) master
 ? 1 additional replica(s)
M: 3e8c289a64638da55b0e812f924aaa242efbc315 192.168.1.200:7001
 ? slots:[5461-10922] (5462 slots) master
 ? 1 additional replica(s)
S: 2a642a2e14d15ae8cb77b3ef04f58d32cfb09933 192.168.1.234:7004
 ? slots: (0 slots) slave
 ? replicates 3e8c289a64638da55b0e812f924aaa242efbc315
S: 9882ec7d971ec8b159ff126ac8e1ce480a6ba9e0 192.168.1.200:7003
 ? slots: (0 slots) slave
 ? replicates 3af55accd31b6a057945c4b9e9378c7932d5f1f8
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

redis群集的管理命令

cluster nodes ? 列出集群節點

[root@nfs1 ruby-2.4.5]# redis-cli -c -h 192.168.1.234 -p 7000 ? ? ? ? 登入任意一個redis中
192.168.1.234:7000> CLUSTER NODES
a3c7b8a393f2b4d3b9838753fb6496c3861bbb69 192.168.1.200:7005@17005 slave d8418ab100f8f32539fc60a7c8ead5927591f52a 0 1542270025000 6 connected
d8418ab100f8f32539fc60a7c8ead5927591f52a 192.168.1.234:7002@17002 master - 0 1542270024896 2 connected 10923-16383
3e8c289a64638da55b0e812f924aaa242efbc315 192.168.1.200:7001@17001 master - 0 1542270027918 4 connected 5461-10922
2a642a2e14d15ae8cb77b3ef04f58d32cfb09933 192.168.1.234:7004@17004 slave 3e8c289a64638da55b0e812f924aaa242efbc315 0 1542270026906 4 connected
3af55accd31b6a057945c4b9e9378c7932d5f1f8 192.168.1.234:7000@17000 myself,master - 0 1542270024000 1 connected 0-5460
9882ec7d971ec8b159ff126ac8e1ce480a6ba9e0 192.168.1.200:7003@17003 slave 3af55accd31b6a057945c4b9e9378c7932d5f1f8 0 1542270028928 5 connected

cluster info ? 查看集群信息

192.168.1.234:7000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:1594
cluster_stats_messages_pong_sent:1603
cluster_stats_messages_sent:3197
cluster_stats_messages_ping_received:1598
cluster_stats_messages_pong_received:1594
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:3197

所有的新添加的redis節點,都是為master角色
cluster meet ? ip ?port
在192.168.1.200上新啟動一個redis,監聽端口為7007。登入群集中執行redis節點添加命令

[root@nfs1 redis_data]# redis-cli -c -h 192.168.1.234 -p 7000
192.168.1.234:7000> cluster meet 192.168.1.200 7007
OK
192.168.1.234:7000> cluster nodes ? ? ? ? ? ? ? ?  //第一條就是新添加的節點信息
cff62edb40a4df11d8390bc28d254abca16188ba 192.168.1.200:7007@17007 master - 0 1542270675527 0 connected
a3c7b8a393f2b4d3b9838753fb6496c3861bbb69 192.168.1.200:7005@17005 slave d8418ab100f8f32539fc60a7c8ead5927591f52a 0 1542270677550 6 connected
d8418ab100f8f32539fc60a7c8ead5927591f52a 192.168.1.234:7002@17002 master - 0 1542270678560 2 connected 10923-16383
----------------省略

再添加一個新的redis節點,並將新節點指定為從節點
cluster reolicate ?master node info ? ? ? ? ? ?//登入需要設定為從節點的redis,執行命令指定主節點
指定從節點需要登入到那個redis節點當中操作,而不是在集群中指定,在登入節點中指定本節點的master
再次添加一個redis節點

192.168.1.234:7000> cluster meet 192.168.1.234 7006
OK
cff62edb40a4df11d8390bc28d254abca16188ba 192.168.1.200:7007@17007 master - 0 1542270675527 0 connected
24f1c355bc92a0ab15a6077b1bbf4bd510dfa132 192.168.1.234:7006@17006 master - 0 1542270869918 7 connected

退出節點登入端口為7006的節點,

192.168.1.234:7000> exit
[root@nfs1 redis_data]# redis-cli -c -h 192.168.1.234 -p 7006
192.168.1.234:7006> CLUSTER REPLICATE cff62edb40a4df11d8390bc28d254abca16188ba
OK

再次查看群集節點狀態信息,可以看到端口7006的redis已經成為7007redis的slave節點了

192.168.1.234:7006> cluster nodes
cff62edb40a4df11d8390bc28d254abca16188ba 192.168.1.200:7007@17007 master - 0 1542271290173 0 connected
24f1c355bc92a0ab15a6077b1bbf4bd510dfa132 192.168.1.234:7006@17006 myself,slave cff62edb40a4df11d8390bc28d254abca16188ba 0 1542271292000 7 connected

移除一個redis節點
cluster forget? ? node_id
在移除某個redis節點之前,首先不能在登入該節點當中,否則不能正常移除該節點,當前處於7006的redis當中,移除節點提示信息:

24f1c355bc92a0ab15a6077b1bbf4bd510dfa132 192.168.1.234:7006@17006 myself,slave cff62edb40a4df11d8390bc28d254abca16188ba 0 1542271292000 7 connected
192.168.1.234:7006> cluster forget 24f1c355bc92a0ab15a6077b1bbf4bd510dfa132
(error) ERR I tried hard but I can‘t forget myself...

必須登入其他節點移除端口為7006的redis節點

[root@nfs1 redis_data]# redis-cli -c -h 192.168.1.234 -p 7000

移除7006節點之前查看所有節點信息

192.168.1.234:7000> CLUSTER NODES

技術分享圖片

拷貝7006節點的node_id,移除7006節點

192.168.1.234:7000> cluster forget 24f1c355bc92a0ab15a6077b1bbf4bd510dfa132
OK

移除7006節點後

192.168.1.234:7000> CLUSTER NODES

技術分享圖片

保存redis集群的配置文件
cluster saveconfig ?

保存配置文件,會在當前登入的redis數據存儲路徑下增加redis配置內容

192.168.1.234:7000> cluster saveconfig
OK
192.168.1.234:7000> exit

技術分享圖片

redis群集節點刪除和添加