1. 程式人生 > >redis-cluster叢集詳解

redis-cluster叢集詳解

一 叢集環境搭建
1 叢集環境介紹:
虛擬機器6臺:
192.168.4.51
192.168.4.52
192.168.4.53
192.168.4.54
192.168.4.55
192.168.4.56
每臺虛擬機器安裝redis服務,修改redis配置檔案IP地址為自身IP地址,取消迴環IP,不開啟密碼驗證。

2 修改主配置檔案 (6臺)

[[email protected] ~]# vim /etc/redis/6379.conf
815 cluster-enabled yes                                                //開啟叢集
823 cluster-config-file nodes-6351.conf                    //本機用來儲存叢集資訊的檔名
829 cluster-node-timeout 5000                                   //叢集節點超時時間

3 重啟redis服務,檢視服務埠(6臺)

[[email protected] ~]# ss -antulp | grep redis
tcp    LISTEN     0      128    192.168.4.51:6351                  *:*                   users:(("redis-server",pid=2122,fd=6))
tcp    LISTEN     0      128    192.168.4.51:16351                 *:*                  users:(("redis-server",pid=2122,fd=8))

4 檢視叢集資訊檔案原始狀態:

[[email protected] ~]#  cd /var/lib/redis/6379
[[email protected] 6379]# cat nodes-6355.conf 
e0f9f77ba2bd2468baa1a2e9d88d494210d406bf :[email protected] myself,master - 0 0 0 connected               //顯示自己的ID值 
vars currentEpoch 0 lastVoteEpoch 0

檢視叢集原始狀態2:

192.168.4.55:6355> cluster info
cluster_state:fail                                                   //建立集群后才顯示OK
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0

192.168.4.55:6355> cluster nodes                        //檢視叢集中的節點(目前只能看到主機,還沒有建立叢集)
e0f9f77ba2bd2468baa1a2e9d88d494210d406bf :
[email protected]
myself,master - 0 0 0 connected

二 建立叢集
1 部署管理主機:
在51做redis伺服器的同時,同時做管理叢集的的伺服器
rubygems安裝.gem結尾的軟體包。ruby是一個直譯器,解釋ruby指令碼

[[email protected] ~]# yum -y install ruby rubygems
[[email protected] redis-cluster]# rpm -ivh ruby-devel-2.0.0.648-30.el7.x86_64.rpm
[[email protected] redis-cluster]# gem install redis-3.2.1.gem

2 為建立叢集的指令碼redis-trib.rb設定全域性環境變數

[[email protected] src]# echo $PATH
/root/perl5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
[[email protected] ~]# cd /root/redis2/redis-4.0.8/src
[[email protected] src]# ls *.rb
redis-trib.rb
[[email protected] ~]# mkdir /root/bin
[[email protected] src]# cp redis-trib.rb /root/bin
[[email protected] src]# ls /root/bin
redis-trib.rb
[[email protected] src]# redis-trib.rb help          //指令碼可以在全域性環境下單獨使用

3 建立叢集:(環境為3主3從)

[[email protected] ~]# redis-trib.rb create --replicas 1 \    //replicas 1代表一主一從
> 192.168.4.51:6351 192.168.4.52:6352 \
> 192.168.4.53:6353 192.168.4.54:6354 \
> 192.168.4.55:6355 192.168.4.56:6356

>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.4.51:6351
192.168.4.52:6352
192.168.4.53:6353
Adding replica 192.168.4.55:6355 to 192.168.4.51:6351
Adding replica 192.168.4.56:6356 to 192.168.4.52:6352
Adding replica 192.168.4.54:6354 to 192.168.4.53:6353
M: 88e7b3ed90d604cb3170309b084c82defcb32e8e 192.168.4.51:6351
   slots:0-5460 (5461 slots) master                    //第一個槽的範圍0-5460
M: 8cb2e1a1c9df060d95ec5f7c0094bed8d69979b6 192.168.4.52:6352
   slots:5461-10922 (5462 slots) master                //第二個槽的範圍5461-10922
M: aa1dd1ea0e5328541764f81541046bfd14acf20e 192.168.4.53:6353
   slots:10923-16383 (5461 slots) master              //第三個槽的範圍10923-16383
S: f71b5bc43cf4be71516f87ca56224d81a10795c8 192.168.4.54:6354
   replicates aa1dd1ea0e5328541764f81541046bfd14acf20e
S: e0f9f77ba2bd2468baa1a2e9d88d494210d406bf 192.168.4.55:6355
   replicates 88e7b3ed90d604cb3170309b084c82defcb32e8e
S: f169df56c739577d2d0cba2f883840a32631d511 192.168.4.56:6356
   replicates 8cb2e1a1c9df060d95ec5f7c0094bed8d69979b6
Can I set the above configuration? (type 'yes' to accept): yes
2.168.4.55:6355 192.168.4.56:6356

[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.                     //建立成功,總槽數16384個

################################
建立叢集時報錯解析:

Can I set the above configuration? (type 'yes' to accept): yes
/usr/local/share/gems/gems/redis-3.2.1/lib/redis/client.rb:113:in `call': ERR Slot 16287 is already busy (Redis::CommandError)
	from /usr/local/share/gems/gems/redis-3.2.1/lib/redis.rb:2556:in `block in method_missing'
	from /usr/local/share/gems/gems/redis-3.2.1/lib/redis.rb:37:in `block in synchronize'

檢視51上的叢集日誌檔案,在叢集建立未完成時,連線槽均為0

[[email protected] ~]# cat /var/lib/redis/6379/nodes-6351.conf
刪除/var/lib/redis/6379/目錄下的兩個檔案 dump.rdb  nodes-6351.conf
重新啟動redis服務
繼續執行建立叢集命令即可

#################################################
4 測試:
1)檢測叢集中的某個redid的master主機:

[[email protected] ~]# redis-trib.rb check 192.168.4.52:6352
>>> Performing Cluster Check (using node 192.168.4.52:6352)
M: 8cb2e1a1c9df060d95ec5f7c0094bed8d69979b6 192.168.4.52:6352
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
M: aa1dd1ea0e5328541764f81541046bfd14acf20e 192.168.4.53:6353
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: f169df56c739577d2d0cba2f883840a32631d511 192.168.4.56:6356
   slots: (0 slots) slave
   replicates 8cb2e1a1c9df060d95ec5f7c0094bed8d69979b6
S: f71b5bc43cf4be71516f87ca56224d81a10795c8 192.168.4.54:6354
   slots: (0 slots) slave
   replicates aa1dd1ea0e5328541764f81541046bfd14acf20e
S: e0f9f77ba2bd2468baa1a2e9d88d494210d406bf 192.168.4.55:6355
   slots: (0 slots) slave
   replicates 88e7b3ed90d604cb3170309b084c82defcb32e8e
M: 88e7b3ed90d604cb3170309b084c82defcb32e8e 192.168.4.51:6351
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

通過檢視
主庫192.168.4.51 它的從庫為:192.168.4.55
主庫192.168.4.52 它的從庫為:192.168.4.56
主庫192.168.4.53 它的從庫為:192.168.4.54

2 )客戶端192.168.4.50 訪問叢集管理伺服器51,檢測redis叢集的分散式儲存策略

 [[email protected] ~]# redis-cli -c -h 192.168.4.51 -p 6351            //訪問叢集時需要加-c選項
192.168.4.51:6351> keys *
(empty list or set)
192.168.4.51:6351> set x 100
-> Redirected to slot [16287] located at 192.168.4.53:6353   (自動分配到了資料庫53主庫)
OK
192.168.4.53:6353> keys *           
1) "x"
192.168.4.53:6353> set q 455
OK
192.168.4.53:6353> set w 400
-> Redirected to slot [3696] located at 192.168.4.51:6351
OK
192.168.4.51:6351> set q 88
-> Redirected to slot [11958] located at 192.168.4.53:6353
OK
192.168.4.53:6353> get w
-> Redirected to slot [3696] located at 192.168.4.51:6351
"400"

關於儲存的原理:
關於總槽數16384,無論有多少個主庫,這個值是不變的。
5461個槽並不代表可以儲存5461個變數。具體存多少個變數,根據主機的記憶體決定。
儲存時,做一個計算,CRC16演算法,算出hash槽的值,決定這個變數存在哪裡,決定去哪裡取值
將變數名和CRC16演算法做一個運算,得出來一個數和16384做取餘運算,得出來的餘數即為每次設定或者查詢變數時Redirected重定向的槽數,從而根據槽數決定重定向到哪個主庫。

redis高可用,當主庫宕機後,它對應從庫會自動升級為主庫

三 管理叢集
1 master選舉測試:
當52宕機後,52的從庫56會自動被選舉為主庫
當52修好啟動後,52會自動成為56的從庫,自動同步宕機期間在主庫52上產生的資料
在正常哪個的訪問過程中,為了實現真正的高可用,還需要寫一個指令碼,當主庫宕機後,使用者訪問主資料庫的ip地址自動切換為之前從庫的IP地址,不然訪問會出錯。

叢集不能用的情況:
有半數或者半數以上的主庫機器掛掉,叢集就不能用了
把一個從庫升級成主,沒有從庫,叢集不能用(前提是:有半數或者半數以上的主庫機器掛掉)
一個主庫掛掉,它的從庫自動頂替為主庫,正常使用(前提是:有半數或者半數以上的主庫機器能用),掛掉的主庫修復好後,會成為從庫,不會搶佔為主
當一組組從全掛掉之後,叢集不可用
可以設定一主兩從

2 新增新的節點:
準備兩個新的redis節點
1)將57新增進叢集為主庫

 [[email protected] ~]# redis-trib.rb add-node 192.168.4.57:6357 192.168.4.51:6351

2)為57分配hash槽(重新分片)手動對叢集進行分片遷移 選項:reshard 重新分配hash槽

[[email protected] ~]# redis-trib.rb reshard 192.168.4.51:6351
How many slots do you want to move (from 1 to 16384)? 4096       //指定移出hash槽的個數
What is the receiving node ID? 57488ae2bc84ce1b9cf1956c5e47e4d7c7372808 //指定接受hash槽的主機ID
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:all     //指定移出hash槽的主機ID
do you want to proceed with the proposed reshard plan(yes/no) yes //你確定要接受這個分片方式嗎?

3)檢視叢集狀態:再次檢視發現4.57有4096個hash slot

[[email protected] ~]# redis-trib.rb check 192.168.4.51:6351
>>> Performing Cluster Check (using node 192.168.4.51:6351)
M: 88e7b3ed90d604cb3170309b084c82defcb32e8e 192.168.4.51:6351
   slots:1365-5460 (4096 slots) master
   1 additional replica(s)
S: e0f9f77ba2bd2468baa1a2e9d88d494210d406bf 192.168.4.55:6355
   slots: (0 slots) slave
   replicates 88e7b3ed90d604cb3170309b084c82defcb32e8e
M: f71b5bc43cf4be71516f87ca56224d81a10795c8 192.168.4.54:6354
   slots:12288-16383 (4096 slots) master
   1 additional replica(s)

S: 8cb2e1a1c9df060d95ec5f7c0094bed8d69979b6 192.168.4.52:6352
   slots: (0 slots) slave
   replicates f169df56c739577d2d0cba2f883840a32631d511
S: aa1dd1ea0e5328541764f81541046bfd14acf20e 192.168.4.53:6353
   slots: (0 slots) slave
   replicates f71b5bc43cf4be71516f87ca56224d81a10795c8
M: 57488ae2bc84ce1b9cf1956c5e47e4d7c7372808 192.168.4.57:6357
   slots:0-1364,5461-6826,10923-12287 (4096 slots) master
   0 additional replica(s)
M: f169df56c739577d2d0cba2f883840a32631d511 192.168.4.56:6356
   slots:6827-10922 (4096 slots) master
   1 additional replica(s)

[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

[[email protected] ~]# redis-cli -c -h 192.168.4.57 -p 6357
192.168.4.57:6357> keys *
1) "uq"
2) "rl"
3) "yc"
4) "uu"
5) "uer"
6) "we"
7) "q"
資料來源於分過來的hash槽

4) 新增從庫並指定主庫:(預設情況下,不指定主庫ID時,誰的從庫最少,那麼就會成為它的從庫)

[[email protected] ~]# redis-trib.rb add-node --slave 192.168.4.58:6358 192.168.4.51:6351

另一種指定主庫ID的寫法:
[[email protected] ~]# redis-trib.rb add-node --slave --master-id 2e1584d4a4885288f26e314197f97d12e99576ce 192.168.4.58:6358 192.168.4.51:6351


[[email protected] ~]# redis-trib.rb check 192.168.4.51:6351
>>> Performing Cluster Check (using node 192.168.4.51:6351)
M: 88e7b3ed90d604cb3170309b084c82defcb32e8e 192.168.4.51:6351
   slots:1365-5460 (4096 slots) master
   1 additional replica(s)
S: e0f9f77ba2bd2468baa1a2e9d88d494210d406bf 192.168.4.55:6355
   slots: (0 slots) slave
   replicates 88e7b3ed90d604cb3170309b084c82defcb32e8e
M: f71b5bc43cf4be71516f87ca56224d81a10795c8 192.168.4.54:6354
   slots:12288-16383 (4096 slots) master
   1 additional replica(s)
S: 4726211191d55c54bfb72d9a56f7f0ed040a36a4 192.168.4.58:6358
   slots: (0 slots) slave
   replicates 57488ae2bc84ce1b9cf1956c5e47e4d7c7372808
S: 8cb2e1a1c9df060d95ec5f7c0094bed8d69979b6 192.168.4.52:6352
   slots: (0 slots) slave
   replicates f169df56c739577d2d0cba2f883840a32631d511
S: aa1dd1ea0e5328541764f81541046bfd14acf20e 192.168.4.53:6353
   slots: (0 slots) slave
   replicates f71b5bc43cf4be71516f87ca56224d81a10795c8
M: 57488ae2bc84ce1b9cf1956c5e47e4d7c7372808 192.168.4.57:6357
   slots:0-1364,5461-6826,10923-12287 (4096 slots) master
   1 additional replica(s)
M: f169df56c739577d2d0cba2f883840a32631d511 192.168.4.56:6356
   slots:6827-10922 (4096 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered

5)移除從節點:

[[email protected] ~]# redis-trib.rb del-node 192.168.4.58:6358 4726211191d55c54bfb72d9a56f7f0ed040a36a4
>>> Removing node 4726211191d55c54bfb72d9a56f7f0ed040a36a4 from cluster 192.168.4.58:6358
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

移除的同時會關閉被移除主機的redis服務
登入58主機的redis服務,能看到之前的資料,但是沒辦法獲取值

[[email protected] utils]# redis-cli
192.168.4.58:6358> keys *
1) "uer"
2) "q"
3) "we"
4) "uu"
5) "yc"
6) "rl"
7) "uq"
192.168.4.58:6358> get q;
(error) MOVED 15440 192.168.4.54:6354

192.168.4.58:6358> cluster reset 重置叢集資訊(所有叢集的資訊將在本資料庫清除)
進入配置檔案 /etc/redis/6379.conf 將配置叢集的三個選項註釋,
[[email protected] utils]# rm -rf /var/lib/redis/6379/nodes-6358.conf
再重啟redis服務即可

當完成以上步驟後,這個58主機的redis資料庫就時一臺單獨的資料庫伺服器,和叢集沒有關係了,也沒有資料了。

注意:當沒有進行cluster reset 重置時,先進入配置檔案註釋掉叢集資訊,再重啟服務後,叢集的資料會在資料庫內保留,而且可以正常獲取資料
192.168.4.58:6358> get yc
“09”
192.168.4.58:6358> get q
“88”
192.168.

6)移除主節點:
先移除57的hash槽

[[email protected] ~]# redis-trib.rb reshard 192.168.4.51:6351
How many slots do you want to move (from 1 to 16384)? 4096       //移出多少個hash槽
What is the receiving node ID?88e7b3ed90d604cb3170309b084c82defcb32e8e //把hash槽移給誰
Source node #1:57488ae2bc84ce1b9cf1956c5e47e4d7c7372808  //從哪裡取hash槽
Source node #2:done  // 結束,指定從57一臺主機上取hash
Do you want to proceed with the proposed reshard plan (yes/no)?yes  //是否接受這個方式

[[email protected] ~]# redis-trib.rb check 192.168.4.51:6351
M: 57488ae2bc84ce1b9cf1956c5e47e4d7c7372808 192.168.4.57:6357
   slots: (0 slots) master
   0 additional replica(s)

刪除叢集主機:

[[email protected] ~]# redis-trib.rb del-node 192.168.4.57:6357 57488ae2bc84ce1b9cf1956c5e47e4d7c7372808

主庫被移除後,資料庫內資料被清空,因為hash槽被釋放了
192.168.4.57:6357> cluster reset
進入配置檔案註釋掉叢集資訊,刪除叢集日誌檔案/var/lib/redis/6379/nodes-6357.conf
重啟redis服務即可。

##################################################

怎麼樣把拿出來的主機再次放入叢集
cluster reset
刪除叢集日誌檔案/var/lib/redis/6379/nodes-6357.conf
重啟redis服務
再往