久前, 玩過redis叢集服務的搭建, 那種哨兵模式有點類似資料庫的讀寫分離, 但是業務寫入資料還是全部由一個邏輯上的master承擔的, 具體方法可見 [哨兵]; 今天來依據Redis的官方說明部署把玩下分散式叢集, 這種模式可以配置多個master均分16384個hash slot, 使得不同hash key分落到不同的master節點上, 真正意義的分散式;

本文的部署環境是 ubuntu16.04, redis-3.2.8

部署方案及環境

本示例中採用6節點, 三個master和三個slaves兩兩成組的方式共同形成一個分散式叢集, 對外提供儲存服務;

獲取redis

下載 redis, 得到應該是原始碼, 先進行解壓, 然後make即可;

redis 配置介紹

# redis 提供服務時繫結的本機網路地址，可以是多個滿足多塊網絡卡分別對外網內網的訪問, 按需要改
bind 127.0.0.1
# 提供服務的埠, 供客戶端連線
port 7000
# 啟動redis服務時,程序pid儲存位置
pidfile /disk/develop/redis/cluster/7000/redis.pid
# redis 日誌檔案所在
logfile "/disk/develop/redis/cluster/7000/log.txt"
# 持久化檔案儲存目錄
dir /disk/develop/redis/cluster/7000
 

# 開啟叢集模式
cluster-enabled yes
# 叢集模式下的節點配置資訊
cluster-config-file nodes.conf
# 叢集中各節點間連線超時時間
cluster-node-timeout 5000
# 允許資料持久化追加
appendonly yes

建立叢集目錄

比如上面我把下載的redis原始碼解壓到了 ‘/disk/develop/redis/cluster/redis-3.2.8’, 同時建立其他六個節點目錄

cd /disk/develop/redis/cluster/
mkdir 7000 7001 7002 7003 7004 7005 

開始配置

copy 原始的配置檔案

cp redis-3.2.8/redis.conf 7000/
cp redis-3.2.8/redis.conf 7001/
cp redis-3.2.8/redis.conf 7002/
cp redis-3.2.8/redis.conf 7003/
cp redis-3.2.8/redis.conf 7004/
cp redis-3.2.8/redis.conf 7005/

修改配置檔案, 分別進入到上面的六個新建的資料夾中, 按照上面的配置介紹中的樣例進行對應修改

啟動服務

cd /disk/develop/redis/cluster/redis-3.2.8
src/redis-server ../7000/redis.conf
src/redis-server ../7001/redis.conf
src/redis-server ../7002/redis.conf
src/redis-server ../7003/redis.conf
src/redis-server ../7004/redis.conf
src/redis-server ../7005/redis.conf

檢視下服務呢

ps -ef|grep redis-server
hinsteny 21971  2242  0 22:48 ?        00:00:02 src/redis-server 127.0.0.1:7000 [cluster]
hinsteny 22062  2242  0 22:50 ?        00:00:02 src/redis-server 127.0.0.1:7001 [cluster]
hinsteny 22149  2242  0 22:53 ?        00:00:02 src/redis-server 127.0.0.1:7002 [cluster]
hinsteny 22226  2242  0 22:55 ?        00:00:02 src/redis-server 127.0.0.1:7003 [cluster]
hinsteny 22257  2242  0 22:57 ?        00:00:02 src/redis-server 127.0.0.1:7004 [cluster]
hinsteny 22280  2242  0 22:58 ?        00:00:02 src/redis-server 127.0.0.1:7005 [cluster]
hinsteny 23831 11259  0 23:35 pts/10   00:00:00 grep --color=auto redis-server

雖然六個服務都起來了, 但這時他們之間還不知道彼此的存在, 無法組成叢集提供服務

src/redis-cli -c -p 7000
127.0.0.1:7000> cluster info
cluster_state:fail   //叢集狀態失敗
cluster_slots_assigned:0 // hash 槽點分佈也是失敗的
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0
127.0.0.1:7000> 

接下來讓我們用一個工具 ‘redis-trib.rb’ 將上面的六個節點連線起來

插播, redis-trib是需要ruby環境並在其環境下安裝對應的redis外掛

# 先安裝ruby環境
apt-get install ruby
# 再安裝redis需要的外掛
gem install redis

關聯組建叢集

src/redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7000
127.0.0.1:7001
127.0.0.1:7002
Adding replica 127.0.0.1:7003 to 127.0.0.1:7000
Adding replica 127.0.0.1:7004 to 127.0.0.1:7001
Adding replica 127.0.0.1:7005 to 127.0.0.1:7002
M: dc5da8bfce379f9cc2c11966709d7d9f275d06b1 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
M: 2b22012f983698f504b9e61d0958fc645112c805 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
M: df396e45f08019525bf668ff4704d626014db2d6 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
S: 90740536f6e5b191300253b84fb4d9b908d48d54 127.0.0.1:7003
   replicates dc5da8bfce379f9cc2c11966709d7d9f275d06b1
S: 52ba80c45ffd165b9cec01c8409c605386d1a8b2 127.0.0.1:7004
   replicates 2b22012f983698f504b9e61d0958fc645112c805
S: 888848820f300331a76803755dc1c15302f2ce98 127.0.0.1:7005
   replicates df396e45f08019525bf668ff4704d626014db2d6
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: dc5da8bfce379f9cc2c11966709d7d9f275d06b1 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
M: 2b22012f983698f504b9e61d0958fc645112c805 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 90740536f6e5b191300253b84fb4d9b908d48d54 127.0.0.1:7003
   slots: (0 slots) slave
   replicates dc5da8bfce379f9cc2c11966709d7d9f275d06b1
S: 52ba80c45ffd165b9cec01c8409c605386d1a8b2 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 2b22012f983698f504b9e61d0958fc645112c805
M: df396e45f08019525bf668ff4704d626014db2d6 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: 888848820f300331a76803755dc1c15302f2ce98 127.0.0.1:7005
   slots: (0 slots) slave
   replicates df396e45f08019525bf668ff4704d626014db2d6
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

驗證叢集配置

上面顯示已經成功組成叢集了, 節點[7000,7001,7002]為三個master, 剩下的[7003,7004,7005]分別為前三者的複製節點, 並且可以看到16384個槽點被全部覆蓋分佈, 現在再檢視下叢集狀態

src/redis-cli -c -p 7000127.0.0.1:7000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_sent:487
cluster_stats_messages_received:487

最後通過set與get操作驗證下

src/redis-cli -c -p 7000
127.0.0.1:7000> set hisoka welcome
-> Redirected to slot [9822] located at 127.0.0.1:7001
OK
127.0.0.1:7001> get hisoka
"welcome"
127.0.0.1:7001> set hinsteny 'love sing'
OK
127.0.0.1:7001> get hinsteny
"love sing"
127.0.0.1:7001> set k3 k3_val
-> Redirected to slot [4576] located at 127.0.0.1:7000
OK
127.0.0.1:7000> set k5 k5_val
-> Redirected to slot [12582] located at 127.0.0.1:7002
OK
127.0.0.1:7002> 

That’s all, Thanks!