Linux Redis集群搭建與集群客戶端實現
阿新 • • 發佈：2019-04-01
分享 2.0 redis安裝 strong trac gif oot complete install 
硬件環境
本文適用的硬件環境如下
Linux版本：CentOS release 6.7 (Final) Redis版本：3.2.1
Redis已經成功安裝，安裝路徑為/home/idata/yangfan/local/redis-3.2.1。
我們要在單臺機器上搭建Redis集群，方式是通過不同的TCP端口啟動多個實例，然後組成集群。
1、啟動Redis多個實例
我們在Redis安裝目錄下創建目錄cluster，並編寫7000.conf~7005.conf 6個配置文件，這6個配置文件用來啟動6個實例，後面將使用這6個實例組成集群。
以7000.conf為例，配置文件需要填寫如下幾項。
port 7000 //端口7000,7002,7003 bind 10.93.84.53 //默認ip為127.0.0.1 需要改為其他節點機器可訪問的ip 否則創建集群時無法訪問對應的端口，無法創建集群 daemonize yes //redis後臺運行 pidfile ./redis_7000.pid //pidfile文件對應7000,7001,7002 cluster-enabled yes //開啟集群 把註釋#去掉 cluster-config-file nodes_7000.conf //集群的配置 配置文件首次啟動自動生成 7000,7001,7002 cluster-node-timeout 15000 //請求超時 默認15秒，可自行設置 appendonly yes //aof日誌開啟 有需要就開啟，它會每次寫操作都記錄一條日誌
分別啟動6個實例
./bin/redis-server cluster/conf/7000.conf ./bin/redis-server cluster/conf/7001.conf ./bin/redis-server cluster/conf/7002.conf ./bin/redis-server cluster/conf/7003.conf ./bin/redis-server cluster/conf/7004.conf ./bin/redis-server cluster/conf/7005.conf
啟動成功後，看一下進程
# ps -ef | grep redis | grep cluster idata 15711 22329 0 18:40 pts/10 00:00:00 ./bin/redis-server 10.93.84.53:7000 [cluster] idata 15740 22329 0 18:40 pts/10 00:00:00 ./bin/redis-server 10.93.84.53:7001 [cluster] idata 15810 22329 0 18:40 pts/10 00:00:00 ./bin/redis-server 10.93.84.53:7002 [cluster] idata 17023 22329 0 18:42 pts/10 00:00:00 ./bin/redis-server 10.93.84.53:7003 [cluster] idata 17030 22329 0 18:42 pts/10 00:00:00 ./bin/redis-server 10.93.84.53:7004 [cluster] idata 17035 22329 0 18:42 pts/10 00:00:00 ./bin/redis-server 10.93.84.53:7005 [cluster]
至此，ip=10.93.84.53機器上創建了6個實例，端口號為port=7000~7005。
Redis 3.0以上的集群方式是通過Redis安裝目錄下的bin/redis-trib.rb腳本搭建。
這個腳本是用Ruby編寫的，嘗試運行，如果打印如下，你可以跳過本文的第二部分。
[email protected]:~/yangfan/local/redis-3.2.1/bin$ ruby redis-trib.rb Usage: redis-trib <command> <options> <arguments ...> create host1:port1 ... hostN:portN --replicas <arg> check host:port info host:port fix host:port --timeout <arg> reshard host:port --from <arg> --to <arg> --slots <arg> --yes --timeout <arg> --pipeline <arg> rebalance host:port --weight <arg> --auto-weights --use-empty-masters --timeout <arg> --simulate --pipeline <arg> --threshold <arg> add-node new_host:new_port existing_host:existing_port --slave --master-id <arg> del-node host:port node_id set-timeout host:port milliseconds call host:port command arg arg .. arg import host:port --from <arg> --copy --replace help (show this help) For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.
如果執行失敗，那麽不幸的是你的機器沒有Ruby運行的環境，那麽你需要安裝Ruby。進入第二部分。
2、安裝ruby
下面的過程都是在root權限下完成的。
1）yum安裝ruby和依賴的包。
# yum -y install ruby ruby-devel rubygems rpm-build
一般來說，這一步是能正常完成的。
2）使用gem這個命令來安裝redis接口
gem是ruby的一個工具包
# gem install redis
安裝過程出錯
郁悶，看樣子要升級ruby版本。
3）升級Ruby的版本
安裝rvm，我不知道這是個什麽東西，但是感覺像是Ruby的一個包管理器。
# curl -L get.rvm.io | bash -s stable
WTF，又出問題了
氣急敗壞的照著他說的做
# gpg2 --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 gpg: keyring `/root/.gnupg/secring.gpg‘ created gpg: requesting key D39DC0E3 from hkp server keys.gnupg.net gpg: /root/.gnupg/trustdb.gpg: trustdb created gpg: key D39DC0E3: public key "Michal Papis (RVM signing) <[email protected]>" imported gpg: no ultimately trusted keys found gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1)
然後重新下載rvm安裝，成功了。
# curl -L get.rvm.io | bash -s stable
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 24090 100 24090 0 0 18023 0 0:00:01 0:00:01 --:--:-- 129k
Downloading https://github.com/rvm/rvm/archive/1.29.3.tar.gz
Downloading https://github.com/rvm/rvm/releases/download/1.29.3/1.29.3.tar.gz.asc
gpg: Signature made Mon 11 Sep 2017 04:59:21 AM CST using RSA key ID BF04FF17
gpg: Good signature from "Michal Papis (RVM signing) <[email protected]>"
gpg: aka "Michal Papis <[email protected]>"
gpg: aka "[jpeg image of size 5015]"
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 409B 6B17 96C2 7546 2A17 0311 3804 BB82 D39D C0E3
Subkey fingerprint: 62C9 E5F4 DA30 0D94 AC36 166B E206 C29F BF04 FF17
GPG verified ‘/usr/local/rvm/archives/rvm-1.29.3.tgz‘
Creating group ‘rvm‘
Installing RVM to /usr/local/rvm/
Installation of RVM in /usr/local/rvm/ is almost complete:
* First you need to add all users that will be using rvm to ‘rvm‘ group,
and logout - login again, anyone using rvm will be operating with `umask u=rwx,g=rwx,o=rx`.
* To start using RVM you need to run `source /etc/profile.d/rvm.sh`
in all your open shell windows, in rare cases you need to reopen all shell windows.
接著，source環境，讓rvm可用。
# source /usr/local/rvm/scripts/rvm
查看Ruby可用版本
# rvm list known # MRI Rubies [ruby-]1.8.6[-p420] [ruby-]1.8.7[-head] # security released on head [ruby-]1.9.1[-p431] [ruby-]1.9.2[-p330] [ruby-]1.9.3[-p551] [ruby-]2.0.0[-p648] [ruby-]2.1[.10] [ruby-]2.2[.7] [ruby-]2.3[.4] [ruby-]2.4[.1]
可以看到最新的版本是2.4.1，那麽我們裝最新的吧。
# rvm install 2.4.1
Searching for binary rubies, this might take some time.
No binary rubies available for: centos/6/x86_64/ruby-2.4.1.
Continuing with compilation. Please read ‘rvm help mount‘ to get more information on binary rubies.
Checking requirements for centos.
Installing requirements for centos.
Installing required packages: libffi-devel, libyaml-devel......
Requirements installation successful.
Installing Ruby from source to: /usr/local/rvm/rubies/ruby-2.4.1, this may take a while depending on your cpu(s)...
ruby-2.4.1 - #downloading ruby-2.4.1, this may take a while depending on your connection...
curl: (35) SSL connect error
There was an error(35).
Checking fallback: https://ftp.ruby-lang.org/pub/ruby/2.4/ruby-2.4.1.tar.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11.9M 100 11.9M 0 0 1753k 0 0:00:07 0:00:07 --:--:-- 2919k
ruby-2.4.1 - #extracting ruby-2.4.1 to /usr/local/rvm/src/ruby-2.4.1....
ruby-2.4.1 - #applying patch /usr/local/rvm/patches/ruby/2.4.1/random_c_using_NR_prefix.patch.
ruby-2.4.1 - #configuring..................................................................
ruby-2.4.1 - #post-configuration..
ruby-2.4.1 - #compiling..............................................................................................
ruby-2.4.1 - #installing.........................
ruby-2.4.1 - #making binaries executable..
ruby-2.4.1 - #downloading rubygems-2.6.14
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 751k 100 751k 0 0 443k 0 0:00:01 0:00:01 --:--:-- 628k
No checksum for downloaded archive, recording checksum in user configuration.
ruby-2.4.1 - #extracting rubygems-2.6.14....
ruby-2.4.1 - #removing old rubygems.........
ruby-2.4.1 - #installing rubygems-2.6.14...........................
ruby-2.4.1 - #gemset created /usr/local/rvm/gems/ruby-2.4.1@global
ruby-2.4.1 - #importing gemset /usr/local/rvm/gemsets/global.gems...............................................
ruby-2.4.1 - #generating global wrappers........
ruby-2.4.1 - #gemset created /usr/local/rvm/gems/ruby-2.4.1
ruby-2.4.1 - #importing gemsetfile /usr/local/rvm/gemsets/default.gems evaluated to empty gem list
ruby-2.4.1 - #generating default wrappers........
ruby-2.4.1 - #adjusting #shebangs for (gem irb erb ri rdoc testrb rake).
Install of ruby-2.4.1 - #complete
Ruby was built without documentation, to build it run: rvm docs generate-ri
至此，我們升級了Ruby的版本。
4）安裝gem redis接口，成功！
gem install redis Fetching: redis-4.0.1.gem (100%) Successfully installed redis-4.0.1 Parsing documentation for redis-4.0.1 Installing ri documentation for redis-4.0.1 Done installing documentation for redis after 0 seconds 1 gem installed
5）安裝rubygems，成功！
# yum install -y rubygems Loaded plugins: fastestmirror, security Setting up Install Process Loading mirror speeds from cached hostfile base | 3.7 kB 00:00 didi_jenkins_enable | 1.5 kB 00:00 didi_op_toa_enable | 1.5 kB 00:00 didi_txjenkins_enable | 1.5 kB 00:00 didi_update | 1.5 kB 00:00 epel | 4.3 kB 00:00 extras | 3.4 kB 00:00 tmprepo | 1.5 kB 00:00 updates | 3.4 kB 00:00 Package rubygems-1.3.7-5.el6.noarch already installed and latest version Nothing to do
至此，我們的Ruby和運行redis-trib.rb需要的環境安裝完成了。
3、Redis集群搭建
有了Ruby執行環境，可以開始將之前的6個實例組建成集群了。
命令方式：
ruby ./bin/redis-trib.rb create --replicas 1 10.93.84.53:7000 10.93.84.53:7001 10.93.84.53:7002 10.93.84.53:7003 10.93.84.53:7004 10.93.84.53:7005
--replicas 1表示為集群的master節點創建1個副本。那麽6個實例裏，有三個master，有三個是slave。
後面跟上6個實例就好了，形式就是ip:port
執行情況：
# ruby ./bin/redis-trib.rb create --replicas 1 10.93.84.53:7000 10.93.84.53:7001 10.93.84.53:7002 10.93.84.53:7003 10.93.84.53:7004 10.93.84.53:7005 >>> Creating cluster >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 10.93.84.53:7000 10.93.84.53:7001 10.93.84.53:7002 Adding replica 10.93.84.53:7003 to 10.93.84.53:7000 Adding replica 10.93.84.53:7004 to 10.93.84.53:7001 Adding replica 10.93.84.53:7005 to 10.93.84.53:7002 M: 6346ae8c7af7949658619fcf4021cc7aca454819 10.93.84.53:7000 slots:0-5460 (5461 slots) master M: 5ac973bceab0d486c497345fe884ff54d1bb225a 10.93.84.53:7001 slots:5461-10922 (5462 slots) master M: cc46a4a1c0ec3f621b6b5405c6c10b7cffe73932 10.93.84.53:7002 slots:10923-16383 (5461 slots) master S: 92f62ec93a0550d962f81213ca7e9b3c9c996afd 10.93.84.53:7003 replicates 6346ae8c7af7949658619fcf4021cc7aca454819 S: 942c9f97dc68198c39f425d13df0d8e3c40c5a58 10.93.84.53:7004 replicates 5ac973bceab0d486c497345fe884ff54d1bb225a S: a92a81532b63652bbd862be6f19a9bd8832e5e05 10.93.84.53:7005 replicates cc46a4a1c0ec3f621b6b5405c6c10b7cffe73932 Can I set the above configuration? (type ‘yes‘ to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join... >>> Performing Cluster Check (using node 10.93.84.53:7000) M: 6346ae8c7af7949658619fcf4021cc7aca454819 10.93.84.53:7000 slots:0-5460 (5461 slots) master 1 additional replica(s) S: a92a81532b63652bbd862be6f19a9bd8832e5e05 10.93.84.53:7005 slots: (0 slots) slave replicates cc46a4a1c0ec3f621b6b5405c6c10b7cffe73932 M: 5ac973bceab0d486c497345fe884ff54d1bb225a 10.93.84.53:7001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: 942c9f97dc68198c39f425d13df0d8e3c40c5a58 10.93.84.53:7004 slots: (0 slots) slave replicates 5ac973bceab0d486c497345fe884ff54d1bb225a S: 92f62ec93a0550d962f81213ca7e9b3c9c996afd 10.93.84.53:7003 slots: (0 slots) slave replicates 6346ae8c7af7949658619fcf4021cc7aca454819 M: cc46a4a1c0ec3f621b6b5405c6c10b7cffe73932 10.93.84.53:7002 slots:10923-16383 (5461 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
可以看到16384個slot都已經創建完成，並且建立了3個master和對應的replica:
Using 3 masters: 10.93.84.53:7000 10.93.84.53:7001 10.93.84.53:7002 Adding replica 10.93.84.53:7003 to 10.93.84.53:7000 Adding replica 10.93.84.53:7004 to 10.93.84.53:7001 Adding replica 10.93.84.53:7005 to 10.93.84.53:7002
。。。
[OK] All 16384 slots covered.
4、驗證集群狀態
登錄集群客戶端，-c標識以集群方式登錄
./bin/redis-cli -h 10.93.84.53 -p 7000 -c
查看集群狀態
10.93.84.53:7000> cluster info cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:3 cluster_current_epoch:8 cluster_my_epoch:8 cluster_stats_messages_sent:215 cluster_stats_messages_received:215 10.93.84.53:7000> cluster nodes 942c9f97dc68198c39f425d13df0d8e3c40c5a58 10.93.84.53:7004 slave 5ac973bceab0d486c497345fe884ff54d1bb225a 0 1507806791940 5 connected 5ac973bceab0d486c497345fe884ff54d1bb225a 10.93.84.53:7001 master - 0 1507806788937 2 connected 5461-10922 a92a81532b63652bbd862be6f19a9bd8832e5e05 10.93.84.53:7005 slave cc46a4a1c0ec3f621b6b5405c6c10b7cffe73932 0 1507806790939 6 connected cc46a4a1c0ec3f621b6b5405c6c10b7cffe73932 10.93.84.53:7002 master - 0 1507806789937 3 connected 10923-16383 6346ae8c7af7949658619fcf4021cc7aca454819 10.93.84.53:7000 myself,slave 92f62ec93a0550d962f81213ca7e9b3c9c996afd 0 0 1 connected 92f62ec93a0550d962f81213ca7e9b3c9c996afd 10.93.84.53:7003 master - 0 1507806792941 8 connected 0-5460
一些原理
redis cluster在設計的時候，就考慮到了去中心化，去中間件，也就是說，集群中的每個節點都是平等的關系，都是對等的，每個節點都保存各自的數據和整個集群的狀態。每個節點都和其他所有節點連接，而且這些連接保持活躍，這樣就保證了我們只需要連接集群中的任意一個節點，就可以獲取到其他節點的數據。
Redis集群沒有並使用傳統的一致性哈希來分配數據，而是采用另外一種叫做哈希槽（hash slot）的方式來分配的，一致性哈希對向集群中新增和刪除實例的支持很好，但是哈希槽對向集群新增實例或者刪除實例的話，需要額外的操作，需要手動的將slot重新平均的分配到新集群的實例中。
redis cluster 默認分配了 16384 個slot，當我們set一個key時，會用CRC16算法來取模得到所屬的slot，然後將這個key分到哈希槽區間的節點上，具體算法就是：CRC16(key)%16384。
Redis 集群會把數據存在一個master節點，然後在這個master和其對應的salve之間進行數據同步。當讀取數據時，也根據一致性哈希算法到對應的master節點獲取數據。只有當一個master 掛掉之後，才會啟動一個對應的salve節點，充當master。
需要註意的是：必須要3個或以上的主節點，否則在創建集群時會失敗，並且當存活的主節點數小於總節點數的一半時，整個集群就無法提供服務了。
5、python集群客戶端
以Python Redis Cluster集群的使用方式為例，簡單說明一下如何使用，讓大家更直觀的了解一下Redis集群。
Redis集群方式與單機方式在python客戶端實現上是有很大不同的。
- 包不同，單機依賴包redis-py，集群依賴包redis-py-cluster
- 對同時操作多個keys的命令(mset, mget, sinter, ...)，redis-py-cluster重寫StriceRedis(單機Redis操作類)了其方法，而這些方法喪失了命令的原子性。
- Pipelines在集群中的表現也不同了。在單機版中，pipeline是批量batch提交redis批量執行的，但是在集群版中，是one by one提交redis執行完成直接返回，在客戶端重新組合成一個列表返回。所以集群中只是看起來像是批量執行。
- 一些命令是會Fanout(扇形)發送到集群中各個shard中執行並匯總的，如ping，keys等。一些命令是不提倡使用(blocked)，如watch，unwatch等。
我只是簡單翻譯了官網的一些內容，相信信息大家可以參考：
http://redis-py-cluster.readthedocs.io/en/master/commands.html
http://redis-py-cluster.readthedocs.io/en/master/limitations-and-differences.html
1）安裝redis-py-cluster
簡單的通過pip安裝redis-py-cluster包。如果安裝失敗，可以自助下載安裝。
# pip install redis-py-cluster Collecting redis-py-cluster Downloading redis_py_cluster-1.3.4-py2.py3-none-any.whl Requirement already satisfied: redis>=2.10.2 in /home/idata/pythonEnv/idataPlatEnv/lib/python2.7/site-packages/redis-2.10.5-py2.7.egg (from redis-py-cluster) Installing collected packages: redis-py-cluster Successfully installed redis-py-cluster-1.3.4
2）一個簡單的demo
封裝了RedisCluster操作類，實現了一些方法，其實就是做了一層封裝。
封裝的意義是：我喜歡對這些封裝增加一些裝飾器，控制異常和重試等邏輯。
# -*- coding:utf-8 -*-
from rediscluster import StrictRedisCluster
redis_nodes = [
{‘host‘: ‘10.93.84.53‘, ‘port‘: 7000},
{‘host‘: ‘10.93.84.53‘, ‘port‘: 7001},
{‘host‘: ‘10.93.84.53‘, ‘port‘: 7002},
{‘host‘: ‘10.93.84.53‘, ‘port‘: 7003},
{‘host‘: ‘10.93.84.53‘, ‘port‘: 7004},
{‘host‘: ‘10.93.84.53‘, ‘port‘: 7005},
]
class RedisCluster(object):
def __init__(self, redis_nodes):
self.cluster = StrictRedisCluster(startup_nodes=redis_nodes)
# 無差別的方法
def set(self, name, value, ex=None, px=None, nx=False, xx=False):
return self.cluster.set(name, value, ex, px, nx, xx)
# 無差別的方法
def get(self, name):
return self.cluster.get(name)
# 扇形發送的命令
def cluster_info(self):
return self.cluster.cluster_info()
# 重寫StrictRedis的方法
def mset(self, *args, **kwargs):
return self.cluster.mset(args, kwargs)
# 重寫StrictRedis的方法
def mget(self, keys, *args):
return self.cluster.mget(keys, args)
cluster = RedisCluster(redis_nodes)
cluster.cluster_info()
Linux Redis集群搭建與集群客戶端實現