Mongodb叢集架構之分片架構
本文介紹了熱門的NoSQL資料庫Mongodb的分片架構模式的相關概念以及分片環境搭建方法。分片就是在分散式資料架構中常見的Sharding這個詞。Mongodb分片的方式包括範圍分片、雜湊分片和標籤分片三種。
1.分片相關概念
Mongodb分片是什麼？
分片即sharding。在Mongodb中，一個集合可以根據特定的規則將其資料分成幾個不同的部分，然後每個組成部分分佈在不同的Mongodb伺服器上。在查詢時，自動從這些組成部分中查詢並給給出彙總結果。
分片跟副本集是不同的概念。
分片後，每個伺服器上的資料只是一個集合的一部分資料，必須將所有伺服器上的資料集中後才能得到完整的資料。
副本集方式部署時，每個伺服器上的資料都是完整的，不需要彙總即可得到一個集合的完整資料。
分片方式部署的節點結構。
在Mongodb分片方式部署時，需要三種類型的節點：
分片伺服器。用於實際存放集合的資料。直接查詢某個分片伺服器是不能得到一個集合的完整資料的。至少應該有2個分片伺服器。
配置伺服器。用於存放各種配置資料。至少應該有3個配置伺服器以副本集方式部署。
路由伺服器。用於定位分片並向外提供資料服務。直接查詢路由伺服器的資料可以得到一個集合的完整資料。至少應該有2個路由伺服器。
Mongodb分片的三種應用方式。
範圍分片。使用集合的某個鍵作為分片欄位，根據範圍分片。
雜湊分片。使用集合的某個鍵作為分片欄位，根據雜湊值分片。
標籤分片。使用集合的某個鍵作為分片欄位，根據標籤值分片。
2.分片環境搭建
為了操作方便，假定所有伺服器均在同一個機器上執行。
在搭建Mongodb分片方式的叢集時，按照以下步驟進行。
（1）建立如圖所示的目錄結構，每個數字目錄代表一個伺服器所使用的埠。
圖1
config目錄中的埠用於配置伺服器。route目錄中的埠用於路由伺服器。shard目錄中的埠用於分片伺服器。因此，這個mongodb叢集中總共會執行9個mongodb伺服器，每種型別的伺服器各有3個。
（2）編寫配置伺服器的啟動指令碼。
圖2
（3）編寫分片伺服器的啟動指令碼。
圖3
編寫路由伺服器的啟動指令碼。
圖4
（5）啟動配置伺服器節點。
首先應該啟動配置伺服器節點，再啟動分片節點，最後啟動路由節點。
[root@coe2coe data]# ./config/startconfig.sh 27117 27118 27119
starting mongodb configsvr @HOMEDIR:{/data/mongo/data/config/27117}
about to fork child process, waiting until server is ready for connections.
forked process: 6403
child process started successfully, parent exiting
starting mongodb configsvr @HOMEDIR:{/data/mongo/data/config/27118}
about to fork child process, waiting until server is ready for connections.
forked process: 6437
child process started successfully, parent exiting
starting mongodb configsvr @HOMEDIR:{/data/mongo/data/config/27119}
about to fork child process, waiting until server is ready for connections.
forked process: 6471
child process started successfully, parent exiting
3個配置伺服器需要配置成為副本集的方式，因此將它們組成一個副本集，以27117節點為主節點，另外兩個節點均為從節點(SECONDARY)。具體配置方式請參考之前的關於副本集的部落格。此處不再贅述。
（6）啟動分片伺服器節點。
[root@coe2coe data]# ./shard/startshard.sh 27017 27018 27019
starting mongodb shardsvr @HOMEDIR:{/data/mongo/data/shard/27017}
about to fork child process, waiting until server is ready for connections.
forked process: 6324
child process started successfully, parent exiting
starting mongodb shardsvr @HOMEDIR:{/data/mongo/data/shard/27018}
about to fork child process, waiting until server is ready for connections.
forked process: 6350
child process started successfully, parent exiting
starting mongodb shardsvr @HOMEDIR:{/data/mongo/data/shard/27019}
about to fork child process, waiting until server is ready for connections.
forked process: 6376
child process started successfully, parent exiting
（7）啟動路由伺服器節點。
[root@coe2coe data]# ./route/startroute.sh 11.1.1.11:27117,11.1.1.11:27118,11.1.1.11:27119 27217 27218 27219
CONFIG_SERVERS:{11.1.1.11:27117,11.1.1.11:27118,11.1.1.11:27119}
DBPORTS:{27217}
starting mongodb routing @HOMEDIR:{/data/mongo/data/route/27217}
about to fork child process, waiting until server is ready for connections.
forked process: 6643
child process started successfully, parent exiting
starting mongodb routing @HOMEDIR:{/data/mongo/data/route/27218}
about to fork child process, waiting until server is ready for connections.
forked process: 6674
child process started successfully, parent exiting
starting mongodb routing @HOMEDIR:{/data/mongo/data/route/27219}
about to fork child process, waiting until server is ready for connections.
forked process: 6705
child process started successfully, parent exiting
啟動路由伺服器節點後需要將三個分片伺服器節點加入到這個路由中。
[root@coe2coe data]# mongo --quiet --port=27217
mongos> sh.addShard("11.1.1.11:27017");
{
"shardAdded" : "shard0000",
"ok" : 1,
"operationTime" : Timestamp(1538403870, 4),
"$clusterTime" : {
"clusterTime" : Timestamp(1538403870, 4),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
mongos> sh.addShard("11.1.1.11:27018");
{
"shardAdded" : "shard0001",
"ok" : 1,
"operationTime" : Timestamp(1538403879, 3),
"$clusterTime" : {
"clusterTime" : Timestamp(1538403879, 4),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
mongos> sh.addShard("11.1.1.11:27019");
{
"shardAdded" : "shard0002",
"ok" : 1,
"operationTime" : Timestamp(1538403887, 2),
"$clusterTime" : {
"clusterTime" : Timestamp(1538403887, 2),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
此時可以檢視分片狀態：
mongos> sh.status();
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5bb22c83322f90f844d3defc")
}
shards:
{ "_id" : "shard0000", "host" : "11.1.1.11:27017", "state" : 1 }
{ "_id" : "shard0001", "host" : "11.1.1.11:27018", "state" : 1 }
{ "_id" : "shard0002", "host" : "11.1.1.11:27019", "state" : 1 }
active mongoses:
"4.0.2-76-g279a1f51b9" : 3
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "config", "primary" : "config", "partitioned" : true }
三個分片節點27017,27018,27019均處於正常狀態。至此一個正常的分片環境基本搭建完畢。
對資料庫啟用分片功能
在實際應用分片前，首先應該對含有需要分片的集合所在的資料庫啟用分片功能。在啟用分片功能之後可以對其中的集合進行分片的實際操作。
圖5
3.範圍分片
對t1集合進行範圍分片，分片欄位為shard。
圖6
4.雜湊分片
對t1表進行雜湊分片，分片欄位為shard。
圖7
向t2集合中插入一些資料：
mongos> db.t2.find();
{ "_id" : ObjectId("5bb2341df116ff2bb6014822"), "name" : "a", "shard" : 2 }
{ "_id" : ObjectId("5bb2341ff116ff2bb6014823"), "name" : "a", "shard" : 3 }
{ "_id" : ObjectId("5bb23444f116ff2bb6014824"), "name" : "a", "shard" : 4 }
{ "_id" : ObjectId("5bb2344af116ff2bb6014826"), "name" : "a", "shard" : 6 }
{ "_id" : ObjectId("5bb23419f116ff2bb6014821"), "name" : "a", "shard" : 1 }
{ "_id" : ObjectId("5bb23448f116ff2bb6014825"), "name" : "a", "shard" : 5 }
分別通過mongo客戶端直接向三個分片伺服器節點查詢，可以看到三個節點上各自有一部分資料。
[root@coe2coe route]# mongo --quiet --port=27017
> use test
switched to db test
> db.t2.find();
{ "_id" : ObjectId("5bb2344af116ff2bb6014826"), "name" : "a", "shard" : 6 }
[root@coe2coe route]# mongo --quiet --port=27018
> use test
switched to db test
> db.t2.find();
{ "_id" : ObjectId("5bb2341df116ff2bb6014822"), "name" : "a", "shard" : 2 }
{ "_id" : ObjectId("5bb2341ff116ff2bb6014823"), "name" : "a", "shard" : 3 }
{ "_id" : ObjectId("5bb23444f116ff2bb6014824"), "name" : "a", "shard" : 4 }
[root@coe2coe route]# mongo --quiet --port=27019
> use test
switched to db test
> db.t2.find();
{ "_id" : ObjectId("5bb23419f116ff2bb6014821"), "name" : "a", "shard" : 1 }
{ "_id" : ObjectId("5bb23448f116ff2bb6014825"), "name" : "a", "shard" : 5 }
至此雜湊分片功能已經實現。
5.標籤分片
對t3集合進行標籤分片，分片欄位為shard。
先新增幾個標籤。
mongos> use config
switched to db config
mongos> db.shards.find();
{ "_id" : "shard0000", "host" : "11.1.1.11:27017", "state" : 1 }
{ "_id" : "shard0001", "host" : "11.1.1.11:27018", "state" : 1 }
{ "_id" : "shard0002", "host" : "11.1.1.11:27019", "state" : 1 }
mongos> sh.addShardTag("shard0000","tag1");
{
"ok" : 1,
"operationTime" : Timestamp(1538405808, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1538405808, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
mongos> sh.addShardTag("shard0001","tag2");
{
"ok" : 1,
"operationTime" : Timestamp(1538405813, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1538405813, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
mongos> sh.addShardTag("shard0002","tag3");
{
"ok" : 1,
"operationTime" : Timestamp(1538405817, 2),
"$clusterTime" : {
"clusterTime" : Timestamp(1538405817, 2),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
為t3集合的shard欄位設定標籤。
shard值小於100的文件分配標籤tag1，100到1000的分配標籤tag2,1000以上的分配tag3。
mongos> use test
switched to db test
mongos> sh.addTagRange("test.t3", {shard:MinKey},{shard:100} ,"tag1");
{
"ok" : 1,
"operationTime" : Timestamp(1538405898, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1538405898, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
mongos> sh.addTagRange("test.t3", {shard:100},{shard:1000} ,"tag2");
{
"ok" : 1,
"operationTime" : Timestamp(1538405913, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1538405913, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
mongos> sh.addTagRange("test.t3", {shard:1000},{shard:MaxKey} ,"tag3");
{
"ok" : 1,
"operationTime" : Timestamp(1538405924, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1538405924, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
向t3集合中增加一些文件，並直接訪問三個分片節點予以驗證標籤分片結果。
t3集合中的文件列表：
mongos> db.t3.find();
{ "_id" : ObjectId("5bb2368cf116ff2bb6014827"), "name" : "a", "shard" : 1 }
{ "_id" : ObjectId("5bb23697f116ff2bb6014828"), "name" : "a", "shard" : 99 }
{ "_id" : ObjectId("5bb2369bf116ff2bb6014829"), "name" : "a", "shard" : 100 }
{ "_id" : ObjectId("5bb2369ef116ff2bb601482a"), "name" : "a", "shard" : 101 }
{ "_id" : ObjectId("5bb236a4f116ff2bb601482b"), "name" : "a", "shard" : 999 }
{ "_id" : ObjectId("5bb236a9f116ff2bb601482c"), "name" : "a", "shard" : 1000 }
{ "_id" : ObjectId("5bb236abf116ff2bb601482d"), "name" : "a", "shard" : 1001 }
{ "_id" : ObjectId("5bb236b3f116ff2bb601482e"), "name" : "a", "shard" : 9999 }
分別直接訪問三個分片節點：
[root@coe2coe route]# mongo --quiet --port=27017
> use test
switched to db test
> db.t3.find();
{ "_id" : ObjectId("5bb2368cf116ff2bb6014827"), "name" : "a", "shard" : 1 }
{ "_id" : ObjectId("5bb23697f116ff2bb6014828"), "name" : "a", "shard" : 99 }
> exit
[root@coe2coe route]# mongo --quiet --port=27018
> use test
switched to db test
> db.t3.find();
{ "_id" : ObjectId("5bb2369bf116ff2bb6014829"), "name" : "a", "shard" : 100 }
{ "_id" : ObjectId("5bb2369ef116ff2bb601482a"), "name" : "a", "shard" : 101 }
{ "_id" : ObjectId("5bb236a4f116ff2bb601482b"), "name" : "a", "shard" : 999 }
> exit
[root@coe2coe route]# mongo --quiet --port=27019
> use test
switched to db test
> db.t3.find();
{ "_id" : ObjectId("5bb236a9f116ff2bb601482c"), "name" : "a", "shard" : 1000 }
{ "_id" : ObjectId("5bb236abf116ff2bb601482d"), "name" : "a", "shard" : 1001 }
{ "_id" : ObjectId("5bb236b3f116ff2bb601482e"), "name" : "a", "shard" : 9999 }
> exit
結果表明，分片成功。
也可以直接通過config資料庫檢視分片情況和標籤情況。
mongos> use config;
switched to db config
mongos> db.shards.find();
{ "_id" : "shard0000", "host" : "11.1.1.11:27017", "state" : 1, "tags" : [ "tag1" ] }
{ "_id" : "shard0001", "host" : "11.1.1.11:27018", "state" : 1, "tags" : [ "tag2" ] }
{ "_id" : "shard0002", "host" : "11.1.1.11:27019", "state" : 1, "tags" : [ "tag3" ] }
mongos> db.tags.find();
{ "_id" : { "ns" : "test.t3", "min" : { "shard" : { "$minKey" : 1 } } }, "ns" : "test.t3", "min" : { "shard" : { "$minKey" : 1 } }, "max" : { "shard" : 100 }, "tag" : "tag1" }
{ "_id" : { "ns" : "test.t3", "min" : { "shard" : 100 } }, "ns" : "test.t3", "min" : { "shard" : 100 }, "max" : { "shard" : 1000 }, "tag" : "tag2" }
{ "_id" : { "ns" : "test.t3", "min" : { "shard" : 1000 } }, "ns" : "test.t3", "min" : { "shard" : 1000 }, "max" : { "shard" : { "$maxKey" : 1 } }, "tag" : "tag3" }
6.分片的優點
Mongodb的分片叢集架構具有以下的優點：
在分片叢集架構中，可以輕鬆實現多點寫入。
在分片架構中，可以有多個路由節點，因此，連線到任意一個路由節點均可以提供資料的讀寫功能。
在分片叢集架構中，可以提供寫資料的負載均衡。
在分片結構中，資料實際儲存在分片節點中，而一個叢集中可以有多個分片節點，而集合中的資料實際儲存到哪一個節點是有分片的鍵來決定的，因此可以通過分片鍵來調整資料的儲存位置，從而實現一定的寫均衡的功能。
7.分片的缺點
Mongodb的分片叢集架構具有以下的缺點：
在分片叢集架構中，存在分片節點的單點故障問題。
每一份資料僅僅儲存在某個特定的分片伺服器節點中,如果這個分片分片節點宕機，則這部分資料無法讀取。解決辦法稍後給出。
在分片叢集架構中，如果需要讀取完整的資料，只能通過路由節點讀取。而資料實際儲存在分片節點中，因此其中必然會多出一些節點間的網路資料傳輸的消耗。
8.分片叢集總控指令碼
由於叢集中節點個數和種類比較多，編寫了一個總控指令碼，用於簡化叢集的啟動和停止以及狀態檢視操作。
完整指令碼內容如下：
[root@coe2coe data]# cat cluster.sh
#!/bin/bash
##################################################################
# FileName ：startcluster.sh
# Author : [email protected]
# Created ：2018-10-02
# Description ：http://www.cnblogs.com/coe2coe/
#################################################################
start()
{
IP=$(ip addr |grep inet |grep brd |awk -F' ' '{ print $2}'|awk -F'/' '{print $1}')
if [ "$IP" == "" ]
then
echo -e "Failed to get IP on this host."
exit 1
fi
CONFIG_PORTS="27117 27118 27119"
SHARD_PORTS="27017 27018 27019"
ROUTE_PORTS="27217 27218 27219"
CONFIG_ADDRESSES="$IP:27117,$IP:27118,$IP:27119"
echo -e "Starting mongodb cluster at {$IP}....."
echo -e "Starting config nodes @{$CONFIG_PORTS} ..."
/data/mongo/data/config/startconfig.sh $CONFIG_PORTS
echo -e "Starting shard nodes @{$SHARD_PORTS}...."
/data/mongo/data/shard/startshard.sh $SHARD_PORTS
echo -e "Starting route nodes @{$ROUTE_PORTS} with CONFIG:{$CONFIG_ADDRESSES}...."
/data/mongo/data/route/startroute.sh $CONFIG_ADDRESSES $ROUTE_PORTS
echo -e "===ALL DONE====="
}
stop()
{
PIDS=$(pidof mongod mongos 2>/dev/null )
if [ "$PIDS" == "" ]
then
echo -e "NO such process found!"
exit 1
fi
echo -e "Stopping mongod and mongos:{$PIDS} ...."
kill -9 ${PIDS}
exit 0
}
status()
{
C_PIDS=$(ps -elf |grep mongod |grep configsvr |grep -v grep |awk '{print $4}' |xargs )
D_PIDS=$(ps -elf |grep mongod |grep shardsvr |grep -v grep |awk '{print $4}' |xargs )
R_PIDS=$(ps -elf |grep mongos |grep -v grep |awk '{print $4}' |xargs )
if [ "$C_PIDS" == "" ]
then
C_STATUS="NOT running"
else
C_STATUS="Running"
fi
if [ "$D_PIDS" == "" ]
then
D_STATUS="NOT running"
else
D_STATUS="Running"
fi
if [ "$R_PIDS" == "" ]
then
R_STATUS="NOT running"
else
R_STATUS="Running"
fi
echo -e "config nodes:{$C_PIDS}:{${C_STATUS}}"
echo -e "shard nodes :{$D_PIDS}:{${D_STATUS}}"
echo -e "route nodes :{$R_PIDS}:{${R_STATUS}}"
exit 0
}
usage()
{
echo -e "Usage: $0 [start|stop|status]"
exit 1
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status
;;
*)
usage
;;
esac
指令碼功能如下：
啟動叢集：
[root@coe2coe data]# ./cluster.sh start
Starting mongodb cluster at {11.1.1.11}.....
Starting config nodes @{27117 27118 27119} ...
starting mongodb configsvr @HOMEDIR:{/data/mongo/data/config/27117}
about to fork child process, waiting until server is ready for connections.
forked process: 5569
child process started successfully, parent exiting
starting mongodb configsvr @HOMEDIR:{/data/mongo/data/config/27118}
about to fork child process, waiting until server is ready for connections.
forked process: 5652
child process started successfully, parent exiting
starting mongodb configsvr @HOMEDIR:{/data/mongo/data/config/27119}
about to fork child process, waiting until server is ready for connections.
forked process: 5737
child process started successfully, parent exiting
Starting shard nodes @{27017 27018 27019}....
starting mongodb shardsvr @HOMEDIR:{/data/mongo/data/shard/27017}
about to fork child process, waiting until server is ready for connections.
forked process: 5826
child process started successfully, parent exiting
starting mongodb shardsvr @HOMEDIR:{/data/mongo/data/shard/27018}
about to fork child process, waiting until server is ready for connections.
forked process: 5888
child process started successfully, parent exiting
starting mongodb shardsvr @HOMEDIR:{/data/mongo/data/shard/27019}
about to fork child process, waiting until server is ready for connections.
forked process: 5934
child process started successfully, parent exiting
Starting route nodes @{27217 27218 27219} with CONFIG:{11.1.1.11:27117,11.1.1.11:27118,11.1.1.11:27119}....
CONFIG_SERVERS:{11.1.1.11:27117,11.1.1.11:27118,11.1.1.11:27119}
DBPORTS:{27217}
starting mongodb routing @HOMEDIR:{/data/mongo/data/route/27217}
about to fork child process, waiting until server is ready for connections.
forked process: 5982
child process started successfully, parent exiting
starting mongodb routing @HOMEDIR:{/data/mongo/data/route/27218}
about to fork child process, waiting until server is ready for connections.
forked process: 6015
child process started successfully, parent exiting
starting mongodb routing @HOMEDIR:{/data/mongo/data/route/27219}
about to fork child process, waiting until server is ready for connections.
forked process: 6044
child process started successfully, parent exiting
===ALL DONE=====
檢視叢集狀態：
[root@coe2coe data]# ./cluster.sh status
config nodes:{5569 5652 5737}:{Running}
shard nodes :{5826 5888 5934}:{Running}
route nodes :{5982 6015 6044}:{Running}
停止叢集：
[root@coe2coe data]# ./cluster.sh stop
Stopping mongod and mongos:{5934 5888 5826 5737 5652 5569 6044 6015 5982} ....
停止集群后檢視狀態：
[root@coe2coe data]# ./cluster.sh status
config nodes:{}:{NOT running}
shard nodes :{}:{NOT running}
route nodes :{}:{NOT running}