1. 程式人生 > >Mongodb副本集分片叢集模式環境部署

Mongodb副本集分片叢集模式環境部署

分片(sharding)是指將資料庫拆分,將其分散在不同的機器上的過程。將資料分散到不同的機器上,不需要功能強大的伺服器就可以儲存更多的資料和處理更大的負載。基本思想就是將集合切成小塊,這些塊分散到若干片裡,每個片只負責總資料的一部分,最後通過一個均衡器來對各個分片進行均衡(資料遷移)。通過一個名為mongos的路由程序進行操作,mongos知道資料和片的對應關係(通過配置伺服器)。大部分使用場景都是解決磁碟空間的問題,對於寫入有可能會變差(+++裡面的說明+++),查詢則儘量避免跨分片查詢。使用分片的時機:

1,機器的磁碟不夠用了。使用分片解決磁碟空間的問題。
2,單個mongod已經不能滿足寫資料的效能要求。通過分片讓寫壓力分散到各個分片上面,使用分片伺服器自身的資源。
3,想把大量資料放到記憶體裡提高效能。和上面一樣,通過分片使用分片伺服器自身的資源。

MongoDB sharding Cluster(分片叢集),需要三種角色

Shard Server: mongod 例項,用於儲存實際的資料塊,實際生產環境中一個shard server角色可由幾臺機器組個一個relica set承擔,防止主機單點故障; Config Server: mongod 例項,儲存了整個 Cluster Metadata,其中包括 chunk 資訊。 Route Server: mongos 例項,前端路由,客戶端由此接入,且讓整個叢集看上去像單一資料庫,前端應用可以透明使用。 叢集搭建伺服器規劃(共3個分片叢集,每個叢集3個副本集,另外一個3結點配置副本集,一個3結點路由副本集):

 

 

1、安裝Mongodb(3臺機器都要操作,本次用的4.0.3版本)
下載地址:https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-4.0.3.tgz

解壓
[[email protected] ~]$ tar -zxvf mongodb-linux-x86_64-4.0.3.tgz 
修改解壓目錄名
[[email protected] ~]$ mv mongodb-linux-x86_64-4.0.3 mongodb

2、建立相應叢集所需目錄(3臺機器根據上述伺服器規劃表,分別操作)

[[email protected] ~]$ mkdir services
[[email protected] ~]$ mv mongodb services/
[[email protected] ~]$ mkdir -p services/shard1 services/shard1/db services/shard2 services/shard2/db services/shard3 services/shard3/db services/configsvr services/configsvr/db services/mongos

3、配置relica sets

進入到servcies目錄

[[email protected] ~]$ cd services/

 在三臺機器上分別先執行(1)(2)(3)(4)步,注意配置檔案中bingIp屬性,需不同機器不同配置

(1)配置並啟動Shard1

[[email protected] services]$ vi shard1/mongod.conf 

 

#shard1 mongod.conf
storage:
    dbPath: /home/mongo/services/shard1/db
    journal:
        enabled: true
systemLog:
    destination: file
    logAppend: true
    path: /home/mongo/services/shard1/mongod.log
net:
    port: 27018
    bindIp: 172.16.0.192
processManagement:
  fork:
true
  pidFilePath:
/home/mongo/services/shard1/pid
replication:
  replSetName: shard1
sharding:
  clusterRole: shardsvr

啟動mongod程序,檢視是否成功,若不成功,可檢視配置的mongod.log,根據錯誤日誌進行配置檔案修改

[[email protected] services]$ mongodb/bin/mongod -f shard1/mongod.conf 
about to fork child process, waiting until server is ready for connections.
forked process: 19834
child process started successfully, parent exiting

(2)配置並啟動Shard2

[[email protected] services]$ vi shard2/mongod.conf 

 

#shard2 mongod.conf
storage:
    dbPath: /home/mongo/services/shard2/db
    journal:
        enabled: true
systemLog:
    destination: file
    logAppend: true
    path: /home/mongo/services/shard2/mongod.log
net:
    port: 27019
    bindIp: 127.0.0.1
processManagement:
    fork: true
    pidFilePath: /home/mongo/services/shard2/pid
replication:
    replSetName: shard2
sharding:
    clusterRole: shardsvr

啟動mongod程序,檢視是否成功,若不成功,可檢視配置的mongod.log,根據錯誤日誌進行配置檔案修改

[[email protected] services]$ mongodb/bin/mongod -f shard2/mongod.conf 
about to fork child process, waiting until server is ready for connections.
forked process: 27144
child process started successfully, parent exiting

(3)配置並啟動Shard3

[[email protected] services]$ vi shard3/mongod.conf 
#shard31 mongod.conf
storage:
    dbPath: /home/mongo/services/shard3/db
    journal:
        enabled: true
systemLog:
    destination: file
    logAppend: true
    path: /home/mongo/services/shard3/mongod.log
net:
    port: 27020
    bindIp: 127.0.0.1
processManagement:
    fork: true
    pidFilePath: /home/mongo/services/shard3/pid
replication:
    replSetName: shard3
sharding:
    clusterRole: shardsvr

 啟動mongod程序

[[email protected] services]$ mongodb/bin/mongod -f shard3/mongod.conf 
about to fork child process, waiting until server is ready for connections.
forked process: 1101
child process started successfully, parent exiting

(4)配置並啟動configsvr

[[email protected] services]$ vi configsvr/cfg.conf
#configsvr cfg.conf
storage:
    dbPath: /home/mongo/services/configsvr/db
    journal:
        enabled: true
systemLog:
    destination: file
    logAppend: true
    path: /home/mongo/services/configsvr/mongod.log
net:
    port: 20000
    bindIp: 127.0.0.1
processManagement:
    fork: true
    pidFilePath: /home/mongo/services/configsvr/pid
replication:
    replSetName: cfg
sharding:
    clusterRole: configsvr

 啟動mongod程序

[[email protected] services]$ mongodb/bin/mongod -f configsvr/cfg.conf 
about to fork child process, waiting until server is ready for connections.
forked process: 7940
child process started successfully, parent exiting

 

(5)關閉防火牆或者開放防火牆上述所用埠

各種版本liunx的防火牆關閉指令不同,目前我這裡使用的是CentOS7

[[email protected] services]$ systemctl stop firewalld
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: root
Password: 
==== AUTHENTICATION COMPLETE ===

(6)初始化Shard副本集

進入其中一個Shard1容器配置Shard1副本集

[[email protected] services]$ mongodb/bin/mongo 172.16.0.194:27018
MongoDB shell version v4.0.3
connecting to: mongodb://172.16.0.194:27018/test
Implicit session: session { "id" : UUID("d24db7a9-132f-4db4-8b20-0a082f72486d") }
MongoDB server version: 4.0.3
Server has startup warnings: 
2018-10-30T23:41:10.942+0800 I CONTROL  [initandlisten] 
2018-10-30T23:41:10.942+0800 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-10-30T23:41:10.942+0800 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-10-30T23:41:10.942+0800 I CONTROL  [initandlisten] 
2018-10-30T23:41:10.942+0800 I CONTROL  [initandlisten] 
2018-10-30T23:41:10.943+0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-10-30T23:41:10.943+0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-10-30T23:41:10.943+0800 I CONTROL  [initandlisten] 
2018-10-30T23:41:10.943+0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-10-30T23:41:10.943+0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-10-30T23:41:10.943+0800 I CONTROL  [initandlisten] 
> 
> rs.initiate({
...     "_id":"shard1",
...     "members":[
...         {
...             "_id":0,
...             "host":"172.16.0.192:27018"
...         },
...         {
...             "_id":1,
...             "host":"172.16.0.193:27018"
...         },
...         {
...             "_id":2,
...             "host":"172.16.0.194:27018"
...         }
...     ]
... })
{
        "ok" : 1,
        "operationTime" : Timestamp(1540915476, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1540915476, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}

進入其中一個Shard2容器配置Shard2副本集

[[email protected] services]$ mongodb/bin/mongo 172.16.0.193:27019
> rs.initiate({
...     "_id":"shard3",
...     "members":[
...         {
...             "_id":0,
...             "host":"172.16.0.192:27020"
...         },
...         {
...             "_id":1,
...             "host":"172.16.0.193:27020"
...         },
...         {
...             "_id":2,
...             "host":"172.16.0.194:27020"
...         }
...     ]
... })

進入其中一個Shard3容器配置Shard3副本集

[[email protected] services]$ mongodb/bin/mongo 172.16.0.192:27020
2018-10-30T23:32:26.519+0800 I CONTROL  [initandlisten] 
> rs.initiate({
...     "_id":"shard3",
...     "members":[
...         {
...             "_id":0,
...             "host":"172.16.0.192:27020"
...         },
...         {
...             "_id":1,
...             "host":"172.16.0.193:27020"
...         },
...         {
...             "_id":2,
...             "host":"172.16.0.194:27020"
...         }
...     ]
... })

(7)初始化Configsvr副本集

進入其中一個Configsvr容器配置Configsvr副本集

[[email protected] services]$ mongodb/bin/mongo 172.16.0.193:20000
> rs.initiate({
...     "_id":"cfg",
...     "members":[
...         {
...             "_id":0,
...             "host":"172.16.0.192:20000"
...         },
...         {
...             "_id":1,
...             "host":"172.16.0.193:20000"
...         },
...         {
...             "_id":2,
...             "host":"172.16.0.194:20000"
...         }
...     ]
... })

4、配置並啟動路由程序(分別在三臺機器上操作,注意bindIp的值)

[[email protected] services]$ vi mongos/mongos.conf
#mongos mongos.conf
systemLog:
    destination: file
    logAppend: true
    path: /home/mongo/services/mongos/mongos.log
net:
    port: 27017
    bindIp: 172.16.0.192
processManagement:
    fork: true
    pidFilePath: /home/mongo/services/mongos/pid
sharding:
    configDB: cfg/172.16.0.192:20000,172.16.0.193:20000,172.16.0.194:20000

啟動路由程序

[[email protected] services]$ mongodb/bin/mongos -f mongos/mongos.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27963
child process started successfully, parent exiting

 5、新增分片叢集

(1)從3臺機器中任意找一臺,連線mongod

 

[[email protected] services]$ mongodb/bin/mongo 172.16.0.192:27017

 

(2)新增3個分片叢集

mongos> sh.addShard("shard1/172.16.0.192:27018,172.16.0.193:27018,172.16.0.194:27018")
{
        "shardAdded" : "shard1",
        "ok" : 1,
        "operationTime" : Timestamp(1540981435, 6),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1540981435, 6),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}
mongos> sh.addShard("shard2/172.16.0.192:27019,172.16.0.193:27019,172.16.0.194:27019")
{
        "shardAdded" : "shard2",
        "ok" : 1,
        "operationTime" : Timestamp(1540981449, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1540981449, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}
mongos> sh.addShard("shard3/172.16.0.192:27020,172.16.0.193:27020,172.16.0.194:27020")
{
        "shardAdded" : "shard3",
        "ok" : 1,
        "operationTime" : Timestamp(1540981455, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1540981455, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}

 (3)通過sh.status()檢查上面配置是否正確

mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("5bd8849513e9f7fb57277071")
  }
  shards:
        {  "_id" : "shard1",  "host" : "shard1/172.16.0.192:27018,172.16.0.193:27018,172.16.0.194:27018",  "state" : 1 }
        {  "_id" : "shard2",  "host" : "shard2/172.16.0.192:27019,172.16.0.193:27019,172.16.0.194:27019",  "state" : 1 }
        {  "_id" : "shard3",  "host" : "shard3/172.16.0.192:27020,172.16.0.193:27020,172.16.0.194:27020",  "state" : 1 }
  active mongoses:
        "4.0.3" : 3
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours: 
                No recent migrations
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard1  1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shard1 Timestamp(1, 0) 

mongos> 

(4)檢視分片叢集資料庫資訊

 

mongos> show dbs;
admin   0.000GB
config  0.001GB
mongos> db
test
mongos> use config
switched to db config
mongos> show collections
changelog
chunks
collections
lockpings
locks
migrations
mongos
shards
tags
transactions
version
mongos> 

 

至此,MongoDB分片叢集部署成功,生產部署還需要調整一些引數,這部分內容可以通過--help檢視引數詳情。

 

  6、測試分片叢集

 (1)新建資料庫,且向集合中插入文件

mongos> use testShard;
switched to db testShard
mongos> db.users.insert({userid:1,username:"ChavinKing",city:"beijing"})
WriteResult({ "nInserted" : 1 })
mongos> db.users.find()
{ "_id" : ObjectId("5bd99310e1a54b9bca661075"), "userid" : 1, "username" : "ChavinKing", "city" : "beijing" }
mongos> 

(2)檢視分片叢集狀態

mongos> sh.status();
--- Sharding Status --- 
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("5bd8849513e9f7fb57277071")
  }
  shards:
        {  "_id" : "shard1",  "host" : "shard1/172.16.0.192:27018,172.16.0.193:27018,172.16.0.194:27018",  "state" : 1 }
        {  "_id" : "shard2",  "host" : "shard2/172.16.0.192:27019,172.16.0.193:27019,172.16.0.194:27019",  "state" : 1 }
        {  "_id" : "shard3",  "host" : "shard3/172.16.0.192:27020,172.16.0.193:27020,172.16.0.194:27020",  "state" : 1 }
  active mongoses:
        "4.0.3" : 3
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours: 
                No recent migrations
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard1  1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shard1 Timestamp(1, 0) 
        {  "_id" : "testShard",  "primary" : "shard2",  "partitioned" : false,  "version" : {  "uuid" : UUID("842f5428-9e0a-49b4-9c18-ff1c95d1bfea"),  "lastMod" : 1 } }
        #資料庫testShard目前不支援分片("partitioned" :false),資料庫檔案儲存在shard2片上("primary" : "shard2"

3、MongoDB分片是針對集合的,要想使集合支援分片,首先需要使其資料庫支援分片,為資料庫testShard啟用分片

mongos> sh.enableSharding("testShard")
{
        "ok" : 1,
        "operationTime" : Timestamp(1540986138, 3),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1540986138, 3),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}

4、為分片欄位建立索引,同時為集合指定片鍵

mongos> db.users.ensureIndex({city:1})
{
        "raw" : {
                "shard3/172.16.0.192:27020,172.16.0.193:27020,172.16.0.194:27020" : {
                        "createdCollectionAutomatically" : false,
                        "numIndexesBefore" : 1,
                        "numIndexesAfter" : 2,
                        "ok" : 1
                }
        },
        "ok" : 1,
        "operationTime" : Timestamp(1540986455, 8),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1540986455, 8),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}
mongos>  sh.shardCollection("testShard.users",{city:1})  
{
        "ok" : 0,
        "errmsg" : "Please create an index that starts with the proposed shard key before sharding the collection",
        "code" : 72,
        "codeName" : "InvalidOptions",
        "operationTime" : Timestamp(1540986499, 4),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1540986499, 4),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}
mongos> 

5、再次檢視分片叢集狀態

mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("5bd8849513e9f7fb57277071")
  }
  shards:
        {  "_id" : "shard1",  "host" : "shard1/172.16.0.192:27018,172.16.0.193:27018,172.16.0.194:27018",  "state" : 1 }
        {  "_id" : "shard2",  "host" : "shard2/172.16.0.192:27019,172.16.0.193:27019,172.16.0.194:27019",  "state" : 1 }
        {  "_id" : "shard3",  "host" : "shard3/172.16.0.192:27020,172.16.0.193:27020,172.16.0.194:27020",  "state" : 1 }
  active mongoses:
        "4.0.3" : 3
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours: 
                No recent migrations
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard1  1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shard1 Timestamp(1, 0) 
        {  "_id" : "test",  "primary" : "shard3",  "partitioned" : false,  "version" : {  "uuid" : UUID("fabfdcc3-0757-4042-be35-39efe9691f86"),  "lastMod" : 1 } }
        {  "_id" : "testShard",  "primary" : "shard2",  "partitioned" : true,  "version" : {  "uuid" : UUID("842f5428-9e0a-49b4-9c18-ff1c95d1bfea"),  "lastMod" : 1 } }
                testShard.users
                        shard key: { "city" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard2  1
                        { "city" : { "$minKey" : 1 } } -->> { "city" : { "$maxKey" : 1 } } on : shard2 Timestamp(1, 0) 
         #資料庫testShard支援分片("partitioned" :true),資料庫檔案儲存在shard2片上 mongos
>

6、向叢集插入測試資料

 

mongos> for(var i=1;i<1000000;i++) db.users.insert({userid:i,username:"chavin"+i,city:"beijing"})
mongos> for(var i=1;i<1000000;i++) db.users.insert({userid:i,username:"dbking"+i,city:"changsha"})

 7、再次檢視分片叢集狀態

mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("5bd8849513e9f7fb57277071")
  }
  shards:
        {  "_id" : "shard1",  "host" : "shard1/172.16.0.192:27018,172.16.0.193:27018,172.16.0.194:27018",  "state" : 1 }
        {  "_id" : "shard2",  "host" : "shard2/172.16.0.192:27019,172.16.0.193:27019,172.16.0.194:27019",  "state" : 1 }
        {  "_id" : "shard3",  "host" : "shard3/172.16.0.192:27020,172.16.0.193:27020,172.16.0.194:27020",  "state" : 1 }
  active mongoses:
        "4.0.3" : 3
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  5
        Last reported error:  Could not find host matching read preference { mode: "primary" } for set shard2
        Time of Reported error:  Thu Nov 01 2018 18:22:23 GMT+0800 (CST)
        Migration Results for the last 24 hours: 
                2 : Success
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard1  1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shard1 Timestamp(1, 0) 
        {  "_id" : "test",  "primary" : "shard3",  "partitioned" : false,  "version" : {  "uuid" : UUID("fabfdcc3-0757-4042-be35-39efe9691f86"),  "lastMod" : 1 } }
        {  "_id" : "testShard",  "primary" : "shard2",  "partitioned" : true,  "version" : {  "uuid" : UUID("842f5428-9e0a-49b4-9c18-ff1c95d1bfea"),  "lastMod" : 1 } }
                testShard.users
                        shard key: { "city" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard1  1
                                shard2  1
                                shard3  1
                        { "city" : { "$minKey" : 1 } } -->> { "city" : "beijing" } on : shard3 Timestamp(3, 0) 
                        { "city" : "beijing" } -->> { "city" : "shanghai" } on : shard2 Timestamp(3, 1) 
                        { "city" : "shanghai" } -->> { "city" : { "$maxKey" : 1 } } on : shard1 Timestamp(2, 0) 
          #資料庫testShard支援分片("partitioned" :true),資料庫檔案儲存在shard1、shard2片、shard3

測試完畢