1. 程式人生 > >YCSB 對MongoDB 進行壓力測試

YCSB 對MongoDB 進行壓力測試

 執行環境:CentOS7.5 +YCSB 0.16版本+MongoDB 3.6.6

注意:
參考YCSB 0.15.0的版本釋出說明資訊,
Tested in previous releases, unchanged in this release中MongoDB支援的版本
為3.6,而早於3.6版本之前的則不支援。
1.安裝JDK:
為了便於使用直接使用RPM:
#rpm -ivh jdk-8u172-linux-x64.rpm
# java -version
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)

2.maven
#wget http://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz
#tar -xzf apache-maven-3.5.4-bin.tar.gz -C /usr/local/apache-maven-3.5.4-bin
#mv  /usr/local/apache-maven-3.5.4-bin /usr/local/maven
#vim  /etc/profile.d/mvn.sh
#/bin/bash
export M2_HOME=/usr/local/maven
export MAVEN_OPTS="-Xms256m -Xmx512m"
export PATH=$M2_HOME/bin:$PATH
#source  /etc/profile.d/mvn.sh 
--驗證:
# mvn --version
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00)
Maven home: /usr/local/maven
Java version: 1.8.0_172, vendor: Oracle Corporation, runtime: /usr/local/jdk/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-327.el7.x86_64", arch: "amd64", family: "unix"
3.YCSB:
#git clone https://github.com/brianfrankcooper/YCSB.git
# cd YCSB
--建立所有的:
# mvn clean package
--只針對某個包進行編譯:
mvn -pl com.yahoo.ycsb:mongodb-binding -am clean package

編譯資訊:

[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] YCSB Root                                                          [pom]
[INFO] Core YCSB                                                          [jar]
[INFO] Per Datastore Binding descriptor                                   [jar]
[INFO] YCSB Datastore Binding Parent                                      [pom]
[INFO] Accumulo 1.6 DB Binding                                            [jar]
[INFO] Accumulo 1.7 DB Binding                                            [jar]
[INFO] Accumulo 1.8 DB Binding                                            [jar]
[INFO] Aerospike DB Binding                                               [jar]
[INFO] ArangoDB Binding                                                   [jar]
[INFO] AsyncHBase Client Binding for Apache HBase                         [jar]
[INFO] Cassandra 2.1+ DB Binding                                          [jar]
[INFO] Cloud Spanner DB Binding                                           [jar]
[INFO] Couchbase Binding                                                  [jar]
[INFO] Couchbase Java SDK 2.x Binding                                     [jar]
[INFO] Azure DocumentDB Binding                                           [jar]
[INFO] Azure table storage Binding                                        [jar]
[INFO] DynamoDB DB Binding                                                [jar]
[INFO] Elasticsearch Binding                                              [jar]
[INFO] Elasticsearch 5.x Binding                                          [jar]
[INFO] FoundationDB Binding                                               [jar]
[INFO] Geode DB Binding                                                   [jar]
[INFO] Google Cloud Datastore Binding                                     [jar]
[INFO] Google Cloud Bigtable Binding                                      [jar]
[INFO] HBase 0.98.x DB Binding                                            [jar]
[INFO] HBase 1.0 DB Binding                                               [jar]
[INFO] HBase 1.2 DB Binding                                               [jar]
[INFO] HBase 1.4 DB Binding                                               [jar]
[INFO] HBase 2.0 DB Binding                                               [jar]
[INFO] Hypertable DB Binding                                              [jar]
[INFO] Apache Ignite                                                      [jar]
[INFO] Infinispan DB Binding                                              [jar]
[INFO] JDBC DB Binding                                                    [jar]
[INFO] Kudu DB Binding                                                    [jar]
[INFO] memcached binding                                                  [jar]
[INFO] MapRDB Binding                                                     [jar]
[INFO] MapR JSON DB Binding                                               [jar]
[INFO] MongoDB Binding                                                    [jar]
[INFO] Oracle NoSQL Database Binding                                      [jar]
[INFO] OrientDB Binding                                                   [jar]
[INFO] rados of Ceph FS binding                                           [jar]
[INFO] Redis DB Binding                                                   [jar]
[INFO] Rest Client Binding                                                [jar]
[INFO] Riak KV Binding                                                    [jar]
[INFO] RocksDB Java Binding                                               [jar]
[INFO] S3 Storage Binding                                                 [jar]
[INFO] Solr Binding                                                       [jar]
[INFO] Solr 6 Binding                                                     [jar]
[INFO] Tarantool DB Binding                                               [jar]
[INFO] YCSB Release Distribution Builder                                  [pom]
[INFO] 
[INFO] ------------------------< com.yahoo.ycsb:root >-------------------------
[INFO] Building YCSB Root 0.16.0-SNAPSHOT                                [1/49]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
...

只編譯單個包:

# mvn -pl com.yahoo.ycsb:mongodb-binding -am clean package
[INFO] Building tar: /data/soft/YCSB/mongodb/target/ycsb-mongodb-binding-0.16.0-SNAPSHOT.tar.gz
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] YCSB Root 0.16.0-SNAPSHOT .......................... SUCCESS [04:49 min]
[INFO] Core YCSB .......................................... SUCCESS [03:05 min]
[INFO] Per Datastore Binding descriptor ................... SUCCESS [  0.192 s]
[INFO] YCSB Datastore Binding Parent ...................... SUCCESS [01:21 min]
[INFO] MongoDB Binding 0.16.0-SNAPSHOT .................... SUCCESS [03:17 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:34 min
[INFO] Finished at: 2018-08-25T20:37:34+08:00
[INFO] ------------------------------------------------------------------------

YCSB提供的測試模式:

檔名     測試場景                  讀取更新插入比 應用示例 
workloada Update heavy workload      50/50   Session store recording recent actions
workloadb Read mostly workload       95/5    photo tagging; add a tag is an update, but most operations are to read tags
workloadc Read only                  100/0   user profile cache, where profiles are constructed elsewhere (e.g., Hadoop)
workloadd Read latest workload       95/0/5  user status updates; people want to read the latest
workloade Short ranges               95/5    threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id)
workloadf Read-modify-write workload 50/50   user database, where user records are read and modified by the user or to record user activity.
註釋:
其他的預設引數:
Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
Request distribution: zipfian

YCSB提供的配置引數說明:

引數                      說明
fieldcount                單條記錄欄位個數,預設10
fieldlength               每個欄位的大小,預設100B
readallfields             是否讀取所有記錄,預設true
readproportion            讀比例(預設0.95)
updateproportion          更新比例(預設0.05)
insertproportion          插入比例(預設0)
scanproportion            掃描比例(預設0)
readmodifywriteproportion 同一記錄讀、修改、回寫比例(預設值:0)
requestdistribution       記錄選擇策略:uniform,zipfian,latest(預設值:uniform)
maxscanlength             最大scan記錄數(預設值:100)
scanlengthdistribution    scan記錄選擇策略(預設值:uniform)
insertorder               記錄插入策略:ordered,hashed(預設值:hashed)
operationcount            操作執行數
maxexecutiontime          最大執行時間,單位:秒
table                     表名(預設值值usertable)
recordcount               初始化記錄條數(預設值:0)

MongDB的配置引數:

引數                說明
mongodb.url          MongoDB URI或者連線串
mongodb.batchsize    對大量inert操作的時候,分批commit提交,提高吞吐量
mongodb.upsert       預設是false.true表示update操作,false表示insert操作.
mongodb.writeConcern 預設值是acknowledged。
                     可選的值:
                     - `errors_ignored`
                     - `unacknowledged`
                     - `acknowledged`
                     - `journaled`
                     - `replica_acknowledged`
                     - `majority`
mongodb.readPreference 預設值是primary
                       可選值:
                        - `primary`
                        - `primary_preferred`
                        - `secondary`
                        - `secondary_preferred`
                        - `nearest`
mongodb.maxconnections 預設值是100. mongodb.url選型中的maxPoolSize的值。
mongodb.threadsAllowedToBlockForConnectionMultiplier
                       預設值是5. mongodb.url 選型中的waitQueueMultiple的值。

YCSB執行檔案的幫助資訊:

# cd /data/soft/YCSB/mongodb/target/ycsb-mongodb-binding-0.16.0-SNAPSHOT
# ./bin/ycsb --help
usage: ./bin/ycsb command database [options]

Commands:
    load           Execute the load phase
    run            Execute the transaction phase
    shell          Interactive mode

Databases:
    accumulo       https://github.com/brianfrankcooper/YCSB/tree/master/accumulo
    accumulo1.6    https://github.com/brianfrankcooper/YCSB/tree/master/accumulo1.6
...
Options:
    -P file        Specify workload file
    -cp path       Additional Java classpath entries
    -jvm-args args Additional arguments to the JVM
    -p key=value   Override workload property
    -s             Print status to stderr
    -target n      Target ops/sec (default: unthrottled)
    -threads n     Number of client threads (default: 1)

Workload Files:
    There are various predefined workloads under workloads/ directory.
    See https://github.com/brianfrankcooper/YCSB/wiki/Core-Properties
    for the list of workload properties.

positional arguments:
  {load,run,shell}      Command to run.
  {accumulo,accumulo1.6,accumulo1.7,accumulo1.8,aerospike,arangodb,arangodb3,asynchbase,azuredocumentdb,azuretablestorage,basic,basicts,cassandra-cql,cassandra2-cql,cloudspanner,couchbase,couchbase2,dynamodb,elasticsearch,elasticsearch5,foundationdb,geode,googlebigtable,googledatastore,hbase098,hbase10,hbase12,hbase14,hbase20,hypertable,ignite,ignite-sql,infinispan,infinispan-cs,jdbc,kudu,maprdb,maprjsondb,memcached,mongodb,mongodb-async,nosqldb,orientdb,rados,redis,rest,riak,rocksdb,s3,solr,solr6,tarantool}
                        Database to test.

optional arguments:
  -h, --help            show this help message and exit
  -cp CLASSPATH         Additional classpath entries, e.g. '-cp
                        /tmp/hbase-1.0.1.1/conf'. Will be prepended to the
                        YCSB classpath.
  -jvm-args JVM_ARGS    Additional arguments to pass to 'java', e.g. '-Xmx4g'

YCSB的workload配置和使用:

方法1:壓測引數和monogBD的配置資訊寫到一起

場景說明:記錄數為1000,操作次數1000,讀寫場景為讀取和更新平衡,
每個記錄為2k(250bytes*8列),併發數量為100,write concern為acknowledged
# cat workloads/workloada | grep -v ^#
recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0
fieldlength=250
fieldcount=8
requestdistribution=zipfian
maxexecutiontime=180
threadcount=100
mongodb.url=mongodb://192.168.0.151:27017/ycsb   
mongodb.database=ycsb
mongodb.writeConcern=replica_acknowledged
--使用方法:
#./bin/ycsb --load mongodb -s -P workloads/workloada

方法2:將壓測引數和mongoDB的配置資訊分開,mongoDB的配置資訊寫到ycsb的後面:

# cat workloads/workloadb | grep -v ^#     
recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=0.95
updateproportion=0.05
scanproportion=0
insertproportion=0
--使用方法:
#./bin/ycsb --load mongodb -s -P workloads/workloadb  -p mongodb.url=mongodb://user:[email protected]:27017/ycsb

針對MongoDB的壓測模式:

非同步模式:
#./bin/ycsb.sh load mongodb-async -s -P workloads/workloada
同步模式:(預設)
#./bin/ycsb.sh load mongodb -s -P workloads/workloada

針對MongoDB的壓力測試的步驟:

--壓力測試的步驟:
1.啟動mongoDB,建立測試資料庫,庫名為ycsb,使用者和密碼均為ycsb
2.編寫workload檔案:
3.載入資料:load
3.執行壓測:run
4.壓測過程中使用mongostat和iostat一起觀察
mongostat -uycsb -pycsb --authenticationDatabase=ycsb -h192.168.0.151
iostat -xmt 2
推薦使用pmm等圖形介面的監控。
5.在不同併發下壓力測試。
6.出具測試報告。

相關推薦

no