YCSB 對MongoDB 進行壓力測試
阿新 • • 發佈:2019-01-11
執行環境:CentOS7.5 +YCSB 0.16版本+MongoDB 3.6.6
注意:
參考YCSB 0.15.0的版本釋出說明資訊,
Tested in previous releases, unchanged in this release中MongoDB支援的版本
為3.6,而早於3.6版本之前的則不支援。
1.安裝JDK: 為了便於使用直接使用RPM: #rpm -ivh jdk-8u172-linux-x64.rpm # java -version java version "1.8.0_172" Java(TM) SE Runtime Environment (build 1.8.0_172-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode) 2.maven #wget http://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz #tar -xzf apache-maven-3.5.4-bin.tar.gz -C /usr/local/apache-maven-3.5.4-bin #mv /usr/local/apache-maven-3.5.4-bin /usr/local/maven #vim /etc/profile.d/mvn.sh #/bin/bash export M2_HOME=/usr/local/maven export MAVEN_OPTS="-Xms256m -Xmx512m" export PATH=$M2_HOME/bin:$PATH #source /etc/profile.d/mvn.sh --驗證: # mvn --version Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00) Maven home: /usr/local/maven Java version: 1.8.0_172, vendor: Oracle Corporation, runtime: /usr/local/jdk/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-327.el7.x86_64", arch: "amd64", family: "unix" 3.YCSB: #git clone https://github.com/brianfrankcooper/YCSB.git # cd YCSB --建立所有的: # mvn clean package --只針對某個包進行編譯: mvn -pl com.yahoo.ycsb:mongodb-binding -am clean package
編譯資訊:
[INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] YCSB Root [pom] [INFO] Core YCSB [jar] [INFO] Per Datastore Binding descriptor [jar] [INFO] YCSB Datastore Binding Parent [pom] [INFO] Accumulo 1.6 DB Binding [jar] [INFO] Accumulo 1.7 DB Binding [jar] [INFO] Accumulo 1.8 DB Binding [jar] [INFO] Aerospike DB Binding [jar] [INFO] ArangoDB Binding [jar] [INFO] AsyncHBase Client Binding for Apache HBase [jar] [INFO] Cassandra 2.1+ DB Binding [jar] [INFO] Cloud Spanner DB Binding [jar] [INFO] Couchbase Binding [jar] [INFO] Couchbase Java SDK 2.x Binding [jar] [INFO] Azure DocumentDB Binding [jar] [INFO] Azure table storage Binding [jar] [INFO] DynamoDB DB Binding [jar] [INFO] Elasticsearch Binding [jar] [INFO] Elasticsearch 5.x Binding [jar] [INFO] FoundationDB Binding [jar] [INFO] Geode DB Binding [jar] [INFO] Google Cloud Datastore Binding [jar] [INFO] Google Cloud Bigtable Binding [jar] [INFO] HBase 0.98.x DB Binding [jar] [INFO] HBase 1.0 DB Binding [jar] [INFO] HBase 1.2 DB Binding [jar] [INFO] HBase 1.4 DB Binding [jar] [INFO] HBase 2.0 DB Binding [jar] [INFO] Hypertable DB Binding [jar] [INFO] Apache Ignite [jar] [INFO] Infinispan DB Binding [jar] [INFO] JDBC DB Binding [jar] [INFO] Kudu DB Binding [jar] [INFO] memcached binding [jar] [INFO] MapRDB Binding [jar] [INFO] MapR JSON DB Binding [jar] [INFO] MongoDB Binding [jar] [INFO] Oracle NoSQL Database Binding [jar] [INFO] OrientDB Binding [jar] [INFO] rados of Ceph FS binding [jar] [INFO] Redis DB Binding [jar] [INFO] Rest Client Binding [jar] [INFO] Riak KV Binding [jar] [INFO] RocksDB Java Binding [jar] [INFO] S3 Storage Binding [jar] [INFO] Solr Binding [jar] [INFO] Solr 6 Binding [jar] [INFO] Tarantool DB Binding [jar] [INFO] YCSB Release Distribution Builder [pom] [INFO] [INFO] ------------------------< com.yahoo.ycsb:root >------------------------- [INFO] Building YCSB Root 0.16.0-SNAPSHOT [1/49] [INFO] --------------------------------[ pom ]--------------------------------- [INFO] ...
只編譯單個包:
# mvn -pl com.yahoo.ycsb:mongodb-binding -am clean package [INFO] Building tar: /data/soft/YCSB/mongodb/target/ycsb-mongodb-binding-0.16.0-SNAPSHOT.tar.gz [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] YCSB Root 0.16.0-SNAPSHOT .......................... SUCCESS [04:49 min] [INFO] Core YCSB .......................................... SUCCESS [03:05 min] [INFO] Per Datastore Binding descriptor ................... SUCCESS [ 0.192 s] [INFO] YCSB Datastore Binding Parent ...................... SUCCESS [01:21 min] [INFO] MongoDB Binding 0.16.0-SNAPSHOT .................... SUCCESS [03:17 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 12:34 min [INFO] Finished at: 2018-08-25T20:37:34+08:00 [INFO] ------------------------------------------------------------------------
YCSB提供的測試模式:
檔名 測試場景 讀取更新插入比 應用示例
workloada Update heavy workload 50/50 Session store recording recent actions
workloadb Read mostly workload 95/5 photo tagging; add a tag is an update, but most operations are to read tags
workloadc Read only 100/0 user profile cache, where profiles are constructed elsewhere (e.g., Hadoop)
workloadd Read latest workload 95/0/5 user status updates; people want to read the latest
workloade Short ranges 95/5 threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id)
workloadf Read-modify-write workload 50/50 user database, where user records are read and modified by the user or to record user activity.
註釋:
其他的預設引數:
Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
Request distribution: zipfian
YCSB提供的配置引數說明:
引數 說明
fieldcount 單條記錄欄位個數,預設10
fieldlength 每個欄位的大小,預設100B
readallfields 是否讀取所有記錄,預設true
readproportion 讀比例(預設0.95)
updateproportion 更新比例(預設0.05)
insertproportion 插入比例(預設0)
scanproportion 掃描比例(預設0)
readmodifywriteproportion 同一記錄讀、修改、回寫比例(預設值:0)
requestdistribution 記錄選擇策略:uniform,zipfian,latest(預設值:uniform)
maxscanlength 最大scan記錄數(預設值:100)
scanlengthdistribution scan記錄選擇策略(預設值:uniform)
insertorder 記錄插入策略:ordered,hashed(預設值:hashed)
operationcount 操作執行數
maxexecutiontime 最大執行時間,單位:秒
table 表名(預設值值usertable)
recordcount 初始化記錄條數(預設值:0)
MongDB的配置引數:
引數 說明
mongodb.url MongoDB URI或者連線串
mongodb.batchsize 對大量inert操作的時候,分批commit提交,提高吞吐量
mongodb.upsert 預設是false.true表示update操作,false表示insert操作.
mongodb.writeConcern 預設值是acknowledged。
可選的值:
- `errors_ignored`
- `unacknowledged`
- `acknowledged`
- `journaled`
- `replica_acknowledged`
- `majority`
mongodb.readPreference 預設值是primary
可選值:
- `primary`
- `primary_preferred`
- `secondary`
- `secondary_preferred`
- `nearest`
mongodb.maxconnections 預設值是100. mongodb.url選型中的maxPoolSize的值。
mongodb.threadsAllowedToBlockForConnectionMultiplier
預設值是5. mongodb.url 選型中的waitQueueMultiple的值。
YCSB執行檔案的幫助資訊:
# cd /data/soft/YCSB/mongodb/target/ycsb-mongodb-binding-0.16.0-SNAPSHOT
# ./bin/ycsb --help
usage: ./bin/ycsb command database [options]
Commands:
load Execute the load phase
run Execute the transaction phase
shell Interactive mode
Databases:
accumulo https://github.com/brianfrankcooper/YCSB/tree/master/accumulo
accumulo1.6 https://github.com/brianfrankcooper/YCSB/tree/master/accumulo1.6
...
Options:
-P file Specify workload file
-cp path Additional Java classpath entries
-jvm-args args Additional arguments to the JVM
-p key=value Override workload property
-s Print status to stderr
-target n Target ops/sec (default: unthrottled)
-threads n Number of client threads (default: 1)
Workload Files:
There are various predefined workloads under workloads/ directory.
See https://github.com/brianfrankcooper/YCSB/wiki/Core-Properties
for the list of workload properties.
positional arguments:
{load,run,shell} Command to run.
{accumulo,accumulo1.6,accumulo1.7,accumulo1.8,aerospike,arangodb,arangodb3,asynchbase,azuredocumentdb,azuretablestorage,basic,basicts,cassandra-cql,cassandra2-cql,cloudspanner,couchbase,couchbase2,dynamodb,elasticsearch,elasticsearch5,foundationdb,geode,googlebigtable,googledatastore,hbase098,hbase10,hbase12,hbase14,hbase20,hypertable,ignite,ignite-sql,infinispan,infinispan-cs,jdbc,kudu,maprdb,maprjsondb,memcached,mongodb,mongodb-async,nosqldb,orientdb,rados,redis,rest,riak,rocksdb,s3,solr,solr6,tarantool}
Database to test.
optional arguments:
-h, --help show this help message and exit
-cp CLASSPATH Additional classpath entries, e.g. '-cp
/tmp/hbase-1.0.1.1/conf'. Will be prepended to the
YCSB classpath.
-jvm-args JVM_ARGS Additional arguments to pass to 'java', e.g. '-Xmx4g'
YCSB的workload配置和使用:
方法1:壓測引數和monogBD的配置資訊寫到一起
場景說明:記錄數為1000,操作次數1000,讀寫場景為讀取和更新平衡,
每個記錄為2k(250bytes*8列),併發數量為100,write concern為acknowledged
# cat workloads/workloada | grep -v ^#
recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0
fieldlength=250
fieldcount=8
requestdistribution=zipfian
maxexecutiontime=180
threadcount=100
mongodb.url=mongodb://192.168.0.151:27017/ycsb
mongodb.database=ycsb
mongodb.writeConcern=replica_acknowledged
--使用方法:
#./bin/ycsb --load mongodb -s -P workloads/workloada
方法2:將壓測引數和mongoDB的配置資訊分開,mongoDB的配置資訊寫到ycsb的後面:
# cat workloads/workloadb | grep -v ^#
recordcount=1000
operationcount=1000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=0.95
updateproportion=0.05
scanproportion=0
insertproportion=0
--使用方法:
#./bin/ycsb --load mongodb -s -P workloads/workloadb -p mongodb.url=mongodb://user:[email protected]:27017/ycsb
針對MongoDB的壓測模式:
非同步模式:
#./bin/ycsb.sh load mongodb-async -s -P workloads/workloada
同步模式:(預設)
#./bin/ycsb.sh load mongodb -s -P workloads/workloada
針對MongoDB的壓力測試的步驟:
--壓力測試的步驟:
1.啟動mongoDB,建立測試資料庫,庫名為ycsb,使用者和密碼均為ycsb
2.編寫workload檔案:
3.載入資料:load
3.執行壓測:run
4.壓測過程中使用mongostat和iostat一起觀察
mongostat -uycsb -pycsb --authenticationDatabase=ycsb -h192.168.0.151
iostat -xmt 2
推薦使用pmm等圖形介面的監控。
5.在不同併發下壓力測試。
6.出具測試報告。