Jmeter Redis外掛開發 -- 讀寫資料
阿新 • • 發佈：2018-12-06
背景
最近一段時間在接觸效能壓測，遇到一個棘手的問題。效能需求在30KQPS，要求進行單介面壓測，介面之間依賴不可避免（下一個介面發壓資料需要使用上一介面的返回），還不能通過做資料的方式準備。只能將上一介面返回的資料，儲存起來，用於下一介面的引數。
在一開始的時候，犯了一個很二的錯誤，將資料寫入到Jmeter的日誌中，再進行提取（發壓端檔案IO影響效能不是一點點），然後將受影響的效能指標作為測試結果（可進行兩次測試，第一次，不寫日誌，效能指標作為測試結果，第二次，寫日誌，採集的資料作為下一介面的配置元）。
當然了，這樣的工作，為什麼要每次都做兩遍呢，當然可以考慮開發一個外掛來做了。首先考慮的就是寫記憶體記錄下來。
主要考慮兩個方向：
① Jmeter記憶體
② Redis
其中，Jmeter記憶體儲存，僅在本次測試有效，還需要在測試結束時，將記憶體中資料，儲存到檔案。考慮用redis，網上搜索了一把，jmeter-plugins-redis能滿足讀取redis資料的需求（不重複造輪子，就用它了），只要自己完成一個寫入資料到redis的外掛就可以了。
準備工作
下載Redis外掛
官方提供的網址：https://jmeter-plugins.org/wiki/RedisDataSet/
下載下來之後，繼續下載依賴的jedis版本，官方依賴的是jedis 2.2.1
然鵝，在jmeter 3.0版本下，竟然用不起來，丟擲了一票的異常。度娘了一把（請原諒，公司翻牆要自備梯子），是jedis的版本過低，jedis 2.4.1 以上解決了。正好，專案組其他外掛依賴2.8.1，索性就用這個版本。
原始碼修改
用Jedis 2.8.1後，又拋異常，真是沮喪啊。怎麼辦？思來想去，改原始碼，就不信這個邪！
github上一搜索，還真就有。二話不說，下載原始碼：https://github.com/undera/jmeter-plugins
這位老哥undera，還是很厚道的，外掛原始碼都來了。
開啟原始碼，進行編譯，提示下面3個函式已經沒有了：
那就看下怎麼修改吧，setMaxActive -> setMaxTotal，setMaxWait -> setMaxWaitMillis，最後一個實在沒有找到，先註釋掉。
編譯，生成jar包，放進去跑一把，介面出來了。讀取資料，就算搞定了，下面去開發寫資料。
二次開發
由於前面已經下載了原始碼，而且jmeter外掛網站上，提供了原始碼，怎麼配合Redis Data Set的get mode，插入資料。
那就先做簡單的，證明這種方式可以用。考慮用後置處理器來解決。
簡單應用
1、新增資料操作類：RedisDataWriter
2、新增介面顯示：RedisDataWriterGUI
想法很簡單，先用起來，能在Jmeter上，找到這個後置處理器，能將資料插入到redis裡面。連線資訊之類的，都先給寫死。
跑完第一把，發現數據竟然能寫到redis，很滿足了。
新增介面
有了上面的經驗，那就可以慢慢的往上新增介面，優化連線。
RedisDataWriterGUI.java
import com.bilibili.redis.RedisDataWriter;
import org.apache.jmeter.processor.gui.AbstractPostProcessorGui;
import org.apache.jmeter.testelement.TestElement;
import java.awt.*;
import java.util.List;
import javax.swing.BorderFactory;
import javax.swing.Box;
import javax.swing.JComponent;
import javax.swing.JPanel;
import org.apache.jmeter.util.JMeterUtils;
import org.apache.jorphan.gui.JLabeledField;
import org.apache.jorphan.gui.JLabeledTextField;
import org.apache.jorphan.gui.JLabeledChoice;
public class RedisDataWriterGUI extends AbstractPostProcessorGui{
private JLabeledTextField redisKeyField;
private JLabeledTextField variableNamesField;
private JLabeledChoice addModeField;
private JLabeledTextField redisServerHostField;
private JLabeledTextField redisServerPortField;
private JLabeledTextField redisServerTimeoutField;
private JLabeledTextField redisServerPasswdField;
private JLabeledTextField redisServerDbField;
private JLabeledTextField redisPoolMinIdleField;
private JLabeledTextField redisPoolMaxIdleField;
private JLabeledTextField redisPoolMaxActiveField;
private JLabeledTextField redisPoolMaxWaitField;
public RedisDataWriterGUI() {
super();
init();
}
@Override
public void configure(TestElement el) {
super.configure(el);
if (el instanceof RedisDataWriter){
RedisDataWriter re = (RedisDataWriter) el;
redisKeyField.setText(re.getRedisKey());
variableNamesField.setText(re.getVariableNames());
addModeField.setText(re.getRedisAddMode());
redisServerHostField.setText(re.getRedisHost());
redisServerPortField.setText(re.getRedisPort());
redisServerTimeoutField.setText(re.getRedisTimeout());
redisServerPasswdField.setText(re.getRedisPasswd());
redisServerDbField.setText(re.getRedisDatabase());
redisPoolMinIdleField.setText(Integer.toString(re.getRedisMinIdle()));
redisPoolMaxIdleField.setText(Integer.toString(re.getRedisMaxIdle()));
redisPoolMaxActiveField.setText(Integer.toString(re.getRedisMaxActive()));
redisPoolMaxWaitField.setText(Integer.toString(re.getRedisMaxWait()));
redisKeyField.setLabel("Redis key: ");
variableNamesField.setLabel("Redis value (可以是變數列表, 逗號分隔, 例如：${token},${host}): ");
addModeField.setLabel("Redis寫入模式: ");
redisServerHostField.setLabel("Redis host: ");
redisServerPortField.setLabel("Redis port: ");
redisServerTimeoutField.setLabel("連線超時設定(ms): ");
redisServerPasswdField.setLabel("連線密碼: ");
redisServerDbField.setLabel("資料庫: ");
redisPoolMinIdleField.setLabel("minIdle: ");
redisPoolMaxIdleField.setLabel("maxIdle: ");
redisPoolMaxActiveField.setLabel("maxActive: ");
redisPoolMaxWaitField.setLabel("maxWait(s):");
}
}
/**
* Implements JMeterGUIComponent.clearGui
*/
@Override
public void clearGui() {
super.clearGui();
redisKeyField.setText(""); //$NON-NLS-1$
variableNamesField.setText(""); //$NON-NLS-1$
addModeField.setText("");
redisServerHostField.setText(""); //$NON-NLS-1$
redisServerPortField.setText(""); //$NON-NLS-1$
redisServerTimeoutField.setText("");
redisServerPasswdField.setText("");
redisServerDbField.setText("");
redisPoolMinIdleField.setText("");
redisPoolMaxIdleField.setText("");
redisPoolMaxActiveField.setText("");
redisPoolMaxWaitField.setText("");
}
@Override
public TestElement createTestElement() {
// TODO Auto-generated method stub
RedisDataWriter extractor = new RedisDataWriter();
modifyTestElement(extractor);
return extractor;
}
@Override
public String getLabelResource() {
// TODO Auto-generated method stub
return this.getClass().getName();
}
@Override
public String getStaticLabel() {//設定顯示名稱
// TODO Auto-generated method stub
return "Redis資料錄入器";
}
@Override
public void modifyTestElement(TestElement extractor) {
// TODO Auto-generated method stub
super.configureTestElement(extractor);
if (extractor instanceof RedisDataWriter) {
RedisDataWriter r = (RedisDataWriter) extractor;
r.setRediskey(redisKeyField.getText());
r.setVariableNames(variableNamesField.getText());
r.setRedisAddMode(addModeField.getText());
r.setRedisHost(redisServerHostField.getText());
r.setRedisPort(redisServerPortField.getText());
r.setRedisTimeout(redisServerTimeoutField.getText());
r.setRedisPasswd(redisServerPasswdField.getText());
r.setRedisDatabase(redisServerDbField.getText());
r.setRedisMinIdle(Integer.parseInt(redisPoolMinIdleField.getText()));
r.setRedisMaxIdle(Integer.parseInt(redisPoolMaxIdleField.getText()));
r.setRedisMaxActive(Integer.parseInt(redisPoolMaxActiveField.getText()));
r.setRedisMaxWait(Integer.parseInt(redisPoolMaxWaitField.getText()));
}
}
private void init() {
setLayout(new BorderLayout());
setBorder(makeBorder());
Box box = Box.createVerticalBox();
box.add(makeTitlePanel());
box.add(makeRedisDataPanel());
add(box, BorderLayout.NORTH);
box.add(makeRedisConnectionPanel());
box.add(makeRedisPoolPanel());
}
private JPanel makeRedisDataPanel() {
redisKeyField = new JLabeledTextField(JMeterUtils.getResString("rediskey_field")); //$NON-NLS-1$
variableNamesField = new JLabeledTextField(JMeterUtils.getResString("variable_names_field")); //$NON-NLS-1$
addModeField = new JLabeledChoice(JMeterUtils.getResString("add_mode_field"), false);
addModeField.addValue("LST_RPUSH");
addModeField.addValue("SET_ADD");
JPanel panel = new JPanel(new GridBagLayout());
panel.setBorder(BorderFactory.createTitledBorder("Redis資料配置")); //$NON-NLS-1$
GridBagConstraints gbc = new GridBagConstraints();
initConstraints(gbc);
addField(panel, redisKeyField, gbc);
resetContraints(gbc);
addField(panel, variableNamesField, gbc);
resetContraints(gbc);
addField(panel, addModeField, gbc);
return panel;
}
private JPanel makeRedisConnectionPanel() {
redisServerHostField = new JLabeledTextField(JMeterUtils.getResString("redis_server_host_field")); //$NON-NLS-1$
redisServerPortField = new JLabeledTextField(JMeterUtils.getResString("redis_server_port_field")); //$NON-NLS-1$
redisServerTimeoutField = new JLabeledTextField(JMeterUtils.getResString("redis_server_timeout_field"));
redisServerPasswdField = new JLabeledTextField(JMeterUtils.getResString("redis_server_passwd_field"));
redisServerDbField = new JLabeledTextField(JMeterUtils.getResString("redis_server_database_field"));
JPanel panel = new JPanel(new GridBagLayout());
panel.setBorder(BorderFactory.createTitledBorder("Redis連線配置")); //$NON-NLS-1$
GridBagConstraints gbc = new GridBagConstraints();
initConstraints(gbc);
addField(panel, redisServerHostField, gbc);
resetContraints(gbc);
addField(panel, redisServerPortField, gbc);
resetContraints(gbc);
addField(panel, redisServerTimeoutField, gbc);
resetContraints(gbc);
addField(panel, redisServerPasswdField, gbc);
resetContraints(gbc);
addField(panel, redisServerDbField, gbc);
return panel;
}
private JPanel makeRedisPoolPanel() {
redisPoolMinIdleField = new JLabeledTextField(JMeterUtils.getResString("redis_pool_minidle"));
redisPoolMaxIdleField = new JLabeledTextField(JMeterUtils.getResString("redis_pool_maxidle"));
redisPoolMaxActiveField = new JLabeledTextField(JMeterUtils.getResString("redis_pool_maxactive"));
redisPoolMaxWaitField = new JLabeledTextField(JMeterUtils.getResString("redis_pool_maxwait"));
JPanel panel = new JPanel(new GridBagLayout());
panel.setBorder(BorderFactory.createTitledBorder("Redis連線池配置")); //$NON-NLS-1$
GridBagConstraints gbc = new GridBagConstraints();
initConstraints(gbc);
addField(panel, redisPoolMinIdleField, gbc);
resetContraints(gbc);
addField(panel, redisPoolMaxIdleField, gbc);
resetContraints(gbc);
addField(panel, redisPoolMaxActiveField, gbc);
resetContraints(gbc);
addField(panel, redisPoolMaxWaitField, gbc);
return panel;
}
private void addField(JPanel panel, JLabeledField field, GridBagConstraints gbc) {
List<JComponent> item = field.getComponentList();
panel.add(item.get(0), gbc.clone());
gbc.gridx++;
gbc.weightx = 1;
gbc.fill=GridBagConstraints.HORIZONTAL;
panel.add(item.get(1), gbc.clone());
}
// Next line
private void resetContraints(GridBagConstraints gbc) {
gbc.gridx = 0;
gbc.gridy++;
gbc.weightx = 0;
gbc.fill=GridBagConstraints.NONE;
}
private void initConstraints(GridBagConstraints gbc) {
gbc.anchor = GridBagConstraints.NORTHWEST;
gbc.fill = GridBagConstraints.NONE;
gbc.gridheight = 1;
gbc.gridwidth = 1;
gbc.gridx = 0;
gbc.gridy = 0;
gbc.weightx = 0;
gbc.weighty = 0;
}
}RedisDataWriter.java
/*
* Created on 2018/01/26
* @author: ben大神點C
*/
package com.bilibili.redis;
import java.io.Serializable;
import org.apache.jmeter.processor.PostProcessor;
import org.apache.jmeter.samplers.SampleResult;
import org.apache.jmeter.testelement.AbstractScopedTestElement;
import org.apache.jmeter.testelement.TestStateListener;
import org.apache.jmeter.threads.JMeterContext;
import org.apache.jorphan.logging.LoggingManager;
import org.apache.jorphan.util.JOrphanUtils;
import org.apache.log.Logger;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.JedisPoolConfig;
import redis.clients.jedis.Protocol;
public class RedisDataWriter extends AbstractScopedTestElement implements PostProcessor, Serializable, TestStateListener {
private static final Logger log = LoggingManager.getLoggerForClass();
private static final String REDISKEY = "RedisData.redisKey"; // $NON-NLS-1$
private static final String VARIABLE_NAMES = "RedisData.variableNames";
private static final String REDISADDMODE = "RedisData.addMode";
private static final String REDISHOST = "RedisServer.host";
private static final String REDISPORT = "RedisServer.port";
private static final String REDISTIMEOUT = "RedisServer.timeout";
private static final String REDISPASSWD = "RedisServer.passwd";
private static final String REDISDATABSE = "RedisServer.database";
private static final String REDISMINIDLE = "RedisPool.minIdle";
private static final String REDISMAXIDLE = "RedisPool.maxIdle";
private static final String REDISMAXACTIVE = "RedisPool.maxActive";
private static final String REDISMAXWAIT = "RedisPool.maxWait";
private transient JedisPool pool;
public enum AddMode {
LST_RPUSH((byte)0),
SET_ADD((byte)1);
private byte value;
private AddMode(byte value) {
this.value = value;
}
public byte getValue() {
return value;
}
}
@Override
public void testStarted(String distributedHost) {
JedisPoolConfig config = new JedisPoolConfig();
config.setMaxTotal(getRedisMaxActive());
config.setMaxIdle(getRedisMaxIdle());
config.setMinIdle(getRedisMinIdle());
config.setMaxWaitMillis(getRedisMaxWait()*1000);
String host = getRedisHost();
int port = Protocol.DEFAULT_PORT;
if(!JOrphanUtils.isBlank(getRedisPort())) {
port = Integer.parseInt(getRedisPort());
}
int timeout = Protocol.DEFAULT_TIMEOUT;
if(!JOrphanUtils.isBlank(getRedisTimeout())) {
timeout = Integer.parseInt(getRedisTimeout());
}
int database = Protocol.DEFAULT_DATABASE;
if(!JOrphanUtils.isBlank(getRedisDatabase())) {
database = Integer.parseInt(getRedisDatabase());
}
String password = null;
if(!JOrphanUtils.isBlank(getRedisPasswd())) {
password = getRedisPasswd();
}
this.pool = new JedisPool(config, host, port, timeout, password, database);
//System.out.println("testStarted:" + this.pool + "this:" + this);
}
@Override
public void testEnded() {
testEnded("");
}
@Override
public void testEnded(String host) {
pool.destroy();
}
@Override
public void testStarted() {
testStarted("");
}
@Override
public Object clone() {
RedisDataWriter clonedElement = (RedisDataWriter)super.clone();
clonedElement.pool = this.pool;
return clonedElement;
}
@Override
public void process() {
// TODO Auto-generated method stub
JMeterContext context = getThreadContext();
SampleResult previousResult = context.getPreviousResult();
if (previousResult == null) {
return;
}
log.debug("RedisDataWriter processing result");
try {
String redisKey = getRedisKey();
String redisValue = getVariableNames();
Jedis connection = this.pool.getResource();
//System.out.println("testStarted:" + this.pool + "this:" + this);
try {
Enum<AddMode> mode = getAddMode();
if(mode.equals(AddMode.LST_RPUSH)) {
connection.rpush(redisKey, redisValue);
} else {
connection.sadd(redisKey, redisValue);
}
} finally {
if (connection != null) {
this.pool.returnResource(connection);
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
public void setRediskey(String redisKey) {
setProperty(REDISKEY, redisKey);
}
public String getRedisKey() {
return getPropertyAsString(REDISKEY);
}
public void setVariableNames(String variableNames) {
setProperty(VARIABLE_NAMES, variableNames);
}
public String getVariableNames() {
return getPropertyAsString(VARIABLE_NAMES);
}
public void setRedisHost(String host) {
setProperty(REDISHOST, host);
}
public String getRedisHost() {
return getPropertyAsString(REDISHOST);
}
public void setRedisPort(String port) {
setProperty(REDISPORT, port);
}
public String getRedisPort() {
String port = getPropertyAsString(REDISPORT);
if(JOrphanUtils.isBlank(port)) {
port = Integer.toString(Protocol.DEFAULT_PORT);
}
return port;
}
public void setRedisTimeout(String timeout) {
setProperty(REDISTIMEOUT, timeout);
}
public String getRedisTimeout() {
String timeout = getPropertyAsString(REDISTIMEOUT);
if (JOrphanUtils.isBlank(timeout)) {
timeout = Integer.toString(Protocol.DEFAULT_TIMEOUT);
}
return timeout;
}
public void setRedisPasswd(String password) {
setProperty(REDISPASSWD, password);
}
public String getRedisPasswd() {
return getPropertyAsString(REDISPASSWD, null);
}
public void setRedisDatabase(String db) {
setProperty(REDISDATABSE, db);
}
public String getRedisDatabase() {
String database = getPropertyAsString(REDISDATABSE);
if (JOrphanUtils.isBlank(database)) {
database = Integer.toString(Protocol.DEFAULT_DATABASE);
}
return database;
}
public void setRedisAddMode(String mode) {
for(Enum<AddMode> e : AddMode.values()) {
final String propName = e.toString();
//System.out.println("propName:" + propName + ", mode:" + mode + ", name:" + e.name());
if (mode.equals(propName)) {
final int tmpMode = e.ordinal();
if (log.isDebugEnabled()) {
log.debug("Converted " + "addMode=" + mode + " to mode=" + tmpMode);
}
super.setProperty(REDISADDMODE, e.toString());
return;
}
}
super.setProperty(REDISADDMODE, AddMode.LST_RPUSH.ordinal());
}
public String getRedisAddMode() {
return getPropertyAsString(REDISADDMODE, AddMode.LST_RPUSH.toString());
}
public Enum<AddMode> getAddMode() {
String mode = getRedisAddMode();
for(Enum<AddMode> e : AddMode.values()) {
final String propName = e.toString();
if (mode.equals(propName)) {
return e;
}
}
return AddMode.LST_RPUSH;
}
public void setRedisMinIdle(int minIdle) {
setProperty(REDISMINIDLE, minIdle);
}
public int getRedisMinIdle() {
return getPropertyAsInt(REDISMINIDLE, 0);
}
public void setRedisMaxIdle(int maxIdle) {
setProperty(REDISMAXIDLE, maxIdle);
}
public int getRedisMaxIdle() {
return getPropertyAsInt(REDISMAXIDLE, 10);
}
public void setRedisMaxActive(int maxActive) {
setProperty(REDISMAXACTIVE, maxActive);
}
public int getRedisMaxActive() {
return getPropertyAsInt(REDISMAXACTIVE, 500);
}
public void setRedisMaxWait(int maxWait) {
setProperty(REDISMAXWAIT, maxWait);
}
public int getRedisMaxWait() {
return getPropertyAsInt(REDISMAXWAIT, 30000);
}
}
看下介面，基本上，需要設定的地方，都可以用了。
① 位置：Sampler -> 後置處理器：
② Rdis資料錄入器：
優化介面
看了下程式碼，感覺還是有改進的可能性。起碼介面這塊，可以使用BenInfoSupport。
這裡把程式碼連線貼一下：
https://github.com/eyotang/jmeter-plugins
寫起來很簡單，介面出起來也很快，還支撐多語言。最主要的是，和RedisDataSet放到同一個jar包裡面。
測試效能
既然是效能測試使用，那外掛本身也需要有一個性能指標吧。
這裡使用本地的一個java程式產生的內容，使用Redis資料錄入器寫入單節點的redis中。
峰值：
QPS：62565.8 (6萬)，Redis的QPS已達到極限
CPU：33.2%, redis: 96%~97%, java：770%~780%
資料：18522114（1852萬）（記憶體：890M）java：1241M
外掛下載
為了讓大家能直接下載到該版本的外掛，這裡將jedis-2.81和jmeter-plugins-redis都放進去了。
下載地址：http://download.csdn.net/download/periodtang/10233148