sentinel控制臺監控數據持久化【MySQL】
根據官方wiki文檔,sentinel控制臺的實時監控數據,默認僅存儲 5 分鐘以內的數據。如需持久化,需要定制實現相關接口。
https://github.com/alibaba/Sentinel/wiki/在生產環境中使用-Sentinel-控制臺 也給出了指導步驟:
1.自行擴展實現 MetricsRepository 接口;
2.註冊成 Spring Bean 並在相應位置通過 @Qualifier 註解指定對應的 bean name 即可。
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
首先看接口定義:
repository.metric包下的MetricsRepository<T>接口
該接口定義了4個方法,分別用於保存和查詢sentinel的metric數據。註釋其實很清楚了,這裏簡單過一下:
save:保存單個metric
saveAll:保存多個metric
queryByAppAndResourceBetween:通過應用名名稱、資源名稱、開始時間、結束時間查詢metric列表
listResourcesOfApp:通過應用名稱查詢資源列表
註:發現跟接口定義跟Spring Data JPA用法很像,即某個實體類Xxx對應一個XxxRepository,方法的命令也很規範,save、queryBy...
結合控制臺【實時監控】菜單的界面,大概能猜到列表頁面的查詢流程:
菜單屬於某一個應用,這裏應用名稱是sentinel-dashborad;
先通過應用名稱查詢應用下所有的資源,圖中看到有2個,資源名稱分別是/resource/machineResource.json、/flow/rules.json;// listResourcesOfApp方法
再通過應用名稱、資源名稱、時間等查詢metric列表用於呈現統計圖表;// queryByAppAndResourceBetween方法
在MetricsRepository類名上Ctrl+H查看類繼承關系(Type Hiberarchy):
默認提供了一個用內存存儲的實現類:InMemoryMetricsRepository
在MetricsRepository類的各個方法上,通過Ctrl+Alt+H 查看方法調用關系(Call Hierarchy) :
可以看到,MetricsRepository接口的
save方法被它的實現類InMemoryMetricsRepository的saveAll調用,再往上走被MetricFetcher調用,用於保存metric數據;
queryByAppAndResourceBetween、listResourcesOfApp被MetricController調用,用於查詢metric數據;
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
OK,以上初步梳理了MetricsRepository接口的方法和流程,接下來我們使用MySQL數據庫,實現一個MetricsRepository接口。
首先,參考MetricEntity類設計一張表sentinel_metric來存儲監控的metric數據,表ddl如下:
-- 創建監控數據表 CREATE TABLE `sentinel_metric1` ( `id` INT NOT NULL AUTO_INCREMENT COMMENT ‘id,主鍵‘, `gmt_create` DATETIME COMMENT ‘創建時間‘, `gmt_modified` DATETIME COMMENT ‘修改時間‘, `app` VARCHAR(100) COMMENT ‘應用名稱‘, `timestamp` DATETIME COMMENT ‘統計時間‘, `resource` VARCHAR(500) COMMENT ‘資源名稱‘, `pass_qps` INT COMMENT ‘通過qps‘, `success_qps` INT COMMENT ‘成功qps‘, `block_qps` INT COMMENT ‘限流qps‘, `exception_qps` INT COMMENT ‘發送異常的次數‘, `rt` DOUBLE COMMENT ‘所有successQps的rt的和‘, `_count` INT COMMENT ‘本次聚合的總條數‘, `resource_code` INT COMMENT ‘資源的hashCode‘, INDEX app_idx(`app`) USING BTREE, INDEX resource_idx(`resource`) USING BTREE, INDEX timestamp_idx(`timestamp`) USING BTREE, PRIMARY KEY (`id`) ) ENGINE=INNODB DEFAULT CHARSET=utf8;
註:app、resource、timestamp在查詢語句的where條件中用到,因此給它們建立索引提高查詢速度;
count是MySQL的關鍵字,因此加上_前綴。
持久層選用Spring Data JPA框架,在pom中引入starter依賴:
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-jpa</artifactId> <version>${spring.boot.version}</version> </dependency>
在datasource.entity包下,新建jpa包,下面新建sentinel_metric表對應的實體類MetricPO:
package com.taobao.csp.sentinel.dashboard.datasource.entity.jpa; import javax.persistence.*; import java.io.Serializable; import java.util.Date; /** * @author cdfive * @date 2018-09-14 */ @Entity @Table(name = "sentinel_metric") public class MetricPO implements Serializable { private static final long serialVersionUID = 7200023615444172715L; /**id,主鍵*/ @Id @GeneratedValue @Column(name = "id") private Long id; /**創建時間*/ @Column(name = "gmt_create") private Date gmtCreate; /**修改時間*/ @Column(name = "gmt_modified") private Date gmtModified; /**應用名稱*/ @Column(name = "app") private String app; /**統計時間*/ @Column(name = "timestamp") private Date timestamp; /**資源名稱*/ @Column(name = "resource") private String resource; /**通過qps*/ @Column(name = "pass_qps") private Long passQps; /**成功qps*/ @Column(name = "success_qps") private Long successQps; /**限流qps*/ @Column(name = "block_qps") private Long blockQps; /**發送異常的次數*/ @Column(name = "exception_qps") private Long exceptionQps; /**所有successQps的rt的和*/ @Column(name = "rt") private Double rt; /**本次聚合的總條數*/ @Column(name = "_count") private Integer count; /**資源的hashCode*/ @Column(name = "resource_code") private Integer resourceCode; // getter setter省略 }
該類也是參考MetricEntity創建,加上了JPA的註解,比如@Table指定表名,@Entity標識為實體,@Id、@GeneratedValue設置id字段為自增主鍵等;
在resources目錄下的application.properties文件中,增加數據源和JPA(hibernate)的配置:
# datasource spring.datasource.driver-class-name=com.mysql.jdbc.Driver spring.datasource.url=${spring.datasource.url} spring.datasource.username=${spring.datasource.username} spring.datasource.password=${spring.datasource.password} # spring data jpa spring.jpa.hibernate.ddl-auto=none spring.jpa.hibernate.use-new-id-generator-mappings=false spring.jpa.database-platform=org.hibernate.dialect.MySQLDialect spring.jpa.show-sql=false
這裏數據庫連接(url)、用戶名(username)、密碼(password)用${xxx}占位符,這樣可以通過maven的pom.xml添加profile配置不同環境(開發、測試、生產) 或 從配置中心讀取參數。
接著在InMemoryMetricsRepository所在的repository.metric包下新建JpaMetricsRepository類,實現MetricsRepository<MetricEntity>接口:
package com.taobao.csp.sentinel.dashboard.repository.metric; import com.alibaba.csp.sentinel.util.StringUtil; import com.taobao.csp.sentinel.dashboard.datasource.entity.MetricEntity; import com.taobao.csp.sentinel.dashboard.datasource.entity.jpa.MetricPO; import org.springframework.beans.BeanUtils; import org.springframework.stereotype.Repository; import org.springframework.transaction.annotation.Transactional; import org.springframework.util.CollectionUtils; import javax.persistence.EntityManager; import javax.persistence.PersistenceContext; import javax.persistence.Query; import java.time.Instant; import java.util.*; import java.util.stream.Collectors; /** * @author cdfive * @date 2018-09-17 */ @Transactional @Repository("jpaMetricsRepository") public class JpaMetricsRepository implements MetricsRepository<MetricEntity> { @PersistenceContext private EntityManager em; @Override public void save(MetricEntity metric) { if (metric == null || StringUtil.isBlank(metric.getApp())) { return; } MetricPO metricPO = new MetricPO(); BeanUtils.copyProperties(metric, metricPO); em.persist(metricPO); } @Override public void saveAll(Iterable<MetricEntity> metrics) { if (metrics == null) { return; } metrics.forEach(this::save); } @Override public List<MetricEntity> queryByAppAndResourceBetween(String app, String resource, long startTime, long endTime) { List<MetricEntity> results = new ArrayList<MetricEntity>(); if (StringUtil.isBlank(app)) { return results; } if (StringUtil.isBlank(resource)) { return results; } StringBuilder hql = new StringBuilder(); hql.append("FROM MetricPO"); hql.append(" WHERE app=:app"); hql.append(" AND resource=:resource"); hql.append(" AND timestamp>=:startTime"); hql.append(" AND timestamp<=:endTime"); Query query = em.createQuery(hql.toString()); query.setParameter("app", app); query.setParameter("resource", resource); query.setParameter("startTime", Date.from(Instant.ofEpochMilli(startTime))); query.setParameter("endTime", Date.from(Instant.ofEpochMilli(endTime))); List<MetricPO> metricPOs = query.getResultList(); if (CollectionUtils.isEmpty(metricPOs)) { return results; } for (MetricPO metricPO : metricPOs) { MetricEntity metricEntity = new MetricEntity(); BeanUtils.copyProperties(metricPO, metricEntity); results.add(metricEntity); } return results; } @Override public List<String> listResourcesOfApp(String app) { List<String> results = new ArrayList<>(); if (StringUtil.isBlank(app)) { return results; } StringBuilder hql = new StringBuilder(); hql.append("FROM MetricPO"); hql.append(" WHERE app=:app"); hql.append(" AND timestamp>=:startTime"); long startTime = System.currentTimeMillis() - 1000 * 60; Query query = em.createQuery(hql.toString()); query.setParameter("app", app); query.setParameter("startTime", Date.from(Instant.ofEpochMilli(startTime))); List<MetricPO> metricPOs = query.getResultList(); if (CollectionUtils.isEmpty(metricPOs)) { return results; } List<MetricEntity> metricEntities = new ArrayList<MetricEntity>(); for (MetricPO metricPO : metricPOs) { MetricEntity metricEntity = new MetricEntity(); BeanUtils.copyProperties(metricPO, metricEntity); metricEntities.add(metricEntity); } Map<String, MetricEntity> resourceCount = new HashMap<>(32); for (MetricEntity metricEntity : metricEntities) { String resource = metricEntity.getResource(); if (resourceCount.containsKey(resource)) { MetricEntity oldEntity = resourceCount.get(resource); oldEntity.addPassQps(metricEntity.getPassQps()); oldEntity.addRtAndSuccessQps(metricEntity.getRt(), metricEntity.getSuccessQps()); oldEntity.addBlockQps(metricEntity.getBlockQps()); oldEntity.addExceptionQps(metricEntity.getExceptionQps()); oldEntity.addCount(1); } else { resourceCount.put(resource, MetricEntity.copyOf(metricEntity)); } } // Order by last minute b_qps DESC. return resourceCount.entrySet() .stream() .sorted((o1, o2) -> { MetricEntity e1 = o1.getValue(); MetricEntity e2 = o2.getValue(); int t = e2.getBlockQps().compareTo(e1.getBlockQps()); if (t != 0) { return t; } return e2.getPassQps().compareTo(e1.getPassQps()); }) .map(Map.Entry::getKey) .collect(Collectors.toList()); } }
參考InMemoryMetricsRepository類來實現,將其中用map存儲和查詢的部分改為用JPA實現:
save方法,將MetricEntity轉換為MetricPO類,調用EntityManager類的persist方法即可;
saveAll方法,循環調用save;
queryByAppAndResourceBetween、listResourcesOfApp編寫查詢即可。
最後一步,在MetricsRepository、MetricController類註入的metricStore屬性,加上@Qualifier("jpaMetricsRepository")註解:
@Qualifier("jpaMetricsRepository") @Autowired private MetricsRepository<MetricEntity> metricStore;
至此,監控數據MySQL持久化就完成了,得益於sentinel良好的Repository接口設計,是不是很簡單:)
來驗證下成果:
設置sentinel-dashboard工程啟動參數:-Dserver.port=8080 -Dcsp.sentinel.dashboard.server=localhost:8080 -Dproject.name=sentinel-dashboard
啟動工程,打開http://localhost:8080,查看不同的頁面均顯示正常,執行sql查詢sentinel_metric表已有數據。
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
思考:
個人感覺sentinel控制臺默認的實現類InMemoryMetricsRepository挺贊的,雖然內存存儲重啟會清空數據,如果沒有對歷史數據查詢的需求應用於生產環境是沒問題的;
其中如何用內存存儲,包括保存、查詢以及排序等代碼都值得學習。
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
參考:
https://github.com/alibaba/Sentinel/wiki/控制臺
https://github.com/alibaba/Sentinel/wiki/在生產環境中使用-Sentinel-控制臺
sentinel控制臺監控數據持久化【MySQL】