1. 程式人生 > >SLS機器學習最佳實戰:日誌聚類+異常告警

SLS機器學習最佳實戰:日誌聚類+異常告警

摘要: 圍繞日誌,挖掘其中更大價值,一直是我們團隊所關注。在原有日誌實時查詢基礎上,今年SLS在DevOps領域完善了如下功能: - 上下文查詢 - 實時Tail和智慧聚類,以提高問題調查效率 - 提供多種時序資料的異常檢測和預測函式,來做更智慧的檢查和預測 - 資料分析的結果視覺化 - 強大的告...

0.文章系列連結



1.手中的錘子都有啥?

圍繞日誌,挖掘其中更大價值,一直是我們團隊所關注。在原有日誌實時查詢基礎上,今年SLS在DevOps領域完善了如下功能:

  • 上下文查詢
  • 實時Tail和智慧聚類,以提高問題調查效率
  • 提供多種時序資料的異常檢測和預測函式,來做更智慧的檢查和預測
  • 資料分析的結果視覺化
  • 強大的告警設定和通知,通過呼叫webhook進行關聯行動

今天我們重點介紹下,日誌只能聚類和異常告警如何配合,更好的進行異常發現和告警

2.平臺實驗

2.1 實驗資料

一份Sys Log的原始資料,,並且開啟了日誌聚類服務,具體的狀態截圖如下:

通過調整下面截圖中紅色框1的大小,可以改變圖中紅色框2的結果,但是對於每個最細粒度的pattern並不會改變,也就是說:子Pattern的結果是穩定且唯一的,我們可以通過子Pattern的Signature找到對應的原始日誌條目。

2.2 生成子模式的時序資訊

假設,我們對這個子Pattern要進行監控:

msg:vm-111932.tc su: pam_unix(*:session): session closed for user root
對應的 signature_id : __log_signature__: 1814836459146662485

我們得到了上述pattern對應的原始日誌,可以看下具體的數量在時間軸上的直返圖:

上圖中,我們可以發現,這個模式的日誌分佈不是很均衡,其中還有一些是沒有的,如果直接按照時間視窗統計數量,得到的時序圖如下:

__log_signature__: 1814836459146662485 |  
select 
    date_trunc('minute', __time__) as time, 
    COUNT(*) as num 
from log GROUP BY time order by time ASC limit 10000

上述圖中我們發現時間上並不是連續的。因此,我們需要對這條時序進行補點操作。

__log_signature__: 1814836459146662485 | 
select 
    time_series(time, '1m', '%Y-%m-%d %H:%i:%s', '0') as time, 
    avg(num) as num 
from  ( 
    select 
        __time__ - __time__ % 60 as time, 
        COUNT(*) as num 
    from log GROUP BY time order by time desc ) 
GROUP by time order by time ASC limit 10000

2.3 對時序進行異常檢測

使用時序異常檢測函式: ts_predicate_arma

__log_signature__: 1814836459146662485 | 
select 
    ts_predicate_arma(to_unixtime(time), num, 5, 1, 1, 1, 'avg') 
from  ( 
    select 
        time_series(time, '1m', '%Y-%m-%d %H:%i:%s', '0') as time, 
        avg(num) as num 
    from  ( 
        select 
            __time__ - __time__ % 60 as time, 
            COUNT(*) as num 
        from log GROUP BY time order by time desc ) 
    GROUP by time order by time ASC ) limit 10000

2.4 告警該如何設定

  • 將機器學習函式的結果拆解開
__log_signature__: 1814836459146662485 | 
select 
    t1[1] as unixtime, t1[2] as src, t1[3] as pred, t1[4] as up, t1[5] as lower, t1[6] as prob 
from  ( 
    select 
        ts_predicate_arma(to_unixtime(time), num, 5, 1, 1, 1, 'avg') as res 
    from  ( 
        select 
            time_series(time, '1m', '%Y-%m-%d %H:%i:%s', '0') as time, 
            avg(num) as num 
        from  ( 
            select 
                __time__ - __time__ % 60 as time, 
                COUNT(*) as num 
            from log GROUP BY time order by time desc ) 
        GROUP by time order by time ASC )) , unnest(res) as t(t1)

  • 針對最近兩分鐘的結果進行告警
__log_signature__: 1814836459146662485 | 
select 
    unixtime, src, pred, up, lower, prob 
from  ( 
    select 
        t1[1] as unixtime, t1[2] as src, t1[3] as pred, t1[4] as up, t1[5] as lower, t1[6] as prob 
    from  ( 
        select 
            ts_predicate_arma(to_unixtime(time), num, 5, 1, 1, 1, 'avg') as res 
        from  ( 
            select 
                time_series(time, '1m', '%Y-%m-%d %H:%i:%s', '0') as time, 
                avg(num) as num 
            from  ( 
                select 
                    __time__ - __time__ % 60 as time, COUNT(*) as num 
                from log GROUP BY time order by time desc ) 
            GROUP by time order by time ASC )) , unnest(res) as t(t1) ) 
    where is_nan(src) = false order by unixtime desc limit 2

  • 針對上升點進行告警,並設定兜底策略
__log_signature__: 1814836459146662485 | 
select 
    sum(prob) as sumProb, max(src) as srcMax, max(up) as upMax 
from ( 
    select 
        unixtime, src, pred, up, lower, prob 
    from  ( 
        select 
            t1[1] as unixtime, t1[2] as src, t1[3] as pred, t1[4] as up, t1[5] as lower, t1[6] as prob 
        from  ( 
            select 
                ts_predicate_arma(to_unixtime(time), num, 5, 1, 1, 1, 'avg') as res 
            from  ( 
                select 
                    time_series(time, '1m', '%Y-%m-%d %H:%i:%s', '0') as time, avg(num) as num 
                from  ( 
                    select 
                        __time__ - __time__ % 60 as time, COUNT(*) as num 
                    from log GROUP BY time order by time desc ) 
                GROUP by time order by time ASC )) , unnest(res) as t(t1) ) 
        where is_nan(src) = false order by unixtime desc limit 2 )

具體的告警設定如下:


3.硬廣時間

3.1 日誌進階

這裡是日誌服務的各種功能的演示 日誌服務整體介紹,各種Demo

更多日誌進階內容可以參考:日誌服務學習路徑

作者:悟冥

原文連結

本文為雲棲社群原創內容,未經