OAL 直譯器實現

OAL 直譯器是基於 Antlr4 實現的,我們先來了解下 Antlr4

Antlr4 基本介紹

Antlr4 使用案例

參考Antlr4的使用簡介這篇文章,我們實現了一個簡單的案例:antlr案例:簡單的計算器,下面來講講這個案例。

首先,裝好ANTLR v4(IDEA外掛)外掛,這個之後驗證語法樹的時候會用到。

pom.xml 中配置 antlr4 的依賴和外掛

<dependency>
<groupId>org.antlr</groupId>
<artifactId>antlr4-runtime</artifactId>
<version>4.7.1</version>
</dependency>
<plugin>
<groupId>org.antlr</groupId>
<artifactId>antlr4-maven-plugin</artifactId>
<version>${antlr.version}</version>
<executions>
<execution>
<id>antlr</id>
<goals>
<goal>antlr4</goal>
</goals>
</execution>
</executions>
</plugin>

src/main/antlr4/com/switchvov/antlr/demo/calc 目錄下新增一個 Calc.g4 檔案

grammar Calc;   //名稱需要和檔名一致

root : expr EOF;   //解決問題: no viable alternative at input '<EOF>'

expr
: expr '+' expr #add //標籤會生成對應訪問方法方便我們實現呼叫邏輯編寫
| expr '-' expr #sub
| INT #int
; INT : [0-9]+ //定義整數
; WS : [ \r\n\t]+ -> skip //跳過空白類字元
;

執行一下: mvn compile -Dmaven.test.skip=true ,在 target/generated-sources/antlr4 會生成相應的 Java 程式碼。

使用方式預設是監聽器模式,也可以配置成訪問者模式。

監聽器模式:主要藉助了 ParseTreeWalker 這樣一個類,相當於是一個 hook ,每經過一個樹的節點,便會觸發對應節點的方法。好處就算是比較方便,但是靈活性不夠,不能夠自主性的呼叫任意節點進行使用。

訪問者模式:將每個資料的節點型別高度抽象出來夠,根據你傳入的上下文型別來判斷你想要訪問的是哪個節點,觸發對應的方法

PS:結論,簡單語法監聽器模式就可以了,如果語法比較靈活可以考慮使用訪問者模式。

antlr4
├── Calc.tokens
├── CalcLexer.tokens
└── com
└── switchvov
└── antlr
└── demo
└── calc
├── Calc.interp
├── CalcBaseListener.java # 監聽模式下生成的監聽器基類,實現類監聽器介面,通過繼承該類可以實現相應的功能
├── CalcLexer.interp
├── CalcLexer.java # 詞法解析器
├── CalcListener.java # 監聽模式下生成的監聽器介面
└── CalcParser.java # 語法解析器

繼承 com.switchvov.antlr.demo.calc.CalcBaseListener ,實現計算器相應功能

package com.switchvov.antlr.demo.calc;

import java.util.ArrayDeque;
import java.util.Deque; /**
* @author switch
* @since 2021/6/30
*/
public class CalcExecuteListener extends CalcBaseListener { Deque<Integer> queue = new ArrayDeque<>(16); @Override
public void exitInt(CalcParser.IntContext ctx) {
queue.add(Integer.parseInt(ctx.INT().getText()));
} @Override
public void exitAdd(CalcParser.AddContext ctx) {
int r = queue.pop();
int l = queue.pop();
queue.add(l + r);
} @Override
public void exitSub(CalcParser.SubContext ctx) {
int r = queue.pop();
int l = queue.pop();
queue.add(l - r);
} public int result() {
return queue.pop();
}
}

測試一下

package com.switchvov.antlr.demo.calc;

import org.antlr.v4.runtime.CharStreams;
import org.antlr.v4.runtime.CodePointCharStream;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.tree.ParseTree;
import org.antlr.v4.runtime.tree.ParseTreeWalker;
import org.junit.Test; /**
* @author switch
* @since 2021/6/30
*/
public class CalcTest { public static int exec(String input) {
// 讀入字串
CodePointCharStream cs = CharStreams.fromString(input);
// 詞法解析
CalcLexer lexer = new CalcLexer(cs);
CommonTokenStream tokens = new CommonTokenStream(lexer);
// 語法解析
CalcParser parser = new CalcParser(tokens); // 監聽器觸發獲取執行結果
ParseTree tree = parser.expr();
ParseTreeWalker walker = new ParseTreeWalker();
CalcExecuteListener listener = new CalcExecuteListener();
walker.walk(listener, tree);
return listener.result();
} @Test
public void testCalc() {
String input = "1+2";
// 輸出結果:3
System.out.println(exec(input));
} }

Antlr4 IDEA 外掛使用

Calc.g4 語法定義檔案中,滑鼠右擊可以選擇 Test Rule root ,然後在 ANTLR Preview 的輸入框中填入 1 + 2 就可以校驗語法檔案是否 OK ,並且也可以看到相應的語法樹

Antlr4 在 Skywalking 的應用

通過“ Antlr4 基本介紹”一節,基本上對 Antlr4 使用有了個大概的認識。下面來看看 Skywalking 中 Antlr4 是如何使用的。

詞法定義

oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALLexer.g4 檔案中,我們能看到 OAL 的詞法定義

// Observability Analysis Language lexer
lexer grammar OALLexer; @Header {package org.apache.skywalking.oal.rt.grammar;} // Keywords FROM: 'from';
FILTER: 'filter';
DISABLE: 'disable';
SRC_ALL: 'All';
SRC_SERVICE: 'Service';
SRC_SERVICE_INSTANCE: 'ServiceInstance';
SRC_ENDPOINT: 'Endpoint';
SRC_SERVICE_RELATION: 'ServiceRelation';
SRC_SERVICE_INSTANCE_RELATION: 'ServiceInstanceRelation';
SRC_ENDPOINT_RELATION: 'EndpointRelation';
SRC_SERVICE_INSTANCE_JVM_CPU: 'ServiceInstanceJVMCPU';
SRC_SERVICE_INSTANCE_JVM_MEMORY: 'ServiceInstanceJVMMemory';
SRC_SERVICE_INSTANCE_JVM_MEMORY_POOL: 'ServiceInstanceJVMMemoryPool';
SRC_SERVICE_INSTANCE_JVM_GC: 'ServiceInstanceJVMGC';
SRC_SERVICE_INSTANCE_JVM_THREAD: 'ServiceInstanceJVMThread';
SRC_SERVICE_INSTANCE_JVM_CLASS:'ServiceInstanceJVMClass';
SRC_DATABASE_ACCESS: 'DatabaseAccess';
SRC_SERVICE_INSTANCE_CLR_CPU: 'ServiceInstanceCLRCPU';
SRC_SERVICE_INSTANCE_CLR_GC: 'ServiceInstanceCLRGC';
SRC_SERVICE_INSTANCE_CLR_THREAD: 'ServiceInstanceCLRThread';
SRC_ENVOY_INSTANCE_METRIC: 'EnvoyInstanceMetric'; // Browser keywords
SRC_BROWSER_APP_PERF: 'BrowserAppPerf';
SRC_BROWSER_APP_PAGE_PERF: 'BrowserAppPagePerf';
SRC_BROWSER_APP_SINGLE_VERSION_PERF: 'BrowserAppSingleVersionPerf';
SRC_BROWSER_APP_TRAFFIC: 'BrowserAppTraffic';
SRC_BROWSER_APP_PAGE_TRAFFIC: 'BrowserAppPageTraffic';
SRC_BROWSER_APP_SINGLE_VERSION_TRAFFIC: 'BrowserAppSingleVersionTraffic'; // Constructors symbols DOT: '.';
LR_BRACKET: '(';
RR_BRACKET: ')';
LS_BRACKET: '[';
RS_BRACKET: ']';
COMMA: ',';
SEMI: ';';
EQUAL: '=';
DUALEQUALS: '==';
ALL: '*';
GREATER: '>';
LESS: '<';
GREATER_EQUAL: '>=';
LESS_EQUAL: '<=';
NOT_EQUAL: '!=';
LIKE: 'like';
IN: 'in';
CONTAIN: 'contain';
NOT_CONTAIN: 'not contain'; // Literals BOOL_LITERAL: 'true'
| 'false'
; NUMBER_LITERAL : Digits+; CHAR_LITERAL: '\'' (~['\\\r\n] | EscapeSequence) '\''; STRING_LITERAL: '"' (~["\\\r\n] | EscapeSequence)* '"'; DelimitedComment
: '/*' ( DelimitedComment | . )*? '*/'
-> channel(HIDDEN)
; LineComment
: '//' ~[\u000A\u000D]*
-> channel(HIDDEN)
; SPACE: [ \t\r\n]+ -> channel(HIDDEN); // Identifiers IDENTIFIER: Letter LetterOrDigit*; // Fragment rules fragment EscapeSequence
: '\\' [btnfr"'\\]
| '\\' ([0-3]? [0-7])? [0-7]
| '\\' 'u'+ HexDigit HexDigit HexDigit HexDigit
; fragment HexDigits
: HexDigit ((HexDigit | '_')* HexDigit)?
; fragment HexDigit
: [0-9a-fA-F]
; fragment Digits
: [0-9] ([0-9_]* [0-9])?
; fragment LetterOrDigit
: Letter
| [0-9]
; fragment Letter
: [a-zA-Z$_] // these are the "java letters" below 0x7F
| ~[\u0000-\u007F\uD800-\uDBFF] // covers all characters above 0x7F which are not a surrogate
| [\uD800-\uDBFF] [\uDC00-\uDFFF] // covers UTF-16 surrogate pairs encodings for U+10000 to U+10FFFF
;

語法定義

oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALParser.g4 檔案中,我們能看到 OAL 的語法定義

parser grammar OALParser;

@Header {package org.apache.skywalking.oal.rt.grammar;}

options { tokenVocab=OALLexer; }

// Top Level Description

root
: (aggregationStatement | disableStatement)*
; aggregationStatement
: variable (SPACE)? EQUAL (SPACE)? metricStatement DelimitedComment? LineComment? (SEMI|EOF)
; disableStatement
: DISABLE LR_BRACKET disableSource RR_BRACKET DelimitedComment? LineComment? (SEMI|EOF)
; metricStatement
: FROM LR_BRACKET source (sourceAttributeStmt+) RR_BRACKET (filterStatement+)? DOT aggregateFunction
; filterStatement
: DOT FILTER LR_BRACKET filterExpression RR_BRACKET
; filterExpression
: expression
; source
: SRC_ALL | SRC_SERVICE | SRC_DATABASE_ACCESS | SRC_SERVICE_INSTANCE | SRC_ENDPOINT |
SRC_SERVICE_RELATION | SRC_SERVICE_INSTANCE_RELATION | SRC_ENDPOINT_RELATION |
SRC_SERVICE_INSTANCE_JVM_CPU | SRC_SERVICE_INSTANCE_JVM_MEMORY | SRC_SERVICE_INSTANCE_JVM_MEMORY_POOL | SRC_SERVICE_INSTANCE_JVM_GC | SRC_SERVICE_INSTANCE_JVM_THREAD | SRC_SERVICE_INSTANCE_JVM_CLASS |// JVM source of service instance
SRC_SERVICE_INSTANCE_CLR_CPU | SRC_SERVICE_INSTANCE_CLR_GC | SRC_SERVICE_INSTANCE_CLR_THREAD |
SRC_ENVOY_INSTANCE_METRIC |
SRC_BROWSER_APP_PERF | SRC_BROWSER_APP_PAGE_PERF | SRC_BROWSER_APP_SINGLE_VERSION_PERF |
SRC_BROWSER_APP_TRAFFIC | SRC_BROWSER_APP_PAGE_TRAFFIC | SRC_BROWSER_APP_SINGLE_VERSION_TRAFFIC
; disableSource
: IDENTIFIER
; sourceAttributeStmt
: DOT sourceAttribute
; sourceAttribute
: IDENTIFIER | ALL
; variable
: IDENTIFIER
; aggregateFunction
: functionName LR_BRACKET ((funcParamExpression (COMMA funcParamExpression)?) | (literalExpression (COMMA literalExpression)?))? RR_BRACKET
; functionName
: IDENTIFIER
; funcParamExpression
: expression
; literalExpression
: BOOL_LITERAL | NUMBER_LITERAL | IDENTIFIER
; expression
: booleanMatch | stringMatch | greaterMatch | lessMatch | greaterEqualMatch | lessEqualMatch | notEqualMatch | booleanNotEqualMatch | likeMatch | inMatch | containMatch | notContainMatch
; containMatch
: conditionAttributeStmt CONTAIN stringConditionValue
; notContainMatch
: conditionAttributeStmt NOT_CONTAIN stringConditionValue
; booleanMatch
: conditionAttributeStmt DUALEQUALS booleanConditionValue
; stringMatch
: conditionAttributeStmt DUALEQUALS (stringConditionValue | enumConditionValue)
; greaterMatch
: conditionAttributeStmt GREATER numberConditionValue
; lessMatch
: conditionAttributeStmt LESS numberConditionValue
; greaterEqualMatch
: conditionAttributeStmt GREATER_EQUAL numberConditionValue
; lessEqualMatch
: conditionAttributeStmt LESS_EQUAL numberConditionValue
; booleanNotEqualMatch
: conditionAttributeStmt NOT_EQUAL booleanConditionValue
; notEqualMatch
: conditionAttributeStmt NOT_EQUAL (numberConditionValue | stringConditionValue | enumConditionValue)
; likeMatch
: conditionAttributeStmt LIKE stringConditionValue
; inMatch
: conditionAttributeStmt IN multiConditionValue
; multiConditionValue
: LS_BRACKET (numberConditionValue ((COMMA numberConditionValue)*) | stringConditionValue ((COMMA stringConditionValue)*) | enumConditionValue ((COMMA enumConditionValue)*)) RS_BRACKET
; conditionAttributeStmt
: conditionAttribute ((DOT conditionAttribute)*)
; conditionAttribute
: IDENTIFIER
; booleanConditionValue
: BOOL_LITERAL
; stringConditionValue
: STRING_LITERAL
; enumConditionValue
: IDENTIFIER DOT IDENTIFIER
; numberConditionValue
: NUMBER_LITERAL
;

Antlr4 生成 Java 程式碼

oap-server/oal-grammar 下執行 mvn compile -Dmaven.test.skip=true 會在 oap-server/oal-grammar/target/generated-sources/antlr4 目錄下生成相應的 Java 程式碼

.
├── OALLexer.tokens
├── OALParser.tokens
└── org
└── apache
└── skywalking
└── oal
└── rt
└── grammar
├── OALLexer.interp
├── OALLexer.java # 詞法解析器
├── OALParser.interp
├── OALParser.java # 語法解析器
├── OALParserBaseListener.java # 監聽器
└── OALParserListener.java

Skywalking 的使用

通過“ Antlr4 使用案例”一節,可以知道 Antlr4 有兩種功能實現方式:監聽器或者訪問器。

通過“ Antlr4 生成 Java 程式碼”一節,知道 Skywalking 使用的是監聽器模式。

Skywalking 關於 OAL 的相應的程式碼都在 oap-server/oal-rt 模組中。

org.apache.skywalking.oal.rt.grammar.OALParserBaseListener 的繼承類座標是 org.apache.skywalking.oal.rt.parser.OALListener

package org.apache.skywalking.oal.rt.parser;

import java.util.Arrays;
import java.util.List;
import org.antlr.v4.runtime.misc.NotNull;
import org.apache.skywalking.oal.rt.grammar.OALParser;
import org.apache.skywalking.oal.rt.grammar.OALParserBaseListener;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine; public class OALListener extends OALParserBaseListener {
private List<AnalysisResult> results;
private AnalysisResult current;
private DisableCollection collection; private ConditionExpression conditionExpression; private final String sourcePackage; public OALListener(OALScripts scripts, String sourcePackage) {
this.results = scripts.getMetricsStmts();
this.collection = scripts.getDisableCollection();
this.sourcePackage = sourcePackage;
} @Override
public void enterAggregationStatement(@NotNull OALParser.AggregationStatementContext ctx) {
current = new AnalysisResult();
} @Override
public void exitAggregationStatement(@NotNull OALParser.AggregationStatementContext ctx) {
DeepAnalysis deepAnalysis = new DeepAnalysis();
results.add(deepAnalysis.analysis(current));
current = null;
} @Override
public void enterSource(OALParser.SourceContext ctx) {
current.setSourceName(ctx.getText());
current.setSourceScopeId(DefaultScopeDefine.valueOf(metricsNameFormat(ctx.getText())));
} @Override
public void enterSourceAttribute(OALParser.SourceAttributeContext ctx) {
current.getSourceAttribute().add(ctx.getText());
} @Override
public void enterVariable(OALParser.VariableContext ctx) {
} @Override
public void exitVariable(OALParser.VariableContext ctx) {
current.setVarName(ctx.getText());
current.setMetricsName(metricsNameFormat(ctx.getText()));
current.setTableName(ctx.getText().toLowerCase());
} @Override
public void enterFunctionName(OALParser.FunctionNameContext ctx) {
current.setAggregationFunctionName(ctx.getText());
} @Override
public void enterFilterStatement(OALParser.FilterStatementContext ctx) {
conditionExpression = new ConditionExpression();
} @Override
public void exitFilterStatement(OALParser.FilterStatementContext ctx) {
current.addFilterExpressionsParserResult(conditionExpression);
conditionExpression = null;
} @Override
public void enterFuncParamExpression(OALParser.FuncParamExpressionContext ctx) {
conditionExpression = new ConditionExpression();
} @Override
public void exitFuncParamExpression(OALParser.FuncParamExpressionContext ctx) {
current.addFuncConditionExpression(conditionExpression);
conditionExpression = null;
} /////////////
// Expression
////////////
@Override
public void enterConditionAttribute(OALParser.ConditionAttributeContext ctx) {
conditionExpression.getAttributes().add(ctx.getText());
} @Override
public void enterBooleanMatch(OALParser.BooleanMatchContext ctx) {
conditionExpression.setExpressionType("booleanMatch");
} @Override
public void enterStringMatch(OALParser.StringMatchContext ctx) {
conditionExpression.setExpressionType("stringMatch");
} @Override
public void enterGreaterMatch(OALParser.GreaterMatchContext ctx) {
conditionExpression.setExpressionType("greaterMatch");
} @Override
public void enterGreaterEqualMatch(OALParser.GreaterEqualMatchContext ctx) {
conditionExpression.setExpressionType("greaterEqualMatch");
} @Override
public void enterLessMatch(OALParser.LessMatchContext ctx) {
conditionExpression.setExpressionType("lessMatch");
} @Override
public void enterLessEqualMatch(OALParser.LessEqualMatchContext ctx) {
conditionExpression.setExpressionType("lessEqualMatch");
} @Override
public void enterNotEqualMatch(final OALParser.NotEqualMatchContext ctx) {
conditionExpression.setExpressionType("notEqualMatch");
} @Override
public void enterBooleanNotEqualMatch(final OALParser.BooleanNotEqualMatchContext ctx) {
conditionExpression.setExpressionType("booleanNotEqualMatch");
} @Override
public void enterLikeMatch(final OALParser.LikeMatchContext ctx) {
conditionExpression.setExpressionType("likeMatch");
} @Override
public void enterContainMatch(final OALParser.ContainMatchContext ctx) {
conditionExpression.setExpressionType("containMatch");
} @Override
public void enterNotContainMatch(final OALParser.NotContainMatchContext ctx) {
conditionExpression.setExpressionType("notContainMatch");
} @Override
public void enterInMatch(final OALParser.InMatchContext ctx) {
conditionExpression.setExpressionType("inMatch");
} @Override
public void enterMultiConditionValue(final OALParser.MultiConditionValueContext ctx) {
conditionExpression.enterMultiConditionValue();
} @Override
public void exitMultiConditionValue(final OALParser.MultiConditionValueContext ctx) {
conditionExpression.exitMultiConditionValue();
} @Override
public void enterBooleanConditionValue(OALParser.BooleanConditionValueContext ctx) {
enterConditionValue(ctx.getText());
} @Override
public void enterStringConditionValue(OALParser.StringConditionValueContext ctx) {
enterConditionValue(ctx.getText());
} @Override
public void enterEnumConditionValue(OALParser.EnumConditionValueContext ctx) {
enterConditionValue(ctx.getText());
} @Override
public void enterNumberConditionValue(OALParser.NumberConditionValueContext ctx) {
conditionExpression.isNumber();
enterConditionValue(ctx.getText());
} private void enterConditionValue(String value) {
if (value.split("\\.").length == 2 && !value.startsWith("\"")) {
// Value is an enum.
value = sourcePackage + value;
}
conditionExpression.addValue(value);
} /////////////
// Expression end.
//////////// @Override
public void enterLiteralExpression(OALParser.LiteralExpressionContext ctx) {
if (ctx.IDENTIFIER() == null) {
current.addFuncArg(new Argument(EntryMethod.LITERAL_TYPE, Arrays.asList(ctx.getText())));
return;
}
current.addFuncArg(new Argument(EntryMethod.IDENTIFIER_TYPE, Arrays.asList(ctx.getText().split("\\."))));
} private String metricsNameFormat(String source) {
source = firstLetterUpper(source);
int idx;
while ((idx = source.indexOf("_")) > -1) {
source = source.substring(0, idx) + firstLetterUpper(source.substring(idx + 1));
}
return source;
} /**
* Disable source
*/
@Override
public void enterDisableSource(OALParser.DisableSourceContext ctx) {
collection.add(ctx.getText());
} private String firstLetterUpper(String source) {
return source.substring(0, 1).toUpperCase() + source.substring(1);
}
}

簡單來說,就是通過監聽器封裝了個 org.apache.skywalking.oal.rt.parser.OALScripts 物件

package org.apache.skywalking.oal.rt.parser;

import java.util.LinkedList;
import java.util.List;
import lombok.Getter; @Getter
public class OALScripts {
// 解析出來的分析結果集合
private List<AnalysisResult> metricsStmts;
// 禁用表示式集合
private DisableCollection disableCollection; public OALScripts() {
metricsStmts = new LinkedList<>();
disableCollection = new DisableCollection();
}
}

org.apache.skywalking.oal.rt.parser.ScriptParser 類讀取 oal 檔案,使用 Antlr 生成的 Java 類進行解析

package org.apache.skywalking.oal.rt.parser;

import java.io.IOException;
import java.io.Reader;
import org.antlr.v4.runtime.CharStreams;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.tree.ParseTree;
import org.antlr.v4.runtime.tree.ParseTreeWalker;
import org.apache.skywalking.oal.rt.grammar.OALLexer;
import org.apache.skywalking.oal.rt.grammar.OALParser; /**
* Script reader and parser.
*/
public class ScriptParser {
private OALLexer lexer; private String sourcePackage; private ScriptParser() { } public static ScriptParser createFromFile(Reader scriptReader, String sourcePackage) throws IOException {
ScriptParser parser = new ScriptParser();
parser.lexer = new OALLexer(CharStreams.fromReader(scriptReader));
parser.sourcePackage = sourcePackage;
return parser;
} public static ScriptParser createFromScriptText(String script, String sourcePackage) throws IOException {
ScriptParser parser = new ScriptParser();
parser.lexer = new OALLexer(CharStreams.fromString(script));
parser.sourcePackage = sourcePackage;
return parser;
} public OALScripts parse() throws IOException {
OALScripts scripts = new OALScripts(); CommonTokenStream tokens = new CommonTokenStream(lexer); OALParser parser = new OALParser(tokens); ParseTree tree = parser.root();
ParseTreeWalker walker = new ParseTreeWalker(); walker.walk(new OALListener(scripts, sourcePackage), tree); return scripts;
} public void close() {
}
}

參考文件

分享並記錄所學所見