[4] Hive3.x SemanticAnalyzer and CalcitePlanner 物化檢視相關原始碼-01
阿新 • • 發佈:2019-01-11
查詢Hive3.x Materialized view中構建的物化檢視的例子, debug檢視詳細執行過程
1)查詢語句
SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
SET hive.support.concurrency=true;
SET hive.enforce.bucketing=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.compactor. initiator.on=true;
SET hive.compactor.worker.threads=2;
SET hive.query.results.cache.enabled=false;
SELECT deptno, count(1) as deptno_cnt from depts group by deptno;
2)查詢語句
hive> SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
hive> SET hive.support.concurrency=true;
hive> SET hive.enforce.bucketing=true;
hive> SET hive.exec.dynamic.partition.mode=nonstrict;
hive> SET hive.compactor.initiator.on=true;
hive> SET hive.compactor.worker.threads=2;
hive> SET hive.query.results.cache.enabled=false;
hive> explain SELECT deptno, count(1) as deptno_cnt from depts group by deptno;
FAILED: SemanticException [Error 10001]: Line 1:52 Table not found 'depts'
hive> explain SELECT deptno, count(1) as deptno_cnt from hive3_test.depts group by deptno;
OK
STAGE DEPENDENCIES:
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
TableScan
alias: hive3_test.depts_agg
Statistics: Num rows: 2 Data size: 24 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: deptno (type: int), deptno_cnt (type: bigint)
outputColumnNames: _col0, _col1
Statistics: Num rows: 2 Data size: 24 Basic stats: COMPLETE Column stats: NONE
ListSink
Time taken: 1.806 seconds, Fetched: 17 row(s)
SemanticAnalyzer
void analyzeInternal(ASTNode ast, PlannerContextFactory pcf) {
....
// 1. Generate Resolved Parse tree from syntax tree
boolean needsTransform = needsTransform();
// 2. Gen OP Tree from resolved Parse Tree
Operator sinkOp = genOPTree(ast, plannerCtx);//進入CalcitePlanner::getOPTree
//---待續
}
CalcitePlanner::getOPTree
這裡入參為hive的ASTNode
Operator genOPTree(ASTNode ast, PlannerContext plannerCtx){
......
// 1. Gen Optimized AST
ASTNode newAST = getOptimizedAST();
}
CalcitePlanner::getOptimizedAST
ASTNode getOptimizedAST() throws SemanticException {
//用calcite優化查詢,生成calcite的RelNode
RelNode optimizedOptiqPlan = logicalPlan();
//將RelNode轉化為hive的ASTNode
ASTNode optiqOptimizedAST = ASTConverter.convert(optimizedOptiqPlan, resultSchema,
HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_COLUMN_ALIGNMENT));
return optiqOptimizedAST;
}
CalcitePlanner:: logicalPlan
RelNode logicalPlan() throws SemanticException {
RelNode optimizedOptiqPlan = null;
CalcitePlannerAction calcitePlannerAction = null;
/**
* Map of table name to names of accessed columns
*/
Map<String, Set<String>> this.columnAccessInfo = new ColumnAccessInfo();
/**
* CalcitePlannerAction is code responsible for Calcite plan generation and optimization.
*/
calcitePlannerAction = new CalcitePlannerAction(
prunedPartitions,
ctx.getOpContext().getColStatsCache(),
this.columnAccessInfo);
//calcite 優化plan,主要優化工作在CalcitePlanner::CalcitePlannerAction::apply()
optimizedOptiqPlan = Frameworks.withPlanner(calcitePlannerAction, Frameworks
.newConfigBuilder().typeSystem(new HiveTypeSystemImpl()).build());
return optimizedOptiqPlan;
}
CalcitePlanner::CalcitePlannerAction::apply()
待續。。。