【大資料安全】Apache Kylin 安全配置(Kerberos)
阿新 • • 發佈：2018-12-09
1. 概述
本文首先會簡單介紹Kylin的安裝配置，然後介紹啟用Kerberos的CDH叢集中如何部署及使用Kylin。
Apache Kylin™是一個開源的分散式分析引擎，提供Hadoop/Spark之上的SQL查詢介面及多維分析（OLAP）能力以支援超大規模資料，最初由eBay Inc. 開發並貢獻至開源社群。它能在亞秒內查詢巨大的Hive表。
1.2 環境說明
- CDH版本：5.11.2
- Linux版本：7.4.1708
- Docker版本：Docker version 18.06.0-ce
- JDK版本：1.8
- 操作使用者：root
2.Kylin安裝配置
2.1 安裝
此處以Kylin 2.0.0版本為例。 社群版Kylin地址：https://archive.apache.org/dist/kylin/
[[email protected] ~]# cd /usr/local/src/
[[email protected] src]# wget https://archive.apache.org/dist/kylin/apache-kylin-2.0.0/apache-kylin-2.0.0-bin-cdh57.tar.gz
[[email protected] src]# tar xf apache-kylin-2.0.0-bin-cdh57.tar.gz
[[email protected] src]# cp -a apache-kylin-2.0.0-bin /usr/local/
[[email protected] 2.2 環境配置
export BASE_PATH_BIG=/opt/cloudera/parcels/CDH/lib #added by Hbase export HBASE_HOME=$BASE_PATH_BIG/hbase #added by HCat export HCAT_HOME=/opt/cloudera/parcels/CDH/lib/hive-hcatalog/share/hcatalog #added by Kylin export KYLIN_HOME=/usr/local/kylin export PATH=$HBASE_HOME/bin:$PATH
然後執行source /etc/profile生效。
2.3 Kylin配置
編輯/usr/local/kylin/conf/kylin.properites檔案，新增以下配置：
Kylin2.0+版本配置的名稱有變化，具體參考：https://github.com/apache/kylin/blob/2.0.x/core-common/src/main/resources/kylin-backward-compatibility.properties
## 修改配置（替換地址）
kylin.rest.servers=192.168.100.102:7070
## 新增配置
kylin.job.jar=/usr/local/apache-kylin-2.0.0-bin/lib/kylin-job-2.0.0.jar
kylin.coprocessor.local.jar=/usr/local/apache-kylin-2.0.0-bin/lib/kylin-coprocessor-2.0.0.jar
## 替換地址
kylin.job.yarn.app.rest.check.status.url=http://cdh-node-2:8088/ws/v1/cluster/apps/${job_id}?anonymous=true
kylin.job.mr.lib.dir=/opt/cloudera/parcels/CDH-5.11.2-1.cdh5.11.2.p0.4/lib/sentry/lib
配置說明：
kylin.rest.servers：kylin例項伺服器列表，注意：不包括以job模式執行的伺服器例項！kylin.job.jar：MR jobs依賴kylin.coprocessor.local.jar：Hbase協同處理依賴，用於提高效能。kylin.job.yarn.app.rest.check.status.url：yarn工作區kylin.job.mr.lib.dir：Hive/Hbase依賴目錄。（在沒有安裝Hive/Hbase的節點上構建Cube會因為找不到依賴報ClassNotFoundException錯誤，需要此配置。具體參考這邊博文《上傳Kylin MR依賴》)。
2.4 建立使用者
在每個節點建立Kylin使用者
useradd kylin3. Kerberos配置
3.1 建立kylin賬號
在Kerberos server上建立kylin賬號：
[[email protected] /]# kadmin.local
Authenticating as principal admin/[email protected] with password.
kadmin.local: addprinc kylin
WARNING: no policy specified for [email protected]; defaulting to no policy
Enter password for principal "[email protected]":
Re-enter password for principal "[email protected]":
add_principal: Principal or policy already exists while creating "[email protected]".
kadmin.local:3.2 生成keytab檔案
生成kylin賬號keytab檔案：
xst -norandkey -k klin.keytab [email protected]將kylin.keytab複製到每個kylin節點上。
3.3 定時重新整理kt
在kylin節點上配置定時任務進行kinit命令：
kinit -k -t /root/kylin.keytab [email protected]
新增定時任務,這裡設定每天凌晨1點執行一次，可根據kerberos的過期時間自己配置。
- 編寫shell指令碼：
[[email protected] security]# cat init_kt.sh
#!/bin/bash
kinit -kt ./kylin.keytab [email protected]
- 新增定時任務：
[[email protected] security]# crontab -e
# 新增以下內容
0 1 * * * root /home/security/init_kt.sh > /tmp/kylin-ktinit.log 2>&1
# 檢視定時任務
[[email protected] security]# crontab -l
0 1 * * * root /home/security/init_kt.sh > /tmp/kylin-ktinit.log 2>&13.4 新增Hive許可權
- 登入Hive 使用擁有操作hive許可權的kerberos賬戶登入beeline：
[[email protected] hive]# beeline -u 'jdbc:hive2://cdh-node-3:10000/default;principal=hive/[email protected]'
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
scan complete in 2ms
Connecting to jdbc:hive2://cdh-node-3:10000/default;principal=hive/[email protected]
Connected to: Apache Hive (version 1.1.0-cdh5.11.2)
Driver: Hive JDBC (version 1.1.0-cdh5.11.2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.1.0-cdh5.11.2 by Apache Hive
0: jdbc:hive2://cdh-node-3:10000/default>- 新增kylin許可權
將
admin角色許可權賦予kylin：
0: jdbc:hive2://cdh-node-3:10000/default> grant role admin to user kylin;
INFO : Compiling command(queryId=hive_20180914174141_6359e1a1-251f-4116-b646-0aa9f55dcfa8): grant role admin to user leili
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling command(queryId=hive_20180914174141_6359e1a1-251f-4116-b646-0aa9f55dcfa8); Time taken: 0.069 seconds
INFO : Executing command(queryId=hive_20180914174141_6359e1a1-251f-4116-b646-0aa9f55dcfa8): grant role admin to user leili
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=hive_20180914174141_6359e1a1-251f-4116-b646-0aa9f55dcfa8); Time taken: 0.331 seconds
INFO : OK
No rows affected (0.528 seconds)3.5 新增Hbase許可權
使用擁有操作hbase許可權的kerberos賬戶登入hbase shell：
[[email protected] hive]# hbase shell
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
18/09/14 17:44:40 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.2.0-cdh5.11.2, rUnknown, Fri Aug 18 14:09:37 PDT 2017- 給kylin使用者授權
base(main):001:0> grant 'kylin','RWXCA'
0 row(s) in 0.5830 seconds切換到kylin使用者重新登入hbase shell，測試一下：
hbase(main):001:0> whoami
[email protected] (auth:KERBEROS)
groups: kylin
hbase(main):002:0> create 'test', 'cf'
0 row(s) in 2.6930 seconds
=> Hbase::Table - test
hbase(main):003:0> list
TABLE
test
1 row(s) in 0.0310 seconds3.6 執行Kylin檢查
[[email protected] bin]# ./check-env.sh
Retrieving hadoop conf dir...
KYLIN_HOME is set to /usr/local/apache-kylin-2.0.0-bin3.7 啟動kylin服務
先確認主機使用kerberos憑據為kylin，再啟動：
[[email protected] bin]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: [email protected]
Valid starting Expires Service principal
09/14/2018 17:48:28 09/15/2018 17:48:28 krbtgt/[email protected]
renew until 09/21/2018 17:48:28
[[email protected] bin]# ./kylin.sh start
Retrieving hadoop conf dir...
KYLIN_HOME is set to /usr/local/kylin
Retrieving hive dependency...
Retrieving hbase dependency...
Retrieving hadoop conf dir...
Retrieving kafka dependency...
Retrieving Spark dependency...
KYLIN_JVM_SETTINGS is -Xms1024M -Xmx4096M -Xss1024K -XX:MaxPermSize=128M -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/usr/local/kylin/logs/kylin.gc.3982 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M
A new Kylin instance is started by root. To stop it, run 'kylin.sh stop'
Check the log at /usr/local/kylin/logs/kylin.log
Web UI is at http://<hostname>:7070/kylin4. 使用Kylin Sample測試
Kylin本身自帶了一個測試例子，建立流程如下： 執行sample.sh指令碼，這個主要是建立kylin的project、model、cube以及相關的hive表等。
[[email protected] bin]# ./sample.sh
Retrieving hadoop conf dir...
Retrieving hadoop conf dir...
KYLIN_HOME is set to /usr/local/kylin
Loading sample data into HDFS tmp path: /tmp/kylin/sample_cube/d
...
Sample cube is created successfully in project 'learn_kylin'.
Restart Kylin server or reload the metadata from web UI to see the change.進入Kylin Web介面system - reload metadata 然後構建示例的cube，如果構建成功，並且能成功執行查詢命令，則表示整個配置全部完成。
[1] Cloudera Configuring Authentication Doc： https://www.cloudera.com/documentation/enterprise/5-11-x/topics/sg_authentication.html [2] Kylin官網配置指南： http://kylin.apache.org/cn/docs/install/configuration.html [3] Linux下的crontab定時執行任務命令詳解： https://www.cnblogs.com/longjshz/p/5779215.html