Pentaho Kettle 連線 CDH Hive (No suitable driver found for jdbc:hive2 錯誤解決方法)
關鍵點:
清理Kettle快取:
rm -rf /home/user/data-integration/./system/karaf/caches
rm -rf /home/user/data-integration/./system/karaf/data
配置Kettle big data setting:
在 Kettle安裝目錄/data-integration/plugins/pentaho-big-data-plugin/plugin.properties 中找到配置項
active.hadoop.configuration=cdh513
這裡的"cdh513"就是 Kettle安裝目錄/data-integration/plugins/pentaho-big-data-plugin/hadoop-configurations 下的子目錄名字, 又稱shim, 它相當於hadoop不同版本的驅動
預設有4個shim, 分別對應hadoop的4個發行版, 用哪個就在上述的plugin.properties裡配置好
配置shim的Hadoop setting檔案:
Kettle安裝目錄/data-integration/plugins/pentaho-big-data-plugin/hadoop-configurations/cdh513/下6個xml檔案, 如core-site.xml, hbase-site.xml, hdfs-site.xml, yarn-site.xml, hive-site.xml, mapred-site.xml
或者直接從叢集那裡拷貝覆蓋.(CDP各元件配置檔案路徑: Hadoop: /etc/hadoop/conf, hbase: /etc/hbase/conf, hive: /etc/hive/conf)
拷貝jar包:
執行kitchen或pan的時候如果報錯: no suitable driver found for jdbc:hive2
可以複製一遍jar包到kettle的lib目錄下 以及 active shim的lib目錄下
CDH的Hive在 /opt/cloudera/parcels/CDH/lib/hive
可以把/opt/cloudera/parcels/CDH/lib/hive/lib下所有hive開頭的jar包複製到 Kettle安裝目錄/data-integration/lib 和 Kettle安裝目錄/data-integration/plugins/pentaho-big-data-plugin/hadoop-configurations/cdh513/lib
例如(這裡假設kettle安裝在/opt下, 且當前生效的shim叫cdh513)
cp /opt/cloudera/parcels/CDH/lib/hive/lib/hive*.jar /opt/data-integration/plugins/pentaho-big-data-plugin/hadoop-configurations/cdh513/lib
cp /opt/cloudera/parcels/CDH/lib/hive/lib/hive*.jar /opt/data-integration/lib然後再清理一遍kettle的快取, 否則Kettle可能會不識別剛才拷貝的jar檔案:
rm -rf /home/fr-renjie.wei/data-integration/./system/karaf/caches
rm -rf /home/fr-renjie.wei/data-integration/./system/karaf/dataNo suitable driver found for jdbc:hive2
似乎linux上的Kettle本身在呼叫Hive jar包的過程中有什麼bug, 這個問題經常出現, 網上也有很多人問到這個bug.
我遇到情況是, Kitchen呼叫job會報這個錯, pan不會報錯.
我的解決辦法:
- 清理一次快取(rm -rf /home/user/data-integration/./system/karaf/caches)
- 執行kitchen
還有一個一勞永逸的辦法, 直接改Kitchen.sh, 加上rm這句
#!/bin/sh
# *****************************************************************************
#
# Pentaho Data Integration
#
# Copyright (C) 2005-2018 by Hitachi Vantara : http://www.pentaho.com
#
# *****************************************************************************
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# *****************************************************************************
INITIALDIR="`pwd`"
BASEDIR="`dirname $0`"
cd "$BASEDIR"
DIR="`pwd`"
cd - > /dev/null
rm -rf $BASEDIR/./system/karaf/caches #add this!
if [ "$1" = "-x" ]; then
set LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$BASEDIR/lib
export LD_LIBRARY_PATH
export OPT="-Xruntracer $OPT"
shift
fi
export IS_KITCHEN="true"
"$DIR/spoon.sh" -main org.pentaho.di.kitchen.Kitchen -initialDir "$INITIALDIR/" "[email protected]"