kettle-java程式碼執行hive相關ktr時報錯： database type with plugin id [HIVE2] couldn't be found!

轉

2018年08月13日 16:47:30 lisery1993 閱讀數：305

1.在java專案中執行hive相關的ktr:

1.  

2. KettleEnvironment.init();

3.  

4. TransMeta transMeta = new TransMeta("C:\\Users\\lixiang\\Desktop\\hivetorizhi.ktr");

5. // 設定執行模式

6. TransExecutionConfiguration config = new TransExecutionConfiguration();

7. config.setExecutingClustered(true);

8. config.setExecutingLocally(false);

9. config.setExecutingRemotely(false);

10. config.setClusterPosting(true);

11. config.setClusterPreparing(true);

12. config.setClusterStarting(true);

13. TransSplitter transSplitter = Trans.executeClustered(transMeta, config);

14. System.out.println(transSplitter.getCarteObjectMap());

15. System.out.println(transSplitter.getMaster());

16. System.out.println(transSplitter.getSlaves()[0]);

17. System.out.println(transSplitter.getSlaves()[1].getStepNames()[0]);

18. }

2.報錯：

1. Exception in thread "main" org.pentaho.di.core.exception.KettleXMLException:

2. 錯誤從XML檔案讀取轉換

3.  

4. 錯誤從XML檔案讀取轉換

5.  

6. Unable to load database connection info from XML node

7.  

8. Unable to create new database interface

9.  

10. database type with plugin id [HIVE2] couldn't be found!

11.  

12. at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:3402)

13. at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2758)

14. at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2710)

15. at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2687)

16. at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2667)

17. at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2632)

18. at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2595)

19. at com.kettle.KettleExamle.getRemoteServer(KettleExamle.java:55)

20. at com.kettle.KettleExamle.main(KettleExamle.java:140)

21. Caused by: org.pentaho.di.core.exception.KettleXMLException:

22. 錯誤從XML檔案讀取轉換

23.  

24. Unable to load database connection info from XML node

25.  

26. Unable to create new database interface

27.  

28. database type with plugin id [HIVE2] couldn't be found!

29.  

30. at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:3384)

31. ... 8 more

32. Caused by: org.pentaho.di.core.exception.KettleXMLException:

33. Unable to load database connection info from XML node

34.  

35. Unable to create new database interface

36.  

37. database type with plugin id [HIVE2] couldn't be found!

38.  

39. at org.pentaho.di.core.database.DatabaseMeta.<init>(DatabaseMeta.java:1013)

40. at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:2976)

41. ... 8 more

42. Caused by: org.pentaho.di.core.exception.KettleXMLException:

43. Unable to create new database interface

44.  

45. database type with plugin id [HIVE2] couldn't be found!

46.  

47. at org.pentaho.di.core.database.DatabaseMeta.<init>(DatabaseMeta.java:977)

48. ... 9 more

49. Caused by: org.pentaho.di.core.exception.KettleDatabaseException:

50. database type with plugin id [HIVE2] couldn't be found!

51.  

52. at org.pentaho.di.core.database.DatabaseMeta.findDatabaseInterface(DatabaseMeta.java:584)

53. at org.pentaho.di.core.database.DatabaseMeta.getDatabaseInterface(DatabaseMeta.java:558)

54. at org.pentaho.di.core.database.DatabaseMeta.<init>(DatabaseMeta.java:974)

55. ... 9 more

3.報錯分析：kettle程式在讀取database type的過程中讀不到hive2這個型別。

檢視kettle-core-7.1.0.0-12.jar中的kettle-database-types.xml檔案，此檔案定義了database type ,其中沒hive2.。

1. <database-types>

2.  

3. <database-type id="AS/400">

4. <description>AS/400</description>

5. <classname>org.pentaho.di.core.database.AS400DatabaseMeta</classname>

6. </database-type>

7.  

8. <database-type id="CACHE">

9. <description>Intersystems Cache</description>

10. <classname>org.pentaho.di.core.database.CacheDatabaseMeta</classname>

11. </database-type>

12.  

13. <database-type id="DB2">

14. <description>IBM DB2</description>

15. <classname>org.pentaho.di.core.database.DB2DatabaseMeta</classname>

16. </database-type>

17.  

18. <database-type id="DBASE">

19. <description>dBase III, IV or 5</description>

20. <classname>org.pentaho.di.core.database.DbaseDatabaseMeta</classname>

21. </database-type>

22.  

23. <database-type id="DERBY">

24. <description>Apache Derby</description>

25. <classname>org.pentaho.di.core.database.DerbyDatabaseMeta</classname>

26. </database-type>

27.  

28. <database-type id="EXASOL4">

29. <description>Exasol 4</description>

30. <classname>org.pentaho.di.core.database.Exasol4DatabaseMeta</classname>

31. </database-type>

32.  

33. <database-type id="EXTENDB">

34. <description>ExtenDB</description>

35. <classname>org.pentaho.di.core.database.ExtenDBDatabaseMeta</classname>

36. </database-type>

37.  

38. <database-type id="FIREBIRD">

39. <description>Firebird SQL</description>

40. <classname>org.pentaho.di.core.database.FirebirdDatabaseMeta</classname>

41. </database-type>

42.  

43. <database-type id="GENERIC">

44. <description>Generic database</description>

45. <classname>org.pentaho.di.core.database.GenericDatabaseMeta</classname>

46. </database-type>

47.  

48. <database-type id="GREENPLUM">

49. <description>Greenplum</description>

50. <classname>org.pentaho.di.core.database.GreenplumDatabaseMeta</classname>

51. </database-type>

52.  

53. <database-type id="INFINIDB">

54. <description>Calpont InfiniDB</description>

55. <classname>org.pentaho.di.core.database.InfiniDbDatabaseMeta</classname>

56. </database-type>

57.  

58. <database-type id="SQLBASE">

59. <description>Gupta SQL Base</description>

60. <classname>org.pentaho.di.core.database.GuptaDatabaseMeta</classname>

61. </database-type>

62.  

63. <database-type id="H2">

64. <description>H2</description>

65. <classname>org.pentaho.di.core.database.H2DatabaseMeta</classname>

66. </database-type>

67.  

68. <database-type id="HYPERSONIC">

69. <description>Hypersonic</description>

70. <classname>org.pentaho.di.core.database.HypersonicDatabaseMeta</classname>

71. </database-type>

72.  

73. <database-type id="INFOBRIGHT">

74. <description>Infobright</description>

75. <classname>org.pentaho.di.core.database.InfobrightDatabaseMeta</classname>

76. </database-type>

77.  

78. <database-type id="INFORMIX">

79. <description>Informix</description>

80. <classname>org.pentaho.di.core.database.InformixDatabaseMeta</classname>

81. </database-type>

82.  

83. <database-type id="INGRES">

84. <description>Ingres</description>

85. <classname>org.pentaho.di.core.database.IngresDatabaseMeta</classname>

86. </database-type>

87.  

88. <database-type id="VECTORWISE">

89. <description>Ingres VectorWise</description>

90. <classname>org.pentaho.di.core.database.VectorWiseDatabaseMeta</classname>

91. </database-type>

92.  

93. <database-type id="INTERBASE">

94. <description>Borland Interbase</description>

95. <classname>org.pentaho.di.core.database.InterbaseDatabaseMeta</classname>

96. </database-type>

97.  

98. <database-type id="KINGBASEES">

99. <description>KingbaseES</description>

100. <classname>org.pentaho.di.core.database.KingbaseESDatabaseMeta</classname>

101. </database-type>

102.  

103. <database-type id="LucidDB">

104. <description>LucidDB</description>

105. <classname>org.pentaho.di.core.database.LucidDBDatabaseMeta</classname>

106. </database-type>

107.  

108. <database-type id="MARIADB">

109. <description>MariaDB</description>

110. <classname>org.pentaho.di.core.database.MariaDBDatabaseMeta</classname>

111. </database-type>

112.  

113. <database-type id="MONETDB">

114. <description>MonetDB</description>

115. <classname>org.pentaho.di.core.database.MonetDBDatabaseMeta</classname>

116. </database-type>

117.  

118. <database-type id="MSACCESS">

119. <description>MS Access</description>

120. <classname>org.pentaho.di.core.database.MSAccessDatabaseMeta</classname>

121. </database-type>

122.  

123. <database-type id="MSSQLNATIVE">

124. <description>MS SQL Server (Native)</description>

125. <classname>org.pentaho.di.core.database.MSSQLServerNativeDatabaseMeta</classname>

126. </database-type>

127.  

128. <database-type id="MSSQL">

129. <description>MS SQL Server</description>

130. <classname>org.pentaho.di.core.database.MSSQLServerDatabaseMeta</classname>

131. </database-type>

132.  

133. <database-type id="MYSQL">

134. <description>MySQL</description>

135. <classname>org.pentaho.di.core.database.MySQLDatabaseMeta</classname>

136. </database-type>

137.  

138. <database-type id="MONDRIAN">

139. <description>Native Mondrian</description>

140. <classname>org.pentaho.di.core.database.MondrianNativeDatabaseMeta</classname>

141. </database-type>

142.  

143. <database-type id="NEOVIEW">

144. <description>Neoview</description>

145. <classname>org.pentaho.di.core.database.NeoviewDatabaseMeta</classname>

146. </database-type>

147.  

148. <database-type id="NETEZZA">

149. <description>Netezza</description>

150. <classname>org.pentaho.di.core.database.NetezzaDatabaseMeta</classname>

151. </database-type>

152.  

153. <database-type id="ORACLE">

154. <description>Oracle</description>

155. <classname>org.pentaho.di.core.database.OracleDatabaseMeta</classname>

156. </database-type>

157.  

158. <database-type id="ORACLERDB">

159. <description>Oracle RDB</description>

160. <classname>org.pentaho.di.core.database.OracleRDBDatabaseMeta</classname>

161. </database-type>

162.  

163. <database-type id="POSTGRESQL">

164. <description>PostgreSQL</description>

165. <classname>org.pentaho.di.core.database.PostgreSQLDatabaseMeta</classname>

166. </database-type>

167.  

168. <database-type id="REDSHIFT">

169. <description>Redshift</description>

170. <classname>org.pentaho.di.core.database.RedshiftDatabaseMeta</classname>

171. </database-type>

172.  

173. <database-type id="REMEDY-AR-SYSTEM">

174. <description>Remedy Action Request System</description>

175. <classname>org.pentaho.di.core.database.RemedyActionRequestSystemDatabaseMeta</classname>

176. </database-type>

177.  

178. <database-type id="SAPDB">

179. <description>MaxDB (SAP DB)</description>

180. <classname>org.pentaho.di.core.database.SAPDBDatabaseMeta</classname>

181. </database-type>

182.  

183. <database-type id="SQLITE">

184. <description>SQLite</description>

185. <classname>org.pentaho.di.core.database.SQLiteDatabaseMeta</classname>

186. </database-type>

187.  

188. <database-type id="SYBASE">

189. <description>Sybase</description>

190. <classname>org.pentaho.di.core.database.SybaseDatabaseMeta</classname>

191. </database-type>

192.  

193. <database-type id="SYBASEIQ">

194. <description>SybaseIQ</description>

195. <classname>org.pentaho.di.core.database.SybaseIQDatabaseMeta</classname>

196. </database-type>

197.  

198. <database-type id="TERADATA">

199. <description>Teradata</description>

200. <classname>org.pentaho.di.core.database.TeradataDatabaseMeta</classname>

201. </database-type>

202.  

203. <database-type id="UNIVERSE">

204. <description>UniVerse database</description>

205. <classname>org.pentaho.di.core.database.UniVerseDatabaseMeta</classname>

206. </database-type>

207.  

208. <database-type id="VERTICA">

209. <description>Vertica</description>

210. <classname>org.pentaho.di.core.database.VerticaDatabaseMeta</classname>

211. </database-type>

212.  

213. <database-type id="VERTICA5">

214. <description>Vertica 5+</description>

215. <classname>org.pentaho.di.core.database.Vertica5DatabaseMeta</classname>

216. </database-type>

217.  

218. </database-types>

4.報錯解決：

hive的api的連線方式是通過jdbc連線的。具體思路與postgresql的連線思路相同。因此仿照postgresql的連線方法自定義一個database-type為hive2。因此，在kettle-core-7.1.0.0-12.jar的kettle-database-types.xml檔案中新增

1. <database-type id="HIVE2">

2. <description>HIVE2</description>

3. <classname>org.pentaho.di.core.database.Hive2SQLDatabaseMeta</classname>

4. </database-type>

其中會有對應的實現類org.pentaho.di.core.database.Hive2SQLDatabaseMeta。

仿照postgresql的實現類org.pentaho.di.core.database.PostgreSQLDatabaseMeta來寫即可，只需將連線的url以及jdbc driver名稱更換為hive相關。

將org.pentaho.di.core.database.Hive2SQLDatabaseMeta類編譯為.class檔案放進kettle-core-7.1.0.0-12.jar的org\pentaho\di\core\database路徑下

6.再次執行ktr：

執行成功：結果：

1. {/hivetorizhi (master)=1cc5f5bb-b0c4-4938-96f7-85363f88564d, /hivetorizhi (local:slave1-8081)=09ea3633-f5bd-4e43-9352-62b958255469, /hivetorizhi (local:slave2-8082)=35e00dc2-f570-4f26-8540-156fb84d9524}

2. /hivetorizhi (master)

3. /hivetorizhi (local:slave2-8082)

4.