1. 程式人生 > >8springboot 呼叫hive 操作資料庫的bug

8springboot 呼叫hive 操作資料庫的bug

1  java.lang.NoClassDefFoundError: org/apache/hive/service/cli/thrift/TCLIService$Iface

解決載入類載入不進去的方法:

https://blog.csdn.net/u010429286/article/details/78780145

 

特此:粘出最終板的pom檔案

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.pengjunlee</groupId>
	<artifactId>spring-boot-hive</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<packaging>jar</packaging>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>2.0.4.RELEASE</version>
		<relativePath />
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>com.alibaba</groupId>
			<artifactId>druid-spring-boot-starter</artifactId>
			<version>1.1.1</version>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-jdbc</artifactId>

		</dependency>
		<dependency>
			<groupId>org.springframework.data</groupId>
			<artifactId>spring-data-hadoop</artifactId>
			<version>2.5.0.RELEASE</version>
		</dependency>
		 <!--<dependency>-->
		    <!--<groupId>org.apache.hive</groupId>-->
		    <!--<artifactId>hive-service</artifactId>-->
		    <!--<version>1.1.0-cdh5.11.0</version>-->
		<!--</dependency>-->
		<dependency>
		    <groupId>org.apache.thrift</groupId>
		    <artifactId>libthrift</artifactId>
		    <version>0.11.0</version>
		    <type>pom</type>
		</dependency>
		<dependency>
			<groupId>org.apache.hive</groupId>
			<artifactId>hive-jdbc</artifactId>
			<exclusions>
				<exclusion>
					<groupId>org.apache.hive</groupId>
					<artifactId>hive-service-rpc</artifactId>
				</exclusion>
				<exclusion>
					<groupId>org.apache.hive</groupId>
					<artifactId>hive-service</artifactId>
				</exclusion>
			</exclusions>
			<version>1.1.0</version>
		</dependency>
		<dependency>
			<groupId>org.apache.hive</groupId>
			<artifactId>hive-service</artifactId>
			<version>1.1.0</version>
		</dependency>

		<dependency>
			<groupId>org.apache.tomcat</groupId>
			<artifactId>tomcat-jdbc</artifactId>
		</dependency>
		
	</dependencies>
</project>

 

============================================================

總結:進入了誤區

<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-jdbc</artifactId>
    <version>1.1.0-cdh5.11.0</version>
</dependency>

並不能載入進去,反而錯誤越來越多

需要版本一樣<version>1.1.0</version>,並不是<version>1.1.0-cdh5.11.0</version>

遇到的問題::

1問題:

java.sql.SQLException: Could not establish connection to jdbc:hive2://hadoop2:10000/default: Required field 'serverProtocolVersion' is unset! Struct:TOpenSessionResp(status:TStatus(statusCode:SUCCESS_STATUS), serverProtocolVersion:null, sessionHandle:TSessionHandle(sessionId:THandleIdentifier(guid:AC 5A 2E B8 FB ED 48 35 BA F5 0C 58 28 E2 FB 56, secret:BD 9D 67 70 F3 4A 4B 03 A4 E1 58 86 AF 02 BE 2C)), configuration:{})
    at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:246)
    at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:132)
    at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:247)
    at cn.orcale.com.project.CDHHiveJdbcCli.main(CDHHiveJdbcCli.java:24)
Caused by: org.apache.thrift.protocol.TProtocolException: Required field 'serverProtocolVersion' is unset! Struct:TOpenSessionResp(status:TStatus(statusCode:SUCCESS_STATUS), serverProtocolVersion:null, sessionHandle:TSessionHandle(sessionId:THandleIdentifier(guid:AC 5A 2E B8 FB ED 48 35 BA F5 0C 58 28 E2 FB 56, secret:BD 9D 67 70 F3 4A 4B 03 A4 E1 58 86 AF 02 BE 2C)), configuration:{})
    at org.apache.hive.service.cli.thrift.TOpenSessionResp.validate(TOpenSessionResp.java:578)
    at org.apache.hive.service.cli.thrift.TOpenSessionResp$TOpenSessionRespStandardScheme.read(TOpenSessionResp.java:676)
    at org.apache.hive.service.cli.thrift.TOpenSessionResp$TOpenSessionRespStandardScheme.read(TOpenSessionResp.java:612)
    at org.apache.hive.service.cli.thrift.TOpenSessionResp.read(TOpenSessionResp.java:520)
    at org.apache.hive.service.cli.thrift.TCLIService$OpenSession_result$OpenSession_resultStandardScheme.read(TCLIService.java:2041)
    at org.apache.hive.service.cli.thrift.TCLIService$OpenSession_result$OpenSession_resultStandardScheme.read(TCLIService.java:2026)
    at org.apache.hive.service.cli.thrift.TCLIService$OpenSession_result.read(TCLIService.java:1973)
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
    at org.apache.hive.service.cli.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:144)
    at org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:131)
    at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:237)
    ... 5 more

這是由於mysql聯結器驅動和伺服器中新增的不符合造成的

 

解決辦法:

在cdh中或者apache版本中將下列hive包取出來,載入到Eclipse中

伺服器用cloudera搭的:

@可以檢視/usr/share/java/目錄下是否有mysql的驅動存在

      

@ 找到sql-connection.jar

      /opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/hive/lib/    可找到mysql-connect的版本

@ sz  hive-jdbc.jar

 @ 將pom檔案中的依賴更改

<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-jdbc</artifactId>
    <version>1.1.0-cdh5.11.0</version>
</dependency>

即將jar包新增到maven中

=========================================

2

java.lang.ClassNotFoundException: org.apache.hive.service.cli.thrift.TCLIService$Iface
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_151]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_151]
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) ~[na:1.8.0_151]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_151]
	at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) ~[hive-jdbc-1.1.0-cdh5.11.0.jar:1.1.0-cdh5.11.0]
	at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1464) ~[druid-1.1.0.jar:1.1.0]
	at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1529) ~[druid-1.1.0.jar:1.1.0]
	at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:787) ~[druid-1.1.0.jar:1.1.0]
	at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1122) ~[druid-1.1.0.jar:1.1.0]
	at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1118) ~[druid-1.1.0.jar:1.1.0]
	at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:104) ~[druid-1.1.0.jar:1.1.0]
	at com.pengjunlee.hive.controller.HiveDataSourceController.listAllTables(HiveDataSourceController.java:42) ~[classes/:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877) ~[spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783) ~[spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) ~[spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877) ~[spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) ~[spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.5.32.jar:8.5.32]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123) ~[druid-1.1.0.jar:1.1.0]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) [tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800) [tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800) [tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471) [tomcat-embed-core-8.5.32.jar:8.5.32]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.32.jar:8.5.32]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.32.jar:8.5.32]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]

所以去看下引入的依賴包,發現有hive-service和hive-service-rpc兩個包中有相同的原始碼,所以在這裡會拋異常,我就先把這其中一個依賴從hive-jdbc中去除掉,但是發現還是拋異常,我這裡用的hive-jdbc1.1版本,發現hive-service是2.1版本,所以立馬把兩個都去除掉,重新引入一個hive-service 1.1進來就好拉,
======================================================