1. 程式人生 > >spark on yarn 異常記錄二

spark on yarn 異常記錄二

通過spark-submit 向叢集提交任務時,–num-executors 數量大於1即會出現異常,異常如下:

17/06/04 11:47:03 WARN DefaultChannelPipeline: An exception 'java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while
handling the following exception: java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel; at org.apache.spark.rpc.netty.NettyRpcHandler.channelInactive(NettyRpcEnv.scala:621) at org.apache.spark.network.server.TransportRequestHandler.channelInactive(TransportRequestHandler.java:99
) at org.apache.spark.network.server.TransportChannelHandler.channelInactive(TransportChannelHandler.java:103) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:251) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:237
) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:230) at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:257) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:251) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:237) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:230) at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:251) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:237) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:230) at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at org.apache.spark.network.util.TransportFrameDecoder.channelInactive(TransportFrameDecoder.java:182) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:251) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:237) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:230) at org.spark_project.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1289) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:251) at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:237) at org.spark_project.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:893) at org.spark_project.io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:691) at org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:376) at org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:680) at org.spark_project.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) at org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) at org.spark_project.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:745) 17/06/04 11:47:03 INFO ShutdownHookManager: Shutdown hook called

google半天,有很多說法,但是依舊沒有解決我的問題
參考 jira中各種方法,問題並沒有得到解決。想著,異常原因是方法找不到,那麼我把這個類單獨提取出來,放到我要提交的jar中,會不會得到解決,畢竟以前遇到過這種情況,就是這麼解決的。試了試,問題並沒有得到解決。暫時先記著,等請教幾個大牛,再來更新……