Spark2 Failed to send RPC 5346982634 to /ns1:58312: java.nio.channels.ClosedChannelException
將spark任務執行與yarn上出現以下錯誤：
scala> 18/11/21 16:20:11 ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!
18/11/21 16:20:11 ERROR client.TransportClient: Failed to send RPC 5346982634168622865 to /192.168.88.155:58312: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
18/11/21 16:20:11 ERROR cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful
java.io.IOException: Failed to send RPC 5346982634168622865 to /192.168.88.155:58312: java.nio.channels.ClosedChannelException
at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237)
1.因為spark on yarn，首先檢視ResourceMangaer的日誌：
2018-11-21 16:20:12,048 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Null container completed...
2018-11-21 16:20:14,714 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Null container completed...
說明container出現了問題，但具體原因尚不知
2.檢視NodeMangaer日誌：
2018-11-21 16:19:49,777 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=7203,containerID=container_1542787555988_0001_01_000001] is running beyond virtual memory limits. Current usage: 173.8 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.
很明顯的看出container使用的虛擬記憶體超過了設定的2.1G
container使用的虛擬記憶體是由以下公式計算的：
虛擬記憶體=yarn.scheduler.minimum-allocation-mb * yarn.nodemanager.vmem-pmem-ratio
引數都是在yarn-site.xml中配置的，如果需要使用的虛擬記憶體總量超過這個公式計算的值，就會Killing container.
此外，我的yarn.scheduler.minimum-allocation-mb值並沒有設定，因此預設為1G，yarn.nodemanager.vmem-pmem-ratio也沒設定，預設為2.1，所以就出現了日誌中的用了1G裡的360M實體記憶體，用了2.1G裡的2.4G虛擬記憶體。
然後修改yarn-site.xml如下配置
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>9000</value>
<discription>每個任務最多可用記憶體,預設8182MB</discription>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>3072</value>
<discription>每個任務最小可用記憶體</discription>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>3</value> ###實體記憶體和虛擬記憶體比率
</property>