018-04-20 11:36:01:ERROR http-bio-8888-exec-8 cn.shibei.feixia.handler.ControllerExceptionHandler.handledException(ControllerExceptionHandler.java:117) - 攔截到異常請求:
org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:978)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at cn.shibei.core.filter.AccessControlFilter.doFilter(AccessControlFilter.java:36)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:957)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620)
at 

org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at 

com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:271)
at com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:155)
at com.alibaba.com.caucho.hessian.io.SerializerFactory.readObject(SerializerFactory.java:397)
at com.alibaba.com.caucho.hessian.io

.Hessian2Input.readObjectInstance(Hessian2Input.java:2070)
at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2005)
at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1990)

從日誌中可以看出 heap space發生了OOM

登陸飛俠伺服器，通過jmap -heap發現堆記憶體使用率不到10%，完全正常。然後通過jstat -gc檢視fullGC頻率，也完全正常。  想看下GC日誌發現feixia沒加gc日誌相關的jvm引數。 新增-XX:+PrintGCDetails  -Xloggc:XXX.log  -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC，重啟，看下GC日誌，GC發生前後的堆資訊也完全正常。 

沒線索了 只能新增-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=java_oom_dump.hprof 將oom時的heap space dump下來。

等待OOM再次發生。

幾分鐘後 OOM又發生了  但是沒有找到java_oom_dump.hprof  問題很明確了  如果是java heap space報的OOM 那麼dump檔案不可能不存在。所以懷疑是native heap報的OOM。

看日誌中的com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate  可以聯想到Dubbo使用的netty。 netty會建立大量的DirectByteBuffer佔用直接記憶體。 而飛俠並沒有設定-XX:MaxDirectMemorySize，可知直接記憶體只有預設的64M。 調大MaxDirectMemorySize 等待OOM的再次發生

等待的過程中， big uncle the smart感覺可能是dubbo將其他機器的異常資訊拋到了feixia。於是檢視re，10bei等機器， 發現re的堆記憶體使用率到了99%，相當可疑了。然後看下re的fullGC次數  一天裡面竟然高達近5000次。。。而re日誌確實也有OOM的錯誤存在。把re的xmx調成1G 問題解決。

總結下： 使用Dubbo等類似服務的情況下，除了排查本機的記憶體洩漏以外，還需要考慮整個呼叫鏈的記憶體洩漏情況。