記一次jvm crash

阿新 • • 發佈：2019-02-20

昨天下午4：30，系統發生了有史以來最刺激的崩潰，一開始是２臺掛了（線上共５臺），這個基本上很少見，一般情況下都是偶爾一臺掛掉。

趕緊手動重啟，搞笑的是重啟起來的機器裡馬也掛了，結果就出現了一種尷尬的局面，我們不停的重啟，機器不斷地報警，不斷地掛，整個過程持續了20多分鐘。

其實上個禮拜系統就掛過兩次，第１次掛了無任何資訊沒有出現常見的hs*.log這樣的jvm crash日誌，cpu、記憶體、流量等指標也沒有異常，打算開啟ulimit -c 獲取系統core,還沒弄上去第二次crash又發生了，於是把所有能看得日誌都查了一遍，也在/var/log/messages中找到了

"Jul 5 10:25:46 cms-web1 kernel:java[24016]: segfault at 00000000457ebd28 rip 00002aaab75a35cd rsp 00000000457ebd10 error 6"

臭名昭著的segfault出現了，於是趕緊在每臺機器加上了ulimit -c unlimited

20分鐘的忙碌終於過去了，於是趕緊分析core日誌

gdb java core-java-**.log

看到重要資訊

Core was generated by `/usr/xx/java/bin/java -Dprogram.name=run.sh -server -Xmx2g -Xms2g -Xmn512m'.

Program terminated with signal 11, Segmentation fault.

接下來通過gdb的bt看棧資訊

(gdb) bt
#0  0x00002aaab6cc75cd in Java_java_net_SocketOutputStream_socketWrite0 () from /usr/xx/install/jdk1.6.0_25/jre/lib/amd64/libnet.so
#1  0x00002aaaab041b0c in ?? ()
#2  0x0000000000000000 in ?? ()

只能看到最後在訪問網路的時候掛了，但棧資訊太少了

於是通過jstack /path/to/java coredumpfile 終於看到了有用的資訊

Thread 26089: (state = IN_NATIVE)

- java.net.SocketOutputStream.socketWrite0(java.io.FileDescriptor, byte[], int, int) @bci=0 (Interpreted frame)

- java.net.SocketOutputStream.socketWrite(byte[], int, int) @bci=44, line=92 (Interpreted frame)

- java.net.SocketOutputStream.write(byte[], int, int) @bci=4, line=136 (Interpreted frame)

- oracle.net.ns.DataPacket.send(int) @bci=50 (Interpreted frame)

- oracle.net.ns.NetOutputStream.flush() @bci=15 (Interpreted frame)

- oracle.net.ns.NetInputStream.getNextPacket() @bci=41 (Interpreted frame)

- oracle.net.ns.NetInputStream.read(byte[], int, int) @bci=33 (Compiled frame)

- oracle.net.ns.NetInputStream.read(byte[]) @bci=5 (Compiled frame)

- oracle.net.ns.NetInputStream.read() @bci=5 (Compiled frame)

- oracle.jdbc.driver.T4CMAREngine.unmarshalUB1() @bci=6, line=1099 (Compiled frame)

- oracle.jdbc.driver.T4CMAREngine.unmarshalSB1() @bci=1, line=1070 (Interpreted frame)

- oracle.jdbc.driver.T4C8Oall.receive() @bci=52, line=478 (Interpreted frame)

- oracle.jdbc.driver.T4CPreparedStatement.doOall8(boolean, boolean, boolean, boolean) @bci=619, line=216 (Interpreted frame)

- oracle.jdbc.driver.T4CPreparedStatement.executeForRows(boolean) @bci=16, line=955 (Interpreted frame)

- oracle.jdbc.driver.OracleStatement.executeMaybeDescribe() @bci=154, line=1060 (Interpreted frame)

- oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe() @bci=11, line=839 (Interpreted frame)

- oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout() @bci=132, line=1132 (Interpreted frame)

- oracle.jdbc.driver.OraclePreparedStatement.executeInternal() @bci=94, line=3316 (Interpreted frame)

- oracle.jdbc.driver.OraclePreparedStatement.executeQuery() @bci=17, line=3361 (Interpreted frame)

- com.xx.druid.filter.FilterChainImpl.preparedStatement_executeQuery(com.xx.druid.proxy.jdbc.PreparedStatementProxy) @bci=29, line=2564 (Interpreted frame)

- com.xx.druid.filter.FilterAdapter.preparedStatement_executeQuery(com.xx.druid.filter.FilterChain, com.xx.druid.proxy.jdbc.PreparedStatementProxy) @bci=2, line=931 (Interpreted frame)

- com.xx.druid.filter.FilterEventAdapter.preparedStatement_executeQuery(com.xx.druid.filter.FilterChain, com.xx.druid.proxy.jdbc.PreparedStatementProxy) @bci=14, line=459 (Interpreted frame)

- com.xx.druid.filter.FilterChainImpl.preparedStatement_executeQuery(com.xx.druid.proxy.jdbc.PreparedStatementProxy) @bci=17, line=2561 (Interpreted frame)

- com.xx.druid.proxy.jdbc.PreparedStatementProxyImpl.executeQuery() @bci=5, line=83 (Interpreted frame)

- org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeQuery() @bci=4, line=90 (Interpreted frame)

- org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery() @bci=12, line=236 (Interpreted frame)

- com.xx.china.cmshollywood.cms.engine.resourceloader.PreviewTemplateResourceLoader.getResourceStream(java.lang.String) @bci=80, line=65 (Interpreted frame)

- org.apache.velocity.Template.process() @bci=20, line=98 (Interpreted frame)

- com.xx.china.cmshollywood.cms.engine.resourceloader.PreviewResourceManagerImpl.loadResource(java.lang.String, int, java.lang.String) @bci=115, line=116 (Interpreted frame)

- org.apache.velocity.runtime.resource.ResourceManagerImpl.getResource(java.lang.String, int, java.lang.String) @bci=132, line=354 (Interpreted frame)

- org.apache.velocity.runtime.RuntimeInstance.getTemplate(java.lang.String, java.lang.String) @bci=11, line=1400 (Interpreted frame)

- org.apache.velocity.runtime.RuntimeInstance.getTemplate(java.lang.String) @bci=6, line=1380 (Interpreted frame)

- org.apache.velocity.app.VelocityEngine.getTemplate(java.lang.String) @bci=5, line=401 (Interpreted frame)

- com.xx.china.cmshollywood.cms.engine.DefaultVelocityTemplateEngine.mergeTemplate(org.apache.velocity.context.Context, java.lang.String, int) @bci=15, line=415 (Interpreted frame)

- com.xx.china.cmshollywood.cms.engine.DefaultVelocityTemplateEngine.previewTemplate(java.util.Map, java.lang.String, int) @bci=13, line=310 (Interpreted frame)

- com.xx.china.cmshollywood.cms.biz.tdprender.OfferRender.getOfferAndTradeInfoByOfferIds(java.lang.String, java.lang.String, java.lang.String, java.lang.String) @bci=129, line=275 (Interpreted frame)

- sun.reflect.GeneratedMethodAccessor283.invoke(java.lang.Object, java.lang.Object[]) @bci=64 (Interpreted frame)

- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Compiled frame)

- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=161, line=597 (Compiled frame)

- org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(java.lang.Object, java.lang.Object[]) @bci=6, line=389 (Interpreted frame)

- org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=66, line=378 (Interpreted frame)

- org.apache.velocity.runtime.parser.node.ASTMethod.execute(java.lang.Object, org.apache.velocity.context.InternalContextAdapter) @bci=449, line=270 (Interpreted frame)

- org.apache.velocity.runtime.parser.node.ASTReference.execute(java.lang.Object, org.apache.velocity.context.InternalContextAdapter) @bci=192, line=252 (Compiled frame)

- org.apache.velocity.runtime.parser.node.ASTReference.render(org.apache.velocity.context.InternalContextAdapter, java.io.Writer) @bci=30, line=332 (Interpreted frame)

- org.apache.velocity.runtime.parser.node.SimpleNode.render(org.apache.velocity.context.InternalContextAdapter, java.io.Writer) @bci=21, line=336 (Interpreted frame)

- org.apache.velocity.Template.merge(org.apache.velocity.context.Context, java.io.Writer, java.util.List) @bci=251, line=328 (Interpreted frame)

- org.apache.velocity.Template.merge(org.apache.velocity.context.Context, java.io.Writer) @bci=4, line=235 (Interpreted frame)

- com.xx.china.cmshollywood.cms.engine.DefaultVelocityTemplateEngine.mergeTemplate(org.apache.velocity.context.Context, java.lang.String, int) @bci=25, line=416 (Interpreted frame)