併發過大druid異常:Thread pool is EXHAUSTED!
阿新 • • 發佈:2018-12-10
當併發量太高,就會導致部分請求呼叫服務失敗,我們就需要定位瓶頸在哪裡,再對症下藥,本篇就是為了說明執行緒池資源耗盡的情況下該怎麼辦。 專案原始碼地址
2. 異常描述
執行,會看到部分請求失敗: 在專案控制檯會看到報錯資訊:threadpool is exhausted,執行緒池耗盡。
2018-09-14 11:04:30.365 [http-nio-1111-exec-435] ERROR cn.lilyssh.common.exception.ExceptionAdviceHandler - 處理出現異常
com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method save in the service cn. lilyssh.order.api.service.OrderServiceApi. Tried 3 times of the providers [192.168.31.31:3333] (1/1) from the registry ssh.qianxunclub.com:2181 on the consumer 192.168.31.31 using the dubbo version 2.6.0. Last error is: Failed to invoke remote method: save, provider: dubbo://192.168.31.31:3333/cn.lilyssh. order.api.service.OrderServiceApi?anyhost=true&application=order-consumer&check=false&default.check=false&dubbo=2.6.0&generic=false&interface=cn.lilyssh.order.api.service.OrderServiceApi&methods=save,list&pid=2372®ister.ip=192.168.31.31&remote.timestamp=1536893088718 &side=consumer×tamp=1536893890966, cause: Server side(192.168.31.31,3333) threadpool is exhausted ,detail msg:Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-192.168.31.31:3333, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 4395 (completed: 4195), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://192.168.31.31:3333!
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:101)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: com.alibaba.dubbo.remoting.RemotingException: Server side(192.168.31.31,3333) threadpool is exhausted ,detail msg:Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-192.168.31.31:3333, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 4395 (completed: 4195), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://192.168.31.31:3333!
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.returnFromResponse(DefaultFuture.java:218)
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77)
... 69 common frames omitted
3. 異常分析
從異常的資訊看,有兩種可能:
dubbo
執行緒池耗盡druid
連線池耗盡 先增加dubbo
執行緒池的數量,預設為200。改大後又報超時的錯誤:
2018-09-14 15:04:45.223 [http-nio-1111-exec-194] ERROR cn.lilyssh.common.exception.ExceptionAdviceHandler - 處理出現異常
com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method exist in the service cn.lilyssh.user.api.service.UserServiceApi. Tried 3 times of the providers [192.168.31.31:6666] (1/1) from the registry ssh.qianxunclub.com:2181 on the consumer 192.168.31.31 using the dubbo version 2.6.0. Last error is: Invoke remote method timeout. method: exist, provider: dubbo://192.168.31.31:6666/cn.lilyssh.user.api.service.UserServiceApi?anyhost=true&application=order-consumer&check=false&default.check=false&dubbo=2.6.0&generic=false&interface=cn.lilyssh.user.api.service.UserServiceApi&methods=exist,list&pid=3307®ister.ip=192.168.31.31&remote.timestamp=1536908653179&side=consumer×tamp=1536908663628, cause: Waiting server-side response timeout. start time: 2018-09-14 15:04:43.837, end time: 2018-09-14 15:04:44.860, client elapsed: 1 ms, server elapsed: 1022 ms, timeout: 1000 ms, request: Request [id=590, version=2.0.0, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=exist, parameterTypes=[class java.lang.Integer], arguments=[4], attachments={path=cn.lilyssh.user.api.service.UserServiceApi, interface=cn.lilyssh.user.api.service.UserServiceApi, version=0.0.0}]], channel: /192.168.31.31:61708 -> /192.168.31.31:6666
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:101)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: com.alibaba.dubbo.remoting.TimeoutException: Waiting server-side response timeout. start time: 2018-09-14 15:04:43.837, end time: 2018-09-14 15:04:44.860, client elapsed: 1 ms, server elapsed: 1022 ms, timeout: 1000 ms, request: Request [id=590, version=2.0.0, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=exist, parameterTypes=[class java.lang.Integer], arguments=[4], attachments={path=cn.lilyssh.user.api.service.UserServiceApi, interface=cn.lilyssh.user.api.service.UserServiceApi, version=0.0.0}]], channel: /192.168.31.31:61708 -> /192.168.31.31:6666
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:134)
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77)
... 69 common frames omitted
把timeout
預設為1秒,改為5秒:
spring:
dubbo:
registry:
address: zookeeper://ssh.qianxunclub.com:2181
provider:
port: 6666
# threads: 1000
timeout: 5000
# protocol:
# threads: 1000
依舊報錯:threadpool is exhausted. druid最大連線數max-active預設為8,改為20:
spring:
datasource:
druid:
url: jdbc:mysql://db.qianxunclub.com:3306/demo
username: xxx
password: xxx
max-active: 20
繼續測試,jmeter
報錯:
java.net.NoRouteToHostException: Can't assign requested address (Address not available)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.jmeter.protocol.http.sampler.hc.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:318)
at org.apache.jmeter.protocol.http.sampler.MeasuringConnectionManager$MeasuredConnection.open(MeasuringConnectionManager.java:114)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:697)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:455)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1189)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1178)
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:490)
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:416)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:250)
at java.lang.Thread.run(Thread.java:748)
修改了jmeter
的implementation
引數設定,就解決了報錯的問題。修改如下:
4. 解決辦法
druid
最大連線數max-active
預設為8,改為20:
spring:
datasource:
druid:
url: xxx
username: xxx
password: xxx
max-active: 20
大功告成!