redis連線異常的解決方案

最近在做高併發架構，發現程式偶爾報錯：

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
        at redis.clients.util.Pool.getResource(Pool.java:50) ~[Pool.class:na]
        at redis.clients.jedis.JedisPool.getResource(JedisPool.java:86) ~[JedisPool.class:na]
        at com.qingmayun.common.dbConfig.JedisManager.getDbJedis(JedisManager.java:56) ~[JedisManager.class:na]
        at com.qingmayun.common.dbConfig.PipelineManager.getDbPipeline(PipelineManager.java:21) [PipelineManager.class:na]
        at com.qingmayun.common.dbConfig.RedisDao.smsMonitWhenSend(RedisDao.java:1227) [RedisDao.class:na]
        at com.qingmayun.mqconsumer.channel.base.ChannelWatcher.sendMonit(ChannelWatcher.java:79) [ChannelWatcher.class:na]
        at com.qingmayun.mqconsumer.channel.base.proxy.SmsProxy.sendMsg(SmsProxy.java:71) [SmsProxy.class:na]
        at com.qingmayun.mqconsumer.TemplateSmsConsumer.sendTemplateSms(TemplateSmsConsumer.java:205) [TemplateSmsConsumer.class:na]
        at com.qingmayun.mqconsumer.TemplateSmsConsumer.access$100(TemplateSmsConsumer.java:50) [TemplateSmsConsumer.class:na]
        at com.qingmayun.mqconsumer.TemplateSmsConsumer$1.handleDelivery(TemplateSmsConsumer.java:82) [TemplateSmsConsumer$1.class:na]
        at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:144) [rabbitmq-client.jar:na]
        at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:99) [rabbitmq-client.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: connect timed out
        at redis.clients.jedis.Connection.connect(Connection.java:155) ~[Connection.class:na]
        at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:83) ~[BinaryClient.class:na]
        at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1643) ~[BinaryJedis.class:na]
        at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:85) ~[JedisFactory.class:na]
        at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861) ~[commons-pool2-2.3.jar:2.3]
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435) ~[commons-pool2-2.3.jar:2.3]
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.3.jar:2.3]
        at redis.clients.util.Pool.getResource(Pool.java:48) ~[Pool.class:na]
        ... 14 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.7.0_71]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[na:1.7.0_71]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[na:1.7.0_71]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[na:1.7.0_71]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.7.0_71]
        at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_71]
        at redis.clients.jedis.Connection.connect(Connection.java:149) ~[Connection.class:na]
        ... 21 common frames omitted
 

百度了redis的Could not get a resource from the pool和java.net.SocketTimeoutException: connect timed out始終得不到理想答案。網路上的說法無非就是使用執行緒池，增加連線時長，增加執行緒池數量。已經照做了，但是並沒啥鳥用，壓力山大，再解決不了沒臉見人了。

自力更生吧，檢視cpu，30%；記憶體，還剩很多；網路，同一臺機器；硬碟，讀寫無瓶頸；Redis配置，使用預設的最大10000連線，當前活動連線也就1K多；swap，並無使用，Redis也已經刪除資料重啟了；jvm，XMX配置了4G，才使用1G；tcpdump，抓到了包，然而看不懂。網路上一個團隊說他們通過抓包最終找到了問題是他們顯示卡驅動不行，找供應商升級了驅動

哥只是一個java小屌而已，基本的問題查查還行，太深了還真沒那個水平，而且據說還可能是redis版本或者相容性這種讓人徹底無語的原因。

自己寫了個小程式for迴圈開2000個執行緒去連線池拿連線，拿到後休息10秒，列印成功和失敗。多次測試發現，前面1800+的連線總能拿成功，後面就失敗了，失敗量隨機的，而且RP爆發的一次竟然全部獲取成功了。執行緒數改到1000，RP好的時全部成功，最常見的還是後面幾十個失敗。看來執行緒數多一些少一些也總有一些拿不到連線。記得一句話，JVM線上程很多的時候不穩定，而且網路偶爾丟包或者中斷那麼幾毫秒還是很常見的，作為程式設計師需要考慮這種極端的環境，而不要太相信機器太相信邏輯。

好，改造了下獲取連線的程式碼。

	public static Jedis getJedis()
	{
		int timeoutCount = 0;
		while (true) // 如果是網路超時則多試幾次
		{
			try
			{
				Jedis jedis = RedisPool.getResource();
				return jedis;
			} catch (Exception e)
			{
				// 底層原因是SocketTimeoutException，不過redis已經捕捉且丟擲JedisConnectionException，不繼承於前者
				if (e instanceof JedisConnectionException || e instanceof SocketTimeoutException)
				{
					timeoutCount++;
					log.warn("getJedis timeoutCount={}", timeoutCount);
					if (timeoutCount > 3)
					{
						break;
					}
				}else
				{
					log.warn("jedisInfo。NumActive=" + RedisPool.getBalancePool().getNumActive() + ", NumIdle="
							+ RedisPool.getNumIdle() + ", NumWaiters="
							+ RedisPool.getNumWaiters() + ", isClosed="
							+ RedisPool.isClosed());
					log.error("getJedis error", e);
					break;
				}
			}
		}
		return null;
	}

同時，連線池的獲取超時時間改成5秒（預設2秒）。扔到伺服器一跑，再無ERROR。偶爾會列印getJedis timeoutCount=1，極少數getJedis timeoutCount=2。領盒飯去了。