1. 程式人生 > >記一次jstack命令定位問題

記一次jstack命令定位問題

今天天氣不錯,但是趕上惡意加班心情就不爽,懷著不爽的心情幹活,總能創造出更多的問題,這不,今天就自己挖了一個坑,自己跳進去了,好在上來了

經過是這樣的,開始除錯canal採集binlog時,由於添加了一個上報數量大小,隨手列印了一個日誌

logger.info("batchData:{}", batchData);
pipeline.put(batchData);

發到sit環境測試一把,臥槽。。。,怎麼不同步資料了,開始排查問題,是我改的程式碼有問題?於是回退了,還是不同步資料,但是發現在停止任務時會中斷讀執行緒,這時資料同步過來了,試了幾次都是這樣,開始懷疑讀執行緒被阻塞了,於是乎祭出許久未用的jstack去看一下讀執行緒狀態:jstack pid

"canal-binlog-encrypt-wangbd-test-001,219025,ReaderTask_CanalReader_0_0" #32 prio=5 os_prio=0 tid=0x00007fdfdc010800 nid=0x4199 waiting on condition [0x00007fe044653000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000800ad988> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at com.data.integration.engine.stream.core.cache.memory.MemoryDataCache.get(MemoryDataCache.java:135)
        at com.data.integration.engine.stream.core.cache.DataCache.get(DataCache.java:65)
        at com.data.integration.engine.stream.core.cache.DataCache.get(DataCache.java:61)
        at com.data.integration.engine.stream.core.config.pipeline.Pipeline.getBatchData(Pipeline.java:74)
        at com.alibaba.fastjson.serializer.ASMSerializer_4_Pipeline.write(Unknown Source)
        at com.alibaba.fastjson.serializer.ASMSerializer_1_DefaultBatchData.write(Unknown Source)
        at com.alibaba.fastjson.serializer.JSONSerializer.write(JSONSerializer.java:285)
        at com.alibaba.fastjson.JSON.toJSONString(JSON.java:740)
        at com.alibaba.fastjson.JSON.toJSONString(JSON.java:678)
        at com.alibaba.fastjson.JSON.toJSONString(JSON.java:643)
        at com.data.integration.engine.stream.plugin.reader.canalreader.CanalReader.start(CanalReader.java:127)
        at com.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

發現居然在使用fastjson轉換json時阻塞在快取佇列上了,說一下快取佇列讀執行緒只會向裡面寫,寫現場才會讀,於是乎開始懷疑列印的這個日誌,順著思路找到了如下程式碼:

pipeline的batchData屬性居然是從快取中獲取的物件

public BatchData getBatchData(){
        return dataCache.get();
    }

這就難怪了,由於快取佇列是空的,在toJson時阻塞了,導致整個讀執行緒被阻塞,把這個日誌去掉後就正常了,正常的讀執行緒棧是這樣的:

"canal-binlog-encrypt-wangbd-test-001,219023,ReaderTask_CanalReader_0_0" #32 prio=5 os_prio=0 tid=0x00007f7c8c00b000 nid=0xefb waiting on condition [0x00007f7d0476d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000807bf660> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at com.alibaba.otter.canal.store.memory.MemoryEventStoreWithBuffer.get(MemoryEventStoreWithBuffer.java:219)
        at com.alibaba.otter.canal.server.embedded.CanalServerWithEmbedded.getEvents(CanalServerWithEmbedded.java:538)
        at com.alibaba.otter.canal.server.embedded.CanalServerWithEmbedded.getWithoutAck(CanalServerWithEmbedded.java:347)
        - locked <0x00000000808a3448> (a com.alibaba.otter.canal.instance.manager.CanalInstanceWithManager)
        at com.data.integration.engine.stream.plugin.reader.canalreader.CanalReader.start(CanalReader.java:101)
        at com.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

&n