最近在使用tensorflow的時候出現了以下所示的bug：

2017-11-08 12:24:52.838039: E tensorflow/stream_executor/cuda/cuda_driver.cc:1080] failed to synchronize the stop event: CUDA_ERROR_ILLEGAL_ADDRESS
2017-11-08 12:24:52.838090: E tensorflow/stream_executor/cuda/cuda_timer.cc:54] Internal: error destroying CUDA event in context 0x51f18f0: CUDA_ERROR_ILLEGAL_ADDRESS
2017-11-08 12:24:52.838106: E tensorflow/stream_executor/cuda/cuda_timer.cc:59] Internal: error destroying CUDA event in context 0x51f18f0: CUDA_ERROR_ILLEGAL_ADDRESS
2017-11-08 12:24:52.838137: F tensorflow/stream_executor/cuda/cuda_dnn.cc:3218] failed to set stream for cudnn handle: CUDNN_STATUS_MAPPING_ERROR
 

由於我的模型中使用了tf.contrib.image.transform層，而該層在tensorflow1.7,1.8中，ubuntu下gpu版本沒問題。windows下cpu版本正常，gpu版本存在bug。解決思路如https://github.com/tensorflow/tensorflow/issues/17485所示，將圖中的tf.contrib.image.transform放在cpu上，改為：

with tf.device('/cpu:0'):
    data_node_transformed = tf.contrib.image.transform(imgs, rtm, "BILINEAR")
 

重新進行模型訓練，部署。

期待後續tensorflow和nvidia可以解決該bug。