2018.02.27 leaning journal
by 趙木木

1. caffe-segnet 訓練探索

鍵入訓練命令：

relaybot@ubuntu:~/mumu$ ./SegNet/caffe-segnet/build/tools/caffe train -gpu 0 -solver ./SegNet/Models/segnet_solver.prototxt

segnet-caffe 訓練遇到錯誤：Check failed: error == cudaSuccess (2 vs. 0) out of memory

網上有解釋是：batch_size太大了，一次性讀入的圖片太多了，所以就超出了視訊記憶體。因此需要將train.prototxt

然後是訓練時間耗時很長，我在Models/segnet_solver.prototxt檔案中修改了下訓練的大小，有些名字不太懂，其實這類問題都可以上 github 的 Issues 中找，後面有些問題也是通過檢視Issues解決的。

前面在Models/segnet_solver.prototxt檔案中，將max_iter設定為1000，也就是需要迭代1000次才能訓練結束，display設定為20，也就是每迭代20次顯示一次迭代資訊，我的筆記本這次訓練耗時一個小時，換成別的顯示卡肯定快許多。

I0227 09
 
:27:29.151870  3754 solver.cpp:228] Iteration 960, loss = 0.465969
I0227 09:27:29.151957  3754 solver.cpp:244]     Train net output #0: accuracy = 0.255446
I0227 09:27:29.151971  3754 solver.cpp:244]     Train net output #1: loss = 0.465969 (* 1 = 0.465969 loss)
I0227 09:27:29.151979  3754 solver.cpp:244]     Train net output #2: per_class_accuracy = 0
 

I0227 09:27:29.151986  3754 solver.cpp:244]     Train net output #3: per_class_accuracy = 0.0160278
I0227 09:27:29.151993  3754 solver.cpp:244]     Train net output #4: per_class_accuracy = 0
I0227 09:27:29.151998  3754 solver.cpp:244]     Train net output #5: per_class_accuracy = 0.949324
I0227 09:27:29.152004  3754 solver.cpp:244]     Train net output #6: per_class_accuracy = 0.106586
I0227 09:27:29.152011  3754 solver.cpp:244]     Train net output #7: per_class_accuracy = 0
I0227 09:27:29.152017  3754 solver.cpp:244]     Train net output #8: per_class_accuracy = 0
I0227 09:27:29.152024  3754 solver.cpp:244]     Train net output #9: per_class_accuracy = 0
I0227 09:27:29.152029  3754 solver.cpp:244]     Train net output #10: per_class_accuracy = 0
I0227 09:27:29.152035  3754 solver.cpp:244]     Train net output #11: per_class_accuracy = 0
I0227 09:27:29.152040  3754 solver.cpp:244]     Train net output #12: per_class_accuracy = 0
I0227 09:27:29.152048  3754 sgd_solver.cpp:106] Iteration 960, lr = 0.01
I0227 09:28:46.873960  3754 solver.cpp:228] Iteration 980, loss = 0.859281
I0227 09:28:46.874079  3754 solver.cpp:244]     Train net output #0: accuracy = 0.473837
I0227 09:28:46.874097  3754 solver.cpp:244]     Train net output #1: loss = 0.859281 (* 1 = 0.859281 loss)
I0227 09:28:46.874106  3754 solver.cpp:244]     Train net output #2: per_class_accuracy = 0.978743
I0227 09:28:46.874114  3754 solver.cpp:244]     Train net output #3: per_class_accuracy = 0.108868
I0227 09:28:46.874120  3754 solver.cpp:244]     Train net output #4: per_class_accuracy = 0.0167785
I0227 09:28:46.874127  3754 solver.cpp:244]     Train net output #5: per_class_accuracy = 0.918752
I0227 09:28:46.874135  3754 solver.cpp:244]     Train net output #6: per_class_accuracy = 0.675187
I0227 09:28:46.874141  3754 solver.cpp:244]     Train net output #7: per_class_accuracy = 0.00214325
I0227 09:28:46.874147  3754 solver.cpp:244]     Train net output #8: per_class_accuracy = 0.915285
I0227 09:28:46.874155  3754 solver.cpp:244]     Train net output #9: per_class_accuracy = 0
I0227 09:28:46.874161  3754 solver.cpp:244]     Train net output #10: per_class_accuracy = 0
I0227 09:28:46.874168  3754 solver.cpp:244]     Train net output #11: per_class_accuracy = 0.919846
I0227 09:28:46.874174  3754 solver.cpp:244]     Train net output #12: per_class_accuracy = 0
I0227 09:28:46.874182  3754 sgd_solver.cpp:106] Iteration 980, lr = 0.01
I0227 09:30:00.935937  3754 solver.cpp:454] Snapshotting to binary proto file /home/relaybot/mumu/SegNet/Models/Training/segnet_iter_1000.caffemodel
I0227 09:30:03.815346  3754 sgd_solver.cpp:273] Snapshotting solver state to binary proto file /home/relaybot/mumu/SegNet/Models/Training/segnet_iter_1000.solverstate
I0227 09:30:05.194144  3754 solver.cpp:317] Iteration 1000, loss = 0.789985
I0227 09:30:05.194195  3754 solver.cpp:337] Iteration 1000, Testing net (#0)
I0227 09:31:08.651491  3754 solver.cpp:404]     Test net output #0: accuracy = 0.474696
I0227 09:31:08.651664  3754 solver.cpp:404]     Test net output #1: loss = 0.772579 (* 1 = 0.772579 loss)
I0227 09:31:08.651679  3754 solver.cpp:404]     Test net output #2: per_class_accuracy = 0.945665
I0227 09:31:08.651688  3754 solver.cpp:404]     Test net output #3: per_class_accuracy = 0.0743196
I0227 09:31:08.651698  3754 solver.cpp:404]     Test net output #4: per_class_accuracy = 0.0442652
I0227 09:31:08.651707  3754 solver.cpp:404]     Test net output #5: per_class_accuracy = 0.868098
I0227 09:31:08.651716  3754 solver.cpp:404]     Test net output #6: per_class_accuracy = 0.348668
I0227 09:31:08.651726  3754 solver.cpp:404]     Test net output #7: per_class_accuracy = 0.0093155
I0227 09:31:08.651734  3754 solver.cpp:404]     Test net output #8: per_class_accuracy = 0.809386
I0227 09:31:08.651743  3754 solver.cpp:404]     Test net output #9: per_class_accuracy = 0
I0227 09:31:08.651751  3754 solver.cpp:404]     Test net output #10: per_class_accuracy = 0.0136846
I0227 09:31:08.651759  3754 solver.cpp:404]     Test net output #11: per_class_accuracy = 0.655417
I0227 09:31:08.651768  3754 solver.cpp:404]     Test net output #12: per_class_accuracy = 0.0351428
I0227 09:31:08.651777  3754 solver.cpp:322] Optimization Done.
I0227 09:31:08.651783  3754 caffe.cpp:254] Optimization Done.
relaybot@ubuntu:~/mumu$ 

訓練結束，在/SegNet/Models/Training資料夾底下輸出結果：

/SegNet/Models/Inference/

鍵入命令：

relaybot@ubuntu:~/mumu$ python ./SegNet/Scripts/compute_bn_statistics.py ./SegNet/Models/segnet_train.prototxt ./SegNet/Models/Training/segnet_iter_1000.caffemodel ./SegNet/Models/Inference/

生成權重檔案：

鍵入命令顯示影象輸出結果，此部分程式碼是python檔案：test_segmentation_camvid.py，其中事先在檔案中更改目錄；

relaybot@ubuntu:~/mumu$ python ./SegNet/Scripts/test_segmentation_camvid.py --model ./SegNet/Models/segnet_inference.prototxt --weights ./SegNet/Models/Inference/test_weights.caffemodel --iter 233

會生成三張圖，需要將三張圖都關閉後，才會顯示下一個圖片物件的輸入輸出結果；