【深度學習】【caffe實用工具3】筆記25 Windows下caffe中將影象資料集合轉換為DB(LMDB/LEVELDB)檔案格式之convert_imageset
阿新 • • 發佈：2019-01-09
/********************************************************************************************************************************* 檔案說明: 【1】This program converts a set of images to a lmdb/leveldb by storing them as Datum proto buffers. 【2】這個程式用於將影象資料轉化為DB(lmdb/leveldb)資料庫檔案 【3】where ROOTFOLDER is the root folder that holds all the images, and LISTFILE should be a list of files as well as their labels, in the format as subfolder1/file1.JPEG 7 用 法: 【注意1】*.bat檔案的使用方法: convert_imageset [FLAGS] ROOTFOLDER/ LISTFILE DB_NAME 【1】convert_imageset:convert_imageset.exe檔案所在的路徑 【2】[FLAGS]:可選引數的配置 【3】ROOTFOLDER/:圖片集合所在的資料夾所在的路徑，如下所示: argv[1] = "E://caffeInstall2013//caffe-master//data//myself//train//"; 【4】LISTFILE:圖片檔案列表檔案所在的路徑，如下所示: argv[2] = "E://caffeInstall2013//caffe-master//data//myself//train.txt"; 【5】DB_NAME:生成的DB檔案存放的路徑以及生成DB資料庫的資料庫名字 【注意2】直接在程式碼中進行配置，執行: 【1】首先根據自己要轉換圖片資料集合的圖片規格等，對可選引數進行配置 【2】新增如下的程式碼: argv[1] = "E://caffeInstall2013//caffe-master//data//myself//train//"; argv[2] = "E://caffeInstall2013//caffe-master//data//myself//train.txt"; argv[3] = "E://caffeInstall2013//caffe-master//examples//myself//myself_train_lmdb"; 【注意3】:*.bat檔案形式的具體使用方法，可以參考下面的部落格: http://www.cnblogs.com/LiuSY/p/5761781.html 操作步驟: 【1】首先，在目錄E:\caffeInstall2013\caffe-master\data下建立自己的影象資料集合檔案，例如train,val 【2】其次，再建立這些影象檔案的檔案列表train.txt val.txt 【3】然後，配置程式(就是本程式) 【4】執行程式，生成DB資料庫檔案 開發環境: windows+cuda7.5+cuDnnV5+opencv+caffe1+vs2013 時間地點: 陝西師範大學 文津樓 2017.8.9 作 者: 九 月 **********************************************************************************************************************************/ #include <algorithm> #include <fstream> // NOLINT(readability/streams) #include <string> #include <utility> #include <vector> #include "boost/scoped_ptr.hpp" #include "gflags/gflags.h" #include "glog/logging.h" #include "caffe/proto/caffe.pb.h" #include "caffe/util/db.hpp" #include "caffe/util/format.hpp" #include "caffe/util/io.hpp" #include "caffe/util/rng.hpp" using namespace caffe; // NOLINT(build/namespaces) using std::pair; using boost::scoped_ptr; /********************************************************************************************************************** 模組說明: 可選引數的定義 引數說明: 【1】-gray: 是否以灰度圖的方式開啟圖片。程式呼叫opencv庫中的imread()函式來開啟圖片，預設為false 【2】-shuffle: 是否隨機打亂圖片順序。預設為false 【3】-backend:需要轉換成的db檔案格式，可選為leveldb或lmdb,預設為lmdb 【4】-resize_width/resize_height: 改變圖片的大小。在執行中，要求所有圖片的尺寸一致，因此需要改變圖片大小。 程式呼叫opencv庫的resize（）函式來對圖片放大縮小，預設為0，不改變 【5】-check_size: 檢查所有的資料是否有相同的尺寸。預設為false,不檢查 【6】-encoded: 是否將原圖片編碼放入最終的資料中，預設為false 【7】-encode_type: 與前一個引數對應，將圖片編碼為哪一個格式：‘png','jpg'...... ***********************************************************************************************************************/ DEFINE_bool(gray, false,"When this option is on, treat images as grayscale ones"); DEFINE_bool(shuffle, false,"Randomly shuffle the order of images and their labels"); DEFINE_string(backend, "lmdb","The backend {lmdb, leveldb} for storing the result"); DEFINE_int32(resize_width, 256, "Width images are resized to"); DEFINE_int32(resize_height, 256, "Height images are resized to"); DEFINE_bool(check_size, false,"When this option is on, check that all the datum have the same size"); DEFINE_bool(encoded, false,"When this option is on, the encoded image will be save in datum"); DEFINE_string(encode_type, "", "Optional: What type should we encode the image as ('png','jpg',...)."); int main(int argc, char** argv) { #ifdef USE_OPENCV ::google::InitGoogleLogging(argv[0]); // Print output to stderr (while still logging) FLAGS_alsologtostderr = 1; #ifndef GFLAGS_GFLAGS_H_ namespace gflags = google; #endif gflags::SetUsageMessage("Convert a set of images to the leveldb/lmdb\n" "format used as input for Caffe.\n" "Usage:\n" " convert_imageset [FLAGS] ROOTFOLDER/ LISTFILE DB_NAME\n" "The ImageNet dataset for the training demo is at\n" "http://www.image-net.org/download-images\n"); argv[1] = "E://caffeInstall2013//caffe-master//data//myself//train//"; argv[2] = "E://caffeInstall2013//caffe-master//data//myself//train.txt"; argv[3] = "E://caffeInstall2013//caffe-master//examples//myself//myself_train_lmdb"; const bool is_color = !FLAGS_gray; const bool check_size = FLAGS_check_size; const bool encoded = FLAGS_encoded; const string encode_type = FLAGS_encode_type; std::ifstream infile(argv[2]); std::vector<std::pair<std::string, int> > lines; std::string line; size_t pos; int label; while (std::getline(infile, line)) { pos = line.find_last_of(' '); label = atoi(line.substr(pos + 1).c_str()); lines.push_back(std::make_pair(line.substr(0, pos), label)); } if (FLAGS_shuffle) { // randomly shuffle data LOG(INFO) << "Shuffling data"; shuffle(lines.begin(), lines.end()); } LOG(INFO) << "A total of " << lines.size() << " images."; if (encode_type.size() && !encoded) { LOG(INFO) << "encode_type specified, assuming encoded=true."; } int resize_height = std::max<int>(0, FLAGS_resize_height); int resize_width = std::max<int>(0, FLAGS_resize_width); // Create new DB scoped_ptr<db::DB> db(db::GetDB(FLAGS_backend)); db->Open(argv[3], db::NEW); scoped_ptr<db::Transaction> txn(db->NewTransaction()); // Storing to db std::string root_folder(argv[1]); Datum datum; int count = 0; int data_size = 0; bool data_size_initialized = false; for (int line_id = 0; line_id < lines.size(); ++line_id) { bool status; std::string enc = encode_type; if (encoded && !enc.size()) { // Guess the encoding type from the file name string fn = lines[line_id].first; size_t p = fn.rfind('.'); if ( p == fn.npos ) LOG(WARNING) << "Failed to guess the encoding of '" << fn << "'"; enc = fn.substr(p); std::transform(enc.begin(), enc.end(), enc.begin(), ::tolower); } status = ReadImageToDatum(root_folder + lines[line_id].first,lines[line_id].second, resize_height, resize_width, is_color, enc, &datum); if (status == false) continue; if (check_size) { if (!data_size_initialized) { data_size = datum.channels() * datum.height() * datum.width(); data_size_initialized = true; } else { const std::string& data = datum.data(); CHECK_EQ(data.size(), data_size) << "Incorrect data field size "<< data.size(); } } // sequential string key_str = caffe::format_int(line_id, 8) + "_" + lines[line_id].first; // Put in db string out; CHECK(datum.SerializeToString(&out)); txn->Put(key_str, out); if (++count % 1000 == 0) { // Commit db txn->Commit(); txn.reset(db->NewTransaction()); LOG(INFO) << "Processed " << count << " files."; } } // write the last batch if (count % 1000 != 0) { txn->Commit(); LOG(INFO) << "Processed " << count << " files."; } std::system("pause"); #else LOG(FATAL) << "This tool requires OpenCV; compile with USE_OPENCV."; #endif // USE_OPENCV return 0; }
convert.bat的格式為
convert_imageset.exe的位置+空格+FLAGS+空格+圖片所在的位置+空格+你生成的list的位置+空格+將要生成的db格式要儲存的位置
建議都使用絕對位置！！！
例子：
D:/deeptools/caffe-windows-master/bin/convert_imageset.exe --shuffle --resize_height=256 --resize_width=256 D:/deeptools/caffe-windows-master/data/re/ D:/deeptools/caffe-windows-master/examples/myfile/train.txt D:/deeptools/caffe-windows-master/examples/myfile/train_db pause
其中FLAGS可以選擇為：
（1）--shuffle 是否隨機打亂圖片順序 【預設為false】
D:/deeptools/caffe-windows-master/bin/convert_imageset.exe --shuffle D:/deeptools/caffe-windows-master/data/mnist/train-images/ D:/deeptools/caffe-windows-master/examples/mymnist/train.txt D:/deeptools/caffe-windows-master/examples/mymnist/train_lmdb
pause
為什麼要隨機打亂圖片順序？
待答。。。
（2）--gray 是否以灰度圖片的方式開啟【預設為false】
D:/deeptools/caffe-windows-master/bin/convert_imageset.exe --gray D:/deeptools/caffe-windows-master/data/mnist/train-images/ D:/deeptools/caffe-windows-master/examples/mymnist/train.txt D:/deeptools/caffe-windows-master/examples/mymnist/traingray_lmdb pause
（3）--resize_width
--resize_height 改變圖片大小（縮放）【預設為原圖】
（4）--backend 需要轉換成什麼格式的db，可選為leveldb與lmdb格式【預設為lmdb】
D:/deeptools/caffe-windows-master/bin/convert_imageset.exe --backend=leveldb D:/deeptools/caffe-windows-master/data/mnist/train-images/ D:/deeptools/caffe-windows-master/examples/mymnist/train.txt D:/deeptools/caffe-windows-master/examples/mymnist/trainbackend_leveldb
pause
結果：
現在我們認真解讀一下這個leveldb格式：
http://www.2cto.com/kf/201607/527860.html
待續。。。不知道里面的糾結是什麼東西
（5）--check_size 檢查所有的資料是否為同一個size【預設為false，不檢查】
（6）--encoded 是否將原圖編碼放入最終的資料中【預設為false】
（7）--encode_type 與前邊呼應，將圖片改為哪種格式【png，jpg。。】
貌似這個得需要opencv。。我沒有安裝opencv出錯如下
