Select Via Proxy: Efficient Data Selection For Training DeepNetworks

2019ICLR的文章，介紹了Select Via Proxy（SVP）通過較小規模的模型來確定樣本的uncertainty進而決定複雜模型訓練使用樣本的策略。

ABSTRACT

we develop a novel approach to efficiently select a subset of training data to achieve faster training with no loss in model predictive performance.

這是論文的主要貢獻。

In our approach, we first train a small proxy model quickly, which we then use to estimate the utility of individual training data points, and then select the most informative ones for training the large target model.

先訓練一些小的proxy模型去估計訓練樣本的實用性，然後用資訊最豐富（uncertainty最高）的樣本去訓練目標模型。

1 INTRODUCTION

介紹了一下SVP的目的和uniform subsampling和core-set selection兩種方法的缺點。簡單介紹了SVP的流程（圖1）和結果。

2 RELATED WORK

介紹了關於sampling和training data selection的一些文獻，有興趣的可以看一下。

3 SELECT VIA PROXY

3.1 CREATING A PROXY MODEL

總共就三步：

1）Create a proxy model that is fast to train and to provide us with an approximate decision boundary, 2) use the proxy model to select a subset of uncertain data points around the decision boundary, and 3) train the large target model on the selected subset via proxy to refine the decision boundary and get the final accurate model.

這裡作者提出來兩個觀點：

It has been observed that for deep models with many layers, reducing the dimension (narrowing) or number of hidden layers (shortening) leads to a considerably reduced training times with only a small drop in accuracy.

a significant amount of training is spent to obtain a relatively small reduction in error.

這是使用SVP的基本立足點。

3.2 SUBSET SELECTION VIA PROXY

Quantifying uncertainty

介紹了有以下幾種方法：

然後根據Algorithm 1對目標模型進行更新：

4 RESULTS

使用不同的proxy和不同的資料集進行了實驗，結果顯示在達到同樣誤差下，使用了SVP的訓練時間短訓練樣本少。圖5展示不同的uncertainty metrics效果的差異。表2展示了採用了proxy model進行example selection導致的樣本類別不均衡。隨著訓練的繼續和子訓練集規模的增加資料將變得均衡，但從每一個類別裡通過uncertainty選取相同數量的examples往往會導致模型表現不佳。

5 CONCLUSION

提出了一種通過proxy model來選擇子訓練集來訓練深度學習模型的一種方法買這種方法會使得在同樣精度的情況下訓練速度的加快。由於不改變目標模型的訓練，因此此方法可以與任意的模型配合使用。