1 Visualizing Activations

Show the activations of the network during the forward pass. It turns out that the activations usually start out looking relatively blobby and dense, but as the training progresses the activations usually become more sparse and localized.

symptom of high learning rates.

2 Visualize Patches that Maximally Activate Neurons

3 Visualize the Raw Weights

Only interpretable on the first layer’s weights (Gabor like features) since the first layer has contact with the raw image. You can still do it for higher layers, it is just not that interesting because it looks at the previous activation values but not the raw image.

Noisy patterns can be an indicator of a network that hasn’t been trained for long enough, or possibly a very low regularization strength that may have led to overfitting.

4 Visualize the Representation Space
For example, use t-SNE visualization on fc7 features (crucially, features after the ReLU non-linearity). It embed high-dimensional points so that locallly, pairwise distances are
conserved. 

ConvNets can be interpreted as gradually transforming the images into a representation in which the classes are separable by a linear classifier. It shows that the similarities are more often class-based and semantic rather than pixel and color-based.

5 Occlusion Experiments

See [Zeiler and Fergus. 2013]

6 Deconv Approaches
This topic will be covered in [Simonyan et al. 2014], [Dosovitskiy et al. 2015], and [Zeiler and Fergus. 2013]. This approach is image-specific. It only requires a single backward
pass.

6.1 Goal
Compute the gradient of any arbitrary neuron in the network wrt the image.

It shows the input pattern that is most strongly activating this neuron (and hence the part that is most discriminative to it). That is, the gradient is telling that making a small step of the image towards the gradient’s direction will have a locally positive influence on this neuron’s activation. The gradient indicates which pixels need to be changed the least to affect the class score the most.

6.2 Method
Pick a layer, set the gradient there to be all zero except for one 1 for some neuron of interest. Then backprop to image. We will modify the back pass of relu layer, and otherwise we will back through the image.

7 Optimization Approaches
See [Simonyan et al. 2014]. This approach is not specific to any particular image.

8
References
[1]. F.-F. Li, A. Karpathy, and J. Johnson. http://cs231n.stanford.edu/slides/winter1516_lecture9.pdf.