此次主要由自我學習過度到深度學習，簡單記錄如下：

(1)深度學習比淺層網路學習對特徵具有更優異的表達能力和緊密簡潔的表達了比淺層網路大的多的函式集合。

(2)將傳統的淺層神經網路進行擴充套件會存在資料獲取、區域性最值和梯度彌散的缺點。

(3)棧式自編碼神經網路是由多層稀疏自編碼器構成的神經網路(最後一層採用的softmax迴歸或者logistic迴歸分類)，採用逐層貪婪的訓練方法得到初始的引數，這樣在資料獲取方面就可以充分利用無標籤的資料。通過逐層貪婪的訓練方法又稱為預訓練，然後可以使用有標籤的資料集對其進行微調。

習題答案：

(1) %% STEP 2: Train the first sparse autoencoder

addpath minFunc/
options.Method = 'lbfgs'; % Here, we use L-BFGS to optimize our cost
                          % function. Generally, for minFunc to work, you
                          % need a function pointer with two outputs: the
                          % function value and the gradient. In our problem,
                          % sparseAutoencoderCost.m satisfies this.
options.maxIter = 400;	  % Maximum number of iterations of L-BFGS to run 
options.display = 'on';



[sae1OptTheta, cost] = minFunc( @(p) sparseAutoencoderCost(p, ...
                                   inputSize, hiddenSizeL1, ...
                                   lambda, sparsityParam, ...
                                   beta, trainData), ...
                              sae1Theta, options);




W1 = reshape(sae1OptTheta(1:inputSize*hiddenSizeL1), hiddenSizeL1, inputSize);
display_network(W1', 12); 
 

(2) % STEP 2: Train the second sparse autoencoder

options.Method = 'lbfgs'; % Here, we use L-BFGS to optimize our cost
                          % function. Generally, for minFunc to work, you
                          % need a function pointer with two outputs: the
                          % function value and the gradient. In our problem,
                          % sparseAutoencoderCost.m satisfies this.
options.maxIter = 400;	  % Maximum number of iterations of L-BFGS to run 
options.display = 'on';



[sae2OptTheta, cost2] = minFunc( @(p) sparseAutoencoderCost(p, ...
                                   hiddenSizeL1, hiddenSizeL2, ...
                                   lambda, sparsityParam, ...
                                   beta, sae1Features), ...
                              sae2Theta, options);
 

(3) %% STEP 3: Train the softmax classifier

addpath minFunc/
options.Method = 'lbfgs'; % Here, we use L-BFGS to optimize our cost
                          % function. Generally, for minFunc to work, you
                          % need a function pointer with two outputs: the
                          % function value and the gradient. In our problem,
                          % softmaxCost.m satisfies this.
minFuncOptions.display = 'on';
lambda = 1e-4;  
[saeSoftmaxTheta, cost3] = minFunc( @(p) softmaxCost(p, ...
                                   numClasses, hiddenSizeL2, lambda, ...
                                   sae2Features, trainLabels), ...                                   
                              saeSoftmaxTheta, options);

(4) %% STEP 5: Finetune softmax model

addpath minFunc/
options.Method = 'lbfgs'; % Here, we use L-BFGS to optimize our cost
                          % function. Generally, for minFunc to work, you
                          % need a function pointer with two outputs: the
                          % function value and the gradient. In our problem,
                          % softmaxCost.m satisfies this.
minFuncOptions.display = 'on';

[stackedAEOptTheta, cost3] = minFunc( @(p) stackedAECost(p, ...
                                   inputSize, hiddenSizeL2, ...
                                   numClasses, netconfig, ...
                                   lambda, trainData, trainLabels), ...                                   
                              stackedAETheta, options);

StackedAECost.m

depth = numel(stack);
z = cell(depth+1, 1);         %輸入+隱藏層的z
a = cell(depth+1, 1);        %輸入+隱藏層的激勵函式
a{1} = data;
for i = 1:depth          % 計算隱藏層的z和激勵a
    z{i+1} = stack{i}.w * a{i} + repmat(stack{i}.b, 1, numCases);
    a{i+1} = sigmoid(z{i+1});
end

M = softmaxTheta * a{depth+1};     % 計算softmax對應的激勵值
M = bsxfun(@minus, M, max(M, [],1)); 
M = exp(M);   % 
p = bsxfun(@rdivide, M, sum(M));  

cost = -1/numCases .* sum(groundTruth(:)'*log(p(:))) + lambda/2 *sum(softmaxTheta(:).^2);    % cost function
softmaxThetaGrad = -1/numCases .* (groundTruth - p) * a{depth+1}' + lambda * softmaxTheta;               % grad   softmax對應的引數

delta = cell(depth+1);          % 誤差項 只需要計算隱藏層的就可以了
delta{depth+1} =  -(softmaxTheta' * (groundTruth-p)) .* a{depth+1} .* (1-a{depth+1});  %最後一個隱藏層所對應的error 可以推匯出來

for layer = depth: -1: 2
    delta{layer} = (stack{layer}.w * delta{layer+1}) .* a{layer} .* (1-a{layer});    %% 計算前面各層所對應的error  沒有考慮係數項及貝葉斯學派的引數
end

for layer = depth : -1 :1                 % 計算各隱層引數w和b所對應的梯度
    stackgrad{layer}.w = delta{layer+1} * a{layer}' ./ numCases;
    stackgrad{layer}.b = sum(delta{layer+1}, 2) ./numCases;
end

(5) %% STEP 6: Test

numCases = size(data, 2);
depth = numel(stack);
z = cell(depth+1, 1);         %杈撳靉+闅愯棌灞傜殑z
a = cell(depth+1, 1);        %杈撳靉+闅愯棌灞傜殑嬋?姳鍑芥暟
a{1} = data;
for i = 1:depth          % 璁＄畻闅愯棌灞傜殑z鍜屾縺鍔盿
    z{i+1} = stack{i}.w * a{i} + repmat(stack{i}.b, 1, numCases);
    a{i+1} = sigmoid(z{i+1});
end

[~, pred] = max(softmaxTheta * a{depth+1});

最終我得到的結果是：BeforeFinetuning Test Accuracy: 92.150%，After Finetuning Test Accuracy: 96.680% 和練習給的答案有點誤差。

下面將反向傳播法中進行微調的公式進行推導下：------貌似有誤 最後一層softmax計算了激勵函式值，不是線性的a=z

宣告：個人認為ufldl教程中《微調多層自編碼演算法》有誤：

個人認為不應該說是輸出層，應該是最後一個隱藏層，這樣後面的公式才是正確的(個人觀點)。

由反向傳導演算法一節中的公式：

由於在此處最後一層softmax迴歸中，我們並沒有求解z_i的sigmoid函式，所以：

再由最後一層隱藏層：

最後的結果再向量化即為程式碼中的結果，也與教程中給出的公式是一致的。(公式最後一步的f(zi)求導錯了，是f(zi)*f(1-zi))