該部分程式碼從輸出節點的增量開始，計算隱藏節點的輸出誤差，並將其用於下一次誤差的計算。

This process starts from the delta of theoutput node, calculates the error of the hidden node, and uses it for the nexterror.

該過程在delta3、delta2和delta1之間迴圈重複相同的步驟。

It repeats the same steps through delta3,delta2, and delta1.

…

e = d - y;

delta = e;

e3 = W4’*delta;

delta3 = (v3 > 0).*e3;

e2 = W3’*delta3;

delta2 = (v2 > 0).*e2;

e1 = W2’*delta2;

delta1 = (v1 > 0).*e1;

…

程式碼中需要注意的是函式ReLU的導數。

Something noticeable from the code is thederivative of the function ReLU.

例如，在計算第三隱藏層delta3的增量時，ReLU函式的導數對應的程式碼如下：

For example, in the calculation of thedelta of the third hidden layer, delta3, the derivative of the ReLU function iscoded as follows:

(v3> 0)

讓我們看看這行程式碼是如何成為ReLU函式的導數。

Let’s see how this line becomes the derivativeof the ReLU function.

如果括號中的表示式分別為真和假時，則MATLAB對應返回1和0。

MATLAB returns a unity and zero if theexpressions in the brackets are true and false, respectively.

因此，如果v3 > 0，則該行程式碼返回1，否則返回0。

Therefore, this line becomes 1 if v3 > 0and 0 otherwise.

與下面所示的ReLU函式的導數定義一樣，輸出結果是相同的：

The same result is produced as thedefinition of the derivative of the ReLU function shown here:

以下程式碼為TestDeepReLU.m檔案的詳細清單，該程式碼用於DeepReLU函式的測試。

The following listing shows theTestDeepReLU.m file, which tests the DeepReLU function.

該程式碼呼叫DeepReLU函式，並訓練網路10000次。

This program calls the DeepReLU functionand trains the network 10,000 times.

它將訓練資料輸入到訓練網路中並顯示輸出。

It enters the training data into thetrained network and displays the output.

我們通過比較訓練輸出和正確的輸出來驗證神經網路訓練的充分性。

We verify the adequacy of the training bycomparing the output and correct output.

clear all

X = zeros(5, 5, 5);

X(:, :, 1) = [0 1 1 0 0;

            0 0 1 0 0;

            0 0 1 0 0;

            0 0 1 0 0;

            0 1 1 1 0

           ];

X(:, :, 2) = [1 1 1 1 0;

            0 0 0 0 1;

            0 1 1 1 0;

            1 0 0 0 0;

            1 1 1 1 1

           ];

X(:, :, 3) = [1 1 1 1 0;

           0 0 0 0 1;

            0 1 1 1 0;

            0 0 0 0 1;

            1 1 1 1 0

           ];

X(:, :, 4) = [0 0 0 1 0;

            0 0 1 1 0;

            0 1 0 1 0;

            1 1 1 1 1;

            0 0 0 1 0

           ];

X(:, :, 5) = [1 1 1 1 1;

            1 0 0 0 0;

            1 1 1 1 0;

            0 0 0 0 1;

            1 1 1 1 0

           ];

D = [ 1 0 0 00;

    0 1 0 0 0;

    0 0 1 0 0;

    0 0 0 1 0;

    0 0 0 0 1

   ];

W1 = 2*rand(20, 25) - 1;

W2 = 2*rand(20, 20) - 1;

W3 = 2*rand(20, 20) - 1;

W4 = 2*rand( 5, 20) - 1;

for epoch = 1:10000 % train

   [W1,W2, W3, W4] = DeepReLU(W1, W2, W3, W4, X, D);

end

N = 5; % inference

for k = 1:N

   x= reshape(X(:, :, k), 25, 1);

   v1= W1*x;

   y1 = ReLU(v1);

   v2 = W2*y1;

   y2 = ReLU(v2);

   v3 = W3*y2;

   y3 = ReLU(v3);

   v = W4*y3;

   y= Softmax(v)

end

由於該程式碼與以前測試程式的程式碼幾乎相同，因此不再詳細說明。

As this code is also almost identical tothe previous test programs, a detailed explanation is omitted.

有時候該程式碼可能會訓練失敗，產生錯誤輸出，而在我們使用sigmoid啟用函式時是從來沒有發生過的。

This code occasionally fails to trainproperly and yields wrong outputs, which has never happened with the sigmoidactivation function.

ReLU函式對初始權值的敏感特性導致了這種異常現象。

The sensitivityof the ReLU function to the initial weight values seems to cause this anomaly.

——本文譯自Phil Kim所著的《Matlab Deep Learning》

更多精彩文章請關注微訊號：