1. 經常使用類

class tf.contrib.rnn.BasicLSTMCell

BasicLSTMCell 是最簡單的一個LSTM類。沒有實現clipping，projection layer。peep-hole等一些LSTM的高級變種，僅作為一個主要的basicline結構存在，假設要使用這些高級變種，需用class tf.contrib.rnn.LSTMCell這個類。

使用方式：

lstm = rnn.BasicLSTMCell(lstm_size, forget_bias=1.0, state_is_tuple=True)

Args:

• num_units: int, The number of units in the LSTM cell.

• forget_bias: float, The bias added to forget gates.

• state_is_tuple: If True, accepted and returned states are 2-tuples of the c_state and m_state. If False, they are concatenated along the column axis. The latter behavior will soon be deprecated.

• activation: Activation function of the inner states.

說明：

• num_units 是指一個Cell中神經元的個數，並非循環層的Cell個數。

  這裏有人可能會疑問：循環層的Cell數目怎麽表示？答案是通過例如以下代碼中的 time_step_size確定（X_split 中劃分出的arrays數量為循環層的Cell個數）：

    X_split = tf.split(XR, time_step_size, 0)

• 在隨意時刻 $t$ ，LSTM Cell會產生兩個內部狀態 $c_{t}$和$h_{t}$ （關於RNN與LSTM的介紹可參考：循環神經網絡與LSTM）。當state_is_tuple=True時，上面講到的狀態$c_{t}$和$h_{t}$ 就是分開記錄，放在一個二元tuple中返回，假設這個參數沒有設定或設置成False，兩個狀態就按列連接起來返回。官方說這樣的形式立即就要被deprecated了，全部我們在使用LSTM的時候要加上state_is_tuple=True。

class tf.contrib.rnn.DropoutWrapper

RNN中的dropout和cnn不同，在RNN中。時間序列方向不進行dropout，也就是說從t-1時刻的狀態傳遞到t時刻進行計算時，這個中間不進行memory的dropout。例如以下圖所看到的，Dropout僅應用於虛線方向的輸入，即僅針對於上一層的輸出做Dropout。

lstm = tf.nn.rnn_cell.DropoutWrapper(lstm, output_keep_prob=keep_prob)

Args:

• cell: an RNNCell, a projection to output_size is added to it.

• input_keep_prob: unit Tensor or float between 0 and 1, input keep probability; if it is float and 1, no input dropout will be added.

• output_keep_prob: unit Tensor or float between 0 and 1, output keep probability; if it is float and 1, no output dropout will be added.

• seed: (optional) integer, the randomness seed.

class tf.contrib.rnn.MultiRNNCell

假設希望整個網絡的層數很多其它（比如上圖表示一個兩層的RNN，第一層Cell的output還要作為下一層Cell的輸入），應該堆疊多個LSTM Cell，tensorflow給我們提供了MultiRNNCell，因此堆疊多層網絡僅僅生成這個類就可以：

lstm = tf.nn.rnn_cell.MultiRNNCell([lstm] * num_layers, state_is_tuple=True)

2. 代碼

MNIST數據集的格式與數據預處理代碼 input_data.py的解說請參考 :Tutorial (2)

# -*- coding: utf-8 -*-
import tensorflow as tf
from tensorflow.contrib import rnn

import numpy as np
import input_data

# configuration
#                        O * W + b -> 10 labels for each image, O[? 28], W[28 10], B[10]
#                       ^ (O: output 28 vec from 28 vec input)
#                       |
#      +-+  +-+       +--+
#      |1|->|2|-> ... |28| time_step_size = 28
#      +-+  +-+       +--+
#       ^    ^    ...  ^
#       |    |         |
# img1:[28] [28]  ... [28]
# img2:[28] [28]  ... [28]
# img3:[28] [28]  ... [28]
# ...
# img128 or img256 (batch_size or test_size 256)
#      each input size = input_vec_size=lstm_size=28

# configuration variables
input_vec_size = lstm_size = 28 # 輸入向量的維度
time_step_size = 28 # 循環層長度

batch_size = 128
test_size = 256

def init_weights(shape):
    return tf.Variable(tf.random_normal(shape, stddev=0.01))


def model(X, W, B, lstm_size):
    # X, input shape: (batch_size, time_step_size, input_vec_size)
    # XT shape: (time_step_size, batch_size, input_vec_size)
    XT = tf.transpose(X, [1, 0, 2])  # permute time_step_size and batch_size,[28, 128, 28]

    # XR shape: (time_step_size * batch_size, input_vec_size)
    XR = tf.reshape(XT, [-1, lstm_size]) # each row has input for each lstm cell (lstm_size=input_vec_size)

    # Each array shape: (batch_size, input_vec_size)
    X_split = tf.split(XR, time_step_size, 0) # split them to time_step_size (28 arrays),shape = [(128, 28),(128, 28)...]


    # Make lstm with lstm_size (each input vector size). num_units=lstm_size; forget_bias=1.0
    lstm = rnn.BasicLSTMCell(lstm_size, forget_bias=1.0, state_is_tuple=True)

    # Get lstm cell output, time_step_size (28) arrays with lstm_size output: (batch_size, lstm_size)
    # rnn..static_rnn()的輸出相應於每個timestep。假設僅僅關心最後一步的輸出，取outputs[-1]就可以
    outputs, _states = rnn.static_rnn(lstm, X_split, dtype=tf.float32)  # 時間序列上每個Cell的輸出:[... shape=(128, 28)..]

    # Linear activation
    # Get the last output
    return tf.matmul(outputs[-1], W) + B, lstm.state_size # State size to initialize the stat

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True) # 讀取數據

# mnist.train.images是一個55000 * 784維的矩陣, mnist.train.labels是一個55000 * 10維的矩陣
trX, trY, teX, teY = mnist.train.images, mnist.train.labels, mnist.test.images, mnist.test.labels

# 將每張圖用一個28x28的矩陣表示,(55000,28,28,1)
trX = trX.reshape(-1, 28, 28) 
teX = teX.reshape(-1, 28, 28) 

X = tf.placeholder("float", [None, 28, 28])
Y = tf.placeholder("float", [None, 10])

# get lstm_size and output 10 labels
W = init_weights([lstm_size, 10])  # 輸出層權重矩陣28×10
B = init_weights([10])  # 輸出層bais

py_x, state_size = model(X, W, B, lstm_size)

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=py_x, labels=Y))
train_op = tf.train.RMSPropOptimizer(0.001, 0.9).minimize(cost)
predict_op = tf.argmax(py_x, 1)

session_conf = tf.ConfigProto()
session_conf.gpu_options.allow_growth = True

# Launch the graph in a session
with tf.Session(config=session_conf) as sess:
    # you need to initialize all variables
    tf.global_variables_initializer().run()

    for i in range(100):
        for start, end in zip(range(0, len(trX), batch_size), range(batch_size, len(trX)+1, batch_size)):
            sess.run(train_op, feed_dict={X: trX[start:end], Y: trY[start:end]})

        test_indices = np.arange(len(teX))  # Get A Test Batch
        np.random.shuffle(test_indices)
        test_indices = test_indices[0:test_size]

        print(i, np.mean(np.argmax(teY[test_indices], axis=1) ==
                         sess.run(predict_op, feed_dict={X: teX[test_indices]})))

3. 參考資料

• https:[email protected]/hello-world-rnn-83cd7105b767
• http://www.cnblogs.com/wuzhitj/p/6434171.html
• http://blog.csdn.net/mydear_11000/article/details/52414342

Tensorflow - Tutorial (7) : 利用 RNN/LSTM 進行手寫數字識別