Udacity機器學習筆記——深度學習（1）

深度學習是機器學習的一個熱門分支，它使用大量的資料裡解決與人類感知相關的問題，例如影象識別，語音識別、理解自然語言等。許多公司都將深度學習作為研究的一部分，例如Facebook、Google、Microsoft和百度等。上個世紀八九十年代雖然有關於神經網路的研究，但是由於當時資料量小和計算機運算能力有限，並沒有得到廣泛的應用，在二十一世紀的前十年關於神經網路的研究也比較邊緣化。但進入第二個十年，隨著語音識別（2009）、計算機視覺（2012）和機器翻譯（2014）等領域的興起，神經網路才以深度學習的身份變得流行起來，其背後也是因為我們現在擁有了海量資料以及更加快速的處理器。

工具

1. 課程使用TensorFlow，Google的開源庫。可以使用Pip、Virtualenv或者Docker方法安裝。

深度學習簡介

1. 在此之前學習了監督學習、無監督學習和強化學習。監督學習基於一組帶有標籤的資料學習預測新的資料的標籤；無監督學習基於一組無標籤的資料學習如何分類；強化學習則是建立了一個模型，可以在環境中學習得到最優的結果。深度學習也是基於一組資料進行學習，從資料中抽取出更高等級的抽象出來。

2. 在深度學習課程中，首先介紹了深度神經網路，一種多層人工神經網路結構模型；然後，介紹了卷積神經網路模型，適用於影象和語音識別領域的神經網路模型；最後，介紹了序列神經網路模型，適用於識別書寫和語言文字等領域。

TensorFlow

1. 使用Conda安裝TensorFlow，在OS X或者Linux系統：

conda create -n tensorflow python=3.5
source activate tensorflow
conda install pandas matplotlib jupyter notebook scipy scikit-learn
pip install tensorflow

在Windows系統：

conda create -n tensorflow python=3.5
activate tensorflow
conda install pandas matplotlib jupyter notebook scipy scikit-learn
pip install tensorflow
 

2. Hello, world!程式：

import tensorflow as tf

#Create TensorFlow object called tensor
hello_constant = tf.constant('Hello World!')

with tf.Session() as sess:
    #Run the tf.constant operation in the session
    output = sess.run(hello_constant)
    print(output)

3. 資料型別Tensor的定義：

#A is a 0-dimensional int32 tensor
A = tf.constant(1234) 
#B is a 1-dimensional int32 tensor
B = tf.constant([123,456,789]) 
#C is a 2-dimensional int32 tensor
C = tf.constant([ [123,456,789], [222,333,444] ])

4. TensorFlow計算圖執行環境Session：

with tf.Session() as sess:
    output = sess.run(hello_constant)
    print(output)

線性函式

1. 神經網路中最常見的運算就是計算輸入、權重和偏差的線性函式：
   $y = xW + b$
2. 訓練神經網路的目標就是通過修改權重和偏差，從而更好地預測新輸入的標籤或者型別，在TensorFlow中定義權重和偏差，需要使用可以修改的變數，而不是上面定義的tf.Constant。另外，初始化權重時可以選擇正交分佈隨機值；初始化偏差可以不用隨機值，直接使用最簡單的0。

#Define variable x and initialize all variables
x = tf.Variable(5)
init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
#Initialize weights
n_features = 120
n_labels = 5
weights = tf.Variable(tf.truncated_normal((n_features, n_labels)))
#Initialize bias
n_labels = 5
bias = tf.Variable(tf.zeros(n_labels))

3. 下面演示一個分類函式對手寫數字0、1、2進行識別，其中資料集源自MNIST資料集。

# Import modules
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data


# Define weights, biases, and linear function
def get_weights(n_features, n_labels):
    return tf.Variable(tf.truncated_normal((n_features, n_labels)))


def get_biases(n_labels):
    return tf.Variable(tf.zeros(n_labels))


def linear(input, w, b):
    return tf.add(tf.matmul(input, w), b)


# Extract part of MNIST data
def mnist_features_labels(n_labels):
    mnist_features = []
    mnist_labels = []
    mnist = input_data.read_data_sets('/datasets/ud730/mnist', one_hot=True)
    # We only look at 10000 images
    for mnist_feature, mnist_label in zip(*mnist.train.next_batch(10000)):
        if mnist_label[:n_labels].any():
            mnist_features.append(mnist_feature)
            mnist_labels.append(mnist_label[:n_labels])
    return mnist_features, mnist_labels


# Number of features (28*28 image is 784 features)
n_features = 784
# Number of labels
n_labels = 3

# Features and Labels
features = tf.placeholder(tf.float32)
labels = tf.placeholder(tf.float32)

# Weights and Biases
w = get_weights(n_features, n_labels)
b = get_biases(n_labels)

# Linear Function xW+b
logits = linear(features, w, b)

# Training data
train_features, train_labels = mnist_features_labels(n_labels)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    # Softmax
    prediction = tf.nn.softmax(logits)

    # Cross entropy
    cross_entropy = -tf.reduce_sum(labels * tf.log(prediction), reduction_indices=1)

    # Training loss
    loss = tf.reduce_mean(cross_entropy)

    # Rate at which the weights are changed
    learning_rate = 0.08

    # Gradient Descent
    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

    # Run optimizer and get loss
    _, l = sess.run([optimizer, loss], feed_dict={features: train_features, labels: train_labels})

# Print loss
print('Loss: {}'.format(l))

• sigmoid函式作為啟用函式，存在讓學習速度變慢的問題。上面程式碼中的啟用函式使用softmax函式，該函式將輸入轉換為0與1之間的概率分佈，相比於sigmoid函式，softmax適合於多分類問題；softmax函式形式如下:
  $\sigma(z)_{j} = \frac{e^{z_{j}}}{\sum_{k=1}^{K}e^{z_{k}}} \quad j=1,2,...,K$
• One-Hot Encoding獨熱碼，就是將離散型資料進行編碼，每條資料變成只有一個值為1，其他值為0的方法。下面程式碼演示了使用scikit-learn的LabelBinarizer方法進行One-Hot Encoding：

import numpy as np
from sklearn import prepocessing
labels = np.array([1,5,3,2,1,4,2,1,3])
lb = preprocessing.LabelBinarizer()
lb.fit(labels)
lb.transform(labels)

• 上面程式碼使用了Cross Entropy成本函式：
  $C(y',y)=-\sum_{j}y_{j}lny'_{j}$

4. 上面僅對前10000個數據進行訓練，如果資料量更大，可以使用mini-batching和SGD技術對資料集進行訓練。程式碼如下：

import math
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf
import numpy as np


#Define print epoch status function
def print_epoch_stats(epoch_i, sess, last_features, last_labels):
    #Print cost and validation accuracy of an epoch
    current_cost = sess.run(cost, feed_dict={features: last_features, labels: last_labels})
    valid_accuracy = sess.run(accuracy, feed_dict={features: valid_features, labels: valid_labels})
    print('Epoch: {:<4} - Cost: {:<8.3} Valid Accuracy: {:<5.3}'.format(epoch_i, current_cost, valid_accuracy))

#Define batches function
def batches(batch_size, features, labels):
    assert len(features) == len(labels)
    output_batches = []

    sample_size = len(features)
    for start_i in range(0, sample_size, batch_size):
        end_i = start_i + batch_size
        batch = [features[start_i:end_i], labels[start_i:end_i]]
        output_batches.append(batch)
    return output_batches

n_input = 784
n_classes = 10

#Import MNIST data
mnist = input_data.read_data_sets('/dataset/ud730/mnist', one_hot=True)

#The features are already scaled and the data is shuffled
train_features = mnist.train.images
valid_features = mnist.validation.images
test_features = mnist.test.images

train_labels = mnist.train.labels.astype(np.float32)
valid_labels = mnist.validation.labels.astype(np.float32)
test_labels = mnist.test.labels.astype(np.float32)

#Features and Labels
features = tf.placeholder(tf.float32, [None, n_input])
labels = tf.placeholder(tf.float32, [None, n_classes])

#Weights and Biaes
weights = tf.Variable(tf.random_normal([n_input, n_classes]))
bias = tf.Variable(tf.random_normal([n_classes]))

#Logits - xW+b
logits = tf.add(tf.matmul(features, weights), bias)

#Define loss and optimizer
learning_rate = tf.placeholder(tf.float32)
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=labels))
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(cost)

#Calculate accuray
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(labels, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

#Set batch size
batch_size = 128
epochs = 80
learn_rate = 0.001

assert batch_size is not None, 'You must set the batch size'
init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)

    #Training cycle
    for epoch_i in range(epochs):

        #Train optimizer on all batches
        for batch_features, batch_labels in batches(batch_size, train_features, train_labels):
            sess.run(optimizer, feed_dict={features: batch_features, labels: batch_labels, learning_rate: learn_rate})
        #Print cost and validation accuracy of an epoch
        print_epoch_stats(epoch_i, sess, batch_features, batch_labels)

    #Calculate accuracy for test dataset
    test_accuracy = sess.run(accuracy, feed_dict={features: test_features, labels: test_labels})

print('Test Accuracy: {}'.format(test_accuracy))