一、Tensor 類簡介

• Tensor 定義

  • A Tensor is a symbolic handle to one of the outputs of an Operation. It does not hold the values of that operation’s output, but instead provides a means of computing those values in a TensorFlow tf.Session.
  • 在 TensorFlow 中，所有在節點之間傳遞的資料都為 Tensor 物件(可以看作 n 維的陣列)，常用影象資料的表示形式 為：batch*height*width*channel
    
    

• Tensor-like objects

  • tf.Tensor
  • tf.Variable
  • numpy.ndarray
  • list (and lists of tensor-like objects)
  • Scalar Python types: bool, float, int, str

  Note: By default, TensorFlow will create a new tf.Tensor each time you use the same tensor-like object.

• Some special tensors

  • tf.constant()：返回一個常量 tensor
  • tf.Variable()：返回一個 tensor-like 物件，表示變數
  • tf.SparseTensor()：返回一個tensor-like 物件
  • tf.placeholder()：return a tensor that may be used as a handle for feeding a value, but not evaluated directly.

二、Tensor 建立

• TF op：可接收標準 Python 資料型別，如整數、字串、由它們構成的列表或者Numpy 陣列，並將它們自動轉化為張量。單個數值將被轉化為0階張量（或標量），數值列表將被轉化為1階張量（向量），由列表構成的列表
  將被轉化為2階張量（矩陣），以此類推。
• Note：shape 要以 list 或 tuple 的形式傳入

1、常量 Tensor 的建立

• Constant Value Tensors

# 產生全 0 的張量
tf.zeros(shape, dtype=tf.float32, name=None)
tf.zeros_like(tensor, dtype=None, name=None)


# 產生全 1 的張量
tf.ones(shape, dtype=tf.float32, name=None)
tf.ones_like(tensor, dtype=None, name=None)


# Creates a tensor of shape  and fills it with value
tf.fill(dims, value, name=None)
tf.fill([2, 3], 9) ==> [[9, 9, 9]
                        [9, 9, 9]]


# 產生常量 Tensor, value 值可為 python 標準資料型別、Numpy 等
tf.constant(value, dtype=None, shape=None, name='Const')
tf.constant(-1.0, shape=[2, 3]) => [[-1., -1., -1.]  # Note: 注意 shape 的用法(廣播機制) 
                                    [-1., -1., -1.]]
tf.constant([1,2,3,4,5,6], shape=[2,3]) => [[1, 2, 3]
                                            [4, 5, 6]]

• Sequences

# 產生 num 個等距分佈在 [start, stop] 間元素組成的陣列，包括 start & stop (需為 float 型別)
# increase by (stop - start) / (num - 1)
tf.linspace(start, stop, num,, name=None)


# []為可選引數，步長 delta 預設為 1，start 預設為 0, limit 的值取不到，它產生一個數字序列
tf.range([start], limit, delta=1, dtype=None, name='range')

# eg
tf.range(start=3, limit=18, delta=3)  # [3, 6, 9, 12, 15]
tf.range(limit=5)  # [0, 1, 2, 3, 4]

• Random Tensors

# 正態分佈，預設均值為0，標準差為1.0，資料型別為float32
tf.random_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None)


# 正態分佈，但那些到均值的距離超過2倍標準差的隨機數將被丟棄，然後重新抽取，直到取得足夠數量的隨機數為止, 隨機數 x 
# 的取值範圍是$[mean - 2*stddev, mean + 2*stddev]$, 從而可以防止有元素與該張量中的其他元素顯著不同的情況出現
tf.truncated_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None)


# 產生在[minval, maxval)之間形狀為 shape 的均勻分佈, 預設是[0, 1)之間形狀為 shape 的均勻分佈
tf.random_uniform(shape, minval=0.0, maxval=1, dtype=tf.float32, seed=None, name=None)


# Randomly shuffles a tensor along its first dimension
tf.random_shuffle(value, seed=None, name=None)


# Randomly crops a tensor to a given size
tf.random_crop(value, size, seed=None, name=None)
# Note：If a dimension should not be cropped, pass the full size of that dimension. 
# For example, RGB images can be cropped with size = [crop_height, crop_width, 3]


# Sets the graph-level random seed
tf.set_random_seed(seed)
# 1. To generate the same repeatable sequence for an op across sessions
# set the seed for the op, a = tf.random_uniform([1], seed=1)
# 2. To make the random sequences generated by all ops be repeatable across sessions
# set a graph-level seed, tf.set_random_seed(1234)


# 其它
tf.multinomial(logits, num_samples, seed=None, name=None)
tf.random_gamma(shape,alpha,beta=None,dtype=tf.float32,seed=None,name=None)

2、變數 Tensor 的建立

I、Class tf.Variable()

• 常用屬性

  • dtype、shape、name
  • initial_value：Returns the Tensor used as the initial value for the variable.
  • initializer：The initializer operation for this variable，用於初始化此變數 sess.run(v.initializer)
  • op：The Operation that produces this tensor as an output.
  • device：The name of the device on which this tensor will be produced, or None.
  • graph：The Graph that contains this tensor.

• 常用方法

  • eval(session=None)：Evaluates this tensor in a Session. Returns A numpy ndarray with a copy of the value of this variable
  • get_shape()：Alias of Tensor.shape.
  • set_shape(shape)： It can be used to provide additional information about the shape of this tensor that cannot be inferred from the graph alone。
  • initialized_value()：Returns the value of the initialized variable.
  • read_value()：Returns the value of this variable, read in the current context.
  • assign(value, use_locking=False)：Assigns a new value to the variable.
  • assign_add(delta, use_locking=False)
  • assign_sub(delta, use_locking=False)

• Class Variable 定義

# tf.constant 是 op，而 tf.Variable() 是一個類，初始化的物件有多個op
var_obj = tf.Variable(
    initial_value, 
    dtype=None, 
    name=None, 
    trainable=True,
    collections=None,
    validate_shape=True
)

# 初始化引數
initial_value：可由 Python 內建資料型別提供，也可由常量 Tensor 的內建 op 來快速構建，但所有這些 op 都需要提供 shape

trainable：指明瞭該變數是否可訓練, 會加入 `GraphKeys.TRAINABLE_VARIABLES` collection 中去。

collections: List of graph collections keys. The new variable is added to these collections. Defaults to [GraphKeys.GLOBAL_VARIABLES].

validate_shape: If False, allows the variable to be initialized with a value of unknown shape. If True, the default, the shape of initial_value must be known.

# 返回值
變數例項物件(Tensor-like)

II、tf.get_variable()

# Gets an existing variable with these parameters or create a new one
tf.get_variable(
    name,
    shape=None,
    dtype=None,
    initializer=None,
    trainable=True,
    regularizer=None,
    collections=None,
    caching_device=None,
    partitioner=None,
    validate_shape=True,
    use_resource=None,
    custom_getter=None
)

# 初始化引數
name: The name of the new or existing variable.
shape: Shape of the new or existing variable.
dtype: Type of the new or existing variable (defaults to DT_FLOAT).
initializer: Initializer for the variable if one is created.

trainable: If True also add the variable to the graph collection tf.GraphKeys.TRAINABLE_VARIABLES.

regularizer: A (Tensor -> Tensor or None) function; the result of applying it on a newly created variable will be added to the collection tf.GraphKeys.REGULARIZATION_LOSSES and can be used for regularization.

collections: List of graph collections keys to add the Variable to. Defaults to [GraphKeys.GLOBAL_VARIABLES] (see tf.Variable). 

# 返回值
The created or existing Variable, 擁有變數類的所有屬性和方法。


# Note:
>>> name 引數必須要指定，如果僅給出 shape 引數而未指定 initializer，那麼它的值將由 tf.glorot_uniform_initializer 隨機產生，資料型別為tf.float32; 
>>> 另外，initializer 可以為一個張量，這種情況下，變數的值和形狀即為此張量的值和形狀(就不必指定shape 了)。
>>> 此函式經常和 tf.variable_scope() 一起使用，產生共享變數

III、initializer 引數的初始化

一般要在 tf.get_variable() 函式中指定shape，因為initializer要用到。

• tf.constant_initializer()、tf.zeros_initializer()、tf.ones_initializer()

tf.constant_initializer(
    value=0, 
    dtype=dtypes.float32, 
    verify_shape=False
)

# 通常偏置項就是用它初始化的。由它衍生出的兩個初始化方法：
I、tf.zeros_initializer()
II、tf.ones_initializer()

init = tf.constant_initializer()
x = tf.get_variable(name='v_x', shape=[2, 3], initializer=init) # 必須指定shape
sess.run(x.initializer)
sess.run(x)
>>> array([[ 0.,  0.,  0.],
           [ 0.,  0.,  0.]], dtype=float32)

• tf.truncated_normal_initializer()、tf.random_normal_initializer()

# 生成截斷正態分佈的隨機數，方差一般選0.01等比較小的數
tf.truncated_normal_initializer(
    mean=0.0,
    stddev=1.0,
    seed=None,
    dtype=tf.float32
)


# 生成標準正態分佈的隨機數，方差一般選0.01等比較小的數
tf.random_normal_initializer(
    mean=0.0,
    stddev=1.0,
    seed=None,
    dtype=tf.float32
)

• tf.random_uniform_initializer()、tf.uniform_unit_scaling_initializer()

# 生成均勻分佈的隨機數
tf.random_uniform_initializer(
    minval=0,
    maxval=None,
    seed=None,
    dtype=tf.float32
)


# 和均勻分佈差不多，只是這個初始化方法不需要指定最小最大值，是通過計算出來的
# 它的分佈區間為[-max_val, max_val]
tf.uniform_unit_scaling_initializer(
    factor=1.0,
    seed=None,
    dtype=tf.float32
)

max_val = math.sqrt(3 / input_size) * self.factor
# input size is obtained by multiplying W's all dimensions but the last one
# for a linear layer factor is 1.0, relu: ~1.43, tanh: ~1.15

• tf.variance_scaling_initializer()

tf.variance_scaling_initializer(
    scale=1.0,
    mode='fan_in',
    distribution='normal',
    seed=None,
    dtype=tf.float32
)

# 初始化引數
scale: Scaling factor (positive float).
mode: One of "fan_in", "fan_out", "fan_avg".
distribution: Random distribution to use. One of "normal", "uniform".

# 1、當 distribution="normal" 的時候：
生成 truncated normal distribution（截斷正態分佈）的隨機數，其中mean = 0, stddev = sqrt(scale / n)，
n 的計算與 mode 引數有關：
    如果mode = "fan_in"， n 為輸入單元的結點數         
    如果mode = "fan_out"，n 為輸出單元的結點數
    如果mode = "fan_avg",n 為輸入和輸出單元結點數的平均值

# 2、當distribution="uniform”的時候：
生成均勻分佈的隨機數，假設分佈區間為[-limit, limit]，則limit = sqrt(3 * scale / n)

• tf.glorot_uniform_initializer()、tf.glorot_normal_initializer()
  

  為了使得在經過多層網路後，訊號不被過分放大或過分減弱，我們儘可能保持每個神經元的輸入和輸出的方差一致! 從數學角度來講，就是讓權重滿足均值為 0，方差為 2fanin+fanout$\frac{2}{fa{n}_{in}+fa{n}_{out}}$，隨機分佈的形式可以為均勻分佈或者高斯分佈。

# 又稱 Xavier uniform initializer
tf.glorot_uniform_initializer(
    seed=None,
    dtype=tf.float32
)

# It draws samples from a uniform distribution within [a=-limit, b=limit] 
limit： sqrt(6 / (fan_in + fan_out)) 
fan_in：the number of input units in the weight tensor 
fan_out：the number of output units in the weight tensor
mean = (b + a) / 2
stddev = (b - a)**2 /12

# 又稱 Xavier normal initializer
tf.glorot_normal_initializer(
    seed=None,
    dtype=tf.float32
)

# It draws samples from a truncated normal distribution centered on 0 with 
# stddev = sqrt(2 / (fan_in + fan_out)) 
fan_in：the number of input units in the weight tensor 
fan_out：the number of output units in the weight tensor

三、 Tensor 初始化及訪問

1、Constants 初始化

• Constants are initialized when you call tf.constant, and their value can never change.

2、Variables 初始化

• Variables are not initialized when you call tf.Variable. To initialize all the variables in a TensorFlow program, you must explicitly call a special operation as follows:

# 變數使用前一定要初始化
init = tf.global_variables_initializer() # 初始化全部變數
sess.run(init)

# 使用變數的 initializer 屬性初始化
sess.run(v.initializer)

• 用另一個變數的初始化值給當前變數初始化
  
  • 由於tf.global_variables_initializer()是並行地初始化所有變數，所以直接使用另一個變數的初始化值來初始化當前變數會報錯(因為你用另一個變數的值時，它沒有被初始化)
  • 在這種情況下需要使用另一個變數的initialized_value()方法。你可以直接把已初始化的值作為新變數的初始值，或者把它當做tensor計算得到一個值賦予新變數。

# Create a variable with a random value.
weights = tf.Variable(tf.random_normal([784, 200], stddev=0.35), name="weights")

# Create another variable with the same value as 'weights'.
w2 = tf.Variable(weights.initialized_value(), name="w2")

# Create another variable with twice the value of 'weights'
w_twice = tf.Variable(weights.initialized_value() * 0.2, name="w_twice")

• 改變變數的值：通過 TF 中的賦值操作,update = tf.assign(old_variable, new_value) or v.assign(new_value)

3、Tensor 的訪問

• 索引
  
  • 一維 Tensor 的索引和 Python 列表類似(可以逆序索引(arr[ : : -1])和負索引arr[-3])
  • 二維 Tensor 的索引： arr[i, j] == arr[i][j]
  • 在多維 Tensor 中，如果省略了後面的索引，則返回的物件會是一個維度低一點的ndarray(但它含有高一級維度上的某條軸上的所有資料)
  • 條件索引：arr[conditon] # conditon 可以使用 & | 進行多條件組合
• 切片
  
  • 一維 Tensor 的切片和 Python 列表類似
  • 二維 Tensor 的索引：arr[r1:r2, c1:c2:step] # 也可指定 step 進行切片

四、Tensor 常用屬性

• dtype
  
  • tf.float32/64、tf.int8/16/32/64
  • tf.string、tf.bool、tf.complex64、tf.qint8
  • 不帶小數點的數會被預設為tf.int32，帶小數點的會預設為tf.float32
  • 可使用tf.cast(x, dtype, name=None)轉換資料型別

• shape

  • Tensor 的 shape 刻畫了張量每一維的長度，張量的維數由tf.rank(tensor)來表示
    

  • 取得Tensor shape 的值

    • 使用shape 屬性或者 get_shape() 方法， This method returns a TensorShape object This can be used for debugging, and providing early error messages
    • 設計計算圖時，使用tf.shape()函式， returns a tensor
    • Use batch_size = tf.shape(input)[0] to extract the batch dimension from a Tensor called input, and store it in a Tensor called batch_size.
  • 改變 Tensor shape
    
    • 使用tf.reshape(tensor, shape, name=None)函式：返回一個新的 tensor，shape 中的某一維可以用-1指定讓 reshape 函式取自動計算此維的長度
    • 使用 Tensor.set_shape() 方法：In some cases, the inferred shape may have unknown dimensions. If the caller has additional information about the values of these dimensions, Tensor.set_shape() can be used to augment the inferred shape.
      
  • 將 Tensor shape 轉化為 list： Tensor.shape.as_list()
  • passing in the value None as a shape (instead of using a list/tuple that contains None), will tell TensorFlow to allow a tensor of any shape

• name

  • eg: w1 = tf.Variable(tf.random_normal([2, 3], stddev=1), name='weight1'), 這裡面定義了變數 w1，為什麼又給了它一個 name='weight1'？ 這個 tensor 中的 name 屬性和其變數名有什麼區別呢？為什麼要這樣做呢？
  • 答：w1是程式碼中的變數名（識別符號），程式碼中都用這個。name='weight1'這個是引數名(權重)，在引數儲存或讀取的時候使用，方便在其它環境(C++等)中部署。還有個作用是跟 scope 配合使用的，用於引數共享
• op
  
  • The Operation that produces this tensor as an output.
  • 在上面 name 的例子中， tf.Operation named： w1.op.name='weight1' ，tf.Tensor named： w1.name='weight1:0'
  • Note that： tf.Tensor objects are implicitly named after the tf.Operation that produces the tensor as output. A tensor name has the form <OP_NAME>:<i> where:
    
    • <OP_NAME> ：節點的名稱
    • <i> ：表示當前張量來自節點的第幾個輸出
• device
  
  • The name of the device on which this tensor will be produced, or None.
• graph
  
  • The Graph that contains this tensor.

五、Tensor 常用方法

• eval(feed_dict=None, session=None)
  
  • Evaluates this tensor in a Session，通常需要指定 session=sess
  • 當在互動式環境中使用 sess = tf.InteractiveSession()，系統會自動將生成的會話註冊為預設會話，此時就不需要指定 session=sess了
• get_shape()
  
  • Alias of Tensor.shape
• set_shape(shape)
  
  • It can be used to provide additional information about the shape of this tensor that cannot be inferred from the graph alone

_, image_data = tf.TFRecordReader(...).read(...)
image = tf.image.decode_png(image_data, channels=3)

# The height and width dimensions of `image` are data dependent, and
# cannot be computed without executing the op.
print(image.shape)
==> TensorShape([Dimension(None), Dimension(None), Dimension(3)])

# We know that each image in this dataset is 28 x 28 pixels.
image.set_shape([28, 28, 3])
print(image.shape)
==> TensorShape([Dimension(28), Dimension(28), Dimension(3)])

六、Tensor 變換常用方法

1.Casting：資料型別轉換

tf.string_to_number(string_tensor, out_type=None, name=None)
tf.to_double(x, name='ToDouble')
tf.to_float(x, name='ToFloat')
tf.to_int32(x, name='ToInt32')
tf.to_int64(x, name='ToInt64')


tf.cast(x, dtype, name=None)  # Casts a tensor to a new type
# tensor `a` is [1.8, 2.2], dtype=tf.float
tf.cast(a, tf.int32) ==> [1, 2]  # dtype=tf.int32


# 其它
tf.bitcast
tf.saturate_cast
tf.to_bfloat16

2. Shapes and Shaping：取得張量形狀和改變張量形狀

# 改變 Tensor 的形狀
tf.reshape(tensor, shape, name=None)
# Flatten：令 shape=[-1] 即可
# Reshape：shape 乘積不變即可，當某一維傳入-1時，它會自動推得此維度的大小


# 轉置
tf.transpose(a, perm=None, name='transpose')


# 返回 tensor 各個維度的大小
tf.shape(input, name=None)  
# 't' is [[[1, 1, 1], [2, 2, 2]], [[3, 3, 3], [4, 4, 4]]]
sess.run(tf.shape(t)) ==> array([2, 2, 3], dtype=int32)   # 必須要 run 才能得出結果
# 亦可以使用 TF 變數物件 Var 的get_shape() 方法來實現Var.get_shape() 


# 返回 tensor 中元素的總數
tf.size(input, name=None)  


# 返回 Tensor 的維度(軸)的個數，類似於 Numpy 中的 ndim 屬性
tf.rank(input, name=None)


# inserts a dimension of 1 into a tensor's shape
tf.expand_dims(
    input,
    axis=None,
    name=None,
)
# 例1，'t' is a tensor of shape [2]
tf.shape(tf.expand_dims(t, 0))  # [1, 2]
tf.shape(tf.expand_dims(t, 1))  # [2, 1]
tf.shape(tf.expand_dims(t, -1))  # [2, 1]，支援負索引

# 例2，'t2' is a tensor of shape [2, 3, 5]
tf.shape(tf.expand_dims(t2, 0))  # [1, 2, 3, 5]， make it a batch of 1 image
tf.shape(tf.expand_dims(t2, 2))  # [2, 3, 1, 5]
tf.shape(tf.expand_dims(t2, 3))  # [2, 3, 5, 1]


# 若 axis 沒指定，則移除 shape 中所有的 1，若指定某個軸，則只移除相應位置shape 中的 1 
tf.squeeze(
    input,
    axis=None,
    name=None,
)
# 例1，'t' is a tensor of shape [1, 2, 1, 3, 1, 1]
tf.shape(tf.squeeze(t))  # [2, 3]

# 例2， remove specific size 1 dimensions
tf.shape(tf.squeeze(t, axis=[2, 4]))  # [1, 2, 3, 1]


# 其它
tf.broadcast_dynamic_shape
tf.broadcast_static_shape
tf.shape_n
tf.meshgrid

3. Slicing and Joining：切片和連線

• 切片：可使用 TF 函式實現，也可使用 python 原始切片方式實現(切出 1 份)

tf.slice(input_, begin, size, name=None)
# begin(zero-based)：切片的起點座標，一般用 list 來表示
# size(one-based)：切出多大，size[i] is the number of elements of the 'i'th dimension of input that you want to slice
# If size[i] is -1, all remaining elements in dimension i are included in the slice

For example:
# 'input' is [[[1, 1, 1], [2, 2, 2]],
#             [[3, 3, 3], [4, 4, 4]],
#             [[5, 5, 5], [6, 6, 6]]]
tf.slice(input, [1, 0, 0], [1, 1, 3]) ==> [[[3, 3, 3]]]
tf.slice(input, [1, 0, 0], [1, 2, 3]) ==> [[[3, 3, 3],  
                                            [4, 4, 4]]]
tf.slice(input, [1, 0, 0], [2, 1, -1]) ==> [[[3, 3, 3]],
                                           [[5, 5, 5]]]
# 亦可以使用 python 原始切片方式實現，eg: input[1, 0:2, 0:3]和第三個效果相同

• 分割：沿著座標軸將 Tensor 分割成尺寸相同的 n 等份或者尺寸不同的 n 份

tf.split(value, num_or_size_splits, axis=0, num=None, name='split')
# num_or_size_splits 
integer：splits value along dimension axis into integer smaller tensors
list：plits value along dimension axis into len(list) smaller tensors.等份每一份的大小是list[i] 

For example:
# 'value' is a tensor with shape [5, 30]
# Split 'value' into 3 tensors with sizes [4, 15, 11] along dimension 1
split0, split1, split2 = tf.split(value, [4, 15, 11], 1)
tf.shape(split0) ==> [5, 4]
tf.shape(split1) ==> [5, 15]
tf.shape(split2) ==> [5, 11]
# Split 'value' into 3 tensors along dimension 1
split0, split1, split2 = tf.split(value, num_or_size_splits=3, axis=1)
tf.shape(split0) ==> [5, 10]

• 連線：沿著某座標軸連線 N 個張量(Numpy 連線傳入的是 tuple, 此處為 list )

tf.concat(values, axis, name='concat')  # 維度不變 
For example:
t1 = [[1, 2, 3], [4, 5, 6]]
t2 = [[7, 8, 9], [10, 11, 12]]
tf.concat([t1, t2], 0) ==> [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]]
tf.concat([t1, t2], 1) ==> [[1, 2, 3, 7, 8, 9], [4, 5, 6, 10, 11, 12]]

# tensor t3 with shape [2, 3]
# tensor t4 with shape [2, 3]
tf.shape(tf.concat([t3, t4], 0)) ==> [4, 3]
tf.shape(tf.concat([t3, t4], 1)) ==> [2, 6]


tf.stack(values, axis, name='concat')  # 維度+1
# Stacks a list of rank-R tensors into one rank-(R+1) tensor
# Given a list of length N=2 of tensors of shape (3, 3);
if axis == 0 then the output tensor will have the shape (N, 3, 3). 
if axis == 1 then the output tensor will have the shape (3, N, 3).
if axis == 1 then the output tensor will have the shape (3, 3, N).

a = np.array([[1, 2, 3],
              [4, 5, 6],
              [7, 8, 9]])

b = np.array([[ 2,  4,  6],
              [ 8, 10, 12],
              [14, 16, 18]])

# Note: 做 stack 之前把 a, b 的維度+1變為(1, 3, 3) 
# 沿著 x 軸(垂直向下)連線 a, b 的第 0 維元素
sess.run(tf.stack([a,b], axis=0))
array([[[ 1,  2,  3],
        [ 4,  5,  6],
        [ 7,  8,  9]],

       [[ 2,  4,  6],
        [ 8, 10, 12],
        [14, 16, 18]]])

# 沿著 y 軸(水平向右)連線 a, b 的第 1 維元素
sess.run(tf.stack([a,b], axis=1)) 
array([[[ 1,  2,  3],
        [ 2,  4,  6]],

       [[ 4,  5,  6],
        [ 8, 10, 12]],

       [[ 7,  8,  9],
        [14, 16, 18]]])

# 沿著 z 軸(豎直向上)連線 a, b 的第 2 維元素
sess.run(tf.stack([a,b], axis=2))
array([[[ 1,  2],
        [ 2,  4],
        [ 3,  6]],

       [[ 4,  8],
        [ 5, 10],
        [ 6, 12]],

       [[ 7, 14],
        [ 8, 16],
        [ 9, 18]]])

• 補零

tf.pad(tensor, paddings, mode='CONSTANT', name=None)
paddings： is an integer tensor with shape [n, 2]，n是 tensor 的維度
For example:
# 't' is [[1, 2, 3], [4, 5, 6]].
# 'paddings' is [[1, 1,], [2, 2]].
# paddings[0, 0/1]: 沿著第 0 維(x軸)在 tensor 上方/下方補 1 圈零
# paddings[1, 0/1]: 沿著第 1 維(y軸)在 tensor 左方/右方補 2 圈零
tf.pad(t, paddings, "CONSTANT") ==> [[0, 0, 0, 0, 0, 0, 0],
                                     [0, 0, 1, 2, 3, 0, 0],
                                     [0, 0, 4, 5, 6, 0, 0],
                                     [0, 0, 0, 0, 0, 0, 0]]

• one_hot 向量的生成

tf.one_hot(indices, depth, on_value=1, off_value=0, axis=-1, dtype=None, name=None)
# 將 indices 中的每個元素 j 擴充套件成一個深度為 depth 的向量，輸出維度+1
# 此向量中索引位置 j 的取值為 1,其餘位置的取值為 0 
# If indices is a scalar the output shape will be a vector of length depth

# If indices is a vector of length features, the output shape will be:
features x depth if axis == -1
depth x features if axis == 0

# If indices is a matrix (batch) with shape [batch, features], the output shape will be:
batch x features x depth if axis == -1
batch x depth x features if axis == 1
depth x batch x features if axis == 0

# 使用onehot的直接原因是：現在多分類cnn網路的輸出通常是softmax層，而它的輸出是一個概率分佈
# 從而要求輸入的標籤也以概率分佈的形式出現，進而計算交叉熵之類

• 其它

tf.extract_image_patches
tf.strided_slice
tf.tile
tf.parallel_stack
tf.unstack
tf.reverse_sequence
tf.reverse
tf.reverse_v2
tf.space_to_batch_nd
tf.space_to_batch
tf.required_space_to_batch_paddings
tf.batch_to_space_nd
tf.batch_to_space
tf.space_to_depth
tf.depth_to_space
tf.gather
tf.gather_nd
tf.unique_with_counts
tf.scatter_nd
tf.dynamic_partition
tf.dynamic_stitch
tf.boolean_mask
tf.sequence_mask
tf.dequantize
tf.quantize_v2
tf.quantized_concat
tf.setdiff1d

七、Numpy VS TensorFLow

• 相同點：Both are N-d array libraries，建立、訪問、常用屬性和方法都非常相似
• 不同點：
  
  • Numpy has Ndarray support, but doesn’t offer methods to create tensor functions and automatically compute derivatives (+ no GPU support)
  • Numpy 方法中 shape 通常傳入的是一個 tuple, 而 Tensor 中shape 通常傳入一個 list
    

八、參考資料