基於MNIST資料集上從一對數字中訓練一個 Siamese MLP。

Siamese ,連體的，相似的。

Siamese Net,孿生網路、連體網路

MLP，多層感知機，（多個隱藏層的全連線的神經網路）

詳解

程式碼註釋

'''Trains a Siamese MLP on pairs of digits from the MNIST dataset.
基於MNIST資料集上從一對數字中訓練一個 Siamese MLP。
Siamese網路是一種相似性度量方法,當類別數多,但每個類別的樣本數量少的情況下可用於類別的識別、分類等。

It follows Hadsell-et-al.'06 [1] by computing the Euclidean distance on the
output of the shared network and by optimizing the contrastive loss (see paper
for mode details).
通過計算共享網路的輸出上的歐幾里德距離並通過優化對比損耗（見模式細節論文）來跟蹤Ha售貨等。'06（1）。
# References
參考
- Dimensionality Reduction by Learning an Invariant Mapping
基於不變對映學習的降維
    http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf

Gets to 97.2% test accuracy after 20 epochs.
20週期後97.2%測試準確率
2 seconds per epoch on a Titan X Maxwell GPU
2秒/週期，基於Titan X Maxwell GPU （執行硬體）
'''
from __future__ import absolute_import
from __future__ import print_function
import numpy as np

import random
from keras.datasets import mnist
from keras.models import Model
from keras.layers import Input, Flatten, Dense, Dropout, Lambda
from keras.optimizers import RMSprop
from keras import backend as K

num_classes = 10
epochs = 20


def euclidean_distance(vects):
    x, y = vects
    return K.sqrt(K.maximum(K.sum(K.square(x - y), axis=1, keepdims=True), K.epsilon()))


def eucl_dist_output_shape(shapes):
    shape1, shape2 = shapes
    return (shape1[0], 1)


def contrastive_loss(y_true, y_pred):
    '''Contrastive loss from Hadsell-et-al.'06
    Hadsell-et-al.'06 的對比損失
    http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf

    人工智慧專家：
    Raia Hadsell
    http://raiahadsell.com/index.html

    Sumit Chopra
    https://in.linkedin.com/in/schoprasumit

    Yann LeCun
    http://yann.lecun.com/

    '''
    margin = 1
    return K.mean(y_true * K.square(y_pred) +
                  (1 - y_true) * K.square(K.maximum(margin - y_pred, 0)))


def create_pairs(x, digit_indices):
    '''Positive and negative pair creation.
    正面和負面的創作。
    Alternates between positive and negative pairs.
    在正和負對之間交替。
    '''
    pairs = []
    labels = []
    n = min([len(digit_indices[d]) for d in range(num_classes)]) - 1
    for d in range(num_classes):
        for i in range(n):
            z1, z2 = digit_indices[d][i], digit_indices[d][i + 1]
            pairs += [[x[z1], x[z2]]]
            inc = random.randrange(1, num_classes)
            dn = (d + inc) % num_classes
            z1, z2 = digit_indices[d][i], digit_indices[dn][i]
            pairs += [[x[z1], x[z2]]]
            labels += [1, 0]
    return np.array(pairs), np.array(labels)


def create_base_network(input_shape):
    '''Base network to be shared (eq. to feature extraction).
    共享的基本網路（相當於特徵提取）。
    '''
    input = Input(shape=input_shape)
    x = Flatten()(input)
    x = Dense(128, activation='relu')(x)
    x = Dropout(0.1)(x)
    x = Dense(128, activation='relu')(x)
    x = Dropout(0.1)(x)
    x = Dense(128, activation='relu')(x)
    return Model(input, x)


def compute_accuracy(y_true, y_pred):
    '''Compute classification accuracy with a fixed threshold on distances.
    用固定的閾值計算距離的分類精度。
    '''
    pred = y_pred.ravel() < 0.5
    return np.mean(pred == y_true)


def accuracy(y_true, y_pred):
    '''Compute classification accuracy with a fixed threshold on distances.
    用固定的閾值計算距離的分類精度。
    '''
    return K.mean(K.equal(y_true, K.cast(y_pred < 0.5, y_true.dtype)))


# the data, shuffled and split between train and test sets
# 用於訓練和測試的資料集，經過了篩選（清洗、資料樣本順序打亂）和劃分（劃分為訓練和測試集）
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
input_shape = x_train.shape[1:]

# create training+test positive and negative pairs
# 建立訓練+測試正負兩對
digit_indices = [np.where(y_train == i)[0] for i in range(num_classes)]
tr_pairs, tr_y = create_pairs(x_train, digit_indices)

digit_indices = [np.where(y_test == i)[0] for i in range(num_classes)]
te_pairs, te_y = create_pairs(x_test, digit_indices)

# network definition
# 網路定義
base_network = create_base_network(input_shape)

input_a = Input(shape=input_shape)
input_b = Input(shape=input_shape)

# because we re-use the same instance `base_network`,
# the weights of the network
# will be shared across the two branches
# 因為我們重新使用同一個例項“base_network”，網路的權重將在兩個分支之間共享。
processed_a = base_network(input_a)
processed_b = base_network(input_b)

distance = Lambda(euclidean_distance,
                  output_shape=eucl_dist_output_shape)([processed_a, processed_b])

model = Model([input_a, input_b], distance)

# train
# 訓練
rms = RMSprop()
model.compile(loss=contrastive_loss, optimizer=rms, metrics=[accuracy])
model.fit([tr_pairs[:, 0], tr_pairs[:, 1]], tr_y,
          batch_size=128,
          epochs=epochs,
          validation_data=([te_pairs[:, 0], te_pairs[:, 1]], te_y))

# compute final accuracy on training and test sets
# 計算訓練和測試集的最終準確率
y_pred = model.predict([tr_pairs[:, 0], tr_pairs[:, 1]])
tr_acc = compute_accuracy(tr_y, y_pred)
y_pred = model.predict([te_pairs[:, 0], te_pairs[:, 1]])
te_acc = compute_accuracy(te_y, y_pred)

print('* Accuracy on training set: %0.2f%%' % (100 * tr_acc))
print('* Accuracy on test set: %0.2f%%' % (100 * te_acc))

程式碼執行

C:\ProgramData\Anaconda3\python.exe E:/keras-master/examples/mnist_siamese.py
Using TensorFlow backend.
Train on 108400 samples, validate on 17820 samples
Epoch 1/20

   128/108400 [..............................] - ETA: 19:18 - loss: 1.1304 - accuracy: 0.4531
   640/108400 [..............................] - ETA: 4:00 - loss: 0.4385 - accuracy: 0.5234 
  1024/108400 [..............................] - ETA: 2:36 - loss: 0.3660 - accuracy: 0.5322
  1408/108400 [..............................] - ETA: 1:57 - loss: 0.3319 - accuracy: 0.5405
  1792/108400 [..............................] - ETA: 1:35 - loss: 0.3121 - accuracy: 0.5474
  2176/108400 [..............................] - ETA: 1:20 - loss: 0.2967 - accuracy: 0.5588
  2688/108400 [..............................] - ETA: 1:07 - loss: 0.2839 - accuracy: 0.5666
  2944/108400 [..............................] - ETA: 1:07 - loss: 0.2796 - accuracy: 0.5703
  3328/108400 [..............................] - ETA: 1:01 - loss: 0.2725 - accuracy: 0.5802
  3840/108400 [>.............................] - ETA: 54s - loss: 0.2658 - accuracy: 0.5859 
  4352/108400 [>.............................] - ETA: 49s - loss: 0.2585 - accuracy: 0.5974
  4736/108400 [>.............................] - ETA: 46s - loss: 0.2540 - accuracy: 0.6041
  5248/108400 [>.............................] - ETA: 42s - loss: 0.2479 - accuracy: 0.6151
  5632/108400 [>.............................] - ETA: 40s - loss: 0.2438 - accuracy: 0.6239
  6016/108400 [>.............................] - ETA: 38s - loss: 0.2400 - accuracy: 0.6303
  6528/108400 [>.............................] - ETA: 36s - loss: 0.2360 - accuracy: 0.6374
  7040/108400 [>.............................] - ETA: 34s - loss: 0.2321 - accuracy: 0.6440
  7680/108400 [=>............................] - ETA: 31s - loss: 0.2278 - accuracy: 0.6530
  8192/108400 [=>............................] - ETA: 30s - loss: 0.2248 - accuracy: 0.6592
  8704/108400 [=>............................] - ETA: 29s - loss: 0.2217 - accuracy: 0.6629
  9344/108400 [=>............................] - ETA: 27s - loss: 0.2176 - accuracy: 0.6728
  9984/108400 [=>............................] - ETA: 26s - loss: 0.2139 - accuracy: 0.6801
 10496/108400 [=>............................] - ETA: 25s - loss: 0.2114 - accuracy: 0.6855
 11008/108400 [==>...........................] - ETA: 24s - loss: 0.2082 - accuracy: 0.6920
 11392/108400 [==>...........................] - ETA: 24s - loss: 0.2064 - accuracy: 0.6951
 11776/108400 [==>...........................] - ETA: 23s - loss: 0.2043 - accuracy: 0.6994
 12160/108400 [==>...........................] - ETA: 23s - loss: 0.2022 - accuracy: 0.7034
 12672/108400 [==>...........................] - ETA: 22s - loss: 0.2000 - accuracy: 0.7080
 13184/108400 [==>...........................] - ETA: 22s - loss: 0.1977 - accuracy: 0.7125
 13696/108400 [==>...........................] - ETA: 21s - loss: 0.1954 - accuracy: 0.7169
 14080/108400 [==>...........................] - ETA: 21s - loss: 0.1941 - accuracy: 0.7188
 14592/108400 [===>..........................] - ETA: 20s - loss: 0.1919 - accuracy: 0.7229
 15104/108400 [===>..........................] - ETA: 20s - loss: 0.1894 - accuracy: 0.7274
 15616/108400 [===>..........................] - ETA: 19s - loss: 0.1878 - accuracy: 0.7301
 16000/108400 [===>..........................] - ETA: 19s - loss: 0.1863 - accuracy: 0.7328
 16384/108400 [===>..........................] - ETA: 19s - loss: 0.1853 - accuracy: 0.7350
 16768/108400 [===>..........................] - ETA: 19s - loss: 0.1837 - accuracy: 0.7380
 17024/108400 [===>..........................] - ETA: 19s - loss: 0.1829 - accuracy: 0.7399
 17408/108400 [===>..........................] - ETA: 19s - loss: 0.1818 - accuracy: 0.7421
 17920/108400 [===>..........................] - ETA: 18s - loss: 0.1801 - accuracy: 0.7457
 18432/108400 [====>.........................] - ETA: 18s - loss: 0.1783 - accuracy: 0.7491
 18944/108400 [====>.........................] - ETA: 18s - loss: 0.1770 - accuracy: 0.7516
 19456/108400 [====>.........................] - ETA: 17s - loss: 0.1757 - accuracy: 0.7539
 19968/108400 [====>.........................] - ETA: 17s - loss: 0.1745 - accuracy: 0.7560
 20480/108400 [====>.........................] - ETA: 17s - loss: 0.1733 - accuracy: 0.7580
 20992/108400 [====>.........................] - ETA: 16s - loss: 0.1719 - accuracy: 0.7604
 21504/108400 [====>.........................] - ETA: 16s - loss: 0.1705 - accuracy: 0.7630
 22016/108400 [=====>........................] - ETA: 16s - loss: 0.1693 - accuracy: 0.7651
 22528/108400 [=====>........................] - ETA: 16s - loss: 0.1683 - accuracy: 0.7667
 23040/108400 [=====>........................] - ETA: 15s - loss: 0.1672 - accuracy: 0.7686
 23552/108400 [=====>........................] - ETA: 15s - loss: 0.1662 - accuracy: 0.7704
 24064/108400 [=====>........................] - ETA: 15s - loss: 0.1652 - accuracy: 0.7724
 24576/108400 [=====>........................] - ETA: 15s - loss: 0.1641 - accuracy: 0.7741
 24960/108400 [=====>........................] - ETA: 15s - loss: 0.1634 - accuracy: 0.7754
 25472/108400 [======>.......................] - ETA: 15s - loss: 0.1624 - accuracy: 0.7773
 25984/108400 [======>.......................] - ETA: 15s - loss: 0.1614 - accuracy: 0.7791
 26496/108400 [======>.......................] - ETA: 14s - loss: 0.1607 - accuracy: 0.7805
 26880/108400 [======>.......................] - ETA: 14s - loss: 0.1600 - accuracy: 0.7818
 27136/108400 [======>.......................] - ETA: 14s - loss: 0.1596 - accuracy: 0.7825
 27520/108400 [======>.......................] - ETA: 14s - loss: 0.1589 - accuracy: 0.7836
 27904/108400 [======>.......................] - ETA: 14s - loss: 0.1582 - accuracy: 0.7851
 28416/108400 [======>.......................] - ETA: 14s - loss: 0.1573 - accuracy: 0.7866
 28544/108400 [======>.......................] - ETA: 14s - loss: 0.1571 - accuracy: 0.7871
 28928/108400 [=======>......................] - ETA: 14s - loss: 0.1564 - accuracy: 0.7882
 29440/108400 [=======>......................] - ETA: 14s - loss: 0.1554 - accuracy: 0.7898
 29952/108400 [=======>......................] - ETA: 14s - loss: 0.1546 - accuracy: 0.7911
 30592/108400 [=======>......................] - ETA: 13s - loss: 0.1535 - accuracy: 0.7931
 31232/108400 [=======>......................] - ETA: 13s - loss: 0.1524 - accuracy: 0.7951
 31744/108400 [=======>......................] - ETA: 13s - loss: 0.1515 - accuracy: 0.7966
 32384/108400 [=======>......................] - ETA: 13s - loss: 0.1505 - accuracy: 0.7985
 32896/108400 [========>.....................] - ETA: 13s - loss: 0.1498 - accuracy: 0.7993
 33408/108400 [========>.....................] - ETA: 12s - loss: 0.1488 - accuracy: 0.8009
 34048/108400 [========>.....................] - ETA: 12s - loss: 0.1479 - accuracy: 0.8027
 34688/108400 [========>.....................] - ETA: 12s - loss: 0.1469 - accuracy: 0.8044
 35200/108400 [========>.....................] - ETA: 12s - loss: 0.1462 - accuracy: 0.8054
 35712/108400 [========>.....................] - ETA: 12s - loss: 0.1456 - accuracy: 0.8065
 36224/108400 [=========>....................] - ETA: 11s - loss: 0.1450 - accuracy: 0.8076
 36736/108400 [=========>....................] - ETA: 11s - loss: 0.1443 - accuracy: 0.8091
 37120/108400 [=========>....................] - ETA: 11s - loss: 0.1437 - accuracy: 0.8101
 37504/108400 [=========>....................] - ETA: 11s - loss: 0.1432 - accuracy: 0.8112
 38016/108400 [=========>....................] - ETA: 11s - loss: 0.1425 - accuracy: 0.8123
 38400/108400 [=========>....................] - ETA: 11s - loss: 0.1419 - accuracy: 0.8133
 38656/108400 [=========>....................] - ETA: 11s - loss: 0.1416 - accuracy: 0.8137
 39040/108400 [=========>....................] - ETA: 11s - loss: 0.1410 - accuracy: 0.8147
 39424/108400 [=========>....................] - ETA: 11s - loss: 0.1406 - accuracy: 0.8156
 39808/108400 [==========>...................] - ETA: 11s - loss: 0.1401 - accuracy: 0.8163
 40192/108400 [==========>...................] - ETA: 11s - loss: 0.1396 - accuracy: 0.8172
 40704/108400 [==========>...................] - ETA: 11s - loss: 0.1390 - accuracy: 0.8182
 41216/108400 [==========>...................] - ETA: 10s - loss: 0.1383 - accuracy: 0.8193
 41728/108400 [==========>...................] - ETA: 10s - loss: 0.1378 - accuracy: 0.8202
 42112/108400 [==========>...................] - ETA: 10s - loss: 0.1372 - accuracy: 0.8211
 42368/108400 [==========>...................] - ETA: 10s - loss: 0.1370 - accuracy: 0.8216
 42624/108400 [==========>...................] - ETA: 10s - loss: 0.1367 - accuracy: 0.8221
 43008/108400 [==========>...................] - ETA: 10s - loss: 0.1363 - accuracy: 0.8228
 43136/108400 [==========>...................] - ETA: 10s - loss: 0.1362 - accuracy: 0.8230
 43520/108400 [===========>..................] - ETA: 10s - loss: 0.1356 - accuracy: 0.8238
 44032/108400 [===========>..................] - ETA: 10s - loss: 0.1350 - accuracy: 0.8250
 44544/108400 [===========>..................] - ETA: 10s - loss: 0.1345 - accuracy: 0.8258
 45056/108400 [===========>..................] - ETA: 10s - loss: 0.1341 - accuracy: 0.8263
 45568/108400 [===========>..................] - ETA: 10s - loss: 0.1336 - accuracy: 0.8272
 45952/108400 [===========>..................] - ETA: 10s - loss: 0.1332 - accuracy: 0.8279
 46208/108400 [===========>..................] - ETA: 10s - loss: 0.1329 - accuracy: 0.8284
 46592/108400 [===========>..................] - ETA: 9s - loss: 0.1325 - accuracy: 0.8291 
 46976/108400 [============>.................] - ETA: 9s - loss: 0.1321 - accuracy: 0.8297
 47488/108400 [============>.................] - ETA: 9s - loss: 0.1317 - accuracy: 0.8303
 47872/108400 [============>.................] - ETA: 9s - loss: 0.1313 - accuracy: 0.8309
 48256/108400 [============>.................] - ETA: 9s - loss: 0.1310 - accuracy: 0.8314
 48640/108400 [============>.................] - ETA: 9s - loss: 0.1306 - accuracy: 0.8321
 49024/108400 [============>.................] - ETA: 9s - loss: 0.1303 - accuracy: 0.8324
 49408/108400 [============>.................] - ETA: 9s - loss: 0.1299 - accuracy: 0.8331
 49792/108400 [============>.................] - ETA: 9s - loss: 0.1295 - accuracy: 0.8337
 50176/108400 [============>.................] - ETA: 9s - loss: 0.1291 - accuracy: 0.8345
 50560/108400 [============>.................] - ETA: 9s - loss: 0.1286 - accuracy: 0.8351
 50944/108400 [=============>................] - ETA: 9s - loss: 0.1284 - accuracy: 0.8356
 51456/108400 [=============>................] - ETA: 9s - loss: 0.1279 - accuracy: 0.8362
 51840/108400 [=============>................] - ETA: 9s - loss: 0.1276 - accuracy: 0.8368
 52224/108400 [=============>................] - ETA: 8s - loss: 0.1272 - accuracy: 0.8374
 52608/108400 [=============>................] - ETA: 8s - loss: 0.1269 - accuracy: 0.8378
 52992/108400 [=============>................] - ETA: 8s - loss: 0.1266 - accuracy: 0.8384
 53504/108400 [=============>................] - ETA: 8s - loss: 0.1262 - accuracy: 0.8392
 54016/108400 [=============>................] - ETA: 8s - loss: 0.1257 - accuracy: 0.8399
 54528/108400 [==============>...............] - ETA: 8s - loss: 0.1252 - accuracy: 0.8408
 55040/108400 [==============>...............] - ETA: 8s - loss: 0.1248 - accuracy: 0.8415
 55680/108400 [==============>...............] - ETA: 8s - loss: 0.1242 - accuracy: 0.8425
 56192/108400 [==============>...............] - ETA: 8s - loss: 0.1238 - accuracy: 0.8432
 56704/108400 [==============>...............] - ETA: 8s - loss: 0.1234 - accuracy: 0.8437
 57216/108400 [==============>...............] - ETA: 7s - loss: 0.1230 - accuracy: 0.8444
 57728/108400 [==============>...............] - ETA: 7s - loss: 0.1227 - accuracy: 0.8448
 58240/108400 [===============>..............] - ETA: 7s - loss: 0.1223 - accuracy: 0.8455
 58880/108400 [===============>..............] - ETA: 7s - loss: 0.1218 - accuracy: 0.8464
 59520/108400 [===============>..............] - ETA: 7s - loss: 0.1213 - accuracy: 0.8473
 60032/108400 [===============>..............] - ETA: 7s - loss: 0.1210 - accuracy: 0.8479
 60544/108400 [===============>..............] - ETA: 7s - loss: 0.1206 - accuracy: 0.8484
 61184/108400 [===============>..............] - ETA: 7s - loss: 0.1200 - accuracy: 0.8494
 61696/108400 [================>.............] - ETA: 7s - loss: 0.1196 - accuracy: 0.8501
 62080/108400 [================>.............] - ETA: 7s - loss: 0.1193 - accuracy: 0.8506
 62592/108400 [================>.............] - ETA: 6s - loss: 0.1189 - accuracy: 0.8511
 63232/108400 [================>.............] - ETA: 6s - loss: 0.1184 - accuracy: 0.8520
 63872/108400 [================>.............] - ETA: 6s - loss: 0.1180 - accuracy: 0.8527
 64384/108400 [================>.............] - ETA: 6s - loss: 0.1177 - accuracy: 0.8532
 64896/108400 [================>.............] - ETA: 6s - loss: 0.1172 - accuracy: 0.8539
 65408/108400 [=================>............] - ETA: 6s - loss: 0.1169 - accuracy: 0.8544
 65920/108400 [=================>............] - ETA: 6s - loss: 0.1165 - accuracy: 0.8549
 66560/108400 [=================>............] - ETA: 6s - loss: 0.1161 - accuracy: 0.8556
 66944/108400 [=================>............] - ETA: 6s - loss: 0.1159 - accuracy: 0.8559
 67328/108400 [=================>............] - ETA: 6s - loss: 0.1157 - accuracy: 0.8563
 67840/108400 [=================>............] - ETA: 6s - loss: 0.1154 - accuracy: 0.8568
 68352/108400 [=================>............] - ETA: 5s - loss: 0.1150 - accuracy: 0.8573
 68864/108400 [==================>...........] - ETA: 5s - loss: 0.1147 - accuracy: 0.8577
 69376/108400 [==================>...........] - ETA: 5s - loss: 0.1143 - accuracy: 0.8583
 69888/108400 [==================>...........] - ETA: 5s - loss: 0.1140 - accuracy: 0.8588
 70400/108400 [==================>...........] - ETA: 5s - loss: 0.1137 - accuracy: 0.8592
 70784/108400 [==================>...........] - ETA: 5s - loss: 0.1134 - accuracy: 0.8597
 71296/108400 [==================>...........] - ETA: 5s - loss: 0.1132 - accuracy: 0.8601
 71936/108400 [==================>...........] - ETA: 5s - loss: 0.1128 - accuracy: 0.8608
 72576/108400 [===================>..........] - ETA: 5s - loss: 0.1124 - accuracy: 0.8615
 73216/108400 [===================>..........] - ETA: 5s - loss: 0.1120 - accuracy: 0.8621
 73856/108400 [===================>..........] - ETA: 4s - loss: 0.1116 - accuracy: 0.8626
 74496/108400 [===================>..........] - ETA: 4s - loss: 0.1112 - accuracy: 0.8633
 75136/108400 [===================>..........] - ETA: 4s - loss: 0.1108 - accuracy: 0.8638
 75648/108400 [===================>..........] - ETA: 4s - loss: 0.1105 - accuracy: 0.8644
 76160/108400 [====================>.........] - ETA: 4s - loss: 0.1101 - accuracy: 0.8650
 76800/108400 [====================>.........] - ETA: 4s - loss: 0.1098 - accuracy: 0.8655
 77312/108400 [====================>.........] - ETA: 4s - loss: 0.1095 - accuracy: 0.8658
 77952/108400 [====================>.........] - ETA: 4s - loss: 0.1091 - accuracy: 0.8663
 78592/108400 [====================>.........] - ETA: 4s - loss: 0.1088 - accuracy: 0.8669
 79232/108400 [====================>.........] - ETA: 4s - loss: 0.1085 - accuracy: 0.8673
 79872/108400 [=====================>........] - ETA: 4s - loss: 0.1081 - accuracy: 0.8680
 80384/108400 [=====================>........] - ETA: 3s - loss: 0.1078 - accuracy: 0.8685
 81024/108400 [=====================>........] - ETA: 3s - loss: 0.1074 - accuracy: 0.8691
 81664/108400 [=====================>........] - ETA: 3s - loss: 0.1071 - accuracy: 0.8696
 82304/108400 [=====================>........] - ETA: 3s - loss: 0.1066 - accuracy: 0.8702
 82944/108400 [=====================>........] - ETA: 3s - loss: 0.1062 - accuracy: 0.8708
 83584/108400 [======================>.......] - ETA: 3s - loss: 0.1059 - accuracy: 0.8714
 84224/108400 [======================>.......] - ETA: 3s - loss: 0.1055 - accuracy: 0.8719
 84864/108400 [======================>.......] - ETA: 3s - loss: 0.1051 - accuracy: 0.8724
 85504/108400 [======================>.......] - ETA: 3s - loss: 0.1048 - accuracy: 0.8729
 86144/108400 [======================>.......] - ETA: 3s - loss: 0.1045 - accuracy: 0.8734
 86784/108400 [=======================>......] - ETA: 2s - loss: 0.1042 - accuracy: 0.8738
 87424/108400 [=======================>......] - ETA: 2s - loss: 0.1039 - accuracy: 0.8743
 87808/108400 [=======================>......] - ETA: 2s - loss: 0.1038 - accuracy: 0.8746
 88320/108400 [=======================>......] - ETA: 2s - loss: 0.1035 - accuracy: 0.8750
 88960/108400 [=======================>......] - ETA: 2s - loss: 0.1031 - accuracy: 0.8754
 89600/108400 [=======================>......] - ETA: 2s - loss: 0.1028 - accuracy: 0.8759
 90112/108400 [=======================>......] - ETA: 2s - loss: 0.1026 - accuracy: 0.8763
 90624/108400 [========================>.....] - ETA: 2s - loss: 0.1023 - accuracy: 0.8767
 91264/108400 [========================>.....] - ETA: 2s - loss: 0.1019 - accuracy: 0.8772
 91904/108400 [========================>.....] - ETA: 2s - loss: 0.1016 - accuracy: 0.8776
 92416/108400 [========================>.....] - ETA: 2s - loss: 0.1014 - accuracy: 0.8780
 92928/108400 [========================>.....] - ETA: 2s - loss: 0.1012 - accuracy: 0.8782
 93440/108400 [========================>.....] - ETA: 2s - loss: 0.1010 - accuracy: 0.8786
 93952/108400 [=========================>....] - ETA: 1s - loss: 0.1007 - accuracy: 0.8790
 94464/108400 [=========================>....] - ETA: 1s - loss: 0.1005 - accuracy: 0.8794
 94848/108400 [=========================>....] - ETA: 1s - loss: 0.1003 - accuracy: 0.8796
 95360/108400 [=========================>....] - ETA: 1s - loss: 0.1000 - accuracy: 0.8800
 95744/108400 [=========================>....] - ETA: 1s - loss: 0.0998 - accuracy: 0.8803
 96128/108400 [=========================>....] - ETA: 1s - loss: 0.0997 - accuracy: 0.8805
 96512/108400 [=========================>....] - ETA: 1s - loss: 0.0995 - accuracy: 0.8807
 96896/108400 [=========================>....] - ETA: 1s - loss: 0.0994 - accuracy: 0.8809
 97024/108400 [=========================>....] - ETA: 1s - loss: 0.0993 - accuracy: 0.8810
 97152/108400 [=========================>....] - ETA: 1s - loss: 0.0992 - accuracy: 0.8811
 97408/108400 [=========================>....] - ETA: 1s - loss: 0.0992 - accuracy: 0.8812
 97792/108400 [==========================>...] - ETA: 1s - loss: 0.0990 - accuracy: 0.8814
 98304/108400 [==========================>...] - ETA: 1s - loss: 0.0988 - accuracy: 0.8817
 98944/108400 [==========================>...] - ETA: 1s - loss: 0.0985 - accuracy: 0.8820
 99584/108400 [==========================>...] - ETA: 1s - loss: 0.0983 - accuracy: 0.8824
100224/108400 [==========================>...] - ETA: 1s - loss: 0.0980 - accuracy: 0.8828
100864/108400 [==========================>...] - ETA: 1s - loss: 0.0977 - accuracy: 0.8831
101504/108400 [===========================>..] - ETA: 0s - loss: 0.0975 - accuracy: 0.8834
102144/108400 [===========================>..] - ETA: 0s - loss: 0.0973 - accuracy: 0.8838
102784/108400 [===========================>..] - ETA: 0s - loss: 0.0970 - accuracy: 0.8840
103424/108400 [===========================>..] - ETA: 0s - loss: 0.0967 - accuracy: 0.8844
104064/108400 [===========================>..] - ETA: 0s - loss: 0.0964 - accuracy: 0.8849
104704/108400 [===========================>..] - ETA: 0s - loss: 0.0961 - accuracy: 0.8853
105216/108400 [============================>.] - ETA: 0s - loss: 0.0959 - accuracy: 0.8856
105728/108400 [============================>.] - ETA: 0s - loss: 0.0958 - accuracy: 0.8858
106240/108400 [============================>.] - ETA: 0s - loss: 0.0956 - accuracy: 0.8861
106624/108400 [============================>.] - ETA: 0s - loss: 0.0954 - accuracy: 0.8863
107136/108400 [============================>.] - ETA: 0s - loss: 0.0952 - accuracy: 0.8866
107776/108400 [============================>.] - ETA: 0s - loss: 0.0950 - accuracy: 0.8868
108400/108400 [==============================] - 15s 138us/step - loss: 0.0948 - accuracy: 0.8871 - val_loss: 0.0449 - val_accuracy: 0.9506
Epoch 2/20

   128/108400 [..............................] - ETA: 15s - loss: 0.0733 - accuracy: 0.9219
   640/108400 [..............................] - ETA: 13s - loss: 0.0597 - accuracy: 0.9344
  1152/108400 [..............................] - ETA: 13s - loss: 0.0567 - accuracy: 0.9401
  1664/108400 [..............................] - ETA: 12s - loss: 0.0517 - accuracy: 0.9477
  2176/108400 [..............................] - ETA: 13s - loss: 0.0502 - accuracy: 0.9490
  2432/108400 [..............................] - ETA: 14s - loss: 0.0511 - accuracy: 0.9457
  2816/108400 [..............................] - ETA: 14s - loss: 0.0520 - accuracy: 0.9457
  3328/108400 [..............................] - ETA: 13s - loss: 0.0513 - accuracy: 0.9471
  3840/108400 [>.............................] - ETA: 13s - loss: 0.0523 - accuracy: 0.9448
  4352/108400 [>.............................] - ETA: 13s - loss: 0.0508 - accuracy: 0.9472
  4864/108400 [>.............................] - ETA: 13s - loss: 0.0506 - accuracy: 0.9480
  5248/108400 [>.............................] - ETA: 13s - loss: 0.0497 - accuracy: 0.9495
  5632/108400 [>.............................] - ETA: 13s - loss: 0.0496 - accuracy: 0.9496
  6144/108400 [>.............................] - ETA: 12s - loss: 0.0498 - accuracy: 0.9492
  6656/108400 [>.............................] - ETA: 12s - loss: 0.0503 - accuracy: 0.9483
  7168/108400 [>.............................] - ETA: 12s - loss: 0.0511 - accuracy: 0.9480
  7680/108400 [=>............................] - ETA: 12s - loss: 0.0513 - accuracy: 0.9478
  8192/108400 [=>............................] - ETA: 12s - loss: 0.0509 - accuracy: 0.9487
  8704/108400 [=>............................] - ETA: 12s - loss: 0.0508 - accuracy: 0.9492
  9344/108400 [=>............................] - ETA: 11s - loss: 0.0510 - accuracy: 0.9488
  9984/108400 [=>............................] - ETA: 11s - loss: 0.0510 - accuracy: 0.9488
 10624/108400 [=>............................] - ETA: 11s - loss: 0.0508 - accuracy: 0.9493
 11264/108400 [==>...........................] - ETA: 10s - loss: 0.0510 - accuracy: 0.9487
 11904/108400 [==>...........................] - ETA: 10s - loss: 0.0511 - accuracy: 0.9486
 12416/108400 [==>...........................] - ETA: 10s - loss: 0.0510 - accuracy: 0.9485
 13056/108400 [==>...........................] - ETA: 10s - loss: 0.0507 - accuracy: 0.9488
 13696/108400 [==>...........................] - ETA: 10s - loss: 0.0509 - accuracy: 0.9486
 14336/108400 [==>...........................] - ETA: 10s - loss: 0.0509 - accuracy: 0.9482
 14976/108400 [===>..........................] - ETA: 10s - loss: 0.0504 - accuracy: 0.9491
 15616/108400 [===>..........................] - ETA: 9s - loss: 0.0504 - accuracy: 0.9486 
 16256/108400 [===>..........................] - ETA: 9s - loss: 0.0501 - accuracy: 0.9490
 16896/108400 [===>..........................] - ETA: 9s - loss: 0.0501 - accuracy: 0.9488
 17536/108400 [===>..........................] - ETA: 9s - loss: 0.0499 - accuracy: 0.9492
 18176/108400 [====>.........................] - ETA: 9s - loss: 0.0496 - accuracy: 0.9497
 18816/108400 [====>.........................] - ETA: 9s - loss: 0.0496 - accuracy: 0.9495
 19456/108400 [====>.........................] - ETA: 9s - loss: 0.0496 - accuracy: 0.9496
 20096/108400 [====>.........................] - ETA: 9s - loss: 0.0492 - accuracy: 0.9500
 20736/108400 [====>.........................] - ETA: 8s - loss: 0.0490 - accuracy: 0.9502
 21376/108400 [====>.........................] - ETA: 8s - loss: 0.0488 - accuracy: 0.9502
 22016/108400 [=====>........................] - ETA: 8s - loss: 0.0486 - accuracy: 0.9503
 22528/108400 [=====>........................] - ETA: 8s - loss: 0.0485 - accuracy: 0.9504
 23168/108400 [=====>........................] - ETA: 8s - loss: 0.0482 - accuracy: 0.9506
 23808/108400 [=====>........................] - ETA: 8s - loss: 0.0481 - accuracy: 0.9506
 24448/108400 [=====>........................] - ETA: 8s - loss: 0.0481 - accuracy: 0.9506
 25088/108400 [=====>........................] - ETA: 8s - loss: 0.0481 - accuracy: 0.9507
 25728/108400 [======>.......................] - ETA: 8s - loss: 0.0479 - accuracy: 0.9508
 26368/108400 [======>.......................] - ETA: 8s - loss: 0.0477 - accuracy: 0.9512
 27008/108400 [======>.......................] - ETA: 8s - loss: 0.0477 - accuracy: 0.9512
 27648/108400 [======>.......................] - ETA: 8s - loss: 0.0478 - accuracy: 0.9512
 28288/108400 [======>.......................] - ETA: 7s - loss: 0.0478 - accuracy: 0.9513
 28800/108400 [======>.......................] - ETA: 7s - loss: 0.0477 - accuracy: 0.9513
 29440/108400 [=======>......................] - ETA: 7s - loss: 0.0476 - accuracy: 0.9515
 29952/108400 [=======>......................] - ETA: 7s - loss: 0.0475 - accuracy: 0.9515
 30592/108400 [=======>......................] - ETA: 7s - loss: 0.0473 - accuracy: 0.9517
 31232/108400 [=======>......................] - ETA: 7s - loss: 0.0473 - accuracy: 0.9517
 31744/108400 [=======>......................] - ETA: 7s - loss: 0.0473 - accuracy: 0.9519
 32000/108400 [=======>......................] - ETA: 7s - loss: 0.0473 - accuracy: 0.9521
 32256/108400 [=======>......................] - ETA: 7s - loss: 0.0473 - accuracy: 0.9519
 32512/108400 [=======>......................] - ETA: 7s - loss: 0.0473 - accuracy: 0.9520
 32896/108400 [========>.....................] - ETA: 7s - loss: 0.0473 - accuracy: 0.9522
 33280/108400 [========>.....................] - ETA: 7s - loss: 0.0472 - accuracy: 0.9523
 33536/108400 [========>.....................] - ETA: 7s - loss: 0.0472 - accuracy: 0.9524
 33920/108400 [========>.....................] - ETA: 7s - loss: 0.0472 - accuracy: 0.9524
 34176/108400 [========>.....................] - ETA: 7s - loss: 0.0472 - accuracy: 0.9524
 34432/108400 [========>.....................] - ETA: 8s - loss: 0.0472 - accuracy: 0.9524
 34688/108400 [========>.....................] - ETA: 8s - loss: 0.0472 - accuracy: 0.9525
 34944/108400 [========>.....................] - ETA: 8s - loss: 0.0471 - accuracy: 0.9526
 35328/108400 [========>.....................] - ETA: 8s - loss: 0.0470 - accuracy: 0.9527
 35584/108400 [========>.....................] - ETA: 8s - loss: 0.0470 - accuracy: 0.9527
 35968/108400 [========>.....................] - ETA: 8s - loss: 0.0469 - accuracy: 0.9527
 36352/108400 [=========>....................] - ETA: 8s - loss: 0.0468 - accuracy: 0.9529
 36736/108400 [=========>....................] - ETA: 8s - loss: 0.0468 - accuracy: 0.9529
 36864/108400 [=========>....................] - ETA: 8s - loss: 0.0468 - accuracy: 0.9528
 37120/108400 [=========>....................] - ETA: 8s - loss: 0.0467 - accuracy: 0.9529
 37376/108400 [=========>....................] - ETA: 8s - loss: 0.0467 - accuracy: 0.9530
 37760/108400 [=========>....................] - ETA: 8s - loss: 0.0467 - accuracy: 0.9531
 38016/108400 [=========>....................] - ETA: 8s - loss: 0.0466 - accuracy: 0.9532
 38272/108400 [=========>....................] - ETA: 8s - loss: 0.0466 - accuracy: 0.9532
 38528/108400 [=========>....................] - ETA: 8s - loss: 0.0466 - accuracy: 0.9532
 38912/108400 [=========>....................] - ETA: 8s - loss: 0.0465 - accuracy: 0.9533
 39296/108400 [=========>....................] - ETA: 8s - loss: 0.0465 - accuracy: 0.9534
 39680/108400 [=========>....................] - ETA: 8s - loss: 0.0463 - accuracy: 0.9535
 40064/108400 [==========>...................] - ETA: 8s - loss: 0.0463 - accuracy: 0.9536
 40448/108400 [==========>...................] - ETA: 8s - loss: 0.0463 - accuracy: 0.9537
 40832/108400 [==========>...................] - ETA: 8s - loss: 0.0462 - accuracy: 0.9538
 41088/108400 [==========>...................] - ETA: 8s - loss: 0.0462 - accuracy: 0.9538
 41472/108400 [==========>...................] - ETA: 8s - loss: 0.0461 - accuracy: 0.9538
 41856/108400 [==========>...................] - ETA: 8s - loss: 0.0461 - accuracy: 0.9538
 42240/108400 [==========>...................] - ETA: 8s - loss: 0.0461 - accuracy: 0.9536
 42624/108400 [==========>...................] - ETA: 8s - loss: 0.0462 - accuracy: 0.9536
 43136/108400 [==========>...................] - ETA: 8s - loss: 0.0462 - accuracy: 0.9535
 43648/108400 [===========>..................] - ETA: 8s - loss: 0.0462 - accuracy: 0.9533
 44032/108400 [===========>..................] - ETA: 8s - loss: 0.0462 - accuracy: 0.9535
 44416/108400 [===========>..................] - ETA: 8s - loss: 0.0461 - accuracy: 0.9535
 44800/108400 [===========>..................] - ETA: 8s - loss: 0.0461 - accuracy: 0.9536
 45184/108400 [===========>..................] - ETA: 8s - loss: 0.0460 - accuracy: 0.9537
 45696/108400 [===========>..................] - ETA: 7s - loss: 0.0459 - accuracy: 0.9537
 46208/108400 [===========>..................] - ETA: 7s - loss: 0.0459 - accuracy: 0.9536
 46720/108400 [===========>..................] - ETA: 7s - loss: 0.0458 - accuracy: 0.9538
 47232/108400 [============>.................] - ETA: 7s - loss: 0.0458 - accuracy: 0.9539
 47744/108400 [============>.................] - ETA: 7s - loss: 0.0457 - accuracy: 0.9540
 48256/108400 [============>.................] - ETA: 7s - loss: 0.0457 - accuracy: 0.9541
 48768/108400 [============>.................] - ETA: 7s - loss: 0.0457 - accuracy: 0.9540
 49280/108400 [============>.................] - ETA: 7s - loss: 0.0456 - accuracy: 0.9543
 49920/108400 [============>.................] - ETA: 7s - loss: 0.0456 - accuracy: 0.9544
 50432/108400 [============>.................] - ETA: 7s - loss: 0.0455 - accuracy: 0.9545
 51072/108400 [=============>................] - ETA: 7s - loss: 0.0453 - accuracy: 0.9548
 51584/108400 [=============>................] - ETA: 7s - loss: 0.0452 - accuracy: 0.9549
 51968/108400 [=============>................] - ETA: 7s - loss: 0.0452 - accuracy: 0.9549
 52480/108400 [=============>................] - ETA: 6s - loss: 0.0452 - accuracy: 0.9549
 53120/108400 [=============>................] - ETA: 6s - loss: 0.0452 - accuracy: 0.9548
 53760/108400 [=============>................] - ETA: 6s - loss: 0.0451 - accuracy: 0.9549
 54272/108400 [==============>...............] - ETA: 6s - loss: 0.0451 - accuracy: 0.9549
 54784/108400 [==============>...............] - ETA: 6s - loss: 0.0451 - accuracy: 0.9548
 55424/108400 [==============>...............] - ETA: 6s - loss: 0.0450 - accuracy: 0.9551
 56064/108400 [==============>...............] - ETA: 6s - loss: 0.0449 - accuracy: 0.9552
 56704/108400 [==============>...............] - ETA: 6s - loss: 0.0448 - accuracy: 0.9553
 57344/108400 [==============>...............] - ETA: 6s - loss: 0.0448 - accuracy: 0.9553
 57984/108400 [===============>..............] - ETA: 6s - loss: 0.0448 - accuracy: 0.9553
 58624/108400 [===============>..............] - ETA: 6s - loss: 0.0447 - accuracy: 0.9554
 59264/108400 [===============>..............] - ETA: 5s - loss: 0.0447 - accuracy: 0.9553
 59904/108400 [===============>..............] - ETA: 5s - loss: 0.0446 - accuracy: 0.9554
 60544/108400 [===============>..............] - ETA: 5s - loss: 0.0447 - accuracy: 0.9553
 61184/108400 [===============>..............] - ETA: 5s - loss: 0.0446 - accuracy: 0.9554
 61824/108400 [================>.............] - ETA: 5s - loss: 0.0446 - accuracy: 0.9554
 62336/108400 [================>.............] - ETA: 5s - loss: 0.0445 - accuracy: 0.9555
 62848/108400 [================>.............] - ETA: 5s - loss: 0.0444 - accuracy: 0.9556
 63360/108400 [================>.............] - ETA: 5s - loss: 0.0444 - accuracy: 0.9555
 63872/108400 [================>.............] - ETA: 5s - loss: 0.0443 - accuracy: 0.9555
 64384/108400 [================>.............] - ETA: 5s - loss: 0.0443 - accuracy: 0.9556
 64896/108400 [================>.............] - ETA: 5s - loss: 0.0442 - accuracy: 0.9557
 65280/108400 [=================>............] - ETA: 5s - loss: 0.0442 - accuracy: 0.9556
 65792/108400 [=================>............] - ETA: 5s - loss: 0.0442 - accuracy: 0.9556
 66432/108400 [=================>............] - ETA: 4s - loss: 0.0441 - accuracy: 0.9557
 67072/108400 [=================>............] - ETA: 4s - loss: 0.0440 - accuracy: 0.9558
 67712/108400 [=================>............] - ETA: 4s - loss: 0.0439 - accuracy: 0.9559
 68352/108400 [=================>............] - ETA: 4s - loss: 0.0440 - accuracy: 0.9559
 68992/108400 [==================>...........] - ETA: 4s - loss: 0.0439 - accuracy: 0.9559
 69632/108400 [==================>...........] - ETA: 4s - loss: 0.0439 - accuracy: 0.9558
 70272/108400 [==================>...........] - ETA: 4s - loss: 0.0439 - accuracy: 0.9560
 70912/108400 [==================>...........] - ETA: 4s - loss: 0.0438 - accuracy: 0.9561
 71552/108400 [==================>...........] - ETA: 4s - loss: 0.0437 - accuracy: 0.9562
 72192/108400 [==================>...........] - ETA: 4s - loss: 0.0437 - accuracy: 0.9562
 72832/108400 [===================>..........] - ETA: 4s - loss: 0.0437 - accuracy: 0.9562
 73472/108400 [===================>..........] - ETA: 4s - loss: 0.0437 - accuracy: 0.9563
 74112/108400 [===================>..........] - ETA: 3s - loss: 0.0436 - accuracy: 0.9563
 74752/108400 [===================>..........] - ETA: 3s - loss: 0.0436 - accuracy: 0.9564
 75392/108400 [===================>..........] - ETA: 3s - loss: 0.0435 - accuracy: 0.9565
 76032/108400 [====================>.........] - ETA: 3s - loss: 0.0435 - accuracy: 0.9565
 76672/108400 [====================>.........] - ETA: 3s - loss: 0.0434 - accuracy: 0.9566
 77312/108400 [====================>.........] - ETA: 3s - loss: 0.0433 - accuracy: 0.9567
 77952/108400 [====================>.........] - ETA: 3s - loss: 0.0433 - accuracy: 0.9566
 78592/108400 [====================>.........] - ETA: 3s - loss: 0.0433 - accuracy: 0.9566
 78848/108400 [====================>.........] - ETA: 3s - loss: 0.0433 - accuracy: 0.9566
 79104/108400 [====================>.........] - ETA: 3s - loss: 0.0433 - accuracy: 0.9567
 79616/108400 [=====================>........] - ETA: 3s - loss: 0.0432 - accuracy: 0.9567
 79872/108400 [=====================>........] - ETA: 3s - loss: 0.0432 - accuracy: 0.9568
 80128/108400 [=====================>........] - ETA: 3s - loss: 0.0432 - accuracy: 0.9568
 80384/108400 [=====================>........] - ETA: 3s - loss: 0.0432 - accuracy: 0.9568
 80768/108400 [=====================>........] - ETA: 3s - loss: 0.0432 - accuracy: 0.9567
 81152/108400 [=====================>........] - ETA: 3s - loss: 0.0432 - accuracy: 0.9567
 81664/108400 [=====================>........] - ETA: 3s - loss: 0.0431 - accuracy: 0.9567
 82176/108400 [=====================>........] - ETA: 3s - loss: 0.0431 - accuracy: 0.9567
 82688/108400 [=====================>........] - ETA: 3s - loss: 0.0431 - accuracy: 0.9568
 83200/108400 [======================>.......] - ETA: 2s - loss: 0.0430 - accuracy: 0.9568
 83712/108400 [======================>.......] - ETA: 2s - loss: 0.0430 - accuracy: 0.9568
 84224/108400 [======================>.......] - ETA: 2s - loss: 0.0430 - accuracy: 0.9569
 84736/108400 [======================>.......] - ETA: 2s - loss: 0.0429 - accuracy: 0.9569
 85248/108400 [======================>.......] - ETA: 2s - loss: 0.0429 - accuracy: 0.9570
 85760/108400 [======================>.......] - ETA: 2s - loss: 0.0428 - accuracy: 0.9570
 86272/108400 [======================>.......] - ETA: 2s - loss: 0.0428 - accuracy: 0.9571
 86784/108400 [=======================>......] - ETA: 2s - loss: 0.0427 - accuracy: 0.9572
 87296/108400 [=======================>......] - ETA: 2s - loss: 0.0427 - accuracy: 0.9573
 87808/108400 [=======================>......] - ETA: 2s - loss: 0.0426 - accuracy: 0.9574
 88320/108400 [=======================>......] - ETA: 2s - loss: 0.0426 - accuracy: 0.9574
 88832/108400 [=======================>......] - ETA: 2s - loss: 0.0425 - accuracy: 0.9575
 89344/108400 [=======================>......] - ETA: 2s - loss: 0.0425 - accuracy: 0.9574
 89856/108400 [=======================>......] - ETA: 2s - loss: 0.0425 - accuracy: 0.9575
 90368/108400 [========================>.....] - ETA: 2s - loss: 0.0425 - accuracy: 0.9575
 90880/108400 [========================>.....] - ETA: 2s - loss: 0.0424 - accuracy: 0.9575
 91392/108400 [========================>.....] - ETA: 2s - loss: 0.0424 - accuracy: 0.9575
 91904/108400 [========================>.....] - ETA: 1s - loss: 0.0424 - accuracy: 0.9575
 92544/108400 [========================>.....] - ETA: 1s - loss: 0.0423 - accuracy: 0.9575
 93184/108400 [========================>.....] - ETA: 1s - loss: 0.0423 - accuracy: 0.9576
 93824/108400 [========================>.....] - ETA: 1s - loss: 0.0423 - accuracy: 0.9576
 94208/108400 [=========================>....] - ETA: 1s - loss: 0.0422 - accuracy: 0.9577
 94848/108400 [=========================>....] - ETA: 1s - loss: 0.0422 - accuracy: 0.9577
 95488/108400 [=========================>....] - ETA: 1s - loss: 0.0422 - accuracy: 0.9577
 96128/108400 [=========================>....] - ETA: 1s - loss: 0.0421 - accuracy: 0.9579
 96384/108400 [=========================>....] - ETA: 1s - loss: 0.0421 - accuracy: 0.9579
 96768/108400 [=========================>....] - ETA: 1s - loss: 0.0420 - accuracy: 0.9579
 97408/108400 [=========================>....] - ETA: 1s - loss: 0.0419 - accuracy: 0.9580
 98048/108400 [==========================>...] - ETA: 1s - loss: 0.0419 - accuracy: 0.9580
 98560/108400 [==========================>...] - ETA: 1s - loss: 0.0419 - accuracy: 0.9581
 99200/108400 [==========================>...] - ETA: 1s - loss: 0.0419 - accuracy: 0.9582
 99840/108400 [==========================>...] - ETA: 0s - loss: 0.0418 - accuracy: 0.9582
100480/108400 [==========================>...] - ETA: 0s - loss: 0.0418 - accuracy: 0.9582
101120/108400 [==========================>...] - ETA: 0s - loss: 0.0418 - accuracy: 0.9582
101760/108400 [===========================>..] - ETA: 0s - loss: 0.0417 - accuracy: 0.9583
102400/108400 [===========================>..] - ETA: 0s - loss: 0.0417 - accuracy: 0.9584
103040/108400 [===========================>..] - ETA: 0s - loss: 0.0416 - accuracy: 0.9584
103680/108400 [===========================>..] - ETA: 0s - loss: 0.0416 - accuracy: 0.9585
104320/108400 [===========================>..] - ETA: 0s - loss: 0.0415 - accuracy: 0.9586
104960/108400 [============================>.] - ETA: 0s - loss: 0.0414 - accuracy: 0.9586
105600/108400 [============================>.] - ETA: 0s - loss: 0.0414 - accuracy: 0.9587
106112/108400 [============================>.] - ETA: 0s - loss: 0.0413 - accuracy: 0.9588
106752/108400 [============================>.] - ETA: 0s - loss: 0.0413 - accuracy: 0.9588
107392/108400 [============================>.] - ETA: 0s - loss: 0.0412 - accuracy: 0.9589
107904/108400 [============================>.] - ETA: 0s - loss: 0.0412 - accuracy: 0.9589
108400/108400 [==============================] - 13s 121us/step - loss: 0.0412 - accuracy: 0.9589 - val_loss: 0.0320 - val_accuracy: 0.9685
Epoch 3/20

   128/108400 [..............................] - ETA: 13s - loss: 0.0147 - accuracy: 1.0000
   768/108400 [..............................] - ETA: 9s - loss: 0.0338 - accuracy: 0.9688 
  1408/108400 [..............................] - ETA: 9s - loss: 0.0344 - accuracy: 0.9666
  2048/108400 [..............................] - ETA: 9s - loss: 0.0327 - accuracy: 0.9678
  2688/108400 [..............................] - ETA: 9s - loss: 0.0328 - accuracy: 0.9684
  3328/108400 [..............................] - ETA: 9s - loss: 0.0329 - accuracy: 0.9691
  3968/108400 [>.............................] - ETA: 9s - loss: 0.0325 - accuracy: 0.9698
  4608/108400 [>.............................] - ETA: 9s - loss: 0.0330 - accuracy: 0.9685
  5248/108400 [>.............................] - ETA: 9s - loss: 0.0328 - accuracy: 0.9688
  5888/108400 [>.............................] - ETA: 8s - loss: 0.0325 - accuracy: 0.9689
  6528/108400 [>.............................] - ETA: 8s - loss: 0.0329 - accuracy: 0.9681
  7168/108400 [>.............................] - ETA: 8s - loss: 0.0331 - accuracy: 0.9682
  7808/108400 [=>............................] - ETA: 8s - loss: 0.0327 - accuracy: 0.9682
  8448/108400 [=>............................] - ETA: 8s - loss: 0.0326 - accuracy: 0.9684
  9088/108400 [=>............................] - ETA: 8s - loss: 0.0326 - accuracy: 0.9683
  9728/108400 [=>............................] - ETA: 8s - loss: 0.0325 - accuracy: 0.9685
 10368/108400 [=>............................] - ETA: 8s - loss: 0.0323 - accuracy: 0.9686
 10880/108400 [==>...........................] - ETA: 8s - loss: 0.0323 - accuracy: 0.9689
 11392/108400 [==>...........................] - ETA: 8s - loss: 0.0324 - accuracy: 0.9686
 11904/108400 [==>...........................] - ETA: 8s - loss: 0.0325 - accuracy: 0.9683
 12416/108400 [==>...........................] - ETA: 8s - loss: 0.0326 - accuracy: 0.9686
 12800/108400 [==>...........................] - ETA: 8s - loss: 0.0324 - accuracy: 0.9688
 13312/108400 [==>...........................] - ETA: 8s - loss: 0.0323 - accuracy: 0.9689
 13824/108400 [==>...........................] - ETA: 9s - loss: 0.0323 - accuracy: 0.9690
 14336/108400 [==>...........................] - ETA: 9s - loss: 0.0321 - accuracy: 0.9692
 14848/108400 [===>..........................] - ETA: 9s - loss: 0.0324 - accuracy: 0.9689
 15232/108400 [===>..........................] - ETA: 9s - loss: 0.0325 - accuracy: 0.9686
 15616/108400 [===>..........................] - ETA: 9s - loss: 0.0325 - accuracy: 0.9684
 16128/108400 [===>..........................] - ETA: 9s - loss: 0.0326 - accuracy: 0.9684
 16640/108400 [===>..........................] - ETA: 9s - loss: 0.0327 - accuracy: 0.9681
 17152/108400 [===>..........................] - ETA: 9s - loss: 0.0327 - accuracy: 0.9682
 17664/108400 [===>..........................] - ETA: 9s - loss: 0.0328 - accuracy: 0.9680
 18176/108400 [====>.........................] - ETA: 8s - loss: 0.0326 - accuracy: 0.9681
 18688/108400 [====>.........................] - ETA: 8s - loss: 0.0326 - accuracy: 0.9682
 19328/108400 [====>.........................] - ETA: 8s - loss: 0.0324 - accuracy: 0.9685
 19968/108400 [====>.........................] - ETA: 8s - loss: 0.0325 - accuracy: 0.9682
 20352/108400 [====>.........................] - ETA: 8s - loss: 0.0327 - accuracy: 0.9680
 20992/108400 [====>.........................] - ETA: 8s - loss: 0.0329 - accuracy: 0.9677
 21632/108400 [====>.........................] - ETA: 8s - loss: 0.0329 - accuracy: 0.9676
 22144/108400 [=====>........................] - ETA: 8s - loss: 0.0327 - accuracy: 0.9676
 22656/108400 [=====>........................] - ETA: 8s - loss: 0.0327 - accuracy: 0.9676
 23296/108400 [=====>........................] - ETA: 8s - loss: 0.0327 - accuracy: 0.9677
 23936/108400 [=====>........................] - ETA: 8s - loss: 0.0325 - accuracy: 0.9681
 24576/108400 [=====>........................] - ETA: 8s - loss: 0.0322 - accuracy: 0.9685
 25216/108400 [=====>........................] - ETA: 8s - loss: 0.0322 - accuracy: 0.9686
 25856/108400 [======>.......................] - ETA: 8s - loss: 0.0320 - accuracy: 0.9689
 26496/108400 [======>.......................] - ETA: 8s - loss: 0.0320 - accuracy: 0.9689
 27008/108400 [======>.......................] - ETA: 8s - loss: 0.0319 - accuracy: 0.9690
 27520/108400 [======>.......................] - ETA: 7s - loss: 0.0319 - accuracy: 0.9690
 28160/108400 [======>.......................] - ETA: 7s - loss: 0.0319 - accuracy: 0.9689
 28800/108400 [======>.......................] - ETA: 7s - loss: 0.0318 - accuracy: 0.9690
 29440/108400 [=======>......................] - ETA: 7s - loss: 0.0317 - accuracy: 0.9691
 30080/108400 [=======>......................] - ETA: 7s - loss: 0.0316 - accuracy: 0.9692
 30720/108400 [=======>......................] - ETA: 7s - loss: 0.0316 - accuracy: 0.9693
 31360/108400 [=======>......................] - ETA: 7s - loss: 0.0316 - accuracy: 0.9695
 32000/108400 [=======>......................] - ETA: 7s - loss: 0.0316 - accuracy: 0.9693
 32640/108400 [========>.....................] - ETA: 7s - loss: 0.0315 - accuracy: 0.9695
 33280/108400 [========>.....................] - ETA: 7s - loss: 0.0316 - accuracy: 0.9693
 33920/108400 [========>.....................] - ETA: 7s - loss: 0.0315 - accuracy: 0.9694
 34560/108400 [========>.....................] - ETA: 7s - loss: 0.0315 - accuracy: 0.9694
 35200/108400 [========>.....................] - ETA: 7s - loss: 0.0314 - accuracy: 0.9696
 35712/108400 [========>.....................] - ETA: 7s - loss: 0.0315 - accuracy: 0.9695
 36224/108400 [=========>....................] - ETA: 7s - loss: 0.0315 - accuracy: 0.9693
 36736/108400 [=========>....................] - ETA: 6s - loss: 0.0315 - accuracy: 0.9695
 37248/108400 [=========>....................] - ETA: 6s - loss: 0.0316 - accuracy: 0.9693
 37760/108400 [=========>....................] - ETA: 6s - loss: 0.0317 - accuracy: 0.9692
 38272/108400 [=========>....................] - ETA: 6s - loss: 0.0316 - accuracy: 0.9694
 38912/108400 [=========>....................] - ETA: 6s - loss: 0.0315 - accuracy: 0.9695
 39552/108400 [=========>....................] - ETA: 6s - loss: 0.0315 - accuracy: 0.9695
 40192/108400 [==========>...................] - ETA: 6s - loss: 0.0314 - accuracy: 0.9696
 40832/108400 [==========>...................] - ETA: 6s - loss: 0.0314 - accuracy: 0.9696
 41472/108400 [==========>...................] - ETA: 6s - loss: 0.0314 - accuracy: 0.9696
 41984/108400 [==========>...................] - ETA: 6s - loss: 0.0315 - accuracy: 0.9694
 42624/108400 [==========>...................] - ETA: 6s - loss: 0.0314 - accuracy: 0.9695
 43264/108400 [==========>...................] - ETA: 6s - loss: 0.0313 - accuracy: 0.9696
 43904/108400 [===========>..................] - ETA: 6s - loss: 0.0312 - accuracy: 0.9697
 44544/108400 [===========>..................] - ETA: 6s - loss: 0.0312 - accuracy: 0.9696
 45184/108400 [===========>..................] - ETA: 6s - loss: 0.0313 - accuracy: 0.9695
 45824/108400 [===========>..................] - ETA: 6s - loss: 0.0313 - accuracy: 0.9694
 46464/108400 [===========>..................] - ETA: 5s - loss: 0.0313 - accuracy: 0.9694
 47104/108400 [============>.................] - ETA: 5s - loss: 0.0313 - accuracy: 0.9695
 47744/108400 [============>.................] - ETA: 5s - loss: 0.0314 - accuracy: 0.9693
 48384/108400 [============>.................] - ETA: 5s - loss: 0.0312 - accuracy: 0.9695
 49024/108400 [============>.................] - ETA: 5s - loss: 0.0311 - accuracy: 0.9696
 49664/108400 [============>.................] - ETA: 5s - loss: 0.0311 - accuracy: 0.9697
 50304/108400 [============>.................] - ETA: 5s - loss: 0.0310 - accuracy: 0.9697
 50944/108400 [=============>................] - ETA: 5s - loss: 0.0310 - accuracy: 0.9697
 51456/108400 [=============>................] - ETA: 5s - loss: 0.0309 - accuracy: 0.9698
 52096/108400 [=============>................] - ETA: 5s - loss: 0.0309 - accuracy: 0.9699
 52608/108400 [=============>................] - ETA: 5s - loss: 0.0309 - accuracy: 0.9698
 53120/108400 [=============>................] - ETA: 5s - loss: 0.0309 - accuracy: 0.9698
 53760/108400 [=============>................] - ETA: 5s - loss: 0.0308 - accuracy: 0.9700
 54400/108400 [==============>...............] - ETA: 5s - loss: 0.0307 - accuracy: 0.9700
 55040/108400 [==============>...............] - ETA: 5s - loss: 0.0307 - accuracy: 0.9700
 55680/108400 [==============>...............] - ETA: 5s - loss: 0.0306 - accuracy: 0.9701
 56320/108400 [==============>...............] - ETA: 4s - loss: 0.0306 - accuracy: 0.9702
 56832/108400 [==============>...............] - ETA: 4s - loss: 0.0306 - accuracy: 0.9702
 57472/108400 [==============>...............] - ETA: 4s - loss: 0.0306 - accuracy: 0.9702
 58112/108400 [===============>..............] - ETA: 4s - loss: 0.0307 - accuracy: 0.9701
 58752/108400 [===============>..............] - ETA: 4s - loss: 0.0307 - accuracy: 0.9701
 59392/108400 [===============>..............] - ETA: 4s - loss: 0.0307 - accuracy: 0.9700
 60032/108400 [===============>..............] - ETA: 4s - loss: 0.0308 - accuracy: 0.9700
 60672/108400 [===============>..............] - ETA: 4s - loss: 0.0307 - accuracy: 0.9700
 61312/108400 [===============>..............] - ETA: 4s - loss: 0.0307 - accuracy: 0.9701
 61952/108400 [================>.............] - ETA: 4s - loss: 0.0306 - accuracy: 0.9702
 62592/108400 [================>.............] - ETA: 4s - loss: 0.0306 - accuracy: 0.9703
 63232/108400 [================>.............] - ETA: 4s - loss: 0.0305 - accuracy: 0.9704
 63872/108400 [================>.............] - ETA: 4s - loss: 0.0304 - accuracy: 0.9705
 64512/108400 [================>.............] - ETA: 4s - loss: 0.0304 - accuracy: 0.9704
 65152/108400 [=================>............] - ETA: 4s - loss: 0.0304 - accuracy: 0.9704
 65792/108400 [=================>............] - ETA: 4s - loss: 0.0303 - accuracy: 0.9704
 66432/108400 [=================>............] - ETA: 3s - loss: 0.0303 - accuracy: 0.9705
 67072/108400 [=================>............] - ETA: 3s - loss: 0.0302 - accuracy: 0.9706
 67712/108400 [=================>............] - ETA: 3s - loss: 0.0302 - accuracy: 0.9707
 68352/108400 [=================>............] - ETA: 3s - loss: 0.0302 - accuracy: 0.9707
 68864/108400 [==================>...........] - ETA: 3s - loss: 0.0301 - accuracy: 0.9707
 69376/108400 [==================>...........] - ETA: 3s - loss: 0.0301 - accuracy: 0.9708
 69760/108400 [==================>...........] - ETA: 3s - loss: 0.0301 - accuracy: 0.9708
 70144/108400 [==================>...........] - ETA: 3s - loss: 0.0301 - accuracy: 0.9708
 70528/108400 [==================>...........] - ETA: 3s - loss: 0.0301 - accuracy: 0.9708
 70912/108400 [==================>...........] - ETA: 3s - loss: 0.0301 - accuracy: 0.9707
 71552/108400 [==================>...........] - ETA: 3s - loss: 0.0300 - accuracy: 0.9708
 72192/108400 [==================>...........] - ETA: 3s - loss: 0.0300 - accuracy: 0.9708
 72832/108400 [===================>..........] - ETA: 3s - loss: 0.0300 - accuracy: 0.9708
 73472/108400 [===================>..........] - ETA: 3s - loss: 0.0300 - accuracy: 0.9708
 74112/108400 [===================>..........] - ETA: 3s - loss: 0.0300 - accuracy: 0.9708
 74752/108400 [===================>..........] - ETA: 3s - loss: 0.0300 - accuracy: 0.9709
 75392/108400 [===================>..........] - ETA: 3s - loss: 0.0299 - accuracy: 0.9709
 76032/108400 [====================>.........] - ETA: 3s - loss: 0.0299 - accuracy: 0.9709
 76672/108400 [====================>.........] - ETA: 2s - loss: 0.0299 - accuracy: 0.9710
 77312/108400 [====================>.........] - ETA: 2s - loss: 0.0299 - accuracy: 0.9710
 77952/108400 [====================>.........] - ETA: 2s - loss: 0.0299 - accuracy: 0.9709
 78592/108400 [====================>.........] - ETA: 2s - loss: 0.0299 - accuracy: 0.9710
 79232/108400 [====================>.........] - ETA: 2s - loss: 0.0298 - accuracy: 0.9710
 79872/108400 [=====================>........] - ETA: 2s - loss: 0.0298 - accuracy: 0.9710
 80512/108400 [=====================>........] - ETA: 2s - loss: 0.0298 - accuracy: 0.9710
 81152/108400 [=====================>........] - ETA: 2s - loss: 0.0298 - accuracy: 0.9710
 81792/108400 [=====================>........] - ETA: 2s - loss: 0.0298 - accuracy: 0.9710
 82432/108400 [=====================>........] - ETA: 2s - loss: 0.0298 - accuracy: 0.9710
 83072/108400 [=====================>........] - ETA: 2s - loss: 0.0297 - accuracy: 0.9710
 83712/108400 [======================>.......] - ETA: 2s - loss: 0.0298 - accuracy: 0.9710
 84352/108400 [======================>.......] - ETA: 2s - loss: 0.0298 - accuracy: 0.9710
 84992/108400 [======================>.......] - ETA: 2s - loss: 0.0297 - accuracy: 0.9710
 85632/108400 [======================>.......] - ETA: 2s - loss: 0.0297 - accuracy: 0.9710
 86272/108400 [======================>.......] - ETA: 2s - loss: 0.0297 - accuracy: 0.9710
 86912/108400 [=======================>......] - ETA: 2s - loss: 0.0297 - accuracy: 0.9711
 87552/108400 [=======================>......] - ETA: 1s - loss: 0.0297 - accuracy: 0.9710
 88192/108400 [=======================>......] - ETA: 1s - loss: 0.0296 - accuracy: 0.9710
 88832/108400 [=======================>......] - ETA: 1s - loss: 0.0296 - accuracy: 0.9710
 89472/108400 [=======================>......] - ETA: 1s - loss: 0.0295 - accuracy: 0.9711
 90112/108400 [=======================>......] - ETA: 1s - loss: 0.0295 - accuracy: 0.9711
 90752/108400 [========================>.....] - ETA: 1s - loss: 0.0295 - accuracy: 0.9711
 91392/108400 [========================>.....] - ETA: 1s - loss: 0.0295 - accuracy: 0.9711
 92032/108400 [========================>.....] - ETA: 1s - loss: 0.0294 - accuracy: 0.9711
 92672/108400 [========================>.....] - ETA: 1s - loss: 0.0294 - accuracy: 0.9711
 93312/108400 [========================>.....] - ETA: 1s - loss: 0.0294 - accuracy: 0.9712
 93952/108400 [=========================>....] - ETA: 1s - loss: 0.0294 - accuracy: 0.9712
 94592/108400 [=========================>....] - ETA: 1s - loss: 0.0293 - accuracy: 0.9712
 95232/108400 [=========================>....] - ETA: 1s - loss: 0.0294 - accuracy: 0.9712
 95872/108400 [=========================>....] - ETA: 1s - loss: 0.0293 - accuracy: 0.9712
 96512/108400 [=========================>....] - ETA: 1s - loss: 0.0293 - accuracy: 0.9712
 97152/108400 [=========================>....] - ETA: 1s - loss: 0.0293 - accuracy: 0.9713
 97792/108400 [==========================>...] - ETA: 0s - loss: 0.0293 - accuracy: 0.9713
 98432/108400 [==========================>...] - ETA: 0s - loss: 0.0293 - accuracy: 0.9713
 99072/108400 [==========================>...] - ETA: 0s - loss: 0.0292 - accuracy: 0.9713
 99712/108400 [==========================>...] - ETA: 0s - loss: 0.0292 - accuracy: 0.9713
100352/108400 [==========================>...] - ETA: 0s - loss: 0.0293 - accuracy: 0.9712
100992/108400 [==========================>...] - ETA: 0s - loss: 0.0292 - accuracy: 0.9713
101504/108400 [===========================>..] - ETA: 0s - loss: 0.0292 - accuracy: 0.9713
102144/108400 [===========================>..] - ETA: 0s - loss: 0.0291 - accuracy: 0.9714
102784/108400 [===========================>..] - ETA: 0s - loss: 0.0291 - accuracy: 0.9714
103424/108400 [===========================>..] - ETA: 0s - loss: 0.0291 - accuracy: 0.9715
104064/108400 [===========================>..] - ETA: 0s - loss: 0.0290 - accuracy: 0.9715
104704/108400 [===========================>..] - ETA: 0s - loss: 0.0290 - accuracy: 0.9715
105344/108400 [============================>.] - ETA: 0s - loss: 0.0290 - accuracy: 0.9715
105984/108400 [============================>.] - ETA: 0s - loss: 0.0290 - accuracy: 0.9716
106624/108400 [============================>.] - ETA: 0s - loss: 0.0289 - accuracy: 0.9716
107264/108400 [============================>.] - ETA: 0s - loss: 0.0290 - accuracy: 0.9716
107904/108400 [============================>.] - ETA: 0s - loss: 0.0290 - accuracy: 0.9715
108400/108400 [==============================] - 11s 98us/step - loss: 0.0290 - accuracy: 0.9716 - val_loss: 0.0288 - val_accuracy: 0.9693
Epoch 4/20

   128/108400 [..............................] - ETA: 13s - loss: 0.0247 - accuracy: 0.9688
   768/108400 [..............................] - ETA: 10s - loss: 0.0217 - accuracy: 0.9805
  1408/108400 [..............................] - ETA: 9s - loss: 0.0215 - accuracy: 0.9815 
  2048/108400 [..............................] - ETA: 9s - loss: 0.0227 - accuracy: 0.9790
  2560/108400 [..............................] - ETA: 10s - loss: 0.0239 - accuracy: 0.9766
  2944/108400 [..............................] - ETA: 10s - loss: 0.0229 - accuracy: 0.9779
  3456/108400 [..............................] - ETA: 11s - loss: 0.0232 - accuracy: 0.9780
  3840/108400 [>.............................] - ETA: 11s - loss: 0.0235 - accuracy: 0.9779
  4224/108400 [>.............................] - ETA: 11s - loss: 0.0236 - accuracy: 0.9780
  4736/108400 [>.............................] - ETA: 11s - loss: 0.0238 - accuracy: 0.9772
  5248/108400 [>.............................] - ETA: 11s - loss: 0.0231 - accuracy: 0.9783
  5888/108400 [>...................