Artificial Neural Networks and Deep Learning

Keras tutorial - 25/10/2019

Francesco Lattari, PhD student (francesco.lattari@polimi.it)

Artificial Intelligence and Robotics Laboratory

2
3
4
5
6
7
8
9
Being able to go from idea to result with the least possible delay is key to doing good research

https://keras.io/

10
11
12
Data structure: tf.Tensor

13
Data structure: tf.Tensor

14
Data structure: tf.Tensor

15
Data structure: tf.Tensor

Create a tensor

● tf.constant(value, dtype, shape)

● tf.zeros(shape, dtype)

● tf.ones(shape, dtype)

● tf.random
○ normal(
shape, mean, stddev, dtype, seed)
○ uniform(
shape, minval, maxval, dtype, seed)
….

● tf.range(start, limit, delta, dtype)

● tf.convert_to_tensor(value, dtype)

16
Operations on tensors - Tensorflow v1

ones:0 Const:0

add:0

17
Operations on tensors - Tensorflow v1

ones:0 Const:0

add:0

18
Operations on tensors - Tensorflow 2.0

Eager execution
● imperative programming environment
○ operations evaluated immediately

● easier debugging

● natural control flow (Python)

19
Operations on tensors - Tensorflow 2.0

Change data type

● tf.cast(x, dtype)

20
Operations on tensors - Tensorflow 2.0

Reshape

● tf.reshape(tensor, shape)

1 2 3 4 5 6 7 8 9

1 2 3

4 5 6

7 8 9

21
Operations on tensors - Tensorflow 2.0

Reshape

● tf.expand_dims(input, axis)

1 2 3 1 2 3
4 5 6 4 5 6
7 8 9 7 8 9

22
Operations on tensors - Tensorflow 2.0

Reshape

● tf.squeeze(input, axis)

1 2 3 1 2 3

4 5 6 4 5 6

7 8 9 7 8 9

23
Operations on tensors - Tensorflow 2.0

Math operations

● *, +, -, / operators (element-wise)

● tf.add(x, y) (element-wise)
○ tf.add_n(inputs)

● tf.multiply(x, y) (element-wise) https://www.tensorflow.org/api_docs/python/tf/math

○ tf.tensordot(a, b, axes) (matrix multiplication)

● tf.abs(x)

● tf.pow(x, y)

● tf.transpose(a, perm, conjugate)

● …

24
Operations on tensors - Tensorflow 2.0

Slicing

axis 1 Indexing

● pythonic way: “[ ]”
0 1 2 ○ [idx1, idx2, …, idxN]
○ [start:stop:step]
0 1 2 3
● tf.slice(input_, begin, size)
axis 0 1 4 5 6

2 7 8 9 ● using a condition: tf.gather_nd(params, indices)

○ indices = tf.where(condition)

25
Operations on tensors - Tensorflow 2.0

Splitting: tf.split(value, num_or_size_splits, axis)

1 2 3

1 2 3 4 5 6 7 8 9 4 5

6 7 8 9
Stacking: tf.concat(values, axis)

1 2 3

4 5 1 2 3 4 5 6 7 8 9

6 7 8 9

26
Deep Learning Framework Training Loop

Data Optimization

Data
Preparation
Training set Model
Validation set
Test set
Batches
Transformations

Visualization

27
Data Loader

Class tf.data.Dataset
https://www.tensorflow.org/api_docs/python/tf/data/Dataset

● Methods (main)

○ __iter__()

○ batch(batch_size, drop_reminder=False)

○ map(map_fn, num_parallel_calls=None)

○ filter(predicate)

○ shuffle(buffer_size, seed=None, reshuffle_each_iteration=False)

○ …

28
Data Loader

Class tf.data.Dataset
https://www.tensorflow.org/api_docs/python/tf/data/Dataset

● Create a Dataset

○ range(start, stop, step)

○ from_tensors(tensors)

○ from_tensor_slices(tensors)

○ ...

29
Data Loader

tf.keras.datasets
https://www.tensorflow.org/api_docs/python/tf/keras/datasets

● Fashion MNIST

○ 28x28 grayscale images

○ 10 classes:
1. T-shirt/top
2. Trouser/pants
3. Pullover shirt
4. Dress
5. Coat
6. Sandal
7. Shirt
8. Sneaker
9. Bag
10. Ankle boot

30
Model

Class tf.keras.layers
https://www.tensorflow.org/api_docs/python/tf/keras/layers

● Classes of basic building blocks

○ e.g., fully-connected layer

tf.keras.layers.Dense(
units,
activation=None,
use_bias=True,
kernel_initializer='glorot_uniform',
bias_initializer='zeros',
kernel_regularizer=None,
bias_regularizer=None,
activity_regularizer=None,
kernel_constraint=None,
bias_constraint=None,
**kwargs,
)

31
Model

Class tf.keras.layers
https://www.tensorflow.org/api_docs/python/tf/keras/layers

● Classes of basic building blocks

○ e.g., fully-connected layer

tf.keras.layers.Dense(
units,
activation=None,
use_bias=True,
kernel_initializer='glorot_uniform',
tf.keras.activations
bias_initializer='zeros',
kernel_regularizer=None, https://www.tensorflow.org/api_docs/python/tf/keras/activations
bias_regularizer=None,
activity_regularizer=None,
kernel_constraint=None,
bias_constraint=None,
**kwargs,
)

32
Model

Class tf.keras.layers
https://www.tensorflow.org/api_docs/python/tf/keras/layers

● Classes of basic building blocks

○ e.g., fully-connected layer

tf.keras.layers.Dense(
units,
activation=None,
use_bias=True,
kernel_initializer='glorot_uniform',
tf.keras.initializers
bias_initializer='zeros',
kernel_regularizer=None, https://www.tensorflow.org/api_docs/python/tf/initializers
bias_regularizer=None,
activity_regularizer=None,
kernel_constraint=None,
bias_constraint=None,
**kwargs,
)

33
Model

Class tf.keras.Model
https://www.tensorflow.org/api_docs/python/tf/keras/Model

● Groups multiple layers

34
Model

Class tf.keras.Model
https://www.tensorflow.org/api_docs/python/tf/keras/Model

● Groups multiple layers

● Create a Model instance (“functional API”):

1. Instantiate keras Input tensor

https://www.tensorflow.org/api_docs/python/tf/keras/Input

x = tf.keras.Input(shape, dtype, …)

35
Model

Class tf.keras.Model
https://www.tensorflow.org/api_docs/python/tf/keras/Model

● Groups multiple layers

● Create a Model instance (“functional API”):

1. Instantiate keras Input tensor

https://www.tensorflow.org/api_docs/python/tf/keras/Input

x = tf.keras.Input(shape, dtype, …)

2. Instantiate and chain keras layers

layer1 = tf.keras.layers.LAYER_NAME(...)(x)
layer2 = tf.keras.layers.LAYER_NAME(...)(layer1)
….
out = ....

36
Model

Class tf.keras.Model
https://www.tensorflow.org/api_docs/python/tf/keras/Model

● Groups multiple layers

● Create a Model instance (“functional API”):

1. Instantiate keras Input tensor 3. Instantiate Model

https://www.tensorflow.org/api_docs/python/tf/keras/Input
model = tf.keras.Model(inputs=x, outputs=out)
x = tf.keras.Input(shape, dtype, …)

2. Instantiate and chain keras layers

layer1 = tf.keras.layers.LAYER_NAME(...)(x)
layer2 = tf.keras.layers.LAYER_NAME(...)(layer1)
….
out = ....

37
Model

Class tf.keras.Model
https://www.tensorflow.org/api_docs/python/tf/keras/Model

● Groups multiple layers

● Create a Model instance (“functional API”):

1. Instantiate keras Input tensor 3. Instantiate Model

https://www.tensorflow.org/api_docs/python/tf/keras/Input
model = tf.keras.Model(inputs=x, outputs=out)
x = tf.keras.Input(shape, dtype, …)

2. Instantiate and chain keras layers

OR...
layer1 = tf.keras.layers.LAYER_NAME(...)(x)
layer2 = tf.keras.layers.LAYER_NAME(...)(layer1)
….
out = ....

38
Model

Class tf.keras.Sequential
https://www.tensorflow.org/api_docs/python/tf/keras/Sequential

● Stacks multiple layers

● Create a Sequential instance (“functional API”):

○ 2 modalities:

1. model = tf.keras.Sequential([layer1, layer2, …, layerN])

2. model = tf.keras.Sequential()
model.add(layer1)
model.add(layer2)
....
model.add(layerN)

39
Training Loop

Class tf.keras.Model
https://www.tensorflow.org/api_docs/python/tf/keras/Model

● 2 main steps through class methods

1. Prepare model for training

model.compile(
optimizer='rmsprop',
loss=None,
metrics=None,
loss_weights=None,
sample_weight_mode=None,
weighted_metrics=None,
target_tensors=None,
distribute=None,
**kwargs)

40
Training Loop

Class tf.keras.Model
https://www.tensorflow.org/api_docs/python/tf/keras/Model tf.keras.optimizers
https://www.tensorflow.org/api_docs/python/tf/keras/optimizers
● 2 main steps through class methods
● class SGD
1. Prepare model for training

model.compile( ● class Adam

optimizer='rmsprop',
loss=None, ● class Adadelta
metrics=None,
loss_weights=None,
sample_weight_mode=None,
● ...
weighted_metrics=None,
target_tensors=None,
distribute=None,
**kwargs)

41
Training Loop

Class tf.keras.Model
https://www.tensorflow.org/api_docs/python/tf/keras/Model tf.keras.losses
https://www.tensorflow.org/api_docs/python/tf/keras/losses
● 2 main steps through class methods
● class BinaryCrossEntropy
1. Prepare model for training

model.compile( ● class CategoricalCrossEntropy

optimizer='rmsprop',
loss=None, ● class MeanSquaredError
metrics=None,
loss_weights=None,
sample_weight_mode=None,
● ...
weighted_metrics=None,
target_tensors=None,
distribute=None,
**kwargs)

42
Training Loop

Class tf.keras.Model
https://www.tensorflow.org/api_docs/python/tf/keras/Model tf.keras.metrics
https://www.tensorflow.org/api_docs/python/tf/keras/metrics
● 2 main steps through class methods
● class Accuracy
1. Prepare model for training

model.compile( ● class MeanIoU

optimizer='rmsprop',
loss=None, ● class Precision
metrics=None,
loss_weights=None,
sample_weight_mode=None,
● class Recall
weighted_metrics=None,
target_tensors=None, ● ...
distribute=None,
**kwargs)

43
Training Loop

Class tf.keras.Model
https://www.tensorflow.org/api_docs/python/tf/keras/Model
2. Training model

model.fit(
x=None,
● 2 main steps through class methods
y=None,
batch_size=None,
1. Prepare model for training epochs=1,
verbose=1,
model.compile( callbacks=None,
optimizer='rmsprop', validation_split=0.0,
loss=None, validation_data=None,
metrics=None, shuffle=True,
loss_weights=None, class_weight=None,
sample_weight_mode=None, sample_weight=None,
weighted_metrics=None, initial_epoch=0,
target_tensors=None, steps_per_epoch=None,
distribute=None, validation_steps=None,
**kwargs) validation_freq=1,
max_queue_size=10,
workers=1,
use_multiprocessing=False,
**kwargs)

44
Training Loop

Class tf.keras.Model
https://www.tensorflow.org/api_docs/python/tf/keras/Model
2. Training model

model.fit( Returns a History object

x=None, containing a record of
● 2 main steps through class methods
y=None,
loss and metric values at
batch_size=None,
1. Prepare model for training epochs=1, each epoch
verbose=1,
model.compile( callbacks=None,
optimizer='rmsprop', validation_split=0.0,
loss=None, validation_data=None,
metrics=None, shuffle=True,
loss_weights=None, class_weight=None,
sample_weight_mode=None, sample_weight=None,
weighted_metrics=None, initial_epoch=0,
target_tensors=None, steps_per_epoch=None,
distribute=None, validation_steps=None,
**kwargs) validation_freq=1,
max_queue_size=10,
workers=1,
use_multiprocessing=False,
**kwargs)

45
Training Loop

Class tf.keras.Model
https://www.tensorflow.org/api_docs/python/tf/keras/Model
2. Training model

model.fit(
x=None,
y=None,
tf.keras.callbacks batch_size=None,
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks epochs=1,
verbose=1,
callbacks=None,
validation_split=0.0,
● Save model
validation_data=None,
shuffle=True,
● Learning visualization class_weight=None,
sample_weight=None,
● ... initial_epoch=0,
steps_per_epoch=None,
validation_steps=None,
validation_freq=1,
max_queue_size=10,
workers=1,
use_multiprocessing=False,
**kwargs)

46
Test Model

Class tf.keras.Model
https://www.tensorflow.org/api_docs/python/tf/keras/Model

● Compute loss and metrics in test mode ● Compute model output:

evaluate( predict(
x=None, x,
y=None, batch_size=None,
batch_size=None, verbose=0,
verbose=1, steps=None,
sample_weight=None, callbacks=None,
steps=None, max_queue_size=10,
callbacks=None, workers=1,
max_queue_size=10, use_multiprocessing=False
workers=1, )
use_multiprocessing=False
)
Return:
Return:
test loss or list of scalars (for multiple predictions as tensors
outputs and metrics)

47
Save Model

Class tf.keras.callbacks.ModelCheckpoint
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint

● Save model during learning

ModelCheckpoint(
filepath,
monitor='val_loss',
verbose=0,
save_best_only=False,
save_weights_only=False,
mode='auto',
save_freq='epoch',
**kwargs
)

48
Save Model

Class tf.keras.callbacks.ModelCheckpoint
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint

True: save only model weights

● Save model during learning
● Save
ModelCheckpoint(
filepath,
model.save_weights(‘checkpoint_path/filename.ckpt’)
monitor='val_loss',
verbose=0, ● Restore.
save_best_only=False, model = Model(...)
save_weights_only=False,
mode='auto', model.load_weights(‘checkpoint_path/filename.ckpt’)
save_freq='epoch',
**kwargs False: save the entire model
)

● Save
model.save(‘model_path/filename.h5’)

● Restore.
model = tf.keras.models.load_model(‘model_path/filename.h5’)

49
Visualize Learning

TensorBoard: Tensorflow’s visualization toolkit

● Plotting scalars
○ e.g., losses, accuracy, gradients, etc.

● Show images
○ e.g., layer activations, segmentation
results, filters, etc.
● Plotting histograms
○ e.g., gradients and weights distribution

● Show the model graph

● ……

50
Visualize Learning

Class tf.keras.callbacks.Tensorboard
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard

Tensorboard(
log_dir='logs',
histogram_freq=0,
write_graph=True,
write_images=False,
update_freq='epoch',
profile_batch=2,
embeddings_freq=0,
embeddings_metadata=None,
**kwargs
)

51
Recap

● Tensors and their manipulation

● Keras principles
○ Backend
○ Execution (Graph vs. Eager)

● Deep Learning framework

○ Dataset and Data Loader ○ Save and Restore models
○ Model creation ■ callbacks.ModelCheckpoint
■ tf.keras.layers ■ model.save_weights
■ tf.keras.Model ■ model.save
■ tf.keras.Sequential ○ Visualize Learning
○ Model training and validation ■ callbacks.Tensorboard
■ model.fit
■ tf.keras.optimizers
■ tf.keras.losses
○ Model test
■ model.evaluate
■ model.metrics
■ model.predict
52