Deep Learning and TensorFlow

1 TensorFlow 2.x Basics

1.1 Introduction

1.1.1 About This Experiment

This experiment helps trainees understand the basic syntax of TensorFlow 2.x by introducing a series of tensor
operations of TensorFlow 2.x, including tensor creation, slicing, and indexing, tensor dimension modification,
tensor arithmetic operations, and tensor sorting.

1.1.2 Objectives

Upon completion of this task, you will be able to:

 Understand how to create a tensor.

 Master the tensor slicing and indexing methods.
 Master the syntax for tensor dimension modification.
 Master arithmetic operations of tensors.
 Master the tensor sorting method.
 Dive deeper into eager execution and AutoGraph based on code.

1.2 Experiment Steps

1.2.1 Introduction to Tensors

In TensorFlow, tensors are classified into constant tensors and variable tensors.

 A defined constant tensor has an unchangeable value and dimension, and a defined variable tensor has
a changeable value and an unchangeable dimension.
 In neural networks, variable tensors are generally used as matrices for storing weights and other
information, and are a type of trainable data. Constant tensors can be used for storing hyperparameters
or other structured data.

1.2.1.1 Tensor Creation

1.2.1.1.1 Creating a Constant Tensor

Common methods for creating a constant tensor include:

 tf.constant(): creates a constant tensor.

 tf.zeros(), tf.zeros_like(), tf.ones(), and tf.ones_like(): create an all-zero or all-one constant tensor.
 tf.fill(): creates a tensor with a user-defined value.
 tf.random: creates a tensor with a known distribution.
 Creating a list object by using NumPy, and then converting the list object into a tensor by using
tf.convert_to_tensor.

Step 1 tf.constant()

tf.constant(value, dtype=None, shape=None, name='Const'):

  value: A constant value (or list) of output type dtype.
 dtype: The type of the elements of the resulting tensor.
 shape: Optional dimensions of resulting tensor.
 name: Optional name for the tensor.

In [11]:
import tensorflow as tf
const_a = tf.constant([[1, 2, 3, 4]],shape=[2,2], dtype=tf.float32) # Create
a 2x2 matrix with values 1, 2, 3, and 4.
const_a
Out[11]:
<tf.Tensor: shape=(2, 2), dtype=float32, numpy=
array([[1., 2.],
[3., 4.]], dtype=float32)>
In [12]:
#View common attributes.
print("value of the constant const_a:", const_a.numpy())
print("data type of the constant const_a:", const_a.dtype)
print("shape of the constant const_a:", const_a.shape)
print("name of the device that is to generate the constant const_a:",
const_a.device)
value of the constant const_a: [[1. 2.]
[3. 4.]]
data type of the constant const_a: <dtype: 'float32'>
shape of the constant const_a: (2, 2)
name of the device that is to generate the constant const_a:
/job:localhost/replica:0/task:0/device:CPU:0
Step 2 tf.zeros(), tf.zeros_like(), tf.ones(), and tf.ones_like()

Usages of tf.ones() and tf.ones_like() are similar to those of tf.zeros() and tf.zeros_like ().

Therefore, the following describes only the usages of tf.ones() and tf.ones_like().

Create a constant with the value 0 0.

tf.zeros(shape, dtype=tf.float32, name=None):

 shape: A list of integers, a tuple of integers, or a 1-D Tensor of type int32.t

 dtype: The DType of an element in the resulting Tensor.
 name: Optional string. A name for the operation.

In [13]:
zeros_b = tf.zeros(shape=[2, 3], dtype=tf.int32) # Create a 2x3 matrix with
all values being 0.
Create a tensor whose value is 0 0 based on the input tensor, with its shape being the same as that of the input
tensor.

tf.zeros_like(input, dtype=None, name=None):  input_tensor: A Tensor or array-like object.

 dtype: A type for the returned Tensor. Must be float16, float32, float64, int8, uint8, int16, uint16, int32,
int64, complex64, complex128, bool or string (optional).

 name: A name for the operation (optional).

In [14]:
zeros_like_c = tf.zeros_like(const_a)
#View generated data.
zeros_like_c.numpy()
Out[14]:
array([[0., 0.],
[0., 0.]], dtype=float32)
Step 3 tf.fill()

Create a tensor and fill it with a scalar value.

tf.fill(dims, value, name=None):

 dims: A 1-D sequence of non-negative numbers. Represents the shape of the output tf.Tensor. Entries
should be of type: int32, int64.

 value: A value to fill the returned tf.Tensor.

 name: Optional string. The name of the output tf.Tensor.

In [15]:
fill_d = tf.fill([3,3], 8) # Create a 2x3 matrix with all values being 8.
#View data.
fill_d.numpy()
Out[15]:
array([[8, 8, 8],
[8, 8, 8],
[8, 8, 8]], dtype=int32)
Step 4 tf.random

Huawei AI Certification Training Lab Guide Page 10

This module is used to generate a tensor with a specific distribution. Common methods in this module include
tf.random.uniform(), tf.random.normal(), and tf.random.shuffle(). The following describes how to use
tf.random.normal().

Create a tensor that conforms to a normal distribution.

tf.random.normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32,seed=None, name=None) :

 shape: A 1-D integer Tensor or Python array. The shape of the output tensor.
 mean: A Tensor or Python value of type dtype, broadcastable with stddev. The mean of the normal
distribution.

 stddev: A Tensor or Python value of type dtype, broadcastable with mean. The standard deviation of
the normal distribution.

 dtype: The type of the output.

 seed: A Python integer. Used to create a random seed for the distribution. See tf.random.set_seed for
behavior.

 name: A name for the operation (optional).

In [16]:
random_e = tf.random.normal([5,5],mean=0,stddev=1.0, seed = 1)
#View the created data.
random_e.numpy()
Out[16]:
array([[-0.8113182 , 1.4845988 , 0.06532937, -2.4427042 , 0.0992484 ],
 [ 0.5912243 , 0.59282297, -2.1229296 , -0.72289723, -0.05627038],
[ 0.6435448 , -0.26432407, 1.8566332 , 0.5678417 , -0.3828359 ],
[-1.4853433 , 1.2617711 , -0.02530608, -0.2646297 , 1.5328138 ],
[-1.7429771 , -0.43789294, -0.56601 , 0.32066926, 1.132831 ]],
dtype=float32)
Step 5 Create a list object by using NumPy, and then convert the list object into a tensor by using
tf.convert_to_tensor.

This method can convert a given value into a tensor. tf.convert_to_tensor can be used to convert a Python data
type into a tensor data type available to TensorFlow.

tf.convert_to_tensor(value,dtype=None,dtype_hint=None,name=None) :

 value: An object whose type has a registered Tensor conversion function.

 dtype: Optional element type for the returned tensor. If missing, the type is inferred from the type of
value.

 dtype_hint: Optional element type for the returned tensor, used when dtype is None. In some cases, a
caller may not have a dtype in mind when converting to a tensor, so dtype_hint can be used as a soft
preference. If the conversion to dtype_hint is not possible, this argument has no effect.

 Name: Optional name to use if a new Tensor is created.

In [17]:
#Create a list.
list_f = [1,2,3,4,5,6]
#View the data type.
type(list_f)
Out[17]:
list
In [18]:
tensor_f = tf.convert_to_tensor(list_f, dtype=tf.float32)
tensor_f
Out[18]:
<tf.Tensor: shape=(6,), dtype=float32, numpy=array([1., 2., 3., 4., 5., 6.],
dtype=float32)>

1.2.1.1.2 Creating a Variable Tensor

In TensorFlow, variables are operated using the tf.Variable class. tf.Variable indicates a tensor. The value of
tf.Variable can be changed by running an arithmetic operation on tf.Variable. Variable values can be read and
changed.
In [19]:
#Create a variable. Only the initial value needs to be provided.
var_1 = tf.Variable(tf.ones([2,3]))
var_1
Out[19]:
<tf.Variable 'Variable:0' shape=(2, 3) dtype=float32, numpy=
array([[1., 1., 1.],
[1., 1., 1.]], dtype=float32)>
In [20]:
#Read the variable value.
print("Value of the variable var_1:",var_1.read_value())
#Assign a variable value.
var_value_1=[[1,2,3],[4,5,6]]
var_1.assign(var_value_1)
print("Value of the variable var_1 after the assignment:",var_1.read_value())
Value of the variable var_1: tf.Tensor(
[[1. 1. 1.]
[1. 1. 1.]], shape=(2, 3), dtype=float32)
Value of the variable var_1 after the assignment: tf.Tensor(
[[1. 2. 3.]
[4. 5. 6.]], shape=(2, 3), dtype=float32)
In [21]:
#Variable addition
var_1.assign_add(tf.ones([2,3]))
var_1
Out[21]:
<tf.Variable 'Variable:0' shape=(2, 3) dtype=float32, numpy=
array([[2., 3., 4.],
[5., 6., 7.]], dtype=float32)>

1.2.1.2 Tensor Slicing and Indexing

1.2.1.2.1 Slicing

Tensor slicing methods include:

 [start: end]: extracts a data slice from the start position to the end position of the tensor.

 [start:end:step] or [::step]: extracts a data slice at an interval of step from the start position to the end
position of the tensor.

 [::-1]: slices data from the last element.

 '...': indicates a data slice of any length.

In [22]:
#Create a 4-dimensional tensor. The tensor contains four images. The size of
each image is 100 x 100 x 3.
tensor_h = tf.random.normal([4,100,100,3])
tensor_h
Out[22]:
<tf.Tensor: shape=(4, 100, 100, 3), dtype=float32, numpy=
array([[[[-0.42564863, 0.452668 , -0.29914168],
[-1.2919582 , 1.8550993 , -0.05315711],
[-1.1958379 , -1.214941 , 0.36617145],
...,
[ 0.58726525, 0.70759076, -0.46401304],
[-0.5949442 , 0.60752153, -0.95319635],
[-0.2574743 , 0.24064586, 0.34597343]],

[[-1.7168367 , -0.23556498, 2.0132248 ],

[ 1.0574174 , 0.25092965, 0.611986 ],
[-0.4694253 , -0.6934754 , 0.13635111],
...,
[ 0.03584324, -0.7154953 , 0.07054269],
[-0.7138733 , -0.31570685, -0.5378517 ],
[-1.0845814 , 0.72623473, -0.8368647 ]],

[[-0.5114272 , 0.29174542, -1.228559 ],

[ 1.4579431 , -0.8066154 , -0.02132352],
[-0.4740772 , 0.52693737, 0.23475689],
...,
[-1.017584 , -0.24304558, -1.5286679 ],
 [-0.46588707, 1.6216873 , 0.45128444],
[ 0.96607655, 0.53378534, -1.0223454 ]],

...,

[[ 0.34742594, -0.44042996, -1.3424541 ],

[ 1.2630217 , -0.15709393, -1.1029688 ],
[ 0.337064 , 1.0246189 , -0.5582971 ],
...,
[-0.42786717, 1.1596836 , -0.3421803 ],
[-1.8006552 , 0.9324209 , 0.05656384],
[-0.43976578, -0.58924365, -0.2823274 ]],

[[ 0.5410462 , 0.5804195 , 0.29743925],

[-0.5645381 , 0.29361114, 1.5866159 ],
[-0.8229178 , -0.5118068 , 0.6232647 ],
...,
[-0.8210146 , -0.5309559 , 1.7510443 ],
[-1.8660572 , 1.0673795 , -0.6420711 ],
[ 1.3523059 , -1.095501 , 0.56174904]],

[[ 1.4924296 , -0.60755205, 0.9630441 ],

[-1.5792804 , 0.33756173, 0.41448846],
[-2.6247635 , 0.54883015, 1.454688 ],
...,
[ 1.0511454 , 1.9847646 , -0.8202082 ],
[ 0.43942735, -1.8400631 , 0.2689787 ],
[ 0.24013405, -1.1960655 , -0.7239898 ]]],

[[[ 0.827751 , -0.8917146 , -1.3633173 ],

[ 0.37120134, -0.31135547, -1.6114012 ],
[-0.3715497 , 0.12056366, 1.2951739 ],
...,
[ 0.50387484, 0.49553663, -0.9391985 ],
[ 0.16923928, 0.38702163, 0.3049653 ],
[ 1.3247155 , -0.7304066 , 0.91027665]],

[[ 0.4397138 , -0.5790663 , 0.20748606],

[-1.2422135 , -0.95965856, -0.23108931],
[-1.3649904 , -1.0418425 , 0.46117795],
...,
[-0.6164832 , -0.8543789 , -0.18572907],
[ 0.2936265 , 0.40962088, -0.41644526],
[ 0.3442094 , 0.12161341, 0.4152682 ]],

[[ 1.6018158 , 0.05495111, 0.71397877],

[-2.4949882 , 1.4684566 , -1.1969898 ],
[-0.89853895, -1.2358171 , -0.84759676],
...,
[ 2.087779 , 0.7366188 , 0.7473216 ],
[-0.3190504 , -0.03293897, -0.1575045 ],
[-0.8604673 , 0.5231678 , 0.23351063]],

...,

[[ 0.08051784, 1.6063491 , -0.01714951],

[ 0.08994591, -1.3978703 , -0.607305 ],
[-1.3601972 , -0.62083566, -0.33745226],
 ...,
[-0.67440724, -0.9736315 , 0.8896246 ],
[ 1.0974662 , -0.09331249, 0.41488403],
[-0.84917855, -1.0173733 , 0.6883804 ]],

[[ 0.82938236, -0.1012913 , -0.9174921 ],

[ 2.0279603 , 1.7132381 , -0.77499837],
[ 0.50818497, -1.3681097 , 1.182135 ],
...,
[ 1.1404238 , -1.9341455 , -0.33085215],
[ 1.7738689 , -0.2727936 , 0.6107232 ],
[ 0.4003075 , -0.49705872, -1.7602799 ]],

[[-1.134393 , -1.4050153 , -0.77682215],

[-1.1580638 , -0.5361828 , -0.3299927 ],
[ 2.5059564 , -2.1682148 , 0.28446195],
...,
[-1.4590594 , 0.68389434, 0.18419999],
[ 0.38754275, 0.95740974, -1.3241694 ],
[ 0.54431945, 0.23139928, 0.8231759 ]]],

[[[-1.4254456 , 0.39137927, -0.13649423],

[-0.5710502 , 0.10289402, 0.36140317],
[-1.1484516 , -2.7555463 , 2.2230012 ],
...,
[-0.56392187, 0.62368715, 0.38970223],
[-1.2613596 , -0.7723032 , 1.2898194 ],
[ 0.2144069 , -1.9822742 , 1.2304467 ]],

[[ 1.7191933 , -1.0121458 , 0.16656664],

[ 0.05821927, -1.3770248 , -0.6377959 ],
[-0.60149914, -0.43310806, 1.4596988 ],
...,
[-0.5678659 , 0.2236976 , 0.16242246],
[-0.7852512 , -0.8143705 , 1.022717 ],
[-0.36592624, -0.25483608, -0.42377204]],

[[-0.52819216, -0.74893945, -0.5176597 ],

[ 0.13868137, 0.26775536, 3.1764894 ],
[ 0.17698635, 0.85923845, 0.33271807],
...,
[ 0.76272476, 1.3833762 , -0.65704817],
[ 0.2640248 , -1.9969002 , 0.2623298 ],
[ 1.3529757 , -0.00963973, 0.846223 ]],

...,

[[-0.6977969 , 0.12629062, 1.0830774 ],

[-0.9938452 , -0.21627773, -1.6369616 ],
[ 0.6734002 , -0.5706922 , 0.10492811],
...,
[-1.4478308 , -0.26070035, 0.1359629 ],
[-0.34297273, 1.3164726 , 0.34523824],
[-0.17374493, 0.4249873 , -0.85981774]],

[[-0.11244927, 0.07421584, 0.2704961 ],

[-1.4734796 , -0.86446565, -0.86091816],
[ 1.4203255 , -0.5859937 , 0.44925168],
 ...,
[ 1.4119536 , 0.46325207, -3.2448297 ],
[-0.12290744, 0.29577062, -0.39216644],
[-0.4729547 , 0.15344724, 1.408864 ]],

[[ 0.21240012, 0.53249377, -0.00389338],

[-0.5422931 , -0.09353203, 0.03279584],
[ 0.05919829, 1.7881966 , 0.50872993],
...,
[-0.49521384, 1.8982345 , -1.2846154 ],
[ 0.3619637 , 0.8716316 , -1.6442288 ],
[-0.85900116, -0.28337526, -0.23599708]]],

[[[ 1.0388501 , -0.21380688, 0.21120134],

[-0.926739 , -0.8133858 , -0.78145933],
[ 0.9289431 , 0.01986507, 0.48339394],
...,
[-0.95561856, 0.7273471 , -1.3942958 ],
[-2.0075085 , -0.769726 , -0.81108063],
[ 2.384009 , -0.3022256 , 0.27821884]],

[[ 0.16614938, 0.82138044, 1.1175478 ],

[ 0.23566669, 0.0094515 , 2.777578 ],
[-0.8147552 , 0.07749964, -2.3900774 ],
...,
[-1.0257695 , -1.1510082 , 0.08606431],
[ 1.2651023 , 0.6840444 , -0.4371489 ],
[ 1.4022624 , -0.41605678, 0.5049461 ]],

[[-0.26702768, -0.94770133, 1.3767972 ],

[ 0.13892083, 0.00883611, -1.4245614 ],
[-1.0700448 , 0.46347347, -2.1798024 ],
...,
[ 0.9927806 , -0.5713902 , -1.5768932 ],
[ 0.44241902, -2.2897117 , 0.4850132 ],
[-0.31186196, -0.46309343, 0.6004569 ]],

...,

[[-0.25572157, -0.481574 , -0.62847406],

[-0.18530175, -0.31511047, -0.17287329],
[-1.172901 , 0.43398705, 0.26872268],
...,
[-0.15042119, -0.45772648, -0.01517491],
[-0.84364545, 0.30128956, -0.29066485],
[ 0.25015455, -0.11824514, 0.46264866]],

[[ 0.51299995, -0.45022494, 0.26325727],

[-0.58410096, -0.53932184, 0.2789639 ],
[-1.2701533 , -0.11238922, -0.40922546],
...,
[-2.1618803 , -1.7747833 , -2.5344229 ],
[ 0.16259412, 1.4623609 , -0.75771034],
[ 0.55835825, -1.0756673 , -0.15727316]],

[[ 1.1368815 , -0.29778495, 0.7555838 ],

[ 0.7729792 , -2.0782697 , -0.25594652],
[-2.5195763 , 1.17271 , -1.8258004 ],
 ...,
[ 1.2047101 , 0.02295183, 0.13764495],
[-0.5464439 , 0.42274612, -0.40945956],
[ 0.18586746, -0.57554436, -0.49103737]]]], dtype=float32)>
In [23]:
#Extract the first image.
tensor_h[0,:,:,:]
Out[23]:
<tf.Tensor: shape=(100, 100, 3), dtype=float32, numpy=
array([[[-0.42564863, 0.452668 , -0.29914168],
[-1.2919582 , 1.8550993 , -0.05315711],
[-1.1958379 , -1.214941 , 0.36617145],
...,
[ 0.58726525, 0.70759076, -0.46401304],
[-0.5949442 , 0.60752153, -0.95319635],
[-0.2574743 , 0.24064586, 0.34597343]],

[[-1.7168367 , -0.23556498, 2.0132248 ],

[ 1.0574174 , 0.25092965, 0.611986 ],
[-0.4694253 , -0.6934754 , 0.13635111],
...,
[ 0.03584324, -0.7154953 , 0.07054269],
[-0.7138733 , -0.31570685, -0.5378517 ],
[-1.0845814 , 0.72623473, -0.8368647 ]],

[[-0.5114272 , 0.29174542, -1.228559 ],

[ 1.4579431 , -0.8066154 , -0.02132352],
[-0.4740772 , 0.52693737, 0.23475689],
...,
[-1.017584 , -0.24304558, -1.5286679 ],
[-0.46588707, 1.6216873 , 0.45128444],
[ 0.96607655, 0.53378534, -1.0223454 ]],

...,

[[ 0.34742594, -0.44042996, -1.3424541 ],

[ 1.2630217 , -0.15709393, -1.1029688 ],
[ 0.337064 , 1.0246189 , -0.5582971 ],
...,
[-0.42786717, 1.1596836 , -0.3421803 ],
[-1.8006552 , 0.9324209 , 0.05656384],
[-0.43976578, -0.58924365, -0.2823274 ]],

[[ 0.5410462 , 0.5804195 , 0.29743925],

[-0.5645381 , 0.29361114, 1.5866159 ],
[-0.8229178 , -0.5118068 , 0.6232647 ],
...,
[-0.8210146 , -0.5309559 , 1.7510443 ],
[-1.8660572 , 1.0673795 , -0.6420711 ],
[ 1.3523059 , -1.095501 , 0.56174904]],

[[ 1.4924296 , -0.60755205, 0.9630441 ],

[-1.5792804 , 0.33756173, 0.41448846],
[-2.6247635 , 0.54883015, 1.454688 ],
...,
[ 1.0511454 , 1.9847646 , -0.8202082 ],
[ 0.43942735, -1.8400631 , 0.2689787 ],
[ 0.24013405, -1.1960655 , -0.7239898 ]]], dtype=float32)>
 In [24]:
#Extract one slice at an interval of two images.
tensor_h[::2,...]
Out[24]:
<tf.Tensor: shape=(2, 100, 100, 3), dtype=float32, numpy=
array([[[[-0.42564863, 0.452668 , -0.29914168],
[-1.2919582 , 1.8550993 , -0.05315711],
[-1.1958379 , -1.214941 , 0.36617145],
...,
[ 0.58726525, 0.70759076, -0.46401304],
[-0.5949442 , 0.60752153, -0.95319635],
[-0.2574743 , 0.24064586, 0.34597343]],

[[-1.7168367 , -0.23556498, 2.0132248 ],

[ 1.0574174 , 0.25092965, 0.611986 ],
[-0.4694253 , -0.6934754 , 0.13635111],
...,
[ 0.03584324, -0.7154953 , 0.07054269],
[-0.7138733 , -0.31570685, -0.5378517 ],
[-1.0845814 , 0.72623473, -0.8368647 ]],

[[-0.5114272 , 0.29174542, -1.228559 ],

[ 1.4579431 , -0.8066154 , -0.02132352],
[-0.4740772 , 0.52693737, 0.23475689],
...,
[-1.017584 , -0.24304558, -1.5286679 ],
[-0.46588707, 1.6216873 , 0.45128444],
[ 0.96607655, 0.53378534, -1.0223454 ]],

...,

[[ 0.34742594, -0.44042996, -1.3424541 ],

[ 1.2630217 , -0.15709393, -1.1029688 ],
[ 0.337064 , 1.0246189 , -0.5582971 ],
...,
[-0.42786717, 1.1596836 , -0.3421803 ],
[-1.8006552 , 0.9324209 , 0.05656384],
[-0.43976578, -0.58924365, -0.2823274 ]],

[[ 0.5410462 , 0.5804195 , 0.29743925],

[-0.5645381 , 0.29361114, 1.5866159 ],
[-0.8229178 , -0.5118068 , 0.6232647 ],
...,
[-0.8210146 , -0.5309559 , 1.7510443 ],
[-1.8660572 , 1.0673795 , -0.6420711 ],
[ 1.3523059 , -1.095501 , 0.56174904]],

[[ 1.4924296 , -0.60755205, 0.9630441 ],

[-1.5792804 , 0.33756173, 0.41448846],
[-2.6247635 , 0.54883015, 1.454688 ],
...,
[ 1.0511454 , 1.9847646 , -0.8202082 ],
[ 0.43942735, -1.8400631 , 0.2689787 ],
[ 0.24013405, -1.1960655 , -0.7239898 ]]],

[[[-1.4254456 , 0.39137927, -0.13649423],

[-0.5710502 , 0.10289402, 0.36140317],
[-1.1484516 , -2.7555463 , 2.2230012 ],
 ...,
[-0.56392187, 0.62368715, 0.38970223],
[-1.2613596 , -0.7723032 , 1.2898194 ],
[ 0.2144069 , -1.9822742 , 1.2304467 ]],

[[ 1.7191933 , -1.0121458 , 0.16656664],

[ 0.05821927, -1.3770248 , -0.6377959 ],
[-0.60149914, -0.43310806, 1.4596988 ],
...,
[-0.5678659 , 0.2236976 , 0.16242246],
[-0.7852512 , -0.8143705 , 1.022717 ],
[-0.36592624, -0.25483608, -0.42377204]],

[[-0.52819216, -0.74893945, -0.5176597 ],

[ 0.13868137, 0.26775536, 3.1764894 ],
[ 0.17698635, 0.85923845, 0.33271807],
...,
[ 0.76272476, 1.3833762 , -0.65704817],
[ 0.2640248 , -1.9969002 , 0.2623298 ],
[ 1.3529757 , -0.00963973, 0.846223 ]],

...,

[[-0.6977969 , 0.12629062, 1.0830774 ],

[-0.9938452 , -0.21627773, -1.6369616 ],
[ 0.6734002 , -0.5706922 , 0.10492811],
...,
[-1.4478308 , -0.26070035, 0.1359629 ],
[-0.34297273, 1.3164726 , 0.34523824],
[-0.17374493, 0.4249873 , -0.85981774]],

[[-0.11244927, 0.07421584, 0.2704961 ],

[-1.4734796 , -0.86446565, -0.86091816],
[ 1.4203255 , -0.5859937 , 0.44925168],
...,
[ 1.4119536 , 0.46325207, -3.2448297 ],
[-0.12290744, 0.29577062, -0.39216644],
[-0.4729547 , 0.15344724, 1.408864 ]],

[[ 0.21240012, 0.53249377, -0.00389338],

[-0.5422931 , -0.09353203, 0.03279584],
[ 0.05919829, 1.7881966 , 0.50872993],
...,
[-0.49521384, 1.8982345 , -1.2846154 ],
[ 0.3619637 , 0.8716316 , -1.6442288 ],
[-0.85900116, -0.28337526, -0.23599708]]]], dtype=float32)>
In [25]:
#Slice data from the last element.
tensor_h[::-1]
Out[25]:
<tf.Tensor: shape=(4, 100, 100, 3), dtype=float32, numpy=
array([[[[ 1.0388501 , -0.21380688, 0.21120134],
[-0.926739 , -0.8133858 , -0.78145933],
[ 0.9289431 , 0.01986507, 0.48339394],
...,
[-0.95561856, 0.7273471 , -1.3942958 ],
[-2.0075085 , -0.769726 , -0.81108063],
[ 2.384009 , -0.3022256 , 0.27821884]],
[[ 0.16614938, 0.82138044, 1.1175478 ],
[ 0.23566669, 0.0094515 , 2.777578 ],
[-0.8147552 , 0.07749964, -2.3900774 ],
...,
[-1.0257695 , -1.1510082 , 0.08606431],
[ 1.2651023 , 0.6840444 , -0.4371489 ],
[ 1.4022624 , -0.41605678, 0.5049461 ]],

[[-0.26702768, -0.94770133, 1.3767972 ],

[ 0.13892083, 0.00883611, -1.4245614 ],
[-1.0700448 , 0.46347347, -2.1798024 ],
...,
[ 0.9927806 , -0.5713902 , -1.5768932 ],
[ 0.44241902, -2.2897117 , 0.4850132 ],
[-0.31186196, -0.46309343, 0.6004569 ]],

...,

[[-0.25572157, -0.481574 , -0.62847406],

[-0.18530175, -0.31511047, -0.17287329],
[-1.172901 , 0.43398705, 0.26872268],
...,
[-0.15042119, -0.45772648, -0.01517491],
[-0.84364545, 0.30128956, -0.29066485],
[ 0.25015455, -0.11824514, 0.46264866]],

[[ 0.51299995, -0.45022494, 0.26325727],

[-0.58410096, -0.53932184, 0.2789639 ],
[-1.2701533 , -0.11238922, -0.40922546],
...,
[-2.1618803 , -1.7747833 , -2.5344229 ],
[ 0.16259412, 1.4623609 , -0.75771034],
[ 0.55835825, -1.0756673 , -0.15727316]],

[[ 1.1368815 , -0.29778495, 0.7555838 ],

[ 0.7729792 , -2.0782697 , -0.25594652],
[-2.5195763 , 1.17271 , -1.8258004 ],
...,
[ 1.2047101 , 0.02295183, 0.13764495],
[-0.5464439 , 0.42274612, -0.40945956],
[ 0.18586746, -0.57554436, -0.49103737]]],

[[[-1.4254456 , 0.39137927, -0.13649423],

[-0.5710502 , 0.10289402, 0.36140317],
[-1.1484516 , -2.7555463 , 2.2230012 ],
...,
[-0.56392187, 0.62368715, 0.38970223],
[-1.2613596 , -0.7723032 , 1.2898194 ],
[ 0.2144069 , -1.9822742 , 1.2304467 ]],

[[ 1.7191933 , -1.0121458 , 0.16656664],

[ 0.05821927, -1.3770248 , -0.6377959 ],
[-0.60149914, -0.43310806, 1.4596988 ],
...,
[-0.5678659 , 0.2236976 , 0.16242246],
[-0.7852512 , -0.8143705 , 1.022717 ],
[-0.36592624, -0.25483608, -0.42377204]],
[[-0.52819216, -0.74893945, -0.5176597 ],
[ 0.13868137, 0.26775536, 3.1764894 ],
[ 0.17698635, 0.85923845, 0.33271807],
...,
[ 0.76272476, 1.3833762 , -0.65704817],
[ 0.2640248 , -1.9969002 , 0.2623298 ],
[ 1.3529757 , -0.00963973, 0.846223 ]],

...,

[[-0.6977969 , 0.12629062, 1.0830774 ],

[-0.9938452 , -0.21627773, -1.6369616 ],
[ 0.6734002 , -0.5706922 , 0.10492811],
...,
[-1.4478308 , -0.26070035, 0.1359629 ],
[-0.34297273, 1.3164726 , 0.34523824],
[-0.17374493, 0.4249873 , -0.85981774]],

[[-0.11244927, 0.07421584, 0.2704961 ],

[-1.4734796 , -0.86446565, -0.86091816],
[ 1.4203255 , -0.5859937 , 0.44925168],
...,
[ 1.4119536 , 0.46325207, -3.2448297 ],
[-0.12290744, 0.29577062, -0.39216644],
[-0.4729547 , 0.15344724, 1.408864 ]],

[[ 0.21240012, 0.53249377, -0.00389338],

[-0.5422931 , -0.09353203, 0.03279584],
[ 0.05919829, 1.7881966 , 0.50872993],
...,
[-0.49521384, 1.8982345 , -1.2846154 ],
[ 0.3619637 , 0.8716316 , -1.6442288 ],
[-0.85900116, -0.28337526, -0.23599708]]],

[[[ 0.827751 , -0.8917146 , -1.3633173 ],

[ 0.37120134, -0.31135547, -1.6114012 ],
[-0.3715497 , 0.12056366, 1.2951739 ],
...,
[ 0.50387484, 0.49553663, -0.9391985 ],
[ 0.16923928, 0.38702163, 0.3049653 ],
[ 1.3247155 , -0.7304066 , 0.91027665]],

[[ 0.4397138 , -0.5790663 , 0.20748606],

[-1.2422135 , -0.95965856, -0.23108931],
[-1.3649904 , -1.0418425 , 0.46117795],
...,
[-0.6164832 , -0.8543789 , -0.18572907],
[ 0.2936265 , 0.40962088, -0.41644526],
[ 0.3442094 , 0.12161341, 0.4152682 ]],

[[ 1.6018158 , 0.05495111, 0.71397877],

[-2.4949882 , 1.4684566 , -1.1969898 ],
[-0.89853895, -1.2358171 , -0.84759676],
...,
[ 2.087779 , 0.7366188 , 0.7473216 ],
[-0.3190504 , -0.03293897, -0.1575045 ],
[-0.8604673 , 0.5231678 , 0.23351063]],
...,

[[ 0.08051784, 1.6063491 , -0.01714951],

[ 0.08994591, -1.3978703 , -0.607305 ],
[-1.3601972 , -0.62083566, -0.33745226],
...,
[-0.67440724, -0.9736315 , 0.8896246 ],
[ 1.0974662 , -0.09331249, 0.41488403],
[-0.84917855, -1.0173733 , 0.6883804 ]],

[[ 0.82938236, -0.1012913 , -0.9174921 ],

[ 2.0279603 , 1.7132381 , -0.77499837],
[ 0.50818497, -1.3681097 , 1.182135 ],
...,
[ 1.1404238 , -1.9341455 , -0.33085215],
[ 1.7738689 , -0.2727936 , 0.6107232 ],
[ 0.4003075 , -0.49705872, -1.7602799 ]],

[[-1.134393 , -1.4050153 , -0.77682215],

[-1.1580638 , -0.5361828 , -0.3299927 ],
[ 2.5059564 , -2.1682148 , 0.28446195],
...,
[-1.4590594 , 0.68389434, 0.18419999],
[ 0.38754275, 0.95740974, -1.3241694 ],
[ 0.54431945, 0.23139928, 0.8231759 ]]],

[[[-0.42564863, 0.452668 , -0.29914168],

[-1.2919582 , 1.8550993 , -0.05315711],
[-1.1958379 , -1.214941 , 0.36617145],
...,
[ 0.58726525, 0.70759076, -0.46401304],
[-0.5949442 , 0.60752153, -0.95319635],
[-0.2574743 , 0.24064586, 0.34597343]],

[[-1.7168367 , -0.23556498, 2.0132248 ],

[ 1.0574174 , 0.25092965, 0.611986 ],
[-0.4694253 , -0.6934754 , 0.13635111],
...,
[ 0.03584324, -0.7154953 , 0.07054269],
[-0.7138733 , -0.31570685, -0.5378517 ],
[-1.0845814 , 0.72623473, -0.8368647 ]],

[[-0.5114272 , 0.29174542, -1.228559 ],

[ 1.4579431 , -0.8066154 , -0.02132352],
[-0.4740772 , 0.52693737, 0.23475689],
...,
[-1.017584 , -0.24304558, -1.5286679 ],
[-0.46588707, 1.6216873 , 0.45128444],
[ 0.96607655, 0.53378534, -1.0223454 ]],

...,

[[ 0.34742594, -0.44042996, -1.3424541 ],

[ 1.2630217 , -0.15709393, -1.1029688 ],
[ 0.337064 , 1.0246189 , -0.5582971 ],
...,
[-0.42786717, 1.1596836 , -0.3421803 ],
[-1.8006552 , 0.9324209 , 0.05656384],
 [-0.43976578, -0.58924365, -0.2823274 ]],

[[ 0.5410462 , 0.5804195 , 0.29743925],

[-0.5645381 , 0.29361114, 1.5866159 ],
[-0.8229178 , -0.5118068 , 0.6232647 ],
...,
[-0.8210146 , -0.5309559 , 1.7510443 ],
[-1.8660572 , 1.0673795 , -0.6420711 ],
[ 1.3523059 , -1.095501 , 0.56174904]],

[[ 1.4924296 , -0.60755205, 0.9630441 ],

[-1.5792804 , 0.33756173, 0.41448846],
[-2.6247635 , 0.54883015, 1.454688 ],
...,
[ 1.0511454 , 1.9847646 , -0.8202082 ],
[ 0.43942735, -1.8400631 , 0.2689787 ],
[ 0.24013405, -1.1960655 , -0.7239898 ]]]], dtype=float32)>

1.2.1.2.2 Indexing

The basic format of an index is a[d1][d2][d3].

In [26]:
#Obtain the pixel in the position [20,40] in the second channel of the first
image.
tensor_h[0][19][39][1]
Out[26]:
<tf.Tensor: shape=(), dtype=float32, numpy=0.32219484>
If the indexes of data to be extracted are nonconsecutive, tf.gather and tf.gather_nd are commonly used for
data extraction in TensorFlow.

To extract data from a particular dimension: tf.gather(params, indices,axis=None):

 params: input tensor

 indices: index of the data to be extracted
 axis: dimension of the data to be extracted

In [27]:
#Extract the first, second, and fourth images from tensor_h ([4,100,100,3]).
indices = [0,1,3]
tf.gather(tensor_h,axis=0,indices=indices)
Out[27]:
<tf.Tensor: shape=(3, 100, 100, 3), dtype=float32, numpy=
array([[[[-0.42564863, 0.452668 , -0.29914168],
[-1.2919582 , 1.8550993 , -0.05315711],
[-1.1958379 , -1.214941 , 0.36617145],
...,
[ 0.58726525, 0.70759076, -0.46401304],
[-0.5949442 , 0.60752153, -0.95319635],
[-0.2574743 , 0.24064586, 0.34597343]],

[[-1.7168367 , -0.23556498, 2.0132248 ],

[ 1.0574174 , 0.25092965, 0.611986 ],
[-0.4694253 , -0.6934754 , 0.13635111],
...,
[ 0.03584324, -0.7154953 , 0.07054269],
[-0.7138733 , -0.31570685, -0.5378517 ],
[-1.0845814 , 0.72623473, -0.8368647 ]],
[[-0.5114272 , 0.29174542, -1.228559 ],
[ 1.4579431 , -0.8066154 , -0.02132352],
[-0.4740772 , 0.52693737, 0.23475689],
...,
[-1.017584 , -0.24304558, -1.5286679 ],
[-0.46588707, 1.6216873 , 0.45128444],
[ 0.96607655, 0.53378534, -1.0223454 ]],

...,

[[ 0.34742594, -0.44042996, -1.3424541 ],

[ 1.2630217 , -0.15709393, -1.1029688 ],
[ 0.337064 , 1.0246189 , -0.5582971 ],
...,
[-0.42786717, 1.1596836 , -0.3421803 ],
[-1.8006552 , 0.9324209 , 0.05656384],
[-0.43976578, -0.58924365, -0.2823274 ]],

[[ 0.5410462 , 0.5804195 , 0.29743925],

[-0.5645381 , 0.29361114, 1.5866159 ],
[-0.8229178 , -0.5118068 , 0.6232647 ],
...,
[-0.8210146 , -0.5309559 , 1.7510443 ],
[-1.8660572 , 1.0673795 , -0.6420711 ],
[ 1.3523059 , -1.095501 , 0.56174904]],

[[ 1.4924296 , -0.60755205, 0.9630441 ],

[-1.5792804 , 0.33756173, 0.41448846],
[-2.6247635 , 0.54883015, 1.454688 ],
...,
[ 1.0511454 , 1.9847646 , -0.8202082 ],
[ 0.43942735, -1.8400631 , 0.2689787 ],
[ 0.24013405, -1.1960655 , -0.7239898 ]]],

[[[ 0.827751 , -0.8917146 , -1.3633173 ],

[ 0.37120134, -0.31135547, -1.6114012 ],
[-0.3715497 , 0.12056366, 1.2951739 ],
...,
[ 0.50387484, 0.49553663, -0.9391985 ],
[ 0.16923928, 0.38702163, 0.3049653 ],
[ 1.3247155 , -0.7304066 , 0.91027665]],

[[ 0.4397138 , -0.5790663 , 0.20748606],

[-1.2422135 , -0.95965856, -0.23108931],
[-1.3649904 , -1.0418425 , 0.46117795],
...,
[-0.6164832 , -0.8543789 , -0.18572907],
[ 0.2936265 , 0.40962088, -0.41644526],
[ 0.3442094 , 0.12161341, 0.4152682 ]],

[[ 1.6018158 , 0.05495111, 0.71397877],

[-2.4949882 , 1.4684566 , -1.1969898 ],
[-0.89853895, -1.2358171 , -0.84759676],
...,
[ 2.087779 , 0.7366188 , 0.7473216 ],
[-0.3190504 , -0.03293897, -0.1575045 ],
[-0.8604673 , 0.5231678 , 0.23351063]],
...,

[[ 0.08051784, 1.6063491 , -0.01714951],

[ 0.08994591, -1.3978703 , -0.607305 ],
[-1.3601972 , -0.62083566, -0.33745226],
...,
[-0.67440724, -0.9736315 , 0.8896246 ],
[ 1.0974662 , -0.09331249, 0.41488403],
[-0.84917855, -1.0173733 , 0.6883804 ]],

[[ 0.82938236, -0.1012913 , -0.9174921 ],

[ 2.0279603 , 1.7132381 , -0.77499837],
[ 0.50818497, -1.3681097 , 1.182135 ],
...,
[ 1.1404238 , -1.9341455 , -0.33085215],
[ 1.7738689 , -0.2727936 , 0.6107232 ],
[ 0.4003075 , -0.49705872, -1.7602799 ]],

[[-1.134393 , -1.4050153 , -0.77682215],

[-1.1580638 , -0.5361828 , -0.3299927 ],
[ 2.5059564 , -2.1682148 , 0.28446195],
...,
[-1.4590594 , 0.68389434, 0.18419999],
[ 0.38754275, 0.95740974, -1.3241694 ],
[ 0.54431945, 0.23139928, 0.8231759 ]]],

[[[ 1.0388501 , -0.21380688, 0.21120134],

[-0.926739 , -0.8133858 , -0.78145933],
[ 0.9289431 , 0.01986507, 0.48339394],
...,
[-0.95561856, 0.7273471 , -1.3942958 ],
[-2.0075085 , -0.769726 , -0.81108063],
[ 2.384009 , -0.3022256 , 0.27821884]],

[[ 0.16614938, 0.82138044, 1.1175478 ],

[ 0.23566669, 0.0094515 , 2.777578 ],
[-0.8147552 , 0.07749964, -2.3900774 ],
...,
[-1.0257695 , -1.1510082 , 0.08606431],
[ 1.2651023 , 0.6840444 , -0.4371489 ],
[ 1.4022624 , -0.41605678, 0.5049461 ]],

[[-0.26702768, -0.94770133, 1.3767972 ],

[ 0.13892083, 0.00883611, -1.4245614 ],
[-1.0700448 , 0.46347347, -2.1798024 ],
...,
[ 0.9927806 , -0.5713902 , -1.5768932 ],
[ 0.44241902, -2.2897117 , 0.4850132 ],
[-0.31186196, -0.46309343, 0.6004569 ]],

...,

[[-0.25572157, -0.481574 , -0.62847406],

[-0.18530175, -0.31511047, -0.17287329],
[-1.172901 , 0.43398705, 0.26872268],
...,
[-0.15042119, -0.45772648, -0.01517491],
 [-0.84364545, 0.30128956, -0.29066485],
[ 0.25015455, -0.11824514, 0.46264866]],

[[ 0.51299995, -0.45022494, 0.26325727],

[-0.58410096, -0.53932184, 0.2789639 ],
[-1.2701533 , -0.11238922, -0.40922546],
...,
[-2.1618803 , -1.7747833 , -2.5344229 ],
[ 0.16259412, 1.4623609 , -0.75771034],
[ 0.55835825, -1.0756673 , -0.15727316]],

[[ 1.1368815 , -0.29778495, 0.7555838 ],

[ 0.7729792 , -2.0782697 , -0.25594652],
[-2.5195763 , 1.17271 , -1.8258004 ],
...,
[ 1.2047101 , 0.02295183, 0.13764495],
[-0.5464439 , 0.42274612, -0.40945956],
[ 0.18586746, -0.57554436, -0.49103737]]]], dtype=float32)>
tf.gather_nd allows data extraction from multiple dimensions: tf.gather_nd(params,indices, batch_dims=0,
name=None):

 params: A Tensor. The tensor from which to gather values.

 indices: A Tensor. Must be one of the following types: int32, int64. Index tensor.
 Name: A name for the operation (optional).
 batch_dims: An integer or a scalar 'Tensor'. The number of batch dimensions.

In [28]:
# Extract the pixel in [1,1] from the first dimension of the first image
# and the pixel in [2,2] from the first dimension of the second image in
tensot_h ([4,100,100,3]).
indices = [[0,1,1,0],[1,2,2,0]]
tf.gather_nd(tensor_h,indices=indices)
Out[28]:
<tf.Tensor: shape=(2,), dtype=float32, numpy=array([ 1.0574174 , -
0.89853895], dtype=float32)>

1.2.1.3 Tensor Dimension Modification

1.2.1.3.1 Dimension Display

In [29]:
const_d_1 = tf.constant([[1, 2, 3, 4]],shape=[2,2], dtype=tf.float32)
#Three common methods for displaying a dimension:
print(const_d_1.shape)
print(const_d_1.get_shape())
#The output is a tensor. The value of the tensor indicates the size of the
tensor dimension to be displayed.
print(tf.shape(const_d_1))
(2, 2)
(2, 2)
tf.Tensor([2 2], shape=(2,), dtype=int32)
As described above, .shape and .get_shape() return TensorShape objects, and tf.shape(x) returns Tensor
objects.

1.2.1.3.2 Dimension Reshaping

tf.reshape(tensor,shape,name=None):
  tensor: input tensor
 shape: dimension of the reshaped tensor

In [30]:
reshape_1 = tf.constant([[1,2,3],[4,5,6]])
print(reshape_1)
tf.reshape(reshape_1, (3,2))
tf.Tensor(
[[1 2 3]
[4 5 6]], shape=(2, 3), dtype=int32)
Out[30]:
<tf.Tensor: shape=(3, 2), dtype=int32, numpy=
array([[1, 2],
[3, 4],
[5, 6]], dtype=int32)>

1.2.1.3.3 Dimension Expansion

tf.expand_dims(input,axis,name=None):

 input: input tensor

 axis: adds a dimension after the axis dimension. When the number of dimensions of the input data is
D, the axis must fall in the range of [–(D + 1), D] (included). A negative value indicates adding a
dimension in reverse order.

In [31]:
#Generate a 100 x 100 x 3 tensor to represent a 100 x 100 three-channel color
image.
expand_sample_1 = tf.random.normal([100,100,3], seed=1)
print("size of the original data:",expand_sample_1.shape)
print("add a dimension before the first dimension ( axis = 0 0):
",tf.expand_dims(expand_sample_1, axis=0).shape)
print("add a dimension before the second dimension ( axis = 1 1):
",tf.expand_dims(expand_sample_1, axis=1).shape)
print("add a dimension after the last dimension ( axis = – –1 1):
",tf.expand_dims(expand_sample_1, axis=-1).shape)
size of the original data: (100, 100, 3)
add a dimension before the first dimension ( axis = 0 0): (1, 100, 100, 3)
add a dimension before the second dimension ( axis = 1 1): (100, 1, 100, 3)
add a dimension after the last dimension ( axis = – –1 1): (100, 100, 3, 1)

1.2.1.3.4 Dimension Squeezing

tf.squeeze(input,axis=None,name=None):

 input: input tensor

 axis: If axis is set to 1, dimension 1 needs to be deleted.

In [32]:
#Generate a 100 x 100 x 3 tensor to represent a 100 x 100 three-channel color
image.
squeeze_sample_1 = tf.random.normal([1,100,100,3])
print("size of the original data:",squeeze_sample_1.shape)
squeezed_sample_1 = tf.squeeze(expand_sample_1)
print("data size after dimension squeezing:",squeezed_sample_1.shape)
size of the original data: (1, 100, 100, 3)
data size after dimension squeezing: (100, 100, 3)
1.2.1.3.5 Transpose

tf.transpose(a,perm=None,conjugate=False,name='transpose'):

 a: input tensor
 perm: tensor size sequence, generally used to transpose high-dimensional arrays
 conjugate: indicates complex number transpose.
 name: tensor name

In [33]:
#Input the tensor to be transposed, and call tf.transpose.
trans_sample_1 = tf.constant([1,2,3,4,5,6],shape=[2,3])
print("size of the original data:",trans_sample_1.shape)
transposed_sample_1 = tf.transpose(trans_sample_1)
print("size of transposed data:",transposed_sample_1.shape)
size of the original data: (2, 3)
size of transposed data: (3, 2)
perm is required for high-dimensional data transpose, and indicates the dimension sequence of the input tensor.

The original dimension sequence of a three-dimensional tensor is [0, 1, 2] ( perm), indicating the length, width,
and height of high-dimensional data, respectively.

Data dimensions can be transposed by changing the sequence of values in perm.

In [34]:
#Generate an $ x 100 x 200 x 3 tensor to represent four 100 x 200 three-
channel color images.
trans_sample_2 = tf.random.normal([4,100,200,3])
print("size of the original data:",trans_sample_2.shape)
#Exchange the length and width for the four images: The original perm value
is [0,1, 2,3], and the new perm value is [0,2,1,3].
transposed_sample_2 = tf.transpose(trans_sample_2,[0,2,1,3])
print("size of transposed data:",transposed_sample_2.shape)
size of the original data: (4, 100, 200, 3)
size of transposed data: (4, 200, 100, 3)

1.2.1.3.6 Broadcast (broadcast_to)

broadcast_to is used to broadcast data from a low dimension to a high dimension.

tf.broadcast_to(input,shape,name=None):

 input: input tensor

 shape: size of the output tensor

In [35]:
broadcast_sample_1 = tf.constant([1,2,3,4,5,6])
print("original data:",broadcast_sample_1.numpy())
broadcasted_sample_1 = tf.broadcast_to(broadcast_sample_1,shape=[4,6])
print("broadcasted data:",broadcasted_sample_1.numpy())
original data: [1 2 3 4 5 6]
broadcasted data: [[1 2 3 4 5 6]
[1 2 3 4 5 6]
[1 2 3 4 5 6]
[1 2 3 4 5 6]]
In [36]:
#During the operation, if two arrays have different shapes, TensorFlow
automatically triggers the broadcast mechanism as NumPy does.
a = tf.constant([[ 0, 0, 0],
[10,10,10],
[20,20,20],
[30,30,30]])
b = tf.constant([1,2,3])
print(a + b)
tf.Tensor(
[[ 1 2 3]
[11 12 13]
[21 22 23]
[31 32 33]], shape=(4, 3), dtype=int32)

1.2.1.4 Arithmetic Operations on Tensors

1.2.1.4.1 Arithmetic Operators

Main arithmetic operations include addition (tf.add), subtraction (tf.subtract), multiplication (tf.multiply),
division ( tf.divide), logarithm (tf.math.log), and powers ( tf.pow). The following describes only one addition
example.
In [37]:
a = tf.constant([[3, 5], [4, 8]])
b = tf.constant([[1, 6], [2, 9]])
print(tf.add(a, b))
tf.Tensor(
[[ 4 11]
[ 6 17]], shape=(2, 2), dtype=int32)

1.2.1.4.2 Matrix Multiplication

Matrix multiplication is implemented by calling tf.matmul.

In [38]:
tf.matmul(a,b)
Out[38]:
<tf.Tensor: shape=(2, 2), dtype=int32, numpy=
array([[13, 63],
[20, 96]], dtype=int32)>

1.2.1.4.3 Tensor Statistics Collection

Methods for collecting tensor statistics include:

 tf.reduce_min/max/mean(): calculates the minimum, maximum, and mean values.

 tf.argmax()/tf.argmin(): calculates the positions of the maximum and minimum values.
 tf.equal(): checks whether two tensors are equal by element.
 tf.unique(): removes duplicate elements from tensors.
 tf.nn.in_top_k(prediction, target, K): calculates whether the predicted value is equal to the actual
value, and returns a Boolean tensor.

The following describes how to use tf.argmax():

Return the position of the maximum value.

tf.argmax(input,axis):

 input: input tensor

 axis: maximum output value in the axis dimension
 In [39]:
argmax_sample_1 = tf.constant([[1,3,2],[2,5,8],[7,5,9]])
print("input tensor:",argmax_sample_1.numpy())
max_sample_1 = tf.argmax(argmax_sample_1, axis=0)
max_sample_2 = tf.argmax(argmax_sample_1, axis=1)
print("locate the maximum value by column:",max_sample_1.numpy())
print("locate the maximum value by row:",max_sample_2.numpy())
input tensor: [[1 3 2]
[2 5 8]
[7 5 9]]
locate the maximum value by column: [2 1 2]
locate the maximum value by row: [1 2 2]

1.2.1.5 Dimension-based Arithmetic Operations

In TensorFlow, a series of operations of tf.reduce_* reduce tensor dimensions. The series of operations can be
performed on dimensional elements of a tensor, for example, calculating the mean value by row and
calculating a product of all elements in the tensor.

Common operations include tf.reduce_sum (addition), tf.reduce_prod (multiplication), tf.reduce_min

(minimum), tf.reduce_max (maximum), tf.reduce_mean (mean value), tf.reduce_all (logical AND),
tf.reduce_any (logical OR), and tf.reduce_logsumexp ( log(sum(exp))).

The methods for using these operations are similar. The following describes how to use tf.reduce_sum.

Calculate the sum of elements in all dimensions of a tensor.

tf.reduce_sum(input_tensor, axis=None, keepdims=False,name=None):

 input_tensor: The tensor to reduce. Should have numeric type.

 axis: The dimensions to reduce. If None (the default), reduces all dimensions. Must be in the range [-
rank(input_tensor),rank(input_tensor)].
 keepdims: If true, retains reduced dimensions with length 1.
 name: A name for the operation (optional).

In [40]:
reduce_sample_1 = tf.constant([1,2,3,4,5,6],shape=[2,3])
print("original data",reduce_sample_1.numpy())
print("calculate the sum of all elements in the tensor ( axis =
None):",tf.reduce_sum(reduce_sample_1,axis=None).numpy())
print("calculate the sum of elements in each column by column ( axis = 0
0):",tf.reduce_sum(reduce_sample_1,axis=0).numpy())
print("calculate the sum of elements in each column by row ( axis = 1
1):",tf.reduce_sum(reduce_sample_1,axis=1).numpy())
original data [[1 2 3]
[4 5 6]]
calculate the sum of all elements in the tensor ( axis = None): 21
calculate the sum of elements in each column by column ( axis = 0 0): [5 7 9]
calculate the sum of elements in each column by row ( axis = 1 1): [ 6 15]

1.2.1.6 Tensor Concatenation and Splitting

1.2.1.6.1 Tensor Concatenation

In TensorFlow, tensor concatenation operations include:

  tf.contact(): concatenates vectors based on the specified dimension, while keeping other dimensions
unchanged.
 tf.stack(): changes a group of R dimensional tensors to R+1 dimensional tensors, with the dimensions
changed after the concatenation.
 tf.concat(values, axis, name='concat'):
 values: input tensor
 axis: dimension to concatenate
 name: operation name

In [41]:
concat_sample_1 = tf.random.normal([4,100,100,3])
concat_sample_2 = tf.random.normal([40,100,100,3])
print("sizes of the original
data:",concat_sample_1.shape,concat_sample_2.shape)
concated_sample_1 = tf.concat([concat_sample_1,concat_sample_2],axis=0)
print("size of the concatenated data:",concated_sample_1.shape)
sizes of the original data: (4, 100, 100, 3) (40, 100, 100, 3)
size of the concatenated data: (44, 100, 100, 3)
A dimension can be added to an original matrix in the same way. axis determines the position of the
dimension.

tf.stack(values, axis=0, name='stack'):

 values: A list of Tensor objects with the same shape and type.
 axis: An int. The axis to stack along. Defaults to the first dimension. Negative values wrap around, so
the valid range is [-(R+1), R+1).
 name: A name for this operation (optional).

In [42]:
stack_sample_1 = tf.random.normal([100,100,3])
stack_sample_2 = tf.random.normal([100,100,3])
print("sizes of the original data: ",stack_sample_1.shape,
stack_sample_2.shape)

#Dimensions increase after the concatenation. If axis is set to 0, a

dimension is added before the first dimension.
stacked_sample_1 = tf.stack([stack_sample_1, stack_sample_2],axis=0)
print("size of the concatenated data:",stacked_sample_1.shape)
sizes of the original data: (100, 100, 3) (100, 100, 3)
size of the concatenated data: (2, 100, 100, 3)

1.2.1.6.2 Tensor Splitting

In TensorFlow, tensor splitting operations include:

 tf.unstack(): splits a tensor by a specific dimension.

 tf.split(): splits a tensor into a specified number of sub tensors based on a specific dimension.
 tf.split() is more flexible than tf.unstack().
 tf.unstack(value,num=None,axis=0,name='unstack'):
 value: input tensor
 num: indicates that a list containing num elements is output. The value of num must be the same as the
number of elements in the specified dimension. This parameter can generally be ignored.
 axis: specifies the dimension based on which the tensor is split.
 name: operation name

In [43]:
#Split data based on the first dimension and output the split data in a list.
tf.unstack(stacked_sample_1,axis=0)
Out[43]:
[<tf.Tensor: shape=(100, 100, 3), dtype=float32, numpy=
array([[[-3.97027731e-01, 6.74600065e-01, -9.82945263e-01],
[ 9.37624693e-01, -9.55090046e-01, -5.13697684e-01],
[-9.67928052e-01, 1.53015286e-01, 3.64312351e-01],
...,
[-1.03265333e+00, 1.74107659e+00, -1.64016807e+00],
[-1.75055861e+00, 3.47322911e-01, 9.39206481e-01],
[-1.96195650e-03, 1.39943630e-01, 1.55450010e+00]],

[[ 1.22875023e+00, 6.00278318e-01, 1.54033911e+00],

[-2.41163418e-01, -5.37025273e-01, 1.26784933e+00],
[ 2.79889494e-01, -7.00754941e-01, -1.19651532e+00],
...,
[ 7.95198321e-01, -1.21067321e+00, -4.07342017e-01],
[-1.84756100e+00, -1.57242790e-01, 1.04297876e+00],
[-7.71217374e-03, 3.69889960e-02, -1.26842833e+00]],

[[ 5.06475747e-01, 1.86357915e-01, -2.59965825e+00],

[-1.07214510e+00, -1.99476910e+00, -4.89632376e-02],
[-8.09971988e-01, -2.89777756e-01, 1.59874094e+00],
...,
[-3.63343477e-01, -1.36213923e+00, -1.09660439e-02],
[ 5.90351760e-01, 1.23041856e+00, -2.86851466e-01],
[-7.10459054e-01, -7.78000414e-01, -1.84206879e+00]],

...,

[[ 4.12264794e-01, -4.01336998e-01, -6.39511645e-01],

[-5.61565280e-01, -1.62996594e-02, -9.60973084e-01],
[ 3.50514203e-01, 8.00278068e-01, -1.99057505e-01],
...,
[ 1.69850362e-03, -8.33531559e-01, -1.23511866e-01],
[-6.08690619e-01, -1.07997842e-01, 9.12157476e-01],
[-2.14688134e+00, 5.77987731e-01, 2.65130854e+00]],

[[ 4.74124663e-02, 5.19689143e-01, -2.63181746e-01],

[ 1.27542603e+00, 9.09324050e-01, -1.39776099e+00],
[-1.02589451e-01, 3.06445956e-01, -9.61245477e-01],
...,
[ 1.55686414e+00, 2.02488914e-01, 2.29738206e-01],
[ 1.36241272e-01, 1.34302926e+00, 7.36615658e-01],
[ 1.24884653e+00, -8.66630435e-01, -8.52142930e-01]],

[[-1.90823221e+00, 2.71346140e+00, 8.70326340e-01],

[ 8.24487209e-01, 1.49598837e+00, -1.67542481e+00],
[ 4.21411693e-01, 1.15748668e+00, -1.21164107e+00],
...,
[-9.88213420e-02, -2.13800266e-01, 1.26440251e+00],
[-1.66945672e+00, -1.21460550e-01, 1.55864549e+00],
[ 1.94681082e-02, 5.55914044e-01, 7.08899617e-01]]],
dtype=float32)>,
<tf.Tensor: shape=(100, 100, 3), dtype=float32, numpy=
array([[[ 0.30169746, -1.7319082 , -0.43603268],
[-0.5338792 , 0.8428312 , 1.0814306 ],
[ 2.1790736 , 0.30569705, 0.44539014],
...,
 [ 0.35723972, -0.0939054 , 0.49738717],
[ 0.2717328 , 0.8460396 , -1.3102132 ],
[-0.091142 , -0.26856703, 2.0321429 ]],

[[-0.32157773, -0.11931638, -0.37498775],

[-0.06142596, 1.5721012 , 0.8801709 ],
[ 0.29117894, 2.561649 , 0.306786 ],
...,
[ 0.35712755, 0.95748806, -1.0789773 ],
[ 0.9116759 , 0.953715 , 0.05896353],
[ 0.4184409 , -1.049223 , 0.47251984]],

[[ 0.29520217, -0.7085034 , 0.73690844],

[ 1.1723362 , -0.13338087, -0.00344406],
[ 0.13018037, 0.58779347, -0.5751934 ],
...,
[ 1.121774 , 2.830822 , 0.28868026],
[ 0.29149428, 0.13069268, 2.4165838 ],
[-1.5932789 , -0.48769972, 2.1506271 ]],

...,

[[-0.22314075, 0.6373088 , -0.32497165],

[ 1.3988369 , 1.8182015 , 0.15126874],
[ 0.3103367 , 0.27434397, -0.20354123],
...,
[ 0.73927456, -0.02547839, 0.50398654],
[-0.70552474, -0.30786243, 0.51335305],
[ 0.56129056, 1.1352968 , 0.24495557]],

[[ 0.2072638 , -0.7941653 , -0.38247064],

[-2.2480016 , -1.1023059 , -0.2764185 ],
[-1.8161541 , -1.5995677 , 0.13260911],
...,
[ 0.02098313, 0.4599898 , 0.30808648],
[ 0.12985219, -0.11058076, 0.01006214],
[-0.29562932, -0.34689054, -0.12698753]],

[[-1.855499 , -0.83091897, 0.6355271 ],

[-0.51226306, -1.0056994 , -0.9904588 ],
[ 1.1799202 , -0.23058845, -0.300566 ],
...,
[-0.8815187 , -0.7091681 , 0.7730639 ],
[-0.3659018 , 0.90598446, -0.7852844 ],
[ 0.934468 , 0.80055004, -1.0498375 ]]], dtype=float32)>]
tf.split(value, num_or_size_splits, axis=0, num=None, name='split'):

 value: The Tensor to split.

 num_or_size_splits: Either an integer indicating the number of splits along axis or a 1-D integer
Tensor or Python list containing the sizes of each output tensor along axis. If a scalar, then it must
evenly divide value.shape[axis]; otherwise the sum of sizes along the split axis must match that of the
value.
 axis: An integer or scalar int32 Tensor. The dimension along which to split. Must be in the range [-
rank(value), rank(value)). Defaults to 0.
 num: Optional, used to specify the number of outputs when it cannot be inferred from the shape of
size_splits.
 name: A name for the operation (optional). tf.split() splits a tensor in either of the following ways:
  If the value of num_or_size_splits is an integer, the tensor is evenly split into sub tensors in the
specified dimension (axis = D).
 If the value of num_or_size_splits is a vector, the tensor is split into sub tensors based on the element
value of the vector in the specified dimension (axis = D).

In [44]:
import numpy as np
split_sample_1 = tf.random.normal([10,100,100,3])
print("size of the original data:",split_sample_1.shape)
splited_sample_1 = tf.split(split_sample_1, num_or_size_splits=5,axis=0)
print("size of the split data when m_or_size_splits is set to 10:
",np.shape(splited_sample_1))
splited_sample_2 = tf.split(split_sample_1,
num_or_size_splits=[3,5,2],axis=0)
print("sizes of the split data when num_or_size_splits is set to [3,5,2]:",
np.shape(splited_sample_2[0]),
np.shape(splited_sample_2[1]),
np.shape(splited_sample_2[2]))
size of the original data: (10, 100, 100, 3)
size of the split data when m_or_size_splits is set to 10: (5, 2, 100, 100,
3)
sizes of the split data when num_or_size_splits is set to [3,5,2]: (3, 100,
100, 3) (5, 100, 100, 3) (2, 100, 100, 3)

1.2.1.7 Tensor Sorting

In TensorFlow, tensor sorting operations include:

 tf.sort(): sorts tensors in ascending or descending order and returns the sorted tensors.
 tf.argsort(): sorts tensors in ascending or descending order, and returns tensor indexes.
 tf.nn.top_k(): returns the first k maximum values.
 tf.sort/argsort(input, direction, axis):
 input: input tensor
 direction: sorting order, which can be set to DESCENDING (descending order) or ASCENDING
(ascending order). The default value is ASCENDING.
 axis: sorting by the dimension specified by axis. The default value of axis is –1, indicating the last
dimension.

In [45]:
sort_sample_1 = tf.random.shuffle(tf.range(10))
print("input tensor:",sort_sample_1.numpy())
sorted_sample_1 = tf.sort(sort_sample_1, direction="ASCENDING")
print("tensor sorted in ascending order:",sorted_sample_1.numpy())
sorted_sample_2 = tf.argsort(sort_sample_1,direction="ASCENDING")
print("indexes of elements in ascending order:",sorted_sample_2.numpy())
input tensor: [9 2 8 3 6 1 4 5 0 7]
tensor sorted in ascending order: [0 1 2 3 4 5 6 7 8 9]
indexes of elements in ascending order: [8 5 1 3 6 7 4 9 2 0]
tf.nn.top_k(input,k=1,sorted=True,name=None):

 input: 1-D or higher Tensor with last dimension at least k.

 K: 0-D int32 Tensor. Number of top elements to look for along the last dimension (along each row for
matrices).
 sorted: If true the resulting k elements will be sorted by the values in descending order.
 name: Optional name for the operation. Return two tensors:
 values: k maximum values in each row
 indices: positions of elements in the last dimension of the input tensor
 In [46]:
values, index = tf.nn.top_k(sort_sample_1,5)
print("input tensor:",sort_sample_1.numpy())
print("first five values in ascending order:", values.numpy())
print("indexes of the first five values in ascending order:", index.numpy())
input tensor: [9 2 8 3 6 1 4 5 0 7]
first five values in ascending order: [9 8 7 6 5]
indexes of the first five values in ascending order: [0 2 9 4 7]

1.2.2 Eager Execution of TensorFlow 2.x

Eager execution mode:

The eager execution mode of TensorFlow is a type of imperative programming, which is the same as native
Python. When you perform a particular operation, the system immediately returns a result.

Graph mode:

TensorFlow 1.0 adopts the graph mode to first build a computational graph, enable a session, and then feed
actual data to obtain a result.

In eager execution mode, code debugging is easier, but the code execution efficiency is lower.

The following implements simple multiplication by using TensorFlow to compare the differences between the
eager execution mode and the graph mode.
In [47]:
x = tf.ones((2, 2), dtype=tf.dtypes.float32)
y = tf.constant([[1, 2],
[3, 4]], dtype=tf.dtypes.float32)
z = tf.matmul(x, y)
print(z)
tf.Tensor(
[[4. 6.]
[4. 6.]], shape=(2, 2), dtype=float32)
In [48]:
#Use the syntax of TensorFlow 1.x in TensorFlow 2.x.
# You can install the v1 compatibility package in TensorFlow 2.0
# to inherit the TensorFlow 1.x code and disable the eager execution mode.

import tensorflow.compat.v1 as tf
tf.disable_eager_execution()

#Create a graph and define it as a computational graph.

a = tf.ones((2, 2), dtype=tf.dtypes.float32)
b = tf.constant([[1, 2], [3, 4]], dtype=tf.dtypes.float32)
c = tf.matmul(a, b)

#Enable the drawing function, and perform the multiplication operation to

obtain data.
with tf.Session() as sess:
print(sess.run(c))
[[4. 6.]
[4. 6.]]
Restart the kernel to restore TensorFlow 2.0 and enable the eager execution mode. Another advantage of
the eager execution mode lies in availability of native Python functions, for example, the following condition
statement:
 In [2]:
import tensorflow as tf
thre_1 = tf.random.uniform([], 0, 1)
x = tf.reshape(tf.range(0, 4), [2, 2])
print(thre_1)
if thre_1.numpy() > 0.5:
y = tf.matmul(x, x)
else:
y = tf.add(x, x)
tf.Tensor(0.07018256, shape=(), dtype=float32)
With the eager execution mode, this dynamic control flow can generate a NumPy value extractable by a tensor,
without using operators such as tf. cond and tf.while provided in graph mode.

1.2.3 AutoGraph of TensorFlow 2.x

When used to comment out a function, the decorator tf.function can be called like any other function.
tf.function will be compiled into a graph, so that it can run more efficiently on a GPU or TPU. In this case, the
function becomes an operation in TensorFlow. The function can be directly called to output a return value.
However, the function is executed in graph mode and intermediate variable values cannot be directly viewed.
In [3]:
@tf.function
def simple_nn_layer(w,x,b):
print(b)
return tf.nn.relu(tf.matmul(w, x)+b)

w = tf.random.uniform((3, 3))
x = tf.random.uniform((3, 3))
b = tf.constant(0.5, dtype='float32')
simple_nn_layer(w,x,b)
Tensor("b:0", shape=(), dtype=float32)
Out[3]:
<tf.Tensor: shape=(3, 3), dtype=float32, numpy=
array([[1.1031461 , 1.1353902 , 0.8399954 ],
[1.1968153 , 1.2687542 , 0.96577394],
[1.1476098 , 1.2371151 , 0.9468075 ]], dtype=float32)>
According to the output result, the value of b b in the function cannot be viewed directly, but the return value
can be viewed using .numpy().

The following compares the performance of the graph mode and eager execution mode by performing the
same operation.
In [6]:
import timeit
x = tf.random.uniform(shape=[20, 20], minval=-1, maxval=2,
dtype=tf.dtypes.int32)

def power(x, y):

result = tf.eye(20, dtype=tf.dtypes.int32)
for _ in range(y):
result = tf.matmul(x, result)
return result

@tf.function
def power_graph(x, y):
return power(x, y)

print("Eager execution:", timeit.timeit(lambda: power(x, 400), number=2000))

print("Graph execution:", timeit.timeit(lambda: power_graph(x, 400),
number=2000))
Eager execution: 17.365101660000164
Graph execution: 6.0053127149999455
The comparison shows that the code execution efficiency in graph mode is much higher. Therefore, the
tf.function function can be used to improve the code execution efficiency.

2 Common Modules of TensorFlow 2.x

2.1 Introduction
This section describes the common modules of TensorFlow 2.x, including:

 tf.data: implements operations on datasets.

 These operations include reading datasets directly from the memory, reading CSV files, reading
TFRecord files, and augmenting data.
 tf.image: implements processing operations on images.
 These operations include image luminance transformation, saturation transformation, image size
transformation, image rotation, and edge detection.
 tf.gfile: implements operations on files.
 These operations include reading, writing, and renaming files, and operating folders.
 tf.keras: a high-level API used to build and train deep learning models.
 tf.distributions and other modules
 This section focuses on the tf.keras module to lay a foundation for deep learning modeling.

2.2 Objectives
Upon completion of this task, you will be able to master the common deep learning modeling interfaces in
tf.keras.

2.3 Experiment Steps

2.3.1 Model Building

2.3.1.1 Stacking a Model (tf.keras.Sequential)

The most common way to build a model is to stack layers by using tf.ke
In [7]:
import tensorflow.keras.layers as layers
model = tf.keras.Sequential()
model.add(layers.Dense(32, activation='relu'))
model.add(layers.Dense(32, activation='relu'))
model.add(layers.Dense(10, activation='softmax'))

2.3.1.2 Building a Functional Model

Functional models are mainly built by using tf.keras.Input and tf.keras.Model, which are more complex than
tf.keras.Sequential but have a good effect. Variables can be input at the same time or in different phases, and
data can be output in different phases. Functional models are preferred if more than one model output is
needed.

Stacked model (.Sequential) vs. functional model (.Model):

The tf.keras.Sequential model is a simple stack of layers that cannot represent arbitrary models. You can use
the Keras functional API to build complex model topologies such as:
  Multi-input models
 Multi-output models
 Models with shared layers
 Models with non-sequential data flows (for example, residual connections)

In [8]:
#Use the output of the previous layer as the input of the next layer.
x = tf.keras.Input(shape=(32,))
h1 = layers.Dense(32, activation='relu')(x)
h2 = layers.Dense(32, activation='relu')(h1)
y = layers.Dense(10, activation='softmax')(h2)
model_sample_2 = tf.keras.models.Model(x, y)

#Print model information.

model_sample_2.summary()
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 32)] 0
_________________________________________________________________
dense_3 (Dense) (None, 32) 1056
_________________________________________________________________
dense_4 (Dense) (None, 32) 1056
_________________________________________________________________
dense_5 (Dense) (None, 10) 330
=================================================================
Total params: 2,442
Trainable params: 2,442
Non-trainable params: 0
_________________________________________________________________

2.3.1.3 Building a Network Layer (tf.keras.layers)

The tf.keras.layers module is used to configure neural network layers. Common classes include:

 tf.keras.layers.Dense: builds a fully connected layer.

 tf.keras.layers.Conv2D: builds a two-dimensional convolutional layer.
 tf.keras.layers.MaxPooling2D/AveragePooling2D: builds a maximum/average pooling layer.
 tf.keras.layers.RNN: builds a recurrent neural network layer.
 tf.keras.layers.LSTM/tf.keras.layers.LSTMCell: builds an LSTM network layer/LSTM unit.
 tf.keras.layers.GRU/tf.keras.layers.GRUCell: builds a GRU unit/GRU network layer.
 tf.keras.layers.Embedding: converts a positive integer (subscript) into a vector of a fixed size, for
example, converts [[4], [20]] into [[0.25, 0.1], [0.6, –0.2]]. The embedding layer can be used only as
the first model layer.
 tf.keras.layers.Dropout: builds the dropout layer.

The following describes tf.keras.layers.Dense, tf.keras.layers.Conv2D, tf.keras.layers.MaxPooling2D/

AveragePooling2D, and tf.keras.layers.LSTM/ tf.keras.layers.LSTMCell.

Main network configuration parameters in tf.keras.layers include:

 activation: sets the activation function for the layer. By default, the system applies no activation
function.
 kernel_initializer and bias_initializer: initialization schemes that create the layer's weights (kernel and
bias). This defaults to the Glorot uniform initializer.
  kernel_regularizer and bias_regularizer: regularization schemes that apply to the layer's weights
(kernel and bias), such as L1 or L2 regularization. By default, the system applies no regularization
function.

2.3.1.3.1 tf.keras.layers.Dense

Main configuration parameters in tf.keras.layers.Dense include:

 units: number of neurons

 activation: sets the activation function.
 use_bias: indicates whether to use bias terms. Bias terms are used by default.
 kernel_initializer: initialization scheme that creates the layer's weight (kernel)
 bias_initializer: initialization scheme that creates the layer's weight (bias)
 kernel_regularizer: regularization scheme that applies to the layer's weight (kernel)
 bias_regularizer: regularization scheme that applies to the layer's weight (bias)
 activity_regularizer: regular item applied to the output, a regularizer object
 kernel_constraint: a constraint applied to a weight
 bias_constraint: a constraint applied to a weight

In [9]:
#Create a fully connected layer that contains 32 neurons. Set the activation
function to sigmoid.
#The activation parameter can be set to a function name string,
# for example, sigmoid or a function object, for example, tf.sigmoid.
layers.Dense(32, activation='sigmoid')
layers.Dense(32, activation=tf.sigmoid)

#Set kernel_initializer.
layers.Dense(32, kernel_initializer=tf.keras.initializers.he_normal)
#Set kernel_regularizer to L2 regularization.
layers.Dense(32, kernel_regularizer=tf.keras.regularizers.l2(0.01))
Out[9]:
<tensorflow.python.keras.layers.core.Dense at 0x7fa3246e8e90>

2.3.1.3.2 tf.keras.layers.Conv2D

Main configuration parameters in tf.keras.layers.Conv2D include:

 filters: number of convolution kernels (output dimensions)

 kernel_size: width and length of a convolution kernel
 strides: convolution step
 padding: zero padding policy
 When padding is set to valid, only valid convolution is performed, that is, boundary data is not
processed. When padding is set to same, the convolution result at the boundary is reserved, and
consequently, the output shape is usually the same as the input shape.
 activation: sets the activation function.
 data_format: data format, set to channels_first or channels_last. For example, for a 128 x 128 RGB
image, data is organized as (3, 128, 128) if the value is channels_first, and (128, 128, 3) if the value is
channels_last. The default value of this parameter is the value specified in ~/.keras/keras.json. If this
parameter has never been set, the default value is channels_last.

Other parameters include use_bias, kernel_initializer, bias_initializer, kernel_regularizer, bias_regularizer,

activity_regularizer, kernel_constraints, and bias_constraints.
In [10]:
layers.Conv2D(64,[1,1],2,padding='same',activation="relu")
Out[10]:
<tensorflow.python.keras.layers.convolutional.Conv2D at 0x7fa3246ea750>
2.3.1.3.3 tf.keras.layers.MaxPool2D/AveragePool2D

Main configuration parameters in tf.keras.layers.MaxPool 2D/ AveragePool 2D include, :

 pool_size: size of the pooled kernel. For example, if the matrix (2, 2) is used, the picture becomes half
of the original length in both dimensions. If this parameter is set to an integer, the integer is the values
of all dimensions.
 strides: Integer, tuple of 2 integers, or None. Strides values. Specifies how far the pooling window
moves for each pooling step. If None, it will default to pool_size.
 padding: One of "valid" or "same" (case-insensitive). "valid" adds no zero padding. "same" adds
padding such that if the stride is 1, the output shape is the same as input shape.
 data_format: A string, one of channels_last (default) or channels_first. The ordering of the dimensions
in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while
channels_first corresponds to inputs with shape (batch, channels, height, width). It defaults to the
image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it,
then it will be "channels_last".

In [11]:
layers.MaxPool2D(pool_size=(2,2),strides=(2,1))
Out[11]:
<tensorflow.python.keras.layers.pooling.MaxPooling2D at 0x7fa324533510>

2.3.1.3.4 tf.keras.layers.LSTM/tf.keras.layers.LSTMCell

Main configuration parameters in tf.keras.layers.LSTM/ tf.keras.layers .LSTMCell include:

 units: output dimension

 activation: sets the activation function.
 recurrent_activation: activation function to use for the recurrent step
 return_sequences: If the value is True, the system returns the full sequence. If the value is False, the
system returns the output in the last cell of the output sequence.
 return_state: Boolean value, indicating whether to return the last state in addition to the output.
 dropout: float between 0 and 1, fraction of the neurons to drop for the linear transformation of the
inputs.
 recurrent_dropout: float between 0 and 1, fraction of the neurons to drop for the linear transformation
of the recurrent state.

In [12]:
import numpy as np

inputs = tf.keras.Input(shape=(3, 1))

lstm = layers.LSTM(1, return_sequences=True)(inputs)
model_lstm_1 = tf.keras.models.Model(inputs=inputs, outputs=lstm)

inputs = tf.keras.Input(shape=(3, 1))

lstm = layers.LSTM(1, return_sequences=False)(inputs)
model_lstm_2 = tf.keras.models.Model(inputs=inputs, outputs=lstm)
#Sequences t1, t2, and t3
data = [[[0.1],[0.2],[0.3]]]
print(data)
print("output when return_sequences is set to
True",model_lstm_1.predict(data))
print("output when return_sequences is set to
False",model_lstm_2.predict(data))
[[[0.1], [0.2], [0.3]]]
output when return_sequences is set to True [[[-0.00885137]
[-0.02279976]
[-0.04005386]]]
output when return_sequences is set to False [[-0.0206602]]
LSTMcell is the implementation unit of the LSTM layer.

 LSTM is an LSTM network layer.

 LSTMcell is a single-step computing unit, that is, an LSTM unit.

In [13]:
#LSTM
tf.keras.layers.LSTM(16, return_sequences=True)
#LSTMCell
x = tf.keras.Input((None, 3))
y = layers.RNN(layers.LSTMCell(16))(x)
model_lstm_3= tf.keras.Model(x, y)

2.3.2 Training and Evaluation

2.3.2.1 Model Compilation

After a model is built, you can call compile to configure the learning process of the
model: compile(optimizer='rmsprop', loss=None, metrics=None, loss_weights=None, weighted_metrics=None,
run_eagerly=None, \*\*kwargs):

 optimizer: String (name of optimizer) or optimizer instance.

 loss: loss function, cross entropy for binary tasks and MSE for regression tasks
 metrics: model evaluation criteria during training and testing For example, metrics can be set to
['accuracy']. To specify multiple evaluation criteria, set a dictionary, for example, set metrics to
{'output_a':'accuracy'}.
 loss_weights: If the model has multiple task outputs, you need to specify a weight for each output
when optimizing the global loss.
 weighted_metrics: List of metrics to be evaluated and weighted by sample_weight or class_weight
during training and testing.
 run_eagerly: Bool. Defaults to False. If True, this Model's logic will not be wrapped in a tf.function.
Recommended to leave this as None unless your Model cannot be run inside a tf.function.
 **kwargs: Any additional arguments. Supported arguments: experimental_steps_per_execution: Int.
The number of batches to run during each tf.function call. Running multiple batches inside a single
tf.function call can greatly improve performance on TPUs or small models with a large Python
overhead. Note that if this value is set to N, Callback.on_batch methods will only be called every N
batches. This currently defaults to 1. At most, one full epoch will be run each execution. If a number
larger than the size of the epoch is passed, the execution will be truncated to the size of the epoch.

sample_weight_mode for backward compatibility.

In [14]:
model = tf.keras.Sequential()
model.add(layers.Dense(10, activation='softmax'))

#Determine the optimizer ( optimizer), loss function ( loss), and model

evaluation method ( metrics).
model.compile(optimizer=tf.keras.optimizers.Adam(0.001),
loss=tf.keras.losses.categorical_crossentropy,
metrics=[tf.keras.metrics.categorical_accuracy])

2.3.2.2 Model Training

fit(x=None, y=None, batch_size=None, epochs=1, verbose=1, callbacks=None, validation_split=0.0,

validation_data=None, shuffle=True, class_weight=None, sample_weight=None, initial_epoch=0,
steps_per_epoch=None, validation_steps=None, validation_batch_size=None, validation_freq=1,
max_queue_size=10, workers=1, use_multiprocessing=False):
  x: input training data
 y: target (labeled) data
 batch_size: number of samples for each gradient update The default value is 32.
 epochs: number of iteration rounds of the training model
 verbose: log display mode, set to 0, 1, or 2.  0: no display  1: progress bar  2: one line for each
round
 callbacks: callback function used during training
 validation_split: fraction of the training data to be used as validation data
 validation_data: validation set. This parameter will overwrite validation_split.
 shuffle: indicates whether to shuffle data before each round of iteration. This parameter is invalid
when steps_per_epoch is not None.
 initial_epoch: epoch at which to start training (useful for resuming a previous training weight)
 steps_per_epoch: set to the dataset size or batch_size
 validation_steps: Total number of steps (batches of samples) to validate before stopping. This
parameter is valid only when steps_per_epoch is specified.
 validation_batch_size: Integer or None. Number of samples per validation batch
 validation_freq: Only relevant if validation data is provided. Integer or collections_abc.
 max_queue_size Integer. Used for generator or keras.utils.Sequence input only. Maximum size for the
generator queue. If unspecified, max_queue_size will default to 10.
 workers: Integer. Used for generator or keras.utils.Sequence input only. Maximum number of
processes to spin up when using process-based threading. If unspecified, workers will default to 1. If
0, will execute the generator on the main thread.
 use_multiprocessing: Boolean. Used for generator or keras.utils.Sequence input only. If True, use
process-based threading. If unspecified, use_multiprocessing will default to False. Note that because
this implementation relies on multiprocessing, you should not pass non-picklable arguments to the
generator as they can't be passed easily to children processes.

In [15]:
import numpy as np
train_x = np.random.random((1000, 36))
train_y = np.random.random((1000, 10))
val_x = np.random.random((200, 36))
val_y = np.random.random((200, 10))
model.fit(train_x, train_y, epochs=10, batch_size=100,
validation_data=(val_x, val_y))
Epoch 1/10
10/10 [==============================] - 1s 49ms/step - loss: 12.3802 -
categorical_accuracy: 0.0844 - val_loss: 12.3513 - val_categorical_accuracy:
0.0950
Epoch 2/10
10/10 [==============================] - 0s 15ms/step - loss: 12.4663 -
categorical_accuracy: 0.0871 - val_loss: 12.3515 - val_categorical_accuracy:
0.0950
Epoch 3/10
10/10 [==============================] - 0s 13ms/step - loss: 12.3072 -
categorical_accuracy: 0.1013 - val_loss: 12.3511 - val_categorical_accuracy:
0.0950
Epoch 4/10
10/10 [==============================] - 0s 13ms/step - loss: 12.2677 -
categorical_accuracy: 0.0932 - val_loss: 12.3513 - val_categorical_accuracy:
0.0950
Epoch 5/10
10/10 [==============================] - 0s 12ms/step - loss: 12.3376 -
categorical_accuracy: 0.1059 - val_loss: 12.3519 - val_categorical_accuracy:
0.0950
Epoch 6/10
10/10 [==============================] - 0s 14ms/step - loss: 12.3756 -
categorical_accuracy: 0.1003 - val_loss: 12.3512 - val_categorical_accuracy:
0.0950
Epoch 7/10
10/10 [==============================] - 0s 13ms/step - loss: 12.3867 -
categorical_accuracy: 0.0877 - val_loss: 12.3505 - val_categorical_accuracy:
0.0950
Epoch 8/10
10/10 [==============================] - 0s 32ms/step - loss: 12.3140 -
categorical_accuracy: 0.0971 - val_loss: 12.3501 - val_categorical_accuracy:
0.0950
Epoch 9/10
10/10 [==============================] - 0s 9ms/step - loss: 12.3372 -
categorical_accuracy: 0.1055 - val_loss: 12.3502 - val_categorical_accuracy:
0.0950
Epoch 10/10
10/10 [==============================] - 0s 12ms/step - loss: 12.3934 -
categorical_accuracy: 0.0943 - val_loss: 12.3496 - val_categorical_accuracy:
0.0950
Out[15]:
<tensorflow.python.keras.callbacks.History at 0x7fa31c7605d0>
You can use tf.data to build training input pipelines for large datasets.
In [16]:
dataset = tf.data.Dataset.from_tensor_slices((train_x, train_y))
dataset = dataset.batch(32)
dataset = dataset.repeat()
val_dataset = tf.data.Dataset.from_tensor_slices((val_x, val_y))
val_dataset = val_dataset.batch(32)
val_dataset = val_dataset.repeat()
model.fit(dataset, epochs=10, steps_per_epoch=30,
validation_data=val_dataset, validation_steps=3)
Epoch 1/10
30/30 [==============================] - 1s 12ms/step - loss: 12.3317 -
categorical_accuracy: 0.0979 - val_loss: 12.2354 - val_categorical_accuracy:
0.1042
Epoch 2/10
30/30 [==============================] - 0s 4ms/step - loss: 12.3113 -
categorical_accuracy: 0.1004 - val_loss: 12.2317 - val_categorical_accuracy:
0.1042
Epoch 3/10
30/30 [==============================] - 0s 4ms/step - loss: 12.3155 -
categorical_accuracy: 0.0962 - val_loss: 12.2275 - val_categorical_accuracy:
0.1042
Epoch 4/10
30/30 [==============================] - 0s 3ms/step - loss: 12.3378 -
categorical_accuracy: 0.0929 - val_loss: 12.2230 - val_categorical_accuracy:
0.1042
Epoch 5/10
30/30 [==============================] - 0s 4ms/step - loss: 12.2941 -
categorical_accuracy: 0.0994 - val_loss: 12.2191 - val_categorical_accuracy:
0.1042
Epoch 6/10
30/30 [==============================] - 0s 3ms/step - loss: 12.2916 -
categorical_accuracy: 0.0929 - val_loss: 12.2161 - val_categorical_accuracy:
0.1042
Epoch 7/10
30/30 [==============================] - 0s 4ms/step - loss: 12.3166 -
categorical_accuracy: 0.0972 - val_loss: 12.2136 - val_categorical_accuracy:
0.1042
Epoch 8/10
30/30 [==============================] - 0s 3ms/step - loss: 12.3050 -
categorical_accuracy: 0.0897 - val_loss: 12.2115 - val_categorical_accuracy:
0.1042
Epoch 9/10
30/30 [==============================] - 0s 4ms/step - loss: 12.2984 -
categorical_accuracy: 0.0929 - val_loss: 12.2101 - val_categorical_accuracy:
0.1042
Epoch 10/10
30/30 [==============================] - 0s 4ms/step - loss: 12.3012 -
categorical_accuracy: 0.0983 - val_loss: 12.2088 - val_categorical_accuracy:
0.1042
Out[16]:
<tensorflow.python.keras.callbacks.History at 0x7fa31c77f650>

2.3.2.3 Callback Functions

A callback function is an object passed to the model to customize and extend the model's behavior during
training. You can customize callback functions or use embedded functions in tf.keras.callbacks. Common
embedded callback functions include:

 tf.keras.callbacks.ModelCheckpoint: periodically saves models.

 tf.keras.callbacks.LearningRateScheduler: dynamically changes the learning rate.
 tf.keras.callbacks.EarlyStopping: stops the training in advance.
 tf.keras.callbacks.TensorBoard: exports and visualizes the training progress and results with
TensorBoard.

In [17]:
#Set hyperparameters.
Epochs = 10

#Define a function for dynamically setting the learning rate.

def lr_Scheduler(epoch):
if epoch > 0.9 * Epochs:
lr = 0.0001
elif epoch > 0.5 * Epochs:
lr = 0.001
elif epoch > 0.25 * Epochs:
lr = 0.01
else:
lr = 0.1

print(lr)
return lr

callbacks = [
#Early stopping:
tf.keras.callbacks.EarlyStopping(
#Metric for determining whether the model performance has no further
improvement
monitor='val_loss',
#Threshold for determining whether the model performance has no
further improvement
min_delta=1e-2,
 #Number of epochs in which the model performance has no further
improvement
patience=2),

#Periodically save models.

tf.keras.callbacks.ModelCheckpoint(
#Model path
filepath='testmodel_{epoch}.h5',
#Whether to save the optimal model.
save_best_only=True,
#Monitored metric
monitor='val_loss'),

#Dynamically change the learning rate.

tf.keras.callbacks.LearningRateScheduler(lr_Scheduler),
#Use TensorBoard.
tf.keras.callbacks.TensorBoard(log_dir='./logs')
]

model.fit(train_x, train_y, batch_size=16, epochs=Epochs,

callbacks=callbacks, validation_data=(val_x, val_y))
Epoch 1/10
0.1
63/63 [==============================] - 1s 9ms/step - loss: 12.6148 -
categorical_accuracy: 0.0930 - val_loss: 13.0896 - val_categorical_accuracy:
0.1150
Epoch 2/10
0.1
63/63 [==============================] - 0s 4ms/step - loss: 12.7532 -
categorical_accuracy: 0.0930 - val_loss: 12.4826 - val_categorical_accuracy:
0.0600
Epoch 3/10
0.1
63/63 [==============================] - 0s 4ms/step - loss: 12.4687 -
categorical_accuracy: 0.0770 - val_loss: 12.8848 - val_categorical_accuracy:
0.0750
Epoch 4/10
0.01
63/63 [==============================] - 0s 4ms/step - loss: 12.4746 -
categorical_accuracy: 0.0980 - val_loss: 12.2670 - val_categorical_accuracy:
0.1100
Epoch 5/10
0.01
63/63 [==============================] - 0s 4ms/step - loss: 12.3216 -
categorical_accuracy: 0.0870 - val_loss: 12.3534 - val_categorical_accuracy:
0.1050
Epoch 6/10
0.01
63/63 [==============================] - 0s 4ms/step - loss: 12.3441 -
categorical_accuracy: 0.0910 - val_loss: 12.2818 - val_categorical_accuracy:
0.1050
Out[17]:
<tensorflow.python.keras.callbacks.History at 0x7fa324465050>

2.3.2.4 Evaluation and Prediction

Evaluation and prediction functions: tf.keras.Model.evaluate and tf.keras.Model.predict.

In [18]:
#Model evaluation
test_x = tf.random.normal((1000, 36))
test_y = tf.random.normal((1000, 10))
model.evaluate(test_x, test_y, batch_size=32)
32/32 [==============================] - 0s 2ms/step - loss: -0.1242 -
categorical_accuracy: 0.0880
Out[18]:
[-0.12419211119413376, 0.08799999952316284]
In [19]:
#Model prediction
pre_x = tf.random.normal((10, 36))
result = model.predict(test_x,)
print(result)
[[0.00694199 0.07880092 0.17122707 ... 0.02300966 0.06266186 0.08449849]
[0.00089358 0.13905603 0.00954125 ... 0.05393827 0.09236031 0.4135703 ]
[0.09597747 0.19185147 0.00829463 ... 0.35801575 0.01247225 0.00903553]
...
[0.0584045 0.05342678 0.32602516 ... 0.01447296 0.32406852 0.10163637]
[0.11030743 0.00930175 0.11492845 ... 0.09250981 0.08169756 0.22693962]
[0.05843578 0.36183682 0.05129315 ... 0.05398244 0.08730852 0.01162697]]

2.3.3 Model Saving and Restoration

2.3.3.1 Saving and Restoring an Entire Model

In [20]:
import numpy as np
import os
# create the file
if not os.path.exists('./model/'):
os.mkdir('./model/')
#Save models.
model.save('./model/the_save_model.h5')
#Import models.
new_model = tf.keras.models.load_model('./model/the_save_model.h5')
new_prediction = new_model.predict(test_x)
#np.testing.assert_allclose: determines whether the similarity between two
objects exceeds the specified tolerance.
#If yes, the system displays an exception.
#atol: specified tolerance
np.testing.assert_allclose(result, new_prediction, atol=1e-6) # Prediction
results are the same.
After a model is saved, you can find the corresponding weight file in the corresponding folder.

2.3.3.2 Saving and Loading Network Weights Only

If the weight name is suffixed with .h5 or .keras, save the weight as an HDF5 file, or otherwise, save the
weight as a TensorFlow checkpoint file by default.
In [21]:
model.save_weights('./model/model_weights')
model.save_weights('./model/model_weights.h5')
#Load the weights.
model.load_weights('./model/model_weights')
model.load_weights('./model/model_weights.h5')
3 Handwritten Digit Recognition with TensorFlow
3.1 Introduction
Handwritten digit recognition is a common image recognition task where computers recognize text in
handwriting images. Different from printed fonts, handwriting of different people has different sizes and
styles, making it difficult for computers to recognize handwriting. This chapter describes the basic process of
TensorFlow computing and basic elements for building a network.

3.2 Objectives
Upon completion of this task, you will be able to: • Master the basic process of TensorFlow computing. • Be
familiar with the basic elements of network building, including dataset, network model building, model
training, and model validation.

3.3 Experiment Steps

This experiment involves the following steps: • Reading the MNIST handwritten digit dataset. • Getting started
with TensorFlow by using simple mathematical models. • Implementing softmax regression by using high-
level APIs. • Building a multi-layer CNN. • Implementing a CNN by using high-level APIs. • Visualizing
prediction results.

3.3.1 Project Description and Dataset Acquisition

3.3.1.1 Description

This project applies deep learning and TensorFlow tools to train and build models based on the MNIST
handwriting dataset.

3.3.1.2 Data Acquisition and Processing

3.3.1.2.1 About the Dataset

The MNIST dataset is provided by the National Institute of Standards and Technology (NIST). The dataset
consists of handwritten digits from 250 individuals, of which 50% are high school students and 50% are staff
from Bureau of the Census. You can download the dataset from http://yann.lecun.com/exdb/mnist/, which
consists of the following parts:

- Training set images: train-images-idx3-ubyte.gz (9.9 MB, 47 MB after decompression, including

60,000 samples)

- Training set labels: train-labels-idx1-ubyte.gz (29 KB, 60 KB after decompression, including 60,000
labels)

- Test set images: t10k-images-idx3-ubyte.gz (1.6 MB, 7.8 MB after decompression, including 10,000
samples)

- Test set labels: t10k-labels-idx1-ubyte.gz (5 KB, 10 KB after decompression, including 10,000 labels)

The MNIST dataset is an entry-level computer vision dataset that contains images of various handwritten
digits.
It also contains one label for each image, to clarify the correct digit. For example, the labels for the preceding
four images are 5, 0, 4, and 1.

3.3.1.2.2 MNIST Dataset Reading

Download the MNIST dataset directly from the official TensorFlow website and decompress it.
In [22]:
import os
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, optimizers, datasets
from matplotlib import pyplot as plt
import numpy as np

(x_train_raw, y_train_raw), (x_test_raw, y_test_raw) =

datasets.mnist.load_data()

print(x_train_raw.shape, y_train_raw.shape)
print(x_test_raw.shape, y_test_raw.shape)
print(y_train_raw[0])

#Convert the labels into one-hot codes.

num_classes = 10
y_train = keras.utils.to_categorical(y_train_raw, num_classes)
y_test = keras.utils.to_categorical(y_test_raw, num_classes)

print(y_train[0])
(60000, 28, 28) (60000,)
(10000, 28, 28) (10000,)
5
[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]
In the MNIST dataset, the images are a tensor in the shape of [60000, 28, 28]. The first dimension is used to
extract images, and the second and third dimensions are used to extract pixels in each image. Each element in
this tensor indicates the strength of a pixel in an image. The value ranges from 0 to 255. Label data is
converted from scalar to one-hot vectors. In a one-hot vector, one digit is 1, and digits in other dimensions are
all 0s. For example, label 1 may be represented as [0,1,0,0,0,0,0,0,0,0,0,0]. Therefore, the labels are a digital
matrix of [60000, 10].

3.3.2 Dataset Preprocessing and Visualization

3.3.2.1 Data Visualization

Draw the first 9 images.

In [23]:
plt.figure()
for i in range(10):
plt.subplot(2,5,i+1)
plt.imshow(x_train_raw[i])
plt.axis('off')
plt.show()

3.3.2.2 Data Preprocessing

An output of a fully connected network must be in the form of vector, instead of the matrix form of the current
images. Therefore, you need to sort the images into vectors.
 In [24]:
#Convert a 28 x 28 image into a 784 x 1 vector.
x_train = x_train_raw.reshape(60000, 784)
x_test = x_test_raw.reshape(10000, 784)
Currently, the dynamic range of pixels is 0 to 255. Image pixels are usually normalized to the range of 0 to 1
during processing of image pixel values.
In [25]:
#Normalize image pixel values.
x_train = x_train.astype('float32')/255
x_test = x_test.astype('float32')/255

3.3.3 DNN Model

3.3.3.1 Building a DNN Model

In [26]:
#Create a deep neural network (DNN) model that consists of three fully
connected layers and two RELU activation functions.
model = keras.Sequential([
layers.Dense(512, activation='relu', input_dim = 784),
layers.Dense(256, activation='relu'),
layers.Dense(124, activation='relu'),
layers.Dense(num_classes, activation='softmax')])

model.summary()
Model: "sequential_2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_11 (Dense) (None, 512) 401920
_________________________________________________________________
dense_12 (Dense) (None, 256) 131328
_________________________________________________________________
dense_13 (Dense) (None, 124) 31868
_________________________________________________________________
dense_14 (Dense) (None, 10) 1250
=================================================================
Total params: 566,366
Trainable params: 566,366
Non-trainable params: 0
_________________________________________________________________
layer.Dense() indicates a fully connected layer, and activation indicates a used activation function.

3.3.3.2 Compiling the DNN Model

In [27]:
Optimizer = optimizers.Adam(0.001)
model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=Optimizer,
metrics=['accuracy'])
In the preceding example, the loss function of the model is cross entropy, and the optimization algorithm is the
Adam gradient descent method.

3.3.3.3 Training the DNN Model

In [28]:
#Fit the training data to the model by using the fit method.
model.fit(x_train, y_train,
batch_size=128,
epochs=10,
verbose=1)
Epoch 1/10
469/469 [==============================] - 11s 22ms/step - loss: 0.4302 -
accuracy: 0.8708
Epoch 2/10
469/469 [==============================] - 11s 24ms/step - loss: 0.0864 -
accuracy: 0.9735
Epoch 3/10
469/469 [==============================] - 11s 23ms/step - loss: 0.0510 -
accuracy: 0.9842
Epoch 4/10
469/469 [==============================] - 10s 22ms/step - loss: 0.0388 -
accuracy: 0.9879
Epoch 5/10
469/469 [==============================] - 10s 22ms/step - loss: 0.0262 -
accuracy: 0.9914
Epoch 6/10
469/469 [==============================] - 10s 21ms/step - loss: 0.0256 -
accuracy: 0.9913
Epoch 7/10
469/469 [==============================] - 11s 24ms/step - loss: 0.0251 -
accuracy: 0.9916
Epoch 8/10
469/469 [==============================] - 11s 24ms/step - loss: 0.0153 -
accuracy: 0.9952
Epoch 9/10
469/469 [==============================] - 11s 23ms/step - loss: 0.0154 -
accuracy: 0.9944
Epoch 10/10
469/469 [==============================] - 12s 25ms/step - loss: 0.0145 -
accuracy: 0.9951
Out[28]:
<tensorflow.python.keras.callbacks.History at 0x7fa3244831d0>
Epoch indicates a specific round of training. In the preceding example, full data is iterated for 10 times.

3.3.3.4 Evaluating the DNN Model

In [30]:
score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Test loss: 0.09210538119077682
Test accuracy: 0.9800000190734863
The evaluation shows that the model accuracy reaches 0.87, and 10 training iterations have been performed.

3.3.3.5 Saving the DNN Model

Create model folder under relative path.

In [31]:
model.save('./final_DNN_model.h5')

3.3.4 CNN Model

The conventional CNN construction method helps you better understand the internal network structure but has
a large code volume. Therefore, attempts to construct a CNN by using high-level APIs are made to simplify
the network construction process.

3.3.4.1 Building a CNN Model

In [32]:
import tensorflow as tf
from tensorflow import keras
import numpy as np

model=keras.Sequential() #Create a network sequence.

##Add the first convolutional layer and pooling layer.
model.add(keras.layers.Conv2D(filters=32,kernel_size = 5,strides = (1,1),
padding = 'same',activation =
tf.nn.relu,input_shape = (28,28,1)))
model.add(keras.layers.MaxPool2D(pool_size=(2,2), strides = (2,2), padding =
'valid'))
##Add the second convolutional layer and pooling layer.
model.add(keras.layers.Conv2D(filters=64,kernel_size = 3,strides =
(1,1),padding = 'same',activation = tf.nn.relu))
model.add(keras.layers.MaxPool2D(pool_size=(2,2), strides = (2,2), padding =
'valid'))
##Add a dropout layer to reduce overfitting.
model.add(keras.layers.Dropout(0.25))
model.add(keras.layers.Flatten())
##Add two fully connected layers.
model.add(keras.layers.Dense(units=128,activation = tf.nn.relu))
model.add(keras.layers.Dropout(0.5))
model.add(keras.layers.Dense(units=10,activation = tf.nn.softmax))
In [33]:
model.summary()
Model: "sequential_3"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_1 (Conv2D) (None, 28, 28, 32) 832
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 14, 14, 32) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 14, 14, 64) 18496
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 7, 7, 64) 0
_________________________________________________________________
dropout (Dropout) (None, 7, 7, 64) 0
_________________________________________________________________
flatten (Flatten) (None, 3136) 0
_________________________________________________________________
dense_15 (Dense) (None, 128) 401536
_________________________________________________________________
dropout_1 (Dropout) (None, 128) 0
_________________________________________________________________
dense_16 (Dense) (None, 10) 1290
=================================================================
Total params: 422,154
Trainable params: 422,154
Non-trainable params: 0
_________________________________________________________________
In the preceding network, two convolutional layers and two pooling layers are first added by using
keras.layers. Afterwards, a dropout layer is added to prevent overfitting. Finally, two fully connected layers are
added.

3.3.4.2 Compiling and Training the CNN Model

In [35]:
#Expand data dimensions to adapt to the CNN model.
X_train=x_train.reshape(60000,28,28,1)
X_test=x_test.reshape(10000,28,28,1)
model.compile(optimizer="adam",loss="categorical_crossentropy",metrics=['accu
racy'])
model.fit(x=X_train,y=y_train,epochs=5,batch_size=128)
Epoch 1/5
469/469 [==============================] - 125s 265ms/step - loss: 0.0977 -
accuracy: 0.9693
Epoch 2/5
469/469 [==============================] - 132s 281ms/step - loss: 0.0664 -
accuracy: 0.9804
Epoch 3/5
469/469 [==============================] - 128s 273ms/step - loss: 0.0550 -
accuracy: 0.9831
Epoch 4/5
469/469 [==============================] - 126s 268ms/step - loss: 0.0442 -
accuracy: 0.9867
Epoch 5/5
469/469 [==============================] - 134s 287ms/step - loss: 0.0412 -
accuracy: 0.9875
Out[35]:
<tensorflow.python.keras.callbacks.History at 0x7fa3082a8610>
During training, the network training data is iterated for only five times. You can increase the number of
network iterations to check the effect.

3.3.4.3 Evaluating the CNN Model

In [36]:
test_loss,test_acc=model.evaluate(x=X_test,y=y_test)
print("Test Accuracy %.2f"%test_acc)
313/313 [==============================] - 9s 27ms/step - loss: 0.0202 -
accuracy: 0.9926
Test Accuracy 0.99
The verification shows that accuracy of the CNN model reaches up to 99%.

3.3.4.4 Saving the CNN Model

In [37]:
model.save('./final_CNN_model.h5')

3.3.5 Prediction Result Visualization

3.3.5.1 Loading the CNN Model

In [38]:
from tensorflow.keras.models import load_model
new_model = load_model('./final_CNN_model.h5')
new_model.summary()
Model: "sequential_3"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_1 (Conv2D) (None, 28, 28, 32) 832
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 14, 14, 32) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 14, 14, 64) 18496
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 7, 7, 64) 0
_________________________________________________________________
dropout (Dropout) (None, 7, 7, 64) 0
_________________________________________________________________
flatten (Flatten) (None, 3136) 0
_________________________________________________________________
dense_15 (Dense) (None, 128) 401536
_________________________________________________________________
dropout_1 (Dropout) (None, 128) 0
_________________________________________________________________
dense_16 (Dense) (None, 10) 1290
=================================================================
Total params: 422,154
Trainable params: 422,154
Non-trainable params: 0
_________________________________________________________________
Visualize prediction results.
In [39]:
#Visualize test set output results.
import matplotlib.pyplot as plt
%matplotlib inline
def res_Visual(n):
final_opt_a=np.argmax(new_model.predict(X_test[0:n]), axis=-1)#Perform
predictions on the test set by using the model.
fig, ax = plt.subplots(nrows=int(n/5),ncols=5 )
ax = ax.flatten()
print('prediction results of the first {} images:'.format(n))
for i in range(n):
print(final_opt_a[i],end=',')
if int((i+1)%5) ==0:
print('\t')
#Visualize image display.
img = X_test[i].reshape((28,28))#Read each row of data in the format
of Ndarry.
plt.axis("off")
ax[i].imshow(img, cmap='Greys',
interpolation='nearest')#Visualization
ax[i].axis("off")
print('first {} images in the test set:'.format(n))
res_Visual(20)
prediction results of the first 20 images:
7,2,1,0,4,
1,4,9,5,9,
0,6,9,0,1,
5,9,7,3,4,
first 20 images in the test set:
4 Image Classification
4.1 Introduction

4.1.1 About This Experiment

This experiment is about a basic task in computer vision, that is, image recognition. The numay and
TensorFlow frameworks are required. The NumPy arrays are used as the image objects. The TensorFlow
framework is mainly used to create deep learning algorithms and build a convolutional neural network (CNN).
This experiment recognizes image categories based on the CIFAR10 dataset.

4.1.2 Objectives

 Upon completion of this task, you will be able to:

 Strengthen the understanding of Keras-based neural network model construction process
 Master the method to load the pre-trained model.
 Learn to use checkpoint function.
 Master how to use a trained model to make predictions.

4.2 Experiment Code

4.2.1 Introducing Dependencies

In [40]:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, optimizers, datasets, Sequential
from tensorflow.keras.layers import
Conv2D,Activation,MaxPooling2D,Dropout,Flatten,Dense
import os
import numpy as np
import matplotlib.pyplot as plt

4.2.2 Data Preprocessing

In [41]:
#download Cifar-10 dataset
(x_train,y_train), (x_test, y_test) = datasets.cifar10.load_data()
#print the size of the dataset
print(x_train.shape, y_train.shape, x_test.shape, y_test.shape)
print(y_train[0])
(50000, 32, 32, 3) (50000, 1) (10000, 32, 32, 3) (10000, 1)
[6]
In [42]:
#Convert the category label into onehot encoding
num_classes = 10
y_train_onehot = keras.utils.to_categorical(y_train, num_classes)
y_test_onehot = keras.utils.to_categorical(y_test, num_classes)
y_train_onehot[0]
Out[42]:
array([0., 0., 0., 0., 0., 0., 1., 0., 0., 0.], dtype=float32)
Show the first 9 images
In [43]:
#Create a image tag list
category_dict =
{0:'airplane',1:'automobile',2:'bird',3:'cat',4:'deer',5:'dog',
6:'frog',7:'horse',8:'ship',9:'truck'}
#Show the first 9 images and their labels
plt.figure()
for i in range(9):
#create a figure with 9 subplots
plt.subplot(3,3,i+1)
#show an image
plt.imshow(x_train[i])
#show the label
plt.ylabel(category_dict[y_train[i][0]])
plt.show()

In [44]:
#Pixel normalization
x_train = x_train.astype('float32')/255
x_test = x_test.astype('float32')/255

4.2.3 Model Creation

In [45]:
def CNN_classification_model(input_size = x_train.shape[1:]):
model = Sequential()
#the first block with 2 convolutional layers and 1 maxpooling layer
'''Conv1 with 32 3*3 kernels
padding="same": it applies zero padding to the input image so that
the input image gets fully covered by the filter and specified stride.
It is called SAME because, for stride 1 , the output will be the same
as the input.
output: 32*32*32'''
model.add(Conv2D(32, (3, 3), padding='same',
input_shape=input_size))
#relu activation function
model.add(Activation('relu'))
#Conv2
model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
#maxpooling
model.add(MaxPooling2D(pool_size=(2, 2),strides =1))

#the second block

model.add(Conv2D(64, (3, 3), padding='same'))
model.add(Activation('relu'))
model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
#maxpooling.the default strides =1
model.add(MaxPooling2D(pool_size=(2, 2)))

#Before sending a feature map into a fully connected network, it should

be flattened into a column vector.
model.add(Flatten())
#fully connected layer
model.add(Dense(128))
model.add(Activation('relu'))
#dropout layer.every neuronis set to 0 with a probability of 0.25
model.add(Dropout(0.25))
 model.add(Dense(num_classes))
#map the score of each class into probability
model.add(Activation('softmax'))

opt = keras.optimizers.Adam(lr=0.0001)

model.compile(loss='sparse_categorical_crossentropy', optimizer=opt,
metrics=['accuracy'])
return model
model=CNN_classification_model()
model.summary()
Model: "sequential_4"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_3 (Conv2D) (None, 32, 32, 32) 896
_________________________________________________________________
activation (Activation) (None, 32, 32, 32) 0
_________________________________________________________________
conv2d_4 (Conv2D) (None, 30, 30, 32) 9248
_________________________________________________________________
activation_1 (Activation) (None, 30, 30, 32) 0
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 29, 29, 32) 0
_________________________________________________________________
conv2d_5 (Conv2D) (None, 29, 29, 64) 18496
_________________________________________________________________
activation_2 (Activation) (None, 29, 29, 64) 0
_________________________________________________________________
conv2d_6 (Conv2D) (None, 27, 27, 64) 36928
_________________________________________________________________
activation_3 (Activation) (None, 27, 27, 64) 0
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 13, 13, 64) 0
_________________________________________________________________
flatten_1 (Flatten) (None, 10816) 0
_________________________________________________________________
dense_17 (Dense) (None, 128) 1384576
_________________________________________________________________
activation_4 (Activation) (None, 128) 0
_________________________________________________________________
dropout_2 (Dropout) (None, 128) 0
_________________________________________________________________
dense_18 (Dense) (None, 10) 1290
_________________________________________________________________
activation_5 (Activation) (None, 10) 0
=================================================================
Total params: 1,451,434
Trainable params: 1,451,434
Non-trainable params: 0
_________________________________________________________________

4.2.4 Model Training

In [47]:
from tensorflow.keras.callbacks import ModelCheckpoint
model_name = "final_cifar10.h5"
model_checkpoint = ModelCheckpoint(model_name, monitor='loss',verbose=1,
save_best_only=True)

#load pretrained models

trained_weights_path = 'cifar10_weights.h5'
if os.path.exists(trained_weights_path):
model.load_weights(trained_weights_path, by_name =True)
#train
model.fit(x_train,y_train, batch_size=32, epochs=5,callbacks =
[model_checkpoint],verbose=1)
Epoch 1/5
1563/1563 [==============================] - 876s 560ms/step - loss: 1.5976 -
accuracy: 0.4237

Epoch 00001: loss improved from inf to 1.59755, saving model to

final_cifar10.h5
Epoch 2/5
1563/1563 [==============================] - 868s 555ms/step - loss: 1.2994 -
accuracy: 0.5388

Epoch 00002: loss improved from 1.59755 to 1.29939, saving model to

final_cifar10.h5
Epoch 3/5
1563/1563 [==============================] - 879s 562ms/step - loss: 1.1582 -
accuracy: 0.5928

Epoch 00003: loss improved from 1.29939 to 1.15817, saving model to

final_cifar10.h5
Epoch 4/5
1563/1563 [==============================] - 878s 562ms/step - loss: 1.0438 -
accuracy: 0.6371

Epoch 00004: loss improved from 1.15817 to 1.04376, saving model to

final_cifar10.h5
Epoch 5/5
1563/1563 [==============================] - 881s 564ms/step - loss: 0.9567 -
accuracy: 0.6660

Epoch 00005: loss improved from 1.04376 to 0.95673, saving model to

final_cifar10.h5
Out[47]:
<tensorflow.python.keras.callbacks.History at 0x7fa2d805ebd0>
The network in this experiment is simple, consisting of four convolutional layers. To improve the performance
of this model, you can increase the number of epochs and the complexity of the model.

4.2.5 Model Evaluation

In [48]:
new_model = CNN_classification_model()
new_model.load_weights('final_cifar10.h5')

model.evaluate(x_test, y_test, verbose=1)

313/313 [==============================] - 64s 205ms/step - loss: 0.9439 -
accuracy: 0.6672
Out[48]:
[0.9438978433609009, 0.6672000288963318]
Predict on a single image.
 In [49]:
#output the possibility of each class
new_model.predict(x_test[0:1])
Out[49]:
array([[0.00488163, 0.00135425, 0.02675421, 0.73704094, 0.01140506,
0.15094 , 0.05759602, 0.0035899 , 0.00389682, 0.00254116]],
dtype=float32)
In [50]:
#output the predicted label
np.argmax(new_model.predict(x_test[0:1]), axis=-1)
Out[50]:
array([3])
Plot the first 4 images in the test set and their corresponding predicted labels.
In [51]:
#label list
pred_list = []

plt.figure()
for i in range(0,10):
plt.subplot(5,2,i+1)
#plot
plt.imshow(x_test[i])
#predict
# pred = new_model.predict_classes(x_test[0:10])
pred = np.argmax(new_model.predict(x_test[0:10]), axis=-1)
pred_list.append(pred)
#Display actual and predicted labels of images
plt.title("pred:"+category_dict[pred[i]]+" actual:"+
category_dict[y_test[i][0]])
plt.axis('off')
plt.show()

4.3 Summary
This chapter describes how to build an image classification model based on TensorFlow 2 and python. It
provides trainees with a basic concept of deep learning model building.