Lecture 6 – Computational Graphs; PyTorch and
Tensorflow
DD2424
April 11, 2019
DD2424 - Lecture 8 1
Outline
• First Part
• Computation Graphs
• TensorFlow
• PyTorch
• Notes
• Second Part
DD2424 - Lecture 8 2
Frameworks
DD2424 - Lecture 8 3
Frameworks
DD2424 - Lecture 8 4
O’Reilly Poll: Most popular framework for machine learning
[ Source: https://www.techrepublic.com/google-amp/article/most-popular-
programming-language-frameworks-and-tools-for-machine-learning/ ]
DD2424 - Lecture 8 5
What are computation graphs?
DD2424 - Lecture 8 6
Computation Graph
• DAG (directed acyclic graph)
• Nodes
• Variables
• Mathematical Operations
var
• Edges
• Feeding input op
var
DD2424 - Lecture 8 7
Computation Graph
•𝑐 = 𝑎+𝑏
𝒄=𝒂+𝒃
DD2424 - Lecture 8 8
Computation Graph
•𝑐 = 𝑎+𝑏∗2
𝒄=𝒂+𝒛
𝒃 𝒛=𝒃∗𝟐
DD2424 - Lecture 8 9
Computation Graph
• Tensors: Multi-dimensional arrays
• 𝒂 = 𝑊𝒙 + 𝒃
𝑧 = 𝑾𝑥 a= 𝒛 + 𝒃
DD2424 - Lecture 8 10
Computation Graph
• A feed-forward neural network
𝑾𝟏
𝑧1 = 𝑾𝟏 𝑥 𝒂𝟏 = 𝒛𝟏 + 𝒃𝟏 𝒔𝟏 = 𝝈(𝒂𝟏 )
𝒃𝟏
DD2424 - Lecture 8 11
Computation Graph
• A multi-layer feed-forward neural network
𝑾𝟏 𝑾𝟐
𝑧1 = 𝑾𝟏 𝑥 𝒂𝟏 = 𝒛𝟏 + 𝒃𝟏 𝒔𝟏 = 𝝈(𝒂𝟏 ) 𝑧2 = 𝑾𝟐 𝒔𝟏 𝒂𝟐 = 𝒛𝟐 + 𝒃𝟐 𝒔𝟐 = 𝝈(𝒂𝟐 )
𝒃𝟏 𝒃𝟐
DD2424 - Lecture 8 12
Python (NumPy)
𝑧 = 𝑾𝑥 a= 𝒛 + 𝒃
𝒃
DD2424 - Lecture 8 13
PyTorch
NumPy
PyTorch
𝑾
𝑧 = 𝑾𝑥 a= 𝒛 + 𝒃
𝒃
DD2424 - Lecture 8 14
PyTorch
NumPy
PyTorch
Not always!
DD2424 - Lecture 8 15
PyTorch-NumPy
• Converting a Torch Tensor to a NumPy array and vice versa is a breeze.
DD2424 - Lecture 8 16
PyTorch-NumPy
• Converting a Torch Tensor to a NumPy array and vice versa is a breeze.
Shared Memory
DD2424 - Lecture 8 17
PyTorch-NumPy
• Converting a Torch Tensor to a NumPy array and vice versa is a breeze.
DD2424 - Lecture 8 18
“Define by Run” Computation Graphs
This kind of computation graph is called “define by run“
Also referred to as “dynamic”
DD2424 - Lecture 8 19
“Define and Run” Computation Graphs
• First define the graph structure
• Then run it by feeding in the (input) variables.
Define graph G Run the graph G
𝑾𝟏
• Run G with 𝑥1 , 𝑊1 , 𝑏1
𝑧1 = 𝑾𝟏 𝑥 𝒂𝟏 = 𝒛𝟏 + 𝒃𝟏 𝒔𝟏 = 𝝈(𝒂𝟏 )
𝒙 • Run G with 𝑥2 , 𝑊2 , 𝑏2
𝒃𝟏
• …
Also known as “static graphs”
DD2424 - Lecture 8 20
Run graph
Define graph
many times
DD2424 - Lecture 8
21
TensorFlow
Data loop
• Dynamic Graph • Static Graph
DD2424 - Lecture 8 22
Why computation graphs at all?!
DD2424 - Lecture 8 23
Why computation graphs?
• In lecture 3, you’ve learnt how to do backprop using the chain rule
DD2424 - Lecture 8 24
Why computation graphs?
• Is it feasible?
DD2424 - Lecture 8 25
Why computation graphs?
• Automatic chain rule
• automatic back-prop using implemented operations
• Each operation has their gradient already implemented
• If you want to use a novel operation, then you have to provide it’s gradient w.r.t. inputs
and its learnable parameters (if any)
DD2424 - Lecture 8 26
Let’s look at examples in PyTorch and TensorFlow
DD2424 - Lecture 8 27
Computation Graph
• A feed-forward neural network
𝑾𝟏
𝑧1 = 𝑾𝟏 𝑥 𝒂𝟏 = 𝒛𝟏 + 𝒃𝟏 𝒔𝟏 = 𝝈(𝒂𝟏 )
𝒃𝟏
DD2424 - Lecture 8 28
Computation Graph
• A feed-forward neural network with squared 𝐿2 loss
𝑾𝟏
𝑧1 = 𝑾𝟏 𝑥 𝒂𝟏 = 𝒛𝟏 + 𝒃𝟏 𝒔𝟏 = 𝝈(𝒂𝟏 ) 𝑙 = |𝑠1 − 𝑦 |2
𝒃𝟏 𝒚
DD2424 - Lecture 8 29
Backprop in Computation Graph
• Learnable parameters
𝑾𝟏
𝑧1 = 𝑾𝟏 𝑥 𝒂𝟏 = 𝒛𝟏 + 𝒃𝟏 𝒔𝟏 = 𝝈(𝒂𝟏 ) 𝑙 = |𝑠1 − 𝑦 |2
𝒃𝟏 𝒚
DD2424 - Lecture 8 30
Backprop in Computation Graph
𝜕𝑙
𝜕𝑊1
𝑾𝟏
𝑧1 = 𝑾𝟏 𝑥 𝒂𝟏 = 𝒛𝟏 + 𝒃𝟏 𝒔𝟏 = 𝝈(𝒂𝟏 ) 𝑙 = |𝑠1 − 𝑦 |2
𝒃𝟏 𝒚
𝜕𝑙
𝜕𝑏1
DD2424 - Lecture 8 31
Backprop in Computation Graph
𝜕𝑙
𝜕𝑧1
𝜕𝑊1
𝑾𝟏 𝜕𝑊1 𝜕𝑎1 𝜕𝑠1 𝜕𝑙
𝜕𝑧1 𝜕𝑎1 𝜕𝑠1
𝑧1 = 𝑾𝟏 𝑥 𝒂𝟏 = 𝒛𝟏 + 𝒃𝟏 𝒔𝟏 = 𝝈(𝒂𝟏 ) 𝑙 = |𝑠1 − 𝑦 |2
𝒙
𝜕𝑎1
𝒃𝟏 𝒚
𝜕𝑏1
𝜕𝑙
𝜕𝑏1
DD2424 - Lecture 8 32
Backprop in Computation Graph
A deep learning framework provides an automatic gradient calculation
of its output variables w.r.t. its input variables
𝜕𝑙
𝜕𝑧1
𝜕𝑊1
𝑾𝟏 𝜕𝑊1 𝜕𝑎1 𝜕𝑠1 𝜕𝑙
𝜕𝑧1 𝜕𝑎1 𝜕𝑠1
𝑧1 = 𝑾𝟏 𝑥 𝒂𝟏 = 𝒛𝟏 + 𝒃𝟏 𝒔𝟏 = 𝝈(𝒂𝟏 ) 𝑙 = |𝑠1 − 𝑦 |2
𝒙
𝜕𝑎1
𝒃𝟏 𝒚
𝜕𝑏1
𝜕𝑙
𝜕𝑏1
DD2424 - Lecture 8 33
Backprop in Computation Graph
• Addition Node
• Forward pass: 𝑎 = 𝑏 + 𝑐
𝜕𝑎 𝜕𝑎
• Backward pass: 𝜕𝑏 = 1 and 𝜕𝑐
=1
DD2424 - Lecture 8 34
Backprop in Computation Graph
• Max Node
• Forward pass: 𝑎 = max 𝑏, 𝑐
• Backward pass:
• If b < c
𝜕𝑎 𝜕𝑎
• 𝜕𝑏
= 0 and
𝜕𝑐
=1
max
• If b > c
𝜕𝑎 𝜕𝑎
• 𝜕𝑏
= 1 and
𝜕𝑐
=0
DD2424 - Lecture 8 35
Variables and Ops
• Ops
• Intermediate or final nodes
• Variables
• intrinsic parameters of the model
• input to the model
𝑾𝟏
𝑧1 = 𝑾𝟏 𝑥 𝒂𝟏 = 𝒛𝟏 + 𝒃𝟏 𝒔𝟏 = 𝝈(𝒂𝟏 ) 𝑙 = |𝑠1 − 𝑦 |2
𝒃𝟏 𝒚
DD2424 - Lecture 8 36
Variables and Ops
• Ops
• Intermediate or final nodes
• Variables
• intrinsic parameters of the model
• input to the model
𝑾𝟏
𝑧1 = 𝑾𝟏 𝑥 𝒂𝟏 = 𝒛𝟏 + 𝒃𝟏 𝒔𝟏 = 𝝈(𝒂𝟏 ) 𝑙 = |𝑠1 − 𝑦 |2
𝒃𝟏 𝒚
DD2424 - Lecture 8 37
Variables and Ops
• Variables
• Intrinsic parameters of the model
• Input to the model
• TensorFlow
• Variables
• Place Holders 𝑾𝟏
𝑧1 = 𝑾𝟏 𝑥 𝒂𝟏 = 𝒛𝟏 + 𝒃𝟏 𝒔𝟏 = 𝝈(𝒂𝟏 ) 𝑙 = |𝑠1 − 𝑦 |2
• PyTorch 𝒙
• Variables
𝒃𝟏 𝒚
DD2424 - Lecture 8 38
Variable
PyTorch Autograd
• package: torch.autograd
Data
Tensor
Gradient
w.r.t.
this variable
Function
that created
this variable DD2424 - Lecture 8 39
Pytorch Autograd
DD2424 - Lecture 8 40
Pytorch Autograd
DD2424 - Lecture 8 41
Pytorch Autograd
DD2424 - Lecture 8 42
Pytorch Autograd
DD2424 - Lecture 8 43
PyTorch Autograd
• Calculate gradient using backward() method of a Variable
• var.backward()
DD2424 - Lecture 8 44
TensorFlow gradients
• Add gradient nodes in the graph where necessary using
Tf.gradients(ys, xs, gs)
• And evaluate it
DD2424 - Lecture 8 45
TensorFlow gradients
• Then update the parameters
DD2424 - Lecture 8 46
TensorFlow gradient
• Use tf.Variable instead
DD2424 - Lecture 8 47
How to use GPU?
DD2424 - Lecture 8 48
PyTorch GPU
Turn variables into “GPU” variables by the following command:
• var = var.cuda(#)
DD2424 - Lecture 8 49
PyTorch GPU
Turn back variables into “CPU” variables by the following command:
• var = var.cpu()
DD2424 - Lecture 8 50
TensorFlow GPU
• In TF variables or operations can sit on specific device
• tf.device(/gpu:0)
• tf.device(/gpu:1)
•…
• tf.device(/cpu:0)
DD2424 - Lecture 8 51
TensorFlow GPU
• In TF variables or operations can sit on specific device
tf.Session(config=tf.ConfigProto(log_device_placement=True))
MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0
2018-04-10 12:59:09.508497: I tensorflow/core/common_runtime/placer.cc:874] MatMul: (MatMul)/job:localhost/replica:0/task:0/device:GPU:0
add: (Add): /job:localhost/replica:0/task:0/device:GPU:0
2018-04-10 12:59:09.508513: I tensorflow/core/common_runtime/placer.cc:874] add: (Add)/job:localhost/replica:0/task:0/device:GPU:0
Maximum: (Maximum): /job:localhost/replica:0/task:0/device:GPU:0
2018-04-10 12:59:09.508525: I tensorflow/core/common_runtime/placer.cc:874] Maximum: (Maximum)/job:localhost/replica:0/task:0/device:GPU:0
Maximum/y: (Const): /job:localhost/replica:0/task:0/device:GPU:0
2018-04-10 12:59:09.508537: I tensorflow/core/common_runtime/placer.cc:874] Maximum/y: (Const)/job:localhost/replica:0/task:0/device:GPU:0
Placeholder_2: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
2018-04-10 12:59:09.508548: I tensorflow/core/common_runtime/placer.cc:874] Placeholder_2: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
Placeholder_1: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
2018-04-10 12:59:09.508558: I tensorflow/core/common_runtime/placer.cc:874] Placeholder_1: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
Placeholder: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
2018-04-10 12:59:09.508567: I tensorflow/core/common_runtime/placer.cc:874] Placeholder: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
DD2424 - Lecture 8 52
TensorFlow GPU
• Some TF operations do not have a CUDA implementation
tf.Session(config=tf.ConfigProto(
allow_soft_placement=True, log_device_placement=True))
DD2424 - Lecture 8 53
How to implement complicated models in practice?
DD2424 - Lecture 8 54
PT High-Level Library
• PyTorch package called nn and class called Module
DD2424 - Lecture 8 55
TF High-Level Libraries
• Keras: highest abstraction
• SLIM: best pre-trained models
• TFLearn,
• Sonnet,
• Pretty Tensor,
•…
DD2424 - Lecture 8 56
Data, storage, and loading
!!!Important!!!
• Always monitor CPU/GPU usage (linux: nvidia-smi, top)
• Make storage more efficient (TF Records, etc.)
• Make reading pipeline more efficient (parallel readers, prefetching,
etc.)
DD2424 - Lecture 8 57
Use Visualization
• Always monitor the loss function on the training and validation sets visually
• Monitor all other important scalars, such as learning rate, regularization loss,
layer activations summary, how full your data queues are, and …
• If you have an imbalanced classification problem, visualize the CE loss separately
for each class.
• If you work with images, time to time visualize samples from the batch, if you do
data augmentation, visualize the original sample as well as the augmented one
• TensorBoard for TF
• TensorBoardX, matplotlib, seaborn, … for PT
DD2424 - Lecture 8 58
Use Visualization
You can have the configuration shown as a text file in tensorboard!
DD2424 - Lecture 8 59
Which one is better? PyTorch or TensorFlow?
DD2424 - Lecture 8 60
pros and cons
• PyTorch: easier for prototyping
• PyTorch: much easier to implement flexible graphs
• PyTorch: different structures in each iteration (dependent on data). This is possible with TF too, but is a pain.
• PyTorch: manipulating weight and gradients
• PyTorch: code-level debugging (breakpoints, imperative, tracing your own code instead of TF kernels)
• PyTorch: probably better abstractions for dataset, variable, parallelism, etc. but TF has many high-level wrappers with better abstractions
• Tie?!: Faster run-time, (NHWC v.s. NCHW)
• TF: TensorBoard
• TF: research-level debugging (TensorBoard)
• TF: windows
• TF: distributed training (PyTorch has it now too, but seems not as developed as the TF version)
• TF: easier with distributing the code over multiple devices (GPUs/CPU) (maybe not anymore)
• TF: online community is noticeably larger
• TF: data readers
• TF: supposedly more optimizations of the graph (done by the engine)
• TF: documentation and tutorials
• TF: more models available
• TF: Serialization, code and portability (saving and loading models for across platforms, or checkpoints)
• TF: Deployment: Server, Mobile, etc. (TensorFlow Serving, TensorFlow Lite)
• TF: Richer API (e.g. FFT)
• TF: Automatic shape inference
• TF has a MOOC: https://eu.udacity.com/course/deep-learning--ud730
DD2424 - Lecture 8 61
TensorFlow Eager execution
• Eager Execution
• Dynamic!
• tf.enable_eager_execuation()
• Considerably Slower (being worked on)
• https://www.tensorflow.org/guide/eager
DD2424 - Lecture 8 62
Caffe(2)
• Portability is seamless (e.g. mobile apps)
• Simplest framework for fine-tuning or feature extraction
• Used to be fastest (Caffe)
DD2424 - Lecture 8 63
Summary
• Don’t take the following statements too seriously! -- it depends on many factors
• If you want to use pretrained classic deep networks (AlexNet, VGG, ResNet, …) for feature extraction and/or fine-
tuning → Use Caffe and/or Caffe2
• If you have a mobile application in mind → Use Caffe/Caffe2 or TensorFlow
• If you want more pythonic → use PyTorch
• If you are familiar with Matlab and don’t need much flexibility or advanced layers → use MatConvNet
• If you don’t want so much of flexibility and still use python → use Keras
• If you are working on NLP applications or complicated RNNs → use PyTorch
• If you want large community help, sustainable learning of a framework → use TensorFlow
• If you want to work on bleeding-edge papers → See what framework has the original and/or cleanest
implementation (most likely TensorFlow)
• If you want to prototype many different novel setups → Use PyTorch or TF Eager
DD2424 - Lecture 8 64
