Large-scale Intelligent Systems Laboratory
Large-scale Intelligent Systems Laboratory (Li Lab)
National Science Foundation Center for Big Learning (CBL)
Department of Electrical and Computer Engineering (ECE)
Department of Computer & Information Science & Engineering (CISE)
Pytorch Tutorial
Xiaoyong Yuan, Xiyao Ma
2018/01
some slides copied from cs231n.stanford.edu
Large-scale Intelligent Systems Laboratory
Introduction
■ Why we need deep learning frameworks
(1) Easily build big computational graphs
(2) Easily compute gradients in computational graphs
(3) Run it all efficiently on GPU
■ Why we need Pytorch?
(1) Easy to implement, code, and debug
(2) More flexible due to its dynamic computational graph.
(3) High execution efficiency, since it developed from C.
Large-scale Intelligent Systems Laboratory
PyTorch: Three Levels of Abstraction
Tensor: Like array in Numpy, but runs on GPU
Variable: Node in a computational graph; stores
data and gradient
Module: A neural network layer; may store state or learnable weights
Large-scale Intelligent Systems Laboratory
PyTorch: Tensors
■ PyTorch Tensors are just like numpy arrays,
but they can run on GPU.
■ No built-in notion of computational graph, or
gradients, or deep learning.
■ Here we fit a two-layer net using PyTorch
Tensors
Large-scale Intelligent Systems Laboratory
PyTorch: Tensors
To run on GPU, just cast tensors to a
cuda data type!
(E,g torch.cuda.FloatTensor)
Create random tensors for data and
weights.
Forward pass:
compute predictions and loss
Backward pass:
manually compute gradients
Gradient descent step on weights
Large-scale Intelligent Systems Laboratory
Tensor: Operations http://pytorch.org/docs/
master/index.html
Create a tensor:
Math Operation:
Other operations:
Large-scale Intelligent Systems Laboratory
PyTorch: Autograd
■ A PyTorch Variable is a node in a
computational graph
■ x.data is a Tensor
■ x.grad is a Variable of gradients
(same shape as x.data)
■ x.grad.data is a Tensor of gradients
■ PyTorch Tensors and Variables have
the same API!
■ Variables remember how they
were created (for backprop)
Large-scale Intelligent Systems Laboratory
PyTorch: Autograd
We will not want gradients (of loss)
with respect to data
Do want gradients with respect to
weights
Forward pass looks exactly the same
as the Tensor version, but everything
is a variable now
Compute gradient of loss with respect
to w1 and w2 (zero out grads first)
Make gradient step on weights
Large-scale Intelligent Systems Laboratory
PyTorch: New Autograd Functions
Large-scale Intelligent Systems Laboratory
PyTorch: New Autograd Functions
Define your own autograd functions
by writing forward and backward for
Tensors
Large-scale Intelligent Systems Laboratory
PyTorch: nn
Higher-level wrapper for working with neural
nets
Large-scale Intelligent Systems Laboratory
PyTorch: nn
Define our model as a sequence of
layers
nn also defines loss functions
Forward pass: feed data to model,
and prediction to loss function
Backward pass:
compute all gradients
Make gradient step on each model
parameter
Large-scale Intelligent Systems Laboratory
Example: Define your own Module
You can find more details about API in the following link:
http://pytorch.org/docs/master/index.html
Large-scale Intelligent Systems Laboratory
PyTorch: optim
Use an optimizer for different update
rules
Update all parameters after
computing gradients
Large-scale Intelligent Systems Laboratory
PyTorch: nn Define new Modules
Define our whole model as a single
Module
Initializer sets up two children
(Modules can contain modules)
Note: No need to define backward -
autograd will handle it
Define forward pass using child
modules and autograd ops on
Variables
Construct and train an instance of our
model
Large-scale Intelligent Systems Laboratory
MNIST Example
https://github.com/pytorch/examples/blob/master/mnist/main.py
Large-scale Intelligent Systems Laboratory
Extending PyTorch
■ torch.autograd
■ customize forward() and backward()
■ check gradient
■ from torch.autograd import
gradcheck
Large-scale Intelligent Systems Laboratory
Extending PyTorch
■ torch.autograd
■ Adding function to a
Module
Large-scale Intelligent Systems Laboratory
Extending PyTorch
■ C-extension
a. prepare your C code
header with all functions
c file
Large-scale Intelligent Systems Laboratory
Extending PyTorch
■ C-extension
b. build custom extension c. include it in your Python code
run it and get a new folder with a .so file
Large-scale Intelligent Systems Laboratory
Torchvision
■ popular datasets
■ cifar10, coco, lsun, mnist, ...
■ popular model architectures (pretrained)
■ alexnet, densenet, inception, resnet, squeezenet, vgg
■ common image transformations
■ Normalize, Scale, CenterCrop, Pad, RandomCrop,
RandomFlip, …
■ Compose
Large-scale Intelligent Systems Laboratory
Visualization
■ TensorboardX
■ pip install tensorboardX
■ tensorboard for pytorch (and chainer,
mxnet, numpy, ...)
■ Support scalar, image, histogram, audio,
text, graph, onnx_graph, embedding and
pr_curve
■ demo http://35.197.26.245:6006/
class tensorboardX.SummaryWriter(log_dir=None, comment='')
Large-scale Intelligent Systems Laboratory
Visualization
■ TensorboardX
a. Run the demo script: python demo.py
b. Use TensorBoard with tensorboard --logdir runs (needs to install
TensorFlow)
Large-scale Intelligent Systems Laboratory
Visualization
■ Other choices
Visdom Seaborn
Large-scale Intelligent Systems Laboratory
Onnx
■ Open Neural Network Exchange (ONNX)
■ an open source format to move models between tools
■ defines an extensible computation graph model, built-in operators and
standard data types
■ Hardware Optimizations
■ Supported Tools
Frameworks
Converters Runtimes
Large-scale Intelligent Systems Laboratory
References
1. http://pytorch.org/
2. http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture
8.pdf
3. https://github.com/pytorch/pytorch
Large-scale Intelligent Systems Laboratory
Thank
You!
