Pytorch Mobile Performance Recipes

Introduction

Performance (aka latency) is crucial to most, if not all,

applications and use-cases of ML model inference on mobile devices.

Today, PyTorch executes the models on the CPU backend pending availability

of other hardware backends such as GPU, DSP, and NPU.

Model preparation

Next recipes you can take (offline) while preparing the model

to have an optimized model that will probably have shorter execution time

(higher performance, lower latency) on the mobile device.

Setup

First we need to installed pytorch using conda or pip with version at least 1.5.0.

::

or

::

Code your model:

::

``torch.quantization.QuantStub`` and ``torch.quantization.DeQuantStub()`` are no-op stubs, which will be used for quantization step.

1. Fuse operators using ``torch.quantization.fuse_modules``

Do not be confused that fuse_modules is in the quantization package.

It works for all ``torcn.nn.Module``.

``torch.quantization.fuse_modules`` fuses a list of modules into a single module.

It fuses only the following sequence of modules:

- Convolution, Batch normalization

- Convolution, Batch normalization, Relu

- Convolution, Relu

- Linear, Relu

This script will fuse Convolution, Batch Noarmalization and Relu in previously declared model.

::

2. Quantize your model

You can find more about PyTorch quantization in

`the dedicated tutorial <https://pytorch.org/blog/introduction-to-quantization-on-pytorch/>`_.

Quantization of the model not only moves computation to int8,

but also reduces the size of your model on a disk.

That size reduction helps to reduce disk read operations during the first load of the model and decreases the amount of RAM.

Both of those resources can be crucial for the performance of mobile applications.

::

3. Use torch.utils.mobile_optimizer

Torch mobile_optimizer package does several optimizations with the scripted model,

which will help to conv2d and linear operations.

It pre-packs model weights in an optimized format and fuses ops above with relu

if it is the next operation.

First we script the result model from previous step:

::

Next we call ``optimize_for_mobile`` and save model on the disk.

::

4. Android. Reusing tensors for forward.

This recipe is Android only.

Memory is a critical resource for android performance, especially on old devices.

Tensors can need a significant amount of memory.

For example, standard computer vision tensor contains 1*3*224*224 elements,

assuming that data type is float and will need 588Kb of memory.

::

Here we allocate native memory as ``java.nio.FloatBuffer`` and creating ``org.pytorch.Tensor`` which storage will be pointing to the memory of the allocated buffer.

For most of the use cases, we do not do model forward only once, repeating it with some frequency or as fast as possible.

If we are doing new memory allocation for every module forward - that will be suboptimal.

Instead of this, we can reuse the same memory that we allocated on the previous step, fill it with new data, and run module forward again on the same tensor object.

You can check how it looks in code in `pytorch android application example <https://github.com/pytorch/android-demo-app/blob/master/PyTorchDemoApp/app/src/main/java/org/pytorch/demo/vision/ImageClassificationActivity.java#L174>`_.

::

Member fields ``mModule``, ``mInputTensorBuffer`` and ``mInputTensor`` are initialized only once

and buffer is refilled using ``org.pytorch.torchvision.TensorImageUtils.imageYUV420CenterCropToFloatBuffer``.

Benchmarking

The best way to benchmark (to check if optimizations helped your use case) - to measure your particular use case that you want to optimize, as performance behavior can vary in different environments.

PyTorch distribution provides a way to benchmark naked binary that runs the model forward,

this approach can give more stable measurements rather than testing inside the application.

Android

For this you first need to build benchmark binary:

::

You should have arm64 binary at: ``build_android/bin/speed_benchmark_torch``.

This binary takes ``--model=<path-to-model>``, ``--input_dim="1,3,224,224"`` as dimension information for the input and ``--input_type="float"`` as the type of the input as arguments.

Once you have your android device connected,

push speedbenchark_torch binary and your model to the phone:

::

Now we are ready to benchmark your model:

::

14. mobile_perf.py

PyTorch Mobile Performance Recipes

https://pytorch.org/tutorials/recipes/mobile_perf.html

.. customcarditem::

:header: PyTorch Mobile Performance Recipes

:card_description: List of recipes for performance optimizations for using PyTorch on Mobile.

:image: ../_static/img/thumbnails/cropped/zeroing-out-gradients.PNG

:link: ../recipes/mobile_perf.html

:tags: Mobile,Model-Optimization