Skip to content

Commit 061351a

Browse files
xta0Pat Mellon
xta0
authored and
Pat Mellon
committed
Merge remote-tracking branch 'upstream/mobile-launch' into mobile-launch
2 parents 4a91e9b + 232f796 commit 061351a

File tree

1 file changed

+310
-5
lines changed

1 file changed

+310
-5
lines changed

‎_mobile/android.md

+310-5
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,316 @@ published: true
1010

1111
# Android
1212

13-
{% highlight python %}
13+
## Quick start with Hello World
1414

15-
#!/usr/bin/python3
16-
print('Hello World!')
15+
The easiest way to start playing with pytorch android is to checkout on githyb our ['Hello World' application](https://github.com/pytorch/android-demo-app/tree/master/HelloWorldApp)
1716

18-
{% endhighlight %}
17+
This application runs [torchscript serialized](https://github.com/pytorch/android-demo-app/blob/master/HelloWorldApp/trace_model.py)
18+
[torch vision pretrained resnet18 model](https://pytorch.org/docs/stable/torchvision/models.html) on [image.jpg](https://github.com/pytorch/android-demo-app/blob/master/HelloWorldApp/app/src/main/assets/image.jpg) which packaged inside the app as android asset.
19+
20+
```
21+
git clone https://github.com/pytorch/android-demo-app.git
22+
cd HelloWorldApp
23+
```
24+
If [android sdk]() and [android ndk]() is already installed you can install this application to connected android device or emulator with:
25+
```
26+
./gradlew installDebug
27+
```
28+
29+
We recommend you to open this project in [Android Studio](https://developer.android.com/studio),
30+
in that case you will be able to install android ndk and android sdk using Android Studio UI.
31+
32+
The easiest way to add 'pytorch android' to the app is adding [gradle dependencies](https://github.com/pytorch/android-demo-app/blob/master/HelloWorldApp/app/build.gradle#L28-L29) to your build.gradle:
33+
34+
```
35+
repositories {
36+
jcenter()
37+
}
38+
39+
dependencies {
40+
implementation 'org.pytorch:pytorch_android:0.0.8'
41+
implementation 'org.pytorch:pytorch_android_torchvision:0.0.8'
42+
}
43+
```
44+
Where org.pytorch:pytorch_android is the main dependency with android api, including libtorch native library for all 4 android abis (armeabi-v7a, arm64-v8a, x86, x86_64).
45+
Further in this doc you can find how to rebuild it only for specific list of android abis.
46+
47+
org.pytorch:pytorch_android_torchvision - library with several utility functions for converting `android.media.Image` and `android.graphics.Bitmap` to tensors.
48+
49+
All logic happens in [org.pytorch.helloworld.MainActivity](https://github.com/pytorch/android-demo-app/blob/master/HelloWorldApp/app/src/main/java/org/pytorch/helloworld/MainActivity.java#L31-L69):
50+
```
51+
Bitmap bitmap = BitmapFactory.decodeStream(getAssets().open("image.jpg"));
52+
```
53+
Reading image.jpg to `android.graphics.Bitmap` using standard android api.
54+
55+
```
56+
Module module = Module.load(assetFilePath(this, "model.pt"));
57+
```
58+
`org.pytorch.Module` represents `torch::jit::script::Module` that can be loaded with `load` method specifying file path to the serialized to file model.
59+
60+
```
61+
Tensor inputTensor = TensorImageUtils.bitmapToFloat32Tensor(bitmap,
62+
TensorImageUtils.TORCHVISION_NORM_MEAN_RGB, TensorImageUtils.TORCHVISION_NORM_STD_RGB);
63+
```
64+
`org.pytorch.torchvision.TensorImageUtils` is part of 'org.pytorch:pytorch_android_torchvision' library.
65+
`TensorImageUtils#bitmapToFloat32Tensor` method creates tensor in [torch vision format](https://pytorch.org/docs/stable/torchvision/models.html) using `android.graphics.Bitmap` as a source.
66+
67+
> All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224.
68+
> The images have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]
69+
70+
`inputTensor`'s shape is 1x3xHxW, where H and W are bitmap height and width appropriately.
71+
72+
```
73+
Tensor outputTensor = module.forward(IValue.tensor(inputTensor)).getTensor();
74+
float[] scores = outputTensor.getDataAsFloatArray();
75+
```
76+
77+
`org.pytorch.Module#forward` method runs loaded module's `forward` method and gets result as `org.pytorch.Tensor` outputTensor with shape 1x1000.
78+
It's content is retrieved using `org.pytorch.Tensor#getDataAsFloatArray()` method that returns java array of floats with scores for every image net class.
79+
80+
After that we just find index with maximum score and retrieve predicted class name from `ImageNetClasses.IMAGENET_CLASSES` array that contains all ImageNet classes.
81+
82+
```
83+
float maxScore = -Float.MAX_VALUE;
84+
int maxScoreIdx = -1;
85+
for (int i = 0; i < scores.length; i++) {
86+
if (scores[i] > maxScore) {
87+
maxScore = scores[i];
88+
maxScoreIdx = i;
89+
}
90+
}
91+
String className = ImageNetClasses.IMAGENET_CLASSES[maxScoreIdx];
92+
```
93+
94+
Now you are ready to start using other torchsript models on android, changing [`model.pt`](https://github.com/pytorch/android-demo-app/blob/master/HelloWorldApp/app/src/main/assets/model.pt) and rebuilding the project :)
95+
96+
To prepare serialize the model you can use python [script](https://github.com/pytorch/android-demo-app/blob/master/HelloWorldApp/trace_model.py):
97+
```
98+
import torch
99+
import torchvision
100+
101+
model = torchvision.models.resnet18(pretrained=True)
102+
model.eval()
103+
example = torch.rand(1, 3, 224, 224)
104+
traced_script_module = torch.jit.trace(model, example)
105+
traced_script_module.save("app/src/main/assets/model.pt")
106+
```
107+
108+
More details about torchscript you can find in [tutorials and docs on pytorch.org](https://pytorch.org/docs/stable/jit.html)
109+
110+
In the following sections you can find detailed explanation of pytorch android api, code walk through for bigger [demo application](https://github.com/pytorch/android-demo-app/tree/master/PyTorchDemoApp), implementation details of api and how to customize and build it from the source.
111+
112+
## Pytorch demo app
113+
114+
Bigger example of application that does image classification from android camera output and text classification you can find in the [same github repo](https://github.com/pytorch/android-demo-app/tree/master/PyTorchDemoApp).
115+
116+
To get device camera output in it uses [android cameraX api](https://developer.android.com/training/camerax
117+
).
118+
[github code point](https://github.com/pytorch/android-demo-app/blob/master/PyTorchDemoApp/app/src/main/java/org/pytorch/demo/vision/AbstractCameraXActivity.java#L71-L92)
119+
120+
```
121+
void setupCameraX() {
122+
final PreviewConfig previewConfig = new PreviewConfig.Builder().build();
123+
final Preview preview = new Preview(previewConfig);
124+
preview.setOnPreviewOutputUpdateListener(output -> mTextureView.setSurfaceTexture(output.getSurfaceTexture()));
125+
126+
final ImageAnalysisConfig imageAnalysisConfig =
127+
new ImageAnalysisConfig.Builder()
128+
.setTargetResolution(new Size(224, 224))
129+
.setCallbackHandler(mBackgroundHandler)
130+
.setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE)
131+
.build();
132+
final ImageAnalysis imageAnalysis = new ImageAnalysis(imageAnalysisConfig);
133+
imageAnalysis.setAnalyzer(
134+
(image, rotationDegrees) -> {
135+
analyzeImage(image, rotationDegrees);
136+
});
137+
138+
CameraX.bindToLifecycle(this, preview, imageAnalysis);
139+
}
140+
141+
void analyzeImage(android.media.Image, int rotationDegrees)
142+
```
143+
144+
Where [analyzeImage](https://github.com/pytorch/android-demo-app/blob/master/PyTorchDemoApp/app/src/main/java/org/pytorch/demo/vision/ImageClassificationActivity.java#L128) method analyzes camera output, `android.media.Image`.
145+
146+
It uses aforementioned [`TensorImageUtils#imageYUV420CenterCropToFloat32Tensor`](https://github.com/pytorch/pytorch/blob/master/android/pytorch_android_torchvision/src/main/java/org/pytorch/torchvision/TensorImageUtils.java#L90) method to convert `android.media.Image` in `YUV420` format to input tensor.
147+
148+
After getting predicted scores from the model it [finds top K classes](https://github.com/pytorch/android-demo-app/blob/master/PyTorchDemoApp/app/src/main/java/org/pytorch/demo/vision/ImageClassificationActivity.java#L153-L161) with the highest scores and shows on the UI.
149+
150+
## Building pytorch android from source
151+
152+
In some cases you might want to use a local build of pytorch android, for example you may build custom libtorch binary with another set of operators or to make local changes.
153+
154+
For this you can use `./scripts/build_pytorch_android.sh` script.
155+
```
156+
git clone https://github.com/pytorch/pytorch.git
157+
cd pytorch
158+
sh ./scripts/build_pytorch_android.sh
159+
```
160+
161+
Its workflow contains several steps:
162+
1. Builds libtorch for android for all 4 android abis (armeabi-v7a, arm64-v8a, x86, x86_64)
163+
2. Creates symbolic links to the results of those builds:
164+
`android/pytorch_android/src/main/jniLibs/${abi}` to the directory with output libraries
165+
`android/pytorch_android/src/main/cpp/libtorch_include/${abi}` to the directory with headers. These directories are used to build `libpytorch.so` library that will be loaded on android device.
166+
3. And finally runs `gradle` in `android/pytorch_android` directory with task `assembleRelease`
167+
168+
Script requires that android sdk, android ndk and gradle are installed.
169+
They are specified as environment variables:
170+
171+
`ANDROID_HOME` - path to [android sdk](https://developer.android.com/studio/command-line/sdkmanager.html)
172+
173+
`ANDROID_NDK` - path to [android ndk](https://developer.android.com/studio/projects/install-ndk)
174+
175+
`GRADLE_HOME` - path to [gradle](https://gradle.org/releases/)
176+
177+
178+
After successful build you should see the result as aar file:
179+
180+
```
181+
$ find pytorch_android/build/ -type f -name *aar
182+
pytorch_android/build/outputs/aar/pytorch_android.aar
183+
pytorch_android_torchvision/build/outputs/aar/pytorch_android.aar
184+
libs/fbjni_local/build/outputs/aar/pytorch_android_fbjni.aar
185+
```
186+
187+
It can be used directly in android projects, as a gradle dependency:
188+
```
189+
allprojects {
190+
repositories {
191+
flatDir {
192+
dirs 'libs'
193+
}
194+
}
195+
}
196+
197+
android {
198+
...
199+
packagingOptions {
200+
pickFirst "**/libfbjni.so"
201+
}
202+
...
203+
}
204+
205+
dependencies {
206+
implementation(name:'pytorch_android', ext:'aar')
207+
implementation(name:'pytorch_android_torchvision', ext:'aar')
208+
implementation(name:'pytorch_android_fbjni', ext:'aar')
209+
}
210+
```
211+
212+
At the moment for the case of using aar files directly we need additional configuration due to packaging specific (libfbjni.so is packaged in both pytorch_android_fbjni.aar and pytorch_android.aar).
213+
```
214+
packagingOptions {
215+
pickFirst "**/libfbjni.so"
216+
}
217+
```
218+
219+
## API Details
220+
221+
Main part of java api includes 3 classes:
222+
```
223+
org.pytorch.Module
224+
org.pytorch.IValue
225+
org.pytorch.Tensor
226+
```
227+
228+
If the reader is familiar with pytorch python api, we can think that org.pytorch.Tensor represents torch.tensor, org.pytorch.Module torch.Module<?>, while org.pytorch.IValue represents value of torchscript variable, supporting all its types. ( https://pytorch.org/docs/stable/jit.html#types )
229+
230+
### org.pytorch.Tensor (Tensor)
231+
[github](https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/Tensor.java)
232+
233+
Tensor supports dtypes `uint8, int8, float32, int32, float64, int64`.
234+
Tensor holds data in DirectByteBuffer of proper type with native bit order.
235+
236+
To create a Tensor user can use one of the factory methods:
237+
```
238+
Tensor newUInt8Tensor(long[] shape, ByteBuffer data)
239+
Tensor newUInt8Tensor(long[] shape, byte[] data)
240+
241+
Tensor newInt8Tensor(long[] shape, ByteBuffer data)
242+
Tensor newInt8Tensor(long[] shape, byte[] data)
243+
244+
Tensor newFloat32Tensor(long[] shape, FloatBuffer data)
245+
Tensor newFloat32Tensor(long[] shape, float[] data)
246+
247+
248+
Tensor newInt32Tensor(long[] shape, IntBuffer data)
249+
Tensor newInt32Tensor(long[] shape, int[] data)
250+
251+
Tensor newFloat64Tensor(long[] shape, DoubleBuffer data)
252+
Tensor newFloat64Tensor(long[] shape, double[] data)
253+
254+
255+
Tensor newInt64Tensor(long[] shape, LongBuffer data)
256+
Tensor newInt64Tensor(long[] shape, long[] data)
257+
```
258+
Where the first parameter `long[] shape` is shape of the Tensor as array of longs.
259+
260+
Content of the Tensor can be provided either as (a) java array or (b) as java.nio.DirectByteBuffer of proper type with native bit order.
261+
262+
In case of (a) proper DirectByteBuffer will be created internally. (b) case has an advantage that user can keep the reference to DirectByteBuffer and change its content in future for the next run, avoiding allocation of DirectByteBuffer for repeated runs.
263+
264+
Java’s primitive type byte is signed and java does not have unsigned 8 bit type. For dtype=uint8 api uses byte that will be reinterpretted as uint8 on native side. On java side unsigned value of byte can be read as (byte & 0xFF).
265+
266+
#### Tensor content layout
267+
268+
Tensor content is represented as a one dimensional array (buffer),
269+
where the first element has all zero indexes T\[0, ... 0\].
270+
271+
Lets assume tensor shape is {d<sub>0</sub>, ... d<sub>n-1</sub>} and d<sub>n-1</sub> > 0.
272+
273+
The second element will be T\[0, ... 1\] and the last one T\[d<sub>0</sub>-1, ... d<sub>n-1</sub> - 1\]
274+
275+
Tensor has methods to check its dtype:
276+
```
277+
int dtype()
278+
```
279+
That returns one of the dtype codes:
280+
```
281+
Tensor.DTYPE_UINT8
282+
Tensor.DTYPE_INT8
283+
Tensor.DTYPE_INT32
284+
Tensor.DTYPE_FLOAT32
285+
Tensor.DTYPE_INT64
286+
Tensor.DTYPE_FLOAT64
287+
```
288+
289+
The data of Tensor can be read as java array:
290+
```
291+
byte[] getDataAsUnsignedByteArray()
292+
byte[] getDataAsByteArray()
293+
int[] getDataAsIntArray()
294+
long[] getDataAsLongArray()
295+
float[] getDataAsFloatArray()
296+
double[] getDataAsDoubleArray()
297+
```
298+
These methods throw IllegalStateException if called for inappropriate dtype.
299+
300+
### org.pytorch.IValue (IValue)
301+
[github](https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/IValue.java)
302+
303+
IValue represents a torchscript variable that can be one of the supported (by torchscript) types ( https://pytorch.org/docs/stable/jit.html#types ). IValue is a tagged union. For every supported type it has a factory method, method to check the type and a getter method to retrieve a value.
304+
Getters throw IllegalStateException if called for inappropriate type.
305+
306+
### org.pytorch.Module (Module)
307+
[github](https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/Module.java)
308+
309+
Module is a wrapper of torch.jit.ScriptModule (`torch::jit::script::Module` in pytorch c++ api) which can be constructed with factory method load providing absolute path to the file with serialized torchscript.
310+
```
311+
IValue IValue.runMethod(String methodName, IValue... inputs)
312+
```
313+
for running a particular method of the script module.
314+
```
315+
IValue IValue.forward(IValue... inputs)
316+
```
317+
Shortcut to run 'forward' method.
318+
319+
```
320+
IValue IValue.destroy()
321+
```
322+
Explicitly destructs native (C++) part of the Module, `torch::jit::script::Module`.
323+
324+
As fbjni library destructs native part automatically when current `org.pytorch.Module` instance will be collected by Java GC, the instance will not leak if this method is not called, but timing of deletion and the thread will be at the whim of the Java GC. If you want to control the thread and timing of the destructor, you should call this method explicitly.
19325

20-
<script page-id="android" src="{{ site.baseurl }}/assets/menu-tab-selection.js"></script>

0 commit comments

Comments
 (0)