Add Custom Build tutorial for iOS and Android (#325)

Author: Jessica Lin

_mobile/android.md

@@ -34,10 +34,10 @@ example = torch.rand(1, 3, 224, 224)
 traced_script_module = torch.jit.trace(model, example)
 traced_script_module.save("app/src/main/assets/model.pt")
 ```
-If everything works well, we should have our model - `model.pt` generated in the assets folder of android application. 
+If everything works well, we should have our model - `model.pt` generated in the assets folder of android application.
 That will be packaged inside android application as `asset` and can be used on the device.
 
-More details about TorchScript you can find in [tutorials on pytorch.org](https://pytorch.org/docs/stable/jit.html) 
+More details about TorchScript you can find in [tutorials on pytorch.org](https://pytorch.org/docs/stable/jit.html)
 
 #### 2. Cloning from github
 ```
@@ -67,14 +67,14 @@ dependencies {
 }
 ```
 Where `org.pytorch:pytorch_android` is the main dependency with PyTorch Android API, including libtorch native library for all 4 android abis (armeabi-v7a, arm64-v8a, x86, x86_64).
-Further in this doc you can find how to rebuild it only for specific list of android abis. 
+Further in this doc you can find how to rebuild it only for specific list of android abis.
 
 `org.pytorch:pytorch_android_torchvision` - additional library with utility functions for converting `android.media.Image` and `android.graphics.Bitmap` to tensors.
 
 #### 4. Reading image from Android Asset
 
 All the logic happens in [`org.pytorch.helloworld.MainActivity`](https://github.com/pytorch/android-demo-app/blob/master/HelloWorldApp/app/src/main/java/org/pytorch/helloworld/MainActivity.java#L31-L69).
-As a first step we read `image.jpg` to `android.graphics.Bitmap` using the standard Android API. 
+As a first step we read `image.jpg` to `android.graphics.Bitmap` using the standard Android API.
 ```
 Bitmap bitmap = BitmapFactory.decodeStream(getAssets().open("image.jpg"));
 ```
@@ -93,13 +93,13 @@ Tensor inputTensor = TensorImageUtils.bitmapToFloat32Tensor(bitmap,
 `org.pytorch.torchvision.TensorImageUtils` is part of `org.pytorch:pytorch_android_torchvision` library.
 The `TensorImageUtils#bitmapToFloat32Tensor` method creates tensors in the [torchvision format](https://pytorch.org/docs/stable/torchvision/models.html) using `android.graphics.Bitmap` as a source.
 
-> All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. 
+> All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224.
 > The images have to be loaded in to a range of `[0, 1]` and then normalized using `mean = [0.485, 0.456, 0.406]` and `std = [0.229, 0.224, 0.225]`
 
-`inputTensor`'s shape is `1x3xHxW`, where `H` and `W` are bitmap height and width appropriately. 
+`inputTensor`'s shape is `1x3xHxW`, where `H` and `W` are bitmap height and width appropriately.
 
 #### 7. Run Inference
- 
+
 ```
 Tensor outputTensor = module.forward(IValue.from(inputTensor)).toTensor();
 float[] scores = outputTensor.getDataAsFloatArray();
@@ -109,7 +109,7 @@ float[] scores = outputTensor.getDataAsFloatArray();
 
 #### 8. Processing results
 Its content is retrieved using `org.pytorch.Tensor.getDataAsFloatArray()` method that returns java array of floats with scores for every image net class.
- 
+
 After that we just find index with maximum score and retrieve predicted class name from `ImageNetClasses.IMAGENET_CLASSES` array that contains all ImageNet classes.
 
 ```
@@ -123,8 +123,8 @@ for (int i = 0; i < scores.length; i++) {
 }
 String className = ImageNetClasses.IMAGENET_CLASSES[maxScoreIdx];
 ```
- 
-In the following sections you can find detailed explanations of PyTorch Android API, code walk through for a bigger [demo application](https://github.com/pytorch/android-demo-app/tree/master/PyTorchDemoApp), 
+
+In the following sections you can find detailed explanations of PyTorch Android API, code walk through for a bigger [demo application](https://github.com/pytorch/android-demo-app/tree/master/PyTorchDemoApp),
 implementation details of the API, how to customize and build it from source.
 
 ## PyTorch Demo Application
@@ -169,7 +169,7 @@ After getting predicted scores from the model it finds top K classes with the hi
 #### Language Processing Example
 
 Another example is natural language processing, based on an LSTM model, trained on a reddit comments dataset.
-The logic happens in [`TextClassificattionActivity`](https://github.com/pytorch/android-demo-app/blob/master/PyTorchDemoApp/app/src/main/java/org/pytorch/demo/nlp/TextClassificationActivity.java). 
+The logic happens in [`TextClassificattionActivity`](https://github.com/pytorch/android-demo-app/blob/master/PyTorchDemoApp/app/src/main/java/org/pytorch/demo/nlp/TextClassificationActivity.java).
 
 Result class names are packaged inside the TorchScript model and initialized just after initial module initialization.
 The module has a `get_classes` method that returns `List[str]`, which can be called using method `Module.runMethod(methodName)`:
@@ -199,7 +199,7 @@ Running inference of the model is similar to previous examples:
 Tensor outputTensor = mModule.forward(IValue.from(inputTensor)).toTensor()
 ```
 
-After that, the code processes the output, finding classes with the highest scores. 
+After that, the code processes the output, finding classes with the highest scores.
 
 ## Building PyTorch Android from Source
 
@@ -219,7 +219,7 @@ The workflow contains several steps:
 2\. Create symbolic links to the results of those builds:
 `android/pytorch_android/src/main/jniLibs/${abi}` to the directory with output libraries
 `android/pytorch_android/src/main/cpp/libtorch_include/${abi}` to the directory with headers. These directories are used to build `libpytorch.so` library that will be loaded on android device.
- 
+
 3\. And finally run `gradle` in `android/pytorch_android` directory with task `assembleRelease`
 
 Script requires that Android SDK, Android NDK and gradle are installed.
@@ -263,6 +263,9 @@ dependencies {
     implementation(name:'pytorch_android', ext:'aar')
     implementation(name:'pytorch_android_torchvision', ext:'aar')
     implementation(name:'pytorch_android_fbjni', ext:'aar')
+    ...
+    implementation 'com.android.support:appcompat-v7:28.0.0'
+    implementation 'com.facebook.soloader:nativeloader:0.8.0'
 }
 ```
 
@@ -273,7 +276,35 @@ packagingOptions {
 }
 ```
 
-## More Details
+Also we have to add all transitive dependencies of our aars. As `pytorch_android` depends on `com.android.support:appcompat-v7:28.0.0` and `com.facebook.soloader:nativeloader:0.8.0`, we need to add them. (In case of using maven dependencies they are added automatically from `pom.xml`).
+
+
+## Custom Build
+
+To reduce the size of binaries you can do custom build of PyTorch Android with only set of operators required by your model.
+This includes two steps: preparing the list of operators from your model, rebuilding pytorch android with specified list.
+
+1\. Preparation of the list of operators
+
+List of operators of your serialized torchscript model can be prepared in yaml format using python api function `torch.jit.export_opnames()`:
+```
+import torch, yaml
+m = torch.jit.load("example.pt")
+ops = torch.jit.export_opnames(m)
+f = open('test.yaml', 'w')
+yaml.dump(ops, f)
+```
+2\. Building PyTorch Android with prepared operators list.
+
+To build PyTorch Android with the prepared yaml list of operators, specify it in the environment variable `SELECTED_OP_LIST`. Also in the arguments, specify which Android ABIs it should build; by default it builds all 4 Android ABIs.
+
+```
+SELECTED_OP_LIST=test.yaml sh scripts/build_pytorch_android.sh x86
+```
+
+After successful build you can integrate the result aar files to your android gradle project, following the steps from previous section of this tutorial (Building PyTorch Android from Source).
+
+## API Docs
 
 You can find more details about the PyTorch Android API in the [Javadoc](https://pytorch.org/docs/stable/packages.html).
 

_mobile/ios.md

@@ -1,4 +1,4 @@
----
+\---
 layout: mobile
 title: iOS
 permalink: /mobile/ios/
@@ -10,17 +10,17 @@ published: true
 
 # iOS
 
-To get started with PyTorch on iOS, we recommend exploring the following [HelloWorld](https://github.com/pytorch/ios-demo-app/tree/master/HelloWorld). 
+To get started with PyTorch on iOS, we recommend exploring the following [HelloWorld](https://github.com/pytorch/ios-demo-app/tree/master/HelloWorld).
 
 ## Quickstart with a Hello World Example
 
 HelloWorld is a simple image classification application that demonstrates how to use PyTorch C++ libraries on iOS. The code is written in Swift and uses Objective-C as a bridge.
 
 ### Model Preparation
 
-Let's start with model preparation. If you are familiar with PyTorch, you probably should already know how to train and save your model. In case you don't, we are going to use a pre-trained image classification model - [Resnet18](https://pytorch.org/hub/pytorch_vision_resnet/), which is already packaged in [TorchVision](https://pytorch.org/docs/stable/torchvision/index.html). To install it, run the command below.
+Let's start with model preparation. If you are familiar with PyTorch, you probably should already know how to train and save your model. In case you don't, we are going to use a pre-trained image classification model - [MobileNet v2](https://pytorch.org/hub/pytorch_vision_mobilenet_v2/), which is already packaged in [TorchVision](https://pytorch.org/docs/stable/torchvision/index.html). To install it, run the command below.
 
-> We highly recommend following the [Pytorch Github page](https://github.com/pytorch/pytorch) to set up the Python development environment on your local machine. 
+> We highly recommend following the [Pytorch Github page](https://github.com/pytorch/pytorch) to set up the Python development environment on your local machine.
 
 ```shell
 pip install torchvision
@@ -34,7 +34,7 @@ python trace_model.py
 
 If everything works well, we should have our model - `model.pt` generated in the `HelloWorld` folder. Now copy the model file to our application folder `HelloWorld/model`.
 
-> To find out more details about TorchScript, please visit [tutorials on pytorch.org](https://pytorch.org/tutorials/advanced/cpp_export.html) 
+> To find out more details about TorchScript, please visit [tutorials on pytorch.org](https://pytorch.org/tutorials/advanced/cpp_export.html)
 
 ### Install LibTorch via Cocoapods
 
@@ -94,7 +94,7 @@ private lazy var module: TorchModule = {
     }
 }()
 ```
-Note that the `TorchModule` Class is an Objective-C wrapper of `torch::jit::script::Module`. 
+Note that the `TorchModule` Class is an Objective-C wrapper of `torch::jit::script::Module`.
 
 ```cpp
 torch::jit::script::Module module = torch::jit::load(filePath.UTF8String);
@@ -103,7 +103,7 @@ Since Swift can not talk to C++ directly, we have to either use an Objective-C c
 
 #### Run Inference
 
-Now it's time to run inference and get the results. 
+Now it's time to run inference and get the results.
 
 ```swift
 guard let outputs = module.predict(image: UnsafeMutableRawPointer(&pixelBuffer)) else {
@@ -115,17 +115,16 @@ Again, the `predict` method is just an Objective-C wrapper. Under the hood, it c
 ```cpp
 at::Tensor tensor = torch::from_blob(imageBuffer, {1, 3, 224, 224}, at::kFloat);
 torch::autograd::AutoGradMode guard(false);
-at::AutoNonVariableTypeMode non_var_type_mode(true);
 auto outputTensor = _impl.forward({tensor}).toTensor();
 float* floatBuffer = outputTensor.data_ptr<float>();
 ```
-The C++ function `torch::from_blob` will create an input tensor from the pixel buffer. Note that the shape of the tensor is `{1,3,224,224}` which represents `NxCxWxH` as we discussed in the above section. 
+The C++ function `torch::from_blob` will create an input tensor from the pixel buffer. Note that the shape of the tensor is `{1,3,224,224}` which represents `NxCxWxH` as we discussed in the above section.
 
 ```cpp
 torch::autograd::AutoGradMode guard(false);
 at::AutoNonVariableTypeMode non_var_type_mode(true);
 ```
-The above two lines tells the PyTorch engine to do inference only. This is because by default, PyTorch has built-in support for doing auto-differentiation, which is also known as [autograd](https://pytorch.org/docs/stable/notes/autograd.html). Since we don't do training on mobile, we can just disable the autograd mode. 
+The above two lines tells the PyTorch engine to do inference only. This is because by default, PyTorch has built-in support for doing auto-differentiation, which is also known as [autograd](https://pytorch.org/docs/stable/notes/autograd.html). Since we don't do training on mobile, we can just disable the autograd mode.
 
 Finally, we can call this `forward` function to get the output tensor and convert it to a `float` buffer.
 
@@ -183,16 +182,47 @@ After the build succeeds, all static libraries and header files will be generate
 
 Open your project in XCode, copy all the static libraries as well as header files to your project. Navigate to the project settings, set the value **Header Search Paths** to the path of header files you just copied.
 
-In the build settings, search for **other linker flags**.  Add a custom linker flag below 
+In the build settings, search for **other linker flags**.  Add a custom linker flag below
 
 ```
 -force_load $(PROJECT_DIR)/${path-to-libtorch.a}
 ```
  Finally, disable bitcode for your target by selecting the Build Settings, searching for **Enable Bitcode**, and set the value to **No**.
 
-## API Docs
+### API Docs
+
+Currently, the iOS framework uses the Pytorch C++ front-end APIs directly. The C++ document can be found [here](https://pytorch.org/cppdocs/). To learn more about it, we recommend exploring the [C++ front-end tutorials](https://pytorch.org/tutorials/advanced/cpp_frontend.html) on PyTorch webpage. In the meantime, we're working on providing the Swift/Objective-C API wrappers to PyTorch.
+
+
+### Custom Build
 
-Currently, the iOS framework uses the Pytorch C++ front-end APIs directly. The C++ document can be found here https://pytorch.org/cppdocs/. To learn more about it, we recommend exploring the [C++ front-end tutorials](https://pytorch.org/tutorials/advanced/cpp_frontend.html) on PyTorch webpage. In the meantime, we're working on providing the Swift/Objective-C API wrappers to PyTorch.
+Starting from 1.4.0, PyTorch supports custom build. You can now build the PyTorch library that only contains the operators needed by your model. To do that, follow the steps below
+
+1\. Verify your PyTorch version is 1.4.0 or above. You can do that by checking the value of `torch.__version__`.
+
+2\. To dump the operators in your model, run the following lines of Python code:
+
+```python
+import torch, yaml
+model = torch.jit.load("example.pt")
+ops = torch.jit.export_opnames(m)
+f = open('example.yaml', 'w')
+yaml.dump(ops, f)
+```
+In the snippet above, you first need to load the ScriptModule. Then, use `export_opnames` to return a list of operator names of the ScriptModule and its submodules. Lastly, save the result in a yaml file.
+
+3\. To run the iOS build script locally with the prepared yaml list of operators, pass in the yaml file generate from the last step into the environment variable `SELECTED_OP_LIST`. Also in the arguments, specify `BUILD_PYTORCH_MOBILE=1` as well as the platform/architechture type. Take the arm64 build for example, the command should be:
+
+```
+SELECTED_OP_LIST=example.yaml BUILD_PYTORCH_MOBILE=1 IOS_ARCH=arm64 ./scripts/build_ios.sh
+```
+4\. After the build succeeds, you can integrate the result libraries to your project by following the [XCode Setup](#xcode-setup) section above.
+
+5\. The last step is to add a single line of C++ code before running `forward`. This is because by default JIT will do some optimizations on operators (fusion for example), which might break the consistency with the ops we dumped from the model.
+
+```cpp
+torch::jit::GraphOptimizerEnabledGuard guard(false);
+```
 
 ## Issues and Contribution