Skip to content

update iOS.md #268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 9, 2019
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 42 additions & 22 deletions _mobile/ios.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,42 +10,45 @@ published: true

# iOS

To get started with PyTorch on iOS, we recommend exploring the following a HelloWorld example on Github [Hello Word](https://github.com/pytorch/ios-demo-app/tree/master/HelloWorld).
To get started with PyTorch on iOS, we recommend exploring the following [HelloWorld](https://github.com/pytorch/ios-demo-app/tree/master/HelloWorld).

## Quickstart with a HelloWorld example
## Quickstart with a Hello World example
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capitalize "Example"


HelloWorld is a simple image classification application that demonstrates how to use PyTorch C++ libraries on iOS. The code is written in Swift and uses an Objective-C class as a bridging header.

Before we jump into details, we highly recommend following the Pytorch Github page to set up the Python development environment on your local machine.
HelloWorld is a simple image classification application that demonstrates how to use PyTorch C++ libraries on iOS. The code is written in Swift and uses Objective-C as a bridge.

### Model preparation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capitalize "'Model Preparation"


Let's start with model preparation. If you are familiar with PyTorch, you probably should already know how to train and save your model. In case you don't, we are going to use a pre-trained image classification model(Resnet18), which is packaged in [TorchVision](https://pytorch.org/docs/stable/torchvision/index.html). To install TorchVision, run the command below.
Let's start with model preparation. If you are familiar with PyTorch, you probably should already know how to train and save your model. In case you don't, we are going to use a pre-trained image classification model - Resnet18, which is already packaged in [TorchVision](https://pytorch.org/docs/stable/torchvision/index.html). To install it, run the command below.

> Before running, we highly recommend following the [Pytorch Github page](https://github.com/pytorch/pytorch) to set up the Python development environment on your local machine.

```shell
pip install torchvision
```

Once we have TorchVision installed successfully, let's navigate to the HelloWorld folder and run a python script to generate our model. The `trace_model.py` contains the code of tracing and saving a [torchscript model](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html) that can be run on mobile devices. Run the command below to get our model
Once we have TorchVision installed successfully, let's navigate to the HelloWorld folder and run `trace_model.py`. The script contains the code of tracing and saving a [torchscript model](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html) that can be run on mobile devices.

```shell
python trace_model.py
```

If everything works well, we should have our model - `model.pt` generated in the same folder. Now copy the model file to our application folder `HelloWorld/model`.
If everything works well, we should have our model - `model.pt` generated in the `HelloWorld` folder. Now copy the model file to our application folder `HelloWorld/model`.

> To find out more details about TorchScript, please visit [tutorials on pytorch.org](https://pytorch.org/docs/stable/jit.html)

### Install PyTorch C++ libraries via Cocoapods

The PyTorch C++ library is available in [Cocoapods](https://cocoapods.org/), to integrate it to our project, we can run
The PyTorch C++ library is available in [Cocoapods](https://cocoapods.org/), to integrate it to our project, simply run

```ruby
pod install
```
Now it's time to open the `HelloWorld.xcworkspace` in XCode, select an iOS simulator and hit the build and run button (cmd + R). If everything works well, we should see a wolf picture on the simulator screen along with the prediction result.

Now it's time to open the `HelloWorld.xcworkspace` in XCode, select an iOS simulator and launch it (cmd + R). If everything works well, we should see a wolf picture on the simulator screen along with the prediction result.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion to add link to wolf picture.


### Code Walkthrough

In this part, we are going to walk through the code step by step. The `ViewController.swift` contains most of the code.
In this part, we are going to walk through the code step by step.

#### Image loading
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capitalize "Image Loading"


Expand All @@ -60,7 +63,7 @@ guard var pixelBuffer = resizedImage.normalized() else {
}
```

We first load an image from the bundle and resize it to 224x224, which is the size of the input tensor. Then we call this `normalized()` category method on UIImage to get normalized pixel data from the image. Let's take a look at the code below.
We first load the image from our bundle and resize it to 224x224. Then we call this `normalized()` category method to normalized the pixel buffer. Let's take a closer look at the code below.

```swift
var normalizedBuffer: [Float32] = [Float32](repeating: 0, count: w * h * 3)
Expand All @@ -72,11 +75,12 @@ for i in 0 ..< w * h {
normalizedBuffer[w * h * 2 + i] = (Float32(rawBytes[i * 4 + 2]) / 255.0 - 0.406) / 0.225 // B
}
```
The input data of our model is a 3-channel RGB image of shape (3 x H x W), where H and W are expected to be at least 224. The image have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225].

#### Init JIT interpreter
The code might look weird at first glance, but it’ll make sense once we understand our model. The input data of our model is a 3-channel RGB image of shape (3 x H x W), where H and W are expected to be at least 224. The image have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change have to has.
"The image have..." --> "The image ..."


Now that we have preprocessed our input data and we have a pre-trained TorchScript model, the next step is to use the model and the data to run the predication. To do that, we'll first load our model into the application.
#### TorchScript module
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capitalize "Module"


Now that we have preprocessed our input data and we have a pre-trained TorchScript model, the next step is to use them to run predication. To do that, we'll first load our model into the application.

```swift
private lazy var module: TorchModule = {
Expand All @@ -88,25 +92,43 @@ private lazy var module: TorchModule = {
}
}()
```
The TorchModule Class is an Objective-C wrapper for the C++ class `torch::jit::script::Module`.
Note that the `TorchModule` Class is an Objective-C wrapper of `torch::jit::script::Module`.

```cpp
torch::jit::script::Module module = torch::jit::load(filePath.UTF8String);
```
Since Swift can not talk to C++ directly, we have to either use an Objective-C class as a bride, or create a C wrapper for the C++ library. For the demo purpose, we're going to wrap everything in this Objective-C class, but we are working on bringing the Swift wrapper to PyTorch.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. "For the demo purpose, ..." --> "For demo purposes, ..."
  2. Spelling "bride" --> "bridge"


#### Run Inference

Now it's time to run the inference and get the result. We pass in the pixel buffer object as a raw pointer to the `predict` method and get the result from it.
Now it's time to run the inference and get the results.

```swift
guard let outputs = module.predict(image: UnsafeMutableRawPointer(&pixelBuffer)) else {
return
}
```
Again, the `predict` method on the `module` is an Objective-C method. Under the hood, it calls the C++ version of predict which is `forward`
Again, the `predict` method is just an Objective-C wrapper. Under the hood, it calls the C++ `forward` function. Let's take a look at how it's implemented.

```cpp
auto outputTensor = _impl.forward({inputTensor}).toTensor();
at::Tensor tensor = torch::from_blob(imageBuffer, {1, 3, 224, 224}, at::kFloat);
torch::autograd::AutoGradMode guard(false);
at::AutoNonVariableTypeMode non_var_type_mode(true);
auto outputTensor = _impl.forward({tensor}).toTensor();
void* tensorBuffer = outputTensor.storage().data();
```
The C++ function `torch::from_blob` will create an input tensor from the pixel buffer. Note that the shpae of the tensor is `{1,3,224,224}` which represents `NxCxWxH` as we discuessed in above section.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Spelling "shpae" --> "shape"
  2. Spelling "discuessed" --> "discussed"


```cpp
torch::autograd::AutoGradMode guard(false);
at::AutoNonVariableTypeMode non_var_type_mode(true);
```
The above two lines tells the PyTorch engine to do inference only. This is beacuse By default, PyTorch has built-in support for doing auto-differentiation, which is also known as autograd. Since we don't do training on mobile, we can just disable the autograd mode.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Spelling "becuse" --> "because"
  2. Uncapitalize "By"


Finally, we can call this `forward` function to get the output tensor as the results.

```cpp
auto outputTensor = _impl.forward({tensor}).toTensor();
```

### Collect results
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capitalize "Results"

Expand All @@ -120,9 +142,7 @@ let sortedResults = zippedResults.sorted { $0.1.floatValue > $1.1.floatValue }.p

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capitalize "PyTorch Demo App"

### PyTorch demo app

For more complex use cases, we recommend to check out the PyTorch demo application.

The demo app contains two showcases. A camera app that runs a quantized model to predict the images coming from device’s rear-facing camera in real time. And a text-based app that uses a self-trained NLP model to predict the topic from the input string.
For more complex use cases, we recommend to check out the PyTorch demo application. The demo app contains two showcases. A camera app that runs a quantized model to predict the images coming from device’s rear-facing camera in real time. And a text-based app that uses a text classififcation model to predict the topic from the input string.

## Build PyTorch iOS libraries from source

Expand Down