Skip to content

Commit d3a504d

Browse files
authored
Update README and screenshot (huggingface#34)
* Update README and screenshot.
1 parent 3b75e75 commit d3a504d

File tree

2 files changed

+20
-17
lines changed

2 files changed

+20
-17
lines changed

README.md

+20-17
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,41 @@
11
# Swift Core ML Diffusers 🧨
22

3-
This is a simple app that shows how to integrate Apple's [Core ML Stable Diffusion implementation](https://github.com/apple/ml-stable-diffusion) in a native Swift UI application. The Core ML port is a simplification of the Stable Diffusion implementation from the [diffusers library](https://github.com/huggingface/diffusers). This application can be used for faster iteration, or as sample code for any use cases.
3+
This is a native app that shows how to integrate Apple's [Core ML Stable Diffusion implementation](https://github.com/apple/ml-stable-diffusion) in a native Swift UI application. The Core ML port is a simplification of the Stable Diffusion implementation from the [diffusers library](https://github.com/huggingface/diffusers). This application can be used for faster iteration, or as sample code for any use cases.
44

55
This is what the app looks like on macOS:
66
![App Screenshot](screenshot.jpg)
77

8-
On first launch, the application downloads a zipped archive with a Core ML version of Runway's Stable Diffusion v1.5, from [this location in the Hugging Face Hub](https://huggingface.co/pcuenq/coreml-stable-diffusion/tree/main). This process takes a while, as several GB of data have to be downloaded and unarchived.
8+
On first launch, the application downloads a zipped archive with a Core ML version of Stability AI's Stable Diffusion v2 base, from [this location in the Hugging Face Hub](https://huggingface.co/pcuenq/coreml-stable-diffusion-2-base/tree/main). This process takes a while, as several GB of data have to be downloaded and unarchived.
99

10-
For faster inference, we use a very fast scheduler: [DPM-Solver++](https://github.com/LuChengTHU/dpm-solver) that we ported to Swift. Since this scheduler is still not available in Apple's GitHub repository, the application depends on the following fork instead: https://github.com/pcuenca/ml-stable-diffusion. Our Swift port is based on [Diffusers' DPMSolverMultistepScheduler](https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py), with a number of simplifications.
10+
For faster inference, we use a very fast scheduler: [DPM-Solver++](https://github.com/LuChengTHU/dpm-solver), that we ported to Swift from our [diffusers DPMSolverMultistepScheduler implementation](https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py).
1111

12-
## Compatibility
12+
## Compatibility and Performance
1313

1414
- macOS Ventura 13.1, iOS/iPadOS 16.2, Xcode 14.2.
1515
- Performance (after the initial generation, which is slower)
16-
* ~8s in macOS on MacBook Pro M1 Max (64 GB). Model: Stable Diffusion v2-base, ORIGINAL attention implementation, CPU + GPU.
16+
* ~8s in macOS on MacBook Pro M1 Max (64 GB). Model: Stable Diffusion v2-base, ORIGINAL attention implementation, running on CPU + GPU.
1717
* 23 ~ 30s on iPhone 13 Pro. Model: Stable Diffusion v2-base, SPLIT_EINSUM attention, CPU + Neural Engine, memory reduction enabled.
1818

19-
Performance on iPhone is somewhat erratic, sometimes it's ~20x slower and the phone heats up. This happens because the model could not be scheduled to run on the Neural Engine and everything happens in the CPU. We have not been able to determine the reasons for this problem. If you observe the same, here are some recommendations:
20-
- Detach from Xcode
21-
- Kill apps you are not using.
22-
- Let the iPhone cool down before repeating the test.
23-
- Reboot your device.
19+
See [this post](https://huggingface.co/blog/fast-mac-diffusers) and [this issue](https://github.com/huggingface/swift-coreml-diffusers/issues/31) for additional performance figures.
20+
21+
The application will try to guess the best hardware to run models on. You can override this setting using the `Advanced` section in the controls sidebar.
22+
23+
## How to Run
24+
25+
The easiest way to test the app on macOS is by [downloading it from the Mac App Store](https://apps.apple.com/app/diffusers/id1666309574).
2426

2527
## How to Build
2628

27-
If you clone or fork this repo, please update `common.xcconfig` with your development team identifier. Code signing is required to run on iOS, but it's currently disabled for macOS.
29+
You need [Xcode](https://developer.apple.com/xcode/) to build the app. When you clone the repo, please update `common.xcconfig` with your development team identifier. Code signing is required to run on iOS, but it's currently disabled for macOS.
2830

29-
## Limitations
31+
## Known Issues
3032

31-
- A handful of models are currently supported.
32-
- The Core ML compute units have been hardcoded to CPU and GPU on macOS, and to CPU + Neural Engine on iOS/iPadOS.
33+
Performance on iPhone is somewhat erratic, sometimes it's ~20x slower and the phone heats up. This happens because the model could not be scheduled to run on the Neural Engine and everything happens in the CPU. We have not been able to determine the reasons for this problem. If you observe the same, here are some recommendations:
34+
- Detach from Xcode
35+
- Kill apps you are not using.
36+
- Let the iPhone cool down before repeating the test.
37+
- Reboot your device.
3338

3439
## Next Steps
3540

36-
- Allow users to select compute units to verify the combination that achieves the best performance on their hardware.
37-
- Implement other schedulers, additional options.
38-
- Experiment with smaller distilled models.
41+
- Allow additional models to be downloaded from the Hub.

screenshot.jpg

38.1 KB
Loading

0 commit comments

Comments
 (0)