CoreML stable diffusion image generation example

The example app for running text-to-image or image-to-image models to generate images using Apple's Core ML Stable Diffusion implementation

How to get generated image

1. Place at least one of your prepared split_einsum models into the ‘Local Models’ folder. Find the ‘Document’ folder through the interface by tapping on the ‘Local Models’ button. If the folder is empty, then create a folder named ‘models’. Refer to the folders’ hierarchy in the image below for guidance. The example app supports only split_einsum models. In terms of performance split_einsum is the fastest way to get result.
2. Pick up the model that was placed at the local folder from the list. Click update button if you added a model while app was launched
3. Enter a prompt or pick up a picture and press "Generate" (You don't need to prepare image size manually) It might take up to a minute or two to get the result

File Name	Description
TextEncoder.mlmodelc	Encodes input text into a vector space for further processing.
Unet.mlmodelc	Core model handling the transformation of encoded vectors into intermediate image representations.
UnetChunk1.mlmodelc	First segment of a segmented U-Net model for optimized processing in environments with memory constraints.
UnetChunk2.mlmodelc	Second segment of the segmented U-Net model, completing the tasks started by the first chunk.
VAEDecoder.mlmodelc	Decodes the latent representations into final image outputs.
VAEEncoder.mlmodelc	Compresses input image data into a latent space for reconstruction or further processing.
SafetyChecker.mlmodelc	Ensures generated content adheres to safety guidelines by checking against predefined criteria.
vocab.json	Contains the vocabulary used by the text encoder for tokenization and encoding processes.
merges.txt	Stores the merging rules for byte-pair encoding used in the text encoder.
ControlledUnet.mlmodelc	Modified U-Net model incorporating control mechanisms from the ControlNet to steer output characteristics.
ControlledUnetChunk1.mlmodelc	First part of the controlled U-Net model designed for sequential or partitioned processing.
ControlledUnetChunk2.mlmodelc	Second part of the controlled U-Net model completing the image synthesis process started by the first chunk.
controlnet/	Directory containing various ControlNet models for different control scenarios.
MultilingualTextEncoderProjection.mlmodelc	Specialized model for projecting multilingual text into a consistent vector space, supporting diverse language inputs.

Model set example

coreml-stable-diffusion-2-base

Performance

The speed can be unpredictable. Sometimes a model will suddenly run a lot slower than before. It appears as if Core ML is trying to be smart in how to schedule things, but doesn’t always optimal.

The package source

Deploying Transformers on the Apple Neural Engine Case study