Local models loader #51

ghost · 2023-06-16T21:03:17Z

Removed Path swift package, replaced with FileManager/URL calls.
Reworked ModelInfo to define "built-in" models
Added new ViewModel for tracking the folder where models are stored
Loads models from the models folder creating ModelInfo entries for them.

let deviceSupportsQuantization: Bool = { if #available(macOS 14, *) { return true } else { return false } }() Help the compiler determine the type in this code.

Update to allow the user to put their own downloaded CoreML capable models into the models folder.

ghost · 2023-06-17T07:46:47Z

please do not accept this yet. more work needs to be done on the model / compute units loading.

More work on this branch to better handle built-in model downloading. Added a feature to replace the "Generate" button with a "Download" button when the selected built-in model/compute units combination is not yet downloaded. When compute units change the list of models refreshes to a filtered list of models supporting that variant.

ghost · 2023-06-18T18:30:10Z

This now includes the latest changes to main AND the additional handling of selected model/compute unit cases. The latest commit comment: More work on this branch to better handle built-in model downloading. Added a feature to replace the "Generate" button with a "Download" button when the selected built-in model/compute units combination is not yet downloaded. When compute units change the list of models refreshes to a filtered list of models supporting that variant.

Added Import Model in File menu and Import Model as a "+" button next to the model selection picker. The user can select a zip file or a folder which contains the files merges.txt and vocab.json.

pcuenca

Thanks a lot for working on this, I think it's a very useful addition!

So far I have taken a broad look at the design and asked some questions in the relevant parts of the code. Before diving deeper, I think we need to clarify the scope of the changes to ModelInfo and move from there. In the proposed design, we'd need to duplicate model entries for all variants of a single model, whereas in the previous one the original and split_einsum were part of the same entity. How are you planning to deal with this?

Another general comment is that, at a first glance, the file observation code looks complex to me. I'd be curious to know if you considered a different solution that could potentially be easier (or maybe not, I haven't explored it). It could be like this:

Show a file browser so the user can navigate to a folder with a model they downloaded from the Hub.
Validate whether all necessary files and resources exist in that place.
If they do, copy the contents to the models folder.
If they don't, display an error message. This way we prevent corrupted entries from appearing in the UI.

How do you feel about that approach vs the file observation one?

I briefly tested the code and saw a couple of things:

Crashes on launch when running on iPhone, as expected.
I was able to select a corrupted folder (empty circle in the list), the "Generate" button turned to "Download". When I clicked it, state changed to "Downloading..." but it never progressed or failed.
I'd maybe remove the folder icon next to the list, I feel it's a bit too intrusive.

As stated, I think this is very useful and am looking forward to help you iterate on it! Please, let me know your thoughts and we'll work from there :)

pcuenca · 2023-06-22T09:57:19Z

Common/Downloader.swift

@@ -8,7 +8,6 @@

 import Foundation
 import Combine
-import Path


It's cool to remove dependencies, but it'd been better to have this change be proposed in a different PR for easier review :)

I think we can remove this PR and I'll move the discrete items into their own separate PRs as I should have to start with.

The reasons I felt compelled to remove Path were

where possible eliminate external packages

inconsistent usage. There are places inside the current code where URL performs a similar task yet only a few locations (2? 3?) Path was used. Converting the usage of Path to URL was trivial.

make the code more approachable by increasing consistent API usage and removing a third party API which the next coder would have to get their head around. It's not really a massive API there, but you get the point, right?

pcuenca · 2023-06-22T09:58:11Z

Common/ModelInfo.swift

Thanks for taking the time to reorganize the file hierarchy! However, this makes review of the PR much more difficult because the diff doesn't show the changes done to ModelInfo :(

Thank you for helping me understand how to best post PRs. This was my first and I've already learned a lot from the attempt. I'll separate and publish more concise discrete PRs as follow up.

pcuenca · 2023-06-22T10:01:21Z

Common/ModelInfo.swift

+    /// Which attention variant is this model associated with?
+    let variant: AttentionVariant


The reason the variant was not part of the model abstraction is because ModelInfo instances are presented in the UI as single entities. If you select a different attention variant, it's still the same model. If we go this way, then we'd have duplicate entries in the model selector (one for original, another one for split_einsum) for each model; or we'd need to complicate the UI logic.

(Arguably, the palettized versions could have been handled in exactly the same way (a single model entry would have several variants such as original, original palettized, split_einsum, split_einsum_v2 palettized, ...), but I separated them because the palettized versions are two new and different, and they only work in beta versions of the OS).

I am not a user of the hugging face API and probably have some misunderstandings on how the model and variant combinations work. The reason for separating the ModelInfo into a discrete instance for each variant was due to my user experience.

If I wanted to download a third party model from HF via the web I would go to
https://huggingface.co/coreml/coreml-GTA5_Artwork_Diffusion
There I would find in Files and Versions a split_einsum and an original model. After downloading and extracting these two matched models I would have one folder "gta5ArtworkDiffusion-v1_ema-vae_original_compiled" and another folder "GTA5_Artwork_Diffusion_split-einsum_compiled."

When I deconstruct these two folder names I get one with a missing version and the two mismatching on their name due to a spacing issue (and a caps issue, which is minor as they should be compared in a case insensitive manner.)

How can I programatically look through a models folder in an unknown state that over time has had lots of slightly incorrectly named models landed into it?

Since I found a few of these that would result in multiple models anyway I started down the path of treating each folder as its own ModelInfo unit.

Further the reason for the file monitor is that whenever the folder contents change there is a notification which updates the UI. These two are connected. Let me show you how and maybe then you can update my understanding...

Let's assume a file list of:
elldrethsVividMix-v2-VividEr_original
analogDream3D_1_ema_vae-split_einsum_compiled
coreml-stable-diffusion-1-4_original_compiled
coreml-stable-diffusion-2-1-base_split_einsum_compiled
coreml-stable-diffusion-v1-5_original_compiled
Deliberate_v2_original
deliberate_v2_split-einsum

With CPU and GPU on a Mac you get a file list of
CoreML Stable Diffusion v1.4
CoreML Stable Diffusion v1.5
Deliberate
elderethsVi [note: no idea why this one truncates!]

When I switch to All compute units on Mac I get
CoreML Stable Diffusion 2 1 base Deliberate analogDream3d
Because a filtered list of model info objects are made that align their variant support. Therefore when the UI compute units are updated and that affects which variants should be used the UI refreshes and the list of available models updates.

Does this attempt at filtering the available selections make sense or is it a misunderstanding of the use case for changing the compute units manually?

I see, it's unfortunate that we have these naming inconsistencies.

My initial assumption was that models were always available in the two variants, so the list was independent of the compute units selected. However, if you select a model variant that had not been downloaded yet, the download was triggered.

In your approach the compute units act as a filter. It's not a bad solution for this use case, where we no longer have the guarantee that all variants exist or share the same name. There could be some slightly confusing situations, say you run the app with Deliberate_v2_original selected (and deliberate_v2_split-einsum not downloaded), then select CPU and Neural Engine: the Deliberate model will disappear from the list. I think this can be disconcerting for first-time users that don't really know the details of how everything works. Not sure how to deal with it though.

The long term ideal solution is to have an integrated HF API browser. If the app displays and downloads assets directly then the original approach trying to access a missing file from the hub triggering a known download link makes sense.

However, I found that there are models listed in GitHub which are not submitted to the HF website so I would still suspect the user might download a completely unknown folder with a completely unknown naming structure (or rename it themselves for some reason, or maybe work with their own test models.)

Therefore I think it's best solve for an unknown state local model folder. Even after later work results in a more elegant user experience with a HF browser UI it'll still be useful for the intermediate or advanced users.

The instance where Deliberate_v2_original is found and deliberate_v2_split_einsum is missing would persist and we still wouldn't know how to find the matching file to help the user. The best solution I could envision at the moment is to add a hint in the UI. Instead of the Disclosure section being named "Hub Models" we rename it to reflect the selected filter as it changes "Models (original)" "Models (einsum)" or something to alert the intermediate/advanced user which file they're missing on their system.

pcuenca · 2023-06-22T10:02:30Z

Common/ModelInfo.swift

+    }
+}
+
+struct ModelInfo: Hashable, Identifiable {


Why do we need Hashable and Identifiable?

In the current code the Picker for the selected model uses the model's version, a String. SwiftUI List() does not like a String or something that can be a nil value as a selection type and can result in unknown behaviour. Along the way I was using the ModelInfo itself as the selected object, which required Hashable and Identifiable to be part of a list.
In the latest commit of this PR the selection is actually an integer representing the index of the filtered array of known model info so I think that this can be removed.

pcuenca · 2023-06-22T10:07:14Z

Common/ModelInfo.swift

+    static let v14Base = ModelInfo(modelId: "pcuenq/coreml-stable-diffusion-1-4",
+              variant: AttentionVariant.original,
+              builtin: true,
+              modelVersion: "v1.4",
+              humanReadableFileName: "CoreML Stable Diffusion v1.4",
+              fileSystemFileName: "coreml-stable-diffusion-1-4")


Since we added variant here, we no longer have a version of v14Base that supports split_einsum. The same happens for all other models. If we duplicate them, then they would appear as different entries in the UI, which I believe would be too crowded.

This is hinting at my misalignment on fundamental terms.

coreml-stable-diffusion-1-4_split_einsum_compiled
coreml-stable-diffusion-1-4_original_compiled

In my mind these are the two v14 model files - one for original and one for split_einsum.

pcuenca · 2023-06-22T10:09:31Z

Common/ModelsViewModel.swift

I don't think this observation mechanism will work on iOS. Perhaps we have to make it Mac only?

Well caught, I think it makes sense to leave it in for Mac so that the models folder list auto refreshes when new models are installed in the finder, but remove it for iOS where the new "Import Model" feature to come in a follow on PR would be the primary import mechanism.

pcuenca · 2023-06-22T10:22:30Z

Diffusion-macOS/ControlsView.swift

+                } else if exists {
+                    filledCircle
+                } else {
+                    dottedCircle


what does this state represent? The model is not builtin and does not exist - is it an invalid folder?

In the second check we're not checking against builtin. If the model exists. Meaning if it's on the local filesystem. It could be either built-in or 3rd party all we care here is - you can continue because it's locally available.

In the third check we know it's not a built-in model that's missing and it's not present on the system. This is an unknown / broken state for sure. Rather than ignore I wanted to indicate "something's amiss" which would allow the user to manually intervene at least.

Ok. What do you think about using some other icon that looks more like an error, like a red × or something?

Agreed Also any edge cases where a built-in model is downloaded and corrupt should be identified and handled instead of simply shown as an error. They're some missing checks of state I think. I do think file system awareness should be moved into functions in the ModelInfo itself or into a Model ViewModel management object to centralise the logic on state checks, etc.

pcuenca · 2023-06-22T10:23:00Z

Diffusion-macOS/ControlsView.swift

+
+            return HStack {
+                if model.builtin && !exists {
+                    dl


Nice to separate "needs download" state from local states!

The first check is if model IS builtin and DOES NOT exist. Therefore in the current version it's the only time a download prompt would be offered. We indicate that a resource we're aware of but that you haven't downloaded yet is needed to continue.

pcuenca · 2023-06-22T10:26:47Z

Diffusion-macOS/ControlsView.swift

+                Button {
+                    NSWorkspace.shared.open(modelsViewModel.modelsFolderURL)
+                } label: {
+                    Image(systemName: "folder.fill")
+                }
+                .buttonStyle(.plain)
+


I'd rather keep the "disclose" action inside the model list; otherwise I think it's too prominent in the UI and takes a lot of space that could be used for the human description. We can also add a menu item and shortcut to disclose.

When I removed the "Reveal in Finder" action from the drop down list I was trying to get the ModelInfo object itself to be the selected object of the picker. Now that the selection is an integer we could probably put it back with a tag of -1 to trigger the reveal action (and then reset the picker back to the previous valid selection.)

XCode file order update. Created a new "Common" group at the top level and moved all files that are assigned to both the Diffusion and the Diffusion-macOS targets into the common folder. Extracted from pull 51 - #51

- PromptTextField additions brought into this PR - Preview images brought into this PR - applicationSupportURL brought into this PR - Array extension for safe index access added to utilities. - Converting selected model index to an optional and creating a No Model Selected option in the ControlsView picker fixes some startup misalignment between loaded last selected model versus UI state. - ControlsView has more UI content separated out into support functions. - ControlsView now tracks last selected model to restore selection after Reveal in Finder is selected.

Odd filename issue fixed post PromptTextField addition.

…t-coreml-diffusers-dol into local-models-loader

Fixing manually merged PRs.

a bit of cleanup on comments and spacing

manual merge of SDXL update changes into this PR.

comments update

spacing

dolmere added 2 commits June 15, 2023 21:22

Help the compiler!

ca42591

let deviceSupportsQuantization: Bool = { if #available(macOS 14, *) { return true } else { return false } }() Help the compiler determine the type in this code.

macOS load models folder contents

64bd11f

Update to allow the user to put their own downloaded CoreML capable models into the models folder.

ghost marked this pull request as draft June 17, 2023 07:45

dolmere added 2 commits June 18, 2023 20:26

Merge branch 'huggingface:main' into local-models-loader

df0aff0

ghost marked this pull request as ready for review June 18, 2023 18:30

Added Import Model

71568fb

Added Import Model in File menu and Import Model as a "+" button next to the model selection picker. The user can select a zip file or a folder which contains the files merges.txt and vocab.json.

pcuenca reviewed Jun 22, 2023

View reviewed changes

ghost marked this pull request as draft June 22, 2023 16:12

ghost mentioned this pull request Jun 22, 2023

Created Common group #55

Merged

ghost mentioned this pull request Jul 12, 2023

Model importer #61

Closed

dolmere added 12 commits July 19, 2023 09:59

Merge branch 'main' into local-models-loader

edac160

Filename fix

f6161f9

Odd filename issue fixed post PromptTextField addition.

Merge branch 'local-models-loader' of https://github.com/dolmere/swif…

afe4758

…t-coreml-diffusers-dol into local-models-loader

Merge branch 'main' into local-models-loader

a4cb619

Manual merge fix

5687823

Fixing manually merged PRs.

cleanup

7d9b419

a bit of cleanup on comments and spacing

DRAFT: merge isXL

b2a23f3

manual merge of SDXL update changes into this PR.

comments update

a233bb9

comments update

spacing

fa4fa29

spacing

Merge remote-tracking branch 'upstream/main' into local-models-loader

7911f0e

Merge remote-tracking branch 'upstream/main' into local-models-loader

d4b90e9

		/// Which attention variant is this model associated with?
		let variant: AttentionVariant

Local models loader #51

Local models loader #51

Uh oh!

Conversation

ghost commented Jun 16, 2023

Uh oh!

ghost commented Jun 17, 2023

Uh oh!

ghost commented Jun 18, 2023

Uh oh!

pcuenca left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ghost Jun 22, 2023 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ghost Jun 22, 2023 •

edited by ghost

Loading