You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: unit2/README.md
+10-16
Original file line number
Diff line number
Diff line change
@@ -8,46 +8,40 @@ Here are the steps for this unit:
8
8
9
9
- Make sure you've [signed up for this course](https://huggingface.us17.list-manage.com/subscribe?u=7f57e683fa28b51bfc493d048&id=ef963b4162) so that you can be notified when new material is released
10
10
- Read through the material below for an overview of the key ideas of this unit
11
-
- Check out the _**FINE TUNING AND GUIDANCE NOTEBOOK TODO LINK**_ to fine-tune an existing diffusion model on a new dataset using the 🤗 Diffusers library
12
-
- Read through the **CONDITIONING NOTEBOOK** to see how we can add additional control to the generation process.
11
+
- Check out the _**Fine-Tuning and Guidance**_ notebook below to fine-tune an existing diffusion model on a new dataset using the 🤗 Diffusers library
13
12
- Create your own custom pipeline and share it as a Gradio demo
14
13
15
14
:loudspeaker: Don't forget to join the [Discord](https://huggingface.co/join/discord), where you can discuss the material and share what you've made in the `#diffusion-models-class` channel.
16
15
17
16
## Fine-Tuning
18
17
19
-
As you may have seen in Unit 1, training diffusion models from scratch can be time-consuming! Especially as we push to higher resolutions, the time and data required to train a model from scratch can become impractical. Fortunately, there is a solution: begin with a model that has already been trained! This way we start from a model that has already learnt to denoise images of some kind, and the hope is that this provides a better starting point than beginning from a randomly initialized model.
18
+
As you saw in Unit 1, training diffusion models from scratch can be time-consuming! Especially as we push to higher resolutions, the time and data required to train a model from scratch can become impractical. Fortunately, there is a solution: begin with a model that has already been trained! This way we start from a model that has already learnt to denoise images of some kind, and the hope is that this provides a better starting point than beginning from a randomly initialized model.
20
19
21
20
22
21
23
-
Fine-tuning typically works best if the new data somewhat resembles the base model's original training data (for example, beginning with a model trained on faces is probably a good idea if you're trying to generate cartoon faces) but suprisinggly the benefits persist even if the domain is changed quite drastically. The image above is generated from a [model trained on the LSUN Bedrooms dataset]([todo link](https://huggingface.co/google/ddpm-bedroom-256)) and fine-tuned for 500 steps on [the WikiArt dataset](https://huggingface.co/datasets/huggan/wikiart). The training script is included for reference alongside the notebooks for this unit.
22
+
Fine-tuning typically works best if the new data somewhat resembles the base model's original training data (for example, beginning with a model trained on faces is probably a good idea if you're trying to generate cartoon faces) but suprisinggly the benefits persist even if the domain is changed quite drastically. The image above is generated from a [model](https://huggingface.co/johnowhitaker/sd-class-wikiart-from-bedrooms) trained on the LSUN Bedrooms dataset](https://huggingface.co/google/ddpm-bedroom-256) and fine-tuned for 500 steps on [the WikiArt dataset](https://huggingface.co/datasets/huggan/wikiart). The training script is included for reference alongside the notebooks for this unit.
24
23
25
24
## Guidance
26
25
27
-
Unconditional models don't give much control over what is generated. We can train a conditional model (more on that in the next section) that takes additional inputs to help steer the generation process, but what if we already have a trained unconditional model we'd like to use? Enter guidance, a process by which the model predictions at each step in the generation process are evaluated against some guidance function and modified such that the final generated image is more to our liking.
26
+
Unconditional models don't give much control over what is generated. We can train a conditional model (more on that in the next section) that takes additional inputs to help steer the generation process, but what if we already have a trained unconditional model we'd like to use? Enter **_guidance_**, a process by which the model predictions at each step in the generation process are evaluated against some guidance function and modified such that the final generated image is more to our liking.
28
27
29
28
This guidance function can be almost anything, making this a powerful technique! In the notebook we build up from a simple example to one utilizing a powerful pre-trained model called CLIP which lets us guide generation based on a text description.
30
29
31
-
## Conditioning
32
-
33
-
Guidance is a great way to get some additional mileage from an unconditional diffusion model, but if we have additional information (such as a class label or an image caption) available during training then we can also feed this to the model for it to use as it makes its predictions. In doing so, we create a **conditional** model, which we can control at inference time by controlling what is fed in as conditioning. The notebook shows an example of a class-conditioned model which learns to generate images according to a class label. TODO note about timestep conditioning?
34
-
35
30
## Hands-On Notebook
36
31
37
32
At this point, you know enough to get started with the accompanying notebooks!
| Fine-Tuning and Guidance |[](https://colab.research.google.com/github/huggingface/diffusion-models-class/blob/main/unit1/01_finetuning_and_guidance.ipynb)|[](https://kaggle.com/kernels/welcome?src=https://github.com/huggingface/diffusion-models-class/blob/main/unit1/01_finetuning_and_guidance.ipynb)|[](https://console.paperspace.com/github/huggingface/diffusion-models-class/blob/main/unit1/01_finetuning_and_guidance.ipynb)|[](https://studiolab.sagemaker.aws/import/github/huggingface/diffusion-models-class/blob/main/unit1/01_finetuning_and_guidance.ipynb)|
37
+
40
38
41
39
## Project Time
42
40
43
-
Create a custom pipeline using some or all of the ideas covered in this unit and share it with the community
41
+
The best way to learn is by doing! Fine-tune your own diffusion model on a new image dataset and create a Gradio demo that can generate images from it. Don't forget to share your work with us on Discord, Twitter or elsewhere!
44
42
45
43
## Some Additional Resources
46
44
47
-
GLIDE (text conditioned diffusion model) TODO link
0 commit comments