Understanding Samplers: A Key Component in Unleashing the Power of Stable Diffusion
TLDR: 🎨Stable Diffusion samplers impact image generation speed & accuracy. 🔄 Try different samplers for varied results!
Stable Diffusion-based models have gained attention in recent years due to their ability to generate coherent images from random noise.
The process involves defining and solving differential equations that dictate noise removal. The choice of samplers in this process significantly impacts the tradeoff between speed and accuracy.
Illustration of the diffusion process. Source: NVIDIA Developer.
This blog post dives into the different samplers used in Stable Diffusion, their approaches, and the practical differences observed with each sampler.
What are the different samplers used?
The approaches and variations of different samplers play a crucial role in the stable diffusion process. Here are the different samplers and their approach to sampling:
Euler: This simple and fast sampler is a classic for solving ordinary differential equations (ODEs). It is closely related to Heun, which improves on Euler's accuracy but is half as fast due to additional calculations required.
Heun: A more accurate version of Euler, Heun is slower but provides better detail in the output.
LMS & PLMS: These methods are cousins to Euler and Heun, averaging out steps in the past to boost accuracy. PLMS is an adaptation of the classical LMS method, designed to handle neural network structures better.
DDIM: A neural network-based method, DDIM is fast per step but requires more steps for optimal results.
DPM2: Specifically designed for diffusion models, this method seeks to improve upon DDIM by requiring fewer steps for quality output. However, it is slower due to the need for denoising twice per step.
Ancestral Samplers: These samplers use corresponding algorithms (Euler-A & DPM2-A) in a different context, adding more noise per step and creating more chaotic outputs. DPM2-A is about half as fast as Euler-A.
Changing sampling, while keeping prompt, steps, scale, and seed consistent.
What are some of the differences?
Heun and Euler work well together, with Euler used for rapid iterations and Heun for better detail.
(P)LMS can suffer from artifacts, such as "rainbows" of noise, on lower step settings.
DPM-2s and Euler-A provide coherent outputs at low steps, while high-step non-ancestral samplers yield similar results across styles.
Non-ancestral models are easier to understand, with more steps generally improving results.
Euler-A tends to follow complex prompts better at higher steps, but may focus on less important parts of the prompt.
In practice, sampler choice can impact the quality and speed of image generation. Experimenting with different samplers can help shake up noise patterns and produce variations in the final output.
Additional resources
A good YouTube video covering sampling methods (e.g. Klms, Euler, DPM2, Ancestral): https://www.youtube.com/watch?v=wgVaeg_r2PQ
A research paper on the subject ("Elucidating the Design Space of Diffusion-Based Generative Models") is available here: https://arxiv.org/abs/2206.00364