How Stable Diffusion Runs Faster in JAX / Flax: A Practical Look

Introduction

Stable Diffusion has quickly become a standout in AI-generated imagery, transforming text into high-quality visuals. Initially linked with PyTorch, there’s now a significant shift towards running it with JAX and Flax. This isn’t just a trend—it’s about leveraging JAX’s speed, scalability, and design to enhance research and deployment.

Why Choose JAX/Flax for Diffusion Models?

JAX, a high-performance numerical computing library from Google, excels in machine learning research with features like automatic differentiation, XLA compilation, and native support for GPUs and TPUs. Flax complements JAX by providing a minimal, flexible neural network library.

The Power of JAX in Diffusion Models

Diffusion models, like Stable Diffusion, thrive on repetitive operations—adding noise and reversing it to create images. JAX handles this efficiently through parallelism and function compilation, aided by tools like pmap and vmap for seamless parallel execution across devices. This is crucial for large-scale training or batch image generation.

In JAX, the functional style keeps model logic clean. Unlike PyTorch’s class-based models with hidden states, JAX treats models as pure functions with parameters and states passed explicitly, enhancing predictability and reproducibility—key strengths for scaling experiments or comparing results.

Handling Complex Components

Training large diffusion models involves managing attention mechanisms, U-Net backbones, and language model embeddings. JAX supports these with precise tensor and randomness control through its explicit RNG keys.

Porting Stable Diffusion to JAX: The Process

Porting from PyTorch to JAX is complex; it’s not a simple line-by-line conversion. It requires a shift to JAX’s functional approach—using nested dictionaries for model parameters instead of state_dicts and class objects.

Rebuilding Core Components

The U-Net model, crucial for denoising, adapts well to Flax, though attention layers and residual connections must be carefully reconstructed to preserve original behavior. Minor differences in numerical operations or initialization can affect image outputs.

Ensuring Consistency

The text encoder, utilizing models like CLIP, needs re-implementation or adaptation with compatible Flax models from resources like Hugging Face’s model hub to maintain quality. The noise scheduling process—adding and reversing noise—requires precise implementation to avoid output degradation.

Performance and Practical Benefits

JAX’s standout feature is performance, particularly with TPUs or large-scale experiments. While initial compilation is time-consuming, the resulting speed of compiled functions is often superior to PyTorch’s dynamic execution, benefitting both training and inference.

Clarity and Scalability

JAX enhances code clarity by explicitly handling randomness, model parameters, and states, minimizing hidden side effects and bolstering reproducibility—vital for collaborative or lengthy projects. Functions like pjit and xmap enable operations across multiple devices, facilitating higher-resolution images or longer-generation chains without bottlenecks.

Memory efficiency is another advantage. JAX’s static graph compilation avoids PyTorch’s dynamic overhead, supporting larger batches or more detailed images during training and inference.

Community and Future Outlook

PyTorch remains dominant, but JAX is gaining traction in research, supported by community libraries and tools like Hugging Face’s Transformers and Flax that bridge the ecosystems.

Towards a JAX Future

While many resources start in PyTorch, JAX users are increasingly supported by Flax-based checkpoints and scripts, easing the adaptation process. JAX’s functional approach offers cleaner models and better debugging, vital for building or fine-tuning Stable Diffusion.

Hybrid setups, using PyTorch for components like text encoding and JAX for others like the denoising U-Net, are becoming common, leveraging the strengths of both tools.

Conclusion

Stable Diffusion in JAX and Flax provides a faster, more scalable alternative to traditional PyTorch setups. While the ecosystem continues to grow, JAX already stands out for researchers and developers focused on performance-sensitive or TPU-based projects. With expanding community support and improved tooling, JAX is well-equipped to handle advanced image generation tasks efficiently.

For further exploration, consider visiting Hugging Face’s library for additional resources and community support on JAX and Flax implementations.