The Weight Is Over - Interactive Diffusion on Consumer GPUs

Frieder Ganz; Maximilian Müller

cs.LGSep 18, 2026

The Weight Is Over - Interactive Diffusion on Consumer GPUs

Authors: Frieder Ganz, Maximilian Müller

Organizations: Adobe Hamburg, Germany · NVIDIA Würselen, Germany

Abstract

On-device inference is booming, but the momentum is almost all in language models. Diffusion pipelines are memory hungry, latency-sensitive, and require orchestrating an embedder, a transformer, a decoder, and often further postprocessing that is not as standardized as LLM inference loops are. We navigate the trade-off between performance, quality, and model footprint to reach as many client devices in the wild as possible. We make three contributions: an embedding translator that maps a small text encoder into a large encoder space to cut weight and latency; a reproducible sweep recipe for navigating the speed/quality/memory triangle in diffusion pipelines; and an interactive on-device image generation editor achieving sub-second TTFI on recent GPUs.

Figures & tables

Figure 1. FLUX.2-klein with its 4B text encoder replaced by Qwen3-0.6B plus a 187M-parameter translator (trained post-hoc; transformer and VAE frozen). Prompts: a horned owl at golden hour; an erupting volcano at night; Jupiter; a sumi-e ink-wash raccoon. Four generated images: an owl, an erupting volcano, the planet Jupiter, and an ink-wash anthropomorphic raccoon.

Figure 2. Same prompt and seed across encoders (columns): the 4B reference, and Qwen3-0.6B with a 187M, 29M, and 3.5M translator. The 187M translator tracks the 4B; the 29M drifts to plausible but wrong content; and the 3.5M collapses toward a single dominant style regardless of the prompt. Rows: a red fox in a snowy forest; a bowl of ramen; a lighthouse on a cliff at sunset. All images are $1024\times 1024$ (1K).

Translator	Params	Mem	Encode	LPIPS $\downarrow$	CLIP $\uparrow$
4B (reference)	4000 M	8.0 GB	435 ms	0.553 †	0.328
d3072 (MLP)	187 M	1.77 GB	107 ms	0.514	0.306
d2048 $\times$ 8	158 M	1.72 GB	105 ms	0.550	0.299
d2048 $\times$ 7	98 M	1.60 GB	99 ms	0.564	0.295
d2048 wide	91 M	1.58 GB	98 ms	0.580	0.298
d2048	91 M	1.58 GB	98 ms	0.578	0.295

Table 1. Translator ablation on FLUX.2-klein (Qwen3-0.6B + translator vs. the 4B encoder). Mem = encoder-path bf16 weights (shared 0.6B base

{\approx}1.4

GB + translator, params

\times

2 B); the 4B row is the 4B encoder alone. Encode = text-encode latency (0.6B + translator), PyTorch/MPS, bf16. LPIPS/CLIPScore: end-to-end on 24 PartiPrompts vs. 4B at matched seed. † seed-variance floor (4B at two seeds). Attention rows timed on CPU (MPS SDPA fallback).

Prec.	Step	Weights	Act.	CLIP $\uparrow$	LPIPS $\downarrow$
BF16	105 ms	7,9 GiB	510 MiB	ref.
FP16	108 ms	7,9 GiB	510 MiB	$0.99\pm 0.01$	$0.02\pm 0.02$
FP8	73 ms	4,3 GiB	462 MiB	$0.96\pm 0.05$	$0.10\pm 0.07$
NVFP4	57 ms	2,8 GiB	475 MiB	$0.92\pm 0.08$	$0.23\pm 0.09$

Table 2. Transformer precision on RTX PRO 6000 Blackwell,

1024\times 1024

, single step, 4B encoder. Weights/Act. = weight/peak-activation memory. Quality vs. BF16 on 24 PartiPrompts: CLIP image similarity and LPIPS ( Zhang et al., 2018 ) (mean

\pm

std).

Figure 3. Cross-device sweep (4 steps,

1024^{2}

): peak VRAM vs. per-step latency. Label = resident-weight budget (100% = full); line = one (encoder, precision) trajectory; filled = 0.6B+translator, open = 4B. RTX 4070 Ti (12 GB): above the

{\approx}11.2

GB budget the denoiser pages, latency jumps

{\sim}40\times

(shaded). RTX PRO 6000 (96 GB): all resident—full-resident is fastest and precision drives the front. BF16 exact, FP16

{\geq}0.99

vs BF16.

Encoder	Prec.	Encode	Step	Decode	TTFI	Total	VRAM
4B	BF16	12 ms	105 ms	41 ms	158 ms	0.47 s	15.1 GB
0.6B+tr	BF16	4 ms	105 ms	40 ms	149 ms	0.47 s	10.6 GB
0.6B+tr	FP8	4 ms	73 ms	41 ms	119 ms	0.34 s	7.2 GB
0.6B+tr	NVFP4	4 ms	57 ms	40 ms	102 ms	0.27 s	5.7 GB

Table 3. On-device timings, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, TensorRT-RTX,

1024\times 1024

, 4 steps. TTFI = encode + one denoising step + VAE decode. VRAM = peak device memory across the full pipeline.

Diffusion-Based Image Generation

Diffusion-based image generation creates images by progressively denoising noise, with controls for content, style, and structure.

See all 330 papers

BlazeEdit: Generalist Image Editing on Mobile Devices with Image-to-Image Diffusion Models
May 27, 2026Fei Deng, Yanwu Xu, Zhipeng Bao +4Diffusion Model Acceleration Diffusion-Based Image Editing
Google
DreamLite: A Lightweight On-Device Unified Model for Image Generation and Editing
Mar 30, 2026Kailai Feng, Yuxiang Wei, Bo Chen +5T2I Diffusion Models Diffusion-Based Image Editing
Intelligent Creation Lab, ByteDance
EdgeDiT: Hardware-Aware Diffusion Transformers for Efficient On-Device Image Generation
Mar 30, 2026Sravanth Kodavanti, Manjunath Arveti, Sowmya Vajrala +2Diffusion Transformer Diffusion-Based Image Generation
Samsung Research Institute Bangalore, India

May 27, 2026J/K move · Enter open · S save

BlazeEdit: Generalist Image Editing on Mobile Devices with Image-to-Image Diffusion Models

Fei Deng, Yanwu Xu, Zhipeng Bao, Zhixing Zhang+3

Google

Diffusion Model Acceleration Diffusion-Based Image Editing Image Editing Diffusion Models On-Device Inference

Open paper AskPDF arXiv

The remarkable generation quality of modern diffusion models often comes at the cost of massive parameter counts, which necessitate server-side inference with significant computational costs and potential privacy risks. Consequently, there is growing momentum toward developing efficient on-device alternatives. While recent efforts have optimized text-to-image models for mobile hardware, they remain relatively bulky, typically ranging from 0.5B to 1B parameters. We present BlazeEdit, a highly efficient, generalist image-to-image diffusion model tailored for on-device deployment. By identifying that many practical image editing tasks do not require text-based guidance, we eliminate the text-conditioning components and develop a multi-task architecture that consolidates object removal, outpainting, tone correction, relighting, and sticker generation into a single, compact model of only 195M parameters. BlazeEdit achieves a substantial reduction in download size and memory overhead while maintaining competitive generation quality. It completes a full inference pass in just 290ms on a Pixel 10, delivering a seamless, privacy-preserving, and lightning-fast experience for generalist image editing on the edge.

Abstract

Diffusion-Based Image Generation

Explore similar work

BlazeEdit: Generalist Image Editing on Mobile Devices with Image-to-Image Diffusion Models

DreamLite: A Lightweight On-Device Unified Model for Image Generation and Editing

EdgeDiT: Hardware-Aware Diffusion Transformers for Efficient On-Device Image Generation