Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation

Brian Chao; Lior Yariv; Howard Xiao; Gordon Wetzstein

cs.CVMar 24, 2026

Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation

Authors: Brian Chao, Lior Yariv, Howard Xiao, Gordon Wetzstein

Organizations: Stanford University, USA

Abstract

Diffusion and flow matching models have unlocked unprecedented capabilities for creative content creation, such as interactive image and streaming video generation. The growing demand for higher resolutions, frame rates, and context lengths, however, makes efficient generation increasingly challenging, as computational complexity grows quadratically with the number of generated tokens. Our work seeks to optimize the efficiency of the generation process in settings where the user's gaze location is known or can be estimated, for example, by using eye tracking. In these settings, we leverage the eccentricity-dependent acuity of human vision: while a user perceives very high-resolution visual information in a small region around their gaze location (the foveal region), the ability to resolve detail quickly degrades in the periphery of the visual field. Our approach starts with a mask modeling the foveated resolution to allocate tokens non-uniformly, assigning higher token density to foveal regions and lower density to peripheral regions. An image or video is generated in a mixed-resolution token setting, yielding results perceptually indistinguishable from full-resolution generation, while drastically reducing the token count and generation time. To this end, we develop a principled mechanism for constructing mixed-resolution tokens directly from high-resolution data, allowing a foveated diffusion model to be post-trained from an existing base model while maintaining content consistency across resolutions. We validate our approach through extensive analysis and a carefully designed user study, demonstrating the efficacy of foveation as a practical and scalable axis for efficient generation.

Figures & tables

Figure 1: Foveated Diffusion. (a) Given user-specified masks and text prompts as input, our method generates foveated content using fewer tokens than full high-resolution generation, resulting in faster inference while maintaining comparable perceptual quality. (b, c) Foveated Diffusion is well suited for tasks where salient regions require high-resolution synthesis, while peripheral regions can be generated at a lower resolution.

Figure 2: The Foveated Diffusion Pipeline. In Foveated Generation (a), we iteratively denoise a foveated token sequence of reduced length instead of the full high-resolution sequence. The resulting tokens $z_{0}^{\mathrm{fov}}$ are split into high- and low-resolution grids, decoded by the VAE, and blended using a user-specified foveation mask. We employ Foveated Training (b) to adapt pretrained DiTs to foveated token sequences using low-rank adaptation (LoRA) [ 31 ] . The image and its downsampled version are independently encoded by the VAE encoder and merged into a clean foveated token sequence for flow-matching training.

Figure 3: Adapting RoPE for mixed-resolution attention [ 88 ] .

Figure 4: Failure of Naïve mixed-resolution denoising.

Figure 5: Qualitative comparison for image generation. Our method yields perceptually indistinguishable results from full high-resolution synthesis, whereas the naïve baseline introduces scale inconsistencies and structural artifacts across mixed-resolution regions. The high-resolution regions (fovea) are delineated with white borders.

Figure 6: Foveated visual generation speedup.

Method	Token Ratio	HPSv2.1 $\uparrow$	FID $\downarrow$	Precision $\uparrow$	CLIP $\uparrow$	Runtime $\downarrow$ (Speedup $\uparrow$ )
Full high-res	100 $\%$	0.280	11.38	0.792	0.292	10.45s
Naïve mixed-res	43 $\%$	0.268	10.99	0.769	0.292	6.53s (1.61 $\times$ )
Ours	43 $\%$	0.279	11.38	0.777	0.294	6.53s (1.61 $\times$ )

Naïve mixed-res	30 $\%$	0.270	11.70	0.762	0.292	5.27s (1.98 $\times$ )
Ours	30 $\%$	0.279	11.91	0.789	0.293	5.27s (1.98 $\times$ )

Table 1: Quantitative comparison for image generation. We compare against the naïve mixed-resolution baseline across various token count ratios, highlighting the best result for each. Foveated Diffusion surpasses the baseline, excluding FID which we find less reliable for our task. Our method matches the image quality of full high-resolution generation (top row) while achieving up to a

2\times

speedup.

Figure 7: User study results.

Method	Subject Consistency $\uparrow$	Background Consistency $\uparrow$	Motion Smoothness $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Image Quality $\uparrow$
Full high-res	0.9407	0.9363	0.9943	0.263	0.5434	0.653
Naïve mixed-res	0.9072	0.9239	0.9899	0.465	0.4795	0.522
Ours	0.9446	0.9393	0.9946	0.265	0.5432	0.587

Table 2: Quantitative comparison for video generation (VBench). Foveated Diffusion outperforms the naïve mixed-resolution baseline across key metrics, achieving performance comparable to full-resolution generation. Notably, our framework maintains high subject consistency while providing a

3.5\times

speedup.

Figure 8: Qualitative comparison for video generation. Foveated Diffusion outperforms the naïve mixed-resolution baseline in video generation, which exhibits scale mismatches or duplicate entities near the low–high resolution boundary (white outline).

Figure 9: Image generation with different foveation patterns. We generate images using the same prompt and noise seed while varying the foveation pattern in shape, position, and size. High-resolution regions are delineated with white borders. Our method maintains content consistency across resolution regions, while the naïve mixed-resolution baseline exhibits inconsistencies of scale and structure.

Figure 10: Towards saliency-guided visual generation. We show that the saliency-guided Foveated Diffusion models enable saliency-guided image (a-e) and video generation (f), where salient objects align with the fovea. The randomized masks model do not exhibit such behavior (a). This is potentially useful for generative simulation applications such as VR gaming or robotics simulations (b-e), where only the salient objects have to be generated at the highest resolution, i.e. the machine gun in (b), the robotics arm and box in (c), and the dog in (f).

Figure 12: User study interface. Each trial began with a red gaze fixation dot displayed for 5 seconds ( a ), followed by a pair of images, each displayed for 1 s ( b ). Participants answered visual quality questions after both images were shown. Participants completed two practice trials ( c ) before beginning 60 data collection trials ( d ).

Pair	Preference $P$ (%)	$p$ -value ( $H_{0}:P=0.5$ )
Ours vs. Full high-res	47.3	0.4829
Ours vs. Naïve mixed-res	87.4	$<0.0001$
Full high-res vs. Naïve mixed-res	90.8	$<0.0001$

Table 3: User study statistical analysis. Preference rate for the first-listed method in each pair (higher = more preferred).

p

-values are from a two-sided binomial test under

H_{0}:P=0.5

p>0.05

indicates failure to reject

H_{0}

and implies perceptual indistinguishability.

Figure 13: Extended baseline comparisons. Foveated Diffusion (ours) produces perceptually similar quality to full high-resolution generation, whereas the naïve mixed-resolution baseline exhibits severe scale mismatches and structural inconsistencies across resolutions. All images are uncompressed in this figure.

Figure 14: Extended baseline comparisons. Foveated Diffusion (ours) produces perceptually similar quality to full high-resolution generation, whereas the naïve mixed-resolution baseline exhibits severe scale mismatches and structural inconsistencies across resolutions.

Figure 15: Extended baseline comparisons. Foveated Diffusion (ours) produces perceptually similar quality to full high-resolution generation, whereas the naïve mixed-resolution baseline exhibits severe scale mismatches and structural inconsistencies across resolutions.

Figure 16: Extended baseline comparisons. Foveated Diffusion (ours) produces perceptually similar quality to full high-resolution generation, whereas the naïve mixed-resolution baseline exhibits severe scale mismatches and structural inconsistencies across resolutions.

Figure 17: Varying foveation mask radius. Foveated Diffusion generates coherent content across varying foveation mask radii. Given the same prompt and noise seed, the generated images remain consistent and adhere to the prompt.

Figure 18: Varying foveation mask position. Foveated Diffusion generates coherent content across varying foveation mask positions. Given the same prompt and noise seed, the generated images remain consistent and adhere to the prompt.

Figure 19: Foveation mask containing multiple disjoint regions. Foveated Diffusion generates coherent content with foveation masks containing multiple disjoint regions. Given the same prompt and noise seed, the generated images remain consistent and adhere to the prompt.

Figure 20: Single-object saliency-guided generation. Foveated Diffusion with saliency-guided training enables coarse, controllable single-object generation, where the salient object is approximately aligned with the center of the foveation mask.

Figure 21: Multi-object saliency-guided generation. Foveated Diffusion with saliency-guided training also enables coarse, controllable multi-object generation, where salient objects approximately align with the centers of the disjoint regions in the foveation mask.

Figure 22: Saliency-guided generation for immersive gaming. Foveated Diffusion is well suited for immersive first-person generative gaming applications, where salient objects can be generated near the gaze-tracked location (fovea) and rendered in high resolution, while the remaining regions are rendered at lower resolution.

Figure 23: Saliency-guided generation for robotics simulation. Foveated Diffusion is well suited for generative robotics simulation, where foveated imagery can be used for robotics policy learning. In this setting, only robot arms and manipulated objects are generated at high resolution, while the background is rendered at lower resolution to provide global context.

Figure 24: Saliency-guided generation for autonomous vehicles. Foveated Diffusion is also well suited for generative autonomous driving simulation, where foveated imagery can be used for policy learning in self-driving systems. In this setting, only important objects in the scene (e.g., pedestrians, other vehicles, roadblocks) are generated at high resolution, while the background is rendered at lower resolution to provide global context.

Figure 25: Single-object bounding-box-guided generation. Foveated Diffusion with bounding-box-guided training enables coarse, controllable single-object generation, where the salient object is approximately constrained within the foveation mask.

Figure 26: Multi-object bounding-box-guided generation. Foveated Diffusion with bounding-box-guided training enables coarse, controllable multi-object generation, where salient objects are approximately constrained within the disjoint regions of the foveation mask.

Figure 27: Bounding-box-guided generation and saliency-guided generation. We compare bounding-box-guided and saliency-guided generation. Because bounding-box-derived masks precisely delineate object contours, the bounding-box-guided model generates entire objects within the high-resolution region. In contrast, the saliency-guided model aligns only the most salient parts of objects with the center of the high-resolution region.

Figure 28: Foveated Diffusion artifacts. We delineate the foveation border with a white circular outline. The red dashed lines indicate regions with blending artifacts.

Adaptive Tokenization

See all 11 papers

Level-of-Token Diffusion
Oct 5, 2026Kiyohiro Nakayama, Brian Chao, Jan Ackermann +5Video Diffusion Models Adaptive Tokenization
Stanford University · Google
Contextual Flow Matching: Adaptive Step Selection in Flow Models for Efficient Visual Generation
Oct 2, 2026Divya Jyoti Bajpai, Arun Verma, Manjesh Kumar HanawalEfficient Inference Flow Matching
Department of IEOR IIT Bombay, Mumbai, India · SMART Centre Singapore
MOSAIK: Multi-Patch Content-Aware Spatial Allocation of Image Tokens for Efficient Generation
Aug 5, 2026Mohammadreza Hami, Mohammadreza Samadi, Chao Gao +1Pixel-Space Diffusion Models Diffusion Model Inference Acceleration
Huawei Technologies Canada

Oct 5, 2026J/K move · Enter open · S save

Level-of-Token Diffusion

Kiyohiro Nakayama, Brian Chao, Jan Ackermann, Hansheng Chen+4

Stanford University · Google

Video Diffusion Models Adaptive Tokenization Diffusion Transformer Diffusion-Based Image Generation Diffusion Model Inference Acceleration Visual Token Compression

Open paper AskPDF arXiv

Image and video diffusion models allocate equal computation to every region, even when the intended scene calls for varying levels of detail. The spatial distribution of detail can often be anticipated before generation, indicating where computation can be reduced. We introduce Level-of-Token (LoT) Diffusion, a framework that turns this knowledge into an explicit multiresolution token layout (Level-of-Token layout) for adaptive and efficient generation. Tokens represent rectangular patches of varying sizes and shapes, allocating finer tokens where detail is needed and coarser tokens elsewhere. We adapt pretrained diffusion transformers to LoT layouts through a patch-wise asymmetric flow parametrization and embeddings for multiresolution tokens, preserving full-resolution flow prediction at every denoising step while processing only a reduced token sequence. LoT Diffusion enables layout-adaptive generation while preserving pretrained generative priors. We demonstrate LoT with layouts derived from semantic masks, bounding boxes, texture variance, and depth-of-field cues, as well as agentic plans. Across image and video generation, LoT offers favorable quality-efficiency tradeoffs, with significant speedups determined by the layout's token budget. Our project website is at https://georgenakayama.github.io/lotdiffusion/.

Abstract

Adaptive Tokenization

Explore similar work

Level-of-Token Diffusion

Contextual Flow Matching: Adaptive Step Selection in Flow Models for Efficient Visual Generation

MOSAIK: Multi-Patch Content-Aware Spatial Allocation of Image Tokens for Efficient Generation