Diffusion and flow matching models have unlocked unprecedented capabilities for creative content creation, such as interactive image and streaming video generation. The growing demand for higher resolutions, frame rates, and context lengths, however, makes efficient generation increasingly challenging, as computational complexity grows quadratically with the number of generated tokens. Our work seeks to optimize the efficiency of the generation process in settings where the user's gaze location is known or can be estimated, for example, by using eye tracking. In these settings, we leverage the eccentricity-dependent acuity of human vision: while a user perceives very high-resolution visual information in a small region around their gaze location (the foveal region), the ability to resolve detail quickly degrades in the periphery of the visual field. Our approach starts with a mask modeling the foveated resolution to allocate tokens non-uniformly, assigning higher token density to foveal regions and lower density to peripheral regions. An image or video is generated in a mixed-resolution token setting, yielding results perceptually indistinguishable from full-resolution generation, while drastically reducing the token count and generation time. To this end, we develop a principled mechanism for constructing mixed-resolution tokens directly from high-resolution data, allowing a foveated diffusion model to be post-trained from an existing base model while maintaining content consistency across resolutions. We validate our approach through extensive analysis and a carefully designed user study, demonstrating the efficacy of foveation as a practical and scalable axis for efficient generation.
Figures & tables
Figure 1: Foveated Diffusion. (a) Given user-specified masks and text prompts as input, our method generates foveated content using fewer tokens than full high-resolution generation, resulting in faster inference while maintaining comparable perceptual quality. (b, c) Foveated Diffusion is well suited for tasks where salient regions require high-resolution synthesis, while peripheral regions can be generated at a lower resolution.
Figure 2: The Foveated Diffusion Pipeline. In Foveated Generation (a), we iteratively denoise a foveated token sequence of reduced length instead of the full high-resolution sequence. The resulting tokens z0fov are split into high- and low-resolution grids, decoded by the VAE, and blended using a user-specified foveation mask. We employ Foveated Training (b) to adapt pretrained DiTs to foveated token sequences using low-rank adaptation (LoRA) [ 31 ] . The image and its downsampled version are independently encoded by the VAE encoder and merged into a clean foveated token sequence for flow-matching training.
Figure 4: Failure of Naïve mixed-resolution denoising.
Figure 5: Qualitative comparison for image generation. Our method yields perceptually indistinguishable results from full high-resolution synthesis, whereas the naïve baseline introduces scale inconsistencies and structural artifacts across mixed-resolution regions. The high-resolution regions (fovea) are delineated with white borders.
Figure 6: Foveated visual generation speedup.
Method
Token Ratio
HPSv2.1 ↑
FID ↓
Precision ↑
CLIP ↑
Runtime ↓ (Speedup ↑ )
Full high-res
100 %
0.280
11.38
0.792
0.292
10.45s
Naïve mixed-res
43 %
0.268
10.99
0.769
0.292
6.53s (1.61 × )
Ours
0.279
11.38
0.777
0.294
Naïve mixed-res
30 %
0.270
11.70
0.762
0.292
5.27s (1.98 × )
Ours
0.279
11.91
0.789
0.293
Table 1: Quantitative comparison for image generation. We compare against the naïve mixed-resolution baseline across various token count ratios, highlighting the best result for each. Foveated Diffusion surpasses the baseline, excluding FID which we find less reliable for our task. Our method matches the image quality of full high-resolution generation (top row) while achieving up to a 2× speedup.
Figure 7: User study results.
Method
Subject Consistency ↑
Background Consistency ↑
Motion Smoothness ↑
Dynamic Degree ↑
Aesthetic Quality ↑
Image Quality ↑
Full high-res
0.9407
0.9363
0.9943
0.263
0.5434
0.653
Naïve mixed-res
0.9072
0.9239
0.9899
0.465
0.4795
0.522
Ours
0.9446
0.9393
0.9946
0.265
0.5432
0.587
Table 2: Quantitative comparison for video generation (VBench). Foveated Diffusion outperforms the naïve mixed-resolution baseline across key metrics, achieving performance comparable to full-resolution generation. Notably, our framework maintains high subject consistency while providing a 3.5× speedup.
Figure 8: Qualitative comparison for video generation. Foveated Diffusion outperforms the naïve mixed-resolution baseline in video generation, which exhibits scale mismatches or duplicate entities near the low–high resolution boundary (white outline).
Figure 9: Image generation with different foveation patterns. We generate images using the same prompt and noise seed while varying the foveation pattern in shape, position, and size. High-resolution regions are delineated with white borders. Our method maintains content consistency across resolution regions, while the naïve mixed-resolution baseline exhibits inconsistencies of scale and structure.
Figure 10: Towards saliency-guided visual generation. We show that the saliency-guided Foveated Diffusion models enable saliency-guided image (a-e) and video generation (f), where salient objects align with the fovea. The randomized masks model do not exhibit such behavior (a). This is potentially useful for generative simulation applications such as VR gaming or robotics simulations (b-e), where only the salient objects have to be generated at the highest resolution, i.e. the machine gun in (b), the robotics arm and box in (c), and the dog in (f).
Figure 11: User study setup.
Figure 12: User study interface. Each trial began with a red gaze fixation dot displayed for 5 seconds ( a ), followed by a pair of images, each displayed for 1 s ( b ). Participants answered visual quality questions after both images were shown. Participants completed two practice trials ( c ) before beginning 60 data collection trials ( d ).
Pair
Preference P (%)
p -value ( H0:P=0.5 )
Ours vs. Full high-res
47.3
0.4829
Ours vs. Naïve mixed-res
87.4
<0.0001
Full high-res vs. Naïve mixed-res
90.8
<0.0001
Table 3: User study statistical analysis. Preference rate for the first-listed method in each pair (higher = more preferred). p -values are from a two-sided binomial test under H0:P=0.5 . p>0.05 indicates failure to reject H0 and implies perceptual indistinguishability.
Figure 13: Extended baseline comparisons. Foveated Diffusion (ours) produces perceptually similar quality to full high-resolution generation, whereas the naïve mixed-resolution baseline exhibits severe scale mismatches and structural inconsistencies across resolutions. All images are uncompressed in this figure.
Figure 14: Extended baseline comparisons. Foveated Diffusion (ours) produces perceptually similar quality to full high-resolution generation, whereas the naïve mixed-resolution baseline exhibits severe scale mismatches and structural inconsistencies across resolutions.
Figure 15: Extended baseline comparisons. Foveated Diffusion (ours) produces perceptually similar quality to full high-resolution generation, whereas the naïve mixed-resolution baseline exhibits severe scale mismatches and structural inconsistencies across resolutions.
Figure 16: Extended baseline comparisons. Foveated Diffusion (ours) produces perceptually similar quality to full high-resolution generation, whereas the naïve mixed-resolution baseline exhibits severe scale mismatches and structural inconsistencies across resolutions.
Figure 17: Varying foveation mask radius. Foveated Diffusion generates coherent content across varying foveation mask radii. Given the same prompt and noise seed, the generated images remain consistent and adhere to the prompt.
Figure 18: Varying foveation mask position. Foveated Diffusion generates coherent content across varying foveation mask positions. Given the same prompt and noise seed, the generated images remain consistent and adhere to the prompt.
Figure 19: Foveation mask containing multiple disjoint regions. Foveated Diffusion generates coherent content with foveation masks containing multiple disjoint regions. Given the same prompt and noise seed, the generated images remain consistent and adhere to the prompt.
Figure 20: Single-object saliency-guided generation. Foveated Diffusion with saliency-guided training enables coarse, controllable single-object generation, where the salient object is approximately aligned with the center of the foveation mask.
Figure 21: Multi-object saliency-guided generation. Foveated Diffusion with saliency-guided training also enables coarse, controllable multi-object generation, where salient objects approximately align with the centers of the disjoint regions in the foveation mask.
Figure 22: Saliency-guided generation for immersive gaming. Foveated Diffusion is well suited for immersive first-person generative gaming applications, where salient objects can be generated near the gaze-tracked location (fovea) and rendered in high resolution, while the remaining regions are rendered at lower resolution.
Figure 23: Saliency-guided generation for robotics simulation. Foveated Diffusion is well suited for generative robotics simulation, where foveated imagery can be used for robotics policy learning. In this setting, only robot arms and manipulated objects are generated at high resolution, while the background is rendered at lower resolution to provide global context.
Figure 24: Saliency-guided generation for autonomous vehicles. Foveated Diffusion is also well suited for generative autonomous driving simulation, where foveated imagery can be used for policy learning in self-driving systems. In this setting, only important objects in the scene (e.g., pedestrians, other vehicles, roadblocks) are generated at high resolution, while the background is rendered at lower resolution to provide global context.
Figure 25: Single-object bounding-box-guided generation. Foveated Diffusion with bounding-box-guided training enables coarse, controllable single-object generation, where the salient object is approximately constrained within the foveation mask.
Figure 26: Multi-object bounding-box-guided generation. Foveated Diffusion with bounding-box-guided training enables coarse, controllable multi-object generation, where salient objects are approximately constrained within the disjoint regions of the foveation mask.
Figure 27: Bounding-box-guided generation and saliency-guided generation. We compare bounding-box-guided and saliency-guided generation. Because bounding-box-derived masks precisely delineate object contours, the bounding-box-guided model generates entire objects within the high-resolution region. In contrast, the saliency-guided model aligns only the most salient parts of objects with the center of the high-resolution region.
Figure 28: Foveated Diffusion artifacts. We delineate the foveation border with a white circular outline. The red dashed lines indicate regions with blending artifacts.
Image and video diffusion models allocate equal computation to every region, even when the intended scene calls for varying levels of detail. The spatial distribution of detail can often be anticipated before generation, indicating where computation can be reduced. We introduce Level-of-Token (LoT) Diffusion, a framework that turns this knowledge into an explicit multiresolution token layout (Level-of-Token layout) for adaptive and efficient generation. Tokens represent rectangular patches of varying sizes and shapes, allocating finer tokens where detail is needed and coarser tokens elsewhere. We adapt pretrained diffusion transformers to LoT layouts through a patch-wise asymmetric flow parametrization and embeddings for multiresolution tokens, preserving full-resolution flow prediction at every denoising step while processing only a reduced token sequence. LoT Diffusion enables layout-adaptive generation while preserving pretrained generative priors. We demonstrate LoT with layouts derived from semantic masks, bounding boxes, texture variance, and depth-of-field cues, as well as agentic plans. Across image and video generation, LoT offers favorable quality-efficiency tradeoffs, with significant speedups determined by the layout's token budget. Our project website is at https://georgenakayama.github.io/lotdiffusion/.
Flow Matching enables high-quality visual generation via continuous-time dynamics, but inference remains costly due to multiple sequential function evaluations. Existing acceleration methods reduce the number of function evaluations but often introduce additional training overhead, degrade quality, or fail to account for input-dependent variability. We propose COFLOW, an inference-time method that adaptively selects the step counts each generation based on the prompt features. Our context-aware COFLOW is trained online with an unsupervised reward that balances inference efficiency and generation fidelity. Our method is plug-and-play, requiring no retraining of the underlying generative model. It generalizes to image and video generation, achieving over 2.5x speedup while preserving perceptual and semantic quality. We further provide a theoretical analysis establishing an O(1/K) forward-Euler discretization error bound under standard regularity conditions.
Pixel-space diffusion models avoid the reconstruction ceiling of latent diffusion models by generating directly in image space. However, their substantially higher token count makes generation expensive due to the quadratic complexity of self-attention. Several existing efficiency methods reduce this cost by using larger patches at selected denoising steps, thereby representing the image with fewer tokens. Yet, each step still uses a single patch size uniformly across the entire image, overlooking that different regions suffer different fidelity losses when coarsened. We introduce MOSAIK, a damage-guided framework that varies patch size across regions and denoising steps. MOSAIK adapts the PixelDiT backbone to generate arbitrary heterogeneous patch layouts, and a lightweight predictor uses intermediate denoising features to estimate the fidelity loss caused by coarsening each region. Given a token budget, our damage-guided layout predictor assigns fine patches to sensitive regions and coarse patches elsewhere. Remarkably, while reducing FLOPs by 70% and token count by 83%, MOSAIK matches the full-compute PixelDiT on GenEval and its DPG-Bench score drops by only 1.0 point. Compared to diverse efficiency paradigms, including temporal patch scheduling and feature caching, our approach delivers highly competitive performance at moderate budgets and consistently outperforms these baselines in highly constrained compute regimes.