cs.CVMar 8, 2026
SaveHow Long Can Unified Multimodal Models Generate Images Reliably? Taming Long-Horizon Interleaved Image Generation via Context Curation
Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Adobe Research · Huazhong University of Science and Technology · The Hong Kong University of Science and Technology
Abstract
Unified multimodal models hold the promise of generating extensive, interleaved narratives, weaving text and imagery into coherent long-form stories. However, current systems suffer from a critical reliability gap: as sequences grow, generation quality rapidly collapses. In this work, we investigate the mechanism behind this failure and argue that it is distinct from standard long-context challenges. We reveal that in generation, accumulated visual history acts as a source of active pollution, a decay governed specifically by the number of image events rather than raw token count. We identify a structural vulnerability where dense visual tokens overwhelm the attention mechanism, creating noise that distorts future synthesis. Guided by these mechanistic insights, we propose UniLongGen, a training-free inference strategy that prioritizes safe conditioning over total recall. Instead of retaining all history, UniLongGen dynamically curates the model's memory, identifying and discarding interfering visual signals based on the model's own internal relevance rankings. Extensive experiments demonstrate that this active forgetting approach is essential for stability: UniLongGen significantly outperforms baselines in long-horizon fidelity and consistency, while simultaneously reducing memory footprint and inference time.
Figures & tables
Figure 2: Generation quality degrades as sequence length increases ( ). Left: Normalized scores for four metrics (Consistency, Quality, HPS v3, PickScore) across a 40-image sequence, averaged over multiple runs (shaded bands indicate variance). All metrics remain relatively stable during the first 20 images, then undergo a sharp collapse. Right: Representative samples at different positions. Early shots exhibit high visual fidelity and coherent scene composition, whereas later shots deteriorate into severe artifacts, structural distortions, and ultimately unrecognizable outputs.
Figure 3: Bottleneck: Generation quality depends on image count, not token length. We visualize generation quality (HPS v3, color scale) for long-horizon sequences with different tokens-per-image rates ( , , ). Despite wide disparities in cumulative token count (y-axis), all settings exhibit quality collapse within the same image-index window (20–25 images, shaded region). Horizontal comparisons (dashed gray lines) confirm that matching the token budget does not predict success; instead, the number of semantic events (image index) is the dominant bottleneck.
Figure 4: Token-matched text vs. image history induces distinct attention regimes and failure modes. We compare (A) a 120K-token text-only history and (B) a token-matched image-heavy history (22 images). Top row (diagnostics): similarity-score distributions from current-image queries to historical tokens show that text history produces a consistently concentrated score spectrum, while image-heavy history exhibits much higher variance and a heavier tail (outliers). A concentration plot (cumulative attention vs. top-% tokens) further indicates that attention under image history is more top-heavy (higher Gini). Bottom row (outcomes): token-matched text history mainly leads to passive dilution (blurred, under-conditioned generations), whereas token-matched image history causes active pollution (artifacts, speckle, and structural distortions).
Figure 5: Attention becomes diluted and unfocused as context grows. Attention entropy rises with context length, indicating the model becomes increasingly “confused” about where to attend.
Figure 6: Attention is inherently top-heavy across depth. We measure concentration as the cumulative attention coverage captured by the top- tokens in each layer (shaded bands).
Figure 9: Layer-wise functional clustering. Ward hierarchical clustering on six attention features reveals five functional regimes. C1 (Layer 0) is text-dominant, C2 – C3 balance text/vision, and C4 – C5 (late layers) are VAE-dominant with higher entropy; Bottom: bubble size indicates feature magnitude, and contiguous assignments show a structured shift from grounding to synthesis.
Figure 10: Overview of UniLongGen. Given a long interleaved history, we perform a one-shot probing pass with dense KV and derive two relevance masks from model-internal attention: (i) at an early, text-dependent layer (Layer 1), we score historical text blocks using current VAE queries with a length-normalized Pre-Softmax similarity score; (ii) at a late, generation-dominant layer (default: ), we score historical VAE image blocks . We then keep the masks fixed and apply a layer-split KV visibility throughout generation: early layers use the text mask, while late layers use the image mask, dropping non-selected tokens.
| Variant | Selection design | Downsampling | Quality | Consistency | ||||||||
| Signal | Unit | Budget | Discard | Rate | Interp | HPS v3 | PickScore | Qual. | ID | Style | DINOv2 | |
| A. Baselines (no sparsification / naive heuristics) | ||||||||||||
| Base: Dense KV | – | – | – | – | – | – | 3.1677 | 0.1943 | 5.83 | 5.49 | 3.99 | 0.3164 |
| Sliding window (first + last imgs) | – | Image | Drop | – | – | 4.4886 | 0.1969 | 6.47 | 5.73 | 4.56 | 0.3337 | |
| Sliding window (first + last imgs) | – | Image | Drop | – | – | 3.9126 | 0.1957 | 6.31 | 5.41 | 4.50 | 0.3267 | |
| Sliding window (first + last imgs) | – | Image | Drop | – | – | 4.0844 | 0.1960 | 6.34 | 5.82 | 4.76 | 0.3235 | |
Table 1: Systematic ablations for long interleaved text–image generation. Columns are grouped into Selection design , Downsampling , and Metrics . HPS v2 / PickScore / Qual. measure visual quality; DINOv2 / ID Cons. / Style Cons. measure consistency. Qual., ID Cons., and Style Cons. are GPT-4o based evaluations. Within each block, we vary only one axis while keeping the rest fixed for controlled comparison. Our best configuration uses one-shot dual probing (Text at Layer 1, VAE at a late layer) with a fixed, layer-split KV visibility to retain a small number of reference images ( – ) with direct dropping of discarded tokens (no compression).
Figure 11: Qualitative comparison of long-horizon stability across context-management baselines. We show snapshots from the same 40-image interleaved generation at late positions (columns: 27, 29, 32, 36, 39) at resolution. Rows compare the base model with dense KV (Bagel), UniLongGen, anchored sliding-window baselines (keep first anchor plus last image blocks), token-level selection, a human semantic oracle, and text-to-text retrieval. Dense KV and several heuristics degrade into artifacts or drift at later positions, while UniLongGen better preserves coherent synthesis in this representative example.
| Variant | HPS v3 | PickScore | Qual. | Style Cons. | DINOv2 |
| Baselines | |||||
| Dense KV (Bagel) | 3.1677 | 0.1943 | 5.83 | 3.99 | 0.3164 |
| Sliding window | 4.4886 | 0.1969 | 6.47 | 4.56 | 0.3337 |
| Ours ( ) with varying | |||||
| Ours, | 7.4349 | 0.2019 | 7.53 | 6.12 | 0.4251 |
| Ours, | 7.5701 | 0.2025 | 7.58 | 6.13 | 0.4272 |
Table 2: Ablation on VAE probing layer . We vary the layer at which the VAE attention score is computed ( ) with Layer 15 yields the best trade-off across all metrics.
Figure 12: Runtime scaling vs. historical context (no CFG). Total flow-matching time (seconds; summed over steps) to generate one image at two resolutions (top: , bottom: ) as a function of historical-context tokens (k). Dense KV slows down with longer history, while UniLongGen remains nearly flat due to KV eviction, achieving up to 11 speedup at long contexts.
| Variant | HPS v3 | Qual. | Style | DINOv2 |
| Dense KV (no pruning) | 3.1677 | 5.83 | 3.99 | 0.3164 |
| Semantic oracle (top- by human relevance) | 5.1158 | 6.77 | 5.1 | 0.4205 |
| Text-block matching (current history, top- ) | 1.7063 | 5.35 | 3.06 | 0.3094 |
| UniLongGen (attention-based top- ) | 7.5701 | 7.58 | 6.13 | 0.4272 |
Table 3: Semantic oracle vs. attention-based selection. “Semantic oracle” keeps the same budget ( ) of historical images judged most relevant by human annotation; “Text-block matching” ranks historical turns by the similarity between the current image’s text block and each historical text block, then keeps the images from the top- turns. Our method selects images using model-internal attention signals.
Figure 13: Additional qualitative results on long-horizon interleaved generation (1/7).
Figure 14: Additional qualitative results on long-horizon interleaved generation (2/7).
Figure 15: Additional qualitative results on long-horizon interleaved generation (3/7).
Figure 16: Additional qualitative results on long-horizon interleaved generation (4/7).
Figure 17: Additional qualitative results on long-horizon interleaved generation (5/7).
Figure 18: Additional qualitative results on long-horizon interleaved generation (6/7).
Figure 19: Additional qualitative results on long-horizon interleaved generation (7/7).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 20: Modality attention flow and Text–VAE competition across depth. Left: Modality attention ratios (Text/ViT/VAE) over Transformer depth, showing a shift from early-layer multimodal processing toward VAE-dominant late-layer image synthesis. Right: Text attention ratio versus VAE attention ratio in the historical KV across layers (colored by layer index), exhibiting a strong negative correlation ( ) that indicates zero-sum competition between language grounding and image-token conditioning.
Figure 21: Attention shifts across flow-matching steps. We plot the attention mass allocated to historical-context tokens and current-generation tokens across flow-matching steps. This step-dependent reallocation motivates a uniform context-selection policy across steps: step-varying visibility can cause inconsistent intermediate conditioning and destabilize generation.
Figure 22: Illustration of relevant vs. irrelevant visual history. We construct sequences containing both relevant and irrelevant historical images to study attention misallocation. Top row (Relevant History): Historical images depicting the same subject (an elderly clockmaker) across different shots and contexts (Shot 1, 6, 11, 16), which should receive high attention for identity-consistent generation. Bottom row (Irrelevant History): Unrelated historical images (tools, gears, teapot, cat) that share no semantic connection with the generation target but may still attract spurious attention due to low-level visual similarity. Right (Generation Target): The model’s generation output. When irrelevant history is present, attention can be misallocated to visually salient but semantically irrelevant tokens, leading to degraded identity consistency or visual artifacts.
Figure 23: Lumina-mGPT attention signature during long-horizon interleaved image decoding (pure autoregressive model). Left: layer- and step-wise difference between attention to historical image tokens and historical text tokens (Hist Image Hist Text; red indicates more attention to image history, blue indicates more attention to text history). Middle: distribution of attention mass across content types over generation steps, including the current prompt, history text, history images, and current-image self-attention. Right: history-context versus current-context attention proportion over generation steps. Overall, early layers tend to emphasize historical image content while deeper layers shift toward text, and the total attention share devoted to history is relatively small compared to current-context/self-attention.
| Model | Method | HPS v3 | Qual. |
| Lumina-mGPT (Chameleon-based) | Dense KV | 2.7954 | 0.1904 |
| Sliding Window | 7.2965 | 0.2014 | |
| + UniLongGen | 7.3020 | 0.2015 |
Table 4: Generalization to a pure autoregressive unified model (Lumina-mGPT).
Figure 24: Position-wise performance curves for long-horizon interleaved generation. Top: HPS v3 (quality) and DINO consistency (identity) as a function of image index in a 40-image sequence, comparing the base model (Bagel), anchored sliding-window baselines (keep first + last images), UniLongGen, and UniLongGen with compression. Bottom: HPS v3 ablations for (left) hyperparameters (number of retained image blocks and VAE probing layer), (middle) compression strategies (LERP with different downsampling rates), and (right) token-level selection variants, showing that UniLongGen best preserves long-horizon stability.
Figure 25: Token-level vs. image-level history selection under matched KV budgets. We compare two ways to reduce the visible KV cache when conditioning on long visual history. Top (token level): keep a fixed fraction of historical VAE tokens (e.g., 20%/40%/60% KV), which can weaken the effective reference signal and lead to degraded subject fidelity in this example. Bottom (image level): keep a small number of whole reference images at comparable KV budgets (e.g., 6/10/12 images), which better preserves identity-consistent conditioning. This qualitative comparison supports event/image-level curation over token-level pruning for stable long-horizon synthesis.
| Variant | HPS v3 | PickScore | Style Cons. | DINOv2-ID |
| Baseline | 3.1677 | 0.1943 | 3.99 | 0.3164 |
| Post-Softmax | 5.2779 | 0.1978 | 5.44 | 0.3617 |
| Pre-Softmax (Ours) | 7.5701 | 0.2025 | 6.13 | 0.4272 |
Table 5: Ablation: using Post-Softmax attention vs. Pre-Softmax scores for relevance evaluation. We compare two ways to compute the relevance signal used to select historical reference images: (i) aggregating Post-Softmax attention weights, and (ii) using Pre-Softmax head-averaged similarity scores. We find that using the Pre-Softmax score yields substantially better long-horizon generation quality and identity consistency.
Figure 26: The prompt used for GPT-5.2 based subject consistency evaluation.
Figure 27: The prompt used for GPT-5.2 based artistic style consistency evaluation.
Figure 28: The prompt used for GPT-5.2 based visual quality evaluation.