How Long Can Unified Multimodal Models Generate Images Reliably? Taming Long-Horizon Interleaved Image Generation via Context Curation
Authors: Haoyu Chen, Qing Liu, Yuqian Zhou, He Zhang, Zhaowen Wang, Mengwei Ren, Jingjing Ren, Xiang Wang, +2 more
Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Adobe Research · Huazhong University of Science and Technology · The Hong Kong University of Science and Technology
Unified multimodal models hold the promise of generating extensive, interleaved narratives, weaving text and imagery into coherent long-form stories. However, current systems suffer from a critical reliability gap: as sequences grow, generation quality rapidly collapses. In this work, we investigate the mechanism behind this failure and argue that it is distinct from standard long-context challenges. We reveal that in generation, accumulated visual history acts as a source of active pollution, a decay governed specifically by the number of image events rather than raw token count. We identify a structural vulnerability where dense visual tokens overwhelm the attention mechanism, creating noise that distorts future synthesis. Guided by these mechanistic insights, we propose UniLongGen, a training-free inference strategy that prioritizes safe conditioning over total recall. Instead of retaining all history, UniLongGen dynamically curates the model's memory, identifying and discarding interfering visual signals based on the model's own internal relevance rankings. Extensive experiments demonstrate that this active forgetting approach is essential for stability: UniLongGen significantly outperforms baselines in long-horizon fidelity and consistency, while simultaneously reducing memory footprint and inference time.
Figures & tables
Figure 2: Generation quality degrades as sequence length increases ( 1024×576 ). Left: Normalized scores for four metrics (Consistency, Quality, HPS v3, PickScore) across a 40-image sequence, averaged over multiple runs (shaded bands indicate variance). All metrics remain relatively stable during the first ∼ 20 images, then undergo a sharp collapse. Right: Representative samples at different positions. Early shots exhibit high visual fidelity and coherent scene composition, whereas later shots deteriorate into severe artifacts, structural distortions, and ultimately unrecognizable outputs.
Figure 3: Bottleneck: Generation quality depends on image count, not token length. We visualize generation quality (HPS v3, color scale) for long-horizon sequences with different tokens-per-image rates ( 1024×1024 , 1024×576 , 768×432 ). Despite wide disparities in cumulative token count (y-axis), all settings exhibit quality collapse within the same image-index window (20–25 images, shaded region). Horizontal comparisons (dashed gray lines) confirm that matching the token budget does not predict success; instead, the number of semantic events (image index) is the dominant bottleneck.
Figure 4: Token-matched text vs. image history induces distinct attention regimes and failure modes. We compare (A) a 120K-token text-only history and (B) a token-matched image-heavy history (22 images). Top row (diagnostics): similarity-score distributions from current-image queries to historical tokens show that text history produces a consistently concentrated score spectrum, while image-heavy history exhibits much higher variance and a heavier tail (outliers). A concentration plot (cumulative attention vs. top-% tokens) further indicates that attention under image history is more top-heavy (higher Gini). Bottom row (outcomes): token-matched text history mainly leads to passive dilution (blurred, under-conditioned generations), whereas token-matched image history causes active pollution (artifacts, speckle, and structural distortions).
Figure 5: Attention becomes diluted and unfocused as context grows. Attention entropy rises with context length, indicating the model becomes increasingly “confused” about where to attend.
Figure 6: Attention is inherently top-heavy across depth. We measure concentration as the cumulative attention coverage captured by the top- k% tokens in each layer (shaded bands).
Figure 6
Figure 9: Layer-wise functional clustering. Ward hierarchical clustering on six attention features reveals five functional regimes. C1 (Layer 0) is text-dominant, C2 – C3 balance text/vision, and C4 – C5 (late layers) are VAE-dominant with higher entropy; Bottom: bubble size indicates feature magnitude, and contiguous assignments show a structured shift from grounding to synthesis.
Figure 10: Overview of UniLongGen. Given a long interleaved history, we perform a one-shot probing pass with dense KV and derive two relevance masks from model-internal attention: (i) at an early, text-dependent layer (Layer 1), we score historical text blocks using current VAE queries with a length-normalized Pre-Softmax similarity score; (ii) at a late, generation-dominant layer (default: ℓsyn=15 ), we score historical VAE image blocks . We then keep the masks fixed and apply a layer-split KV visibility throughout generation: early layers use the text mask, while late layers use the image mask, dropping non-selected tokens.
Variant
Selection design
Downsampling
Quality ↑
Consistency ↑
Signal
Unit
Budget
Discard
Rate
Interp
HPS v3
PickScore
Qual.
ID
Style
DINOv2
A. Baselines (no sparsification / naive heuristics)
Base: Dense KV
–
–
–
–
–
–
3.1677
0.1943
5.83
5.49
3.99
0.3164
Sliding window (first + last N imgs)
–
Image
N=4
Drop
–
–
4.4886
0.1969
6.47
5.73
4.56
0.3337
Sliding window (first + last N imgs)
–
Image
N=8
Drop
–
–
3.9126
0.1957
6.31
5.41
4.50
0.3267
Sliding window (first + last N imgs)
–
Image
N=12
Drop
–
–
4.0844
0.1960
6.34
5.82
4.76
0.3235
Table 1: Systematic ablations for long interleaved text–image generation. Columns are grouped into Selection design , Downsampling , and Metrics . HPS v2 / PickScore / Qual. measure visual quality; DINOv2 / ID Cons. / Style Cons. measure consistency. Qual., ID Cons., and Style Cons. are GPT-4o based evaluations. Within each block, we vary only one axis while keeping the rest fixed for controlled comparison. Our best configuration uses one-shot dual probing (Text at Layer 1, VAE at a late layer) with a fixed, layer-split KV visibility to retain a small number of reference images ( K≈4 – 6 ) with direct dropping of discarded tokens (no compression).
Figure 11: Qualitative comparison of long-horizon stability across context-management baselines. We show snapshots from the same 40-image interleaved generation at late positions (columns: 27, 29, 32, 36, 39) at 768×432 resolution. Rows compare the base model with dense KV (Bagel), UniLongGen, anchored sliding-window baselines (keep first anchor plus last N image blocks), token-level selection, a human semantic oracle, and text-to-text retrieval. Dense KV and several heuristics degrade into artifacts or drift at later positions, while UniLongGen better preserves coherent synthesis in this representative example.
Variant
HPS v3 ↑
PickScore ↑
Qual. ↑
Style Cons. ↑
DINOv2 ↑
Baselines
Dense KV (Bagel)
3.1677
0.1943
5.83
3.99
0.3164
Sliding window
4.4886
0.1969
6.47
4.56
0.3337
Ours ( K=4 ) with varying ℓsyn
Ours, ℓsyn=10
7.4349
0.2019
7.53
6.12
0.4251
Ours, ℓsyn=15
7.5701
0.2025
7.58
6.13
0.4272
Table 2: Ablation on VAE probing layer ℓsyn . We vary the layer at which the VAE attention score is computed ( ℓsyn∈{10,15,20} ) with Layer 15 yields the best trade-off across all metrics.
Figure 12: Runtime scaling vs. historical context (no CFG). Total flow-matching time (seconds; summed over steps) to generate one image at two resolutions (top: 512×512 , bottom: 1024×1024 ) as a function of historical-context tokens (k). Dense KV slows down with longer history, while UniLongGen remains nearly flat due to KV eviction, achieving up to ∼ 11 × speedup at long contexts.
Variant
HPS v3 ↑
Qual. ↑
Style ↑
DINOv2 ↑
Dense KV (no pruning)
3.1677
5.83
3.99
0.3164
Semantic oracle (top- K by human relevance)
5.1158
6.77
5.1
0.4205
Text-block matching (current → history, top- K )
1.7063
5.35
3.06
0.3094
UniLongGen (attention-based top- K )
7.5701
7.58
6.13
0.4272
Table 3: Semantic oracle vs. attention-based selection. “Semantic oracle” keeps the same budget ( K ) of historical images judged most relevant by human annotation; “Text-block matching” ranks historical turns by the similarity between the current image’s text block and each historical text block, then keeps the images from the top- K turns. Our method selects images using model-internal attention signals.
Figure 13: Additional qualitative results on long-horizon interleaved generation (1/7).
Figure 14: Additional qualitative results on long-horizon interleaved generation (2/7).
Figure 15: Additional qualitative results on long-horizon interleaved generation (3/7).
Figure 16: Additional qualitative results on long-horizon interleaved generation (4/7).
Figure 17: Additional qualitative results on long-horizon interleaved generation (5/7).
Figure 18: Additional qualitative results on long-horizon interleaved generation (6/7).
Figure 19: Additional qualitative results on long-horizon interleaved generation (7/7).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 20: Modality attention flow and Text–VAE competition across depth. Left: Modality attention ratios (Text/ViT/VAE) over Transformer depth, showing a shift from early-layer multimodal processing toward VAE-dominant late-layer image synthesis. Right: Text attention ratio versus VAE attention ratio in the historical KV across layers (colored by layer index), exhibiting a strong negative correlation ( r≈−0.93 ) that indicates zero-sum competition between language grounding and image-token conditioning.
Figure 21: Attention shifts across flow-matching steps. We plot the attention mass allocated to historical-context tokens and current-generation tokens across flow-matching steps. This step-dependent reallocation motivates a uniform context-selection policy across steps: step-varying visibility can cause inconsistent intermediate conditioning and destabilize generation.
Figure 22: Illustration of relevant vs. irrelevant visual history. We construct sequences containing both relevant and irrelevant historical images to study attention misallocation. Top row (Relevant History): Historical images depicting the same subject (an elderly clockmaker) across different shots and contexts (Shot 1, 6, 11, 16), which should receive high attention for identity-consistent generation. Bottom row (Irrelevant History): Unrelated historical images (tools, gears, teapot, cat) that share no semantic connection with the generation target but may still attract spurious attention due to low-level visual similarity. Right (Generation Target): The model’s generation output. When irrelevant history is present, attention can be misallocated to visually salient but semantically irrelevant tokens, leading to degraded identity consistency or visual artifacts.
Figure 23: Lumina-mGPT attention signature during long-horizon interleaved image decoding (pure autoregressive model). Left: layer- and step-wise difference between attention to historical image tokens and historical text tokens (Hist Image − Hist Text; red indicates more attention to image history, blue indicates more attention to text history). Middle: distribution of attention mass across content types over generation steps, including the current prompt, history text, history images, and current-image self-attention. Right: history-context versus current-context attention proportion over generation steps. Overall, early layers tend to emphasize historical image content while deeper layers shift toward text, and the total attention share devoted to history is relatively small compared to current-context/self-attention.
Model
Method
HPS v3 ↑
Qual. ↑
Lumina-mGPT (Chameleon-based)
Dense KV
2.7954
0.1904
Sliding Window
7.2965
0.2014
+ UniLongGen
7.3020
0.2015
Appendix
Table 4: Generalization to a pure autoregressive unified model (Lumina-mGPT).
Figure 24: Position-wise performance curves for long-horizon interleaved generation. Top: HPS v3 (quality) and DINO consistency (identity) as a function of image index in a 40-image sequence, comparing the base model (Bagel), anchored sliding-window baselines (keep first + last N images), UniLongGen, and UniLongGen with compression. Bottom: HPS v3 ablations for (left) hyperparameters (number of retained image blocks and VAE probing layer), (middle) compression strategies (LERP with different downsampling rates), and (right) token-level selection variants, showing that UniLongGen best preserves long-horizon stability.
Figure 25: Token-level vs. image-level history selection under matched KV budgets. We compare two ways to reduce the visible KV cache when conditioning on long visual history. Top (token level): keep a fixed fraction of historical VAE tokens (e.g., 20%/40%/60% KV), which can weaken the effective reference signal and lead to degraded subject fidelity in this example. Bottom (image level): keep a small number of whole reference images at comparable KV budgets (e.g., 6/10/12 images), which better preserves identity-consistent conditioning. This qualitative comparison supports event/image-level curation over token-level pruning for stable long-horizon synthesis.
Variant
HPS v3 ↑
PickScore ↑
Style Cons. ↑
DINOv2-ID ↑
Baseline
3.1677
0.1943
3.99
0.3164
Post-Softmax
5.2779
0.1978
5.44
0.3617
Pre-Softmax (Ours)
7.5701
0.2025
6.13
0.4272
Appendix
Table 5: Ablation: using Post-Softmax attention vs. Pre-Softmax QK scores for relevance evaluation. We compare two ways to compute the relevance signal used to select historical reference images: (i) aggregating Post-Softmax attention weights, and (ii) using Pre-Softmax head-averaged QK similarity scores. We find that using the Pre-Softmax score yields substantially better long-horizon generation quality and identity consistency.
Figure 26: The prompt used for GPT-5.2 based subject consistency evaluation.
Figure 27: The prompt used for GPT-5.2 based artistic style consistency evaluation.
Figure 28: The prompt used for GPT-5.2 based visual quality evaluation.
Unified multimodal models (UMMs) provide an interface for interleaved text-image generation, where generated visual states can directly inform subsequent reasoning and synthesis. However, existing systems are still primarily demonstrated on short trajectories or simplified visual content. We develop a large UMM natively trained on interleaved multimodal data, with strong capabilities in image synthesis, text rendering, complex composition, and multimodal planning, and study how to extend these native capabilities to substantially longer autoregressive trajectories. The main challenge is that generated images continuously accumulate in context: irrelevant or imperfect visual states interfere with future synthesis, while previously established characters and scenes must still be recovered after long intervals. We introduce a visual-memory framework that maintains persistent visual references, selectively retrieves scene-relevant historical states while preserving the complete semantic history, and uses few-step self-rollout to adapt the model to imperfect generated histories. We further introduce LongMIG-Bench for evaluating long-horizon text-to-multi-image and image-to-multi-image generation. Our model achieves high-fidelity long-horizon interleaved generation with strong character and scene consistency, demonstrating how visual memory can scale native interleaved generation to substantially longer trajectories.
Ziqiao Peng, Yutao Cui, Wang Lin +12
Renmin University of China · Tencent Hy · Tsinghua University
While recent advancements in multimodal language models have enabled image generation from expressive multi-image instructions, existing methods struggle to maintain performance under complex interleaved instructions. This limitation stems from the structural separation of images and text in current paradigms, which forces models to bridge difficult long-range dependencies to match descriptions with visual targets. To address these challenges, we propose Images iNSEnTences (a.k.a, INSET), a unified generation model that seamlessly embeds images as native vocabulary within textual instructions. By positioning visual features directly at their corresponding semantic slots, INSET leverages the contextual locality of transformers for precise object binding, effectively treating images as dense, expressive language tokens. Furthermore, we introduce a scalable data engine that synthesizes 15M high-quality interleaved samples from standard image and video datasets, utilizing VLMs and LLMs to construct rich, long-horizon sequences. Evaluation results on InterleaveBench demonstrate that INSET significantly outperforms state-of-the-art methods in multi-image consistency and text alignment, with performance gaps widening as input complexity increases. Beyond standard generation, our approach inherently extends to multimodal image editing, integrating visual content as part of the instruction to facilitate highly expressive and creative visual manipulations.
Recent unified models have made unprecedented progress in both understanding and generation. However, while most of them accept multi-modal inputs, they typically produce only single-modality outputs. This challenge of producing interleaved content is mainly due to training data scarcity and the difficulty of modeling long-range cross-modal context. To address this issue, we decompose interleaved generation into textual planning and visual consistency modeling, and introduce a framework consisting of a planner and a visualizer. The planner produces dense textual descriptions for visual content, while the visualizer synthesizes images accordingly. Under this guidance, we construct large-scale textual-proxy interleaved data (where visual content is represented in text) to train the planner, and curate reference-guided image data to train the visualizer. These designs give rise to Wan-Weaver, which exhibits emergent interleaved generation ability with long-range textual coherence and visual consistency. Meanwhile, the integration of diverse understanding and generation data into planner training enables Wan-Weaver to achieve robust task reasoning and generation proficiency. To assess the model's capability in interleaved generation, we further construct a benchmark that spans a wide range of use cases across multiple dimensions. Extensive experiments demonstrate that, even without access to any real interleaved data, Wan-Weaver achieves superior performance over existing methods.