Unified multimodal models (UMMs) provide an interface for interleaved text-image generation, where generated visual states can directly inform subsequent reasoning and synthesis. However, existing systems are still primarily demonstrated on short trajectories or simplified visual content. We develop a large UMM natively trained on interleaved multimodal data, with strong capabilities in image synthesis, text rendering, complex composition, and multimodal planning, and study how to extend these native capabilities to substantially longer autoregressive trajectories. The main challenge is that generated images continuously accumulate in context: irrelevant or imperfect visual states interfere with future synthesis, while previously established characters and scenes must still be recovered after long intervals. We introduce a visual-memory framework that maintains persistent visual references, selectively retrieves scene-relevant historical states while preserving the complete semantic history, and uses few-step self-rollout to adapt the model to imperfect generated histories. We further introduce LongMIG-Bench for evaluating long-horizon text-to-multi-image and image-to-multi-image generation. Our model achieves high-fidelity long-horizon interleaved generation with strong character and scene consistency, demonstrating how visual memory can scale native interleaved generation to substantially longer trajectories.
Figures & tables
Figure 1: High-fidelity long-horizon interleaved generation across diverse visual domains. Our model supports photorealistic, stylized, text-rich, and design-oriented generation while maintaining visual consistency over extended trajectories.
Figure 2: Overview of our long-horizon interleaved generation framework. Built on our unified multimodal backbone, the model constructs persistent visual references, retrieves scene-relevant history during autoregressive generation, and uses few-step self-rollout for memory adaptation.
Method
Character Identity ↑
Scene Memory
Visual Fidelity ↑
Visual Memory ↑
Aesthetic Quality ↑
Text Rendering ↑
Scene ↑
Gap-S ↑
Gap-D ↑
Agent-Planned
StoryDiffusion
1.61
1.61
0.0615
0.1196
2.92
1.55
2.31
1.17
Qwen-Image
3.16
3.05
0.0694
0.2127
3.83
3.14
3.73
2.22
UniSpace
2.41
2.33
0.0346
0.1321
2.48
2.03
2.57
1.53
HiDream
3.35
2.58
0.0576
0.1620
3.19
2.76
2.71
2.78
Table 1: Long-horizon generation on LongMIG-Bench. Best and second-best results are shown in bold and underline , respectively.
Figure 3: High-fidelity long-form generation. Left: image-conditioned generation with identity preservation across diverse views. Right: long-form interleaved generation with consistent characters and visual states over an extended trajectory.
Figure 4: Native interleaved generation over extended trajectories. Representative frames from long autoregressive generations across diverse visual domains, including 50+ image trajectories.
Variant
Setting
Identity ↑
Scene ↑
Gap-D ↑
LR Scene ↑
LR Gap-D ↑
Visual Fidelity ↑
Scene-aware retrieval
Predicted Routing
Predicted Scene
3.36
3.26
0.164
3.10
0.119
3.15
GT Scene
Annotated Scene
3.21
3.32
0.173
3.15
0.121
3.03
Noisy Scene
30% Corrupted
3.30
2.81
0.145
2.76
0.099
3.14
Recent-4
Scene-Agnostic
3.15
2.85
0.147
2.79
0.092
2.97
Recent-2
Scene-Agnostic
3.05
2.58
0.114
2.47
0.071
2.89
Table 2: Controlled analysis of visual-memory components. Ablations of scene-aware retrieval, persistent visual references, and self-rollout adaptation on the 24-image Long+Ext stress test.
Figure 5: Effect of self-rollout adaptation. Base and GT-FT exhibit progressive reddish color drift and loss of fine-grained facial details, whereas Self-Rollout better preserves color appearance and local visual structure throughout the trajectory.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Long-form visual process generation for Portuguese egg tarts. All methods are prompted to generate a sequence documenting the preparation process. The comparison highlights differences in step progression, visual diversity, repetition, and consistency across the complete trajectory.
Figure 7: Long-form visual process generation for braised pork. Each method generates a multi-image sequence describing the cooking process. The examples illustrate how models differ in representing successive visual states rather than repeatedly depicting similar intermediate or final results.
Figure 8: Long-form generation across diverse visual domains. Representative trajectories cover (a) cinematic storytelling, (b) cooking photography, (c) minimalist typography, (d) architectural visualization, (e) playful food illustration, (f) graphic identity design, (g) image-conditioned product design, and (h) image-conditioned pet photography. Input reference images for image-conditioned examples are shown separately.
Figure 9: Complete native interleaved planning-and-generation trajectory. Given a request for an eight-page nutrition presentation, the model first forms an overall plan and generates a reusable visual reference, then alternates between textual planning and image generation until completing the requested sequence. Generated images are inserted back into the multimodal context and remain available to subsequent generation steps.
Data type
Fraction
Multi-shot storyboards
45.3%
Illustrated books with visual references
18.5%
Illustrated books without visual references
7.9%
Social-media comics
14.5%
Long comics with background continuity
9.1%
Brand and design sequences
4.7%
Appendix
Table 3: Composition of the long-horizon post-training corpus.
T2MI
I2MI
Stories
50
50
Specified shots
462
460
Avg. shots / story
9.2
Min. revisited shots / story
≥4
Avg. revisited shots
6.16
6.04
Distinct styles
50
50
Appendix
Table 4: Composition of LongMIG-Bench.
Figure 10: Examples from LongMIG-Bench. Each story contains structured character and location definitions together with an ordered sequence of shots. Colors indicate recurring scenes, and arrows highlight long-range revisits after intervening scene transitions. The I2MI track additionally provides a user reference image.
Sec./Image ↓
Last Prefill (s) ↓
Horizon
Full History
Ours
Full History
Ours
8
27.8
26.4
6.8
4.9
16
49.3
37.6
20.7
9.8
24
73.8
46.5
41.4
14.0
32
102.9
54.9
69.0
19.4
Appendix
Table 5: Efficiency across increasing generation horizons. Ours and Full History use identical model weights and generation settings. Sec./Image is the mean wall-clock time per image over the first h images; Last Prefill is the AR prefill time of the h -th image.
Figure 11: Scaling of image-synthesis context and generation latency. Left: selective visual-memory retrieval substantially slows context growth compared with Full History. Right: the corresponding per-image latency gap increases with generation horizon. Solid curves denote measured trajectories, while dotted extensions indicate linear extrapolation.
Unified multimodal models hold the promise of generating extensive, interleaved narratives, weaving text and imagery into coherent long-form stories. However, current systems suffer from a critical reliability gap: as sequences grow, generation quality rapidly collapses. In this work, we investigate the mechanism behind this failure and argue that it is distinct from standard long-context challenges. We reveal that in generation, accumulated visual history acts as a source of active pollution, a decay governed specifically by the number of image events rather than raw token count. We identify a structural vulnerability where dense visual tokens overwhelm the attention mechanism, creating noise that distorts future synthesis. Guided by these mechanistic insights, we propose UniLongGen, a training-free inference strategy that prioritizes safe conditioning over total recall. Instead of retaining all history, UniLongGen dynamically curates the model's memory, identifying and discarding interfering visual signals based on the model's own internal relevance rankings. Extensive experiments demonstrate that this active forgetting approach is essential for stability: UniLongGen significantly outperforms baselines in long-horizon fidelity and consistency, while simultaneously reducing memory footprint and inference time.
Haoyu Chen, Qing Liu, Yuqian Zhou +7
The Hong Kong University of Science and Technology (Guangzhou) · Adobe Research · Huazhong University of Science and Technology +1
Recent unified models have made unprecedented progress in both understanding and generation. However, while most of them accept multi-modal inputs, they typically produce only single-modality outputs. This challenge of producing interleaved content is mainly due to training data scarcity and the difficulty of modeling long-range cross-modal context. To address this issue, we decompose interleaved generation into textual planning and visual consistency modeling, and introduce a framework consisting of a planner and a visualizer. The planner produces dense textual descriptions for visual content, while the visualizer synthesizes images accordingly. Under this guidance, we construct large-scale textual-proxy interleaved data (where visual content is represented in text) to train the planner, and curate reference-guided image data to train the visualizer. These designs give rise to Wan-Weaver, which exhibits emergent interleaved generation ability with long-range textual coherence and visual consistency. Meanwhile, the integration of diverse understanding and generation data into planner training enables Wan-Weaver to achieve robust task reasoning and generation proficiency. To assess the model's capability in interleaved generation, we further construct a benchmark that spans a wide range of use cases across multiple dimensions. Extensive experiments demonstrate that, even without access to any real interleaved data, Wan-Weaver achieves superior performance over existing methods.
Unified vision-language models have made significant progress in multimodal understanding and generation, yet they largely fall short in producing multimodal interleaved outputs, which is a crucial capability for tasks like visual storytelling and step-by-step visual reasoning. In this work, we propose a reinforcement learning-based post-training strategy to unlock this capability in existing unified models, without relying on large-scale multimodal interleaved datasets. We begin with a warm-up stage using a hybrid dataset comprising curated interleaved sequences and limited data for multimodal understanding and text-to-image generation, which exposes the model to interleaved generation patterns while preserving its pretrained capabilities. To further refine interleaved generation, we propose a unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the multimodal setting. Our approach jointly models text and image generation within a single decoding trajectory and optimizes it with our novel hybrid rewards covering textual relevance, visual-text alignment, and structural fidelity. Additionally, we incorporate process-level rewards to provide step-wise guidance, enhancing training efficiency in complex multimodal tasks. Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of multimodal interleaved generation.
Ming Nie, Chunwei Wang, Jianhua Han +2
School of Data Science, Fudan University · Noah’s Ark Lab, Huawei · Yinwang Intelligent Technology Co., Ltd.