Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds

Nan Duan; Haoyang Huang; Weiyang Jin; Haoran Li; Yaowei Li; Yuming Li; Yijun Liu; Xin Lu

cs.CVAug 24, 2026

Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds

Authors: Nan Duan, Haoyang Huang, Weiyang Jin, Haoran Li, Yaowei Li, Yuming Li, Yijun Liu, Xin Lu,

Organizations: Joy Future Academy, JD

Abstract

Video generation is progressing beyond isolated clips toward long-form narratives and interactive worlds, requiring models to preserve identities, follow user controls, and remain stable over extended rollouts. We present JoyAI-Echo-1.5, a unified audio-visual generation system with two purpose-built variants. The long-video variant introduces composable cross-shot memory that aggregates visual evidence across multiple prior shots and speaker cues derived from speech-filtered full-shot audio, enabling persistent character appearance and voice identity across flexible combinations of text, image, and memory conditioning. The world-model variant converts heterogeneous navigation inputs into calibrated metric 6-DoF camera trajectories and injects them through a geometry-aware conditioning pathway, enabling controller-agnostic interaction across flexible viewpoints. To support efficient long-horizon generation, we transform a bidirectional audio-visual backbone into a causal few-step generator using progressive teacher forcing and short- and long-horizon Self-Gradient Forcing on self-generated rollouts. Experiments demonstrate strong performance in both settings. JoyAI-Echo-1.5 achieves improvements over existing long-video baselines in cross-shot consistency, visual quality, text alignment, and speech fidelity. Its world-model variant ranks first on WBench, with an average score of 81.7, and achieves leading visual quality and long-horizon persistence on SANA-WM-Bench. Together, these results indicate that memory, geometric control, and rollout-aware training provide a practical foundation for generating coherent stories and continuously evolving interactive worlds. Project page: https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/.

Figures & tables

Figure 1: Two-stage data construction pipeline of JoyAI-Echo-1.5. Raw videos are quality-filtered and organized into an identity-centric Memory corpus and a high-quality corpus with broader resolution, language, and multi-shot coverage.

Structure	T2V caption	I2V caption
Single shot	ID_A is a young man with short brown hair, wearing a dark gray hoodie, with a clear mid-range voice. He sits slightly off-camera, blinks naturally, and says, “my games often focus on that …” A static chest-up close-up uses shallow depth of field in an indoor gym. No on-screen text or subtitles. No obvious sound effects. No background music.	[Random I2V prefix] A short-haired man speaks with a clear, stable mid-range voice and a natural conversational tone. He sits slightly off-camera, blinks naturally, and says, “my games often focus on that …” A static chest-up close-up keeps him in focus against a softly blurred gym. No on-screen text or subtitles. No obvious sound effects. No background music.
Multi-shot	This video contains two shots. Shot 1: ID_A is a middle-aged man in a dark green dress uniform with a deep voice; ID_B is a middle-aged man in digital camouflage with a bright voice. They walk and talk side by side as a medium shot follows them. OCR: military slogans and a name label. Soft piano music plays. Shot 2: ID_C is a silent middle-aged man in a dark blue police uniform. A police formation advances in a static medium shot. No OCR or subtitles; the piano music continues.	[Random I2V prefix] This video contains two shots. Shot 1: A middle-aged man in a dress uniform and a middle-aged man in camouflage walk and talk side by side. The former has a deep, steady voice and the latter a bright, natural voice. A medium shot follows them; soft piano music plays. Shot 2: A serious middle-aged man in a dark blue police uniform appears silently within an advancing formation. A static medium shot follows the group. No OCR or subtitles; the piano music continues.

Table 1: Representative abbreviated examples of the four caption formats used in JoyAI-Echo-1.5. English renderings are shown for readability, while both Chinese and English captions are retained during training. T2V captions preserve explicit identity definitions, whereas I2V captions use concise natural references because appearance is supplied by the first frame.

Figure 2: Overview of memory-conditioned long-form generation and few-step acceleration in JoyAI-Echo-1.5. Top: unified conditioning with single-frame visual memories, speech-filtered audio memories, and separated RoPE coordinates supports multiple audio-visual generation tasks. Bottom: memory-robust joint DMD combines asymmetric video–audio optimization with audio-specific adversarial and energy constraints.

Figure 3: Illustration of the causal attention masks used in streaming post-training. (a) Audio-visual teacher forcing, where each noisy chunk attends to its preceding clean history and the current noisy chunk, while the clean target and future chunks are masked out. (b) Long-horizon SGF Forward 2, where the causal context is further restricted by the sink-plus-FIFO KV cache.

Figure 4: Overview of the Director Agent workflow. Given a user request and optional reference, the agent performs shot planning and prompt enhancement (PE), and supports both single-shot short-video generation and iterative long-video generation through memory retrieval, shot-level review and revision, and memory admission.

Figure 5: Qualitative effect of the SR stage on generator outputs, concentrated on the two content types that degrade first at the generator’s native resolution: small text and small faces. Smeared strokes on distant storefront signage become legible glyphs, and faces only a few dozen pixels across—including under motion—recover defined features rather than the coarse blobs the generator produces. Each cell shows the generator output with the inspected region boxed (left), followed by that region magnified before and after the SR stage.

	Cross-Shot Consistency			Video Quality		Text Consistency	Speech
Method	ViCLIP $\uparrow$	Self-CIDS $\uparrow$	Voice $\uparrow$	Aesthetic $\uparrow$	Imaging $\uparrow$	CLIP $\uparrow$	Recall $\uparrow$
JavisDiT++ [ 19 ]	0.6955	0.5621	0.7933	0.5019	0.6084	0.2522	0.0951
LTX-2.3 [ 10 ]	0.7047	0.6135	0.6847	0.5436	0.4751	0.2559	0.9304
LTX-2.5 [ 10 ]	0.7775	0.6640	0.7603	0.6019	0.5804	0.2683	0.9545
ShotStream+MMAudio [ 24 ]	0.7988	0.7308	0.7945	0.5255	0.6044	0.1998	0.0297
StoryMem+MMAudio [ 50 ]	0.7897	0.7491	0.7643	0.5265	0.6320	0.2368	0.0301

Table 2: Quantitative comparison of baseline methods. ViCLIP and Self-CIDS assess cross-shot visual and identity consistency, Voice assesses speaker consistency, and Recall measures speech-content coverage. All metrics are higher-is-better. Boldface and underlining indicate the best and second-best results, respectively.

Figure 6: Qualitative visualization of a 10-minute long audio-visual sequence generated under the Director Agent. The 10-minute example follows a wartime evacuation story set in a neutral port city. A radio operator helps two civilians escape as military control tightens and the city gradually descends into chaos. Their escape becomes increasingly entangled with checkpoints, shifting alliances, and the growing danger faced by civilians left behind. The radio operator eventually turns his broadcast network into a broader evacuation effort, connecting scattered groups and helping them find a route out of the city.

Figure 7: Qualitative comparison between JoyAI-Echo-1.5 and JoyAI-Echo-1.0 under the same long-form story. Frames sampled from temporally separated shots demonstrate the improved character consistency, scene stability, and visual rendering of JoyAI-Echo-1.5.

Aspect	JoyAI-Echo-1.5	Similar	HappyOyster
Story-instruction following	38.6%	43.8%	17.7%
Dialogue/background-audio following	39.2%	37.3%	23.5%
Memory/identity consistency	24.8%	59.5%	15.7%
Audio-visual synchronization	48.4%	31.4%	20.3%
Audio quality	44.4%	22.2%	33.3%

Table 3: Human evaluation based on Good–Similar–Bad (GSB) pairwise comparisons between JoyAI-Echo-1.5 (long-video variant) and HappyOyster for long-horizon audio-visual generation. Numbers denote the percentage of user preferences and may not sum to exactly 100% because of rounding.

Figure 8: Qualitative comparison of the undistilled bidirectional multi-step JoyAI-Echo-1.5 on representative WBench Navigation cases across diverse visual domains. Given the same initial observation and navigation condition, our model better preserves scene identity and structure while producing more coherent viewpoint evolution over successive interaction steps.

Figure 9: Multi-turn comparison of JoyAI-Echo-1.5-Causal against representative few-step autoregressive world models on WBench Navigation. All models start from the same initial observation and follow the prescribed multi-turn action sequence. The three cases use (1) A $\rightarrow$ A $\rightarrow$ A $\rightarrow$ A , (2) Left $\rightarrow$ Left $\rightarrow$ Right $\rightarrow$ Right , and (3) Left $\rightarrow$ W $\rightarrow$ Right . Here, Left , Right , Up , and Down denote camera rotations, while W , S , A , and D denote camera translations. JoyAI-Echo-1.5-Causal exhibits stronger action adherence and better preserves scene style, subject identity, and world structure across successive turns, with less accumulated autoregressive drift.

Figure 10: Causal long-horizon generation of JoyAI-Echo-1.5-Causal on representative cases. Each row shows a 60-second causal rollout sampled at different timestamps. Despite prolonged autoregressive generation, the distilled four-step causal model remains responsive to navigation controls while preserving scene identity, subject appearance, visual style, and major spatial structure, demonstrating robustness to long-horizon error accumulation.

Model	Avg.	Quality	Setting	Interaction	Consistency	Physical
JoyAI-Echo-1.5	81.7	81.5	79.4	87.2	89.8	70.6
JoyAI-Echo-1.5-Causal	81.0	81.1	88.3	87.9	77.5	70.1
HiDream-O1-World [ 48 ]	80.7	81.9	81.9	79.5	88.0	72.1
LingBot-World (fast v2) [ 30 ]	79.4	81.8	76.8	82.8	86.5	69.1
Kling 3.0 [ 48 ]	79.0	81.4	91.0	69.4	83.7	69.3
LingBot-World (base-camera) [ 30 ]	78.5	78.9	72.6	80.1	89.9	71.2

Table 4: Comparison on the 158-case WBench Navigation split [ 48 ] . JoyAI-Echo-1.5 denotes our undistilled bidirectional multi-step model, while JoyAI-Echo-1.5-Causal denotes its distilled four-step causal variant. Results are sorted by Average. Best and second-best results are shown in bold and underlined, respectively.

Simple-Trajectory Split
	Trajectory and Visual Quality
Method	VBench $\uparrow$	$R\downarrow$	$T\downarrow$	CMC $\downarrow$
Matrix-Game 3 [ 41 ]	79.41	4.474	0.406	0.453
LingBot-World-v2 [ 30 ]	81.13	3.092	0.282	0.310
HY-WorldPlay [ 36 ]	82.05	2.256	0.284	0.298
DreamX-World [ 6 ]	82.42	1.839	0.261	0.273

Table 5: Short-horizon comparison on SANA-WM-Bench using the first 241 frames of each trajectory. Each split contains 80 scenes. Best and second-best results within each split are shown in bold and underlined, respectively.

Simple-Trajectory Split
	Trajectory and Visual Quality				Revisit Consistency			Temporal Quality
Method	VBench $\uparrow$	$R\downarrow$	$T\downarrow$	CMC $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$\Delta$ IQ $\downarrow$
Infinite-World [ 44 ]	79.18	16.55	1.98	2.08	12.60	0.284	0.595	6.72
LingBot-World [ 30 ]	81.82	10.47	2.01	2.05	14.59	0.366	0.394	0.04
HY-WorldPlay [ 36 ]	68.82	17.89	2.36	2.45	12.83	0.321	0.616	23.59
Matrix-Game 3 [ 41 ]	78.53	12.96	1.83	1.92	12.29	0.326	0.553	2.41

Table 6: Long-horizon comparison on the full SANA-WM-Bench trajectory under the official 961-frame setting (60 seconds at 16 FPS). For JoyAI-Echo-1.5, we report the undistilled multi-step model. Best and second-best results within each split are shown in bold and underlined, respectively.

Simple-Trajectory Split
	Trajectory and Visual Quality				Revisit Consistency			Temporal Quality
Method	VBench $\uparrow$	$R\downarrow$	$T\downarrow$	CMC $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$\Delta$ IQ $\downarrow$
Evoke [ 47 ]	80.11	9.859	2.159	2.190	13.50	0.2774	0.5010	-0.49
LingBot-World-v2 [ 30 ]	79.20	22.516	2.391	2.541	11.90	0.1715	0.5906	2.59
SANA-WM + Causal Refiner [ 53 ]	79.55	17.875	2.222	2.338	14.41	0.2660	0.5748	-0.75
SANA-WM w/o Refiner [ 53 ]	78.21	15.359	2.288	2.393	9.19	0.1661	0.6321	-0.12

Table 7: Long-horizon comparison of distilled causal world models on SANA-WM-Bench under the 961-frame setting (60 seconds at 16 FPS). JoyAI-Echo-1.5-Causal denotes ours 4-step causal model. Best and second-best results within each split are shown in bold and underlined, respectively.

Figure 11: Imaging quality comparison on the Simple- and Hard-Trajectory splits. We report the first-window, mean, minimum, and final-window IQ. Our method achieves consistently higher mean and minimum IQ than Evoke and SANA-WM with Causal Refiner. Although the competing methods exhibit smaller IQ drops, their overall IQ remains lower, indicating that a small drop can also result from consistently low imaging quality rather than stronger long-horizon quality preservation.

Figure 12: Qualitative comparison of 961-frame (60-second at 16 FPS) causal rollouts on representative SANA-WM-Bench trajectories. We compare JoyAI-Echo-1.5-Causal with LingBot-World-v2 (LBW-V2), SANA-WM with and without Causal Refiner, and Evoke. Frames are sampled every 12 seconds from the same initial observation and camera-trajectory condition. Across the extended rollout, JoyAI-Echo-1.5-Causal exhibits less accumulated autoregressive drift, more consistent trajectory following, and stronger preservation of scene identity, visual style, and spatial structure.

Aspect	JoyAI-Echo-1.5	Similar	Happy Oyster
Semantic following	13.63%	56.94%	29.44%
Initial-world consistency	39.50%	45.06%	15.44%
Motion	18.94%	56.51%	24.56%
Spatiotemporal consistency	57.69%	31.88%	10.44%
Visual aesthetics	52.88%	23.32%	23.81%
Overall	63.13%	9.82%	27.06%

Table 8: Human evaluation based on Good–Similar–Bad (GSB) pairwise comparisons between JoyAI-Echo-1.5 (world model variant) and Happy Oyster for world modeling. Numbers denote the percentage of user preferences.

Aspect	JoyAI-Echo-1.5	Similar	LingBot-World-v2
Semantic following	34.00%	50.86%	15.14%
Initial-world consistency	14.64%	77.65%	7.71%
Motion	25.29%	54.71%	20.00%
Spatiotemporal consistency	32.64%	52.65%	14.71%
Visual aesthetics	32.50%	47.57%	19.93%
Overall	46.50%	31.92%	21.57%

Table 9: Human evaluation based on Good–Similar–Bad (GSB) pairwise comparisons between JoyAI-Echo-1.5 (world model variant) and LingBot-World-v2 for world modeling. Numbers denote the percentage of user preferences.

Audio-Video Generation

Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.

See all 157 papers

Sep 8, 2026

DramaAgent: Agentic Storytelling Video Generation

Recent diffusion and autoregressive models have substantially improved text-to-video generation, yet producing coherent long-form story videos with consistent characters and aligned audio remains challenging. Existing methods often suffer from narrative drift, unstable character identity, weak cross-scene continuity, and audio-visual mismatch over extended sequences. We propose DramaAgent, a hierarchical, agentic, and model-agnostic framework for long-form text-to-video-and-audio generation. Rather than improving the underlying video backbone itself, DramaAgent introduces an upper-level control layer that decomposes generation into story planning, persistent character conditioning, scene-wise synthesis, and reflection-guided targeted repair. The framework maintains reusable story and character states across scenes, diagnoses failures such as identity drift, missing scene semantics, temporal discontinuity, and cross-modal mismatch, and repairs problematic clips in a stage-specific manner. Experiments across multiple video generation backbones show that DramaAgent improves long-horizon coherence, character consistency, narrative fidelity, and scene-level audio-visual consistency over direct generation and strong baselines. These results suggest that hierarchical agentic control is a practical direction for controllable long-form audiovisual generation. Code: https://github.com/AIGeeksGroup/DramaAgent. Website: https://aigeeksgroup.github.io/DramaAgent.

Ting Huang, Biao Wu, Ronghao Chen +5

Peking University · UTS AAII · UOL +2

Video Generation Audio-Video Generation Agentic Video Generation Long-Horizon Video Generation Identity-Preserving Video Generation

Ask PDF

Jul 20, 2026

AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report

Unlike conventional video game development, which relies on labor-intensive pipelines for asset production, animation, physics, and programming, video world models generate interactive environments from user inputs instantly. It enable us to create customized, explorable, and continuously evolving virtual world from text, an image, or video. Realizing this vision requires four tightly coupled capabilities: interaction, persistent spatiotemporal consistency, stable long-horizon generation, and efficient response. We present AlayaWorld, an interactive long-horizon video world model that generates 24-fps video at 540p and 720p. Built on a 15B video diffusion transformer, AlayaWorld generates short latent chunks autoregressively under camera trajectories and switchable text prompts. Its bounded visual context combines a persistent sink frame, compressed temporal history, geometry-aligned spatial memory, and recent-frame conditioning. To reduce long-term drift, the model is trained with corrupted histories and prediction residuals collected from its own roll-outs. We further introduce a discrete autoregressive distillation formulation that combines distribution-matching distillation, self-forcing++, and consistency distillation, reducing inference from approximately 30 sampling steps to four steps per chunk. On iWorld-Bench, AlayaWorld achieves the best performance over long-horizon generation. Conceived as a full-stack, open-source, and long-term project, AlayaWorld is intended to provide an extensible foundation for future research on interactive video world models.

AlayaWorld Team, Kaipeng Zhang, Chuanhao Li +15

Alaya Lab

Video Diffusion Models Real-Time Interactive Video Generation Long-Horizon Video Generation Memory-Augmented Video Generation Autoregressive Video Generation Interactive World Models

Ask PDF

May 15, 2026

Echo-Forcing: A Scene Memory Framework for Interactive Long Video Generation

Autoregressive video diffusion models enable open-ended generation through local attention and KV caching. However, existing training-free long-video optimization methods mainly focus on stable extension under a single prompt, making them difficult to handle interactive scenarios involving prompt switching, old scene forgetting, and historical scene recall. We identify the core bottleneck as the functional entanglement of historical KV states: stable anchors and recent dynamics are handled by the same cache policy, leading to outdated background contamination, delayed response to new prompts, and loss of long-range memory. To address this issue, we propose Echo-Forcing, a training-free scene memory framework specifically designed for interactive long video generation with three core mechanisms: (1) Hierarchical Temporal Memory, which decouples stable anchors, compressed history, and recent windows under relative RoPE; (2) Scene Recall Frames, which compresses historical scenes into spatially structured KV representations to support long-term recall; and (3) Difference-aware Memory Decay, which adaptively forgets conflicting tokens according to the discrepancy between old and new scenes. Based on these designs, Echo-Forcing uniformly supports smooth transitions, hard cuts, and long-range scene recall under a bounded cache budget. Extensive evaluations on VBench-Long further demonstrate that Echo-Forcing achieves the best overall performance in both long-video generation and interactive video generation settings. Our code is released in https://github.com/mingqiangWu/Echo-Forcing

Mingqiang Wu, Weilun Feng, Zhefeng Zhang +8

State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · China University of Mining & Technology, Beijing +3

Video Diffusion Models Long-Horizon Video Generation Memory-Augmented Video Generation Video Generation Autoregressive Video Generation

Ask PDF

Abstract

Audio-Video Generation

Explore similar work

DramaAgent: Agentic Storytelling Video Generation

AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report

Echo-Forcing: A Scene Memory Framework for Interactive Long Video Generation