Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds

Nan Duan; Haoyang Huang; Weiyang Jin; Haoran Li; Yaowei Li; Yuming Li; Yijun Liu; Xin Lu

cs.CVAug 24, 2026

Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds

Authors: Nan Duan, Haoyang Huang, Weiyang Jin, Haoran Li, Yaowei Li, Yuming Li, Yijun Liu, Xin Lu,

Organizations: Joy Future Academy, JD

Abstract

Video generation is progressing beyond isolated clips toward long-form narratives and interactive worlds, requiring models to preserve identities, follow user controls, and remain stable over extended rollouts. We present JoyAI-Echo-1.5, a unified audio-visual generation system with two purpose-built variants. The long-video variant introduces composable cross-shot memory that aggregates visual evidence across multiple prior shots and speaker cues derived from speech-filtered full-shot audio, enabling persistent character appearance and voice identity across flexible combinations of text, image, and memory conditioning. The world-model variant converts heterogeneous navigation inputs into calibrated metric 6-DoF camera trajectories and injects them through a geometry-aware conditioning pathway, enabling controller-agnostic interaction across flexible viewpoints. To support efficient long-horizon generation, we transform a bidirectional audio-visual backbone into a causal few-step generator using progressive teacher forcing and short- and long-horizon Self-Gradient Forcing on self-generated rollouts. Experiments demonstrate strong performance in both settings. JoyAI-Echo-1.5 achieves improvements over existing long-video baselines in cross-shot consistency, visual quality, text alignment, and speech fidelity. Its world-model variant ranks first on WBench, with an average score of 81.7, and achieves leading visual quality and long-horizon persistence on SANA-WM-Bench. Together, these results indicate that memory, geometric control, and rollout-aware training provide a practical foundation for generating coherent stories and continuously evolving interactive worlds. Project page: https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/.

Figures & tables

Figure 1: Two-stage data construction pipeline of JoyAI-Echo-1.5. Raw videos are quality-filtered and organized into an identity-centric Memory corpus and a high-quality corpus with broader resolution, language, and multi-shot coverage.

Structure	T2V caption	I2V caption
Single shot	ID_A is a young man with short brown hair, wearing a dark gray hoodie, with a clear mid-range voice. He sits slightly off-camera, blinks naturally, and says, “my games often focus on that …” A static chest-up close-up uses shallow depth of field in an indoor gym. No on-screen text or subtitles. No obvious sound effects. No background music.	[Random I2V prefix] A short-haired man speaks with a clear, stable mid-range voice and a natural conversational tone. He sits slightly off-camera, blinks naturally, and says, “my games often focus on that …” A static chest-up close-up keeps him in focus against a softly blurred gym. No on-screen text or subtitles. No obvious sound effects. No background music.
Multi-shot	This video contains two shots. Shot 1: ID_A is a middle-aged man in a dark green dress uniform with a deep voice; ID_B is a middle-aged man in digital camouflage with a bright voice. They walk and talk side by side as a medium shot follows them. OCR: military slogans and a name label. Soft piano music plays. Shot 2: ID_C is a silent middle-aged man in a dark blue police uniform. A police formation advances in a static medium shot. No OCR or subtitles; the piano music continues.	[Random I2V prefix] This video contains two shots. Shot 1: A middle-aged man in a dress uniform and a middle-aged man in camouflage walk and talk side by side. The former has a deep, steady voice and the latter a bright, natural voice. A medium shot follows them; soft piano music plays. Shot 2: A serious middle-aged man in a dark blue police uniform appears silently within an advancing formation. A static medium shot follows the group. No OCR or subtitles; the piano music continues.

Table 1: Representative abbreviated examples of the four caption formats used in JoyAI-Echo-1.5. English renderings are shown for readability, while both Chinese and English captions are retained during training. T2V captions preserve explicit identity definitions, whereas I2V captions use concise natural references because appearance is supplied by the first frame.

Figure 2: Overview of memory-conditioned long-form generation and few-step acceleration in JoyAI-Echo-1.5. Top: unified conditioning with single-frame visual memories, speech-filtered audio memories, and separated RoPE coordinates supports multiple audio-visual generation tasks. Bottom: memory-robust joint DMD combines asymmetric video–audio optimization with audio-specific adversarial and energy constraints.

Figure 3: Illustration of the causal attention masks used in streaming post-training. (a) Audio-visual teacher forcing, where each noisy chunk attends to its preceding clean history and the current noisy chunk, while the clean target and future chunks are masked out. (b) Long-horizon SGF Forward 2, where the causal context is further restricted by the sink-plus-FIFO KV cache.

Figure 4: Overview of the Director Agent workflow. Given a user request and optional reference, the agent performs shot planning and prompt enhancement (PE), and supports both single-shot short-video generation and iterative long-video generation through memory retrieval, shot-level review and revision, and memory admission.

Figure 5: Qualitative effect of the SR stage on generator outputs, concentrated on the two content types that degrade first at the generator’s native resolution: small text and small faces. Smeared strokes on distant storefront signage become legible glyphs, and faces only a few dozen pixels across—including under motion—recover defined features rather than the coarse blobs the generator produces. Each cell shows the generator output with the inspected region boxed (left), followed by that region magnified before and after the SR stage.

	Cross-Shot Consistency			Video Quality		Text Consistency	Speech
Method	ViCLIP $\uparrow$	Self-CIDS $\uparrow$	Voice $\uparrow$	Aesthetic $\uparrow$	Imaging $\uparrow$	CLIP $\uparrow$	Recall $\uparrow$
JavisDiT++ [ 19 ]	0.6955	0.5621	0.7933	0.5019	0.6084	0.2522	0.0951
LTX-2.3 [ 10 ]	0.7047	0.6135	0.6847	0.5436	0.4751	0.2559	0.9304
LTX-2.5 [ 10 ]	0.7775	0.6640	0.7603	0.6019	0.5804	0.2683	0.9545
ShotStream+MMAudio [ 24 ]	0.7988	0.7308	0.7945	0.5255	0.6044	0.1998	0.0297
StoryMem+MMAudio [ 50 ]	0.7897	0.7491	0.7643	0.5265	0.6320	0.2368	0.0301

Table 2: Quantitative comparison of baseline methods. ViCLIP and Self-CIDS assess cross-shot visual and identity consistency, Voice assesses speaker consistency, and Recall measures speech-content coverage. All metrics are higher-is-better. Boldface and underlining indicate the best and second-best results, respectively.

Figure 6: Qualitative visualization of a 10-minute long audio-visual sequence generated under the Director Agent. The 10-minute example follows a wartime evacuation story set in a neutral port city. A radio operator helps two civilians escape as military control tightens and the city gradually descends into chaos. Their escape becomes increasingly entangled with checkpoints, shifting alliances, and the growing danger faced by civilians left behind. The radio operator eventually turns his broadcast network into a broader evacuation effort, connecting scattered groups and helping them find a route out of the city.

Figure 7: Qualitative comparison between JoyAI-Echo-1.5 and JoyAI-Echo-1.0 under the same long-form story. Frames sampled from temporally separated shots demonstrate the improved character consistency, scene stability, and visual rendering of JoyAI-Echo-1.5.

Aspect	JoyAI-Echo-1.5	Similar	HappyOyster
Story-instruction following	38.6%	43.8%	17.7%
Dialogue/background-audio following	39.2%	37.3%	23.5%
Memory/identity consistency	24.8%	59.5%	15.7%
Audio-visual synchronization	48.4%	31.4%	20.3%
Audio quality	44.4%	22.2%	33.3%

Table 3: Human evaluation based on Good–Similar–Bad (GSB) pairwise comparisons between JoyAI-Echo-1.5 (long-video variant) and HappyOyster for long-horizon audio-visual generation. Numbers denote the percentage of user preferences and may not sum to exactly 100% because of rounding.

Figure 8: Qualitative comparison of the undistilled bidirectional multi-step JoyAI-Echo-1.5 on representative WBench Navigation cases across diverse visual domains. Given the same initial observation and navigation condition, our model better preserves scene identity and structure while producing more coherent viewpoint evolution over successive interaction steps.

Figure 9: Multi-turn comparison of JoyAI-Echo-1.5-Causal against representative few-step autoregressive world models on WBench Navigation. All models start from the same initial observation and follow the prescribed multi-turn action sequence. The three cases use (1) A $\rightarrow$ A $\rightarrow$ A $\rightarrow$ A , (2) Left $\rightarrow$ Left $\rightarrow$ Right $\rightarrow$ Right , and (3) Left $\rightarrow$ W $\rightarrow$ Right . Here, Left , Right , Up , and Down denote camera rotations, while W , S , A , and D denote camera translations. JoyAI-Echo-1.5-Causal exhibits stronger action adherence and better preserves scene style, subject identity, and world structure across successive turns, with less accumulated autoregressive drift.

Figure 10: Causal long-horizon generation of JoyAI-Echo-1.5-Causal on representative cases. Each row shows a 60-second causal rollout sampled at different timestamps. Despite prolonged autoregressive generation, the distilled four-step causal model remains responsive to navigation controls while preserving scene identity, subject appearance, visual style, and major spatial structure, demonstrating robustness to long-horizon error accumulation.

Model	Avg.	Quality	Setting	Interaction	Consistency	Physical
JoyAI-Echo-1.5	81.7	81.5	79.4	87.2	89.8	70.6
JoyAI-Echo-1.5-Causal	81.0	81.1	88.3	87.9	77.5	70.1
HiDream-O1-World [ 48 ]	80.7	81.9	81.9	79.5	88.0	72.1
LingBot-World (fast v2) [ 30 ]	79.4	81.8	76.8	82.8	86.5	69.1
Kling 3.0 [ 48 ]	79.0	81.4	91.0	69.4	83.7	69.3
LingBot-World (base-camera) [ 30 ]	78.5	78.9	72.6	80.1	89.9	71.2

Table 4: Comparison on the 158-case WBench Navigation split [ 48 ] . JoyAI-Echo-1.5 denotes our undistilled bidirectional multi-step model, while JoyAI-Echo-1.5-Causal denotes its distilled four-step causal variant. Results are sorted by Average. Best and second-best results are shown in bold and underlined, respectively.

Simple-Trajectory Split
	Trajectory and Visual Quality
Method	VBench $\uparrow$	$R\downarrow$	$T\downarrow$	CMC $\downarrow$
Matrix-Game 3 [ 41 ]	79.41	4.474	0.406	0.453
LingBot-World-v2 [ 30 ]	81.13	3.092	0.282	0.310
HY-WorldPlay [ 36 ]	82.05	2.256	0.284	0.298
DreamX-World [ 6 ]	82.42	1.839	0.261	0.273

Table 5: Short-horizon comparison on SANA-WM-Bench using the first 241 frames of each trajectory. Each split contains 80 scenes. Best and second-best results within each split are shown in bold and underlined, respectively.

Simple-Trajectory Split
	Trajectory and Visual Quality				Revisit Consistency			Temporal Quality
Method	VBench $\uparrow$	$R\downarrow$	$T\downarrow$	CMC $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$\Delta$ IQ $\downarrow$
Infinite-World [ 44 ]	79.18	16.55	1.98	2.08	12.60	0.284	0.595	6.72
LingBot-World [ 30 ]	81.82	10.47	2.01	2.05	14.59	0.366	0.394	0.04
HY-WorldPlay [ 36 ]	68.82	17.89	2.36	2.45	12.83	0.321	0.616	23.59
Matrix-Game 3 [ 41 ]	78.53	12.96	1.83	1.92	12.29	0.326	0.553	2.41

Table 6: Long-horizon comparison on the full SANA-WM-Bench trajectory under the official 961-frame setting (60 seconds at 16 FPS). For JoyAI-Echo-1.5, we report the undistilled multi-step model. Best and second-best results within each split are shown in bold and underlined, respectively.

Simple-Trajectory Split
	Trajectory and Visual Quality				Revisit Consistency			Temporal Quality
Method	VBench $\uparrow$	$R\downarrow$	$T\downarrow$	CMC $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$\Delta$ IQ $\downarrow$
Evoke [ 47 ]	80.11	9.859	2.159	2.190	13.50	0.2774	0.5010	-0.49
LingBot-World-v2 [ 30 ]	79.20	22.516	2.391	2.541	11.90	0.1715	0.5906	2.59
SANA-WM + Causal Refiner [ 53 ]	79.55	17.875	2.222	2.338	14.41	0.2660	0.5748	-0.75
SANA-WM w/o Refiner [ 53 ]	78.21	15.359	2.288	2.393	9.19	0.1661	0.6321	-0.12

Table 7: Long-horizon comparison of distilled causal world models on SANA-WM-Bench under the 961-frame setting (60 seconds at 16 FPS). JoyAI-Echo-1.5-Causal denotes ours 4-step causal model. Best and second-best results within each split are shown in bold and underlined, respectively.

Figure 11: Imaging quality comparison on the Simple- and Hard-Trajectory splits. We report the first-window, mean, minimum, and final-window IQ. Our method achieves consistently higher mean and minimum IQ than Evoke and SANA-WM with Causal Refiner. Although the competing methods exhibit smaller IQ drops, their overall IQ remains lower, indicating that a small drop can also result from consistently low imaging quality rather than stronger long-horizon quality preservation.

Figure 12: Qualitative comparison of 961-frame (60-second at 16 FPS) causal rollouts on representative SANA-WM-Bench trajectories. We compare JoyAI-Echo-1.5-Causal with LingBot-World-v2 (LBW-V2), SANA-WM with and without Causal Refiner, and Evoke. Frames are sampled every 12 seconds from the same initial observation and camera-trajectory condition. Across the extended rollout, JoyAI-Echo-1.5-Causal exhibits less accumulated autoregressive drift, more consistent trajectory following, and stronger preservation of scene identity, visual style, and spatial structure.

Aspect	JoyAI-Echo-1.5	Similar	Happy Oyster
Semantic following	13.63%	56.94%	29.44%
Initial-world consistency	39.50%	45.06%	15.44%
Motion	18.94%	56.51%	24.56%
Spatiotemporal consistency	57.69%	31.88%	10.44%
Visual aesthetics	52.88%	23.32%	23.81%
Overall	63.13%	9.82%	27.06%

Table 8: Human evaluation based on Good–Similar–Bad (GSB) pairwise comparisons between JoyAI-Echo-1.5 (world model variant) and Happy Oyster for world modeling. Numbers denote the percentage of user preferences.

Aspect	JoyAI-Echo-1.5	Similar	LingBot-World-v2
Semantic following	34.00%	50.86%	15.14%
Initial-world consistency	14.64%	77.65%	7.71%
Motion	25.29%	54.71%	20.00%
Spatiotemporal consistency	32.64%	52.65%	14.71%
Visual aesthetics	32.50%	47.57%	19.93%
Overall	46.50%	31.92%	21.57%

Table 9: Human evaluation based on Good–Similar–Bad (GSB) pairwise comparisons between JoyAI-Echo-1.5 (world model variant) and LingBot-World-v2 for world modeling. Numbers denote the percentage of user preferences.

Audio-Video Generation

Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.

See all 157 papers

DramaAgent: Agentic Storytelling Video Generation
Sep 8, 2026Ting Huang, Biao Wu, Ronghao Chen +5Video Generation Audio-Video Generation
Peking University · UTS AAII · UOL +2
AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report
Jul 20, 2026AlayaWorld Team, Kaipeng Zhang, Chuanhao Li +15Video Diffusion Models Real-Time Interactive Video Generation
Alaya Lab
Echo-Forcing: A Scene Memory Framework for Interactive Long Video Generation
May 15, 2026Mingqiang Wu, Weilun Feng, Zhefeng Zhang +8Video Diffusion Models Long-Horizon Video Generation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · China University of Mining & Technology, Beijing +3

Sep 8, 2026J/K move · Enter open · S save

DramaAgent: Agentic Storytelling Video Generation

Ting Huang, Biao Wu, Ronghao Chen, Zeyu Zhang+4

Peking University · UTS AAII · UOL+2

Video Generation Audio-Video Generation Agentic Video Generation Long-Horizon Video Generation Identity-Preserving Video Generation

Open paper AskPDF arXiv

Recent diffusion and autoregressive models have substantially improved text-to-video generation, yet producing coherent long-form story videos with consistent characters and aligned audio remains challenging. Existing methods often suffer from narrative drift, unstable character identity, weak cross-scene continuity, and audio-visual mismatch over extended sequences. We propose DramaAgent, a hierarchical, agentic, and model-agnostic framework for long-form text-to-video-and-audio generation. Rather than improving the underlying video backbone itself, DramaAgent introduces an upper-level control layer that decomposes generation into story planning, persistent character conditioning, scene-wise synthesis, and reflection-guided targeted repair. The framework maintains reusable story and character states across scenes, diagnoses failures such as identity drift, missing scene semantics, temporal discontinuity, and cross-modal mismatch, and repairs problematic clips in a stage-specific manner. Experiments across multiple video generation backbones show that DramaAgent improves long-horizon coherence, character consistency, narrative fidelity, and scene-level audio-visual consistency over direct generation and strong baselines. These results suggest that hierarchical agentic control is a practical direction for controllable long-form audiovisual generation. Code: https://github.com/AIGeeksGroup/DramaAgent. Website: https://aigeeksgroup.github.io/DramaAgent.

Abstract

Audio-Video Generation

Explore similar work

DramaAgent: Agentic Storytelling Video Generation

AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report

Echo-Forcing: A Scene Memory Framework for Interactive Long Video Generation