Real-Time Joint Audio-Video Generation by Parallel Adapter Composition

Jingyu Li; Xiaoxiao Xiang; Yiwen Guo

cs.CVOct 7, 2026

Real-Time Joint Audio-Video Generation by Parallel Adapter Composition

Authors: Jingyu Li, Xiaoxiao Xiang, Yiwen Guo

Organizations: LIGHTSPEED · Independent Researcher

Abstract

Deploying a joint audio-video diffusion transformer for real-time, interactive generation normally requires two essential modifications: block-autoregressive attention, so frames can be emitted before the whole clip is finished, and few-step sampling, so each block is cheap. Conventionally, the streaming video literature obtains both capabilities from a chained pipeline. It first distills a bidirectional teacher into a causal student, then into a few-step one, or proceeds in reverse order. Each stage of such a chain fine-tunes the weights the previous one produced, so a later objective can undo an earlier capability. Following the idea of model merging, we show that on a packed audio-video backbone the two capabilities can be acquired in parallel. A causal adapter is trained against the frozen backbone, and an off-the-shelf few-step adapter provides the few-step capability. As the two edit different functional axes, we predict, and then verify, that their weight-update directions are near-orthogonal, without any explicit orthogonality constraint during training. Orthogonal updates should combine without interfering, so parallel composition is a direct sum. The two adapters are simply added at inference, with no joint training, yielding few-step, streaming audio-video whose image quality tracks the bidirectional teacher. Compared to the chained baselines, the composed model matches or beats them on most metrics, making parallel composition a practical approach. The resulting streaming system generates joint audio-video in real time, $\approx$ 26 fps at $480\times832$ without quantization, and sustains 30 s of continuous generation with stable image quality.

Figures & tables

Figure 1: Three routes to a few-step streaming model, and what the order costs. Distilling last keeps the chain viable, at two extra stages; the reverse order, trained with a plain regression loss, un-distills. The mechanisms are in § 3.1 .

	Teacher	Whole-seg	Decoupled	Decoupled	Chained-DA	Chained-AD
	$50$ step	$4$ step	$4$ step (ours)	$50$ step	$4$ step	$4$ step
Sync-C $\uparrow$	$8.412$	$8.403$	$7.443$	$7.197$	$6.389$	$\mathbf{8.021}$
AV-IB $\uparrow$	$0.414$	$0.414$	$0.403$	$\mathbf{0.423}$	$0.363$	$0.419$
IQA $\uparrow$	$0.908$	$0.915$	$\mathbf{0.914}$	$0.894$	$0.854$	$0.877$
ASE $\uparrow$	$5.048$	$5.110$	$\mathbf{5.106}$	$5.023$	$5.019$	$5.008$
CLIP $\uparrow$	$0.212$	$0.212$	$\mathbf{0.212}$	$0.210$	$\mathbf{0.212}$	$\mathbf{0.212}$

Table 1: Results on the held-out benchmark (§ 4.1 ). Drift rows are last window over first; CSIM-1st is identity against the clip’s first frame. Speech is transcribed with Qwen3-ASR-1.7B: WER for the English clips, CER for the Chinese ones. Among the four autoregressive systems, best results in bold, second-best underlined.

Figure 2: One clip streamed to $30$ s. Rows are systems and columns are wall-clock seconds; below them the four generated soundtracks share one time axis. Chained-DA stops articulating well before the end while the composed routes speak throughout; Table 2 reports the same comparison over all forty clips.

Figure 3: Long-horizon consistency as a function of generation time. (a) FaceNet similarity and (b) CLIP Image similarity to the first frame of the same video. Curves are means over

40

clips with

95\%

confidence intervals, and terminate at each method’s actual duration.

Method	Sync-C $\uparrow$	Identity $\uparrow$	CLIP $\uparrow$	IQA $\uparrow$	Voice $\uparrow$	RMS
Decoupled	5.69 ( $-$ 29.6%)	0.710 ( $-$ 18.3%)	0.844 ( $-$ 9.7%)	0.871 ( $+$ 2.6%)	0.521 ( $-$ 41.0%)	0.086 ( $-$ 10.6%)
Decoupled-50	6.23 ( $-$ 19.1%)	0.687 ( $-$ 18.9%)	0.856 ( $-$ 8.2%)	0.848 ( $+$ 3.0%)	0.682 ( $-$ 22.3%)	0.109 ( $-$ 16.6%)
Chained-AD	7.82 ( $-$ 12.8%)	0.484 ( $-$ 41.8%)	0.737 ( $-$ 19.7%)	0.526 ( $-$ 35.4%)	0.608 ( $-$ 31.9%)	0.185 ( $+$ 49.0%)
Chained-DA	3.75 ( $-$ 43.7%)	0.683 ( $-$ 21.4%)	0.841 ( $-$ 10.5%)	0.656 ( $-$ 16.2%)	0.603 ( $-$ 32.3%)	0.047 ( $-$ 37.9%)

Table 2: Long-horizon preservation over a

40

-clip,

30

s rollout. Each cell is the score on the final

5

s and, in parentheses, its relative change from the opening

5

s. Both windows are the same length, so Sync-C is comparable between them but not with a full-clip Sync-C. Bold marks the best closing window; for RMS, which is an absolute level, it marks the smallest relative change instead.

	Decoupled	Chained-DA	Chained-AD	null
Projection	$+0.0006\pm 0.0055$	$-0.0022\pm 0.0084$	$-0.0008\pm 0.0349$	$+0.0000\pm 0.0008$
Write interference	$0.0147\pm 0.0126$	$0.0169\pm 0.0159$	$0.0451\pm 0.0308$	$0.0117\pm 0.0042$
Read interference	$0.1356\pm 0.1058$	$0.1061\pm 0.1027$	$0.1844\pm 0.1094$	$0.0237\pm 0.0180$

Table 3: Geometry of each causal update against each distillation update, over all shared (block, projection) pairs, mean

\pm

s.d. Decoupled and Chained-DA measure

\Delta_{d}

against

\Delta_{c}

and

\Delta_{c}^{\prime}

; Chained-AD measures

\Delta_{c}

against its two distillation increments taken together (

\Delta_{cd}\oplus\Delta_{dmd}

). Interference and projection are defined in Appendix F . Projection is a signed alignment, zero meaning none; both interference forms vanish for orthogonal subspaces, so for all three quantities the null column is the value to read against rather than zero.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Attention pattern of Eq. 2 on a three-chunk packed sequence. Clean tokens attend same-or-earlier clean tokens; a noisy chunk attends its own noisy tokens and strictly earlier clean history, never its clean copy. Conditioning is chunk

-1

: visible as a key to every content row, but a conditioning query cannot read later clean tokens. Audio is drawn chunkwise, as video is.

AVSpeech ( $n{=}100$ )
	IQA $\uparrow$	Sync-C $\uparrow$	Sync-D $\downarrow$	AV-IB $\uparrow$	DeSync $\downarrow$	CSIM-r $\uparrow$	FID $\downarrow$	FVD $\downarrow$	FAD $\downarrow$
GT (source)	0.852	6.82	7.77	0.377	0.115	0.867	—	—	—
Teacher	0.857	8.49	7.26	0.333	0.028	0.839	32.2	128.5	6.85
Whole-seg	0.835	7.82	7.62	0.328	0.040	0.864	33.7	119.2	5.96
Decoupled	0.866	7.54	7.85	0.334	0.007	0.791	35.2	129.0	5.23
Decoupled-50	0.836	7.13	7.99	0.324	0.028	0.761	38.9	143.6	5.34

Appendix

Table 4: The systems of Table 1 on three public test sets (

100

clips each), scored against the real source video — so Teacher is a system under test, not the reference. Among the four autoregressive systems, best results in bold, second-best underlined, per column and set; the GT row is a reference, not a competitor. CSIM-r is face identity against the reference portrait, not CSIM-1st. FAD is omitted on HDTF: clip-length mismatch dominates there (whole-segment generation has a

4

s minimum against

{\sim}3.2

s sources); DeSync is undefined on HDTF (clips shorter than Synchformer’s

4.8

s context).

Figure 5: Identity drift over the

30

s long-horizon runs of § 4.2 (

40

clips, probes every

5

s). (a) Face similarity to the reference portrait, with ArcFace. (b) Speaker similarity to the clip’s own opening segment, with CAM++. Both are the complement of Figure 3 , which measures against each clip’s own first frame under different encoders. Face identity is retained equally well by Decoupled , Decoupled-50 and Chained-DA (and worst by Chained-AD ); speaker identity falls furthest for Decoupled , the observation § 5 reports as a limitation.

Explore similar work

Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory
Jul 29, 2026Yanbo Ding, Zhizhi Guo, Quanyue Song +4Audio-Video Generation Long Video Generation
Shenzhen Key Lab of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China · School of Artificial Intelligence, University of Chinese Academy of Sciences +2
Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation
Apr 26, 2026Chunyu Li, Jiaye Li, Ruiqiao Mei +4Audio-Video Generation Video Diffusion Models
Shanghai Innovation Institute · Fudan University · University of Science and Technology of China +2
DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution
Aug 31, 2026Jiashu Zhu, Yanhao Zheng, Ruitian Tian +7Audio-Video Generation Video Diffusion Models
DreamX Team, Alibaba Group

Jul 29, 2026, cs.CVJ/K move · Enter open · S save

Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory

Yanbo Ding, Zhizhi Guo, Quanyue Song, Yishan He+3

Shenzhen Key Lab of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China · School of Artificial Intelligence, University of Chinese Academy of Sciences+2

Audio-Video Generation Long Video Generation Memory-Augmented Video Generation Streaming Video Generation

Open paper AskPDF arXiv

Audio-video generative models achieve impressive quality but suffer from high latency, making them unsuitable for real-time applications. Although several streaming audio-video generation methods have been proposed, they remain costly and fail to support long-form generation. To address this, we propose \textbf{Ripple}, a real-time joint audio-video generation system with a cross-modal recurrent memory mechanism. To enable efficient streaming inference while preserving long-term context, Ripple combines a fixed-length sliding-window attention with modality-specific memory states that continuously summarize audio and video context. Cross-modal memory interaction is further introduced to enhance audio-visual synchronization. To learn this memory-augmented model effectively, we devise a three-stage training recipe: (1) adapting a bidirectional audio-video teacher to block-wise causal attention with simulated memory, (2) optimizing the memory construction and interaction pipeline through end-to-end distillation, and (3) applying online reinforcement post-training tailored for streaming audio-video generation. As a result, Ripple achieves ~28 FPS at 480P resolution, over faster than the teacher, while capable of coherent long-form generation. Extensive experiments on both short-video and long-video benchmarks demonstrate our superior performance over existing offline and online joint audio-video generation methods.

Abstract

Explore similar work

Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory

Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation

DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution