Real-Time Joint Audio-Video Generation by Parallel Adapter Composition

Jingyu Li; Xiaoxiao Xiang; Yiwen Guo

cs.CVOct 7, 2026

Real-Time Joint Audio-Video Generation by Parallel Adapter Composition

Authors: Jingyu Li, Xiaoxiao Xiang, Yiwen Guo

Organizations: LIGHTSPEED · Independent Researcher

Abstract

Deploying a joint audio-video diffusion transformer for real-time, interactive generation normally requires two essential modifications: block-autoregressive attention, so frames can be emitted before the whole clip is finished, and few-step sampling, so each block is cheap. Conventionally, the streaming video literature obtains both capabilities from a chained pipeline. It first distills a bidirectional teacher into a causal student, then into a few-step one, or proceeds in reverse order. Each stage of such a chain fine-tunes the weights the previous one produced, so a later objective can undo an earlier capability. Following the idea of model merging, we show that on a packed audio-video backbone the two capabilities can be acquired in parallel. A causal adapter is trained against the frozen backbone, and an off-the-shelf few-step adapter provides the few-step capability. As the two edit different functional axes, we predict, and then verify, that their weight-update directions are near-orthogonal, without any explicit orthogonality constraint during training. Orthogonal updates should combine without interfering, so parallel composition is a direct sum. The two adapters are simply added at inference, with no joint training, yielding few-step, streaming audio-video whose image quality tracks the bidirectional teacher. Compared to the chained baselines, the composed model matches or beats them on most metrics, making parallel composition a practical approach. The resulting streaming system generates joint audio-video in real time, $\approx$ 26 fps at $480\times832$ without quantization, and sustains 30 s of continuous generation with stable image quality.

Figures & tables

Figure 1: Three routes to a few-step streaming model, and what the order costs. Distilling last keeps the chain viable, at two extra stages; the reverse order, trained with a plain regression loss, un-distills. The mechanisms are in § 3.1 .

	Teacher	Whole-seg	Decoupled	Decoupled	Chained-DA	Chained-AD
	$50$ step	$4$ step	$4$ step (ours)	$50$ step	$4$ step	$4$ step
Sync-C $\uparrow$	$8.412$	$8.403$	$7.443$	$7.197$	$6.389$	$\mathbf{8.021}$
AV-IB $\uparrow$	$0.414$	$0.414$	$0.403$	$\mathbf{0.423}$	$0.363$	$0.419$
IQA $\uparrow$	$0.908$	$0.915$	$\mathbf{0.914}$	$0.894$	$0.854$	$0.877$
ASE $\uparrow$	$5.048$	$5.110$	$\mathbf{5.106}$	$5.023$	$5.019$	$5.008$
CLIP $\uparrow$	$0.212$	$0.212$	$\mathbf{0.212}$	$0.210$	$\mathbf{0.212}$	$\mathbf{0.212}$

Table 1: Results on the held-out benchmark (§ 4.1 ). Drift rows are last window over first; CSIM-1st is identity against the clip’s first frame. Speech is transcribed with Qwen3-ASR-1.7B: WER for the English clips, CER for the Chinese ones. Among the four autoregressive systems, best results in bold, second-best underlined.

Figure 2: One clip streamed to $30$ s. Rows are systems and columns are wall-clock seconds; below them the four generated soundtracks share one time axis. Chained-DA stops articulating well before the end while the composed routes speak throughout; Table 2 reports the same comparison over all forty clips.

Figure 3: Long-horizon consistency as a function of generation time. (a) FaceNet similarity and (b) CLIP Image similarity to the first frame of the same video. Curves are means over

40

clips with

95\%

confidence intervals, and terminate at each method’s actual duration.

Method	Sync-C $\uparrow$	Identity $\uparrow$	CLIP $\uparrow$	IQA $\uparrow$	Voice $\uparrow$	RMS
Decoupled	5.69 ( $-$ 29.6%)	0.710 ( $-$ 18.3%)	0.844 ( $-$ 9.7%)	0.871 ( $+$ 2.6%)	0.521 ( $-$ 41.0%)	0.086 ( $-$ 10.6%)
Decoupled-50	6.23 ( $-$ 19.1%)	0.687 ( $-$ 18.9%)	0.856 ( $-$ 8.2%)	0.848 ( $+$ 3.0%)	0.682 ( $-$ 22.3%)	0.109 ( $-$ 16.6%)
Chained-AD	7.82 ( $-$ 12.8%)	0.484 ( $-$ 41.8%)	0.737 ( $-$ 19.7%)	0.526 ( $-$ 35.4%)	0.608 ( $-$ 31.9%)	0.185 ( $+$ 49.0%)
Chained-DA	3.75 ( $-$ 43.7%)	0.683 ( $-$ 21.4%)	0.841 ( $-$ 10.5%)	0.656 ( $-$ 16.2%)	0.603 ( $-$ 32.3%)	0.047 ( $-$ 37.9%)

Table 2: Long-horizon preservation over a

40

-clip,

30

s rollout. Each cell is the score on the final

5

s and, in parentheses, its relative change from the opening

5

s. Both windows are the same length, so Sync-C is comparable between them but not with a full-clip Sync-C. Bold marks the best closing window; for RMS, which is an absolute level, it marks the smallest relative change instead.

	Decoupled	Chained-DA	Chained-AD	null
Projection	$+0.0006\pm 0.0055$	$-0.0022\pm 0.0084$	$-0.0008\pm 0.0349$	$+0.0000\pm 0.0008$
Write interference	$0.0147\pm 0.0126$	$0.0169\pm 0.0159$	$0.0451\pm 0.0308$	$0.0117\pm 0.0042$
Read interference	$0.1356\pm 0.1058$	$0.1061\pm 0.1027$	$0.1844\pm 0.1094$	$0.0237\pm 0.0180$

Table 3: Geometry of each causal update against each distillation update, over all shared (block, projection) pairs, mean

\pm

s.d. Decoupled and Chained-DA measure

\Delta_{d}

against

\Delta_{c}

and

\Delta_{c}^{\prime}

; Chained-AD measures

\Delta_{c}

against its two distillation increments taken together (

\Delta_{cd}\oplus\Delta_{dmd}

). Interference and projection are defined in Appendix F . Projection is a signed alignment, zero meaning none; both interference forms vanish for orthogonal subspaces, so for all three quantities the null column is the value to read against rather than zero.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Attention pattern of Eq. 2 on a three-chunk packed sequence. Clean tokens attend same-or-earlier clean tokens; a noisy chunk attends its own noisy tokens and strictly earlier clean history, never its clean copy. Conditioning is chunk

-1

: visible as a key to every content row, but a conditioning query cannot read later clean tokens. Audio is drawn chunkwise, as video is.

AVSpeech ( $n{=}100$ )
	IQA $\uparrow$	Sync-C $\uparrow$	Sync-D $\downarrow$	AV-IB $\uparrow$	DeSync $\downarrow$	CSIM-r $\uparrow$	FID $\downarrow$	FVD $\downarrow$	FAD $\downarrow$
GT (source)	0.852	6.82	7.77	0.377	0.115	0.867	—	—	—
Teacher	0.857	8.49	7.26	0.333	0.028	0.839	32.2	128.5	6.85
Whole-seg	0.835	7.82	7.62	0.328	0.040	0.864	33.7	119.2	5.96
Decoupled	0.866	7.54	7.85	0.334	0.007	0.791	35.2	129.0	5.23
Decoupled-50	0.836	7.13	7.99	0.324	0.028	0.761	38.9	143.6	5.34

Appendix

Table 4: The systems of Table 1 on three public test sets (

100

clips each), scored against the real source video — so Teacher is a system under test, not the reference. Among the four autoregressive systems, best results in bold, second-best underlined, per column and set; the GT row is a reference, not a competitor. CSIM-r is face identity against the reference portrait, not CSIM-1st. FAD is omitted on HDTF: clip-length mismatch dominates there (whole-segment generation has a

4

s minimum against

{\sim}3.2

s sources); DeSync is undefined on HDTF (clips shorter than Synchformer’s

4.8

s context).

Figure 5: Identity drift over the

30

s long-horizon runs of § 4.2 (

40

clips, probes every

5

s). (a) Face similarity to the reference portrait, with ArcFace. (b) Speaker similarity to the clip’s own opening segment, with CAM++. Both are the complement of Figure 3 , which measures against each clip’s own first frame under different encoders. Face identity is retained equally well by Decoupled , Decoupled-50 and Chained-DA (and worst by Chained-AD ); speaker identity falls furthest for Decoupled , the observation § 5 reports as a limitation.

Explore similar work

Jul 29, 2026cs.CV

Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory

Audio-video generative models achieve impressive quality but suffer from high latency, making them unsuitable for real-time applications. Although several streaming audio-video generation methods have been proposed, they remain costly and fail to support long-form generation. To address this, we propose \textbf{Ripple}, a real-time joint audio-video generation system with a cross-modal recurrent memory mechanism. To enable efficient streaming inference while preserving long-term context, Ripple combines a fixed-length sliding-window attention with modality-specific memory states that continuously summarize audio and video context. Cross-modal memory interaction is further introduced to enhance audio-visual synchronization. To learn this memory-augmented model effectively, we devise a three-stage training recipe: (1) adapting a bidirectional audio-video teacher to block-wise causal attention with simulated memory, (2) optimizing the memory construction and interaction pipeline through end-to-end distillation, and (3) applying online reinforcement post-training tailored for streaming audio-video generation. As a result, Ripple achieves ~28 FPS at 480P resolution, over faster than the teacher, while capable of coherent long-form generation. Extensive experiments on both short-video and long-video benchmarks demonstrate our superior performance over existing offline and online joint audio-video generation methods.

Yanbo Ding, Zhizhi Guo, Quanyue Song +4

Shenzhen Key Lab of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China · School of Artificial Intelligence, University of Chinese Academy of Sciences +2

Audio-Video Generation Real-Time Systems Fast Inference Cross-Modal Long-Form Generation

Ask PDF

Apr 26, 2026cs.CV

Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation

Real-time text-driven joint audio-video avatar generation requires jointly synthesizing portrait video and speech with high fidelity and precise synchronization, yet existing audio-visual diffusion models remain too slow for interactive use and often degrade noticeably after aggressive acceleration. We present Hallo-Live, a streaming framework for joint audio-visual avatar generation that combines asynchronous dual-stream diffusion with human-centric preference-guided distillation. To reduce articulation lag in causal generation, we introduce Future-Expanding Attention, which allows each video block to access synchronous audio together with a short horizon of future phonetic cues. To mitigate the quality loss of few-step distillation, we further propose Human-Centric Preference-Guided DMD (HP-DMD), which reweights training samples using rewards from visual fidelity, speech naturalness, and audio-visual synchronization. On two NVIDIA H200 GPUs, Hallo-Live runs at 20.38 FPS with 0.94 seconds latency, yielding 16.0x higher throughput and 99.3x lower latency than the teacher model Ovi. Despite this speedup, it retains strong generation quality, reaching comparable VideoAlign overall score and Sync Confidence score while outperforming other accelerated baselines in the overall quality-efficiency trade-off. Qualitative results further show robust generalization across photorealistic, multi-speaker, and stylized scenarios. To the best of our knowledge, Hallo-Live is the first framework to combine streaming dual-stream diffusion with preference-guided distillation for real-time, text-driven audio-visual generation.

Chunyu Li, Jiaye Li, Ruiqiao Mei +4

Shanghai Innovation Institute · Fudan University · University of Science and Technology of China +2

Audio-Video Generation Dual-Stream

Ask PDF

Aug 31, 2026cs.CV

DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution

Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We present DreamX-Creator 1.0, a compact native joint audio-video generation system centered on a 7B generator. Conditioned on a first frame and a text prompt, the generator jointly denoises modality-specialized audio and video streams. The streams are processed independently in the first half of the network and coupled in the latter half through Gated Cross-Modal Attention, whose token- and head-wise output gates modulate each active cross-modal attention-head output. A unified Audio-Video Data System constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools. Progressive Joint Training comprises two audio-video pre-training stages followed by High-Quality Finetuning. Audio-Video Reinforcement Learning further post-trains the generator with Modality-Aware Multimodal Feedback that routes video-, audio-, and cross-modal feedback to the corresponding streams. For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner and distills it into a student requiring one denoising evaluation per temporal chunk. Overall, DreamX-Creator 1.0 achieves native, synchronized audio-video generation with performance competitive with state-of-the-art open-source systems. By releasing our compact 7B generator and 2K Refiner, we seek to democratize native audio-video generation and provide an accessible foundation for future research in unified audio-video generative modeling.

Jiashu Zhu, Yanhao Zheng, Ruitian Tian +7

DreamX Team, Alibaba Group

Audio-Video Generation Multimodal Generation Cross-Modal Attention Dream

Ask PDF