OmniForcing: Unleashing Real-time Joint Audio-Visual Generation

Yaofeng Su; Yuming Li; Zeyue Xue; Jie Huang; Siming Fu; Haoran Li; Ying Li; Zezhong Qian

cs.MMMar 12, 2026

OmniForcing: Unleashing Real-time Joint Audio-Visual Generation

Authors: Yaofeng Su, Yuming Li, Zeyue Xue, Jie Huang, Siming Fu, Haoran Li, Ying Li, Zezhong Qian,

Organizations: JD Explore Academy · Fudan University · Peking University · The University of Hong Kong

Abstract

Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency due to their bidirectional attention dependencies, hindering real-time applications. We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autoregressive generator. However, naively applying causal distillation to such dual-stream architectures triggers severe training instability, due to the extreme temporal asymmetry between modalities and the resulting token sparsity. We address the inherent information density gap by introducing an Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix that prevents multi-modal synchronization drift. The gradient explosion caused by extreme audio token sparsity during the causal shift is further resolved through an Audio Sink Token mechanism equipped with an Identity RoPE constraint. Finally, a Joint Self-Forcing Distillation paradigm enables the model to dynamically self-correct cumulative cross-modal errors from exposure bias during long rollouts. Empowered by a modality-independent rolling KV-cache inference scheme, OmniForcing achieves state-of-the-art streaming generation at $\sim$ 25 FPS on a single GPU, maintaining multi-modal synchronization and visual quality on par with the bidirectional teacher.Project Page: https://omniforcing.com

Figures & tables

Figure 1: OmniForcing breaks the latency barrier for joint audio-visual generation. Top: Our framework achieves real-time streaming at $\sim$ 25 FPS with an ultra-low Time-To-First-Chunk (TTFC) of $\sim$ 0.7s. Bottom: The bidirectional teacher (LTX-2) requires $\sim$ 197s to generate the sequence offline. OmniForcing maintains visual and acoustic fidelity on par with the teacher model.

Figure 2: The three-stage OmniForcing distillation pipeline. Stage I employs Distribution Matching Distillation (DMD) [ 50 , 49 ] to adapt the model for few-step, fast denoising. Stage II utilizes causal ODE regression to adapt the network weights to the asymmetric block-causal mask. Stage III implements joint Self-Forcing [ 9 ] training by autoregressively unrolling the generation process to mitigate exposure bias.

Figure 3: Asymmetric Block-Causal Masking. The vertical axis denotes query tokens and the horizontal axis denotes key tokens. Modalities are synchronized via 1s macro-blocks. Each audio block (

B^{a}

) contains 25 latent frames (one token each), whereas each video block (

B^{v}

) contains 3 latent frames patchified into

3\times 384

tokens. Unmasked tokens include the Global Prefix (orange,

V_{0}/A_{0}

) and Audio Sink tokens (red,

s

). Blue regions denote allowed attention (bidirectional intra-block, strictly causal inter-block), while white regions mask future keys to prevent information leakage.

Figure 4: Qualitative comparison across diverse scenes. Each example shows generated frames with synchronized waveforms and Mel spectrograms. OmniForcing produces voice layered with bird calls at a seaside scene (top-left), sustained speech at a podium presentation (top-right), a precisely timed cat meow (bottom-left), and blended narration with sewing machine sounds (bottom-right).

		AV-Quality		Text-Consistency				AV-Consistency		AV-Synchrony
Model	Size	FVD $\downarrow$	FAD $\downarrow$	TV-IB $\uparrow$	TA-IB $\uparrow$	CLIP $\uparrow$	CLAP $\uparrow$	AV-IB $\uparrow$	AVHScore $\uparrow$	JavisScore $\uparrow$	DeSync $\downarrow$	Runtime $\downarrow$
– T2A+A2V
TempoTkn	1.3B	539.8	–	0.084	–	0.205	–	0.139	0.122	0.103	1.532	20s
TPoS	1.0B	839.7	–	0.201	–	0.229	–	0.124	0.129	0.095	1.493	19s
– T2V+V2A
ReWaS	0.6B	–	9.4	–	0.123	–	0.280	0.110	0.104	0.079	1.071	17s

Table 1: Main results on JavisBench [ 23 ] . Best results in bold , second-best underlined . (

\uparrow

: higher is better;

\downarrow

: lower is better). TTFC and FPS metrics are reported in Tab. 2 .

Model	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$	Motion Smoothness $\uparrow$	Subject Consistency $\uparrow$	Temporal Flickering $\uparrow$	TTFC $\downarrow$	FPS $\uparrow$
	VBench Visual Quality					Streaming
LTX-2 [ 7 ]	0.569	0.574	0.993	0.945	0.988	197.0s	–
OmniForcing	0.595	0.594	0.995	0.955	0.989	0.7s	25

Table 2: Distillation fidelity: OmniForcing vs. the bidirectional LTX-2 teacher on the same prompt set. Visual quality is measured using VBench [ 10 ] , which provides a modular toolkit that officially supports per-video scoring on user-supplied content. Best in bold . (

\uparrow

: higher is better;

\downarrow

: lower is better).

Configuration	Convergence	$\\|\nabla\\|_{\max}$	Visualization	Loss ( $\sigma\!=\!0.5$ , 3k)	Remark
Sink $S\!=\!24$ + Id. RoPE	Stable	9.15	Normal	0.110	–
Sink $S\!=\!16$ + Id. RoPE	Stable	9.23	Normal	0.081	–
Sink $S\!=\!8$ + Id. RoPE	Stable	21.95	Normal	0.129	–
Sink $S\!=\!4$ + Id. RoPE	Stable	49.71	Normal	0.141	–
Sink $S\!=\!2$ + Id. RoPE	NaN	$\infty$	Noise	–	Softmax collapse
Sink $S\!=\!1$ + Id. RoPE	NaN	$\infty$	Noise	–	Softmax collapse

Table 3: Ablation on training stabilization strategies during Stage II. We report convergence status (with failure mode if applicable), maximum gradient norm, visual quality, and one-step denoising loss at

\sigma\!=\!0.5

averaged over the evaluation set after 3k steps. All sink variants use Identity RoPE unless noted.

Audio-Video Generation

Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.

See all 155 papers

Real-Time Joint Audio-Video Generation by Parallel Adapter Composition
Oct 7, 2026Jingyu Li, Xiaoxiao Xiang, Yiwen GuoAudio-Video Generation Real-Time Interactive Video Generation
LIGHTSPEED · Independent Researcher
Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generation
Apr 28, 2026Yupeng Zhou, Lianghua Huang, Zhifan Wu +7Audio-Video Generation Long-Horizon Video Generation
1VCIP, School of Computer Science, Nankai University · 2Tongyi Lab · 2Tongyi Lab,3Peking University +1
Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory
Jul 29, 2026Yanbo Ding, Zhizhi Guo, Quanyue Song +4Audio-Video Generation Long Video Generation
Shenzhen Key Lab of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China · School of Artificial Intelligence, University of Chinese Academy of Sciences +2

Oct 7, 2026J/K move · Enter open · S save

Real-Time Joint Audio-Video Generation by Parallel Adapter Composition

Jingyu Li, Xiaoxiao Xiang, Yiwen Guo

LIGHTSPEED · Independent Researcher

Audio-Video Generation Real-Time Interactive Video Generation Streaming Video Generation Efficient Video Generation Autoregressive Video Generation

Open paper AskPDF arXiv

Deploying a joint audio-video diffusion transformer for real-time, interactive generation normally requires two essential modifications: block-autoregressive attention, so frames can be emitted before the whole clip is finished, and few-step sampling, so each block is cheap. Conventionally, the streaming video literature obtains both capabilities from a chained pipeline. It first distills a bidirectional teacher into a causal student, then into a few-step one, or proceeds in reverse order. Each stage of such a chain fine-tunes the weights the previous one produced, so a later objective can undo an earlier capability. Following the idea of model merging, we show that on a packed audio-video backbone the two capabilities can be acquired in parallel. A causal adapter is trained against the frozen backbone, and an off-the-shelf few-step adapter provides the few-step capability. As the two edit different functional axes, we predict, and then verify, that their weight-update directions are near-orthogonal, without any explicit orthogonality constraint during training. Orthogonal updates should combine without interfering, so parallel composition is a direct sum. The two adapters are simply added at inference, with no joint training, yielding few-step, streaming audio-video whose image quality tracks the bidirectional teacher. Compared to the chained baselines, the composed model matches or beats them on most metrics, making parallel composition a practical approach. The resulting streaming system generates joint audio-video in real time,

\approx

26 fps at

480\times832

without quantization, and sustains 30 s of continuous generation with stable image quality. Demo Page: https://pac-demo-2027.github.io/demo/

Abstract

Audio-Video Generation

Explore similar work

Real-Time Joint Audio-Video Generation by Parallel Adapter Composition

Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generation

Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory