Adaptive Reward Routing: Dynamic Multi-Reward Optimization for Joint Audio-Video Diffusion via Forward-Process RL

Songlin Yang; Xiaotong Zhao; Jiacheng Zhang; Zhe Wang; Toyota Li; Eric Liu; Alan Zhao; Anyi Rao

cs.CVSep 29, 2026

Adaptive Reward Routing: Dynamic Multi-Reward Optimization for Joint Audio-Video Diffusion via Forward-Process RL

Authors: Songlin Yang, Xiaotong Zhao, Jiacheng Zhang, Zhe Wang, Toyota Li, Eric Liu, Alan Zhao, Anyi Rao

Organizations: Tencent Video · The University of Hong Kong

Abstract

Multi-reward guided reinforcement learning (i.e., RL) offers a promising way to improve joint audio-video diffusion models along several complementary objectives, including modality-specific quality, cross-modal semantic alignment, and temporal synchronization. Its effectiveness, however, depends on two quantities that change during training: where reward-driven updates should act, and how competing rewards should be combined. Existing methods tend to rely on fixed routing and reward weights, failing to track evolving model functions. To address these limitations, we propose Adaptive Reward Routing to jointly adapt update locations and reward coordination during forward-process RL (i.e., DiffusionNFT) of joint audio-video diffusion models. Our method consists of two components. (i) Cross-Modal Influence-Guided Routing (Localizing Updates): We use bidirectional cross-attention responses as an efficient proxy for evolving cross-modal influence, dynamically reweighting token-aware losses and scaling gradients across cross-modal layers without additional model interventions. (ii) Preference-Preserving Modality-Aware Reweighting (Coordinating Rewards): We preserve predefined weights as preference priors and use branch-specific reward-gradient interactions as residual corrections after warm-up. This resolves evolving conflicts without letting dominant rewards suppress weak but essential objectives. Extensive experiments demonstrate consistent improvements in modality quality, semantic consistency, and audio-video synchronization over strong RL baselines. Ablations and mechanism analyses further validate the complementary benefits of adaptive update routing and reward coordination.

Figures & tables

Figure 1: Joint audio-video post-training requires adaptive update routing and reward coordination. (a) Forward KV Ablation. We ablate cross-modal K/V outputs by layer range and measure DeSync (lower is better; larger degradation indicates greater synchronization importance). Divergent base/trained profiles reveal shifting synchronization-critical layers. (b) Backward Gradient Analysis. We measure layer-wise $\ell_{2}$ norms of Q (within-branch) and K/V (cross-modal) gradients in both branches; shifted base/trained peaks show evolving cross-modal gradient paths. (c) Reward Conflicts. Rewards conflict when their per-prompt normalized preferences have opposite signs, with larger off-diagonal values indicating more disagreement. The conflicts show that fixed branch assignment is insufficient. (d) Gradient-Based Weight Drift. We track weights derived solely from reward-gradient geometry, where near-zero trajectories indicate objective suppression. The vanishing AV-DeSync weight shows that geometry-only weighting can discard essential objectives. Together, these results motivate dynamic routing and preference-anchored reward coordination.

Figure 2: Overview of Adaptive Reward Routing. Our framework adapts reward-driven optimization at four levels: reward reweighting, modality-branch assignment, token-level credit allocation, and layer-wise cross-modal gradient routing.

Figure 3: Qualitative comparison of joint audio-video generation. For each prompt, we show five temporally ordered frames generated by LTX-2, OmniNFT, and Ours. The examples cover Chinese news delivery, stylized English speech, two-speaker dialogue, and dog barking. Our method better preserves the requested subjects and scene composition while reducing identity and appearance drift throughout the generated sequence.

Backbone	Method	AV-Quality		Text-Consistency				AV-Consistency		AV-Synchrony
Backbone	Method	VQ $\uparrow$	AQ $\uparrow$	TV-IB $\uparrow$	TA-IB $\uparrow$	CLIP $\uparrow$	CLAP $\uparrow$	AV-IB $\uparrow$	AVHScore $\uparrow$	JavisScore $\uparrow$	DeSync $\downarrow$
LTX-2	Base Model	1.883	5.201	0.265	0.143	0.312	0.358	0.180	0.177	0.153	0.604
	+ GDPO	2.722 $\pm 0.0136$	5.450 $\pm 0.0217$	0.261 $\pm 0.0020$	0.138 $\pm 0.0018$	0.312 $\pm 0.0014$	0.347 $\pm 0.0031$	0.174 $\pm 0.0025$	0.175 $\pm 0.0022$	0.155 $\pm 0.0020$	0.671 $\pm 0.0104$
	+ MARBLE	2.384 $\pm 0.0124$	5.100 $\pm 0.0193$	0.265 $\pm 0.0016$	0.138 $\pm 0.0019$	0.311 $\pm 0.0013$	0.365 $\pm 0.0028$	0.182 $\pm 0.0023$	0.182 $\pm 0.0026$	0.158 $\pm 0.0021$	0.618 $\pm 0.0092$
	+ OmniNFT	3.136 $\pm 0.0108$	5.614 $\pm 0.0176$	0.265 $\pm 0.0018$	0.145 $\pm 0.0016$	0.312 $\pm 0.0011$	0.416 $\pm 0.0024$	0.222 $\pm 0.0019$	0.219 $\pm 0.0021$	0.195 $\pm 0.0017$	0.390 $\pm 0.0075$
	+ OmniNFT*	3.278	5.609	0.254	0.165	0.314	0.422	0.220	0.219	0.193	0.378

Table 1: Main results on JavisBench. (Mean ± std over 3 seeds.)

Figure 4: Reward dynamics and component ablations. (a) Training trajectories of five individual rewards (AudioBox, AV-DeSync, CLAP, HPSv3, and VideoAlign) and their average for GDPO, MARBLE, OmniNFT, and Ours. Average denotes the arithmetic mean of the five normalized rewards. (b) Ablation studies of the routing and weighting designs. Starting from GDPO or MARBLE, respectively, each added component yields progressive gains, while the complete method achieves the strongest overall performance.

Study	Configuration	AV-Quality		Text-Consistency				AV-Consistency		AV-Synchrony
Study	Configuration	VQ $\uparrow$	AQ $\uparrow$	TV-IB $\uparrow$	TA-IB $\uparrow$	CLIP $\uparrow$	CLAP $\uparrow$	AV-IB $\uparrow$	AVHScore $\uparrow$	JavisScore $\uparrow$	DeSync $\downarrow$
Routing	+ Token Weighting	3.008 $\pm 0.0127$	5.663 $\pm 0.0202$	0.263 $\pm 0.0018$	0.156 $\pm 0.0016$	0.313 $\pm 0.0013$	0.388 $\pm 0.0027$	0.210 $\pm 0.0022$	0.201 $\pm 0.0024$	0.179 $\pm 0.0019$	0.482 $\pm 0.0088$
	+ Layer Scale	3.192 $\pm 0.0111$	5.784 $\pm 0.0188$	0.264 $\pm 0.0017$	0.161 $\pm 0.0015$	0.313 $\pm 0.0012$	0.409 $\pm 0.0025$	0.221 $\pm 0.0020$	0.227 $\pm 0.0022$	0.189 $\pm 0.0018$	0.366 $\pm 0.0072$
	+ Token + Layer	3.315 $\pm 0.0103$	5.839 $\pm 0.0171$	0.266 $\pm 0.0015$	0.162 $\pm 0.0014$	0.313 $\pm 0.0011$	0.411 $\pm 0.0023$	0.226 $\pm 0.0018$	0.229 $\pm 0.0020$	0.190 $\pm 0.0016$	0.343 $\pm 0.0065$
	+ Branch-Aware	2.612 $\pm 0.0131$	5.290 $\pm 0.0211$	0.264 $\pm 0.0019$	0.149 $\pm 0.0017$	0.311 $\pm 0.0014$	0.384 $\pm 0.0030$	0.198 $\pm 0.0024$	0.188 $\pm 0.0027$	0.177 $\pm 0.0021$	0.492 $\pm 0.0096$
	+ Residual	2.891 $\pm 0.0118$	5.526 $\pm 0.0196$	0.264 $\pm 0.0016$	0.158 $\pm 0.0018$	0.312 $\pm 0.0013$	0.393 $\pm 0.0026$	0.199 $\pm 0.0021$	0.199 $\pm 0.0023$	0.180 $\pm 0.0019$	0.455 $\pm 0.0084$

Table 2: Results of the ablation studies on JavisBench using the LTX-2 backbone. Gray-shaded rows isolate the reward-coordination components from the MARBLE baseline and do not inherit the routing stack. (Mean ± std over 3 seeds.)

Figure 5: The proxy correctly identifies influential cross-modal layers and tokens. (a) Each point is one layer. The x-axis is the layer’s mean pre-gate A2V/V2A response norm, while the y-axis is the relative change in the corresponding final video/audio velocity after disabling that layer. Both axes are min–max normalized across the 48 layers for visualization. Points near the diagonal indicate agreement between the proxy and direct intervention. (b) Higher correlation means better agreement with the current model. The current proxy stays accurate, whereas the proxy fixed at initialization becomes inaccurate as training changes the model. (c) We block the cross-modal information received by the top-scoring, random, or bottom-scoring 10% of target tokens. A larger final prediction change means that the blocked tokens were more influential. The top-scoring tokens consistently cause the largest change, confirming that the proxy correctly identifies important tokens.

Explore similar work

OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation
May 12, 2026Guohui Zhang, XiaoXiao Ma, Jie Huang +9Audio-Video Generation Diffusion-Based Reinforcement Learning Methods
University of Science and Technology of China · JD Explore Academy · Peking University
AV-GRPO: Modality-Anchored Decoupling Diffusion Reinforcement Learning for Joint Audio-Video Generation
Sep 24, 2026Zhiyu Xu, Weilong Yan, Yufei Shi +4Audio-Video Generation Diffusion-Based Reinforcement Learning Methods
Shanghai AI Laboratory · The Hong Kong Polytechnic University · National University of Singapore +2
Diffusion-APO: Trajectory-Aware Direct Preference Alignment for Video Diffusion Transformers
May 8, 2026Jingyuan Zhu, Biaolong Chen, Le Zhang +3Diffusion Alignment Video Diffusion Models
Alibaba Group

May 12, 2026, cs.CVJ/K move · Enter open · S save

OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation

Guohui Zhang, XiaoXiao Ma, Jie Huang, Hang Xu+8

University of Science and Technology of China · JD Explore Academy · Peking University

Audio-Video Generation Diffusion-Based Reinforcement Learning Methods

Open paper AskPDF arXiv

Recent advances in joint audio-video generation have been remarkable, yet real-world applications demand strong per-modality fidelity, cross-modal alignment, and fine-grained synchronization. Reinforcement Learning (RL) offers a promising paradigm, but its extension to multi-objective and multi-modal joint audio-video generation remains unexplored. Notably, our in-depth analysis first reveals that the primary obstacles to applying RL in this stem from: (i) multi-objective advantages inconsistency, where the advantages of multimodal outputs are not always consistent within a group; (ii) multi-modal gradients imbalance, where video-branch gradients leak into shallow audio layers responsible for intra-modal generation; (iii) uniform credit assignment, where fine-grained cross-modal alignment regions fail to get efficient exploration. These shortcomings suggest that vanilla RL fine-tuning strategy with a single global advantage often leads to suboptimal results. To address these challenges, we propose OmniNFT, a novel modality-aware online diffusion RL framework with three key innovations: (1) Modality-wise advantage routing, which routes independent per-reward advantages to their respective modality generation branches. (2) Layer-wise gradient surgery, which selectively detaches video-branch gradients on shallow audio layers while retaining those for cross-modal interaction layers. (3) Region-wise loss reweighting, which modulates policy optimization toward critical regions related to audio-video synchronization and fine-grained alignment. Extensive experiments on JavisBench and VBench with the LTX-2 backbone demonstrate that OmniNFT achieves comprehensive improvements in audio and video perceptual quality, cross-modal alignment, and audio-video synchronization.

Abstract

Explore similar work

OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation

AV-GRPO: Modality-Anchored Decoupling Diffusion Reinforcement Learning for Joint Audio-Video Generation

Diffusion-APO: Trajectory-Aware Direct Preference Alignment for Video Diffusion Transformers