AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

Yuruo Wu; Kaihang Pan; Kai Shen; Jiacheng Li; Shuyuan Tu; Jisheng Dang; Li Tuoyu; Juncheng Li

cs.CVOct 15, 2026

AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

Authors: Yuruo Wu, Kaihang Pan, Kai Shen, Jiacheng Li, Shuyuan Tu, Jisheng Dang, Li Tuoyu, Juncheng Li,

Organizations: Zhejiang University · HiThink Research · Fudan University · Lanzhou University

Abstract

As joint audio-video generation models become more capable, they can generate audio-visual scenes exhibiting three forms of complexity: multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations. However, current evaluations largely reuse metrics from earlier simple and unidirectional tasks. They typically evaluate an audio-video clip as a whole and rely on simple predefined relations, making them unsuitable for these increasingly complex scenes. We propose an event-centric evaluation framework that disentangles audio and visual events at the event level, analyzes each event independently, cross-validates relations across modalities, and aggregates the judgments into a coarse-to-fine, three-level consistency metric. The metric assesses global narrative harmony, event-level temporal synchronization, and fine-grained attribute alignment, providing a general formulation for audio-video consistency evaluation across diverse generation settings and cross-modal relations. We instantiate the metric as AVC-Reward, a trainable audio-video reward model. To evaluate the agreement between automatic judgments and human perception across generation paradigms and relation complexities, we construct AVC-Benchmark, which is primarily built from diverse positive and negative samples generated by various types of audio-video generation models. AVC-Reward achieves state-of-the-art alignment with human judgment among automatic methods, provides interpretable rationales that localize failures, and improves the audio-video consistency of a joint generation model when used as an online reinforcement-learning signal. Project page: https://tsuki0512.github.io/AVC-Reward.github.io/

Figures & tables

Figure 1: Overview of the proposed event-centric audio-video consistency evaluation framework.

Method	Scoring Unit	Audio layers	Relation Evaluated
SyncNet ( Chung and Zisserman, 2016 )	Mouth–speech window	Mixed	Lip–speech synchrony
AV-Align ( Yariv et al., 2024 )	Audio / motion peaks	Mixed	Motion–sound timing
DeSync ( Iashin et al., 2024 )	AV temporal window	Mixed	Temporal offset
ImageBind ( Girdhar et al., 2023 )	Clip	Mixed	Global semantic similarity
AVHScore ( Mao et al., 2024 )	Frame $\times$ whole audio	Mixed	Frame–audio similarity
JavisScore ( Liu et al., 2025 )	Temporal window	Mixed	Window-level AV similarity

Table 1: Comparison of audio-video consistency metrics for challenges exposed by modern JAVG.

Figure 2: Overview of data construction, reward-model training, and generation-side optimization.

Timing-based synchronization
Method	Pairwise $\uparrow$	$\Delta$ -Pairwise $\uparrow$	PLCC $\uparrow$	SRCC $\uparrow$
AV-Align ( Yariv et al., 2024 )	0.577	0.674	0.301	0.215
DeSync ( Iashin et al., 2024 )	0.521	0.580	0.280	0.061
Embedding similarity
ImageBind ( Girdhar et al., 2023 )	0.556	0.625	0.219	0.154
JavisScore ( Liu et al., 2025 )	0.554	0.611	0.180	0.147

Table 2: Main results on AVC-Benchmark (706 samples).

\Delta

-Pairwise (

\Delta\!>\!0.2

) and PLCC are the primary metrics. Best automatic result in bold; Human Upper Bound is the inter-annotator ceiling.

Figure 3: Pairwise accuracy under different margin thresholds

\Delta

. Larger

\Delta

keeps pairs with clearer human-score differences.

(a) AV consistency
Model	AVC-Score $\uparrow$	DeSync $\downarrow$	IB-AV $\uparrow$	AV-align $\uparrow$	Javis $\uparrow$	AVH $\uparrow$	CAVP $\uparrow$
AV-Phys-Bench
LTX-2	0.756	0.376	0.234	0.158	0.189	0.226	0.812
OmniNFT (DeSync)	0.780	0.073	0.248	0.213	0.193	0.233	0.819
OmniNFT gain	+3.26%	+80.60%	+5.84%	+34.17%	+1.96%	+2.91%	+0.88%
Ours (AVC-Reward)	0.819	0.081	0.288	0.246	0.228	0.276	0.818

Table 3: Generation results across three benchmarks. Gain rows report relative improvements over LTX-2. All other entries are absolute metric values. Best results within each benchmark are in bold.

Figure 4: Overall human preference across the three benchmarks.

Figure 5: Representative cases. Left: AVC-Reward produces scores closer to human judgments and identifies specific consistency issues. Right: generation results of LTX-2, OmniNFT, and Ours.

Training Strategy				Score Aggregation
Variant	$\Delta$ -Pair $\uparrow$	PLCC $\uparrow$	SRCC $\uparrow$	Variant	$\Delta$ -Pair $\uparrow$	PLCC $\uparrow$	SRCC $\uparrow$
Base	0.605	0.347	0.154	$S_{\mathrm{mac}}$ only	0.688	0.467	0.251
GRPO only	0.630	0.330	0.161	$S_{\mathrm{temp}}$ only	0.717	0.479	0.235
SFT only	0.679	0.295	0.244	$S_{\mathrm{attr}}$ only	0.715	0.482	0.240
SFT+GRPO	0.728	0.494	0.244	Mean ( $S_{\mathrm{AVC}}$ )	0.728	0.494	0.244

Table 4: Ablations and aggregation analysis on AVC-Benchmark.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Subset	Source model	Paradigm	Raw clips	SFT	Human calib.
General	Ground truth	GT	2.5k	✓
General	Ovi	JAVG	2.5k	✓
General	HunyuanVideo-Foley	V2A	2.5k	✓
Human	Ground truth	GT	1.0k	✓
Human	Ovi	JAVG	1.0k	✓
Human	InfiniteTalk	A2V	1.0k	✓

Appendix

Table 5: Sources and statistics of the training data. The SFT source pool contains 11.5k candidate clips, of which 8k are retained after supervision filtering. The human-calibration source pool contains 16k candidates, from which the 10k GRPO rollout set is constructed.

Stage	Input	Operation	Tool	Output
1. Audio analysis	Audio $A$ + SED hints $T_{\mathrm{SED}}$	Analyze the audio scene, layers, events, and timestamps.	Gemini 2.5 Pro + SED hints	$H_{A}=(G_{A},E_{A})$ : audio context and timestamped audio events.
2. Video analysis	Video $V$	Analyze the visual scene and localize audio-relevant events.	Qwen3-VL-Max	$H_{V}=(G_{V},E_{V})$ : visual context and timestamped visual events.
3. A $\rightarrow$ V verification	$H_{A}$ from Stage 1 + $V$	Verify visual evidence, timing, and applicable event attributes for each audio event.	Qwen3-VL-Max	$R_{A\rightarrow V}$ : audio-to-video verification records.
4. V $\rightarrow$ A verification	$H_{V}$ from Stage 2 + $A$	Verify expected audio, reasonable silence, timing, and applicable event attributes for each visual event.	Gemini 2.5 Pro	$R_{V\rightarrow A}$ : video-to-audio verification records.
5. Aggregation	$\begin{gathered}H_{A},H_{V},\\ R_{A\rightarrow V},R_{V\rightarrow A}\end{gathered}$ + original AV	Re-examine the clip and aggregate the verified evidence.	Gemini 2.5 Pro	$(S_{\mathrm{mac}},S_{\mathrm{temp}},S_{\mathrm{attr}},\kappa)$ .

Appendix

Table 6: Summary of the five-stage automated annotation pipeline. Stages 1–2 produce modality-specific analysis records, Stages 3–4 verify their event relations in the opposite modality, and Stage 5 aggregates all records into the final three-level judgment.

Figure 6: Overview of the five-stage automated annotation pipeline. Audio and video are first analyzed independently, followed by bidirectional event verification and final aggregation into three consistency scores and a diagnostic conclusion.

Dimension	Event type	Label	Annotation meaning
Global narrative harmony	Global	M-A	Globally consistent
		M-B	Narrative-content conflict
		M-C	Emotion or atmosphere mismatch
		M-D	Ambient sound or scene-setting conflict
		M-E	Other global semantic inconsistency
Event-level temporal synchronization	Physical interaction	T-P-A	On-screen event correctly matched

Appendix

Table 7: Human annotation taxonomy with 39 labels. Global narrative harmony evaluates clip-level narrative and scene coherence; event-level temporal synchronization separately covers physical interactions, speaker events, and music or rhythmic events; fine-grained attribute alignment examines the internal audio-visual attributes of temporally matched events.

Label configuration	Pairs	Mean diff.	Median diff.
Same	6,717	0.0321	0.0111
Different	242,148	0.2002	0.1000

Appendix

Table 8: Absolute differences between human scores for sample pairs with the same or different label configurations.

Score	PLCC	$\Delta$ [email protected]
Global narrative harmony	0.591	0.712
Temporal synchronization	0.454	0.816
Fine-grained attribute alignment	0.601	0.798
Overall	0.780	0.905

Appendix

Table 9: Agreement between rewritten and human scores after excluding the manually selected few-shot examples.

Setting	Comparisons	$r_{\mathrm{distill}}$	$1-r_{\mathrm{distill}}$
Rewrite, full evaluation set	706	0.853	0.147
Human leave-one-out	2,118	0.881	0.119

Appendix

Table 10: Score-rewriting agreement measured with the score-distillation reward used during GRPO. Human leave-one-out results provide a reference for annotator variation.

Configuration	Value
Base model	Qwen3-Omni-Thinking
Training framework	MS-SWIFT
Fine-tuning	Full-parameter
Training samples	8k
Epochs	1
Global batch size	64

Appendix

Table 11: Training configuration for the SFT cold-start stage.

Condition	Reward
<think> content is present	+0.2
Each required stage marker is present	$+0.1\times k$ ( $k\leq 5$ )
All three predicted scores are present	+0.2
Diagnostic conclusion is present	+0.1

Appendix

Table 12: Components of the structure reward

r_{\mathrm{struct}}

Configuration	Value
Training framework	MS-SWIFT
Rollout samples	10k
Training steps	1,700
Global batch size	32
Group size $G$	8
Learning rate	$1\times 10^{-6}$

Appendix

Table 13: Training configuration for GRPO human alignment.

Edit operation	$n$
Swap audio tracks between paired clips	10
Overlay the audio track from another clip	10
Shift the audio by $+2$ s	20
Shift the audio by $-2$ s	20
Remove a specified audio event	20
Freeze the video for 1 s at an audio peak	19

Appendix

Table 14: Controlled edit operations used to construct explicit negative samples.

Setting	$\Delta$ -Pairwise $\uparrow$	PLCC $\uparrow$	SRCC $\uparrow$
Raw score aggregation	0.728	0.494	0.244
Z-score aggregation	0.729	0.492	0.245

Appendix

Table 15: Sensitivity analysis of score normalization for Ours on AVC-Benchmark.

Figure 7: Statistics of AVC-Benchmark, including source composition, human-label distribution, scene-complexity coverage, and human-score distribution. The three complexity subsets are non-exclusive.

$\Delta$	0.00	0.05	0.10	0.15	0.20	0.25	0.30	0.50
Valid pairs	227,819	162,008	113,124	84,407	69,180	59,371	52,559	30,831

Appendix

Table 16: Number of valid sample pairs at different

\Delta

-Pairwise margins using the z-score-normalized human reference.

Method	Multi-event	Multi-layer Audio	Diverse Relation
	(678)	(322)	(376)
AV-Align	0.654	0.681	0.670
DeSync	0.587	0.582	0.567
ImageBind	0.626	0.658	0.632
JavisScore	0.615	0.659	0.636
AVHScore	0.597	0.635	0.611

Appendix

Table 17:

\Delta

[email protected] on the three non-exclusive scene-complexity subsets. The best and second-best automatic metrics are shown in bold and underlined, respectively. Human Upper Bound is excluded from ranking.

Human target	$\Delta$ Macro	$\Delta$ Temporal	$\Delta$ Fine-grained
Macro	0.367	0.026	-0.032
Temporal	-0.009	0.078	0.038
Fine-grained	-0.004	0.039	0.102

Appendix

Table 18: Cross-dimensional permutation analysis for validating dimension correspondence. Values denote AUC decrease after permuting each predicted dimension.

Figure 8: Full input–output example of AVC-Reward. For visualization purposes, the semantically empty lines in Stage 4 and Stage 5 are compressed due to space limitations.

Figure 9: Additional evaluation cases of AVC-Reward.

Figure 10: Additional generation cases after AVC-Reward optimization.

Benchmark	Question	Ours	OmniNFT	LTX-2	Ours $>$ OmniNFT	Ours $>$ LTX-2
AVGenBench	Consistency	47.8	24.4	27.8	69.1	65.5
AVGenBench	Overall	46.2	27.5	26.4	66.0	64.5
AV-Phys-Bench	Consistency	50.0	27.8	22.2	73.1	71.4
AV-Phys-Bench	Overall	49.5	26.2	24.3	72.2	68.6
VABench	Consistency	46.4	34.1	19.6	63.5	72.3
VABench	Overall	44.8	36.0	19.2	57.7	70.0

Appendix

Table 19: Normalized human support rates and pairwise win rates. Pairwise win rates exclude double selections containing both compared systems and Tie responses.

Question	Ours	OmniNFT	LTX-2	Ours $>$ OmniNFT / Ours $>$ LTX-2
Audio-video consistency	64.3	7.1	28.6	90.0 / 72.7
Overall quality	56.3	18.8	25.0	75.0 / 69.2

Appendix

Table 20: Normalized support rates and pairwise win rates on the fully consistent subset.

Figure 11: Benchmark-wise normalized human support rates. Each row corresponds to one benchmark, and the two columns report audio-video consistency and overall quality.

Video Generation Evaluation

Video generation evaluation measures visual quality, prompt adherence, temporal consistency, physical plausibility, and task-specific behavior in synthesized videos.

See all 183 papers

AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Models
May 23, 2026Jialiang Yang, Bin Xia, Ruihang Chu +6Audio-Video Generation Audio-Visual Alignment
Tsinghua University · The Chinese University of Hong Kong
SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation
Oct 12, 2026Kaisi Guan, Ying Ba, Wenjing Wang +4Audio-Visual Synchronization Audio-Video Generation
Gaoling School of Artificial Intelligence, Renmin University of China · WeChat, Tencent Inc.
VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation
Aug 19, 2026Yinming Huang, Shuyuan Tu, Xi Yan +6Reward Modeling Audio-Video Generation
Fudan University · Shanghai Innovation Institution · Yinwang Intelligent Technology Co., Ltd +1

May 23, 2026J/K move · Enter open · S save

AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Models

Jialiang Yang, Bin Xia, Ruihang Chu, Dingdong Wang+5

Tsinghua University · The Chinese University of Hong Kong

Audio-Video Generation Audio-Visual Alignment Automated Evaluation

Open paper AskPDF arXiv

Rapid advances in audio-video (AV) generation have enabled high-fidelity synthesis with synchronized sound, particularly for human-related scenarios involving speech and interactions. Yet evaluation for AV generation remains at an early stage, with only a few coarse-grained benchmarks for human-related scenarios and relying on limited preset evaluations with generic multimodal LLMs, leading to inaccurate assessments of model capabilities. To address these issues, we introduce AVBench, a fully automated benchmark tailored for human-centric AV generation. AVBench is built on two key designs for comprehensive and accurate evaluation: (i) Human-centric and fine-grained metrics. AVBench integrates ten evaluation dimensions designed for human-centered real-world scenarios, covering visual quality, audio quality, and multi-level consistency across modalities. These practical metrics capture human-related details that existing benchmarks often overlook. (ii) Specialized evaluators via preference learning. To address the lack of specialized training data, we construct large-scale supervision by transforming real-world videos into diverse training pairs with controlled perturbations. After fine-tuning on this high-quality dataset, the evaluators learn to reliably detect subtle cross-modal inconsistencies. Crucially, instead of producing discrete textual judgment, AVBench derives continuous evaluation scores from the model's prediction confidence on binary decisions. This probabilistic scoring mechanism enables a more reliable assessment than traditional VQA-style evaluation and aligns closely with human judgment. Taken together, AVBench offers automated evaluation for AV generation, demonstrates strong potential for data filtering, and serves as a differentiable reward signal for Reinforcement Learning from Human Feedback (RLHF).

Abstract

Video Generation Evaluation

Explore similar work

AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Models

SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation