SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

Kaisi Guan; Ying Ba; Wenjing Wang; Xihua Wang; Chen Li; Jing Lyu; Ruihau Song

cs.CVOct 12, 2026

SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

Authors: Kaisi Guan, Ying Ba, Wenjing Wang, Xihua Wang, Chen Li, Jing Lyu, Ruihau Song

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · WeChat, Tencent Inc.

Abstract

Recent years have witnessed rapid progress in joint audio--visual generation. However, audio--visual synchronization remains a challenge, as generated speech and sound often temporally misalign with visual events. A key bottleneck is the lack of human-annotated data, reliable evaluation benchmarks, and synchronization-aware evaluation models for generated audio--visual outputs. To address these limitations, we conduct a comprehensive study spanning data, benchmarking, and reward modeling. We first introduce SyncReward-Data-80K, a dataset containing 80K professional synchronization ratings for clips from eight audio--visual generators. A taxonomy-guided construction pipeline covers four sound-event families: Speech, Onset, Instrument, and Ambient. We further construct SyncReward-Bench, a held-out benchmark of 1K clips, each independently rated by multiple professionals. Using this benchmark, we systematically evaluate specialized synchronization metrics and omni-modal models, revealing their substantial misalignment with human judgments. Finally, we develop SyncReward, a reward model that combines semantic--temporal contrastive pretraining on real clips with human supervision on generated clips. As an evaluation metric, SyncReward achieves the strongest agreement with human ratings among the evaluated methods. As a reward, it improves synchronization through inference-time best-of- $N$ selection and reinforcement learning based post-training. We will release the data, benchmark, model, and code at https://github.com/WeChatCV/SyncReward.

Figures & tables

Figure 1: Illustration of SyncReward-Data-80K : human annotations of synchronization in generated audio–visual clips. The dataset contains 80K professional synchronization ratings on outputs from eight generators, covering four sound-event families: Speech, Onset, Instrument, and Ambient. A three-level taxonomy organizes these events into 16 classes and 58 fine-grained categories. Surrounding examples show sampled frames and brief descriptions of visible sound sources and their associated sounds, linked to the corresponding categories.

Figure 2: Sound-event taxonomy and human-rated examples. Examples span four sound-event families, 16 classes, and eight audio–visual generators. Headers report dataset-level rating counts for each family and class. Each card presents three sampled video frames, an event description, the generator name, and a human synchronization score. Scores indicate unsynchronized (0), partially synchronized (1), and fully synchronized (2) clips. Examples are selected to illustrate category coverage and different synchronization levels, rather than the dataset’s score distribution.

Figure 3: Two-stage training of SyncReward. Stage 1 trains the audio and video encoders through contrastive learning with aligned segment pairs, within-clip temporal negatives, and cross-clip semantic negatives from the same batch. Stage 2 freezes both encoders and trains a cross-modal Transformer and score head using human synchronization ratings, combining regression and pairwise ranking losses. The model outputs a clip-level synchronization score in $[0,2]$ . Flame and snowflake icons indicate trainable and frozen modules, respectively.

Synchronization metrics
Method	Params	Speech		Onset		Instrument		Ambient
Method	Params	Spearman $\rho$	Pair Acc	Spearman $\rho$	Pair Acc	Spearman $\rho$	Pair Acc	Spearman $\rho$	Pair Acc
AV-Align ( Yariv et al., 2024 )	–	4.4	51.8	7.0	47.3	14.9	43.2	15.3	55.9
CAVP ( Luo et al., 2023 )	114M	31.1	62.2	36.2	64.2	42.5	66.9	43.7	67.4
SparseSelector ( Iashin et al., 2022 )	55M	47.2	65.5	48.3	68.1	48.4	68.9	53.1	69.3
Synchformer ( Iashin et al., 2024 )	237M	32.9	60.4	47.3	67.9	32.0	61.9	56.5	70.7

Table 1: SyncReward as a metric. We report Spearman’s rank correlation (

\rho\times 100

) and pairwise ranking accuracy (Pair Acc., %) for four on-screen sound categories: Speech, Onset, Instrument, and Ambient. Clips are grouped by the sound category used to construct their prompts. All comparisons are rank-based and require no score calibration. The best and second-best results are shown in bold and underlined , respectively. The Instrument subset is smaller because music-dominant clips were removed from the evaluation set.

Inference-time Best-of-16 selection
Method	Pairwise Preference Score (%)		Automatic Metrics
Method	Human $\uparrow$	Gemini $\uparrow$	SyncR. $\uparrow$	DeSync $\downarrow$	AV-Align $\uparrow$	JavisScore $\uparrow$
LTX 2.3 (Base Model)	32.31	47.97	0.935	0.327	0.306	0.191
AV-Align	36.15	40.83	0.953	0.300	0.448	0.184
Synchformer	38.90	41.16	0.985	0.125	0.293	0.192
SynthSync	52.57	54.20	0.991	0.267	0.292	0.197

Table 2: SyncReward as a reward. We evaluate synchronization rewards for inference-time Best-of-16 selection and DiffusionNFT post-training. Human and Gemini preference scores are measured on matched held-out prompts, while complementary automatic metrics are computed on the same generated videos. The current preference scores are provisional estimates from base-referenced comparisons and will be replaced by complete all-pairs evaluation.

Training stages
Variant	Spearman $\rho\uparrow$	Pair Acc. (%) $\uparrow$
PE-AV initialization	26.6	60.4
Stage 1 only	60.5	74.6
Stage 2 only	40.9	66.3
Stage 1 ablation
w/o semantic	74.1	81.4

Table 3: Key ablations of SyncReward. We evaluate the contributions of the training stages and the semantic-temporal contrastive objective on SyncReward-Bench. Additional design ablations are reported in the Appendix.

Figure 4: Clip representations and human synchronization ratings. Each panel shows a separate PCA projection fitted on validation features, with a class-balanced subset of held-out test clips colored by human synchronization ratings. Curves show class-wise PC1 densities; ellipses indicate one-standard-deviation covariance contours, and connected markers denote class means in rating order. Axis labels report explained variance, and coordinates are not shared across models.

Figure 5: Qualitative effects of SyncReward-guided DiffusionNFT post-training. Each example compares outputs before (top) and after (bottom) post-training using three video frames and the corresponding log-mel spectrogram. Dashed lines mark frame timestamps $t_{1}$ – $t_{3}$ , and displayed scores are SyncReward predictions. Left: Post-training improves visible lip articulation during speech. Right: Vocal correspondence improves, but the instrument specified by the prompt is absent, illustrating a trade-off between synchronization and prompt fidelity.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Category	Subcategories
Speech	Conversation (2)	Monologue; multi-speaker conversation.
	Vocal performance (3)	Singing; expressive speech; challenging tongue twisters.
	Stylized characters (5)	Japanese anime; Western cartoons; children’s animation; 3D film/game characters; puppets and robots without conventional lips.
Onset	Human actions (4)	Body contact; eating and chewing; breathing actions; footsteps.
	Object interactions (5)	Impacts and breakage; object manipulation; material processing; liquid interactions; tool use.
	Mechanical and electronic events (4)	Device startup; electronic notification sounds; vehicle operation; rhythmic mechanisms.

Appendix

Table 4: The SyncReward sound-event taxonomy. The three levels comprise four event families, 16 intermediate categories, and 58 fine-grained subcategories. Numbers in parentheses indicate the number of subcategories in each category. Language tags are recorded separately for Speech.

Data and inputs
Setting	Stage 1	Stage 2
Training data	AudioSet + VGGSound	SyncReward-Data-80K
Filtering / labels	ImageBind similarity $\geq 0.2$	Human ratings: $0/1/2$
Clip duration	$4.75$ s	$4.75$ s
Segmentation	$18\times 0.5$ s	$18\times 0.5$ s
Segment stride	$0.25$ s	$0.25$ s

Appendix

Table 5: SyncReward training hyperparameters. Settings for temporal pretraining (Stage 1) and human alignment (Stage 2).

Generator	LTX-2.3 $22$ B, distilled-1.1, $8$ -step Euler, CFG/STG off
Resolution / length	$512{\times}768$ , $121$ frames @ $24$ fps, $48$ kHz audio
Policy	LoRA $r{=}32$ , $\alpha{=}64$ , all attn. q/k/v/o (video+audio), $214$ M
Group size $k$ / prompts per round	$32$ / $2$ ( $64$ rollouts)
Micro-batch / rollouts per step	$2$ per GPU / $32$
Advantage	group mean, batch std, clip $\pm 5$ , $A_{\max}{=}5$
Trained $\sigma$	$4$ highest of $8$ ( $\sigma{\geq}0.98$ )

Appendix

Table 6: DiffusionNFT post-training recipe.

Audio-Visual Synchronization

Audio-visual synchronization studies how speech, music, and other sounds align in time and meaning with visible actions and events.

See all 64 papers

Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Models
Jul 10, 2026Yijie Qian, Juncheng Wang, Chao Xu +6Audio-Video Generation Group Relative Policy Optimization
Zhejiang University, Hangzhou, China · IROOTECH TECHNOLOGY, China · The Hong Kong Polytechnic University, Hong Kong, China +1
AV-SyncBench: Decoupled Benchmarking of Temporal and Semantic Audio-Visual Synchronization
Jul 1, 2026Tianhong Zhou, Mingyang Han, Boyu Li +8Audio-Visual Synchronization Audio-Visual Alignment
Tsinghua University, Beijing, China · Alibaba Group, China · Fudan University, Shanghai, China
AVC-Reward: Event-Centric Audio-Video Consistency Evaluation
Oct 15, 2026Yuruo Wu, Kaihang Pan, Kai Shen +6Video Generation Evaluation Audio-Video Generation
Zhejiang University · HiThink Research · Fudan University +1

Jul 10, 2026J/K move · Enter open · S save

Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Models

Yijie Qian, Juncheng Wang, Chao Xu, Huihan Wang+5

Zhejiang University, Hangzhou, China · IROOTECH TECHNOLOGY, China · The Hong Kong Polytechnic University, Hong Kong, China+1

Audio-Video Generation Group Relative Policy Optimization Audio-Visual Synchronization Generative AI Evaluation

Open paper AskPDF arXiv

As audio-visual generative models evolve into world simulators, cross-modal synchronization stands as a critical proxy for assessing the consistency of world dynamics and causality in generated content. However, existing evaluation metrics presume structural correctness, reducing synchronization to mere temporal alignment. Consequently, they fail on generative outputs, especially when exhibiting structural hallucinations and asymmetric cross-modal relations, which currently mandate expert human annotation to assess synchronization. This dependency introduces a critical paradox: human evaluators rely on relative, reference-dependent comparisons, whereas automated metrics require reference-free, absolute scalars. We resolve this paradox by proposing a framework that distills relative human perception into a continuous, globally consistent metric. First, we introduce SynthSync, a dataset of generative failures ranked via pairwise human annotations. Second, we adapt the Omni-LLM equipped with a continuous latent projection to translate relative human rankings into continuous absolute values. Third, we propose Real-Valued Group Relative Policy Optimization (

\mathbb{R}

-GRPO) to internalize the global causal structure of synchronization via listwise score distributions. Empirically, our metric achieves state-of-the-art human preference alignment. We leverage this estimator to establish a standardized benchmark, advancing AV-Gen assessment from low-level signal correlation to visually grounded causality.

Abstract

Audio-Visual Synchronization

Explore similar work

Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Models

AV-SyncBench: Decoupled Benchmarking of Temporal and Semantic Audio-Visual Synchronization

AVC-Reward: Event-Centric Audio-Video Consistency Evaluation