SyncRA: Learning Temporal Correspondence in Omni-Modal Models

Zelong Xu; Yan Li; Wenhe Hu; Xiyang Hu

cs.CVSep 28, 2026

SyncRA: Learning Temporal Correspondence in Omni-Modal Models

Authors: Zelong Xu, Yan Li, Wenhe Hu, Xiyang Hu

Organizations: University of Wisconsin–Madison · University of Alberta · Arizona State University

Abstract

Recent omni-modal models demonstrate strong perception of audio and visual inputs, yet often struggle to connect what they hear with what they see at the same moment. This weakness in temporal correspondence can cause models to associate spoken cues with the wrong visual scenes, producing plausible answers grounded in incorrect audio-visual pairings. We diagnose this problem through controlled temporal swaps, revealing that model answers do not reliably follow changes in these pairings. To address it, we propose Synchrony-Guided Representation Alignment (SyncRA), a lightweight method for strengthening temporal correspondence between audio and vision. Specifically, SyncRA contrasts intermediate audio-visual representations within each video, aligning matching moments while separating mismatched ones to capture local temporal correspondence within a shared global context. The objective derives supervision directly from existing input timing, requiring no additional annotations and leaving inference unchanged. We evaluate SyncRA across four open omni-modal models spanning different sizes and architectures on five public video benchmarks. SyncRA consistently outperforms answer-only fine-tuning across all model-benchmark combinations, while substantially improving the ability to track changing audio-visual pairings in controlled evaluations. These results demonstrate that lightweight, targeted supervision can effectively strengthen temporal correspondence and translate into broad improvements in audio-visual question answering.

Figures & tables

Figure 1: A motivating example and the swap diagnostic. Left: exchanging only the audio or only the video windows changes which scene is visible while “heart failure” is spoken (correct option A $\to$ C). Right: Qwen3-Omni accuracy on the four input versions (top; AllFour credits a cue only if all four are answered correctly) and predictions on the two illustrated exchanges (bottom; $\dagger$ / $\ddagger$ ). SyncRA answers both correctly. Fine-tuned checkpoints use Run 1.

Figure 2: Overview of Synchrony-Guided Representation Alignment (SyncRA). Top: the ordinary QA path, trained with $\mathcal{L}_{\mathrm{answer}}$ . Bottom (training only): at a selected block $\ell$ , audio and visual states are mean-pooled within native temporal intervals, projected by a shared $W$ with $L_{2}$ normalization, and contrasted with symmetric InfoNCE against other intervals of the same video.

Backbone	Arch.	Base	Vanilla SFT	SyncRA	$\Delta$
Qwen2.5-Omni	Dense	2.25	11.00	20.75	+9.75
MiniCPM-o 4.5	Dense	35.50	57.25	62.25	+5.00
Qwen3-Omni	MoE	13.25	34.00	62.50	+28.50
Nemotron	MoE	0.50	6.00	16.50	+10.50

Table 1: Correspondence-sensitive answering improves across architectures. AllFour accuracy (%) on the same 100 source videos.

\Delta

is SyncRA minus Vanilla SFT in percentage points. All four paired 95% source-video bootstrap intervals exclude zero (Appendix C.3 ).

Benchmark	$n$	Base	Text CoT-SFT	Vanilla SFT	SyncRA	$\Delta$
Qwen2.5-Omni-7B (dense, full SFT)
WorldSense	3,172	45.27	47.92	$50.63\,\pm\,0.32$	$\mathbf{52.74\,\pm\,0.28}$	+2.11
Daily-Omni	1,197	61.74	63.16	$68.84\,\pm\,0.36$	$\mathbf{72.26\,\pm\,0.29}$	+3.43
OmniVideoBench	1,000	25.10	35.50	$37.77\,\pm\,0.31$	$\mathbf{38.57\,\pm\,0.25}$	+0.80
LVOmniBench	1,014	28.21	31.66	$38.03\,\pm\,0.21$	$\mathbf{39.55\,\pm\,0.30}$	+1.51
AVUT-Human	1,733	64.74	67.86	$67.97\,\pm\,0.31$	$\mathbf{71.46\,\pm\,0.34}$	+3.48

Table 2: SyncRA improves general video QA on every model–benchmark combination. Benchmark accuracy (%) across two dense and two MoE backbones. Vanilla SFT and SyncRA: three-run mean

\pm

sample SD; Base and Text CoT-SFT: single checkpoints. Avg-5 weights all five benchmarks equally;

\Delta

is SyncRA minus Vanilla SFT in percentage points (unrounded means).

Figure 3: Temporal correspondence in dense and MoE backbone states. Cosine similarity between unprojected interval-mean audio (rows) and video (columns) states on a fixed validation clip (Appendix D.1 ); all panels share a 0.4–1 color scale.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Questions	Source videos
Training	9,500	1,367
Validation	500	415
Total	10,000	1,782

Appendix

Table 4: Training data. Each of the ten task categories contributes 1,000 questions to the complete 10,000-question subset. Source videos are disjoint between training and validation.

Setting	Value
Optimizer	AdamW; learning rate $10^{-5}$ ; $(\beta_{1},\beta_{2})=(0.9,0.95)$ ; $\epsilon=10^{-8}$ ; weight decay 0.1
Schedule	Linear decay; 80 warmup steps
Budget	2 epochs; 792 optimizer steps per epoch; 1,584 steps total
Effective optimization batch	12 examples: 2 processes $\times$ microbatch 1 $\times$ accumulation 6
Precision and stabilization	bfloat16; float32 normalized contrastive features, similarities, and contrastive cross-entropy; gradient clipping 1.0; gradient checkpointing
Data order	Fixed example order; no additional shuffling

Appendix

Table 5: Optimization settings for Vanilla SFT, SyncRA, and the three auxiliary controls. Runs 1–3 use different random seeds. Qwen3-Omni diagnostic checkpoints for these methods are from Run 1; the fixed checkpoints for the other backbones are described in Appendix C.3 .

Backbone	Update scope
Qwen2.5-Omni-7B	Full understanding/Thinker fine-tuning, including text, visual, and audio modules; speech generation is not trained.
MiniCPM-o 4.5	Full understanding fine-tuning; text-to-speech is disabled.
Qwen3-Omni-30B-A3B	Routed experts and router gates are frozen; other permitted modules are trained. Trainable backbone parameters: 2,715,593,328 / 31,719,205,488 (8.56%).
Nemotron-3-Nano-Omni-30B-A3B	Routed experts are frozen; routers, shared experts, and other permitted modules are trained. Trainable backbone parameters: 3,640,738,752 / 33,015,546,816 (11.03%).

Appendix

Table 6: Trainable backbone parameters. Parameter counts exclude the auxiliary projection. Each backbone uses the same update scope across the compared objectives.

Backbone	Identifier and revision
Qwen2.5-Omni	Qwen/Qwen2.5-Omni-7B ae9e1690543ffd5c0221dc27f79834d0294cba00
MiniCPM-o 4.5	openbmb/MiniCPM-o-4_5 503e754207c94da6bb26850b4469f367c9ea3582
Qwen3-Omni	Qwen/Qwen3-Omni-30B-A3B-Instruct 26291f793822fb6be9555850f06dfe95f2d7e695
Nemotron	nvidia/Nemotron-3-Nano-Omni-30B-A3B- Reasoning-BF16 e5e9932441de940c9a62185c870ea5bcd4cd24e2

Appendix

Table 7: Pretrained model identifiers and exact revisions used for initialization.

Backbone	$d$	Layer/depth	$\|W\|$	Training video and interval grid
Qwen2.5-Omni	3,584	21/28	229,376	2 FPS, at most 256 frames; native 2 s intervals
MiniCPM-o 4.5	4,096	9/36	262,144	1 FPS, at most 180 one-second chunks; native 1 s intervals
Qwen3-Omni	2,048	36/48	131,072	2 FPS, at most 256 frames; temporal patch size divided by actual sampling FPS
Nemotron	2,688	26/52	172,032	2 FPS, at most 256 frames; timestamp-defined 2 s intervals

Appendix

Table 8: Backbone adaptation and training media. All backbones receive full 16 kHz audio. Layer indices are one-indexed and refer to block outputs.

d

is the hidden dimension; the projection has

64d

trainable weights and no bias.

Backbone	Candidates	Fit/validate	Probe media scope	Selected
Qwen2.5-Omni	7, 14, 21	200/415	Full training clip; 2 FPS, at most 256 frames; native 2 s grid	21
MiniCPM-o 4.5	9, 18, 27	200/415	First 60 s; native 1 s audio–video intervals	9
Qwen3-Omni	12, 24, 36	198/408	Full training clip; 2 FPS, at most 256 frames; actual temporal grid	36
Nemotron	13, 26, 39	200/415	First 120 s; 2 FPS; native 2 s intervals	26

Appendix

Table 9: Frozen-Base layer-selection data and selected layers. Counts are usable independent source videos for probe fitting and validation, respectively. Training itself retains the 9,500/500-question split.

	Quarter depth		Half depth		Three-quarter depth
Backbone	Block	R@1	Block	R@1	Block	R@1
Qwen2.5-Omni	7	11.44	14	16.47	21	20.00
MiniCPM-o 4.5	9	78.61	18	74.39	27	36.44
Qwen3-Omni	12	13.07	24	28.20	36	28.30
Nemotron	13	13.30	26	14.06	39	12.91

Appendix

Table 10: Frozen-Base shared-probe validation R@1 (%) for all twelve candidate layers, using Eq. ( 5 ). All candidates satisfy the eligibility band; bold marks the selected block and score for each backbone.

Backbone	Evaluation instruction	Mode/template setting
Qwen2.5-Omni	Direct option-letter answer	No additional thinking mode
MiniCPM-o 4.5	Step-by-step reasoning; final line Final answer: X	enable_thinking=False ; use_tts_template=False
Qwen3-Omni	Step-by-step reasoning; final line Final answer: X	Instruct weights
Nemotron	Neutral question and options	enable_thinking=False

Appendix

Table 11: Text CoT-SFT benchmark evaluation prompts (no examples). Mode and template settings apply during both training and evaluation.

Benchmark	$n$	Evaluation focus
WorldSense	3,172	Integrated real-world audio–video understanding
Daily-Omni	1,197	Everyday audio–video understanding and reasoning
OmniVideoBench	1,000	Multitask audio–video understanding
LVOmniBench	1,014	Long audio–video understanding
AVUT-Human	1,733	Audio-centered video understanding

Appendix

Table 12: Effective benchmark denominators shared by all four backbones.

Cue	O	A	V	AV
Cue 1	$X_{1}$	$X_{3}$	$X_{3}$	$X_{1}$
Cue 2	$X_{2}$	$X_{4}$	$X_{4}$	$X_{2}$
Cue 3	$X_{3}$	$X_{1}$	$X_{1}$	$X_{3}$
Cue 4	$X_{4}$	$X_{2}$	$X_{2}$	$X_{4}$

Appendix

Table 13: Correct scene for each audio cue across input versions.

X_{i}

denotes the scene paired with cue

i

in the original clip. Each source maps these scenes to a fixed order of option letters.

Qwen2.5-Omni (dense)
Checkpoint	AllFour	Overall	O	A	V	AV
Base	2.25	40.75	58.50	33.25	21.00	50.25
Vanilla SFT	11.00	54.19	73.75	51.50	29.75	61.75
Text CoT-SFT	6.25	46.75	62.00	43.00	30.25	51.75
SyncRA	20.75	64.31	77.75	62.75	46.25	70.50
MiniCPM-o 4.5 (dense)

Appendix

Table 14: Complete correspondence-selection results. All entries are percentages. AllFour requires correct answers in all four versions; Overall averages all 1,600 question–version evaluations. O/A/V/AV denote original, audio-exchanged, video-exchanged, and jointly exchanged inputs.

Backbone	Architecture	$\Delta$	95% interval
Qwen2.5-Omni	Dense	$+9.75$	$[5.50,14.00]$
MiniCPM-o 4.5	Dense	$+5.00$	$[1.00,9.00]$
Qwen3-Omni	MoE	$+28.50$	$[23.75,33.25]$
Nemotron	MoE	$+10.50$	$[6.75,14.50]$

Appendix

Table 15: AllFour improvements across backbones. SyncRA minus Vanilla SFT in percentage points on the same 100 source videos. Paired 95% source-video bootstrap intervals keep the diagnostic checkpoints fixed; all four intervals exclude zero.

	AllFour		Overall accuracy
Comparator	$\Delta$	95% interval	$\Delta$	95% interval
Vanilla SFT	$+28.50$	$[23.75,33.25]$	$+13.25$	$[11.19,15.38]$
Permuted pairs	$+31.50$	$[26.75,36.25]$	$+14.56$	$[12.69,16.44]$
Fixed time codes	$+17.75$	$[13.00,22.75]$	$+5.31$	$[3.44,7.19]$
Clip-level AV	$+30.75$	$[26.25,35.00]$	$+13.75$	$[11.69,15.81]$

Appendix

Table 16: Qwen3-Omni: SyncRA minus each comparator, in percentage points. Intervals are paired 95% source-video bootstrap intervals. All checkpoints are fixed to Run 1.

	LVOmniBench		Daily-Omni
Checkpoint	AllFour	Overall	AllFour	Overall
Base	7.89	50.00	14.51	57.87
Text CoT-SFT	5.26	43.75	13.58	56.48
Vanilla SFT	25.00	67.11	36.11	73.46
Permuted pairs	28.95	67.76	31.48	71.68
Fixed time codes	36.84	76.32	46.60	81.10

Appendix

Table 17: Qwen3-Omni correspondence selection by source benchmark (%). LVOmniBench contributes 19 sources, 76 cues, and 304 questions per checkpoint; Daily-Omni contributes 81 sources, 324 cues, and 1,296 questions.

Figure 5: Qwen3-Omni Run 1 scene-prediction matrices for Vanilla SFT and SyncRA. Each row aggregates one cue index over 100 source videos and sums to 100%. Columns denote original scene identities

X_{1},\ldots,X_{4}

, with option letters mapped back to these identities. Outlined cells mark the correct scene: the diagonal for original and joint-swap inputs, and the exchanged pairing (

1\leftrightarrow 3

2\leftrightarrow 4

) for audio- and video-swap inputs.

	R@1 (%)		MAE (s)
Space	Vanilla	SyncRA	Vanilla	SyncRA
Unprojected states	13.39	96.91	7.176	0.044
Trained SyncRA $W$	9.03	97.68	11.389	0.034
Spectrum-matched random	7.58	80.72	15.332	1.334

Appendix

Table 18: Matching beyond the training projection. Bidirectional same-interval retrieval, averaged equally over 100 validation videos. Each projected readout uses the same matrix for both checkpoints and modalities. MAE is measured in seconds.

Method	AllFour	Native R@1	Daily-Omni
Vanilla SFT	34.00	13.39	75.86
Cosine-only	40.50	17.92	78.20
Pairwise logistic ranking	46.25	27.47	78.36
SyncRA (InfoNCE)	62.50	96.91	78.95

Appendix

Table 19: Loss-function ablation on Qwen3-Omni (Run 1). Scores are percentages. Native R@1 uses unprojected block-36 states.

Checkpoint	Audio input	Unprojected states	SyncRA $W$	$W$ complement
Vanilla	Original	12.75	8.35	12.69
Vanilla	Donor replacement	9.28	6.10	9.10
Vanilla	Two-interval delay	7.82	6.48	7.76
SyncRA	Original	97.07	97.78	96.87
SyncRA	Donor replacement	96.74	97.41	96.59
SyncRA	Two-interval delay	96.45	97.47	96.21

Appendix

Table 20: Interior-query retrieval under audio replacement and delay. Every value is bidirectional, video-equal R@1 (%). Queries exclude the first and last two native intervals; all intervals remain candidates. The complement removes the row-space component of the same SyncRA

W

used for both checkpoints.

	Vanilla		SyncRA
Space	Current interval	Content origin	Current interval	Content origin
Unprojected states	7.82	7.86	96.45	0.01
SyncRA $W$	6.48	6.15	97.47	0.02

Appendix

Table 21: Retrieval targets after a two-interval audio delay. Current-interval and content-origin targets are evaluated on the same interior queries and complete candidate set. Entries are R@1 (%).

Fit $\to$ evaluate	Base MAE	Vanilla MAE	SyncRA MAE	Difference	Paired 95% interval
Audio $\to$ audio	8.996	8.188	6.680	$-1.508$	$[-1.881,-1.168]$
Video $\to$ video	5.675	5.157	4.651	$-0.506$	$[-0.851,-0.163]$
Audio $\to$ video	13.133	12.685	15.741	$+3.056$	$[1.677,4.450]$
Video $\to$ audio	18.107	18.051	17.290	$-0.760$	$[-1.484,-0.007]$

Appendix

Table 22: Elapsed-time ridge decoding. MAEs and differences are seconds; negative differences favor SyncRA. Differences are computed before rounding. Paired intervals resample evaluation videos while holding models and decoders fixed. Source-modal centering and intercept remain fixed under transfer. Base is the released checkpoint before fine-tuning; Vanilla-minus-Base differences are reported in the text.

Control	Base	Vanilla	SyncRA
Shuffled time labels: audio $\to$ audio	30.816	30.249	29.617
Shuffled time labels: video $\to$ video	31.041	30.196	29.044
Shuffled time labels: audio $\to$ video	39.298	54.523	30.236
Shuffled time labels: video $\to$ audio	60.822	55.117	62.424
Training-label-mean constant	29.276	29.276	29.276

Appendix

Table 23: Controls for elapsed-time decoding. Values are validation MAE in seconds under video-balanced evaluation. Time labels are shuffled once within each training video, with the same shuffled labels shared across modalities and checkpoints. The constant predictor uses the training-label mean.

Candidate pool	Space	Vanilla	SyncRA	Clip-level AV
All 100 sources	Unprojected states	8.50	15.00	18.50
All 100 sources	Clip-level AV $W$	5.50	36.50	19.00
All 100 sources	SyncRA $W$	7.50	68.50	20.50
10 duration neighbors	Unprojected states	34.50	41.50	50.00
10 duration neighbors	Clip-level AV $W$	30.00	65.00	54.00
10 duration neighbors	SyncRA $W$	31.50	86.00	57.50

Appendix

Table 24: Clip-level cross-video retrieval, R@1 (%). Both candidate pools use the same 100 validation sources. The reported

W

is shared across checkpoints, including when its training objective differs from the evaluated checkpoint’s objective.

Supervision	Native R@1		AllFour	Daily-Omni
block	Block 24	Block 36
24	68.37	39.87	60.00	77.36
36 (default)	19.96	96.91	62.50	78.95

Appendix

Table 25: Supervision-depth ablation on Qwen3-Omni (Run 1). Scores are percentages. Native R@1 uses unprojected states at the indicated readout block. The AllFour difference (block 36 minus block 24) is +2.50 pp, with a paired 95% source-video bootstrap interval of

[-2.50,+7.75]

. Both configurations use the training and evaluation protocol described above.

$\lambda$	Native R@1	AllFour	Daily-Omni
0.03	90.85	58.00	78.53
0.1 (default)	96.91	62.50	78.95
0.3	93.72	58.75	78.36

Appendix

Table 26: Auxiliary-weight ablation on Qwen3-Omni (seed 1). Native R@1 uses unprojected block-36 states on the 100 validation videos. Paired 95% source-video bootstrap intervals for the AllFour advantage of

\lambda=0.1

over

\lambda=0.03

and

\lambda=0.3

are

[+1.00,+8.00]

and

[+0.25,+7.25]

points, both excluding zero.

Benchmark	Vanilla SFT	Permuted pairs	Fixed time codes	Clip-level AV	SyncRA
WorldSense	$50.63\,\pm\,0.32$	$50.26\,\pm\,0.22$	$49.73\,\pm\,0.17$	$50.42\,\pm\,0.16$	$\mathbf{52.74\,\pm\,0.28}$
Daily-Omni	$68.84\,\pm\,0.36$	$68.50\,\pm\,0.17$	$69.53\,\pm\,0.35$	$69.37\,\pm\,0.26$	$\mathbf{72.26\,\pm\,0.29}$
OmniVideoBench	$37.77\,\pm\,0.31$	$38.23\,\pm\,0.15$	$37.70\,\pm\,0.20$	$37.33\,\pm\,0.15$	$\mathbf{38.57\,\pm\,0.25}$
LVOmniBench	$38.03\,\pm\,0.21$	$39.32\,\pm\,0.21$	$37.34\,\pm\,0.06$	$39.51\,\pm\,0.15$	$\mathbf{39.55\,\pm\,0.30}$
AVUT-Human	$67.97\,\pm\,0.31$	$69.76\,\pm\,0.44$	$68.05\,\pm\,0.22$	$69.28\,\pm\,0.23$	$\mathbf{71.46\,\pm\,0.34}$
Avg-5	$52.65\,\pm\,0.12$	$53.22\,\pm\,0.15$	$52.47\,\pm\,0.09$	$53.18\,\pm\,0.12$	$\mathbf{54.92\,\pm\,0.11}$

Appendix

Table 27: Qwen2.5-Omni-7B: complete training-objective summary. Accuracy in %, mean

\pm

sample SD over three independent training runs with different random seeds. Bold marks the highest mean in each row.

Benchmark	Vanilla SFT	Permuted pairs	Fixed time codes	Clip-level AV	SyncRA
WorldSense	$56.78\,\pm\,0.19$	$51.53\,\pm\,0.18$	$52.39\,\pm\,0.08$	$52.36\,\pm\,0.14$	$\mathbf{57.61\,\pm\,0.13}$
Daily-Omni	$79.67\,\pm\,0.29$	$78.08\,\pm\,0.34$	$77.75\,\pm\,0.10$	$77.08\,\pm\,0.38$	$\mathbf{80.51\,\pm\,0.21}$
OmniVideoBench	$35.53\,\pm\,0.32$	$37.10\,\pm\,0.17$	$35.80\,\pm\,0.30$	$36.83\,\pm\,0.12$	$\mathbf{37.40\,\pm\,0.26}$
LVOmniBench	$31.72\,\pm\,0.21$	$32.15\,\pm\,0.26$	$\mathbf{33.53\,\pm\,0.10}$	$30.24\,\pm\,0.06$	$31.95\,\pm\,0.26$
AVUT-Human	$78.27\,\pm\,0.23$	$77.17\,\pm\,0.03$	$77.15\,\pm\,0.23$	$76.50\,\pm\,0.03$	$\mathbf{79.28\,\pm\,0.21}$
Avg-5	$56.39\,\pm\,0.18$	$55.21\,\pm\,0.02$	$55.32\,\pm\,0.00$	$54.60\,\pm\,0.11$	$\mathbf{57.35\,\pm\,0.15}$

Appendix

Table 28: MiniCPM-o 4.5: complete training-objective summary. Accuracy in %, mean

\pm

sample SD over three independent training runs with different random seeds. Bold marks the highest mean in each row.

Benchmark	Vanilla SFT	Permuted pairs	Fixed time codes	Clip-level AV	SyncRA
WorldSense	$56.26\,\pm\,0.21$	$57.28\,\pm\,0.08$	$56.88\,\pm\,0.12$	$56.95\,\pm\,0.10$	$\mathbf{57.36\,\pm\,0.13}$
Daily-Omni	$76.16\,\pm\,0.27$	$78.72\,\pm\,0.17$	$78.47\,\pm\,0.21$	$78.84\,\pm\,0.13$	$\mathbf{79.09\,\pm\,0.13}$
OmniVideoBench	$46.07\,\pm\,0.32$	$\mathbf{47.17\,\pm\,0.15}$	$46.37\,\pm\,0.25$	$46.83\,\pm\,0.15$	$47.07\,\pm\,0.25$
LVOmniBench	$40.89\,\pm\,0.23$	$41.95\,\pm\,0.25$	$40.76\,\pm\,0.11$	$42.93\,\pm\,0.06$	$\mathbf{43.92\,\pm\,0.21}$
AVUT-Human	$78.09\,\pm\,0.27$	$79.11\,\pm\,0.25$	$79.48\,\pm\,0.12$	$79.00\,\pm\,0.10$	$\mathbf{79.71\,\pm\,0.23}$
Avg-5	$59.50\,\pm\,0.14$	$60.85\,\pm\,0.06$	$60.39\,\pm\,0.14$	$60.91\,\pm\,0.06$	$\mathbf{61.43\,\pm\,0.08}$

Appendix

Table 29: Qwen3-Omni-30B-A3B: complete training-objective summary. Accuracy in %, mean

\pm

sample SD over three independent training runs with different random seeds. Bold marks the highest mean in each row.

Benchmark	Vanilla SFT	Permuted pairs	Fixed time codes	Clip-level AV	SyncRA
WorldSense	$54.32\,\pm\,0.22$	$55.43\,\pm\,0.05$	$\mathbf{55.72\,\pm\,0.07}$	$55.54\,\pm\,0.27$	$55.65\,\pm\,0.21$
Daily-Omni	$75.80\,\pm\,0.26$	$76.33\,\pm\,0.38$	$77.44\,\pm\,0.14$	$76.50\,\pm\,0.05$	$\mathbf{78.42\,\pm\,0.27}$
OmniVideoBench	$43.47\,\pm\,0.15$	$44.20\,\pm\,0.10$	$44.10\,\pm\,0.10$	$43.93\,\pm\,0.15$	$\mathbf{44.93\,\pm\,0.15}$
LVOmniBench	$41.58\,\pm\,0.15$	$41.52\,\pm\,0.10$	$41.39\,\pm\,0.15$	$42.31\,\pm\,0.10$	$\mathbf{42.93\,\pm\,0.15}$
AVUT-Human	$73.88\,\pm\,0.30$	$75.26\,\pm\,0.22$	$75.19\,\pm\,0.12$	$75.30\,\pm\,0.06$	$\mathbf{77.67\,\pm\,0.31}$
Avg-5	$57.81\,\pm\,0.09$	$58.55\,\pm\,0.12$	$58.77\,\pm\,0.05$	$58.72\,\pm\,0.04$	$\mathbf{59.92\,\pm\,0.11}$

Appendix

Table 30: Nemotron-3-Nano-Omni-30B-A3B: complete training-objective summary. Accuracy in %, mean

\pm

sample SD over three independent training runs with different random seeds. Bold marks the highest mean in each row.

Objective	Run 1 $C/n$ (%)	Run 2 $C/n$ (%)	Run 3 $C/n$ (%)	Mean $\pm$ SD
WorldSense ( $n=3,172$ )
Vanilla SFT	1,595/3,172 (50.28)	1,608/3,172 (50.69)	1,615/3,172 (50.91)	$50.63\,\pm\,0.32$
SyncRA	1,670/3,172 (52.65)	1,683/3,172 (53.06)	1,666/3,172 (52.52)	$52.74\,\pm\,0.28$
Permuted pairs	1,593/3,172 (50.22)	1,588/3,172 (50.06)	1,602/3,172 (50.50)	$50.26\,\pm\,0.22$
Fixed time codes	1,583/3,172 (49.91)	1,577/3,172 (49.72)	1,572/3,172 (49.56)	$49.73\,\pm\,0.17$
Clip-level AV	1,595/3,172 (50.28)	1,605/3,172 (50.60)	1,598/3,172 (50.38)	$50.42\,\pm\,0.16$

Appendix

Table 31: Qwen2.5-Omni-7B: three independent training runs with different random seeds. Each run cell gives correct count

C/n

and accuracy in %; means and sample SDs use unrounded accuracies. Run numbers are consistent across tasks.

Objective	Run 1 $C/n$ (%)	Run 2 $C/n$ (%)	Run 3 $C/n$ (%)	Mean $\pm$ SD
WorldSense ( $n=3,172$ )
Vanilla SFT	1,795/3,172 (56.59)	1,807/3,172 (56.97)	1,801/3,172 (56.78)	$56.78\,\pm\,0.19$
SyncRA	1,826/3,172 (57.57)	1,824/3,172 (57.50)	1,832/3,172 (57.76)	$57.61\,\pm\,0.13$
Permuted pairs	1,637/3,172 (51.61)	1,628/3,172 (51.32)	1,639/3,172 (51.67)	$51.53\,\pm\,0.18$
Fixed time codes	1,659/3,172 (52.30)	1,664/3,172 (52.46)	1,662/3,172 (52.40)	$52.39\,\pm\,0.08$
Clip-level AV	1,660/3,172 (52.33)	1,657/3,172 (52.24)	1,666/3,172 (52.52)	$52.36\,\pm\,0.14$

Appendix

Table 32: MiniCPM-o 4.5: three independent training runs with different random seeds. Each run cell gives correct count

C/n

and accuracy in %; means and sample SDs use unrounded accuracies. Run numbers are consistent across tasks.

Objective	Run 1 $C/n$ (%)	Run 2 $C/n$ (%)	Run 3 $C/n$ (%)	Mean $\pm$ SD
WorldSense ( $n=3,172$ )
Vanilla SFT	1,777/3,172 (56.02)	1,789/3,172 (56.40)	1,788/3,172 (56.37)	$56.26\,\pm\,0.21$
SyncRA	1,815/3,172 (57.22)	1,820/3,172 (57.38)	1,823/3,172 (57.47)	$57.36\,\pm\,0.13$
Permuted pairs	1,820/3,172 (57.38)	1,816/3,172 (57.25)	1,815/3,172 (57.22)	$57.28\,\pm\,0.08$
Fixed time codes	1,806/3,172 (56.94)	1,800/3,172 (56.75)	1,807/3,172 (56.97)	$56.88\,\pm\,0.12$
Clip-level AV	1,810/3,172 (57.06)	1,804/3,172 (56.87)	1,805/3,172 (56.90)	$56.95\,\pm\,0.10$

Appendix

Table 33: Qwen3-Omni-30B-A3B: three independent training runs with different random seeds. Each run cell gives correct count

C/n

and accuracy in %; means and sample SDs use unrounded accuracies. Run numbers are consistent across tasks.

Objective	Run 1 $C/n$ (%)	Run 2 $C/n$ (%)	Run 3 $C/n$ (%)	Mean $\pm$ SD
WorldSense ( $n=3,172$ )
Vanilla SFT	1,731/3,172 (54.57)	1,720/3,172 (54.22)	1,718/3,172 (54.16)	$54.32\,\pm\,0.22$
SyncRA	1,760/3,172 (55.49)	1,773/3,172 (55.90)	1,763/3,172 (55.58)	$55.65\,\pm\,0.21$
Permuted pairs	1,757/3,172 (55.39)	1,760/3,172 (55.49)	1,758/3,172 (55.42)	$55.43\,\pm\,0.05$
Fixed time codes	1,768/3,172 (55.74)	1,765/3,172 (55.64)	1,769/3,172 (55.77)	$55.72\,\pm\,0.07$
Clip-level AV	1,765/3,172 (55.64)	1,752/3,172 (55.23)	1,768/3,172 (55.74)	$55.54\,\pm\,0.27$

Appendix

Table 34: Nemotron-3-Nano-Omni-30B-A3B: three independent training runs with different random seeds. Each run cell gives correct count

C/n

and accuracy in %; means and sample SDs use unrounded accuracies. Run numbers are consistent across tasks.

Backbone	Benchmark	Base: $C/n$ (%)	Text CoT-SFT: $C/n$ (%)
Qwen2.5-Omni	WorldSense	1,436/3,172 (45.27)	1,520/3,172 (47.92)
	Daily-Omni	739/1,197 (61.74)	756/1,197 (63.16)
	OmniVideoBench	251/1,000 (25.10)	355/1,000 (35.50)
	LVOmniBench	286/1,014 (28.21)	321/1,014 (31.66)
	AVUT-Human	1,122/1,733 (64.74)	1,176/1,733 (67.86)
MiniCPM-o 4.5	WorldSense	1,750/3,172 (55.17)	1,767/3,172 (55.71)

Appendix

Table 35: Single evaluations of Base and Text CoT-SFT. Each observed cell gives integer

C/n

and accuracy in %. These fixed-checkpoint evaluations have no training-run SD.

Backbone	Macro	Objective	Run 1	Run 2	Run 3	Mean $\pm$ SD
Qwen2.5-Omni	Avg-5	Vanilla SFT	52.54	52.63	52.78	$52.65\,\pm\,0.12$
		SyncRA	55.05	54.87	54.83	$54.92\,\pm\,0.11$
		Permuted pairs	53.20	53.07	53.38	$53.22\,\pm\,0.15$
		Fixed time codes	52.57	52.41	52.43	$52.47\,\pm\,0.09$
		Clip-level AV	53.16	53.31	53.08	$53.18\,\pm\,0.12$
	Avg-3	Vanilla SFT	62.31	62.40	62.74	$62.48\,\pm\,0.23$

Appendix

Table 36: Per-run task-equal macro accuracies (%): dense backbones. Each run is averaged over tasks before computing the across-run mean and sample SD.

Backbone	Macro	Objective	Run 1	Run 2	Run 3	Mean $\pm$ SD
Qwen3-Omni	Avg-5	Vanilla SFT	59.38	59.46	59.65	$59.50\,\pm\,0.14$
		SyncRA	61.34	61.48	61.47	$61.43\,\pm\,0.08$
		Permuted pairs	60.92	60.80	60.82	$60.85\,\pm\,0.06$
		Fixed time codes	60.39	60.25	60.53	$60.39\,\pm\,0.14$
		Clip-level AV	60.97	60.85	60.90	$60.91\,\pm\,0.06$
	Avg-3	Vanilla SFT	70.06	70.15	70.30	$70.17\,\pm\,0.12$

Appendix

Table 37: Per-run task-equal macro accuracies (%): mixture-of-experts backbones. Each run is averaged over tasks before computing the across-run mean and sample SD.

Explore similar work

Sep 17, 2026cs.CV

AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Models

Omni models can describe video content, but can they locate events in time, preserve event order, and judge audio-visual synchronization? We introduce AVTrace (Audio-Visual Temporal Reasoning Assessment and Capability Evaluation), a silver-standard diagnostic suite spanning onset and span grounding, synchronization, next-step prediction, cross-modal localization, chain parsing, and event-conditioned comprehension. It contains 34,114 training examples and category-balanced development and test splits of 3,500 and 7,000 examples. We evaluate five open omni models under their respective input configurations using reference-blind response normalization followed by deterministic scoring. All five off-the-shelf systems score below the test split's majority-label baseline of 0.556 on synchronization verification, and obtain low scores on chain parsing and event-conditioned grounding and comprehension. Development-set perturbations reveal task-dependent sensitivity in Qwen3-Omni-30B to modality removal and changes in visual input processing, without isolating their underlying causes. Parameter-efficient temporal post-training improves Gemma4-E4B-it on several benchmark metrics. On three external image benchmarks, task metrics change modestly, including some degradations, while teacher-forcing perplexity decreases. Together, these findings show that semantic reference-text overlap should not be treated as a proxy for temporal localization, and that AVTrace can identify task-specific weaknesses while providing a testbed for temporal post-training.

Longyin Zhang, Parth Sakhare Mahendra, Chengwei Wei +4

Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · Centre for Frontier AI Research (CFAR), A*STAR, Singapore

Audio-Visual Reasoning Omni-Modal Cross-Modal

Ask PDF

Aug 10, 2026cs.AI

Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models

Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time. Yet existing audio-language models often represent clips as global acoustic events, while vision-language models lack the spatial audio cues needed to localize and track individual sources. To evaluate this missing capability, we introduce ST-OmniQA, a spatio-temporal audio-visual question-answering benchmark built from panoramic videos paired with synchronized first-order Ambisonics (FOA) audio of moving sound sources. It contains 40K videos and 400K question-answer pairs organized into four capability levels covering sound-event recognition, direction of arrival, source distance, motion trajectories, and temporally grounded audio-visual reasoning. Building on this benchmark, we propose ST-Omni-R1, which integrates FOA-derived semantic and trajectory representations with panoramic visual context and is trained through progressive curriculum learning and reasoning-tree reinforcement learning. ST-Omni-R1 achieves 77.83% average semantic accuracy across the four levels, compared with 37.28% for the best evaluated baseline. Results on three public spatial-audio benchmarks further indicate that its learned spatial and motion representations transfer beyond ST-OmniQA.

Zhi Zeng, Cheng Zhang, Zesheng Yang +9

Xi’an Jiaotong University · The Hong Kong Polytechnic University · National University of Singapore +2

Audio-Visual Reasoning Spatial Audio Omni-Modal Omni

Ask PDF

May 21, 2026cs.CL

LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning

Joint audio-visual reasoning is essential for omnimodal understanding, yet current multimodal large language models (MLLMs) still struggle when reasoning requires fine-grained evidence from both modalities. A central limitation is that explicit text-based chain-of-thought (CoT) compresses continuous audio-visual signals into discrete tokens, weakening temporal grounding and shifting intermediate reasoning toward language priors. We argue that a unified latent space is a better medium for such reasoning because it preserves dense sensory information while remaining compatible with autoregressive generation. Based on this insight, we propose \textbf{LatentOmni}, a cross-modal reasoning framework that interleaves textual reasoning with audio-visual latent states. LatentOmni introduces feature-level supervision to align latent reasoning states with task-relevant sensory features and uses Omni-Sync Position Embedding (OSPE) to maintain temporal consistency between latent audio and visual states. We further construct \textbf{LatentOmni-Instruct-35K}, a dataset of audio-visual interleaved reasoning trajectories for supervising latent-space reasoning. Comprehensive evaluation across multiple audio-visual reasoning benchmarks demonstrates that LatentOmni achieves the best performance among the evaluated open-source models and consistently outperforms the Explicit Text CoT baseline, supporting latent-space joint reasoning as a promising path toward stronger omnimodal understanding.

Yifan Dai, Zhenhua Wu, Bohan Zeng +18

School of AI, Shanghai Jiao Tong University · 2Kling Team, Kuaishou Technology · 3Peking University +5

Audio-Visual Reasoning Omni-Modal Multimodal Reasoning Multimodal Large Language Models Latent Space

Ask PDF