OmniGround: Strengthening Omni-Modal Temporal Grounding via Attention-Consistency Training

Zixuan Chen; Depeng Wang; Hao Lin; Ke Xu; Ya Guo; Huijia Zhu; Tanfeng Sun; Xinghao Jiang

cs.CVOct 19, 2026

OmniGround: Strengthening Omni-Modal Temporal Grounding via Attention-Consistency Training

Authors: Zixuan Chen, Depeng Wang, Hao Lin, Ke Xu, Ya Guo, Huijia Zhu, Tanfeng Sun, Xinghao Jiang

Organizations: Shanghai Jiao Tong University · Ant Group · Independent Researcher

Abstract

Omni-modal large language models can jointly understand video, audio, and text, yet they remain weak at temporal grounding, a capability essential for real-world audio-visual applications. To understand this limitation, we look beyond final predictions and examine the model's internal temporal evidence. We identify a temporal generation gap: the internal attention of the model localizes the target moment more accurately than the timestamps it generates. To close this gap, we propose TACG (Temporal Attention-Consistency Grounding), a post-training method that treats internal attention as a reliable temporal pivot and aligns it with both the model prediction and the ground-truth annotation through attention-consistency training. We further build a 580K-sample training dataset by reconstructing temporal QA from dense audio-visual annotations and organizing general omni-modal instruction data. Using TACG and this training dataset, we train OmniGround, a general-purpose omni-modal model with stronger temporal grounding capability, without injecting any additional temporal cues. OmniGround reaches state-of-the-art omni-modal temporal grounding, including 37.5 mIoU on LongVALE and 35.7 mIoU on UnAV-100, while staying competitive on both general audio-visual understanding and vision-only grounding.

Figures & tables

Figure 1: Overview of OmniGround. Left: both the base model and the stronger SFT model exhibit a temporal generation gap : internal attention localizes the target event more accurately than the generated timestamp. TACG closes it to achieve stronger grounding. Right: unlike methods that rely on additional temporal inputs or specialized output formats, OmniGround strengthens temporal grounding while preserving a general-purpose omni-modal interface.

Figure 2: Visualization of the temporal generation gap. Top: the attention averaged over samples and aligned to the ground-truth event (green), overlaid with the averaged prediction. Bottom: the corresponding averaged attention and predicted intervals. In both base and SFT models, the attention localizes the event more accurately than the prediction.

	AVID		LongVALE
Model	Attn. wins †	mIoU	Attn. wins †	mIoU
Qwen2.5-Omni-3B	$73.9\%$	$7.4$	$78.3\%$	$6.3$
Qwen2.5-Omni-3B (SFT)	$57.7\%$	$17.9$	$60.0\%$	$20.9$
Qwen2.5-Omni-7B	$75.7\%$	$6.2$	$72.1\%$	$12.1$
Qwen3-Omni	$65.0\%$	$14.1$	$69.7\%$	$10.6$
OLA	$73.0\%$	$10.3$	$70.3\%$	$7.7$

Table 1: The temporal generation gap across models and datasets. For each benchmark, all statistics are computed on a fixed evaluation subset of 500 examples randomly sampled from its official test split. Attn. wins † reports the fraction of non-tied samples where

\mathrm{IoU}(I_{\text{att}},I_{\text{gt}})>\mathrm{IoU}(I_{\text{pred}},I_{\text{gt}})

. mIoU is the grounding accuracy of the generated interval. This gap is consistent across models and datasets.

Figure 3: Overall architecture and TACG training of OmniGround.

Figure 4: Left: dataset composition. Right: four-stage construction pipeline and representative QA templates for the three grounding formats.

Method	[email protected]	[email protected]	[email protected]	mIoU
VideoLLaMA2	13.3	7.0	2.3	11.5
OLA	13.6	6.6	2.0	13.1
Qwen2.5-Omni (3B)	20.7	11.8	5.5	15.2
Qwen2.5-Omni (7B)	25.9	16.2	6.7	20.1
Qwen3-Omni	26.5	17.5	8.0	21.8
LongVALE	11.3	6.1	3.2	15.0

Table 2: Evaluation on UnAV-100. ⋆ Closed-source, for reference only. † Trained on the same data but with standard SFT (cross-entropy).

Vision-only Models
Method	Omni-modal Grounding								Vision-only Grounding
	AVID-TVG				LongVALE-TVG				Charades-STA				ActivityNet-Captions
	[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	mIoU
VideoChat	7.6	5.2	2.5	5.2	2.2	0.9	0.4	3.0	32.8	8.6	0.0	25.9	23.5	12.6	6.0	17.4
VTimeLLM	9.2	4.1	1.0	7.4	9.1	4.3	1.9	7.9	52.0	27.7	11.1	31.5	44.0	27.8	14.3	30.4
TimeChat	9.8	6.9	3.0	7.8	7.2	3.4	1.6	6.4	47.8	31.6	14.6	31.6	30.2	16.9	8.2	21.8

Table 3: Comparison with state-of-the-art methods on temporal video grounding benchmarks. OmniGround sets a new state of the art on the omni-modal temporal grounding; moreover, despite seeing no vision-only grounding data, it still essentially matches specialized methods on the vision-only Charades-STA and ActivityNet-Captions, reflecting strong generalization. Bold : best; underline : second best. ⋆ Closed-source, for reference only. † Trained on the same data but with standard SFT (cross-entropy).

Figure 5: TACG closes the temporal generation gap. Average ground-truth, attention, and generated intervals for OmniGround. The persistent displacement observed in the preliminary analysis has been largely absorbed.

Method	WorldSense	Daily-Omni
	Acc	Acc
VideoLLaMA2	27.8	35.8
OLA	34.0	54.3
Qwen2.5-Omni (3B)	46.0	62.3
Qwen2.5-Omni (7B)	46.1	62.7
Qwen3-Omni	52.0	68.8

Table 4: Evaluation on general audio-visual understanding benchmarks. ⋆ Closed-source, for reference only. † Trained on the same data but with standard SFT.

Configuration	AVID		LongVALE		$\Delta$ %
Configuration	[email protected]	mIoU	[email protected]	mIoU	$\Delta$ %
Baseline (CE only)	0.3502	0.3153	0.1408	0.1604	0.00
$\mathcal{L}_{\text{tg}}$ only	0.3543	0.3223	0.1563	0.1722	+4.79
$\mathcal{L}_{\text{pg}}$ only	0.3649	0.3282	0.1479	0.1645	+3.32
$\mathcal{L}_{\text{tp}}$ only	0.3671	0.3315	0.1514	0.1663	+4.41
$\mathcal{L}_{\text{tg}}+\mathcal{L}_{\text{pg}}$	0.3671	0.3301	0.1508	0.1672	+4.47

Table 5: Component ablation of the TACG consistency loss.

\Delta

% is the average relative improvement over the SFT baseline across both benchmarks. Bold : best; underline : second best. Full results are provided in the appendix.

Panel of Figure 6 — Figure 6: Training dynamics for timestamp discretization. Argmax spikes mid-training from mode collapse, whereas Round decreases smoothly and monotonically.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: GT-aligned attention at representative layers for the SFT and base models (N=500); the green band marks the ground-truth segment.

Figure 8: GT-aligned temporal attention across all 36 layers ( $N{=}500$ ), for the SFT model (left), the base model (middle), and their difference (right). Each row is a layer; the horizontal axis is the offset from the ground-truth center (GT width normalized to 1; green dashed lines mark the segment). Attention concentrates inside the GT band at nearly every layer and is sharpest around layers 26–28, and the SFT $-$ base difference peaks inside the band at the same layers—fine-tuning sharpens these mid-to-late layers the most.

Variant	Group	Attn. mIoU	Pred. mIoU	$\Delta$ IoU
CE Base	Attn. wins	40.0	5.9	+34.1
	Pred. wins	37.9	68.5	$-$ 30.6
	Ties	0.0	0.0	0.0
	Overall	29.8	31.1	$-$ 1.3
Full TACG	Attn. wins	50.0	8.0	+42.0
	Pred. wins	41.1	74.0	$-$ 32.9

Appendix

Table 6: Attention–prediction alignment of the CE baseline and Full TACG on AVID. All values are reported on the percentage scale. Full TACG improves both localization signals and reduces the aggregate gap between attention and prediction.

Figure 9: Sensitivity of the temporal generation gap to attention extraction choices on Qwen2.5-Omni-3B (

N{=}500

). Left: layer sweep. Right: cumulative-mass cutoff sweep for extracting

I_{\text{att}}

Vision Encoder
Component	Specification
Architecture	Vision Transformer (ViT)
Hidden size	1280
Patch size	14 $\times$ 14
Temporal patch size	2
Spatial merge size	2

Appendix

Table 7: Detailed architecture specifications of OmniGround.

Hyperparameter	Value
Base model	Qwen2.5-Omni-3B / 7B
Optimizer	AdamW
Learning rate	2.5 e- 5
LR scheduler	Cosine
Warmup ratio	0.05
Batch size (per device)	1

Appendix

Table 8: Training hyperparameters and TACG configuration.

Rejected type	Visual-only	Audio-only	Total
Rate	$17\%$	$4\%$	$21\%$

Appendix

Table 9: Text-only audio-visual relevance filtering statistics. Rejection is computed over individual caption/query targets rather than whole videos or QA instances.

Task	Source	Samples
Temporal Grounding Data (from AVID & LongVALE)
Event $\rightarrow$ Time (TVG)	AVID	34,041
Event $\rightarrow$ Time (TVG)	LongVALE	91,863
Time $\rightarrow$ Event (Reverse)	AVID	13,526
Time $\rightarrow$ Event (Reverse)	LongVALE	36,474
Dense Captioning (DVC)	AVID	7,239

Appendix

Table 10: Training data composition of our final model (580,676 samples). Reverse Grounding is produced by inverting a 50,000-sample subset of the forward-TVG pool, so it inherits that pool’s AVID/LongVALE split. The SFT model probed in our preliminary analysis is trained only on the forward-TVG data (125,904 samples), without the Reverse Grounding, DVC, or general-understanding data.

Item	Specification
Hardware
GPUs	32 $\times$ NVIDIA A100-80GB
Parallelism	DeepSpeed ZeRO-2
NVIDIA driver	470.161.03
Software
Python	3.10.15

Appendix

Table 11: Computing infrastructure and software environment.

Figure 10: Best mIoU per discretization against the cross-entropy baseline. Only Round yields a positive gain over the CE baseline; Argmax and Soft fall below it on both benchmarks.

Figure 11: mIoU vs. loss weight

\lambda

for each discretization.

Figure 12: Weight ablation for all three edges (

\mathcal{L}_{\text{tg}}

\mathcal{L}_{\text{pg}}

\mathcal{L}_{\text{tp}}

) on both benchmarks.

Figure 13: Joint mIoU response surface over the

\lambda_{\text{tg}}

–

\lambda_{\text{pg}}

grid (26 evaluated points; dashed contour = SFT baseline) on LongVALE (left) and AVID (right).

Configuration	AVID		LongVALE		$\Delta$ %
Configuration	[email protected]	mIoU	[email protected]	mIoU	$\Delta$ %
Baseline (CE only)	0.3502	0.3153	0.1408	0.1604	0.00
$\mathcal{L}_{\text{tg}}$ only	0.3543	0.3223	0.1563	0.1722	+4.79
$\mathcal{L}_{\text{pg}}$ only	0.3649	0.3282	0.1479	0.1645	+3.32
$\mathcal{L}_{\text{tp}}$ only	0.3671	0.3315	0.1514	0.1663	+4.41
Full w/o $\mathcal{L}_{\text{pg}},\mathcal{L}_{\text{tp}}$	0.3558	0.3229	0.1360	0.1563	$-$ 0.08

Appendix

Table 12: Full component ablation of the TACG consistency loss. Upright rows: addition results with individually optimized weights; italic rows : subtraction results inheriting the full-model weights.

\Delta

%: average relative improvement over the SFT baseline across both benchmarks. Bold / underline : best/second best per column.

Benchmark (mIoU)	OmniGround (3B)
AVID	$56.9_{\pm 0.4}$
LongVALE	$28.6_{\pm 0.5}$
Charades-STA	$38.8_{\pm 0.6}$
ActivityNet-Captions	$23.7_{\pm 0.4}$

Appendix

Table 13: Multi-seed stability of the 3B final model (mean

\pm

std over three seeds).

Configuration	AVID mIoU	LongVALE mIoU
Baseline (CE only)	$0.3147_{\pm 0.0042}$	$0.1598_{\pm 0.0031}$
$\mathcal{L}_{\text{tg}}$ only	$0.3218_{\pm 0.0038}$	$0.1715_{\pm 0.0035}$
$\mathcal{L}_{\text{pg}}$ only	$0.3275_{\pm 0.0045}$	$0.1640_{\pm 0.0029}$
$\mathcal{L}_{\text{tp}}$ only	$0.3309_{\pm 0.0040}$	$0.1658_{\pm 0.0033}$
$\mathcal{L}_{\text{tg}}+\mathcal{L}_{\text{pg}}$	$0.3296_{\pm 0.0036}$	$0.1667_{\pm 0.0030}$
Full TACG	$\mathbf{0.3401_{\pm 0.0034}}$	$\mathbf{0.1753_{\pm 0.0028}}$

Appendix

Table 14: Multi-seed stability of the component ablation (cf. Table 5 ), mean

\pm

std over three seeds.

Temporal Video Grounding

Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.

See all 166 papers

Sep 17, 2026

AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Models

Omni models can describe video content, but can they locate events in time, preserve event order, and judge audio-visual synchronization? We introduce AVTrace (Audio-Visual Temporal Reasoning Assessment and Capability Evaluation), a silver-standard diagnostic suite spanning onset and span grounding, synchronization, next-step prediction, cross-modal localization, chain parsing, and event-conditioned comprehension. It contains 34,114 training examples and category-balanced development and test splits of 3,500 and 7,000 examples. We evaluate five open omni models under their respective input configurations using reference-blind response normalization followed by deterministic scoring. All five off-the-shelf systems score below the test split's majority-label baseline of 0.556 on synchronization verification, and obtain low scores on chain parsing and event-conditioned grounding and comprehension. Development-set perturbations reveal task-dependent sensitivity in Qwen3-Omni-30B to modality removal and changes in visual input processing, without isolating their underlying causes. Parameter-efficient temporal post-training improves Gemma4-E4B-it on several benchmark metrics. On three external image benchmarks, task metrics change modestly, including some degradations, while teacher-forcing perplexity decreases. Together, these findings show that semantic reference-text overlap should not be treated as a proxy for temporal localization, and that AVTrace can identify task-specific weaknesses while providing a testbed for temporal post-training.

Longyin Zhang, Parth Sakhare Mahendra, Chengwei Wei +4

Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · Centre for Frontier AI Research (CFAR), A*STAR, Singapore

Temporal Video Grounding Audio-Visual Understanding Audio-Visual Synchronization Multimodal Model Evaluation

Ask PDF

Oct 13, 2026

OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

With the rapid development of multimodal large language models (MLLMs), evaluating their video understanding capabilities has become increasingly important. Existing video understanding benchmarks cover only part of the evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding. We introduce OmniCompass, a holistic benchmark for omni-modal video grounding and understanding. OmniCompass defines two evaluation tracks: OmniCompass-VL assesses vision-language models on tasks requiring visual evidence, while OmniCompass-AV assesses omni-modal models on tasks requiring joint audio-visual evidence. Each track includes three complementary tasks: temporal grounding (TG), spatial grounding (SG), and question answering (QA), assessing when a referenced target appears, where it is, and what happens. QA covers five perception and three reasoning categories in OmniCompass-VL, and three perception and two reasoning categories in OmniCompass-AV. OmniCompass-VL contains 820 videos and 2,460 instances across domains and three duration types: short, medium, and long. A narrative- and audio-rich subset forms OmniCompass-AV with 240 videos and 720 instances. Each instance is evaluated under two paired settings: text-only and text+image, specifying the same target through text and visual-language guidance. All instances are manually annotated with multi-round quality verification. Evaluations of 50 MLLMs show that open-source models still lag behind proprietary counterparts, particularly in audio-visual understanding. Precise temporal and spatial grounding remains challenging even for leading models, and their predictions still exhibit inconsistencies across the paired target specifications.

Xuyang Liu, Shihao Wang, Xinyu Wei +5

The Hong Kong Polytechnic University · OPPO Research Institute · Colorado School of Mines +2

Video-Language Model Evaluation Multimodal Grounding Audio-Visual Understanding Video QA Temporal Video Grounding

Ask PDF

Sep 28, 2026

SyncRA: Learning Temporal Correspondence in Omni-Modal Models

Recent omni-modal models demonstrate strong perception of audio and visual inputs, yet often struggle to connect what they hear with what they see at the same moment. This weakness in temporal correspondence can cause models to associate spoken cues with the wrong visual scenes, producing plausible answers grounded in incorrect audio-visual pairings. We diagnose this problem through controlled temporal swaps, revealing that model answers do not reliably follow changes in these pairings. To address it, we propose Synchrony-Guided Representation Alignment (SyncRA), a lightweight method for strengthening temporal correspondence between audio and vision. Specifically, SyncRA contrasts intermediate audio-visual representations within each video, aligning matching moments while separating mismatched ones to capture local temporal correspondence within a shared global context. The objective derives supervision directly from existing input timing, requiring no additional annotations and leaving inference unchanged. We evaluate SyncRA across four open omni-modal models spanning different sizes and architectures on five public video benchmarks. SyncRA consistently outperforms answer-only fine-tuning across all model-benchmark combinations, while substantially improving the ability to track changing audio-visual pairings in controlled evaluations. These results demonstrate that lightweight, targeted supervision can effectively strengthen temporal correspondence and translate into broad improvements in audio-visual question answering.

Zelong Xu, Yan Li, Wenhe Hu +1

University of Wisconsin–Madison · University of Alberta · Arizona State University

Audio-Visual Representation Learning Audio-Visual Synchronization Audio-Visual QA Audio-Visual Alignment Multimodal Contrastive Learning

Ask PDF

Abstract

Temporal Video Grounding

Explore similar work

AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Models

OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

SyncRA: Learning Temporal Correspondence in Omni-Modal Models