OmniGround: Strengthening Omni-Modal Temporal Grounding via Attention-Consistency Training

Zixuan Chen; Depeng Wang; Hao Lin; Ke Xu; Ya Guo; Huijia Zhu; Tanfeng Sun; Xinghao Jiang

cs.CVOct 19, 2026

OmniGround: Strengthening Omni-Modal Temporal Grounding via Attention-Consistency Training

Authors: Zixuan Chen, Depeng Wang, Hao Lin, Ke Xu, Ya Guo, Huijia Zhu, Tanfeng Sun, Xinghao Jiang

Organizations: Shanghai Jiao Tong University · Ant Group · Independent Researcher

Abstract

Omni-modal large language models can jointly understand video, audio, and text, yet they remain weak at temporal grounding, a capability essential for real-world audio-visual applications. To understand this limitation, we look beyond final predictions and examine the model's internal temporal evidence. We identify a temporal generation gap: the internal attention of the model localizes the target moment more accurately than the timestamps it generates. To close this gap, we propose TACG (Temporal Attention-Consistency Grounding), a post-training method that treats internal attention as a reliable temporal pivot and aligns it with both the model prediction and the ground-truth annotation through attention-consistency training. We further build a 580K-sample training dataset by reconstructing temporal QA from dense audio-visual annotations and organizing general omni-modal instruction data. Using TACG and this training dataset, we train OmniGround, a general-purpose omni-modal model with stronger temporal grounding capability, without injecting any additional temporal cues. OmniGround reaches state-of-the-art omni-modal temporal grounding, including 37.5 mIoU on LongVALE and 35.7 mIoU on UnAV-100, while staying competitive on both general audio-visual understanding and vision-only grounding.

Figures & tables

Figure 1: Overview of OmniGround. Left: both the base model and the stronger SFT model exhibit a temporal generation gap : internal attention localizes the target event more accurately than the generated timestamp. TACG closes it to achieve stronger grounding. Right: unlike methods that rely on additional temporal inputs or specialized output formats, OmniGround strengthens temporal grounding while preserving a general-purpose omni-modal interface.

Figure 2: Visualization of the temporal generation gap. Top: the attention averaged over samples and aligned to the ground-truth event (green), overlaid with the averaged prediction. Bottom: the corresponding averaged attention and predicted intervals. In both base and SFT models, the attention localizes the event more accurately than the prediction.

	AVID		LongVALE
Model	Attn. wins †	mIoU	Attn. wins †	mIoU
Qwen2.5-Omni-3B	$73.9\%$	$7.4$	$78.3\%$	$6.3$
Qwen2.5-Omni-3B (SFT)	$57.7\%$	$17.9$	$60.0\%$	$20.9$
Qwen2.5-Omni-7B	$75.7\%$	$6.2$	$72.1\%$	$12.1$
Qwen3-Omni	$65.0\%$	$14.1$	$69.7\%$	$10.6$
OLA	$73.0\%$	$10.3$	$70.3\%$	$7.7$

Table 1: The temporal generation gap across models and datasets. For each benchmark, all statistics are computed on a fixed evaluation subset of 500 examples randomly sampled from its official test split. Attn. wins † reports the fraction of non-tied samples where

\mathrm{IoU}(I_{\text{att}},I_{\text{gt}})>\mathrm{IoU}(I_{\text{pred}},I_{\text{gt}})

. mIoU is the grounding accuracy of the generated interval. This gap is consistent across models and datasets.

Figure 3: Overall architecture and TACG training of OmniGround.

Figure 4: Left: dataset composition. Right: four-stage construction pipeline and representative QA templates for the three grounding formats.

Method	[email protected]	[email protected]	[email protected]	mIoU
VideoLLaMA2	13.3	7.0	2.3	11.5
OLA	13.6	6.6	2.0	13.1
Qwen2.5-Omni (3B)	20.7	11.8	5.5	15.2
Qwen2.5-Omni (7B)	25.9	16.2	6.7	20.1
Qwen3-Omni	26.5	17.5	8.0	21.8
LongVALE	11.3	6.1	3.2	15.0

Table 2: Evaluation on UnAV-100. ⋆ Closed-source, for reference only. † Trained on the same data but with standard SFT (cross-entropy).

Vision-only Models
Method	Omni-modal Grounding								Vision-only Grounding
	AVID-TVG				LongVALE-TVG				Charades-STA				ActivityNet-Captions
	[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	mIoU
VideoChat	7.6	5.2	2.5	5.2	2.2	0.9	0.4	3.0	32.8	8.6	0.0	25.9	23.5	12.6	6.0	17.4
VTimeLLM	9.2	4.1	1.0	7.4	9.1	4.3	1.9	7.9	52.0	27.7	11.1	31.5	44.0	27.8	14.3	30.4
TimeChat	9.8	6.9	3.0	7.8	7.2	3.4	1.6	6.4	47.8	31.6	14.6	31.6	30.2	16.9	8.2	21.8

Table 3: Comparison with state-of-the-art methods on temporal video grounding benchmarks. OmniGround sets a new state of the art on the omni-modal temporal grounding; moreover, despite seeing no vision-only grounding data, it still essentially matches specialized methods on the vision-only Charades-STA and ActivityNet-Captions, reflecting strong generalization. Bold : best; underline : second best. ⋆ Closed-source, for reference only. † Trained on the same data but with standard SFT (cross-entropy).

Figure 5: TACG closes the temporal generation gap. Average ground-truth, attention, and generated intervals for OmniGround. The persistent displacement observed in the preliminary analysis has been largely absorbed.

Method	WorldSense	Daily-Omni
	Acc	Acc
VideoLLaMA2	27.8	35.8
OLA	34.0	54.3
Qwen2.5-Omni (3B)	46.0	62.3
Qwen2.5-Omni (7B)	46.1	62.7
Qwen3-Omni	52.0	68.8

Table 4: Evaluation on general audio-visual understanding benchmarks. ⋆ Closed-source, for reference only. † Trained on the same data but with standard SFT.

Configuration	AVID		LongVALE		$\Delta$ %
Configuration	[email protected]	mIoU	[email protected]	mIoU	$\Delta$ %
Baseline (CE only)	0.3502	0.3153	0.1408	0.1604	0.00
$\mathcal{L}_{\text{tg}}$ only	0.3543	0.3223	0.1563	0.1722	+4.79
$\mathcal{L}_{\text{pg}}$ only	0.3649	0.3282	0.1479	0.1645	+3.32
$\mathcal{L}_{\text{tp}}$ only	0.3671	0.3315	0.1514	0.1663	+4.41
$\mathcal{L}_{\text{tg}}+\mathcal{L}_{\text{pg}}$	0.3671	0.3301	0.1508	0.1672	+4.47

Table 5: Component ablation of the TACG consistency loss.

\Delta

% is the average relative improvement over the SFT baseline across both benchmarks. Bold : best; underline : second best. Full results are provided in the appendix.

Panel of Figure 6 — Figure 6: Training dynamics for timestamp discretization. Argmax spikes mid-training from mode collapse, whereas Round decreases smoothly and monotonically.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: GT-aligned attention at representative layers for the SFT and base models (N=500); the green band marks the ground-truth segment.

Figure 8: GT-aligned temporal attention across all 36 layers ( $N{=}500$ ), for the SFT model (left), the base model (middle), and their difference (right). Each row is a layer; the horizontal axis is the offset from the ground-truth center (GT width normalized to 1; green dashed lines mark the segment). Attention concentrates inside the GT band at nearly every layer and is sharpest around layers 26–28, and the SFT $-$ base difference peaks inside the band at the same layers—fine-tuning sharpens these mid-to-late layers the most.

Variant	Group	Attn. mIoU	Pred. mIoU	$\Delta$ IoU
CE Base	Attn. wins	40.0	5.9	+34.1
	Pred. wins	37.9	68.5	$-$ 30.6
	Ties	0.0	0.0	0.0
	Overall	29.8	31.1	$-$ 1.3
Full TACG	Attn. wins	50.0	8.0	+42.0
	Pred. wins	41.1	74.0	$-$ 32.9

Appendix

Table 6: Attention–prediction alignment of the CE baseline and Full TACG on AVID. All values are reported on the percentage scale. Full TACG improves both localization signals and reduces the aggregate gap between attention and prediction.

Figure 9: Sensitivity of the temporal generation gap to attention extraction choices on Qwen2.5-Omni-3B (

N{=}500

). Left: layer sweep. Right: cumulative-mass cutoff sweep for extracting

I_{\text{att}}

Vision Encoder
Component	Specification
Architecture	Vision Transformer (ViT)
Hidden size	1280
Patch size	14 $\times$ 14
Temporal patch size	2
Spatial merge size	2

Appendix

Table 7: Detailed architecture specifications of OmniGround.

Hyperparameter	Value
Base model	Qwen2.5-Omni-3B / 7B
Optimizer	AdamW
Learning rate	2.5 e- 5
LR scheduler	Cosine
Warmup ratio	0.05
Batch size (per device)	1

Appendix

Table 8: Training hyperparameters and TACG configuration.

Rejected type	Visual-only	Audio-only	Total
Rate	$17\%$	$4\%$	$21\%$

Appendix

Table 9: Text-only audio-visual relevance filtering statistics. Rejection is computed over individual caption/query targets rather than whole videos or QA instances.

Task	Source	Samples
Temporal Grounding Data (from AVID & LongVALE)
Event $\rightarrow$ Time (TVG)	AVID	34,041
Event $\rightarrow$ Time (TVG)	LongVALE	91,863
Time $\rightarrow$ Event (Reverse)	AVID	13,526
Time $\rightarrow$ Event (Reverse)	LongVALE	36,474
Dense Captioning (DVC)	AVID	7,239

Appendix

Table 10: Training data composition of our final model (580,676 samples). Reverse Grounding is produced by inverting a 50,000-sample subset of the forward-TVG pool, so it inherits that pool’s AVID/LongVALE split. The SFT model probed in our preliminary analysis is trained only on the forward-TVG data (125,904 samples), without the Reverse Grounding, DVC, or general-understanding data.

Item	Specification
Hardware
GPUs	32 $\times$ NVIDIA A100-80GB
Parallelism	DeepSpeed ZeRO-2
NVIDIA driver	470.161.03
Software
Python	3.10.15

Appendix

Table 11: Computing infrastructure and software environment.

Figure 10: Best mIoU per discretization against the cross-entropy baseline. Only Round yields a positive gain over the CE baseline; Argmax and Soft fall below it on both benchmarks.

Figure 11: mIoU vs. loss weight

\lambda

for each discretization.

Figure 12: Weight ablation for all three edges (

\mathcal{L}_{\text{tg}}

\mathcal{L}_{\text{pg}}

\mathcal{L}_{\text{tp}}

) on both benchmarks.

Figure 13: Joint mIoU response surface over the

\lambda_{\text{tg}}

–

\lambda_{\text{pg}}

grid (26 evaluated points; dashed contour = SFT baseline) on LongVALE (left) and AVID (right).

Configuration	AVID		LongVALE		$\Delta$ %
Configuration	[email protected]	mIoU	[email protected]	mIoU	$\Delta$ %
Baseline (CE only)	0.3502	0.3153	0.1408	0.1604	0.00
$\mathcal{L}_{\text{tg}}$ only	0.3543	0.3223	0.1563	0.1722	+4.79
$\mathcal{L}_{\text{pg}}$ only	0.3649	0.3282	0.1479	0.1645	+3.32
$\mathcal{L}_{\text{tp}}$ only	0.3671	0.3315	0.1514	0.1663	+4.41
Full w/o $\mathcal{L}_{\text{pg}},\mathcal{L}_{\text{tp}}$	0.3558	0.3229	0.1360	0.1563	$-$ 0.08

Appendix

Table 12: Full component ablation of the TACG consistency loss. Upright rows: addition results with individually optimized weights; italic rows : subtraction results inheriting the full-model weights.

\Delta

%: average relative improvement over the SFT baseline across both benchmarks. Bold / underline : best/second best per column.

Benchmark (mIoU)	OmniGround (3B)
AVID	$56.9_{\pm 0.4}$
LongVALE	$28.6_{\pm 0.5}$
Charades-STA	$38.8_{\pm 0.6}$
ActivityNet-Captions	$23.7_{\pm 0.4}$

Appendix

Table 13: Multi-seed stability of the 3B final model (mean

\pm

std over three seeds).

Configuration	AVID mIoU	LongVALE mIoU
Baseline (CE only)	$0.3147_{\pm 0.0042}$	$0.1598_{\pm 0.0031}$
$\mathcal{L}_{\text{tg}}$ only	$0.3218_{\pm 0.0038}$	$0.1715_{\pm 0.0035}$
$\mathcal{L}_{\text{pg}}$ only	$0.3275_{\pm 0.0045}$	$0.1640_{\pm 0.0029}$
$\mathcal{L}_{\text{tp}}$ only	$0.3309_{\pm 0.0040}$	$0.1658_{\pm 0.0033}$
$\mathcal{L}_{\text{tg}}+\mathcal{L}_{\text{pg}}$	$0.3296_{\pm 0.0036}$	$0.1667_{\pm 0.0030}$
Full TACG	$\mathbf{0.3401_{\pm 0.0034}}$	$\mathbf{0.1753_{\pm 0.0028}}$

Appendix

Table 14: Multi-seed stability of the component ablation (cf. Table 5 ), mean

\pm

std over three seeds.

Temporal Video Grounding

Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.

See all 166 papers

AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Models
Sep 17, 2026Longyin Zhang, Parth Sakhare Mahendra, Chengwei Wei +4Temporal Video Grounding Audio-Visual Understanding
Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · Centre for Frontier AI Research (CFAR), A*STAR, Singapore
OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding
Oct 13, 2026Xuyang Liu, Shihao Wang, Xinyu Wei +5Video-Language Model Evaluation Multimodal Grounding
The Hong Kong Polytechnic University · OPPO Research Institute · Colorado School of Mines +2
SyncRA: Learning Temporal Correspondence in Omni-Modal Models
Sep 28, 2026Zelong Xu, Yan Li, Wenhe Hu +1Audio-Visual Representation Learning Audio-Visual Synchronization
University of Wisconsin–Madison · University of Alberta · Arizona State University

Sep 17, 2026J/K move · Enter open · S save

AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Models

Longyin Zhang, Parth Sakhare Mahendra, Chengwei Wei, Ning Zhang+3

Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · Centre for Frontier AI Research (CFAR), A*STAR, Singapore

Temporal Video Grounding Audio-Visual Understanding Audio-Visual Synchronization Multimodal Model Evaluation

Open paper AskPDF arXiv

Omni models can describe video content, but can they locate events in time, preserve event order, and judge audio-visual synchronization? We introduce AVTrace (Audio-Visual Temporal Reasoning Assessment and Capability Evaluation), a silver-standard diagnostic suite spanning onset and span grounding, synchronization, next-step prediction, cross-modal localization, chain parsing, and event-conditioned comprehension. It contains 34,114 training examples and category-balanced development and test splits of 3,500 and 7,000 examples. We evaluate five open omni models under their respective input configurations using reference-blind response normalization followed by deterministic scoring. All five off-the-shelf systems score below the test split's majority-label baseline of 0.556 on synchronization verification, and obtain low scores on chain parsing and event-conditioned grounding and comprehension. Development-set perturbations reveal task-dependent sensitivity in Qwen3-Omni-30B to modality removal and changes in visual input processing, without isolating their underlying causes. Parameter-efficient temporal post-training improves Gemma4-E4B-it on several benchmark metrics. On three external image benchmarks, task metrics change modestly, including some degradations, while teacher-forcing perplexity decreases. Together, these findings show that semantic reference-text overlap should not be treated as a proxy for temporal localization, and that AVTrace can identify task-specific weaknesses while providing a testbed for temporal post-training.

Abstract

Temporal Video Grounding

Explore similar work

AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Models

OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

SyncRA: Learning Temporal Correspondence in Omni-Modal Models