Invert4TVG: A Temporal Video Grounding Framework with Inversion Tasks Preserving Action Understanding Ability

Zhaoyu Chen; Hongnan Lin; Yongwei Nie; Fei Ma; Xuemiao Xu; Fei Yu; Chengjiang Long

cs.AIAug 10, 2025

Invert4TVG: A Temporal Video Grounding Framework with Inversion Tasks Preserving Action Understanding Ability

Authors: Zhaoyu Chen, Hongnan Lin, Yongwei Nie, Fei Ma, Xuemiao Xu, Fei Yu, Chengjiang Long

Organizations: Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, China · School of Computer Science and Engineering, South China University of Technology, Guangzhou, China · ByteDance Inc., San Jose, CA 95110, USA

Abstract

Temporal Video Grounding (TVG) aims to localize video segments corresponding to a given textual query, which often describes human actions. However, we observe that current methods, usually optimizing for high temporal Intersection-over-Union (IoU), frequently struggle to accurately recognize or understand the underlying actions in both the video and query, thus reducing the effectiveness of these methods. To address this, we propose a novel TVG framework that integrates inversion-based TVG as auxiliary objectives to maintain the model's action understanding ability. We introduce three kinds of inversion TVG tasks derived from the original TVG annotations: (1) Verb Completion, predicting masked verbs (actions) in queries given video segments; (2) Action Recognition, identifying query-described actions; and (3) Video Description, generating descriptions containing query-relevant actions given video segments. These inversion tasks are entirely derived from the original TVG tasks and are probabilistically integrated with them within a reinforcement learning framework. By leveraging carefully designed reward functions, the model preserves its ability to understand actions, thereby improving the accuracy of temporal grounding. Experiments show our method outperforms state-of-the-art approaches, achieving a 7.1% improvement in [email protected] on Charades-STA for a 3B model.

Figures & tables

Figure 1: Left side: A specific example of temporal video grounding. According to the model’s reasoning process,it can be seen that our method achieves better understanding of actions in the video compared to VideoChat-R1 and Time-R1. Right side: statistical results demonstrating that Time-R1, which is optimized solely for the IoU loss, reduces action understanding accuracy (where VC, AR, and VD are the proposed three auxiliary inversion TVG tasks measuring multi-granularity action understanding ability). By introducing Inversion-TVG tasks, our method preserves action understanding ability and thus boosts TVG ability (as shown in R1@0.3, R1@0.5, and R1@0.7). Baseline is Qwen-2.5-VL-3B. — Figure 1: Left side: A specific example of temporal video grounding. According to the model’s reasoning process,it can be seen that our method achieves better understanding of actions in the video compared to VideoChat-R1 and Time-R1. Right side: statistical results demonstrating that Time-R1, which is optimized solely for the IoU loss, reduces action understanding accuracy (where VC, AR, and VD are the proposed three auxiliary inversion TVG tasks measuring multi-granularity action understanding ability). By introducing Inversion-TVG tasks, our method preserves action understanding ability and thus boosts TVG ability (as shown in [email protected], [email protected], and [email protected]). Baseline is Qwen-2.5-VL-3B.

Figure 2: We propose three Invert-TVG tasks. By partially reversing the inputs and outputs of the TVG task we obtain Verb Completion, Action Recognition and Video Description, which reuse the original TVG dataset by taking ground truth video segments as input to reconstruct the target query related actions. The prompts for the three invert-TVG tasks are not identical. For VC, the verb in the query is removed, and the model is required to complete and fill in this verb. AR asks the model to directly estimate the verb in the video. VD requires the model to describe the video content containing action verbs in the query.

Figure 3: Overview of the proposed Invert4TVG framework. The LVLM dynamically chooses between TVG tasks and Invert-TVG tasks according to different probabilities. Whenever an Invert-TVG task is selected, one of the three variants VC, AR or VD is chosen with equal probability.

Type	Method	Size	FT	Charades-STA
Type	Method	Size	FT	[email protected]	[email protected]	[email protected]
VLP	2D-TAN*	-	✓	57.3	45.8	27.9
	Moment-DETR*	-	✓	65.8	52.1	30.6
	EaTR*	-	✓	-	68.4	44.9
	SnAG*	-	✓	-	64.6	46.2
SFT	VideoChat-Flash	7B		74.5	53.1	27.6

Table 1: Performance of temporal video grounding on Charades-STA. Methods labeled FT with a ✓were fine-tuned on the Charades-STA training set. Methods marked with * were first pre-trained on extra TVG datasets 1 and then fine-tuned on the Charades-STA training set, while those without * are only trained on Charades-STA. We compare our method against existing 3B, 7B open-source LVLM. We highlight our results and the best-performing baselines using bold and underlining for clear comparison.

Figure 4: Performance of temporal video grounding on ActivityNet and QvHighlight. We compare our method with Time-R1 (the best-performing among previous methods). All the models are zero-shot tested.

Type	Method	Charades-STA
Type	Method	[email protected]	[email protected]	[email protected]
RL	Only-TVG	78.7	64.1	36.9
	Only-VD	79.1	64.3	39.4
	Only-AR	78.2	65.2	43.8
	Only-VC	78.8	68.0	42.0
	AR+VD	79.6	67.9	43.6

Table 2: Ablation study using only TVG (Time-R1), VC (verb Completion), AR (action recognition), VD (video description), and their mixed usage.

Figure 5: The R1 accuracy curves. Blue, orange, and green show how the three R1 metrics evolve as the Invert-TVG task probability

(1-p)

gradually increases.

Reward	[email protected]	[email protected]	[email protected]
Cosine Similarity	76.2	62.2	39.8
Binary 0 or 1	80.8	69.0	44.0

Table 3: The results using binary Invert-TVG reward or cosine similarity reward for training.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: After Invert-SFT, the mIoU of Qwen2.5-VL-3B model and our Invert4TVG method are compared with those without Invert SFT.

Temporal Video Grounding

Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.

See all 169 papers

IAE-VTG: Interaction-Aligned Action-Entity Video Temporal Grounding
Sep 9, 2026Shiwen Zhao, Qi Zhang, Sezer Karaoglu +2Temporal Video Grounding Vision-Language Alignment
School of Computer Science, The University of Sydney, Sydney, NSW 2006, Australia · Informatics Institute at University of Amsterdam, 1098 XH Amsterdam, the Netherlands
DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding
Sep 8, 2026Zhuo Cao, Bingqing Zhang, Sen Wang +1Temporal Video Grounding Test-Time Adaptation of VLMs
The University of Queensland, Australia
EvoGround: Self-Evolving Video Agents for Video Temporal Grounding
May 13, 2026Minjoon Jung, Byoung-Tak Zhang, Lorenzo TorresaniTemporal Video Grounding Self-Supervised Learning
Seoul National University · Northeastern University

Sep 9, 2026J/K move · Enter open · S save

IAE-VTG: Interaction-Aligned Action-Entity Video Temporal Grounding

Shiwen Zhao, Qi Zhang, Sezer Karaoglu, Theo Gevers+1

School of Computer Science, The University of Sydney, Sydney, NSW 2006, Australia · Informatics Institute at University of Amsterdam, 1098 XH Amsterdam, the Netherlands

Temporal Video Grounding Vision-Language Alignment

Open paper AskPDF arXiv

Video Temporal Grounding (VTG) localizes the video segment that matches a natural-language query. Many queries describe an action performed by a particular entity. Existing methods often encode the query as a whole or use general video-text interactions, without explicitly checking whether the action and entity occur together. They may therefore select a segment that contains both concepts but not the event described by the query. We propose Interaction Aligned Action-Entity Video Temporal Grounding (IAE-VTG), which models this rela?tionship at both the representation and training assignment levels. First, the Fine-grained Disentangled Interaction Module (FDIM) separates action and entity related query information and aligns it with complementary motion and appearance features. It then combines token-level interactions to build representations that capture the relationship between the action and entity. Second, Interaction-Sensitive Assignment (ISA) adds this interaction evidence to bipartite matching, so training targets are selected using both temporal overlap and semantic compatibility. This reduces supervision from temporally plausible but semantically incorrect proposals. Experiments on QVHighlights, Charades?STA, and TACoS show that IAE-VTG consistently improves strong baselines and achieves competitive or state-of-the-art performance on standard grounding metrics. Additional analyses show that the method is especially effective when similar actions or entities appear at multiple times and produces more reliable assignments for complex events.

Abstract

Temporal Video Grounding

Explore similar work

IAE-VTG: Interaction-Aligned Action-Entity Video Temporal Grounding

DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding

EvoGround: Self-Evolving Video Agents for Video Temporal Grounding