InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding

Ashutosh Kumar; Rajat Saini; Jingjing Pan; Mustafa Erdogan; Mingfang Zhang; Betty Le Dem; Norimasa Kobori; Quan Kong

cs.CVApr 9, 2026

InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding

Authors: Ashutosh Kumar, Rajat Saini, Jingjing Pan, Mustafa Erdogan, Mingfang Zhang, Betty Le Dem, Norimasa Kobori, Quan Kong

Organizations: Woven by Toyota

Abstract

Current vision-language pre-training (VLP) paradigms excel at global scene understanding but struggle with instance-level reasoning due to global-only supervision. We introduce InstAP, an Instance-Aware Pre-training framework that jointly optimizes global vision-text alignment and fine-grained, instance-level contrastive alignment by grounding textual mentions to specific spatial-temporal regions. To support this, we present InstVL, a large-scale dataset (2 million images, 50,000 videos) with dual-granularity annotations: holistic scene captions and dense, grounded instance descriptions. On the InstVL benchmark, InstAP substantially outperforms existing VLP models on instance-level retrieval, and also surpasses a strong VLP baseline trained on the exact same data corpus, isolating the benefit of our instance-aware objective. Moreover, instance-centric pre-training improves global understanding: InstAP achieves competitive zero-shot performance on multiple video benchmarks, including MSR-VTT and DiDeMo. Qualitative visualizations further show that InstAP localizes textual mentions to the correct instances, while global-only models exhibit more diffuse, scene-level attention.

Figures & tables

Figure 2: Illustration of our InstVL dataset. We display sampled frames with color-coded, temporally-consistent instance trajectories ( e.g. , ID: 0, ID: 1). The top text provides the fine-grained instance captions grounded to these trajectories; the bottom text provides the holistic global caption for the entire scene.

Figure 3: Our instance-aware alignment mechanism. Instance features (Query $Q$ ) from a Trajectory RoI Encoder ( $f_{\theta}$ ) are fused with global context (Key $K$ , Value $V$ ) via an Attention Pool to create an instance-aware embedding. This embedding is contrasted with text features ( $e_{\phi}$ ). The loss forces the model to match positive pairs ( $V_{1T1}$ ) while contrasting against negatives from different videos ( $V_{2T1}$ / $V_{2T2}$ ) and masking potential false-negative pairs from the same video ( $V_{1T2}$ ), enforcing fine-grained discrimination (Eq. 8 ).

Method	Split	InstVL(img)				InstVL(img-zero)				InstVL(video)
		1K		10K		1K		10K		1K
		T2V R@1	V2T R@1	T2V R@1	V2T R@1	T2V R@1	V2T R@1	T2V R@1	V2T R@1	T2V R@1	V2T R@1
VideoPrism [ 70 ]	Instance	28.21	34.52	22.75	29.51	21.32	27.39	13.85	20.04	40.86	39.29
VideoPrism [ 70 ]	Global	97.40	97.60	88.19	89.62	85.70	85.80	73.05	75.11	82.71	83.62
CLIP4Clip [ 38 ]	Instance	25.10	33.21	18.68	28.19	17.82	25.10	9.11	16.30	17.71	24.69
CLIP4Clip [ 38 ]	Global	93.40	96.00	79.22	84.25	78.20	81.70	56.95	63.96	67.50	70.50

Table 1: Comparison of SOTA models and our InstAP on the InstVL test set. We report T2V/V2T R@1 on the instance and global splits across InstVL( img ), InstVL( img-zero ), and InstVL( video ). UMT-L (InstVL; g/g+i) baselines use the same full training corpus as InstAP, trained with only InstVL’s global captions (g) or with all InstVL captions treated as global (g+i).

Method	MSR-VTT	DiDeMo	MSVD	LSMDC	ActivityNet
CLIP4Clip [ 38 ]	32.0 / 57.0 / 66.9	–	38.5 / 66.9 / 76.8	15.1 / 28.5 / 36.4	–
Frozen in Time [ 3 ]	18.7 / 39.5 / 51.6	21.1 / 46.0 / 56.2	38.7 / 70.1 / 80.1	9.3 / 22.0 / 30.1	–
VIOLET [ 16 ]	25.9 / 49.5 / 59.7	23.5 / 49.8 / 59.8	–	–	–
ALPRO [ 26 ]	24.1 / 44.7 / 55.4	23.8 / 47.3 / 57.9	–	–	–
RAP [ 56 ]	28.9 / 47.5 / 56.8	29.5 / 55.7 / 65.6	35.9 / 64.3 / 73.7	12.8 / 26.6 / 33.4	–
Clover [ 21 ]	26.4 / 49.5 / 60.0	29.5 / 55.2 / 66.3	–	14.7 / 29.2 / 38.2	–

Table 2: Zero-shot text-to-video retrieval (R@1 / R@5 / R@10) on standard benchmarks. UMT-L (InstVL; g) and UMT-L (InstVL; g+i) are baselines trained on the full corpus as InstAP.

Method	InstVL(img)			InstVL(img-zero)			InstVL(video)
Method	IoU@50	IoU@70	IoU@90	IoU@50	IoU@70	IoU@90	IoU@50	IoU@70	IoU@90
UMT-L	74.53	63.47	41.64	67.12	54.20	34.05	54.25	40.70	14.44
InstAP (Ours)	76.17	67.04	48.20	68.52	58.91	42.14	60.02	48.85	25.13

Table 3: Grounding metrics (IoU@{50, 70, 90}) on InstVL-1K .

Alignment	DiDeMo	MSR-VTT	LSMDC	InstVL-1K (img-zero)		InstVL-1K (video)
Alignment	DiDeMo	MSR-VTT	LSMDC	Instance	Global	Instance	Global
$\mathcal{L}_{\mathrm{rec}}+\mathcal{L}_{\mathrm{global}}$	65.98	54.65	34.47	49.98	88.82	57.71	91.55
$\mathcal{L}_{\mathrm{rec}}+\mathcal{L}_{\mathrm{global}}+\mathcal{L}_{\mathrm{inst}}$	70.01	56.72	35.75	63.94	89.78	75.32	97.03

Table 4: Effect of adding the instance-aware loss

\mathcal{L}_{\mathrm{inst}}

to the base objectives

\mathcal{L}_{\mathrm{rec}}+\mathcal{L}_{\mathrm{global}}

. We report mean recall (average of R@

1

, R@

5

, R@

10

over T2V and V2T) on standard and InstVL benchmarks.

Method	InstVL-1K (img)	InstVL-1K (img-zero)	InstVL-1K (video)
Baseline	59.10	46.37	45.48
+ Instance temperature	67.19	54.90	55.22
+ Weighted instance loss	68.17	56.00	58.16
+ Caption sub-sampling	71.65	58.42	58.97
+ Instance trajectory	75.03	63.94	75.32

Table 5: Ablation of InstAP components on the InstVL instance-level test sets. We report mean recall, averaged over R@

1

, R@

5

, and R@

10

for both V2T and T2V retrieval.

Figure 4: InstAP tends to attend more closely to caption-relevant regions (e.g., ‘dubai plate 61062’) than the global-only baseline [ 29 ] , which often exhibits diffuse or misaligned attention.

Figure 5: InstAP consistently retrieves correct fine-grained descriptions, whereas the global baseline [ 29 ] is confounded by semantic distractors and mismatches the query.

Vision-Language Pretraining

Vision-language pretraining learns visual representations by relating images, video, or other visual inputs to language.

See all 80 papers

VideoSeeker: Incentivizing Instance-level Video Understanding via Native Agentic Tool Invocation
May 15, 2026Yiming Zhao, Yu Zeng, Wenxuan Huang +11Temporal Video Grounding Tool-Using Vision-Language Agents
University of Science and Technology of China · Xiaohongshu Inc. · East China Normal University +1
Watch Before You Answer: Learning from Visually Grounded Post-Training
Apr 6, 2026Yuxuan Zhang, EunJeong Hwang, Huaisong Zhang +8VLM Post-Training Vision-Language Models
University of British Columbia · Vector Institute · Etude AI +4
Learning to Track Instance from Single Nature Language Description
May 8, 2026Yaozong Zheng, Bineng Zhong, Qihua Liang +3Visual Object Tracking Vision-Language Alignment
Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education Guangxi Normal University, Guilin 541004, China · University Engineering Research Center of Educational Intelligent Technology Guangxi Normal University, Guilin 541004, China · University of Southampton, Southampton, SO17 1BJ, United Kingdom

May 15, 2026J/K move · Enter open · S save

VideoSeeker: Incentivizing Instance-level Video Understanding via Native Agentic Tool Invocation

Yiming Zhao, Yu Zeng, Wenxuan Huang, Zhen Fang+10

University of Science and Technology of China · Xiaohongshu Inc. · East China Normal University+1

Temporal Video Grounding Tool-Using Vision-Language Agents

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have shown significant progress in video understanding, yet they face substantial challenges in tasks requiring precise spatiotemporal localization at the instance level. Existing methods primarily rely on text prompts for human-model interaction, but these prompts struggle to provide precise spatial and temporal references, resulting in poor user experience. Furthermore, current approaches typically decouple visual perception from language reasoning, centering reasoning around language rather than visual content, which limits the model's ability to proactively perceive fine-grained visual evidence. To address these challenges, we propose VideoSeeker, a novel paradigm for instance-level video understanding through visual prompts. VideoSeeker seamlessly integrates agentic reasoning with instance-level video understanding tasks, enabling the model to proactively perceive and retrieve relevant video segments on demand. We construct a four-stage fully automated data synthesis pipeline to efficiently generate large-scale, high-quality instance-level video data. We internalize tool-calling and proactive perception capabilities into the model via cold-start supervision and RL training, building a powerful video understanding model. Experiments demonstrate that our model achieves an average improvement of +13.7% over baselines on instance-level video understanding tasks, surpassing powerful closed-source models such as GPT-4o and Gemini-2.5-Pro, while also showing effective transferability on general video understanding benchmarks. The relevant datasets and code will be released publicly.

Abstract

Vision-Language Pretraining

Explore similar work

VideoSeeker: Incentivizing Instance-level Video Understanding via Native Agentic Tool Invocation

Watch Before You Answer: Learning from Visually Grounded Post-Training

Learning to Track Instance from Single Nature Language Description