SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs

Mohamad Alansari; Naufal Suryanto; Divya Velayudhan; Sajid Javed; Naoufel Werghi; Muzammal Naseer

cs.CVMar 12, 2026

SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs

Authors: Mohamad Alansari, Naufal Suryanto, Divya Velayudhan, Sajid Javed, Naoufel Werghi, Muzammal Naseer

Organizations: Khalifa University, Abu Dhabi, UAE

Abstract

Multimodal large language models (MLLMs) have advanced from image-level reasoning to pixel-level grounding, but extending these capabilities to videos remains challenging as models must achieve spatial precision and temporally consistent reference tracking. Existing video MLLMs often rely on a static segmentation token ([SEG]) for frame-wise grounding, which provides semantics but lacks temporal context, causing spatial drift, identity switches, and unstable initialization when objects move or reappear. We introduce SPARROW, a pixel-grounded video MLLM that unifies spatial accuracy and temporal stability through two key components: (i) Target-Specific Tracked Features (TSF), which inject temporally aligned referent cues during training, and (ii) a dual-prompt design that decodes box ([BOX]) and segmentation ([SEG]) tokens to fuse geometric priors with semantic grounding. SPARROW is supported by a curated referential video dataset of 30,646 videos and 45,231 Q&A pairs and operates end-to-end without external detectors via a class-agnostic SAM2-based proposer. Integrated into three recent open-source video MLLMs (UniPixel, GLUS, and VideoGLaMM), SPARROW delivers consistent gains across six benchmarks, improving up to +8.9 J&F on RVOS, +5 mIoU on visual grounding, and +5.4 CLAIR on GCG. These results demonstrate that SPARROW substantially improves referential stability, spatial precision, and temporal coherence in pixel-grounded video understanding. Project page: https://risys-lab.github.io/SPARROW

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of temporal consistency and initialization quality in video object segmentation. (a) The baseline method [ 38 ] suffers from temporal drift, leading to inconsistent segmentation of the same object across frames. (b) Noisy or unstable initialization propagates segmentation errors through subsequent frames. (c) Our proposed Target-Specific Tracked Feature mitigates drift by maintaining consistent object grounding over time. (d) The Dual-Prompt Initialization strategy improves segmentation precision and stability during early frames.

Figure 2: SPARROW pipeline. Given a video and text prompt, spatial [ 41 ] and temporal [ 57 ] encoders feed V $\!\to$ L adapters and a LoRA‑tuned LLM. The LLM emits [BOX] and [SEG] tokens which are projected (L $\!\to$ V) to condition a class‑agnostic proposer and the SAM2 [ 43 ] pixel decoder. Dashed green modules (GroundingDINO [ 31 ] , CLDTracker [ 2 ] , target cropping, K‑means) are pre-computed offline as pseudo-supervision, used only for target-specific information injection step , and are removed at test time by default.

Figure 3: Illustrative process of the dual-prompt initialization. Given a query (e.g., “Can you segment the truck?”), our module first generates class-agnostic proposals, which are filtered by the [BOX] prompt and then refined by the [SEG] prompt to produce precise segmentation masks. The dual-prompt approach provides tighter localization and sharper boundaries compared with using [SEG] only.

Figure 4: Qualitative comparison on the Referring Video Object Segmentation (RVOS) task. (Left) Dog scene: SPARROW accurately segments “dog … to the right,” “dog … to the left,” and “woman in a yellow jacket” from the first frame, preserving their identities across motion and overlap. (Right) Crowd scene: SPARROW cleanly separates “the woman in yellow,” “the woman in red,” and “the woman in black,” maintaining consistent masks and sharp boundaries under occlusion and scale variation. In contrast, UniPixel [ 32 ] and GLUS [ 28 ] exhibit early inaccuracies, mask swaps, and boundary artifacts across both scenes. The dual-prompt initialization enables precise first-frame grounding and referentially consistent segmentation, producing temporally stable masks throughout the sequence.

Method	MeViS ( val )			MeViS ( val ${}^{\text{u}}$ )
Method	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$
UniPixel Neurips’25 [ 32 ]	50.4	55.7	53.1	56.1	63.2	59.7
+ SPARROW	51.6 +1.2	57.2 +1.5	54.4 +1.3	57.2 +1.1	64.2 +1.0	60.7 +1.0
GLUS CVPR’25 [ 28 ]	48.5	54.2	51.3	56.1	63.4	59.8
+ SPARROW	50.4 +1.9	56.0 +1.8	53.2 +1.9	57.0 +0.9	66.8 +3.4	61.9 +0.3
VideoGLaMM CVPR’24 [ 38 ]	42.1	48.2	45.2	44.0	52.9	48.5

Table 1: MeViS [ 12 ] results on val and val

{}^{\text{u}}

, which assess seen and unseen motion-expression compositions, respectively.

Method	Ref-YTVOS			Ref-DAVIS17
Method	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$
UniPixel Neurips’25 [ 32 ]	68.6	72.3	70.5	70.7	77.8	74.2
+ SPARROW	69.0 +0.4	72.4 +0.1	70.7 +0.2	71.5 +0.8	81.2 +3.4	76.4 +2.2
GLUS CVPR’25 [ 28 ]	65.5	69.0	67.3	69.7	76.1	72.9
+ SPARROW	67.3 +1.8	70.9 +1.9	69.1 +1.8	70.6 +0.9	80.3 +4.2	75.5 +2.6
VideoGLaMM CVPR’24 [ 38 ]	65.4	68.2	66.8	73.3	65.6	69.5

Table 2: Ref-YTVOS [ 45 ] and Ref-DAVIS17 [ 19 ] (val) results, where Ref-YTVOS emphasizes large-scale linguistic grounding and Ref-DAVIS17 focuses on fine-grained boundary accuracy.

Model	mIoU	Recall	METEOR	CIDEr	CLAIR
UniPixel Neurips’25 [ 32 ]	52.0	0.311	0.095	0.55	26.0
+ SPARROW	54.5 +2.5	0.325 +0.01	0.103 +0.01	0.58 +0.03	29.4 +3.4
GLUS CVPR’25 [ 28 ]	45.86	0.245	–	–	–
+ SPARROW	47.91 +2.05	0.265 +0.02	–	–	–
VideoGLaMM CVPR’24 [ 38 ]	62.34	0.375	0.103	0.59	28.2
+ SPARROW	65.59 +3.25	0.383 +0.01	0.112 +0.01	0.63 +0.04	33.6 +5.4

Table 3: VideoGCG benchmark [ 38 ] . We report mask quality (mIoU, Recall) and caption quality (METEOR, CIDEr, CLAIR).

Panel of Figure 5 — Figure 5: Visual grounding on VidSTG (interrogative). SPARROW boosts mIoU by +5.49 on UniPixel (41.25 $\rightarrow$ 46.74), +5.25 on GLUS (29.92 $\rightarrow$ 35.17), and +5.40 on VideoGLaMM (39.66 $\rightarrow$ 45.06), consistently improving spatial and mask quality.

Figure 6: Qualitative comparison on the Grounded Conversation Generation (GCG) task. (Left) Kitchen scene: SPARROW produces more detailed and semantically consistent descriptions (e.g., distinguishing cup , bottle , and mop ) compared to VideoGLaMM’s generic output. (Right) Fencing scene: SPARROW generates context-rich phrases differentiating individuals by appearance and action, while VideoGLaMM produces shorter, less informative text. These examples highlight that SPARROW not only achieves stronger temporal coherence and object grounding, but also narrative completeness across complex scenes.

TSF usage	BOX OFF	BOX ON
No TSF	69.5 (baseline)	72.5 ( +3.0 )
Train only (our default)	72.4 ( +2.9 )	76.8 ( +7.3 )
Train + Inference	75.3 ( +5.8 )	77.7 ( +8.2 )

Table 4: TSF pretraining and [BOX] prompting on Ref-DAVIS17 (val). Values are

\mathcal{J\&F}

(A) Filtration Head supervision token
$[{\rm SEG}]$ features $\rightarrow$ Filtration Head	70.6
$[{\rm BOX}]$ features $\rightarrow$ Filtration Head (ours)	72.5 ( +1.9 )
(B) Proposal selection mechanism
MLLM filtration (language-only)	70.4
Filtration Head (ours)	72.5 ( +2.1 )
(C) Inference prompt composition

Table 5: Selection supervision and prompt composition on Ref-DAVIS17 (val). Values are

\mathcal{J\&F}

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Video Clips	Q&A Pairs
HC-STVG [ 48 ]	10105	10105
MeViS [ 12 ]	1644	4489
A2D Sentences [ 15 ]	2508	5359
LaSOT [ 14 ]	2640	7920
VID-Sentence [ 7 ]	4045	7654
GOT-10k [ 17 ]	8195	8195

Appendix

Table 6: Curated datasets used to construct our TSF dataset.

Figure 7: Qualitative comparison on the MeViS RVOS task. Given the motion-centric query “rabbit moving from middle to left,” our method correctly grounds the intended rabbit from the first frame and maintains its identity throughout the entire sequence, despite the presence of multiple appearance-similar distractors. It produces temporally stable masks with clean boundaries and consistent spatial localization even under close interactions between the rabbits. In contrast, VideoGLaMM, UniPixel, and GLUS frequently drift between rabbits, mix identities, or generate unstable masks across frames. The strong referential grounding of our approach enables precise, distraction-robust tracking and segmentation of the queried rabbit across all frames.

Figure 8: Qualitative comparison on the MeViS RVOS task. For the query “the three horses are moving in the water,” our method accurately grounds all three horses from the first frame and preserves their individual identities throughout the entire sequence. It maintains clean instance separation, stable temporal masks, and sharp boundaries even as the horses move closely, interact, and occlude one another. In contrast, VideoGLaMM, UniPixel, and GLUS frequently merge instances, lose one or more horses across frames, or produce inconsistent and drifting masks. The strong multi-instance grounding of our approach enables reliable segmentation and tracking of all three horses as distinct, temporally consistent targets.

Figure 9: Qualitative comparison on the MeViS RVOS task. For the query “the two zebras playfully chasing each other,” our method accurately grounds both zebras from the first frame and preserves their identities throughout the sequence, despite rapid motion, tight interaction, and repeated occlusions. It maintains clear instance separation, stable temporal masks, and precise spatial localization under fast, dynamic behavior. In contrast, VideoGLaMM, UniPixel, and GLUS frequently merge the two zebras, lose one during high-motion frames, or produce unstable and drifting masks. Our approach delivers robust multi-instance grounding and consistent segmentation of both zebras as distinct targets across the entire video.

Figure 10: Qualitative comparison on the Ref-YTVOS RVOS task. For the query “a black bird flying highest to the left,” our method reliably grounds the correct bird from the first frame and maintains accurate tracking throughout the sequence, despite the densely packed flock and rapid aerial motion. It preserves clean spatial localization and stable masks even when multiple birds share nearly identical appearances and flight patterns. In contrast, VideoGLaMM, GLUS, and UniPixel frequently drift to neighboring birds, lose the target under fast movement, or produce inconsistent and flickering masks. Our approach demonstrates strong referential grounding and robust temporal consistency in challenging multi-instance, motion-heavy scenarios.

Figure 11: Qualitative comparison on the Ref-DAVIS17 RVOS task. Given multiple appearance-based prompts targeting distinct goldfish (“largest,” “smallest,” “center,” “end,” “bottom”), our method accurately grounds each described instance from the first frame and maintains clear separation throughout the sequence. It preserves stable masks, fine-grained localization, and consistent identities even when the fish exhibit similar colors, overlapping motion, and close interactions. In contrast, VideoGLaMM frequently merges instances, confuses similarly colored fish, or produces unstable and drifting segmentations. Our approach demonstrates strong referential grounding under simultaneous multi-query conditions, delivering reliable instance-specific masks across the entire sequence.

Figure 12: Qualitative comparison on the Ref-DAVIS17 RVOS task. For the queries “a back shooting gun,” “a black man,” and “a rope,” our method cleanly grounds all three described targets from the first frame and maintains precise, instance-specific masks throughout the sequence. It preserves accurate boundaries and stable localization even on thin or small objects such as the gun barrel and rope. In contrast, GLUS produces unstable, overlapping masks, frequently blending instances or losing fine structural details under motion and occlusion. Our approach demonstrates stronger fine-grained appearance grounding and consistent multi-object separation across all frames.

Figure 13: Qualitative comparison on Ref-DAVIS17 (RVOS). For the expressions “a man in a suit riding a scooter” and “a black scooter ridden by a man,” our method accurately grounds both the rider and scooter from the first frame and maintains stable, instance-consistent masks throughout the sequence, preserving sharp boundaries despite significant motion and viewpoint changes. In contrast, UniPixel produces fragmented masks, inconsistent localization, and frequent structural loss. Our approach demonstrates stronger appearance-based grounding and greater robustness to multi-object consistency across frames.

Figure 14: Qualitative comparison on the VidSTG VG task. For the query “What is the adult in pink wear hold?,” our method consistently grounds the correct object—the orange cup—from the first frame and maintains accurate, stable masks despite hand motion, partial occlusions, and frequent interaction among multiple people. It preserves precise spatial extent and clean boundaries across the entire sequence. In contrast, VideoGLaMM produces drifting or incomplete masks and struggles to maintain consistent localization of the held object during motion. Our approach demonstrates strong spatio-temporal grounding and reliable object tracking on VidSTG.

Figure 15: Qualitative comparison on the VidSTG VG task. For the query “the child in pink,” our method accurately segments the correct child from the first frame and maintains stable masks throughout the sequence, preserving clean boundaries and consistent localization despite rapid motion, occlusions, and nearby interactions. In contrast, GLUS often drifts to other people or yields incomplete, unstable masks under motion, whereas our approach remains robust to distractors and maintains stronger temporal consistency.

Figure 16: Qualitative comparison on the VidSTG VG task. For the query “Who is the man on the far left?,” our method accurately grounds the correct individual from the first frame and preserves consistent identity assignments throughout the sequence, even during dense group interactions and frequent occlusions. It maintains stable masks and precise spatial localization across all frames. In contrast, UniPixel exhibits identity drift and inconsistent mask assignment, especially in cluttered moments with overlapping people. Our approach demonstrates stronger temporal grounding and robust identity preservation in challenging crowd scenarios.

Figure 17: Qualitative comparison on the Video GCG task. In a cooking sequence where a woman places a potato in water, chops additional potatoes, and transfers the pieces into a pot, our method maintains accurate pixel-level grounding of all manipulated objects across the full series of actions. It produces a coherent, temporally aligned description that faithfully reflects each step of the manipulation process. In contrast, VideoGLaMM generates fluent but loosely grounded narratives, GLUS fails to produce a meaningful grounded response, and UniPixel captures only coarse actions while struggling with temporal transitions and hand–object interactions. Our approach demonstrates robust spatio-temporal grounding and precise correspondence between language and visual evidence throughout the sequence.

Vision + Tracking
Component	Per-unit latency	Count	Added time (s)
GroundingDINO (first frame)	90 ms	$1$	$0.090$
CLDTracker	18 ms / frame / target	$nT$	$0.018\,nT$
CLIP ViT-L/14@336 (crops)	2 ms / crop (batched)	$nT$	$0.002\,nT$
Token construction + LLM
K-means clustering ( $K{=}4$ )	$<1$ ms / target	$n$	$\approx 0$

Appendix

Table 7: Component-wise analysis: TSF overhead at test time on A100.

Figure 18: When to enable TSF at inference (qualitative). Query: segment the “red bmx bike” and the “boy” . TSF Off (default) : the small/fragile referent (bike) is prone to target dropout and drift/false positives as the target becomes small or partially occluded. TSF On (optional max-stability): TSF provides a tracked, target-specific identity cue at inference, preserving consistent referent segmentation.

Scenario	Recommendation
Short/simple clips, limited motion	Keep TSF off
Fast motion or motion blur	Enable TSF if accuracy is critical
Occlusion and re-appearance	Enable TSF
Small or fragile targets	Enable TSF if target dropout is likely
Crowded scenes with similar distractors	Enable TSF

Appendix

Table 8: When to enable TSF at inference.

TSF projection design	Extra tok.	J&F $\uparrow$
No TSF	0	72.5
Temporal-only: $W_{h}(F_{g}(B_{j}))$	+4	72.5
Dual-path (concat): $[W_{g}(\cdot);\,W_{h}(\cdot)]$	+8	74.9
Dual-path (sum): $W_{g}(\cdot)+W_{h}(\cdot)$	+4	74.7
Spatial-only (ours) : $W_{g}(F_{g}(B_{j}))$	+4	76.8

Appendix

Table 9: TSF injection path ablation on Ref-DAVIS17 (val). We vary only the TSF projection path. “tok.” denotes the number of additional TSF tokens per target.

Corruption applied to training trajectories	Level	J&F $\uparrow$
None (clean TSF)	–	76.8
Box jitter (random shift/scale)	$\pm 5\%$	76.5
	$\pm 10\%$	76.0
	$\pm 20\%$	75.2
ID-switch injection (contiguous segments)	5% frames	76.1
	10% frames	75.4

Appendix

Table 10: Robustness of SPARROW to noisy TSF supervision on Ref-DAVIS17 (val). We corrupt only the training-time teacher trajectories before TSF token extraction; the model architecture, training schedule, and inference setting remain unchanged. All results use the default Train-only TSF setting with [BOX] enabled.

Method	+Params (M) $\downarrow$	+Latency (ms) $\downarrow$	$\mathcal{J\&F}\uparrow$
Direct LLM coords (single box)	0.00	0.0	67.8
MLP regressor (single box)	0.35	0.4	68.5
Anchor-free conv head	1.90	1.2	70.4
DETR decoder head	12.33	6.2	71.2
Deformable-DETR decoder head (ours)	11.38	5.6	72.5

Appendix

Table 11: Proposal generation variants (frozen SAM2 Hiera-L features). Performance on Ref-DAVIS17 (val) measured by

\mathcal{J\&F}

. Latency is the incremental overhead of the proposal head only (feature extraction shared).

Method	$K$	[email protected] $\uparrow$	[email protected] $\uparrow$
Direct LLM coords (single box)	1	50.0	20.0
MLP regressor (single box)	1	55.0	28.0
Anchor-free conv head	300	86.0	60.0
DETR decoder head	300	90.0	65.0
Deformable-DETR decoder head (ours)	300	93.0	70.0

Appendix

Table 12: Proposal-set oracle recall under frozen SAM2 Hiera-L features. Recall@IoU measures whether any of the top-

K

boxes overlaps a GT box above the threshold.

Component	Wall-clock	GPU-hours	A100 GPU-days
Offline TSF precomputation (30,646 clips)	51.9 h (1 $\times$ A100)	51.9	2.16
Baseline fine-tuning (per backbone)	10 h (8 $\times$ A100)	80	3.33
Stage 1: TSF injection	12 h (8 $\times$ A100)	96	4.00
Stage 2: Dual-prompt box learning	2 h (8 $\times$ A100)	16	0.67
SPARROW training (S1 + S2)	14 h	112	4.67
Increment over baseline	–	+32	+1.34

Appendix

Table 13: Training cost breakdown for SPARROW. Offline TSF generation is one-time preprocessing and is not part of the training loop or default inference pipeline.

Component	Stage	When executed	Runtime cost
GroundingDINO + CLDTracker	TSF generation	Offline (one-time)	$\sim$ 6.1 s / clip
Crop encoding + K-means	TSF generation	Offline (one-time)	included in total TSF cost
TSF Injection (Stage 1)	Training	Per backbone	$\sim$ 4.0 GPU-days
Dual-Prompt Learning (Stage 2)	Training	Per backbone	$\sim$ 0.67 GPU-days
SPARROW inference (default)	Inference	Per video	no external detector/tracker
TSF-enabled inference (optional)	Inference	Per frame	$\sim$ 20 ms / target / frame

Appendix

Table 14: Execution summary across preprocessing, training, and inference.

Backbone	Params	FPS	GFLOPs/f	+SP Params	+SP FPS	+SP GFLOPs/f
VideoGLaMM	5.435	2.40	38927	5.452	2.40	38962
UniPixel	3.881	15.38	1530	3.898	15.04	1565
GLUS	7.288	6.374	2680	7.305	6.29	2715

Appendix

Table 15: Inference overhead of SPARROW under identical evaluation settings. We report total parameters (in Billion), end-to-end FPS, and vision GFLOPs per frame. +SP denotes the backbone augmented with SPARROW modules.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,069 papers

The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding
Jul 27, 2026Jiameng Zhang, Srikanth MadikeriTemporal Video Grounding VLM Adaptation
University of Zurich
Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding
Apr 9, 2026Xuezhen Tu, Jingyu Wu, Fangyu Kang +4Vision-Language Grounding Temporal Video Grounding
Shanghai Jiao Tong University China · ZTE Corporation China
Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification
Jun 27, 2026Tianshu Zhang, Yan Wang, Ji Qi +1Temporal Video Grounding Efficient Multimodal Inference
Tsinghua University

Jul 27, 2026J/K move · Enter open · S save

The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding

Jiameng Zhang, Srikanth Madikeri

University of Zurich

Temporal Video Grounding VLM Adaptation Vision-Language Grounding Video Frame Selection

Open paper AskPDF arXiv

Large-scale video platforms process millions of uploads hourly, requiring moderation systems that can localize when and where policy violations occur within each video. Processing every frame is infeasible at scale, so systems are constrained to sparse inputs of 8 to 16 frames per video. Yet state-of-the-art multimodal large language models (MLLMs) are pretrained on dense sequences of hundreds of frames, creating a fundamental mismatch between training and deployment conditions. This mismatch causes severe performance collapse: the Qwen3-VL 8B model drops from 56.0% to 22.3% temporal mIoU when frames are reduced to 16, a 60.2% relative degradation. We present a systematic empirical study of training strategies to close this gap for spatial-temporal video grounding. Our results suggest that visual feature extraction is the dominant bottleneck under sparse-frame inputs. Adapting only the final three ViT layers, 4% of total parameters, achieves 68.8% temporal mIoU and surpasses a zero-shot 8B model using dense inputs by 12.8 points. Language model fine-tuning, by contrast, offers negligible or negative returns. A boundary-aware sampling strategy, Hybrid16, further improves temporal mIoU by 26 points over uniform sampling when temporal boundaries are available. We conclude that for sparse-frame video grounding, training strategy dominates model scale: a fine-tuned 2B model consistently outperforms a zero-shot 8B model, with or without dense frame access.

Abstract

Multimodal Large Language Models

Explore similar work

The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding

Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding

Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification