PhysMLLMs: Spatial Priors for Unified Referring Segmentation and Grounded Reasoning of Images and Videos

Siyao Yan; Bo Han; Jisheng Dang; Bimei Wang; Shude Wang; Hong Peng; Yulan Guo; Jianhuang Lai

cs.AIAug 25, 2026

PhysMLLMs: Spatial Priors for Unified Referring Segmentation and Grounded Reasoning of Images and Videos

Authors: Siyao Yan, Bo Han, Jisheng Dang, Bimei Wang, Shude Wang, Hong Peng, Yulan Guo, Jianhuang Lai,

Organizations: School of Information Science and Engineering, Lanzhou University, Lanzhou, 730000, China · School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 510006, China · School of Computing, National University of Singapore, Singapore, 119077, Singapore

Abstract

Video multimodal large language models support language guided video segmentation, but they often show spatio temporal inconsistencies, e.g., jitter, drift, and identity switches. These failures are more common when targets are partly hidden or when similar objects appear nearby.One likely reason is that current training lacks explicit spatial priors, which makes it difficult to maintain stable spatial identity and shape over time. We present PhysMLLMs, a training-stage prior injection architecture that injects physics-inspired spatial continuity priors into Video MLLMs. PhysMLLMs is designed to encourage more stable object-centered representations by aligning the student global visual representation with a frozen teacher model during training. Our core mechanism, Global Representation Prior Alignment (REPA-Global), distills global visual representations from a frozen DINOv2 teacher using an offline embedding cache and a scheduled distillation plan. This design keeps inference unchanged and does not add inference time cost. Across multiple video benchmarks, PhysMLLMs improves video segmentation mask quality and cross-frame consistency, with larger gains on challenging cases involving small targets, fast motion, occlusion, distractors, and reasoning queries. On single-frame referring image segmentation and representative general VLM benchmarks, PhysMLLMs maintains comparable performance, demonstrating that the injected spatial prior improves video consistency without compromising image-level grounding or general multimodal capability. These results suggest that physics-inspired spatial prior injection can improve temporal stability while preserving general capability. The code is available at https://github.com/tusu-code/20260121-icml2026-2.git.

Figures & tables

Figure 1: Motivation and overview of spatial prior injection for video segmentation. Unstable representations can lead to temporally inconsistent masks (left). We inject a continuity prior by distilling global visual representations from a frozen DINOv2 teacher using an offline teacher cache and scheduled distillation weight (middle). The resulting model yields more consistent segmentations with zero inference-time overhead, since the teacher is removed at test time (right).

Figure 2: PhysMLLMs overview. During training, REPA-Global injects a physics-inspired spatial continuity prior by aligning the student global visual embedding $\mathbf{z}_{\mathrm{s}}$ with a frozen DINOv2 teacher embedding $\mathbf{z}_{\mathrm{t}}$ retrieved from an offline cache. Text embeddings condition the LLM and downstream mask prediction, but they do not enter the global visual pooling branch before teacher alignment. At inference, the teacher branch and cache are removed.

Method	MeVIS U	ReVOS	Ref-DAVIS17
PG-Video-LLaVA [ 34 ]	18.9	–	–
GLaMM + SAM2 [ 35 ]	38.7	–	–
VideoGLaMM-3.8B [ 35 ]	45.2	–	–
VISA-13B [ 36 ]	44.5	50.9	70.4
VideoLISA-3B [ 37 ]	44.4	–	68.8
HyperSeg-3B [ 38 ]	–	55.7	71.2

Table 1: Comparison with recent Video-MLLM systems on language-guided video segmentation. We report DAVIS-style J&F on MeVIS U, ReVOS, and Ref-DAVIS17 because most published Video-MLLM papers report aggregate J&F but do not consistently provide separate

\mathcal{J}

and

\mathcal{F}

values across all datasets. Higher is better. “–” means not reported or not available under a comparable protocol.

Setting	ReVOS $\mathcal{J}$	ReVOS $\mathcal{F}$	ReVOS $\mathcal{J\&F}$
Global-only dw012	54.31	60.51	57.41
DINOv2 token-only	53.32	59.21	56.26
DINOv2 global+token	53.01	59.12	56.07

Table 2: Ablation of global and token-level DINOv2 alignment on ReVOS under the same Sa2VA-InternVL3-2B + SAM2 backbone. Global-only REPA-Global outperforms token-only and global+token variants, suggesting that dense patch-level constraints may interfere with language-conditioned target selection.

Setting	KD	ReVOS J&F	MeVIS U J&F
baseline	w/o	56.06	53.92
dw008	w/	56.88	54.51
dw008	w/o	56.46	54.42
dw012	w/	57.41	55.12
dw012	w/o	56.76	54.45
dw020	w/	56.87	55.24

Table 3: Ablation of teacher-prior injection under the same Sa2VA-InternVL3-2B + SAM2 backbone. The symbols dw008 , dw012 , and dw020 denote peak distillation weights

\lambda_{\max}=0.008

0.012

, and

0.020

under the same warmup-ramp-hold schedule. Higher J&F is better.

Stratum	$n$	Official J&F	Ours dw012	$\Delta$
All	5822	54.61	55.91	+1.30
Small target	1459	33.65	35.91	+2.26
Fast motion	1457	49.57	50.94	+1.37
Occlusion or disappearance	1459	31.29	33.22	+1.92
Distractor-heavy	1460	55.95	57.97	+2.02
Reasoning query	2475	51.74	53.50	+1.76

Table 4: Complexity-stratified analysis on ReVOS. Here,

n

denotes the number of ReVOS validation expressions in each stratum, rather than videos or frames. We report the official Sa2VA-InternVL3-2B result, the PhysMLLMs dw012 result, and the absolute J&F improvement. “Hard union” includes expressions satisfying at least one challenging condition.

Figure 3: Qualitative visual video QA examples. Each panel shows the input video, the question, and the model response, illustrating that the injected priors do not noticeably degrade multimodal reasoning on representative cases.

Run ID	RefCOCO	ReVOS	MeVIS U	MMBench
baseline	81.42%	56.06	53.92	0.7876
dw012	81.61%	57.41	55.12	0.7878
dw008	81.73%	56.88	54.51	0.7878
dw020	81.68%	56.87	55.24	0.7878

Table 5: Unified evaluation across image grounding, video segmentation, and MMBench under the InternVL3-2B + SAM2 backbone. We report RefCOCO accuracy, ReVOS and MeVIS U J&F, and MMBench score. Higher is better.

Checkpoint / training stage	MME perception	MME reasoning	MME total	POPE F1	TextVQA val acc
Sa2VA-InternVL3-2B official	1630.838	520.357	2151.195	87.421	76.808
RefCOCO-ft init	1626.126	536.071	2162.198	87.405	76.580
ReVOS calib it200	1610.827	522.500	2133.327	87.575	76.876
ReVOS calib it500	1602.206	527.857	2130.064	87.542	76.796
ReVOS calib it500 + KD dw012	1622.166	530.714	2152.880	–	–

Table 6: General VLM capability check on MME, POPE, and TextVQA. All rows use the same Sa2VA-InternVL3-2B backbone and compare checkpoints from different calibration stages rather than different model backbones. Higher is better.

Model	RefCOCO	RefCOCO+	RefCOCOg
LAVT [ 42 ]	72.7	62.1	61.2
GLaMM-7B [ 43 ]	79.5	72.6	74.2
OMG-LLaVA-7B [ 44 ]	78.0	69.1	72.9
F-LMM-7B [ 45 ]	76.1	65.2	68.5
Sa2VA-InternVL3-2B [ 33 ]	81.4	75.7	80.3
PhysMLLMs	81.9	76.3	80.4

Table 7: Referring image segmentation comparison with representative fine-tuned models. We report validation accuracy on RefCOCO, RefCOCO+, and RefCOCOg. Higher is better.

Setting	Peak weight $\lambda_{\max}$	ReVOS J&F $\uparrow$	tIoU mean $\uparrow$	tIoU var $\downarrow$
dw0002	0.002	56.64	0.6461	0.0841
dw004	0.004	56.53	0.6452	0.0843
dw006	0.006	56.59	0.6396	0.0854
dw008	0.008	56.88	0.6476	0.0839
dw010	0.010	56.66	0.6514	0.0824
dw012	0.012	57.41	0.6419	0.0854

Table 8: Sensitivity to the peak distillation weight under a fixed warmup-ramp-hold schedule. The column “Peak weight

\lambda_{\max}

” denotes the maximum scheduled KD weight. Higher tIoU mean and lower tIoU variance indicate stronger temporal stability.

Figure 4: Qualitative comparison on LV-VIS for language-guided video segmentation. The top three rows show the baseline results and the bottom three rows show PhysMLLMs results. Red boxes highlight baseline failures, and blue boxes highlight improved predictions by PhysMLLMs. The segmentation prompts are: (a) “Please segment a person holding a mop.” (b) “Which creatures are primates? Please answer using segmentation masks.” (c) “Please segment the object(s) carrying ammunition.” (d) “Which gray mat(s) have patterns? Please respond with a segmentation mask.” (e) “Please segment the object fanning its wings on the koala’s snout.” (f) “Please segment the aircraft that is most likely to be in a state of lacking fuel.” Overall, PhysMLLMs improves mask quality and cross-frame consistency under occlusion, distractors, and reasoning-style referring expressions.

Figure 5: Representative failure cases of PhysMLLMs. Each row shows the input image, ground truth, baseline prediction, and our prediction. The three rows correspond to dense similar distractors, persistent occlusion, and small target cases. Red regions indicate false positives, and purple regions indicate false negatives. These examples show that global prior alignment improves temporal stability but may still be insufficient for query-specific identity disambiguation under highly ambiguous visual conditions.

Referring Video Object Segmentation

Referring video object segmentation grounds language descriptions in video frames and produces temporally consistent masks for the described objects.

See all 42 papers

SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs
Mar 12, 2026Mohamad Alansari, Naufal Suryanto, Divya Velayudhan +3Multimodal Large Language Models Vision-Language Grounding
Khalifa University, Abu Dhabi, UAE
Learning Trajectory-Aware Multimodal Large Language Models for Video Reasoning Segmentation
Mar 23, 2026Jingnan Luo, Mingqi Gao, Jun Liu +2Video Object Segmentation Referring Video Object Segmentation
Southern University of Science and Technology, Shenzhen, China. · Tencent YouTu Lab, Shenzhen, China.
SteerSeg: Attention Steering for Reasoning Video Segmentation
May 14, 2026Ali Cheraghian, Hamidreza Dastmalchi, Abdelwahed Khamis +3Video Object Segmentation Referring Video Object Segmentation
1Macquarie University · 2York University · 3CSIRO Data61 +1

Mar 12, 2026J/K move · Enter open · S save

SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs

Mohamad Alansari, Naufal Suryanto, Divya Velayudhan, Sajid Javed+2

Khalifa University, Abu Dhabi, UAE

Multimodal Large Language Models Vision-Language Grounding Referring Video Object Segmentation Temporal Video Grounding

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have advanced from image-level reasoning to pixel-level grounding, but extending these capabilities to videos remains challenging as models must achieve spatial precision and temporally consistent reference tracking. Existing video MLLMs often rely on a static segmentation token ([SEG]) for frame-wise grounding, which provides semantics but lacks temporal context, causing spatial drift, identity switches, and unstable initialization when objects move or reappear. We introduce SPARROW, a pixel-grounded video MLLM that unifies spatial accuracy and temporal stability through two key components: (i) Target-Specific Tracked Features (TSF), which inject temporally aligned referent cues during training, and (ii) a dual-prompt design that decodes box ([BOX]) and segmentation ([SEG]) tokens to fuse geometric priors with semantic grounding. SPARROW is supported by a curated referential video dataset of 30,646 videos and 45,231 Q&A pairs and operates end-to-end without external detectors via a class-agnostic SAM2-based proposer. Integrated into three recent open-source video MLLMs (UniPixel, GLUS, and VideoGLaMM), SPARROW delivers consistent gains across six benchmarks, improving up to +8.9 J&F on RVOS, +5 mIoU on visual grounding, and +5.4 CLAIR on GCG. These results demonstrate that SPARROW substantially improves referential stability, spatial precision, and temporal coherence in pixel-grounded video understanding. Project page: https://risys-lab.github.io/SPARROW

Abstract

Referring Video Object Segmentation

Explore similar work

SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs

Learning Trajectory-Aware Multimodal Large Language Models for Video Reasoning Segmentation

SteerSeg: Attention Steering for Reasoning Video Segmentation