PhysMLLMs: Spatial Priors for Unified Referring Segmentation and Grounded Reasoning of Images and Videos
Authors: Siyao Yan, Bo Han, Jisheng Dang, Bimei Wang, Shude Wang, Hong Peng, Yulan Guo, Jianhuang Lai, Bin Hu, Tat-SengChua
Organizations: School of Information Science and Engineering, Lanzhou University, Lanzhou, 730000, China · School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 510006, China · School of Computing, National University of Singapore, Singapore, 119077, Singapore
Video multimodal large language models support language guided video segmentation, but they often show spatio temporal inconsistencies, e.g., jitter, drift, and identity switches. These failures are more common when targets are partly hidden or when similar objects appear nearby.One likely reason is that current training lacks explicit spatial priors, which makes it difficult to maintain stable spatial identity and shape over time. We present PhysMLLMs, a training-stage prior injection architecture that injects physics-inspired spatial continuity priors into Video MLLMs. PhysMLLMs is designed to encourage more stable object-centered representations by aligning the student global visual representation with a frozen teacher model during training. Our core mechanism, Global Representation Prior Alignment (REPA-Global), distills global visual representations from a frozen DINOv2 teacher using an offline embedding cache and a scheduled distillation plan. This design keeps inference unchanged and does not add inference time cost. Across multiple video benchmarks, PhysMLLMs improves video segmentation mask quality and cross-frame consistency, with larger gains on challenging cases involving small targets, fast motion, occlusion, distractors, and reasoning queries. On single-frame referring image segmentation and representative general VLM benchmarks, PhysMLLMs maintains comparable performance, demonstrating that the injected spatial prior improves video consistency without compromising image-level grounding or general multimodal capability. These results suggest that physics-inspired spatial prior injection can improve temporal stability while preserving general capability. The code is available at https://github.com/tusu-code/20260121-icml2026-2.git.
Figures & tables
Figure 1: Motivation and overview of spatial prior injection for video segmentation. Unstable representations can lead to temporally inconsistent masks (left). We inject a continuity prior by distilling global visual representations from a frozen DINOv2 teacher using an offline teacher cache and scheduled distillation weight (middle). The resulting model yields more consistent segmentations with zero inference-time overhead, since the teacher is removed at test time (right).
Figure 2: PhysMLLMs overview. During training, REPA-Global injects a physics-inspired spatial continuity prior by aligning the student global visual embedding zs with a frozen DINOv2 teacher embedding zt retrieved from an offline cache. Text embeddings condition the LLM and downstream mask prediction, but they do not enter the global visual pooling branch before teacher alignment. At inference, the teacher branch and cache are removed.
Method
MeVIS U
ReVOS
Ref-DAVIS17
PG-Video-LLaVA [ 34 ]
18.9
–
–
GLaMM + SAM2 [ 35 ]
38.7
–
–
VideoGLaMM-3.8B [ 35 ]
45.2
–
–
VISA-13B [ 36 ]
44.5
50.9
70.4
VideoLISA-3B [ 37 ]
44.4
–
68.8
HyperSeg-3B [ 38 ]
–
55.7
71.2
Table 1: Comparison with recent Video-MLLM systems on language-guided video segmentation. We report DAVIS-style J&F on MeVIS U, ReVOS, and Ref-DAVIS17 because most published Video-MLLM papers report aggregate J&F but do not consistently provide separate J and F values across all datasets. Higher is better. “–” means not reported or not available under a comparable protocol.
Setting
ReVOS J
ReVOS F
ReVOS J&F
Global-only dw012
54.31
60.51
57.41
DINOv2 token-only
53.32
59.21
56.26
DINOv2 global+token
53.01
59.12
56.07
Table 2: Ablation of global and token-level DINOv2 alignment on ReVOS under the same Sa2VA-InternVL3-2B + SAM2 backbone. Global-only REPA-Global outperforms token-only and global+token variants, suggesting that dense patch-level constraints may interfere with language-conditioned target selection.
Setting
KD
ReVOS J&F
MeVIS U J&F
baseline
w/o
56.06
53.92
dw008
w/
56.88
54.51
dw008
w/o
56.46
54.42
dw012
w/
57.41
55.12
dw012
w/o
56.76
54.45
dw020
w/
56.87
55.24
Table 3: Ablation of teacher-prior injection under the same Sa2VA-InternVL3-2B + SAM2 backbone. The symbols dw008 , dw012 , and dw020 denote peak distillation weights λmax=0.008 , 0.012 , and 0.020 under the same warmup-ramp-hold schedule. Higher J&F is better.
Stratum
n
Official J&F
Ours dw012
Δ
All
5822
54.61
55.91
+1.30
Small target
1459
33.65
35.91
+2.26
Fast motion
1457
49.57
50.94
+1.37
Occlusion or disappearance
1459
31.29
33.22
+1.92
Distractor-heavy
1460
55.95
57.97
+2.02
Reasoning query
2475
51.74
53.50
+1.76
Table 4: Complexity-stratified analysis on ReVOS. Here, n denotes the number of ReVOS validation expressions in each stratum, rather than videos or frames. We report the official Sa2VA-InternVL3-2B result, the PhysMLLMs dw012 result, and the absolute J&F improvement. “Hard union” includes expressions satisfying at least one challenging condition.
Figure 3: Qualitative visual video QA examples. Each panel shows the input video, the question, and the model response, illustrating that the injected priors do not noticeably degrade multimodal reasoning on representative cases.
Run ID
RefCOCO
ReVOS
MeVIS U
MMBench
baseline
81.42%
56.06
53.92
0.7876
dw012
81.61%
57.41
55.12
0.7878
dw008
81.73%
56.88
54.51
0.7878
dw020
81.68%
56.87
55.24
0.7878
Table 5: Unified evaluation across image grounding, video segmentation, and MMBench under the InternVL3-2B + SAM2 backbone. We report RefCOCO accuracy, ReVOS and MeVIS U J&F, and MMBench score. Higher is better.
Checkpoint / training stage
MME perception
MME reasoning
MME total
POPE F1
TextVQA val acc
Sa2VA-InternVL3-2B official
1630.838
520.357
2151.195
87.421
76.808
RefCOCO-ft init
1626.126
536.071
2162.198
87.405
76.580
ReVOS calib it200
1610.827
522.500
2133.327
87.575
76.876
ReVOS calib it500
1602.206
527.857
2130.064
87.542
76.796
ReVOS calib it500 + KD dw012
1622.166
530.714
2152.880
–
–
Table 6: General VLM capability check on MME, POPE, and TextVQA. All rows use the same Sa2VA-InternVL3-2B backbone and compare checkpoints from different calibration stages rather than different model backbones. Higher is better.
Model
RefCOCO
RefCOCO+
RefCOCOg
LAVT [ 42 ]
72.7
62.1
61.2
GLaMM-7B [ 43 ]
79.5
72.6
74.2
OMG-LLaVA-7B [ 44 ]
78.0
69.1
72.9
F-LMM-7B [ 45 ]
76.1
65.2
68.5
Sa2VA-InternVL3-2B [ 33 ]
81.4
75.7
80.3
PhysMLLMs
81.9
76.3
80.4
Table 7: Referring image segmentation comparison with representative fine-tuned models. We report validation accuracy on RefCOCO, RefCOCO+, and RefCOCOg. Higher is better.
Setting
Peak weight λmax
ReVOS J&F ↑
tIoU mean ↑
tIoU var ↓
dw0002
0.002
56.64
0.6461
0.0841
dw004
0.004
56.53
0.6452
0.0843
dw006
0.006
56.59
0.6396
0.0854
dw008
0.008
56.88
0.6476
0.0839
dw010
0.010
56.66
0.6514
0.0824
dw012
0.012
57.41
0.6419
0.0854
Table 8: Sensitivity to the peak distillation weight under a fixed warmup-ramp-hold schedule. The column “Peak weight λmax ” denotes the maximum scheduled KD weight. Higher tIoU mean and lower tIoU variance indicate stronger temporal stability.
Figure 4: Qualitative comparison on LV-VIS for language-guided video segmentation. The top three rows show the baseline results and the bottom three rows show PhysMLLMs results. Red boxes highlight baseline failures, and blue boxes highlight improved predictions by PhysMLLMs. The segmentation prompts are: (a) “Please segment a person holding a mop.” (b) “Which creatures are primates? Please answer using segmentation masks.” (c) “Please segment the object(s) carrying ammunition.” (d) “Which gray mat(s) have patterns? Please respond with a segmentation mask.” (e) “Please segment the object fanning its wings on the koala’s snout.” (f) “Please segment the aircraft that is most likely to be in a state of lacking fuel.” Overall, PhysMLLMs improves mask quality and cross-frame consistency under occlusion, distractors, and reasoning-style referring expressions.
Figure 5: Representative failure cases of PhysMLLMs. Each row shows the input image, ground truth, baseline prediction, and our prediction. The three rows correspond to dense similar distractors, persistent occlusion, and small target cases. Red regions indicate false positives, and purple regions indicate false negatives. These examples show that global prior alignment improves temporal stability but may still be insufficient for query-specific identity disambiguation under highly ambiguous visual conditions.
Multimodal large language models (MLLMs) have advanced from image-level reasoning to pixel-level grounding, but extending these capabilities to videos remains challenging as models must achieve spatial precision and temporally consistent reference tracking. Existing video MLLMs often rely on a static segmentation token ([SEG]) for frame-wise grounding, which provides semantics but lacks temporal context, causing spatial drift, identity switches, and unstable initialization when objects move or reappear. We introduce SPARROW, a pixel-grounded video MLLM that unifies spatial accuracy and temporal stability through two key components: (i) Target-Specific Tracked Features (TSF), which inject temporally aligned referent cues during training, and (ii) a dual-prompt design that decodes box ([BOX]) and segmentation ([SEG]) tokens to fuse geometric priors with semantic grounding. SPARROW is supported by a curated referential video dataset of 30,646 videos and 45,231 Q&A pairs and operates end-to-end without external detectors via a class-agnostic SAM2-based proposer. Integrated into three recent open-source video MLLMs (UniPixel, GLUS, and VideoGLaMM), SPARROW delivers consistent gains across six benchmarks, improving up to +8.9 J&F on RVOS, +5 mIoU on visual grounding, and +5.4 CLAIR on GCG. These results demonstrate that SPARROW substantially improves referential stability, spatial precision, and temporal coherence in pixel-grounded video understanding. Project page: https://risys-lab.github.io/SPARROW
The prosperity of Multimodal Large Language Models (MLLMs) has stimulated the demand for video reasoning segmentation, which aims to segment video objects based on human instructions. Previous studies rely on unidirectional and implicit text-trajectory alignment, which struggles with trajectory perception when faced with severe video dynamics. In this work, we propose TrajSeg, a simple and unified framework built upon MLLMs. Concretely, we introduce bidirectional text-trajectory alignment, where MLLMs accept grounding-intended (text-to-trajectory) and captioning-intended (trajectory-to-text) instructions. This way, MLLMs can benefit from enhanced correspondence and better perceive object trajectories in videos. The mask generation from trajectories is achieved via a frame-level content integration (FCI) module and a unified mask decoder. The former adapts the MLLM-parsed trajectory-level token to frame-specific information. The latter unifies segmentation for all frames into a single structure, enabling the proposed framework to be simplified and end-to-end trainable. Extensive experiments on referring and reasoning video segmentation datasets demonstrate the effectiveness of TrajSeg, which outperforms all video reasoning segmentation methods on all metrics. The code will be publicly available at https://github.com/haodi19/TrajSeg.
Jingnan Luo, Mingqi Gao, Jun Liu +2
Southern University of Science and Technology, Shenzhen, China. · Tencent YouTu Lab, Shenzhen, China.
Video reasoning segmentation requires localizing objects across video frames from natural language expressions, often involving spatial reasoning and implicit references. Recent approaches leverage frozen large vision-language models (LVLMs) by extracting attention maps and using them as spatial priors for segmentation, enabling training-free grounding. However, these attention maps are optimized for text generation rather than spatial localization, often resulting in diffuse and ambiguous grounding signals. In this work, we introduce SteerSeg, a lightweight framework that identifies attention misalignment as the key bottleneck in attention-based grounding and proposes to steer attention at its source through input-level conditioning. SteerSeg combines learnable soft prompts with reasoning-guided Chain-of-Thought (CoT) prompting. The soft prompts reshape the attention distribution to produce more spatially concentrated maps, while CoT-derived attributes resolve ambiguity among similar objects by guiding attention toward the correct instance. The resulting attention maps are converted into point prompts across keyframes to guide a segmentation model, while candidate tracklets are ranked and selected using correlation-based scoring. Our approach freezes the LVLM and segmentation model parameters and learns only a small set of soft prompts, preserving the model's pretrained reasoning capabilities while significantly improving grounding. Despite being trained only on Ref-YouTube-VOS, SteerSeg generalizes well across diverse benchmarks, significantly improving the spatial grounding capability of LVLMs. Project page: https://steerseg.github.io
Ali Cheraghian, Hamidreza Dastmalchi, Abdelwahed Khamis +3
1Macquarie University · 2York University · 3CSIRO Data61 +1