ATLAS: Learning to Adapt LVLM Attention for Reasoning Video Segmentation

Hamidreza Dastmalchi; Ali Cheraghian; Aijun An; Abdelwahed Khamis; Lars Petersson

cs.CVOct 17, 2026

ATLAS: Learning to Adapt LVLM Attention for Reasoning Video Segmentation

Authors: Hamidreza Dastmalchi, Ali Cheraghian, Aijun An, Abdelwahed Khamis, Lars Petersson

Organizations: York University · Macquarie University · CSIRO Data61

Abstract

Reasoning Video Object Segmentation (RVOS) aims to localize and segment objects referred to by natural-language expressions throughout a video. Although large vision-language models (LVLMs) implicitly encode localization cues in their attention maps, these attentions are often noisy, diffuse, and scattered across layers, heads, and modalities, limiting their direct use for segmentation. Existing methods typically rely on handcrafted attention-selection heuristics or computationally expensive attention refinement. We propose ATLAS, an attention adaptation framework that learns to transform raw LVLM attentions into object-centric localization cues. ATLAS treats attentions from all layers, heads, and modalities as structured features, combining high-resolution frame-level attentions that capture spatial information with low-resolution video-level attentions that encode temporal context. A lightweight U-Net equipped with temporal attention and FiLM-based modulation fuses these complementary representations to generate coarse segmentation heatmaps, which are propagated across the video using SAM2. Extensive experiments on challenging referring and reasoning video segmentation benchmarks show that ATLAS outperforms existing frozen-backbone methods. Code and models are available at https://hamidreza-136.github.io/atlas/.

Figures & tables

Figure 1: Motivation of ATLAS. Directly querying an LVLM for object coordinates produces inconsistent localization, while naively averaging attention maps across layers and heads yields diffuse and noisy activations. ATLAS instead learns to adapt the complete stack of attention maps into object-centric localization cues using a lightweight adaptation network.

Figure 2: Overview of ATLAS. A frozen LVLM extracts visual attentions from sampled keyframes, which are transformed into object-centric localization maps by a lightweight attention adaptation network. The adapted localization maps guide a frozen SAM2 model to generate and propagate candidate segmentation tracklets, from which the most reliable tracklet is selected as the final segmentation result.

Approaches with tuned foundation models
Method	LVLM	Ref-DAVIS			ReasonVOS			ReVOS (Overall)			ReVOS (Referring)			ReVOS (Reasoning)
Method	LVLM	$\boldsymbol{\mathcal{J\&F}}$	$\boldsymbol{\mathcal{J}}$	$\boldsymbol{\mathcal{F}}$	$\boldsymbol{\mathcal{J\&F}}$	$\boldsymbol{\mathcal{J}}$	$\boldsymbol{\mathcal{F}}$	$\boldsymbol{\mathcal{J\&F}}$	$\boldsymbol{\mathcal{J}}$	$\boldsymbol{\mathcal{F}}$	$\boldsymbol{\mathcal{J\&F}}$	$\boldsymbol{\mathcal{J}}$	$\boldsymbol{\mathcal{F}}$	$\boldsymbol{\mathcal{J\&F}}$	$\boldsymbol{\mathcal{J}}$	$\boldsymbol{\mathcal{F}}$
LISA [ 17 ]	LLaVA-7B	64.8	62.2	67.3	31.1	29.1	33.1	40.9	39.1	42.7	45.7	44.3	47.1	36.1	33.8	38.4
VISA [ 37 ]	ChatUniVi-7B	69.4	66.3	72.5	–	–	–	46.9	44.9	49.0	50.9	49.2	52.6	43.0	40.6	45.4
GLUS [ 21 ]	LLaVA-7B	–	–	–	49.9	47.5	52.4	54.9	52.4	57.3	58.3	56.0	60.7	51.4	48.8	53.9
VideoLISA [ 4 ]	LLaVA-Phi-3-V	68.8	64.9	72.7	47.5	45.1	49.9	–	–	–	–	–	–	–	–	–

Table 1: Comparison of LVLM-based VOS methods on Ref-DAVIS [ 15 ] , ReasonVOS [ 4 ] , and ReVOS [ 37 ] . † Loc-Head and DecAF are training-free; ATLAS trains a lightweight adapter with mask supervision. All three keep the LVLM and segmentation model frozen.

Figure 4: Attention adaptation for reasoning video segmentation. From top to bottom: sampled keyframes, averaged LVLM attention maps, U-Net-adapted coarse attention maps, and ground-truth masks.

	Ref-DAVIS			ReasonVOS
Method	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$
Frame only	75.0	80.1	77.5	57.8	63.0	60.4
Frame + Video	76.3	82.0	79.1	62.2	67.7	64.9
Frame + FiLM	75.5	81.3	78.4	58.8	64.1	61.5
Frame + Video + FiLM	76.8	83.4	80.1	64.7	70.2	67.4

Table 4: Temporal and video ablation with Qwen2.5-VL-7B.

	Ref-DAVIS			ReasonVOS
Criterion	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
Random	37.1	45.8	41.4	27.4	34.2	30.8
SAM2 score	51.5	59.8	55.6	37.8	44.3	41.1
Mean prob.	52.8	59.2	56.0	48.0	53.9	50.9
Captured mass	70.2	77.0	73.6	48.4	54.8	51.6
Hard IoU	75.0	81.2	78.1	63.4	68.8	66.1

Table 5: Tracklet selection criterion using Qwen2.5-VL-7B. All variants share identical localization maps and SAM2 propagation.

# Keyframes	4	8	12	16	20	24
$\mathcal{J}$	59.4	61.6	63.0	64.7	63.5	64.2
$\mathcal{F}$	64.5	66.8	68.9	70.2	70.1	70.4
$\mathcal{J}\&\mathcal{F}$	62.0	64.2	65.9	67.4	66.8	67.3

Table 6: Effect of the number of keyframes on ReasonVOS.

	Ref-DAVIS			ReasonVOS
Adapter	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
Linear	53.6	61.5	57.5	45.3	51.3	48.3
MLP	70.1	76.2	73.2	59.4	64.5	61.9
Conv3	71.2	77.3	74.3	58.6	63.9	61.3
U-Net	76.3	82.0	79.1	62.2	67.7	64.9
U-Net + FiLM	76.8	83.4	80.1	64.7	70.2	67.4

Table 7: Adapter architecture ablation with Qwen2.5-VL-7B.

Method	vid/frm	Attn.	Adapt.	Seed	SAM2	E2E	$\mathcal{J}\&\mathcal{F}$
ATLAS (ours)	64 / 1024	2.06	0.003	0.001	13.2	15.3	67.4
ATLAS	256 / 1024	2.46	0.003	0.001	13.1	15.6	67.5
DecAF [ 13 ]	256 / 1024	8.08	–	0.030	33.2	41.3	63.9

Table 8: Runtime and accuracy on an NVIDIA RTX PRO 6000.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimizer	AdamW
Learning rate	$10^{-4}\rightarrow 10^{-5}$ (linear decay)
Weight decay	$10^{-4}$
Epochs	20
Batch size	8 clips
Segmentation loss	BCE-with-Logits + Soft Dice

Appendix

Table 9: Training hyperparameters of the ATLAS adapter.

Stage	Operation	Output	Spec.
Input	Frame / Frame+Video Attn.	$2C\times R_{f}\times W_{f}$	$784\times 24\times 42$
Encoder-1	$3{\times}3$ Conv $\times 2$	$128\times R_{f}\times W_{f}$	BN + ReLU
Downsample	$2{\times}2$ MaxPool	$128\times{R_{f}}/{2}\times{W_{f}}/{2}$	–
Encoder-2	$3{\times}3$ Conv $\times 2$	$256\times{R_{f}}/{2}\times{W_{f}}/{2}$	BN + ReLU
Bottleneck	$3{\times}3$ Conv $\times 2$	$512\times{R_{f}}/{2}\times{W_{f}}/{2}$	–
Temporal	GAP $\rightarrow$ Transformer	$512$	2 layers, 4 heads

Appendix

Table 10: ATLAS Temporal FiLM FV U-Net architecture (

C{=}392

, grid

R_{f}{\times}W_{f}{=}24{\times}42

Figure 6: Training convergence of the U-Net adapter and qualitative outputs before and after training.

Figure 7: Additional qualitative example. From top to bottom: input frames, raw averaged attention maps, adapted attention maps produced by ATLAS, and the corresponding SAM2 segmentation masks.

Figure 8: Additional qualitative example. From top to bottom: input frames, raw averaged attention maps, adapted attention maps produced by ATLAS, and the corresponding SAM2 segmentation masks.

Figure 9: Additional qualitative example. From top to bottom: input frames, raw averaged attention maps, adapted attention maps produced by ATLAS, and the corresponding SAM2 segmentation masks.

Figure 10: Additional qualitative example. From top to bottom: input frames, raw averaged attention maps, adapted attention maps produced by ATLAS, and the corresponding SAM2 segmentation masks.

Figure 11: Additional qualitative example. From top to bottom: input frames, raw averaged attention maps, adapted attention maps produced by ATLAS, and the corresponding SAM2 segmentation masks.

Figure 12: Additional qualitative example. From top to bottom: input frames, raw averaged attention maps, adapted attention maps produced by ATLAS, and the corresponding SAM2 segmentation masks.

Figure 13: Additional qualitative example. From top to bottom: input frames, raw averaged attention maps, adapted attention maps produced by ATLAS, and the corresponding SAM2 segmentation masks.

Figure 14: Additional qualitative example. From top to bottom: input frames, raw averaged attention maps, adapted attention maps produced by ATLAS, and the corresponding SAM2 segmentation masks.

Figure 15: Additional qualitative example. From top to bottom: input frames, raw averaged attention maps, adapted attention maps produced by ATLAS, and the corresponding SAM2 segmentation masks.

Video Object Segmentation

Video object segmentation assigns pixel-level masks to objects across frames while preserving their identities through motion, occlusion, and change.

See all 114 papers

SteerSeg: Attention Steering for Reasoning Video Segmentation
May 14, 2026Ali Cheraghian, Hamidreza Dastmalchi, Abdelwahed Khamis +3Video Object Segmentation Referring Video Object Segmentation
1Macquarie University · 2York University · 3CSIRO Data61 +1
VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation
Jun 5, 2026Ming Dai, Sen Yang, Boqiang Duan +4Video Object Segmentation Referring Video Object Segmentation
School of Automation, Southeast University, Nanjing, China · Baidu Inc, Beijing, China · College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, China
VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation
Mar 28, 2026Jihwan Hong, Jaeyoung DoVideo Object Segmentation Referring Video Object Segmentation
AIDAS Laboratory, 1IPAI, Seoul National University · AIDAS Laboratory, 2ECE, Seoul National University

May 14, 2026J/K move · Enter open · S save

SteerSeg: Attention Steering for Reasoning Video Segmentation

Ali Cheraghian, Hamidreza Dastmalchi, Abdelwahed Khamis, Morteza Saberi+2

1Macquarie University · 2York University · 3CSIRO Data61+1

Video Object Segmentation Referring Video Object Segmentation Prompt Tuning Vision-Language Grounding Reasoning Segmentation

Open paper AskPDF arXiv

Video reasoning segmentation requires localizing objects across video frames from natural language expressions, often involving spatial reasoning and implicit references. Recent approaches leverage frozen large vision-language models (LVLMs) by extracting attention maps and using them as spatial priors for segmentation, enabling training-free grounding. However, these attention maps are optimized for text generation rather than spatial localization, often resulting in diffuse and ambiguous grounding signals. In this work, we introduce SteerSeg, a lightweight framework that identifies attention misalignment as the key bottleneck in attention-based grounding and proposes to steer attention at its source through input-level conditioning. SteerSeg combines learnable soft prompts with reasoning-guided Chain-of-Thought (CoT) prompting. The soft prompts reshape the attention distribution to produce more spatially concentrated maps, while CoT-derived attributes resolve ambiguity among similar objects by guiding attention toward the correct instance. The resulting attention maps are converted into point prompts across keyframes to guide a segmentation model, while candidate tracklets are ranked and selected using correlation-based scoring. Our approach freezes the LVLM and segmentation model parameters and learns only a small set of soft prompts, preserving the model's pretrained reasoning capabilities while significantly improving grounding. Despite being trained only on Ref-YouTube-VOS, SteerSeg generalizes well across diverse benchmarks, significantly improving the spatial grounding capability of LVLMs. Project page: https://steerseg.github.io

Abstract

Video Object Segmentation

Explore similar work

SteerSeg: Attention Steering for Reasoning Video Segmentation

VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation

VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation