Vision-language models (VLMs) that segment from language must pass the location of the referent to a pixel decoder. The dominant interface compresses it into the hidden state of a single [SEG] token, a d-dimensional vector with no spatial index, and leaves the decoder, which never sees the expression, to recover the referent's extent. We show that the VLM already computes a better signal: the [SEG] token's attention over the image tokens, which is defined on the patch grid by construction. Probing a frozen segmentation VLM, we find that this attention yields 18 to 22 more gIoU than the [SEG] hidden state when read by the same probe, that it gains nothing from the image where the hidden state does, and that it is concentrated in the late layers. Building on these findings, we introduce AttnSeg, which replaces the [SEG] embedding with this attention as the decoder's prompt. A learned, signed combination of the attention heads forms a dense spatial prompt, and an attention-guidance objective supervises it directly. A two-stage recipe first trains the VLM to localize without the decoder, then trains the decoder to refine, within the budget of standard fine-tuning. Applied to Sa2VA, AttnSeg improves ReasonSeg by 6.4 gIoU and GS-Eval by 10.2 at 4B (4.2 and 9.6 over the same model fine-tuned on the same data), and by 8.8 and 11.3 at 14B. The gains reach 13 to 18 gIoU on object parts, multiple objects, and single objects, where the referent must be singled out, and are negligible on stuff regions, which the decoder already recovers unaided. Code and models are available at https://eslambakr.github.io/attseg_project_page/
Figures & tables
Figure 2: Which internal signal carries the localization, and where does it reside? Probes trained on a frozen Sa2VA-4B. (a) Read by the same U-Net, the [SEG] attention beats the [SEG] embedding by 18.2 and 21.5 gIoU, and adding the RGB image helps the embedding ( +4.4 , +7.3 ) but not the attention ( −0.3 , +0.2 ). (b) Probes restricted to a five-layer window: the early layers carry almost no localization, and a single late window matches the whole stack (dashed).
Figure 3: How AttnSeg reads and shapes the spatial channel. The [SEG] token’s query row of A , restricted to image columns and folded onto the patch grid, gives one map per layer and head; a 1×1 convolution with signed weights combines them and Latt supervises the result.
ReasonSeg (gIoU)
GS-Eval (gIoU)
Method
Sentence
Phrase
Overall
Stuff
Object parts
Multi obj.
Single obj.
Overall
LISA-7B
47.49
39.08
45.47
80.34
21.25
71.95
42.62
56.34
GLaMM
51.68
42.94
49.58
91.38
24.37
78.50
54.11
65.66
Sa2VA-Qwen3-4B-SAM2
59.42
56.28
58.67
91.41
22.79
76.91
54.77
65.41
Sa2VA-Qwen3-4B-SAM3
65.46
61.47
64.50
89.09
17.06
70.23
48.74
60.29
VISA-7B
50.73
43.50
49.00
85.30
25.39
75.70
50.48
62.10
Table 1: Comparison with prior work , in gIoU. Each highlighted row is AttnSeg on the base model directly above it, and bold marks the best value in each column. Over its base, AttnSeg gains little on stuff and 13 to 18 gIoU on object parts, multiple objects and single objects.
Map pointing
Map extent
Mask gIoU
Stage-one objective
ReasonSeg
GS-Eval
ReasonSeg
GS-Eval
ReasonSeg
GS-Eval
Peak ratio
74.32
78.82
3.26
4.34
60.13
65.99
Soft Dice
63.89
65.65
52.64
56.03
61.91
69.56
Weighted cosine (ours)
69.24
75.77
49.60
59.09
64.11
72.11
Table 2: Why the weighted cosine wins. Attention-map quality and final mask gIoU on Sa2VA-InternVL2.5-4B for the three stage-one objectives, each followed by the same stage two. Pointing is the fraction of queries whose strongest map patch lies inside the target, and extent the gIoU of the map thresholded at 0.5 after min-max normalization. Each alternative fails on one axis; the cosine is the only objective with no severe weakness. Table 7 gives per-split results.
Table 5
Figure 4: The attention map is the prediction, not a diagnostic. For each query we show the aggregated [SEG] attention a^ and the mask the decoder refines from it. More examples in Appendix G .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Probe
Input
Params
What it can do
Gate
attention, (LH)×h×w
560
one scalar per (layer, head), softmax, weighted sum
[SEG] embedding, L×d
672 036
one scalar per layer, then the shared projection
Conv
attention
5 041
a single 3×3 convolution, no norm, no nonlinearity
[SEG] embedding
672 325
the shared projection, then the same 3×3 convolution
U-Net
attention
3.86 M
three stages, base width 64, skip connections
[SEG] embedding
4.23 M
the shared projection, then the same U-Net
Appendix
Table 5: The probes. Section 3.2 uses the convolution and the U-Net, and Section 4.3 adds the gate. Three probes of increasing capacity, applied to both signals with the VLM frozen. The gate has no convolution, bias, nonlinearity or spatial support, so it can only select among maps the model has already drawn.
Task
Dataset
Repetition factor
Semantic segmentation
COCO-Stuff ( Caesar et al., 2018 )
0.5
ADE20K ( Zhou et al., 2017 )
1
Pascal-Part ( Chen et al., 2014 )
4
Referring segmentation
RefCOCO ( Yu et al., 2016 )
2
RefCOCO+ ( Yu et al., 2016 )
2
RefCOCOg ( Mao et al., 2016 )
2
Appendix
Table 6: Training mixture and sampling weights , the thirteen datasets released with Sa2VA ( Yuan et al., 2025 ) , identical in both stages and at both model scales. A factor above one oversamples a dataset within the epoch and a factor below one is a seeded subsample.
ReasonSeg (gIoU)
GS-Eval (gIoU)
Stage-one objective
Sentence
Phrase
Overall
Stuff
Object parts
Multi obj.
Single obj.
Overall
Peak ratio
60.75
58.16
60.13
91.30
28.53
80.64
52.59
65.99
Soft Dice
63.06
58.28
61.91
91.01
28.41
83.94
60.08
69.56
Weighted cosine (ours)
64.80
61.93
64.11
91.20
35.29
86.37
62.97
72.11
Appendix
Table 7: Choice of the attention-guidance objective, per split , on Sa2VA-InternVL2.5-4B, in gIoU. Each row runs both training stages with the area weighting of Eq. ( 5 ) and differs in the stage-one objective. Bold marks the best value per column.
ReasonSeg
GS-Eval
Stage-one objective
Off-target
Top-1%
Particip.
Off-target
Top-1%
Particip.
Peak ratio
76.19
6.28
0.81
70.14
11.29
0.70
Soft Dice
45.42
36.66
0.15
44.79
38.39
0.12
Weighted cosine (ours)
64.81
8.92
0.44
66.20
9.32
0.46
Appendix
Table 8: Where the misplaced attention sits. Properties of the attention mass outside the target for each stage-one objective: its share of the total map (off-target), the share of it held by the strongest 1% of background patches (top-1%), and its normalized participation ratio, which is 1 when it is spread evenly and approaches 0 when it sits on a single patch.
Figure 5: Additional qualitative results on GS-Eval. Query, aggregated [SEG] attention map, and the refined mask.
Figure 6: Additional qualitative results on ReasonSeg. Query, aggregated [SEG] attention map, and the refined mask.