Autoregressive vision-language models (VLMs) struggle to ground generated text in the correct image regions, and existing explainability methods, such as attention maps and Logit Lens, often produce poor spatial explanations due to weak text-to-vision attention and sink tokens. We propose Angular Relevance (RL), a simple visualization measure based on cosine similarity between visual token embeddings and unembedding-matrix rows. Because RL is invariant to token norm, it produces substantially sharper and more faithful heatmaps than attention maps or Logit Lens on out-of-the-box models. Without any training, RL yields competitive referring-expression segmentation (RES), outperforming prior diagnostic maps and CLIP-based training-free methods. To preserve localized object information during cross-modal mixing, we introduce Angular Object-Aware Loss (AOL), an auxiliary loss that directly supervises visual token embeddings to align angularly with the vocabulary directions of their corresponding object concepts. Finetuning with AOL alongside the standard next-token prediction objective significantly sharpens object localization in heatmaps, showing that the VLM's own visual tokens can be used directly for grounding and improving RES, without training any segmentation heads and without degrading general VQA performance. We further demonstrate that the learned grounding also transfers to category-level segmentation of object categories not seen during training.
Figures & tables
Figure 1: Mechanistic comparison of three diagnostic maps for autoregressive vision-language models. Attention maps visualize text-to-vision attention weights, Logit Lens maps visual embeddings through the unembedding matrix and softmax to obtain queried-token probabilities, while Angular Relevance directly measures cosine alignment between final-layer visual embeddings and the queried vocabulary direction (rows of the unembedding matrix).
Figure 2: From left to right: (1) input image with referring expression used to query the Qwen2.5-VL model, (2) attention map from the text-to-vision attention, (3) Logit Lens heatmap, (4) heatmap produced by our relevance measure RL on the base model, and (5) heatmap produced by RL after fine-tuning with NTP+AOL. Columns (2)–(4) are obtained from the base model. Our relevance measure yields more localized and semantically aligned heatmaps than attention and Logit Lens, and fine-tuning with AOL further sharpens object localization. Warmer areas indicate higher relevance.
Methods
Backbone
With SAM
RefCOCO
RefCOCO+
RefCOCOg
val
testA
testB
val
testA
testB
val
test
Cropping [ 50 ]
CLIP
Y
22.7
21.1
23.1
24.1
22.4
23.9
28.7
27.5
TAS [ 40 ]
CLIP
N
29.5
30.3
28.2
33.2
38.8
28.0
35.8
36.2
Ref-Diff [ 32 ]
Stable Diff
Y
35.2
37.4
34.5
35.6
38.7
31.4
38.6
37.5
RESAnything [ 42 ]
Pixtral-12B
Y
68.5
72.2
70.3
60.7
65.6
52.2
60.1
60.5
Logit Lens (Qwen adapt.) [ 19 ]
Qwen2.5-VL-7B
N
63.2
64.8
60.1
52.4
57.1
45.9
53.2
54.1
Table 1: Referring Expression Segmentation result of training-free methods in cIoU (%). For our approach, Heatmap+Thr obtains masks by thresholding (binarizing) the relevance heatmap, while Heatmap+SAM2 uses the heatmap to generate masks via SAM2. “Backbone” denotes the main model family used for grounding, including CLIP-based methods and autoregressive VLMs. “With SAM” indicates whether SAM/SAM2 is used at any stage of the pipeline (Y: yes, N: no). Red/blue denote best/second-best.
Methods
RefCOCO
RefCOCO+
RefCOCOg
val
testA
testB
val
testA
testB
val
test
LISA-7B [ 23 ]
74.1
76.5
71.1
62.4
67.4
56.5
66.4
68.5
GSVA-7B [ 45 ]
76.4
77.4
72.8
64.5
67.7
58.6
71.1
72.0
F-LMM-7B [ 44 ]
75.2
79.1
71.9
63.7
71.8
54.7
67.1
68.1
LISA-13B [ 23 ]
73.4
76.2
69.5
62.3
66.6
56.3
68.2
68.5
GSVA-13B [ 45 ]
77.7
79.9
74.2
68.0
71.5
61.5
73.2
73.9
Table 2: Referring Expression Segmentation results of supervised VLM-based methods . All methods use pixel-level supervision, either by fine-tuning a VLM or by training a task-specific segmentation head. AOL finetuning improves average RES cIoU over the base model by +7.7 (LLaVA, Thr), +4.0 (LLaVA, SAM2), +9.4 (Qwen, Thr), and +6.5 (Qwen, SAM2) points.
Method
Setting
All
Overlap
Disjoint
Logit Lens
Base
28.8
29.9
27.5
NTP
30.2
31.2
28.7
Ang. Rel.
Base
36.9
39.8
33.5
NTP
38.5
41.6
35.2
NTP+AOL
57.4
62.4
51.6
Δ= (NTP+AOL − Base)
+20.5
+22.6
+18.1
Table 3: ImageNet-S50 cross-dataset generalization. Results in mIoU for Qwen2.5-VL trained on RefCOCO and evaluated on ImageNet-S50. Overlap classes have some similarity to MS-COCO classes, whereas Disjoint classes do not.
Figure 3: Angular Relevance heatmaps on COCO-disjoint ImageNet-S50 classes. The model localizes object categories absent from RefCOCO training, including food items and animals.
Figure 4: L2 norms of final-layer visual token embeddings on RefCOCO validation. Compared with NTP-only finetuning, NTP+AOL produces a more compact visual-token norm distribution and fewer high-norm outliers.
Method
VQAv2
POPE
Base Model
80.5
86.77
NTP
80.6
87.50
NTP + AOL
81.2
89.87
Table 4: General VQA and hallucination performance for Qwen2.5-VL-7B.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Layers
First 2
Middle 2
Last 1
cIoU
67.5
70.3
73.4
Appendix
Table 1: Effect of the AOL supervision layer range on RefCOCO validation cIoU (Qwen2.5-VL, NTP+AOL, thresholded heatmaps).
Model setting
RefCOCO val cIoU
Base Qwen2.5-VL
66.2
NTP
66.8
NTP+AOL ( m=0 )
67.9
NTP+AOL (background only)
68.7
NTP+AOL (foreground only)
71.2
NTP+AOL (full)
73.4
Appendix
Table 2: Ablation of the AOL objective on RefCOCO validation (Qwen2.5-VL, thresholded Angular Relevance heatmaps).
Figure 1: Angular Relevance maps for synonym queries. Top: base model; bottom: NTP+AOL. Although AOL supervision only uses the category name “cat”, localization also improves for related queries such as “kitten”, “feline”, and “tabby”.
ImageNet-S50 Class
COCO Category
Goldfinch
bird
Kuvasz
dog
Siamese cat
cat
American black bear
bear
African elephant
elephant
Airliner
airplane
Appendix
Table 3: ImageNet-S50 classes with fine-grained COCO-overlap correspondences.
ImageNet-S50 Class
COCO Category
Giant panda
bear
Container ship
boat
Digital watch
clock
Motor scooter
motorcycle
Streetcar
train
Toilet seat
toilet
Appendix
Table 4: ImageNet-S50 classes with loose COCO-overlap correspondences.
Model setting
AOL supervision
cIoU
Pretrained Qwen2.5-VL
–
66.2
NTP+AOL
SAM3-I pseudo-masks
69.6
NTP+AOL
Ground-truth masks
73.4
Appendix
Table 5: RefCOCO validation cIoU after AOL training with SAM3-I pseudo-mask supervision on RefCOCO train. No frozen segmentation model is used at inference; masks are obtained by Angular Relevance heatmaps + binarization .
Method
Setting
All
Overlap
Disjoint
mIoU
EPG
mIoU
EPG
mIoU
EPG
Logit Lens
Base model
28.8
34.2
29.9
36.0
27.5
32.1
Ang. Rel.
Base model
36.9
39.0
39.8
39.7
33.5
38.2
NTP+AOL
57.4
74.5
62.4
80.4
51.6
67.6
Appendix
Table 6: Full Localization results on ImageNet-S50.
Methods
RefCOCO
RefCOCO+
RefCOCOg
LLaVA-v1.5-7B
Base Model
52.90
42.55
45.00
NTP
53.20
42.90
45.40
NTP + AOL
54.15
43.60
46.20
Qwen2.5-VL-7B
Base Model
90.05
85.40
86.60
Appendix
Table 7: Comparison on the REC (Referring Expression Comprehension) task measured by [email protected].