cs.CVOct 29, 2026
SaveLook at Where You Think: Localization-Head Alignment for Fine-Grained Perception in Thinking VLMs
Organizations: Department of Computer Science and Technology, Tsinghua University · Ant Group
Abstract
Thinking vision-language models reason for thousands of tokens, yet still miss small details in high-resolution images. Zooming methods pay for extra inference passes, and outcome rewards never tell the model where to look at. In Qwen3.5-4B, we find that the bottleneck is selection, not seeing: a few attention heads, which we call localization heads, pick the evidence out of thousands of visual tokens; ablating four of them costs 13.2 points more than ablating four random heads, and their focus fades as the reasoning grows. We therefore propose Localization-Head Alignment (LHA), which makes attention a training signal: whenever the reasoning states a visual fact, these heads learn to look at the focal region rather than the background, as an auxiliary loss in both SFT and RL training. On benchmarks of V*, HR-Bench 4K and TreeBench, LHA lifts Qwen3.5-4B from 74.1% to 76.6% accuracy, the best single-pass 4B result we evaluate, ahead of multi-pass zooming and released 7B models. Alignment alone adds 1.8 points over otherwise identical training while shortening reasoning by 20%, and the gains carry over to Qwen2.5-VL-3B.
Figures & tables
Figure 1: Supervising where a few heads look gives the most accurate single-pass Qwen3.5-4B model, with shorter reasoning. Overall accuracy on V*, HR-Bench 4K and TreeBench against mean generated tokens. Hollow markers use extra passes or crops. The arrow adds the alignment loss to otherwise identical SFT and RL; the inset breaks down LHA’s gain over the base model by benchmark.
Figure 2: Thinking VLMs need small evidence and rely on a few localization heads to select it (Qwen3.5-4B). (a) Masking the evidence hurts far more than masking an equal-area region elsewhere. (b) Ablating four localization heads hurts far more than ablating four random heads of the same layers. (c) The heads’ share of visual attention inside the evidence box declines along the chain of thought. (d) On V*, the heads concentrate on the evidence at the end of the question far more often when the answer is correct. Error bars: 95% bootstrap CIs.
Figure 3: LHA on a TallyQA training question. Top: in the model’s own correct reasoning, each fact phrase names an annotated object (same color in the image). At the query that predicts a fact token, the routing loss raises the attention of the localization heads (red cells in the LLM) on the object’s visual tokens against an equally large background ; the answer token, where the preference is already at chance, is skipped (.75 vs .50 on average). Bottom: the heads are the top ten of 128, ranked on grounding questions; ablating the top four costs 13.2 points more than ablating four random heads. The same loss joins cross-entropy in alignment SFT (flame: trained; snowflake: frozen) and GRPO on correct rollouts in attention RL, where it lifts the heads’ preference from .76 to .86 in 50 steps (Table 3 b). Inference is unchanged.
| Method | V ∗ | HR-Bench 4K | TreeBench | Overall | Tokens | ||||||
| Attr. | Spatial | All | FSP | FCP | All | Perc. | Reas. | All | |||
| Released perception models (own backbones) | |||||||||||
| ZwZ-4B † | 90.4 | 88.2 | 89.5 | 90.0 | 72.2 | 81.1 | 63.8 | 39.5 | 48.4 | 72.8 | 185 |
| DeepEyes-7B (no tools) † | 73.0 | 68.4 | 71.2 | 85.8 | 61.0 | 73.4 | 57.0 | 28.9 | 39.3 | 63.2 | 45 |
| TreeVGR-7B † | 82.6 | 84.2 | 83.2 | 87.8 | 58.8 | 73.2 | 61.1 | 37.5 | 46.2 | 66.8 | 60 |
| Qwen3.5-4B (base) † | 84.3 | 85.5 | 84.8 | 91.0 | 77.0 | 84.0 | 73.8 | 35.5 | 49.6 | 74.1 | 4,678 |
Table 1: Supervising where a few heads look gives the most accurate single-pass Qwen3.5-4B model. Accuracy (%) in thinking mode (temperature 1, 32K-token budget) on V ∗ (attribute and spatial questions), HR-Bench 4K (single- and cross-instance) and TreeBench (perception and reasoning); Overall pools all 1,396 questions and Tokens is the mean completion length. Below EASE, SFT on the same self-generated trajectories uses cross-entropy (CE) alone or adds the alignment loss, and each is followed by the same outcome RL; LHA continues alignment SFT with attention RL. Rows average two decodes (†: one). Bold : best single-pass Qwen3.5-4B model.
| Method | V ∗ | Tree | V ∗ +Tree |
|---|---|---|---|
| Base | 74.3 | 39.8 | 50.8 |
| CE SFT | 77.0 | 39.8 | 51.7 |
| Alignment SFT | 78.0 | 40.5 | 52.5 |
| CE SFT RL | 75.4 | 38.8 | 50.5 |
| Alignment SFT RL | 78.5 | 41.0 | 53.0 |
Table 2: Qwen2.5-VL-3B (direct answers, greedy decoding). Accuracy (%) on V ∗ and TreeBench; both RL rows use the same outcome-reward RL.
| (a) Gradients at the start of RL | |||||
| mean | |||||
| mixed | mixed | correct | mixed | groups | |
| Outcome PG | 1.06 | 0.04–0.04 | 0 | 0.09 | 2/8 |
| KL | 0.11 | 0.01–0.05 | 0.03 | 0.01 | 3/8 |
| Attention | 0.06 | 0.10–0.32 | 0.18 | 0.13 | 8/8 |
Table 3: The attention term supplies the routing signal that outcome RL lacks (Qwen3.5-4B, same alignment-SFT start). (a) Gradients on rollout groups of five: median norm and cosine with the routing direction in mixed groups, mean first-order change of in all-correct and mixed groups, and how many of these eight groups it raises. (b) of the supervised heads on 82 fixed training-domain trajectories; held-out: the 17 not used in SFT; V ∗ : at the object words of the base model’s V ∗ reasoning, against the benchmark’s own boxes (.844 at the start). Last two columns at step 50.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Decoding | 95% CI | / | |
| Alignment SFT RL vs. CE SFT RL | |||
| T=1, decode 1 | [ 0.4, 3.0] | 91/73 | |
| T=1, decode 2 | [0.5, 3.9] | 92/61 | |
| T=1, both decodes | [0.5, 2.9] | – | |
| greedy | [0.4, 4.0] | 89/59 | |
| LHA vs. its control | |||
Table 4: Paired comparisons between runs that differ in one component. Accuracy difference (points), 95% bootstrap interval clustered by source question, and questions gained ( ) and lost ( ).
| Subset | Evid. | Ctrl. | [95% CI] | |
| V* | 191 | 43.5 | 82.2 | 38.7 [ 46.6, 30.4] |
| TreeBench | 333 | 33.3 | 45.9 | 12.6 [ 18.6, 6.6] |
| Area 0.21% | 174 | 42.5 | 70.1 | 27.6 [ 36.2, 19.0] |
| 0.21–2.4% | 175 | 37.1 | 62.3 | 25.1 [ 33.7, 16.6] |
| Area 2.4% | 175 | 31.4 | 45.1 | 13.7 [ 22.9, 4.6] |
| All | 524 | 37.0 | 59.2 | 22.1 [ 27.1, 17.2] |
Table 5: Evidence masking by subset. Accuracy (%) with the evidence or an equal-area control region masked; area is the box’s fraction of the image.
| Visual tokens given | V* | Tree | All |
|---|---|---|---|
| Full image | 83.8 | 46.5 | 60.1 |
| Evidence box only | 77.5 | 51.1 | 60.7 |
| Equal-area control only | 44.5 | 34.2 | 38.0 |
| Full image + box tokens | 83.8 | 47.7 | 60.9 |
| Full image + control tokens | 80.6 | 45.6 | 58.4 |
Table 6: Post-encoder token controls on Qwen3.5-4B (thinking mode, one decode; 191 V* and 333 TreeBench questions). Accuracy (%).
| Base | R4 | L4 | L4 R4 [95% CI] | |
|---|---|---|---|---|
| V* | 84.8 | 83.2 | 69.1 | 14.1 [ 19.9, 8.4] |
| HR-4K | 84.0 | 83.3 | 68.3 | 15.0 [ 18.0, 12.0] |
| Tree | 49.6 | 49.6 | 40.5 | 9.1 [ 13.8, 4.4] |
| All | 74.1 | 73.5 | 60.3 | 13.2 [ 15.5, 10.9] |
Table 7: Localization-head ablation by benchmark. Accuracy (%) of the base model and with four random (R4) or localization (L4) heads ablated.
Figure 4: Question-time evidence enrichment of the four localization heads for correct and incorrect base answers (log scale; bars: medians; dotted: uniform attention).
| Acc. | vs. base [95% CI] | Trunc. | Tokens | |
|---|---|---|---|---|
| 0 (base) | 74.1 | – | 37 | 4,678 |
| 0.25 | 74.1 | 0.1 [ 1.9, 1.7] | 48 | 4,751 |
| 0.5 | 73.9 | 0.3 [ 2.0, 1.4] | 42 | 5,013 |
| 1 | 74.2 | 0.1 [ 1.8, 2.0] | 54 | 5,100 |
Table 8: Length-aware visual logit bias at the four localization heads (1,396 questions, one decode). Trunc.: budget-truncated responses; Tokens: mean completion tokens.
Figure 5: Evidence routing predicts answer correctness (Qwen2.5-VL-3B, 166 V* questions, five fixed localization heads at the answer position). (a) Evidence share for correct and incorrect greedy answers; diamonds: means. (b) Accuracy by quartile of the share.
| Run | s / step | Rollout tokens |
|---|---|---|
| Outcome RL, 100 steps | ||
| CE SFT RL | 448 | 3,170 |
| Alignment SFT RL | 379 ( 15%) | 2,570 ( 19%) |
| Attention RL, 50 steps | ||
| control | 667 | 6,179 |
| LHA | 613 ( 8%) | 5,230 ( 15%) |
Table 9: Training cost of RL (Qwen3.5-4B, four GPUs of one node). Wall-clock time per step and mean rollout length, averaged over training; the two runs of each pair use the same RL data order and hyperparameters.
Figure 6: Evidence preference along RL (Qwen3.5-4B, same alignment-SFT start). The ten supervised heads at the fact tokens of 82 fixed training-domain trajectories, teacher-forced through each checkpoint; dashed: the 17 held out from SFT; bands: 95% bootstrap intervals.
Figure 7: Gradient decomposition at the alignment-SFT policy (Qwen3.5-4B; points: groups of five rollouts). (a) Norm of each objective’s gradient against its cosine with the routing direction . (b) First-order change of under one gradient step (bars: means); all-wrong groups receive neither term.
Figure 8: Alignment SFT raises the evidence share at the answer position, and RL preserves it (Qwen2.5-VL-3B, V*). Grey: CE; red: alignment.
| Method | Passes | +Img. | Tokens | Acc. |
| Base † | 1 | 0 | 4,678 | 74.1 |
| ViCrop | 3 | 3 | 4,230 | 75.9 |
| VCD † | 2 /token | 1 | 4,612 | 75.3 |
| ZoomEye † | 124.2 | 123.2 | 5,371 | 72.1 |
| Fixed-grid zoom † | 1 | 4 | 4,725 | 74.1 |
| GRPO | 1 | 0 | 4,828 | 74.0 |
Table 10: Inference cost per question. Passes: full forward passes over the prompt (VCD: two per decoding step); +Img.: image encodings beyond the original image; Acc.: overall accuracy (%).
| Endpoint | Dec. | V ∗ | HR-4K | Tree | Overall | Tokens | Tr. |
|---|---|---|---|---|---|---|---|
| Qwen3.5-4B (base) | |||||||
| Qwen3.5-4B (base) | 1 | 162 | 672 | 201 | 74.1 (1035) | 4,678 | 37 |
| Head ablation at inference (base model) | |||||||
| Ablate the four localization heads | 1 | 132 | 546 | 164 | 60.3 (842) | 11,471 | 327 |
| Ablate four random heads (same layers) | 1 | 159 | 666 | 201 | 73.5 (1026) | 3,384 | 24 |
| RL from the base model (1,810-question pool, 100 updates) | |||||||
Table 11: All thinking-mode endpoints (temperature 1, 32K budget) used in the paper. V ∗ , HR-4K and Tree give correct answers out of 191, 800 and 405; Overall is accuracy in % with the count out of 1,396; Tokens is the mean completion length and Tr. the number of budget-truncated responses. Dec. indexes the two independent decodes (1 and 2) of the same checkpoint; HR-Bench rotations of one image are scored as separate questions.
| Method | V ∗ | HR-4K | Tree | Overall | Tokens | Trunc. (%) | vs. base [95% CI] |
| Qwen3.5-4B (base) | 78.0 | 75.8 | 42.7 | 66.5 | 7,144 | 19.1 | — |
| ViCrop (rel-att) | 79.1 | 79.1 | 41.2 | 68.1 | 6,498 | 16.8 | [ 0.6, 4.0] |
| GRPO | 73.3 | 76.5 | 42.0 | 66.0 | 7,737 | 20.9 | [ 2.3, 1.5] |
| DAPO | 75.4 | 79.0 | 44.2 | 68.4 | 5,830 | 14.8 | [ 0.1, 4.0] |
| EASE | 73.8 | 74.8 | 44.0 | 65.7 | 8,002 | 21.8 | [ 2.7, 1.2] |
| CE SFT RL | 80.6 | 82.1 | 45.7 | 71.3 | 4,864 | 10.4 | [2.7, 7.1] |
Table 12: Greedy decoding (temperature 0, same budget and scoring; one decode per model). : paired difference to the base model with 95% CIs. Alignment SFT RL leads CE SFT RL by points (95% CI [0.4, 4.0]).
Figure 9: The alignment loss shortens the whole length distribution. Survival curves of response length over two decodes; inset: token saving split by which model finishes; dashed line: 32K budget.
Figure 10: Encoder-side token controls on Qwen2.5-VL-3B (Appendix C.2 ). Accuracy change relative to the full image, with 95% CIs. Red: evidence given as pixels or post-encoder tokens; grey: input-size and position controls; blue: location cues.
Figure 11: Visual attention drifts away from the evidence as the chain grows (Appendix C.6 ; four localization heads, 1,072 same-question rollout pairs). (a) Log share of attention on visual tokens. (b) Image–text logit gap , from all and from non-rotary dimensions. (c) Share of visual attention on the evidence box.
Figure 12: A few deep heads localize the evidence (Appendix D ; Qwen2.5-VL-3B, direct answers). (a) Evidence share of each of the 576 heads on V*; white boxes mark the five strongest. (b) Strongest head and head mean per layer; dotted: mean box area. (c) Attention of each layer on one V* question (cyan box), relative to uniform; red layers have evidence share 0.10.
Figure 13: TreeBench #181: the crop pulls attention onto the occluding signboard (Qwen2.5-VL-3B). (a) From the full image the model answers D, the yellow dragon flag, and its localization heads put 7.1% of their visual attention on the evidence (green box). (b) With the crop appended it answers A, the signboard, and the share rises to 59.7%. (c, d) Attention mass of the same heads over the input sequence.
More in this topic
Fine-Grained Visual Perception
Fine-grained visual perception studies how models detect, distinguish, and ground subtle visual details, attributes, parts, and changes.
See all 154 papers