Can a frontier model repair a vision-language model (VLM) by rewriting its weights? Small reasoning VLMs such as Qwen3.5 need such a repair: in thinking mode they fall into loops on many image questions, repeating themselves until the token limit, spending the whole budget and losing the answer. The usual remedies act on the decoding, leaving the looping model as it was, or must be told what to change. Yet the cause is written in the weights. Keel, the first agentic weight-editing method for VLMs on image tasks, finds and removes it there. Told only the symptom, an editor, a frontier multimodal LLM acting as an agent, locates the cause, edits the weights and tests them, with a failure census and instruments showing it the failing images and where the VLM attends; among several editors, the harness keeps only edits that clearly reduce the failure without lowering accuracy on questions no editor saw. The repair needs no training, keeps the model's size and per-token cost, and applies to already-edited weights. The editors expose where the loops live: from the original weights, independent editors on both models trace them to the attention outputs of the middle layers, and the smaller VLM stops consulting the image inside a loop. On Qwen3.5-0.8B and Qwen3.5-2B, across six image benchmarks from visual mathematics to GUI grounding, the edit removes most loops on every benchmark and shortens answers, and the edited models are significantly more accurate than the thinking models on average; the larger one gains most on GUI grounding, a task absent from the practice questions. Because the repair lives in the weights, it combines with budget forcing where held-out questions favor it: on the best available weights, Keel has the best average accuracy on both models, significantly above every training-free baseline we compare on the larger one.
Figures & tables
Figure 1: The edit stops a thinking loop. Qwen3.5-0.8B repeats the same estimates until it is cut at the token limit (middle); after the Keel edit, the same model closes its thinking and clicks inside the target (right). Selection rule in Sec. E.1 .
Figure 2: Keel repairs a VLM by rewriting its weights, without gradients (real data of the 0.8B run from the original weights). (1) The census shows how the VLM fails; the repair objective targets it. (2) The editor, a frontier LLM agent, edits the weights turn by turn: it acts (code, forward passes) and observes the results. (3) The edits of several editors are tested on held-out questions; the best is kept.
Figure 3: The edit stops the loops on four more benchmarks. Qwen3.5-0.8B, one question per benchmark (rule in Sec. E.1 ); rose: the thinking model repeats itself until it is cut without an answer; green: the end of the edited model’s answer, which is correct.
Figure 4: Keel is the most accurate method on both models. Average accuracy over the six benchmarks of the baselines and of Keel (green), and Keel’s paired gain over each with its 95% interval ( ∗ : excludes zero). Higher is better.
Table 1: Keel has the best average accuracy on both models. Accuracy (%); best average in bold; † : significantly above every baseline (paired bootstrap, 95%). Keel: the repair edit on the held-out-best starting weights (0.8B: the prior edit [ 45 ] ; 2B: the original), with budget forcing on 2B only, chosen on held-out questions. Gain: over Thinking, from unrounded averages. On 2B directional ablation kept no edit.
Figure 7: The edit also stops the loops on Qwen3.5-2B. One question per benchmark (rule in Sec. E.1 ); rose: the thinking model repeats itself until it is cut without an answer; green: the end of the edited model’s answer, which is correct.
Figure 8: Question by question, the edit rescues answers lost at the token limit. Outcome of every question for the thinking model (left) and the Keel edit (right), pooled over the six benchmarks.
Figure 9: Most editors change attention output projections in the middle layers. Every committed edit of the editing runs that start from the original weights, on both models, one row per editor; a cell marks a changed matrix in that layer. ⋆ : the Keel edits.
Figure 10: Inside a loop the image no longer matters. Image advantage: per-token log-likelihood gain of the 0.8B model’s own reasoning trace with the real image instead of a gray one, from the editor’s blank-image reads.
Figure 11: Every step of the edit is needed to remove the loops. Each step of the edit (0.8B, original weights) removed in turn; right of each panel: difference from the full edit ( ∗ : 95% interval excludes zero). Lower loop share and higher accuracy are better.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: A repair is far easier to see than an accuracy change: loops fall by up to 12 standard errors. Paired change of every paired generation pass of the editor of the 0.8B Keel edit (editor 7, Tab. 21 ) in the loop share, the cut share and accuracy, in units of the pass’s own standard error and signed so that right is the repair direction; shaded: within two standard errors; marker colour: token limit of the pass (4,096, 8,192 or the full 16,384).
Figure 13: Each editing session is a long experiment that the editor directs itself. The sessions that produced the Keel edits (numbered as in Tab. 9 ): forward passes used over time, the editor’s code runs (ticks) and the hand-offs to a fresh conversation (dashed).
Model
Editor
Questions
Accuracy (%) ↑
Cut (%) ↓
Cut in a loop (%) ↓
Qwen3.5-0.8B
7 (Keel edit)
1,536
36.0 → 39.1
48.3 → 39.8
37.8 → 5.9
Qwen3.5-2B
1 (in the union)
1,280
64.3 → 67.7
25.8 → 22.2
17.1 → 5.4
Qwen3.5-2B
2 (in the union)
2,560
63.9 → 67.0
26.4 → 22.4
16.3 → 9.6
Appendix
Table 2: The editors of the Keel edits validate their repairs themselves: higher accuracy, far fewer loops (as reported: paired, full token limit, practice questions not used to construct the edit; 0.8B: validation sets d–i of Tab. 21 ).
Setting
Value
Editor and harness
Editor
GPT-6 Astra, maximal reasoning effort
Forward-pass budget per session
40 (protocol (a): 25)
Turns per conversation before a restart
40
Session end
budget used, or 260 turns, or 14 h
Carried over at a restart
last 12,000 characters of notes
Appendix
Table 3: Hyperparameters of the editing harness, the selection step and the evaluation.
Per editing run (mean over 12 runs)
Editor turns
104
Forward passes
40
Editor input tokens (mostly cached)
3.9M
Editor output tokens
168k
Selection passes per edit
1
Appendix
Table 4: Cost of one editing run (first-round runs of protocols (b) and (c)).
Session
Turns
Code runs
Forward passes
Conversations
Hours
Editor input
Editor output
Qwen3.5-0.8B, editor 7
93
52
40
3
7.6
3.45M
155k
Qwen3.5-2B, editor 1
125
77
40
4
10.4
5.92M
236k
Qwen3.5-2B, editor 2
105
61
40
3
11.8
3.73M
168k
Appendix
Table 5: Each session behind the Keel edits finds its edit on its own in under 12 hours. Turns: editor replies (without the final report); code runs: code the harness executed; conversations: the session and its fresh continuations; tokens: API counts.
Model
Comparison
Avg. diff. ↑
95% interval
Qwen3.5-0.8B
Keel vs. Thinking
+7.92 ∗
[+6.80, +9.05]
Qwen3.5-0.8B
Keel vs. Presence penalty
+3.36 ∗
[+2.25, +4.49]
Qwen3.5-0.8B
Keel vs. Reflection suppression
+3.72 ∗
[+2.60, +4.83]
Qwen3.5-0.8B
Keel vs. Budget forcing
+2.56 ∗
[+1.41, +3.70]
Qwen3.5-0.8B
Keel vs. Directional ablation
+3.24 ∗
[+2.12, +4.37]
Qwen3.5-0.8B
Keel vs. Prior edit [ 45 ]
+1.07
[ − 0.002, +2.13]
Appendix
Table 7: Keel’s average accuracy is above every baseline’s on both models (0.8B: including the prior edit it starts from): paired difference in the average over the six benchmarks (points) and 95% bootstrap interval; below, the Keel edit of the original weights against the thinking model it starts from. ∗ : interval excludes zero.
Model
MathVista ↑
MathVerse ↑
MMStar ↑
HallBench ↑
POPE ↑
ScreenSpot ↑
Qwen3.5-0.8B
+3.7 ∗
+2.9
+4.9 ∗
+4.0 ∗
+0.4
+6.5 ∗
Qwen3.5-2B
+2.0
+2.5
+3.2 ∗
+4.7 ∗
+1.8 ∗
+17.1 ∗
Appendix
Table 8: The Keel edit raises accuracy on every benchmark of both models : paired difference from the thinking model (points). ∗ : significant after Holm correction over the six benchmarks ( p<0.05 ).
Model
Editor
Protocol
Visual
Passes
Weights
Δ Acc ↑ (SE)
Δ Cut ↓ (SE)
Outcome
0.8B
1
(a)
✓
25
0.79M
+0.2 (1.5)
− 6.2 (1.7)
passes
0.8B
–
(a)
✓
25
0
–
–
no edit
0.8B
–
(a)
✓
25
0
–
–
no edit
0.8B
–
(a)
×
25
0
–
–
no edit
0.8B
–
(a)
×
25
0
–
–
no edit
0.8B
2
(b)
✓
40
0.26M
+0.9 (1.6)
− 7.2 (1.7)
passes
Appendix
Table 9: Every edit an editor commits lowers the share of cut answers on held-out questions. Every completed editing run that starts from the original weights. Editor: number in Fig. 9 (runs that committed an edit); 1 + 2: the union of the two 2B edits, which selection keeps. Protocols: (a) task only, accuracy-judged; (b) with the failure census; (c) with census and repair objective. Passes: forward passes used. Weights: number of changed weights. Δ Acc and Δ Cut: paired change on the 1,000 held-out practice questions (points, standard error in parentheses). Outcome of selection ( Sec. 2.5 ); the 0.8B pool of Keel is protocol (c) with visual instruments, the 2B pool all visual runs of protocols (b) and (c), because on 2B the (b) runs came first and the completed (c) runs gave no passing edit. Applying the 2B pool rule to 0.8B would keep the same edit: editor 7 remains the most accurate passing edit and shares the layer-15 full-attention output projection with editor 6, so no union is scored.
Figure 14: Selection keeps the most accurate edit on both models. Held-out accuracy of every edit in Tab. 9 , numbered as there: paired change on the 1,000 held-out practice questions (points, 95% intervals). Filled: passes the repair test (cut share lowered by more than two standard errors and accuracy raised); ringed and shaded: kept by selection. Higher is better.
Diagnosis
Visual
Select.
Avg. ↑
Loops ↓
Thinking
48.2
26.1
w/o diagnosis
×
✓
✓
51.3
11.6
w/o visual instruments
✓
×
✓
48.2
26.1
w/o selection
✓
✓
×
51.8
6.3
Keel edit
✓
✓
✓
51.9
4.3
Appendix
Table 10: Every step of the edit is needed. Ablation of the Keel edit of the original 0.8B weights (✓ kept, × removed), evaluated as in Tab. 1 without budget forcing. Diagnosis: census and repair objective (without them, editors are judged by accuracy alone, budget of 25 forward passes); Visual: visual instruments (the text-only edit fails the repair test, so no edit is kept); Select.: selection on held-out questions ( w/o selection : mean of the three single runs of the pool). Avg.: average accuracy over the six benchmarks (%); Loops: answers ending in a verbatim loop, pooled over all questions (%).
Variant minus Keel edit
Avg. accuracy ↑
Loops ↓
w/o diagnosis
− 0.58 [ − 1.70, +0.54]
+7.27 ∗ [+6.46, +8.10]
w/o visual instruments
− 3.73 ∗ [ − 4.81, − 2.68]
+21.84 ∗ [+20.80, +22.85]
w/o selection
− 0.11 [ − 0.76, +0.53]
+2.02 ∗ [+1.63, +2.42]
Appendix
Table 11: Removing any step leaves significantly more loops : paired differences from the Keel edit on Qwen3.5-0.8B (points, 95% bootstrap interval over questions; average accuracy over the six benchmarks, loops pooled). ∗ : interval excludes zero.
Method
Avg. ↑
Thinking
48.2
Keel edit (one round)
51.9
Keel edit (two rounds)
53.4
Appendix
Table 13: A second round improves the edit further on Qwen3.5-0.8B. Average accuracy over the six benchmarks (%); two rounds: the second-round edit selected on top of the first-round edit.
Figure 24
Model
Method
MathVista ↓
MathVerse ↓
MMStar ↓
HallBench ↓
POPE ↓
ScreenSpot ↓
Qwen3.5-0.8B
Budget forcing
29
56
25
13
3
47
Qwen3.5-0.8B
Prior edit + budget forcing
11
31
6
1
0
0
Qwen3.5-0.8B
Keel + budget forcing
6
17
3
0
0
0
Qwen3.5-2B
Budget forcing
17
37
17
9
4
25
Qwen3.5-2B
Keel
10
33
11
4
1
5
Appendix
Table 17: With the edit, budget forcing has to force fewer answers on every benchmark. Share of answers (%) whose thinking reaches the budget and is closed by the forcing sentence. Keel: as in Tab. 1 ; the Keel edit with budget forcing: Fig. 16 .
Model
Comparison
MathVista
MathVerse
MMStar
HallBench
POPE
ScreenSpot
Pooled [95% interval]
Mean generated tokens per answer (thousands) ↓
Qwen3.5-0.8B
Keel edit vs. Thinking
− 0.40
− 0.41
− 0.92 ∗
− 0.88 ∗
− 0.23 ∗
− 4.03 ∗
− 1.20 ∗ [ − 1.33, − 1.06]
Qwen3.5-0.8B
Keel edit + budget forcing vs. Budget forcing
− 0.62 ∗
− 0.27
− 0.83 ∗
− 0.86 ∗
− 0.34 ∗
− 3.16 ∗
− 1.06 ∗ [ − 1.19, − 0.92]
Qwen3.5-0.8B
Keel vs. Prior edit + budget forcing
− 1.42 ∗
− 1.99 ∗
− 1.23 ∗
− 0.72 ∗
− 0.04 ∗
− 1.06 ∗
− 0.99 ∗ [ − 1.06, − 0.91]
Qwen3.5-2B
Keel edit vs. Thinking
− 0.31
− 0.22
− 0.45 ∗
− 0.61 ∗
− 0.19 ∗
− 3.01 ∗
− 0.84 ∗ [ − 0.95, − 0.72]
Qwen3.5-2B
Keel vs. Budget forcing
− 0.53 ∗
− 0.50 ∗
− 0.47 ∗
− 0.52 ∗
− 0.41 ∗
− 2.35 ∗
− 0.82 ∗ [ − 0.93, − 0.70]
Appendix
Table 18: On every benchmark the edit shortens answers, leaves fewer loops, needs fewer forced answers and rescues more answers than it newly cuts : edited minus reference, per benchmark (tokens in thousands, shares in percentage points; ∗ : significant after Holm correction over the six benchmarks) and pooled over all questions with a 95% bootstrap interval ( ∗ : interval excludes zero). Prior edit: the edit released with [ 45 ] .
Figure 17: Keel has the lowest or tied-lowest loop share on every benchmark. Share of answers that are cut at the token limit and end in a verbatim loop, for every method of Tab. 1 ; the number above the Keel bar is its share. Prior edit: the edit released with [ 45 ] ; BF: budget forcing. Lower is better.
Figure 18: The edit rescues more answers from the token limit than it newly cuts on every benchmark. Fig. 8 per benchmark: outcome of the thinking model (left) and of the Keel edit (right) on the same question, as shares of the benchmark’s questions (%). Green: cut by the thinking model and answered correctly after the edit (rescued); rose: answered correctly by the thinking model and cut after the edit (newly cut).
Thinking: correct
Thinking: wrong
Thinking: cut
Model
Benchmark
n
correct ↑
wrong ↓
cut ↓
correct ↑
wrong ↓
cut ↓
correct ↑
wrong ↓
cut ↓
Qwen3.5-0.8B
MathVista
1,000
423
56
27
56
136
20
64
72
146
MathVerse
788
107
34
27
21
102
48
63
93
293
MMStar
1,500
597
84
28
90
276
21
95
140
169
HallusionBench
951
491
82
8
65
174
6
63
48
14
POPE
1,500
1,244
52
1
41
133
0
18
11
0
Appendix
Table 20: On every benchmark the edit rescues more answers than it newly cuts : outcome of every question before and after the edit (counts behind Fig. 8 and Fig. 18 ). Column groups: the outcome of the thinking model; columns within a group: the outcome of the Keel edit on the same question; cut: reached the token limit without a correct answer. Green: rescued; newly cut: correct → cut.
Figure 30
Figure 21: Each choice of the kept edit wins its test: only the attention writes carry the loops, both kinds of attention together repair best, and half strength is the most accurate and cuts the fewest answers. Passes of editor 7 (0.8B Keel edit) by what they tested; paired change, edited minus original, 95% intervals (lower is better except for accuracy); loops: cut in a verbatim loop; green band: the kept choice. (a) Loop direction removed from the writes of layers 8–22 (8,192 tokens, strength 1; amplified: −0.5 ; random: same norm, orthogonal). (b) Removed from all, from the linear-attention (DeltaNet) or from the full-attention writes (full length, strength 0.5). (c) All attention writes, increasing strength. (d) The final edit on validation sets e–i (256 unused questions each); all: pooled.
Figure 22: The edit stops the loops on the remaining benchmarks. The remaining four picks of the rule of Sec. E.1 : the benchmark–model pairs not shown in Figs. 3 and 7 . ScreenSpot-v2: orange box, target element; green ring, the click of the Keel edit. Outputs as in Sec. E.1 .
Figure 23: The edit ends the loop on more MathVista questions. The second and third pick of the rule of Sec. E.1 on each model. Rose box: the end of the thinking model’s answer, cut at the token limit (bottom), and the same lines where they last appeared (top), formulas as plain text; [… n characters …]: the text between them. Green box: the end of the Keel edit’s answer.
Figure 24: The edit ends the loop on more MathVerse questions. The second and third pick of the rule of Sec. E.1 on each model; boxes as in Fig. 23 .
Figure 25: The edit ends the loop on more MMStar questions. The second and third pick of the rule of Sec. E.1 on each model; boxes as in Fig. 23 .
Figure 26: The edit ends the loop on more HallusionBench questions. The second and third pick of the rule of Sec. E.1 on each model; boxes as in Fig. 23 . In the first and third, the thinking model names an answer, then keeps arguing until it is cut.
Figure 27: The edit ends the loop on more POPE questions. The second and third pick of the rule of Sec. E.1 on each model; boxes as in Fig. 23 .
Figure 28: The edit ends the loop on more ScreenSpot-v2 questions. The second and third pick of the rule of Sec. E.1 on each model; boxes as in Fig. 23 . Orange box: target element; green ring: the click of the Keel edit.
In weight-space programming, a language model edits a network's weights directly, without gradients. We ask what a successful repair of this kind relies on. We plant faults in the weights of two vision-language models, SmolVLM-256M and LFM2-VL-450M, and check each edit made by a coding agent against the original weights. Without being told where to look, the agent finds rescaled projector rows and noise in the vision encoder, infers the unknown scale factors, and returns most or all of the rescaled weights to their exact original values and accuracy to near its original level. These repairs depend on a trace of how the faults were computed. The original weights have 16-bit precision, and faults computed in 32-bit arithmetic leave low bits that only the damaged weights carry, a cue that a real failure need not leave. When we round those bits away, each model answers our test questions as before, but the agent, unaided, no longer repairs either model: on LFM2-VL accuracy recovers by about 4 points, against 17 from the unrounded weights. Marking the damaged weights again in their low bits brings the edits back to them. Rounded weights still hold statistical evidence of the rescaling. On SmolVLM the agent uses it once a sentence in the prompt, which we wrote knowing the fault, names such a statistic; on LFM2-VL, whose scale factors are larger, most sessions find the rescaled rows unaided, but none finds the noise. Swapped rows or columns, which keep the set of stored values, are not found unaided on either model within our budgets. We recommend that studies of weight repair report the number format of the weights and include a rounded control.
Vision-language models (VLMs) are fragile under image corruption. We find that the wording of the question affects VLMs in two opposite ways. Verbose questions make VLMs substantially more robust---e.g., rephrasing "Is there a cat?" into "Please look carefully and answer: is there a cat?". Conversely, VLMs become more fragile under corruption when the question is semantically complex or finer-grained, e.g., "what colour is the cup left of the chair?" instead of "is there a cup?". Both effects stem from question-conditioned cross-modal attention, which induces a spectral filter over image patches: verbose questions broaden its frequency support, while fine-grained questions concentrate it onto fewer visual scales. The model's answer drifts most when this filter and the corruption sit on the same spatial frequencies. We test the filter view on Qwen3-VL and LLaVA-OneVision across GQA and CLEVR; verbose paraphrasing reduces drift variance by 70--81% on the 8B models. The practical recipe---pad the prompt---further yields measurable gains in accuracy, even under image corruption.
Farooq Ahmad Wani, Maria Sofia Bucarelli, Mujtaba Hussain Mirza +5
Sapienza University of Rome · Université Côte d’Azur · CNRS +3
Vision-language models (VLMs) often fail to match a point or an object across two images, even when the answer can be decoded from their own hidden states. We trace this failure from the visual tokens to the answer letter, in Qwen3-VL (2B-32B), InternVL3.5-8B, and Gemma-3-12B, on synthetic shape matching, same-instance point correspondence (HPatches), and semantic correspondence (SPair-71k), and find two distinct failure points, a small set of attention heads that carries the comparison, and a repair that touches 0.02% of the parameters. Two failure points: exact copies are compared but not read out. An answer-position probe exceeds the calibrated output by 15 to 25 points in every family, for nameable and unnameable shapes alike, and steering along the probe's direction moves no more answers than random directions. Semantic correspondences are present in the visual tokens but never reach the answer. One pathway: redistributing the cross-image attention of 16-32 heads, with its mass preserved, removes 17-22 points of synthetic and 15-32 points of HPatches accuracy, against at most 7 for random or attention-matched heads. A lever that training finds on its own: training only these heads' query and key projections raises SPair-71k accuracy by 28.3 points (Qwen3-VL-8B) and 25.1 points (InternVL3.5-8B), beats both layer-matched and attention-matched heads, and transfers to HPatches and BLINK. An unconstrained all-layer adapter routes 21-51% of its gain through the attention of the same heads, against at most 7% for random heads, and does so by changing what they read (83-86% of the effect) rather than their weights. Every decision rule was frozen before its run, and every reported number is recomputed by independent verification scripts.