Keel: A Frontier Model Teaches a Vision-Language Model by Rewriting Its Weights
Organizations: UIUC · Yale · NYU · CMU
Abstract
Can a frontier model repair a vision-language model (VLM) by rewriting its weights? Small reasoning VLMs such as Qwen3.5 need such a repair: in thinking mode they fall into loops on many image questions, repeating themselves until the token limit, spending the whole budget and losing the answer. The usual remedies act on the decoding, leaving the looping model as it was, or must be told what to change. Yet the cause is written in the weights. Keel, the first agentic weight-editing method for VLMs on image tasks, finds and removes it there. Told only the symptom, an editor, a frontier multimodal LLM acting as an agent, locates the cause, edits the weights and tests them, with a failure census and instruments showing it the failing images and where the VLM attends; among several editors, the harness keeps only edits that clearly reduce the failure without lowering accuracy on questions no editor saw. The repair needs no training, keeps the model's size and per-token cost, and applies to already-edited weights. The editors expose where the loops live: from the original weights, independent editors on both models trace them to the attention outputs of the middle layers, and the smaller VLM stops consulting the image inside a loop. On Qwen3.5-0.8B and Qwen3.5-2B, across six image benchmarks from visual mathematics to GUI grounding, the edit removes most loops on every benchmark and shortens answers, and the edited models are significantly more accurate than the thinking models on average; the larger one gains most on GUI grounding, a task absent from the practice questions. Because the repair lives in the weights, it combines with budget forcing where held-out questions favor it: on the best available weights, Keel has the best average accuracy on both models, significantly above every training-free baseline we compare on the larger one.
Figures & tables
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Editor | Questions | Accuracy (%) | Cut (%) | Cut in a loop (%) |
| Qwen3.5-0.8B | 7 (Keel edit) | 1,536 | 36.0 39.1 | 48.3 39.8 | 37.8 5.9 |
| Qwen3.5-2B | 1 (in the union) | 1,280 | 64.3 67.7 | 25.8 22.2 | 17.1 5.4 |
| Qwen3.5-2B | 2 (in the union) | 2,560 | 63.9 67.0 | 26.4 22.4 | 16.3 9.6 |
| Setting | Value |
| Editor and harness | |
| Editor | GPT-6 Astra, maximal reasoning effort |
| Forward-pass budget per session | 40 (protocol (a): 25) |
| Turns per conversation before a restart | 40 |
| Session end | budget used, or 260 turns, or 14 h |
| Carried over at a restart | last 12,000 characters of notes |
| Per editing run (mean over 12 runs) | |
| Editor turns | 104 |
| Forward passes | 40 |
| Editor input tokens (mostly cached) | 3.9M |
| Editor output tokens | 168k |
| Selection passes per edit | 1 |
| Session | Turns | Code runs | Forward passes | Conversations | Hours | Editor input | Editor output |
| Qwen3.5-0.8B, editor 7 | 93 | 52 | 40 | 3 | 7.6 | 3.45M | 155k |
| Qwen3.5-2B, editor 1 | 125 | 77 | 40 | 4 | 10.4 | 5.92M | 236k |
| Qwen3.5-2B, editor 2 | 105 | 61 | 40 | 3 | 11.8 | 3.73M | 168k |
| Model | Comparison | Avg. diff. | 95% interval |
| Qwen3.5-0.8B | Keel vs. Thinking | +7.92 ∗ | [+6.80, +9.05] |
| Qwen3.5-0.8B | Keel vs. Presence penalty | +3.36 ∗ | [+2.25, +4.49] |
| Qwen3.5-0.8B | Keel vs. Reflection suppression | +3.72 ∗ | [+2.60, +4.83] |
| Qwen3.5-0.8B | Keel vs. Budget forcing | +2.56 ∗ | [+1.41, +3.70] |
| Qwen3.5-0.8B | Keel vs. Directional ablation | +3.24 ∗ | [+2.12, +4.37] |
| Qwen3.5-0.8B | Keel vs. Prior edit [ 45 ] | +1.07 | [ 0.002, +2.13] |
| Model | MathVista | MathVerse | MMStar | HallBench | POPE | ScreenSpot |
| Qwen3.5-0.8B | +3.7 ∗ | +2.9 | +4.9 ∗ | +4.0 ∗ | +0.4 | +6.5 ∗ |
| Qwen3.5-2B | +2.0 | +2.5 | +3.2 ∗ | +4.7 ∗ | +1.8 ∗ | +17.1 ∗ |
| Model | Editor | Protocol | Visual | Passes | Weights | Acc (SE) | Cut (SE) | Outcome |
| 0.8B | 1 | (a) | ✓ | 25 | 0.79M | +0.2 (1.5) | 6.2 (1.7) | passes |
| 0.8B | – | (a) | ✓ | 25 | 0 | – | – | no edit |
| 0.8B | – | (a) | ✓ | 25 | 0 | – | – | no edit |
| 0.8B | – | (a) | 25 | 0 | – | – | no edit | |
| 0.8B | – | (a) | 25 | 0 | – | – | no edit | |
| 0.8B | 2 | (b) | ✓ | 40 | 0.26M | +0.9 (1.6) | 7.2 (1.7) | passes |
| Diagnosis | Visual | Select. | Avg. | Loops | |
| Thinking | 48.2 | 26.1 | |||
| w/o diagnosis | ✓ | ✓ | 51.3 | 11.6 | |
| w/o visual instruments | ✓ | ✓ | 48.2 | 26.1 | |
| w/o selection | ✓ | ✓ | 51.8 | 6.3 | |
| Keel edit | ✓ | ✓ | ✓ | 51.9 | 4.3 |
| Variant minus Keel edit | Avg. accuracy | Loops |
| w/o diagnosis | 0.58 [ 1.70, +0.54] | +7.27 ∗ [+6.46, +8.10] |
| w/o visual instruments | 3.73 ∗ [ 4.81, 2.68] | +21.84 ∗ [+20.80, +22.85] |
| w/o selection | 0.11 [ 0.76, +0.53] | +2.02 ∗ [+1.63, +2.42] |
| Method | Avg. |
| Thinking | 48.2 |
| Keel edit (one round) | 51.9 |
| Keel edit (two rounds) | 53.4 |
| Model | Method | MathVista | MathVerse | MMStar | HallBench | POPE | ScreenSpot |
| Qwen3.5-0.8B | Budget forcing | 29 | 56 | 25 | 13 | 3 | 47 |
| Qwen3.5-0.8B | Prior edit + budget forcing | 11 | 31 | 6 | 1 | 0 | 0 |
| Qwen3.5-0.8B | Keel + budget forcing | 6 | 17 | 3 | 0 | 0 | 0 |
| Qwen3.5-2B | Budget forcing | 17 | 37 | 17 | 9 | 4 | 25 |
| Qwen3.5-2B | Keel | 10 | 33 | 11 | 4 | 1 | 5 |
| Model | Comparison | MathVista | MathVerse | MMStar | HallBench | POPE | ScreenSpot | Pooled [95% interval] |
| Mean generated tokens per answer (thousands) | ||||||||
| Qwen3.5-0.8B | Keel edit vs. Thinking | 0.40 | 0.41 | 0.92 ∗ | 0.88 ∗ | 0.23 ∗ | 4.03 ∗ | 1.20 ∗ [ 1.33, 1.06] |
| Qwen3.5-0.8B | Keel edit + budget forcing vs. Budget forcing | 0.62 ∗ | 0.27 | 0.83 ∗ | 0.86 ∗ | 0.34 ∗ | 3.16 ∗ | 1.06 ∗ [ 1.19, 0.92] |
| Qwen3.5-0.8B | Keel vs. Prior edit + budget forcing | 1.42 ∗ | 1.99 ∗ | 1.23 ∗ | 0.72 ∗ | 0.04 ∗ | 1.06 ∗ | 0.99 ∗ [ 1.06, 0.91] |
| Qwen3.5-2B | Keel edit vs. Thinking | 0.31 | 0.22 | 0.45 ∗ | 0.61 ∗ | 0.19 ∗ | 3.01 ∗ | 0.84 ∗ [ 0.95, 0.72] |
| Qwen3.5-2B | Keel vs. Budget forcing | 0.53 ∗ | 0.50 ∗ | 0.47 ∗ | 0.52 ∗ | 0.41 ∗ | 2.35 ∗ | 0.82 ∗ [ 0.93, 0.70] |
| Thinking: correct | Thinking: wrong | Thinking: cut | |||||||||
| Model | Benchmark | correct | wrong | cut | correct | wrong | cut | correct | wrong | cut | |
| Qwen3.5-0.8B | MathVista | 1,000 | 423 | 56 | 27 | 56 | 136 | 20 | 64 | 72 | 146 |
| MathVerse | 788 | 107 | 34 | 27 | 21 | 102 | 48 | 63 | 93 | 293 | |
| MMStar | 1,500 | 597 | 84 | 28 | 90 | 276 | 21 | 95 | 140 | 169 | |
| HallusionBench | 951 | 491 | 82 | 8 | 65 | 174 | 6 | 63 | 48 | 14 | |
| POPE | 1,500 | 1,244 | 52 | 1 | 41 | 133 | 0 | 18 | 11 | 0 | |
VLM Reasoning
Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.
See all 875 papers