Tree-Ising: Repairing Language Models with Bit-Level Ising Edits
Organizations: QDeep, Riyadh, Saudi Arabia · LEAD (Lab for Efficient AI and Data), ITMO University
Abstract
Weight-space programming, introduced in recent MIT work by Takehi et al., enables a frontier LLM agent to read a network's weights and write a repair. Their GPT-6 Astra agent corrected the reasoning loops of Qwen3.5-0.8B without gradients, using 158 agent turns, 89.7 million tokens, and 50 forward-pass requests. Our method, Tree-Ising, casts this repair as a sparse discrete optimization problem over the model's own BF16 weights. Forward statistics define an objective that weakens a loop direction, strengthens a concluding direction, and preserves the layer's output on productive tokens. Within a state-tree representation, where lossless BF16 storage acts as a special case, the admissible edits are sign flips and exponent shifts that preserve all shared group exponents and mantissas. We decompose this objective exactly into independent QUBO (Ising) problems, one per output row, which we solve on a GPU using simulated annealing or quantum-inspired simulated bifurcation. On the agent's two layers, Tree-Ising achieves 54.6% on MATH-500, outperforming the agent's edit (51.7%) under the same evaluation protocol, while the base model scores 7.9% as reported. The simulated bifurcation solver yields a statistically indistinguishable repair at 55.1%. Both solvers outperform the agent edit across all five math benchmarks, with statistically significant improvements on MATH-500 and GSM8K after correcting for multiple comparisons. Our approach modifies 13% to 17% fewer weights and reduces BF16 bit flips by a factor of 2.2 to 2.3. It requires only four forward-pass requests and zero LLM tokens, with the annealing recipe costing about $0.90 in GPU time. On Qwen3.5-2B, with layers and dose selected using only the base model, the same recipe increases greedy GSM8K accuracy from 21.6% to 69.4%. As a proof of concept, we execute an eight-variable repair subproblem using QAOA on an IBM Heron processor.
Figures & tables
| Agent edit [ 55 ] | Tree-Ising (ours) | Tree-Ising advantage | ||
| Quality | MATH-500 accuracy (%) | 51.7 | 54.6 | points, |
| Benchmarks ahead (of 5) | – | 5 | 2 significant after correction | |
| MATH-500 answers cut at cap (%) | 31.6 | 23.9 | 1.3 fewer | |
| Cost | Forward-pass requests | 50 | 4 | 12.5 fewer |
| Forward-pass GPU time | 12.2 h | 21.8 min | 34 less | |
| LLM tokens | 89.7M | 0 | none needed |
| Set | Role | Definition |
| loop | In a looping trace, find the first 256-token window in which at least 80% of 8-grams repeat, after at least 300 generated tokens. In the next 1,000 positions, take the final token of every repeated 8-gram. | |
| protected | The 2,000 to 50 tokens before each loop onset, and every generated token of correct completed traces. | |
| concluding | The last 64 thinking tokens before </think> in a correct trace with at least 200 thinking tokens. | |
| continuing | The earlier thinking tokens of the same correct traces. |
| MATH-500@1 | MATH-500@4 | GSM8K@1 | AMC23@16 | Minerva@4 | OlympiadBench@1 | |
| Base model, reported | – | 7.9 | 13.8 | 0.5 | 1.5 | 1.3 |
| Agent edit, reported | – | 48.1 | 67.1 | 23.4 | 25.4 | 21.8 |
| Agent edit, our harness | 51.2 | 51.7 | 65.6 | 22.0 | 24.2 | 21.4 |
| Tree-Ising (SA) | 54.0 | 54.6 | 70.5 | 22.7 | 26.1 | 24.3 |
| Tree-Ising (SB) | 54.4 | 55.1 | 70.8 | 25.0 | 25.6 | 24.0 |
| Cut at cap (%) | ||||||
| Agent Edit | Tree-Ising | |
| Search procedure | LLM agent (GPT-6 Astra, maximal reasoning effort) | Simulated annealing on 2,048 row QUBOs |
| Agent turns and tokens | 158 turns, 89.7M tokens (80.3M cached input, 2.9M output) | None |
| Gradients | None | None |
| Forward passes | 50 | 4 (one generation, two statistics passes, one screen) |
| Model GPU time | 12.2 h | 21.8 min |
| Search time | 1.8 h of agent responses | 8.8 min of GPU annealing |
| Changed Weights | BF16 Bit Flips | State-Tree Bit Flips | Sign Flips | State Changes | Shared Exponents | Mantissa Changes | |
| Agent Edit | 130,188 | 621,388 | 720,586 | 19,299 | 146,981 | 1,261 | 129,668 |
| Tree-Ising (SA) | 112,630 | 278,767 | 181,062 | 57,704 | 99,952 | 0 | 0 |
| Tree-Ising (SB) | 108,327 | 268,090 | 174,245 | 55,666 | 96,007 | 0 | 0 |
| Variant | Change | Accuracy (%) | Cut at cap (%) | Changed Weights | BF16 Bit Flips |
| Tree-Ising | Selected configuration | 57.0 | 22.7 | 112.6k | 279k |
| Loop only | 43.8 | 53.1 | 112.3k | 277k | |
| Weaker conclude term | 53.1 | 28.1 | 115.4k | 285k | |
| Stronger conclude term | 53.9 | 21.1 | 93.2k | 232k | |
| Lower dose | 57.8 | 16.4 | 77.4k | 193k | |
| Higher dose | 54.7 | 26.6 | 118.5k | 301k |
| Model | Layers | Dose | Accuracy (%) | Cut at cap (%) | Only edit / only base / both | Changed weights | State-tree bit flips |
| Qwen3.5-2B | – | – | 21.6 | 77.9 | – | – | – |
| Tree-Ising recipe | 15, 19 | 69.4 | 26.7 | 669 / 38 / 247 | 292,845 | 473,524 | |
| Tree-Ising , stronger dose | 14, 15, 19 | 71.8 | 24.4 | 687 / 25 / 260 | 482,423 | 788,492 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Qwen3.5-0.8B ( Section 4 ) | Qwen3.5-2B ( Section 5 ) |
| Edit configuration | ||
| Target tensors | down_proj , layers 15 and 19 | down_proj , layers 15 and 19. The stronger variant also includes layer 14 |
| Tensor dimensions | per layer | per layer |
| Layer selection | Agent-selected layers, corroborated by a forward scan | Forward scan, excluding the final layer |
| Candidate neurons | of 3,584, with the same candidate columns across all rows within a layer | of 6,144 per row |
| Allowed moves | Eight multiplicative factors from Equation 1 | Same |
| Model | Layer | Own write | Agent dose | Target dose | Achieved dose | Achieved / own | attempts | Changed weights |
| Qwen3.5-0.8B | 15 | 0.328 | 6.8 | 1 | 51,754 | |||
| Qwen3.5-0.8B | 19 | 0.498 | 6.2 | 1 | 60,876 | |||
| Qwen3.5-0.8B, SB | 15 | 0.328 | 6.7 | 1 | 48,860 | |||
| Qwen3.5-0.8B, SB | 19 | 0.498 | 6.2 | 1 | 59,467 | |||
| Qwen3.5-2B | 15 | 1.035 | – | 6.6 | 1 | 111,211 | ||
| Qwen3.5-2B | 19 | 1.755 | – | 6.6 | 2 | 181,634 |
| Backend | ibm_fez , IBM Heron r2, 156 fixed-frequency transmon qubits with tunable couplers and heavy-hexagonal connectivity [ 24 , 26 , 25 ] |
| Access | IBM Quantum Platform Open Plan, us-east , job mode, using Qiskit 2.6.2 and IBM runtime client 0.51.0 |
| Physical qubits | 121, 122, 123, 124, 125, 136, 142, 143 |
| Problem | 8 binary variables and 28 nonzero logical couplings |
| Shots | 1,024 per circuit, with six circuits (three QAOA depths and their controls) submitted in one job |
| Compiled QAOA circuits | has 107 CZ gates and depth 211, has 220 CZ gates and depth 387, and has 350 CZ gates and depth 617 |
| Reference device graph | conf_fez.json from qiskit-ibm-runtime 0.51.0, backend version 1.2.8, with 176 undirected connections [ 23 ] |
| Variable | ||||||||
| Circuit entry | 123 | 124 | 122 | 136 | 125 | 121 | 143 | 142 |
| Readout, | 142 | 143 | 121 | 122 | 136 | 123 | 125 | 124 |
| Readout, | 125 | 121 | 124 | 122 | 136 | 123 | 142 | 143 |
| Readout, | 142 | 143 | 121 | 122 | 136 | 125 | 123 | 124 |
| Cost angles | Mixer angles | Evaluations | Time (s) | |
| 1 | (0.474617) | (2.772206) | 769 | 0.115 |
| 2 | (0.462255, 0.650097) | (2.599518, 2.662857) | 1,975 | 0.430 |
| 3 | (0.342703, 0.641411, 0.645390) | (2.654418, 2.681259, 2.847049) | 3,496 | 1.011 |
| Circuit | Native CZ | Optimum hits | Optimum rate (%) [95% interval] | Ideal rate (%) | Mean energy SE |
| QAOA, | 107 | 33 | 3.22 [2.30, 4.49] | 2.31 | |
| QAOA, | 220 | 110 | 10.74 [8.99, 12.79] | 20.33 | |
| QAOA, | 350 | 55 | 5.37 [4.15, 6.93] | 37.06 | |
| Uniform control, | 0 | 4 | 0.39 [0.15, 1.00] | 0.39 | |
| Uniform control, | 0 | 4 | 0.39 [0.15, 1.00] | 0.39 | |
| Uniform control, | 0 | 7 | 0.68 [0.33, 1.40] | 0.39 |
Language Model Editing
Language model editing changes a model’s knowledge or behavior, or revises its outputs, while aiming to preserve unrelated capabilities.
See all 90 papers