Reasoning Again with a Quarter of the Experts: Retry First, Then Train
Organizations: Independent Researcher
Abstract
Removing three quarters of the routed experts from a 122B-parameter reasoning mixture-of-experts model makes it fit in a 12 GiB file, and breaks it in a specific way: on LiveCodeBench it enters verbatim reasoning loops that run to the token limit on 44 of 120 development problems, where the unpruned model at the same 2-bit quantization loops on none. We study how much of that loss can be recovered without changing the kept experts' quantized bytes, in three steps of increasing cost. On development problems, a sampling penalty recovers part of it. Stopping a reply when it starts to loop and drawing again recovers more on top of it, with no weight change. Our method, B, adds rank-8 trained corrections to the kept experts, loaded as an adapter, and restores two experts per layer; it is fitted layer by layer to the unpruned model's expert block on the pruned model's own activations. In a pre-registered comparison on 130 easy and medium LiveCodeBench problems that no part of the study had touched, with both arms given the same sampling penalty and the same stop-and-redraw policy, B solves 116 problems against 104 (+9.2 pp, 95% CI [+3.1, +15.4]; 15 problems gained, 3 lost). A training-free weight edit aimed at the loops, tested the same way earlier, did not help. B removes more wrong answers than loops, and costs 1.7x the wall time. On two older benchmarks used as regression checks, B stays inside a pre-set 2 pp tolerance but shows no gain (HumanEval+ +0.6 pp, MBPP+ -1.9 pp). The evidence is one model, one quantization and one benchmark family; we state what it does and does not show.
Figures & tables
| Model | Solved | Hit the limit | Limit, looping |
|---|---|---|---|
| Pruned to 25%, 2-bit ( C ) | 71 | 44 | 44 |
| Unpruned, 2-bit | 107 | 11 | 0 |
| Unpruned, 4-bit | 110 | 8 | 0 |
| Arm | Solved | Hit the limit |
|---|---|---|
| C | 74 | 45 |
| C + weight edit | 68 | 47 |
| C + S | 80 | 32 |
| Problems | S +retry | B + S +retry | Difference | Gained / lost | McNemar | |
|---|---|---|---|---|---|---|
| Development (reused) | 120 | 88 | 101 | +10.8 [+3.3, +18.3] | 18 / 5 | 0.011 |
| Confirmation, seed 1 | 64 | 55 | 58 | +4.7 [ 1.6, +12.5] | 4 / 1 | 0.375 |
| Confirmation, seed 2 | 64 | 52 | 57 | +7.8 [ 1.6, +17.2] | 7 / 2 | 0.180 |
| Confirmation, pooled | 64 | +6.2 [+0.8, +13.3] | ||||
| Final (untouched) | 130 | 104 | 116 | +9.2 [+3.1, +15.4] | 15 / 3 | 0.007 |
Mixture-of-Experts Pruning
Mixture-of-Experts pruning removes or trims experts to reduce model memory and computation while preserving useful capabilities.
See all 68 papers