Computational coarse-graining aims to replace a sequence of updates with a cheaper module while preserving task performance. We present a continuation-relative evaluation framework that assesses replacements through the computation that follows them, using downstream task quality and total execution cost as joint criteria. The framework distinguishes boundary representations, summaries of skipped computation, executed operations, and downstream loss. It makes interfaces and permitted deployment information explicit, combines fine-path references with directly trained controls using the same shortened architecture, and accounts for selection overhead. An exact loss decomposition separates replacement effects at a fixed downstream action from effects of changing that action. A finite construction shows how retained information can reduce the stipulated cost of zero-error prediction under fixed information and controller restrictions. Experiments in two frozen pretrained language models show that representation fidelity and task loss can favor different fitting objectives. In a prespecified comparison on a 1.7-billion-parameter model, next-token-loss training lowers in-distribution negative log-likelihood by 0.1901 nats per scored token compared with reconstructing the skipped blocks' output, using the same optimizer and update count. This ranking reverses in all nine paired comparisons under distribution shift. In a recursive expert prototype, the jointly adapted candidate lowers task error relative to frozen-bank and router-only baselines, but trails directly trained controls, and the tested input-only selectors fail the joint quality-cost criteria. Together, the protocol and studies provide explicit criteria and diagnostics for distinguishing useful computational replacement from local approximation gains within specified interfaces and evaluation settings.
Figures & tables
Figure 1: Evaluate a replacement through its continuation and complete execution cost. (A) The recursive prototype illustrates the shared interface. Boundary states are zF,zM ; qF is the fine interval’s measured occupancy QI , and qM is the macro’s prediction without executing those updates. Both paths supply state and occupancy to the same router. Its action a selects an expert, which receives the state and feeds its output to the task head; occupancy is used only by the router. An input-only path selector, distinct from this expert router, can choose between the frozen complete paths. (B) The evaluation protocol compares task quality with both fine references when paths are jointly adapted, includes direct training of the same reduced graph, and charges selector prediction, example partitioning, and dispatch along with path execution. The diagram specifies comparisons, not a guarantee of improvement.
Method
Trainable parts
Objective and comparison role
JOINT-END
all
Base objective; boundary-state-supervised baseline.
SIG
all
Base plus occupancy anchoring and state/occupancy composition.
RESP-ONLY
all
Base plus response matching, weighted uniformly over actions.
SHORTCUT-CONT
all
Base plus 0.1 state-composition MSE and 0.1 MSE between each path’s selected output.
RELEVANT
all
Base plus anchoring, composition, and relevance-weighted response matching; development-selected candidate.
ROUTER-RELEVANT
macro and routers
Full RELEVANT objective; restricted router adaptation.
Table 1: Controls distinguish objectives, trainable components, and budgets. The base objective combines task loss, boundary-state matching, and load balancing. DIRECT trains the reduced graph from initialization; fine-only paths provide quality references.
Figure 2: Boundary fidelity and task loss rank fitting objectives differently. Rows are replacement boundaries within one GPT-2 checkpoint. Panels show (A) WikiText-2 test boundary MSE, (B) WikiText-2 test NLL, and (C) LAMBADA NLL; lower is better. Filled marks are means. FID-LS and FID-SGD have one run; CONT-ZERO has two except at (4,4) (seed 1 only). Open circles show individual CONT-ZERO runs, not confidence intervals. The descriptive FID-SGD comparison matches Adam updates and sampled positions, not backward cost. Table S3 reports all six methods; the numerical supplement retains every metric.
Figure 3: The Qwen3 objective ranking reverses under distribution shift. Each panel reports CONT-ZERO minus FID-SGD NLL at the same three replacement boundaries and three paired training-order seeds. The common horizontal scale is in nats per scored token; negative values favor CONT-ZERO. Each mark is a paired run difference, giving nine marks per distribution from one pretrained model. Shapes identify seeds; vertical offsets separate marks. No confidence bars are shown. All IID differences are negative and all LAMBADA differences positive. NLL for all six methods and fidelity–task comparisons are in tables S4 and S3 ; every metric is preserved in the numerical supplement ( appendix E ).
Figure 4: Complete controls and the measured quality–work trade-off. (A) IID MSE for each of five parents and their mean for each fixed path; lower is better. (B) Candidate fixed paths, selector operating points, and strong DIRECT controls, averaged over the same five parents. The horizontal limit is 1.02 times the candidate’s own-fine mean MSE: a necessary quality condition, not the complete quality–latency test. The plus mark is JOINT-END with its MLP selector at the development-chosen threshold. Selector thresholds are fixed on development data, without retuning quantiles on test data. Affine MACs exclude backward-pass work and non-affine deployment overhead, so the curves do not measure latency. No test-label oracle or population confidence band is shown. Label key: ADAPTED-FINE = continued fine-only; REFERENCE = pretrained fine reference; DEV = development split (DEV-chosen point: the threshold chosen on development data; DEV-frozen curve: thresholds fixed on development data).
Figure 5: Loss-gap decomposition and auxiliary-objective effects in the original synthetic family. (A) Original confirmation fits on IID data: fixed-action, reallocation, and total continuation-loss gaps in 10−3 MSE units. Positive components increase the macro-minus-fine loss gap. Large filled marks are arithmetic means; small open marks show the five parents. Diamonds identify parent 3413, whose negative reallocation determines the negative JOINT-END and SHORTCUT-CONT means. The decomposition describes frozen models and does not identify training effects. DIRECT has no corresponding fine path to decompose. (B) Separate exploratory single-term study with 1,200 fine-pretraining updates: percentage change in IID macro MSE relative to paired JOINT-END fits. Open marks show the five paired parent ratios; filled marks show geometric-mean changes. Bars are descriptive paired 95%t -intervals computed on log ratios and back-transformed. These fits are not pooled with panel A. The occupancy-anchoring interval permits material effects in either direction. Full results and protocols are in sections C.5 and C.9 .
Selector
MSE/fine ≤1.02
MAC saving ≥10%
Latency/fine ≤0.90
MLP (primary)
1.0870
9.71%
1.2378
Linear (control)
1.1511
19.73%
0.8968
Table 2: Neither input-only selector meets the joint quality–cost requirements. IID test results over five parents, relative to the candidate’s own fine path. Both MSE ratios exceed 1.02 and therefore fail the quality rule, which requires meeting the tolerance against both fine references. Column headers give the required thresholds: smaller MSE and latency ratios and larger MAC savings are better. Latency includes selector evaluation, partitioning, and dispatch; affine MAC savings include selector work but exclude non-affine overhead. Ratios use means across parents, with median latency measured per parent.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
c ; s,Δ
external context; start and length of the computation interval
zs,bs
visible state; optional history state needed for closure
E;πs;αs;ds
expert count; soft responsibilities; hard activations; dropped mass
Ψs,Δ;Φθ
fine teacher transition; learned coarse state map
AI,QI
unnormalized route mass; normalized occupancy
BI,CI,ℓI,rI
mass and frequency of adjacent routing pairs; first and last route
Appendix
Table S1: Notation for the routing formulation in sections A.1 and A.2 . Notation for continuation actions and losses is defined separately in section A.4 .
Original
Quadrant
Iterated
Arm or fine reference
IID
Shift
IID
Shift
IID
Shift
Pretrained fine
0.014432
0.020641
0.093523
0.101661
0.024047
0.037823
Continued fine-only
0.009712
0.012471
0.060141
0.067100
0.011832
0.016615
Candidate’s own fine
0.006445
0.010635
0.066664
0.070645
0.012356
0.018408
JOINT-END
0.007917
0.012174
0.089037
0.093989
0.017629
0.032582
SIG
0.009520
0.014035
0.096632
0.099983
0.018670
0.035291
Appendix
Table S2: Direct training of the same fast architecture gives the strongest displayed control. Complete fixed-path MSE for the original confirmation and first two post-hoc replications, averaged over five parent seeds per family (lower is better). DIRECT-COMPUTE has the lowest mean MSE in each displayed family and split. The candidate’s own fine path and continued fine-only model are the quality references; the other controls compare training methods for the fast architecture. Fast paths cost 2,840 forward-affine MACs per example, fine paths 5,176; all three families retain the declared arms and budgets. Shift denotes range shift.
Figure S1: The quality failure persists; auxiliary effects vary by family. Each family uses five parent seeds; IID and range shift are shown separately. (A) Candidate macro MSE divided by own-fine MSE, using ratios of parent means. The 1.02 line is the own-fine tolerance, a necessary part of the quality rule against both fine references, not the complete acceptance test. (B) Geometric-mean change in macro MSE associated with factor A under the paired contrasts in section C.5 ; positive values indicate higher error. Factor A combines occupancy anchoring and self-composition and partly reflects the difference in response weighting, so it does not isolate fidelity’s causal effect. These diagnostics cover one confirmation family, two replication families, and three families used to search for a stronger fine path. They are not six independent real domains or 60 independent training replications. The numerical supplement reports all parent counts and comparisons with named controls.
Figure S2: Longer fine pretraining does not meet the replacement-quality criterion. (A) IID MSE: small marks show the five reused parents; connected large marks are arithmetic means. DIRECT-COMPUTE matches each budget’s pretraining-plus-adaptation forward-affine work. (B) Ratios of macro to own-fine mean MSE on each split. All exceed the 1.02 own-fine limit, failing a necessary part of the two-reference quality rule. Adaptation remains at 1,200 updates. This exploratory comparison uses the rerun 1× condition, not historical confirmation. Lines connect observations without confidence bands or an interpolation guarantee.
A: WikiText-2 (IID)
Arm
(6,2)
(4,2)
(8,2)
(4,4)
(6,4)
DROP
3.879
4.015
3.914
4.910
5.010
FID-LS
3.640
3.670
3.659
4.064
4.075
FID-SGD
3.636
3.661
3.659
4.070
4.086
CONT-ZERO
3.332
3.322
3.354
3.604
3.672
CONT-LS
3.326
3.322
3.341
3.582
3.638
Appendix
Table S3: The IID preference for continuation fitting reverses under shift. GPT-2 task loss for all six arms (NLL in nats per scored token; lower is better), with boundaries (l,k) as columns. CONT-ZERO beats FID-LS at every IID boundary, while FID-LS wins every shifted comparison. The other four arms place this contrast alongside the remaining controls. CONT-ZERO, CONT-LS and END-KL average two runs, except CONT-ZERO at (4,4) (seed 1 only); the other arms have one run. The shared, unadapted FULL reference scores 3.444 (WikiText-2) and 3.611 (LAMBADA). Additional metrics are in the numerical supplement.
Arm
WikiText-2 IID
LAMBADA shift
Boundary (7,2)
DROP
2.52875
3.47792
FID-LS
2.38001
3.37497
FID-SGD
2.38988 (0.00127)
3.38201 (0.000867)
CONT-ZERO
2.21467 (0.00331)
3.54198 (0.00715)
CONT-LS
2.18530 (0.00318)
3.53587 (0.0198)
Appendix
Table S4: The primary IID gain does not identify a universally best fitting objective. NLL for every Qwen3 arm in nats per scored token; lower is better. CONT-ZERO beats FID-SGD on IID but loses on LAMBADA; FID-LS and CONT-LS provide stronger controls. Every replacement has higher shifted NLL than the shared unadapted FULL reference. Parentheses give sample SD over three sampling-order seeds; entries without parentheses are single deterministic runs. FULL is not an equally adapted control.
Figure S3: Fidelity and task quality need separate evaluation. All six replacement methods appear on both distributions at all three boundaries. Axes and bars are defined in the figure; SD bars may be smaller than markers. FULL is an unadapted NLL reference, not an equally adapted control. Points share one model; they are not independent model replications, and no causal relationship is fitted. Complete MSE, KL, and NLL values are reported in the numerical supplement ( appendix E ).
Mixture-of-experts (MoE) models activate only a few experts per token but store the entire expert pool. Pruning this pool requires identifying experts whose removal preserves model behavior. Routing frequency and output magnitude do not fully describe deletion damage, which also depends on how the surviving and replacement experts compensate for the removed output. We introduce RAZOR, a training-free pruning method based on consensus residuals, the deviations of expert outputs from their original weighted mixture. At a fixed layer input, these residuals give the exact output change for a single deletion under survivor renormalization and router refill. RAZOR aggregates this damage by conditional root mean square and selects experts under a layerwise budget using forward computation alone, without gradients, subset search, or recovery training. Against frequency, activation-norm, and REAP baselines on GLM-4.7-Flash and Qwen3.6-35B-A3B at 25% and 50% expert removal, it attains the highest macro average over nine reasoning-intensive tasks in all four model-budget settings, gaining 2.12-5.59 points over REAP and lowering reverse KL in all four. On DeepSeek-V4-Flash-0731 and Hy3, it also achieves the highest macro average among the three residual criteria. Local exactness does not guarantee better joint pruning. Generation analyses show changes in diversity, formatting, and termination despite higher task scores.
Fine-grained mixture-of-experts (MoE) architectures have become a mainstream design for open-weight LLMs, with hundreds of experts and increasingly many selected per token. This shift makes dynamic expert pruning an attractive route to cheaper inference. Yet existing evidence comes largely from coarser architectures and likelihood-scored multiple-choice benchmarks, leaving three central questions open in the fine-grained regime: how redundant per-token expert selection is, how effectively existing pruning methods exploit that redundancy, and what governs a model's sensitivity to pruning. We fill this gap with a systematic empirical study of twelve fine-grained MoE checkpoints spanning nine architecture families, with a core suite of eleven benchmarks covering knowledge QA, mathematics, code generation, and general reasoning. We find that expert selection is far more redundant than the field's operating points assume: uniformly retaining about two thirds of the selected experts preserves 98.8% of unpruned performance on average, requiring only a one-integer change and delivering 1.2-1.7x measured speedup across two serving backends. This simple baseline leaves little room for dynamic allocation at conservative budgets: even the best published rules differ from it by under 1% at matched expert budgets. Their value emerges under aggressive pruning, where the best rules recover up to 3.0% over uniform truncation, with gains concentrated in the generative tasks that suffer the sharpest degradation. Sensitivity to aggressive pruning also depends on the model: larger and thinking models are more resilient, whereas multimodal models are more vulnerable. Together, these findings reveal how much expert computation fine-grained MoEs can dispense with, and establish when dynamic allocation earns its complexity, informing both practical deployment and future pruning methods.
Yuanteng Chen, Qiwei Lai, Chen Tianqi +7
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +1
Evaluator-guided reasoning presents a local control problem at every step: trust the current answer, return a fallback, or spend more inference. We formulate this choice as a compute-aware three-action controller, scoring terminal actions by predicted answer success and expansion by a rollout-based continuation-success surrogate, with compute cost priced explicitly. A local action-gap analysis characterizes when score errors can change a decision, while an expected regret bound accounts for errors along the realized expansion path. Guided by this analysis, we learn and calibrate action-specific confidence estimates, with local uncertainty penalties for ambiguous decisions. On held-out width expansion and revision tasks, the controller improves observed utility over calibration-selected fixed policies while reducing normalized online compute by about 30% and 16%, respectively. Across four Qwen3-8B seeds, compute is lower than fitted Q in both settings, while utility differences are small and mixed. The results frame evaluator-guided reasoning as resource allocation under uncertainty, rather than a uniformly stronger reasoner.
Weida Liang, Kenji Kawaguchi
School of Computing, National University of Singapore