Coarse-Graining Recursive Expert Computation: Continuation-Relative Interfaces and Failure Modes

Shihua Yu; Yanzhi Wang; Tianchi Chen

cs.LGOct 10, 2026

Coarse-Graining Recursive Expert Computation: Continuation-Relative Interfaces and Failure Modes

Authors: Shihua Yu, Yanzhi Wang, Tianchi Chen

Organizations: Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, USA. · Merck & Co., Inc., Rahway, NJ, USA.

Abstract

Computational coarse-graining aims to replace a sequence of updates with a cheaper module while preserving task performance. We present a continuation-relative evaluation framework that assesses replacements through the computation that follows them, using downstream task quality and total execution cost as joint criteria. The framework distinguishes boundary representations, summaries of skipped computation, executed operations, and downstream loss. It makes interfaces and permitted deployment information explicit, combines fine-path references with directly trained controls using the same shortened architecture, and accounts for selection overhead. An exact loss decomposition separates replacement effects at a fixed downstream action from effects of changing that action. A finite construction shows how retained information can reduce the stipulated cost of zero-error prediction under fixed information and controller restrictions. Experiments in two frozen pretrained language models show that representation fidelity and task loss can favor different fitting objectives. In a prespecified comparison on a 1.7-billion-parameter model, next-token-loss training lowers in-distribution negative log-likelihood by 0.1901 nats per scored token compared with reconstructing the skipped blocks' output, using the same optimizer and update count. This ranking reverses in all nine paired comparisons under distribution shift. In a recursive expert prototype, the jointly adapted candidate lowers task error relative to frozen-bank and router-only baselines, but trails directly trained controls, and the tested input-only selectors fail the joint quality-cost criteria. Together, the protocol and studies provide explicit criteria and diagnostics for distinguishing useful computational replacement from local approximation gains within specified interfaces and evaluation settings.

Figures & tables

Panel of Figure 1 — Figure 1: Evaluate a replacement through its continuation and complete execution cost. (A) The recursive prototype illustrates the shared interface. Boundary states are $z_{F},z_{M}$ ; $q_{F}$ is the fine interval’s measured occupancy $Q_{I}$ , and $q_{M}$ is the macro’s prediction without executing those updates. Both paths supply state and occupancy to the same router. Its action $a$ selects an expert, which receives the state and feeds its output to the task head; occupancy is used only by the router. An input-only path selector, distinct from this expert router, can choose between the frozen complete paths. (B) The evaluation protocol compares task quality with both fine references when paths are jointly adapted, includes direct training of the same reduced graph, and charges selector prediction, example partitioning, and dispatch along with path execution. The diagram specifies comparisons, not a guarantee of improvement.

Method	Trainable parts	Objective and comparison role
JOINT-END	all	Base objective; boundary-state-supervised baseline.
SIG	all	Base plus occupancy anchoring and state/occupancy composition.
RESP-ONLY	all	Base plus response matching, weighted uniformly over actions.
SHORTCUT-CONT	all	Base plus $0.1$ state-composition MSE and $0.1$ MSE between each path’s selected output.
RELEVANT	all	Base plus anchoring, composition, and relevance-weighted response matching; development-selected candidate.
ROUTER-RELEVANT	macro and routers	Full RELEVANT objective; restricted router adaptation.

Table 1: Controls distinguish objectives, trainable components, and budgets. The base objective combines task loss, boundary-state matching, and load balancing. DIRECT trains the reduced graph from initialization; fine-only paths provide quality references.

Figure 2: Boundary fidelity and task loss rank fitting objectives differently. Rows are replacement boundaries within one GPT-2 checkpoint. Panels show (A) WikiText-2 test boundary MSE, (B) WikiText-2 test NLL, and (C) LAMBADA NLL; lower is better. Filled marks are means. FID-LS and FID-SGD have one run; CONT-ZERO has two except at

(4,4)

(seed 1 only). Open circles show individual CONT-ZERO runs, not confidence intervals. The descriptive FID-SGD comparison matches Adam updates and sampled positions, not backward cost. Table S3 reports all six methods; the numerical supplement retains every metric.

Figure 3: The Qwen3 objective ranking reverses under distribution shift. Each panel reports CONT-ZERO minus FID-SGD NLL at the same three replacement boundaries and three paired training-order seeds. The common horizontal scale is in nats per scored token; negative values favor CONT-ZERO. Each mark is a paired run difference, giving nine marks per distribution from one pretrained model. Shapes identify seeds; vertical offsets separate marks. No confidence bars are shown. All IID differences are negative and all LAMBADA differences positive. NLL for all six methods and fidelity–task comparisons are in tables S4 and S3 ; every metric is preserved in the numerical supplement ( appendix E ).

Panel of Figure 4 — Figure 4: Complete controls and the measured quality–work trade-off. (A) IID MSE for each of five parents and their mean for each fixed path; lower is better. (B) Candidate fixed paths, selector operating points, and strong DIRECT controls, averaged over the same five parents. The horizontal limit is $1.02$ times the candidate’s own-fine mean MSE: a necessary quality condition, not the complete quality–latency test. The plus mark is JOINT-END with its MLP selector at the development-chosen threshold. Selector thresholds are fixed on development data, without retuning quantiles on test data. Affine MACs exclude backward-pass work and non-affine deployment overhead, so the curves do not measure latency. No test-label oracle or population confidence band is shown. Label key: ADAPTED-FINE = continued fine-only; REFERENCE = pretrained fine reference; DEV = development split (DEV-chosen point: the threshold chosen on development data; DEV-frozen curve: thresholds fixed on development data).

Figure 5: Loss-gap decomposition and auxiliary-objective effects in the original synthetic family. (A) Original confirmation fits on IID data: fixed-action, reallocation, and total continuation-loss gaps in

10^{-3}

MSE units. Positive components increase the macro-minus-fine loss gap. Large filled marks are arithmetic means; small open marks show the five parents. Diamonds identify parent 3413, whose negative reallocation determines the negative JOINT-END and SHORTCUT-CONT means. The decomposition describes frozen models and does not identify training effects. DIRECT has no corresponding fine path to decompose. (B) Separate exploratory single-term study with 1,200 fine-pretraining updates: percentage change in IID macro MSE relative to paired JOINT-END fits. Open marks show the five paired parent ratios; filled marks show geometric-mean changes. Bars are descriptive paired

95\%

t

-intervals computed on log ratios and back-transformed. These fits are not pooled with panel A. The occupancy-anchoring interval permits material effects in either direction. Full results and protocols are in sections C.5 and C.9 .

Selector	MSE/fine $\leq 1.02$	MAC saving $\geq 10\%$	Latency/fine $\leq 0.90$
MLP (primary)	1.0870	$9.71\%$	1.2378
Linear (control)	1.1511	$19.73\%$	0.8968

Table 2: Neither input-only selector meets the joint quality–cost requirements. IID test results over five parents, relative to the candidate’s own fine path. Both MSE ratios exceed

1.02

and therefore fail the quality rule, which requires meeting the tolerance against both fine references. Column headers give the required thresholds: smaller MSE and latency ratios and larger MAC savings are better. Latency includes selector evaluation, partitioning, and dispatch; affine MAC savings include selector work but exclude non-affine overhead. Ratios use means across parents, with median latency measured per parent.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$c$ ; $s,\Delta$	external context; start and length of the computation interval
$z_{s},b_{s}$	visible state; optional history state needed for closure
$E;\pi_{s};\alpha_{s};d_{s}$	expert count; soft responsibilities; hard activations; dropped mass
$\Psi_{s,\Delta};\Phi_{\theta}$	fine teacher transition; learned coarse state map
$A_{I},Q_{I}$	unnormalized route mass; normalized occupancy
$B_{I},C_{I},\ell_{I},r_{I}$	mass and frequency of adjacent routing pairs; first and last route

Appendix

Table S1: Notation for the routing formulation in sections A.1 and A.2 . Notation for continuation actions and losses is defined separately in section A.4 .

	Original		Quadrant		Iterated
Arm or fine reference	IID	Shift	IID	Shift	IID	Shift
Pretrained fine	0.014432	0.020641	0.093523	0.101661	0.024047	0.037823
Continued fine-only	0.009712	0.012471	0.060141	0.067100	0.011832	0.016615
Candidate’s own fine	0.006445	0.010635	0.066664	0.070645	0.012356	0.018408
JOINT-END	0.007917	0.012174	0.089037	0.093989	0.017629	0.032582
SIG	0.009520	0.014035	0.096632	0.099983	0.018670	0.035291

Appendix

Table S2: Direct training of the same fast architecture gives the strongest displayed control. Complete fixed-path MSE for the original confirmation and first two post-hoc replications, averaged over five parent seeds per family (lower is better). DIRECT-COMPUTE has the lowest mean MSE in each displayed family and split. The candidate’s own fine path and continued fine-only model are the quality references; the other controls compare training methods for the fast architecture. Fast paths cost 2,840 forward-affine MACs per example, fine paths 5,176; all three families retain the declared arms and budgets. Shift denotes range shift.

Figure S1: The quality failure persists; auxiliary effects vary by family. Each family uses five parent seeds; IID and range shift are shown separately. (A) Candidate macro MSE divided by own-fine MSE, using ratios of parent means. The

1.02

line is the own-fine tolerance, a necessary part of the quality rule against both fine references, not the complete acceptance test. (B) Geometric-mean change in macro MSE associated with factor A under the paired contrasts in section C.5 ; positive values indicate higher error. Factor A combines occupancy anchoring and self-composition and partly reflects the difference in response weighting, so it does not isolate fidelity’s causal effect. These diagnostics cover one confirmation family, two replication families, and three families used to search for a stronger fine path. They are not six independent real domains or 60 independent training replications. The numerical supplement reports all parent counts and comparisons with named controls.

Figure S2: Longer fine pretraining does not meet the replacement-quality criterion. (A) IID MSE: small marks show the five reused parents; connected large marks are arithmetic means. DIRECT-COMPUTE matches each budget’s pretraining-plus-adaptation forward-affine work. (B) Ratios of macro to own-fine mean MSE on each split. All exceed the

1.02

own-fine limit, failing a necessary part of the two-reference quality rule. Adaptation remains at 1,200 updates. This exploratory comparison uses the rerun

1\times

condition, not historical confirmation. Lines connect observations without confidence bands or an interpolation guarantee.

A: WikiText-2 (IID)
Arm	$(6,2)$	$(4,2)$	$(8,2)$	$(4,4)$	$(6,4)$
DROP	3.879	4.015	3.914	4.910	5.010
FID-LS	3.640	3.670	3.659	4.064	4.075
FID-SGD	3.636	3.661	3.659	4.070	4.086
CONT-ZERO	3.332	3.322	3.354	3.604	3.672
CONT-LS	3.326	3.322	3.341	3.582	3.638

Appendix

Table S3: The IID preference for continuation fitting reverses under shift. GPT-2 task loss for all six arms (NLL in nats per scored token; lower is better), with boundaries

(l,k)

as columns. CONT-ZERO beats FID-LS at every IID boundary, while FID-LS wins every shifted comparison. The other four arms place this contrast alongside the remaining controls. CONT-ZERO, CONT-LS and END-KL average two runs, except CONT-ZERO at

(4,4)

(seed 1 only); the other arms have one run. The shared, unadapted FULL reference scores

3.444

(WikiText-2) and

3.611

(LAMBADA). Additional metrics are in the numerical supplement.

Boundary $(7,2)$
Arm	WikiText-2 IID	LAMBADA shift
DROP	2.52875	3.47792
FID-LS	2.38001	3.37497
FID-SGD	2.38988 (0.00127)	3.38201 (0.000867)
CONT-ZERO	2.21467 (0.00331)	3.54198 (0.00715)
CONT-LS	2.18530 (0.00318)	3.53587 (0.0198)

Appendix

Table S4: The primary IID gain does not identify a universally best fitting objective. NLL for every Qwen3 arm in nats per scored token; lower is better. CONT-ZERO beats FID-SGD on IID but loses on LAMBADA; FID-LS and CONT-LS provide stronger controls. Every replacement has higher shifted NLL than the shared unadapted FULL reference. Parentheses give sample SD over three sampling-order seeds; entries without parentheses are single deterministic runs. FULL is not an equally adapted control.

Figure S3: Fidelity and task quality need separate evaluation. All six replacement methods appear on both distributions at all three boundaries. Axes and bars are defined in the figure; SD bars may be smaller than markers. FULL is an unadapted NLL reference, not an equally adapted control. Points share one model; they are not independent model replications, and no causal relationship is fitted. Complete MSE, KL, and NLL values are reported in the numerical supplement ( appendix E ).

Efficient Language Model Inference

Efficient language model inference reduces the compute, memory, latency, and energy needed to generate useful outputs.

See all 318 papers

RAZOR: Pruning Replaceable Experts in LLMs
Sep 24, 2026Mingyang Song, Mao ZhengMixture-of-Experts Pruning LLM Pruning
Foundation Model Department, Tencent, China
You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
Sep 22, 2026Yuanteng Chen, Qiwei Lai, Chen Tianqi +7Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +1
Confidence-Calibrated Inference Expansion for Evaluator-Guided Test-Time Reasoning
Oct 14, 2026Weida Liang, Kenji KawaguchiEfficient Language Model Reasoning Inference-Time Compute Allocation
School of Computing, National University of Singapore

Sep 24, 2026J/K move · Enter open · S save

RAZOR: Pruning Replaceable Experts in LLMs

Mingyang Song, Mao Zheng

Foundation Model Department, Tencent, China

Mixture-of-Experts Pruning LLM Pruning Mixture-of-Experts Language Models LLM Compression

Open paper AskPDF arXiv

Mixture-of-experts (MoE) models activate only a few experts per token but store the entire expert pool. Pruning this pool requires identifying experts whose removal preserves model behavior. Routing frequency and output magnitude do not fully describe deletion damage, which also depends on how the surviving and replacement experts compensate for the removed output. We introduce RAZOR, a training-free pruning method based on consensus residuals, the deviations of expert outputs from their original weighted mixture. At a fixed layer input, these residuals give the exact output change for a single deletion under survivor renormalization and router refill. RAZOR aggregates this damage by conditional root mean square and selects experts under a layerwise budget using forward computation alone, without gradients, subset search, or recovery training. Against frequency, activation-norm, and REAP baselines on GLM-4.7-Flash and Qwen3.6-35B-A3B at 25% and 50% expert removal, it attains the highest macro average over nine reasoning-intensive tasks in all four model-budget settings, gaining 2.12-5.59 points over REAP and lowering reverse KL in all four. On DeepSeek-V4-Flash-0731 and Hy3, it also achieves the highest macro average among the three residual criteria. Local exactness does not guarantee better joint pruning. Generation analyses show changes in diversity, formatting, and termination despite higher task scores.

Abstract

Efficient Language Model Inference

Explore similar work

RAZOR: Pruning Replaceable Experts in LLMs

You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

Confidence-Calibrated Inference Expansion for Evaluator-Guided Test-Time Reasoning