From Local Prediction Error to Final Outputs: A Theory of Expert Sharing in Multi-Step Reasoning
Organizations: Nanyang Technological University · KTH Royal Institute of Technology · Beijing Institute of Technology
Abstract
Mixture-of-experts (MoE) models can share experts across layers or computational steps, pooling training information from different contexts. This can reduce local prediction error, but multi-step systems are ultimately evaluated by their final outputs. We study when these local gains survive composition. In specified linear regression models, we derive exact sharing and merging criteria by balancing variance savings against bias from sharing across roles with different target behavior. Under a common fixed linear propagation map, we characterize expected squared final-output risk, retaining error directions and correlations. We construct cases where sharing lowers average local error yet increases final-output error. Within this setting, we also identify when orthogonal full pooling with isotropic noise preserves every local improvement at the final output. For general routed computations, we derive conditional end-to-end error bounds that account for distribution shifts, errors in surrounding components, and changes in expert selection. Separately, we show that correct predictions and route labels alone do not establish that individual experts implement the attributed operations. Controlled experiments realize a strict local-to-final risk reversal. In low-data symbolic settings, trained Transformers exhibit statistically supported teacher-forced one-step gains without statistically resolved free-running final-output gains. Separate intervention tests distinguish successful multi-layer actions from single-expert attribution and generalization to unseen transitions. Together, these results distinguish when sharing improves local estimation, when those gains survive to the final output, and what evidence is needed to attribute reusable operations to individual experts.
Figures & tables
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
| Question and entry point | Page | |
|---|---|---|
| A | Symbols and claim types : what does this term or guarantee mean? | A |
| B | Roles and event lifts : how do the definitions fit a concrete model? | B |
| C | Distribution-free statistics : what can independent samples guarantee? | C |
| D | Exact local frontiers : how are pooling risks and selection justified? | D |
| E | Terminal certificates : how do local and routing errors propagate? | E |
| F | Causal actions : why are labels and compressed states insufficient? | F |
| Symbol or convention | Meaning and scope |
|---|---|
| , | A recurring role and an expert parameter-block index. . A role includes the call distribution and environment; it need not coincide with a human-assigned semantic label. |
| , , | Comparator model, its expert count, and its role-to-expert assignment. The compact comparison uses . |
| , , | Roles assigned to one reused expert; the number of roles in a local group; the number of role cells in a stacked model. A role cell can be a role–layer pair. |
| , , , | Event index and horizon ; within-event call index and number of calls in program . An event need not be one token. |
| , in depth models | Layer index and number of layers. elsewhere is a Lipschitz constant, while a matrix in Appendix M is a linear sharing rule. |
| , | Reference and fitted-model pre-event states; is their declared stage space. These are not the regression design matrices . |
| Linear-model symbol | Definition |
|---|---|
| , | True input-to-output operator and its scalar coefficient count. |
| , , | Fixed design matrix, observed responses, and response-noise matrix in . |
| , | Fitted OLS matrix and its parameter error ; this is not an expert function . |
| , | Response-noise variance and coefficient-estimation variance in the orthogonal design. Equal noise levels are homoscedastic; unequal ones are heteroscedastic. |
| , , | Balanced squared sharing bias, no-sharing risk , and variance saving . |
| , | Fixed partial-pooling strength and its population-optimal value. Covariance eigenvalues and resource multipliers are unrelated parameters. |
| Separate (MCSE) | Full (MCSE) | Reference ratio | Observed ratio | ||
|---|---|---|---|---|---|
| 0 | 64 | 0.5329 (0.0096) | 0.0664 (0.0012) | 0.1250 | 0.1246 |
| 0 | 256 | 0.1355 (0.0022) | 0.0169 (0.0003) | 0.1250 | 0.1250 |
| 0 | 1024 | 0.0331 (0.0005) | 0.0042 (0.0001) | 0.1250 | 0.1255 |
| 0.5 | 64 | 0.5329 (0.0096) | 0.1240 (0.0012) | 0.2344 | 0.2327 |
| 0.5 | 256 | 0.1355 (0.0022) | 0.0745 (0.0003) | 0.5625 | 0.5503 |
| 0.5 | 1024 | 0.0331 (0.0005) | 0.0618 (0.0001) | 1.8750 | 1.8644 |
| Selected risk | Excess | Five selection counts | ||
|---|---|---|---|---|
| 0.0 | 64 | 0.0664 | 0.000000 | 0/0/0/0/128 |
| 0.5 | 64 | 0.1241 | 0.000782 | 0/0/0/53/75 |
| 2.0 | 64 | 0.2245 | 0.000638 | 0/0/31/97/0 |
| 0.0 | 256 | 0.0169 | 0.000000 | 0/0/0/0/128 |
| 0.5 | 256 | 0.0567 | 0.000169 | 0/0/23/105/0 |
| 2.0 | 256 | 0.0976 | 0.000373 | 0/107/21/0/0 |
| Feature source | Condition number | Cells | Flagged |
|---|---|---|---|
| trained-local-s0 | 3002 | 54 | 0 |
| trained-tied-s0 | 2625 | 54 | 0 |
| Model / geometry | Reference | Observed | MCSE | |
|---|---|---|---|---|
| reversal/local | 0.0 | 1.0000 | 1.0090 | 0.00590 |
| reversal/local | 0.4 | 0.6533 | 0.6574 | 0.00406 |
| reversal/local | 1.0 | 0.8333 | 0.8359 | 0.00341 |
| reversal/common | 0.0 | 1.0000 | 1.0077 | 0.01022 |
| reversal/common | 0.4 | 1.0000 | 1.0077 | 0.01022 |
| reversal/common | 1.0 | 1.0000 | 1.0077 | 0.01022 |
| Model | Seed | Params (M) | Terminal (%) | Trace (%) | Step (ms) | Peak |
|---|---|---|---|---|---|---|
| local | 0 | 10.5119 | 0.0000 | 0.0000 | 31.7 | 278.3 |
| local | 1 | 10.5119 | 0.0031 | 0.0031 | 32.5 | 279.0 |
| local | 2 | 10.5119 | 0.0000 | 0.0000 | 32.5 | 279.3 |
| tied | 0 | 3.4340 | 0.0000 | 0.0000 | 29.9 | 184.8 |
| tied | 1 | 3.4340 | 0.0000 | 0.0000 | 30.5 | 184.5 |
| tied | 2 | 3.4340 | 0.0000 | 0.0000 | 30.2 | 185.1 |
| Model | Seed | H4 term. | H4 trace | H12 term. | H12 trace | H16 term. | H16 trace |
|---|---|---|---|---|---|---|---|
| local | 0 | 0.000 | 0.000 | 96.826 | 99.951 | 96.973 | 100.000 |
| local | 1 | 0.000 | 0.000 | 96.631 | 99.927 | 96.802 | 100.000 |
| local | 2 | 0.000 | 0.000 | 97.046 | 100.000 | 97.119 | 100.000 |
| tied | 0 | 0.000 | 0.000 | 96.899 | 100.000 | 96.655 | 100.000 |
| tied | 1 | 0.000 | 0.000 | 97.046 | 99.902 | 96.851 | 100.000 |
| tied | 2 | 0.000 | 0.000 | 97.192 | 100.000 | 97.070 | 100.000 |
| Obj. | Seed | Raw | Aligned | Compound | L0 | L1 | L2 | L3 |
|---|---|---|---|---|---|---|---|---|
| B | 0 | 0.2621 | 0.3605 | 0.2567 | 0.267 | 0.267 | 0.266 | 0.267 |
| B | 1 | 0.2051 | 0.3790 | 0.2438 | 0.267 | 0.267 | 0.257 | 0.270 |
| B | 2 | 0.2657 | 0.3297 | 0.2580 | 0.267 | 0.266 | 0.270 | 0.267 |
| R | 0 | 1.0000 | 1.0000 | 0.9632 | 0.279 | 0.267 | 0.293 | 0.269 |
| R | 1 | 1.0000 | 1.0000 | 0.9291 | 0.290 | 0.267 | 0.276 | 0.269 |
| R | 2 | 1.0000 | 1.0000 | 0.9478 | 0.292 | 0.268 | 0.269 | 0.267 |
| Objective | Fidelity | Cross-fidelity | Separation | Target sep. |
|---|---|---|---|---|
| B | 0.2499 0.0093 | 0.1033 0.0095 | 0.5309 0.0270 | 0.9744 0.0000 |
| R | 0.9455 0.0202 | 0.9273 0.0269 | 0.9658 0.0015 | 0.9744 0.0000 |
| I | 1.0000 0.0000 | 1.0000 0.0000 | 0.9744 0.0000 | 0.9744 0.0000 |
| Checkpoint | Alternative | Distance | Harm (pp) | SE (pp) | Spearman |
|---|---|---|---|---|---|
| local-s0 | nearest | 4.458 | 8.618 | 0.439 | 0.172 |
| local-s0 | random | 4.954 | 13.843 | 0.540 | 0.270 |
| local-s0 | farthest | 5.456 | 19.189 | 0.615 | 0.378 |
| tied-s0 | nearest | 3.576 | 0.781 | 0.138 | 0.013 |
| tied-s0 | random | 3.884 | 2.759 | 0.256 | 0.066 |
| tied-s0 | farthest | 4.190 | 5.029 | 0.342 | 0.115 |
| Checkpoint | Alternative | Natural error | Harm (pp) | Repair (%) | Spearman |
|---|---|---|---|---|---|
| EXT-local-B-n128-s0 | nearest | 0.782 | 0.68 | 1.27 | -0.038 |
| EXT-local-B-n128-s0 | farthest | 0.782 | 3.91 | 1.66 | 0.031 |
| EXT-local-B-n128-s1 | nearest | 0.712 | 2.64 | 1.07 | -0.007 |
| EXT-local-B-n128-s1 | farthest | 0.712 | 5.08 | 2.15 | 0.137 |
| EXT-local-B-n128-s2 | nearest | 0.778 | 0.59 | 1.07 | -0.021 |
| EXT-local-B-n128-s2 | farthest | 0.778 | 3.71 | 1.66 | 0.020 |
| Perturbation | Pairs | Certified | Boundary | Not met | Violations | Changed/not met |
|---|---|---|---|---|---|---|
| direct_c=0 | 24576 | 24574 | 2 | 0 | 0 | 0 |
| direct_c=0.24 | 24576 | 24573 | 3 | 0 | 0 | 0 |
| direct_c=0.49 | 24576 | 24515 | 61 | 0 | 0 | 0 |
| direct_c=0.5 | 24576 | 0 | 24576 | 0 | 0 | 0 |
| direct_c=0.51 | 24576 | 0 | 61 | 24515 | 0 | 24515 |
| direct_c=1.0 | 24576 | 0 | 2 | 24574 | 0 | 24574 |
| Candidate/ref. | Seed | Loss | Difference | Lower | Upper |
|---|---|---|---|---|---|
| tied/local | 0 | terminal | 0.0000 | -2.0039 | 2.0039 |
| tied/local | 0 | trace | 0.0000 | -2.0039 | 2.0039 |
| dense/local | 0 | terminal | 0.0000 | -2.0039 | 2.0039 |
| dense/local | 0 | trace | 0.0000 | -2.0039 | 2.0039 |
| R/B | 0 | 1–Fid | 70.6573 | 68.6534 | 72.6613 |
| I/R | 0 | 1–Fid | 3.6774 | 1.6735 | 5.6813 |
| Training traces | Local | Tied | Dense |
|---|---|---|---|
| 16 | 0.9467 0.0034 | 0.9436 0.0032 | 0.9389 0.0065 |
| 24 | 0.9322 0.0008 | 0.9245 0.0039 | 0.9247 0.0110 |
| 32 | 0.9167 0.0035 | 0.8999 0.0087 | 0.8917 0.0130 |
| 48 | 0.8651 0.0107 | 0.8224 0.0239 | 0.8160 0.0524 |
| 64 | 0.7959 0.0220 | 0.7398 0.0293 | 0.7033 0.0488 |
| 96 | 0.4133 0.2122 | 0.3670 0.1186 | 0.1377 0.0459 |
| Training traces | Local | Tied | Dense |
|---|---|---|---|
| 16 | 0.9671 0.0014 | 0.9678 0.0008 | 0.9668 0.0011 |
| 24 | 0.9665 0.0006 | 0.9666 0.0004 | 0.9673 0.0010 |
| 32 | 0.9680 0.0005 | 0.9671 0.0009 | 0.9679 0.0006 |
| 48 | 0.9667 0.0009 | 0.9673 0.0012 | 0.9684 0.0003 |
| 64 | 0.9679 0.0006 | 0.9685 0.0001 | 0.9677 0.0008 |
| 96 | 0.8883 0.1295 | 0.9174 0.0565 | 0.6444 0.1314 |
| Seed | Step diff. | Step lower | Step upper | Term. diff. | Term. lower | Term. upper |
|---|---|---|---|---|---|---|
| 0 | 5.183 | 2.786 | 7.580 | -0.043 | -2.440 | 2.354 |
| 1 | 5.162 | 2.765 | 7.559 | -0.006 | -2.403 | 2.391 |
| 2 | 6.488 | 4.092 | 8.885 | -0.143 | -2.540 | 2.253 |
| Traces | Model | Step error | Terminal error | Lower/higher/unclear |
|---|---|---|---|---|
| 64 | local | 0.7684 0.0233 | 0.9678 0.0009 | – |
| 64 | tied | 0.7127 0.0271 | 0.9679 0.0005 | 10/0/0 |
| 64 | dense | 0.6499 0.0530 | 0.9676 0.0006 | 10/0/0 |
| 96 | local | 0.3736 0.1210 | 0.9129 0.0677 | – |
| 96 | tied | 0.3630 0.0753 | 0.9256 0.0355 | 6/4/0 |
| 96 | dense | 0.1583 0.0518 | 0.6899 0.1248 | 9/0/1 |
| Traces | Obj. | Terminal error | Raw labels | Forced fidelity | Cross-fidelity |
|---|---|---|---|---|---|
| 64 | B | 0.9685 0.0001 | 0.2443 0.0528 | 0.1724 0.0108 | 0.1442 0.0067 |
| 64 | R | 0.9642 0.0006 | 1.0000 0.0000 | 0.3713 0.0267 | 0.3406 0.0298 |
| 64 | I | 0.1257 0.0454 | 1.0000 0.0000 | 0.9943 0.0014 | 0.9978 0.0006 |
| 96 | B | 0.9174 0.0565 | 0.2154 0.0581 | 0.3097 0.0293 | 0.2102 0.0134 |
| 96 | R | 0.5103 0.1574 | 1.0000 0.0000 | 0.7963 0.0260 | 0.7606 0.0351 |
| 96 | I | 0.0215 0.0098 | 1.0000 0.0000 | 0.9991 0.0002 | 0.9997 0.0005 |
| Obj. | Seed | Seen fid. | Withheld fid. | Withheld SE | Clean term. | Uses term. |
|---|---|---|---|---|---|---|
| B | 0 | 0.26882 | 0.07528 | 0.00316 | 0.00000 | 0.98120 |
| B | 1 | 0.15763 | 0.05167 | 0.00252 | 0.00000 | 0.97226 |
| B | 2 | 0.31621 | 0.04687 | 0.00246 | 0.00000 | 0.98362 |
| R | 0 | 0.93353 | 0.02218 | 0.00176 | 0.00000 | 0.98298 |
| R | 1 | 0.96587 | 0.02290 | 0.00179 | 0.00000 | 0.98262 |
| R | 2 | 0.90390 | 0.00000 | 0.00000 | 0.00000 | 0.98435 |
| Obj. | Holdout: seen | Holdout: withheld | Full table: seen | Full table: subset |
|---|---|---|---|---|
| B | 0.2476 0.0814 | 0.0579 0.0152 | 0.1879 0.0129 | 0.2484 0.0708 |
| R | 0.9344 0.0310 | 0.0150 0.0130 | 0.9451 0.0168 | 0.9493 0.0165 |
| I | 1.0000 0.0000 | 0.0021 0.0037 | 1.0000 0.0000 | 1.0000 0.0000 |
| Setting | Seed | Hole | Params (M) | Seen fidelity | Withheld fidelity | Uses error |
|---|---|---|---|---|---|---|
| all | 0 | 16 | 17.89 | 1.00000 | 0.00114 | 0.9875 |
| all | 1 | 16 | 17.89 | 1.00000 | 0.00000 | 0.9794 |
| all | 2 | 16 | 17.89 | 1.00000 | 0.00057 | 0.9916 |
| base | 0 | 32 | 3.43 | 0.99472 | 0.00249 | 0.9774 |
| base | 0 | 4 | 3.43 | 1.00000 | 0.00000 | 0.9990 |
| base | 0 | 8 | 3.43 | 1.00000 | 0.00000 | 0.9935 |
| Withheld | Pairs | Train traces | Updates | Parameters | ||||
|---|---|---|---|---|---|---|---|---|
| 31 | 124 | 0% | 0 | 20,666 | 3,000 | 3,433,984 | 16384 | 0 |
| 31 | 124 | 25% | 31 | 15,500 | 3,000 | 3,433,984 | 1705 | 14679 |
| 31 | 124 | 50% | 62 | 10,334 | 3,000 | 3,433,984 | 54 | 16330 |
| 97 | 388 | 0% | 0 | 64,666 | 9,000 | 3,467,776 | 16384 | 0 |
| 97 | 388 | 25% | 97 | 48,500 | 9,000 | 3,467,776 | 1618 | 14766 |
| 97 | 388 | 50% | 194 | 32,334 | 9,000 | 3,467,776 | 72 | 16312 |
| Withheld | Withheld fidelity | Seen fidelity | Error on traces using withheld pairs | |
|---|---|---|---|---|
| 31 | 0% | — | — | |
| 31 | 25% | |||
| 31 | 50% | |||
| 97 | 0% | — | — | |
| 97 | 25% | |||
| 97 | 50% |
| Withheld | Seed | Withheld fid. | Seen fid. | Clean error | Error on use | |||
|---|---|---|---|---|---|---|---|---|
| 31 | 0% | 0 | — | 1.000000 | 16384 | 0.000000 | 0 | — |
| 31 | 0% | 1 | — | 1.000000 | 16384 | 0.000000 | 0 | — |
| 31 | 0% | 2 | — | 1.000000 | 16384 | 0.000000 | 0 | — |
| 31 | 25% | 0 | 0.000000 | 1.000000 | 1705 | 0.000000 | 14679 | 0.980857 |
| 31 | 25% | 1 | 0.006470 | 1.000000 | 1705 | 0.000000 | 14679 | 0.974317 |
| 31 | 25% | 2 | 0.000244 | 1.000000 | 1705 | 0.000000 | 14679 | 0.975611 |
| Traces | Obj. | Centered variance | Cell-weighted fidelity | Deterministic fraction |
|---|---|---|---|---|
| 64 | B | 0.8221 0.0183 | 0.1724 0.0108 | 0.0000 0.0000 |
| 64 | R | 0.7370 0.0205 | 0.3713 0.0267 | 0.0000 0.0000 |
| 64 | I | 0.0107 0.0024 | 0.9943 0.0014 | 0.7366 0.0123 |
| 96 | B | 0.6418 0.0159 | 0.3097 0.0293 | 0.0000 0.0000 |
| 96 | R | 0.2941 0.0388 | 0.7963 0.0260 | 0.0430 0.0123 |
| 96 | I | 0.0017 0.0003 | 0.9991 0.0002 | 0.9194 0.0689 |
| Groups | Global proxy | Covariance ref. | Observed | MCSE | ||
|---|---|---|---|---|---|---|
| 2.0 | 64 | 0/4 | -0.03990 | 0.00545 | 0.00543 | 0.00007 |
| 2.0 | 64 | 2/6 | -0.03990 | 0.00385 | 0.00377 | 0.00008 |
| 0.5 | 256 | 0/4 | -0.00998 | 0.00136 | 0.00135 | 0.00002 |
| 0.5 | 256 | 2/6 | -0.00997 | 0.00096 | 0.00094 | 0.00002 |
| Separate ref. | Plug-in mean | MCSE | Savings (%) | ||
|---|---|---|---|---|---|
| 0.0 | 64 | 0.5268 | 0.0738 | 0.00110 | 86.0 |
| 0.5 | 64 | 0.5268 | 0.1268 | 0.00101 | 75.9 |
| 2.0 | 64 | 0.5268 | 0.2292 | 0.00115 | 56.5 |
| 0.0 | 256 | 0.1317 | 0.0180 | 0.00026 | 86.4 |
| 0.5 | 256 | 0.1317 | 0.0569 | 0.00027 | 56.8 |
| 2.0 | 256 | 0.1317 | 0.0939 | 0.00054 | 28.7 |
| Architecture subspace | Approximation term | Dimension | Exact risk |
|---|---|---|---|
| Cell-specific layer-local experts | |||
| One globally tied expert | |||
| Role-tied across depth | |||
| One role-shared expert per depth | |||
| Additive depth-conditioned reuse |
| Architecture | Functional route-cost proxy |
|---|---|
| Additive soft gate | over expert outputs evaluated at one common input |
| Uniform top-k | Minimum bipartite matching cost in Equation 132 |
| Capacity-limited additive routing | Augmented transport through a zero-residual atom, charging for dropped mass |
| Row-separable expert-choice assignment | Sum, average, or maximum of row-wise transport costs after token inputs are fixed |
| Full soft dispatch | Output transport plus Lipschitz-weighted input-dispatch transport in Equation 137 |
| Hierarchical additive MoE | Coarse outer transport plus conditional within-group transport |
| Result | Key assumptions | Application conditions | Applicable extension |
|---|---|---|---|
| Q.1 Local estimation and data-dependent choices | |||
| Lemma 6 | Declared roles, interfaces and complete states; probe objects fixed independently of fitting, or covered jointly. | Same-data learned environments need a joint uniform class or independent split. | Appendices B and C . |
| Theorems 1 and 16 | Fixed orthogonal designs, centered independent noise with stated moments, independent tests and fixed pooling. | General designs and covariance need their own risk geometry. | Theorems 38 and 39 . |
| Proposition 7 | Common squared Hilbert geometry; unrestricted or appropriate realizable class for pure variance. | Restricted classes retain distance corrections. | Equation 39 . |
| Theorems 2 , 18 , 19 and 20 | Fixed disjoint groups or linear tying spaces; balanced cells and stated unbiased covariance for dimension penalties. | Adaptive grouping needs selection control; embeddings need realization conditions; greedy optimization needs separate analysis. | Corollary 40 ; independent selection or a uniform family bound. |
| Equations 64 and 10 ; Section D.6 | Gaussian noise and positive variance for sampling laws. The plug-in also uses known , a fixed mean subspace and . | Degeneracies use deterministic identities; searched merges need multiplicity control. | Separate expectation identities and selection guarantees. |
Mixture of Experts
Mixture of Experts (MoE) models route inputs to specialized subnetworks, aiming to scale capacity and adapt computation without activating every parameter.
See all 240 papers