A Strong MoE Is All You Need for Process Supervision

Guanzhi Deng; Zengyan Liu; Haibo Wang; Kuan Wu; Shing Yin Wong; Sichun Luo; Zhuoran Wang; Linqi Song

cs.CLOct 21, 2026

A Strong MoE Is All You Need for Process Supervision

Authors: Guanzhi Deng, Zengyan Liu, Haibo Wang, Kuan Wu, Shing Yin Wong, Sichun Luo, Zhuoran Wang, Linqi Song

Organizations: City University of Hong Kong · Carnegie Mellon University · University of Hong Kong

Abstract

Process reward models (PRMs) provide fine-grained supervision for multi-step reasoning, yet obtaining reliable process labels remains a central bottleneck. Existing approaches typically rely on human annotation, rollout-based estimation, or auxiliary verification. In this work, we explore a different source of process supervision natively exposed by Mixture-of-Experts (MoE) models: expert routing. We show that simple routing statistics provide an intrinsic process-confidence signal that is competitive with conventional output-logit confidence. Building on this observation, we construct routing confidence by combining overall routing mass, head routing mass, and within-step routing variation, and introduce RoCo-Rethink, which identifies the least confident reasoning step and selectively regenerates the subsequent trajectory. We further introduce RoCo-PRM, which uses a strong MoE to automatically annotate reasoning trajectories with routing-derived step scores and distills them into a standalone PRM. Across multiple MoE architectures and diverse reasoning tasks, both RoCo-Rethink and RoCo-PRM consistently outperform strong baselines, while RoCo-PRM transfers effectively across both model architectures and tasks. Our findings suggest that expert routing is more than a computational dispatch mechanism: it provides an architecture-native source of process supervision for reasoning.

Figures & tables

Figure 1: Native MoE routing tracks step-level reliability and is competitive with output-side confidence. (a) Within-trajectory Spearman correlations between verifier-assessed step quality and overall routing mass ( $M$ ), head routing mass ( $H$ ), and within-step routing variation ( $D$ ) across three MoE models and four tasks. GSM8K and MATH results are averaged over two math PRMs. (b) Grouped-CV bad-step AUROC of routing-based versus logit-based predictors. Each point denotes one model–task–verifier setting; points above the diagonal indicate stronger predictive utility from routing features.

Figure 2: Overview of RoCo. RoCo leverages native MoE routing confidence for both test-time reasoning guidance and automatic process supervision. RoCo-Rethink uses routing confidence to identify and revise low-confidence reasoning steps, while RoCo-PRM distills routing-derived supervision into a standalone PRM that guides reasoning across target models.

RoCo-Rethink
Model	Method	GSM8K	MATH500	HumanEval	MMLU	IFEval	LogicBench	Avg.
OLMoE	Vanilla	48.7 $\pm$ 0.3	9.4 $\pm$ 0.5	29.3 $\pm$ 0.4	46.8 $\pm$ 0.1	25.7 $\pm$ 0.4	46.7 $\pm$ 0.3	34.4 $\pm$ 0.3
	Self-Consistency	49.7 $\pm$ 0.5	9.8 $\pm$ 0.7	30.5 $\pm$ 0.7	47.9 $\pm$ 0.1	26.6 $\pm$ 0.6	47.2 $\pm$ 0.2	35.3 $\pm$ 0.5
	LECO	50.5 $\pm$ 0.5	11.4 $\pm$ 0.6	31.1 $\pm$ 0.5	49.0 $\pm$ 0.1	28.7 $\pm$ 0.5	55.4 $\pm$ 0.4	37.7 $\pm$ 0.4
	RoCo-Rethink	51.6 $\pm$ 0.4	12.0 $\pm$ 0.4	32.3 $\pm$ 0.4	50.0 $\pm$ 0.1	30.5 $\pm$ 0.8	56.7 $\pm$ 0.5	38.9 $\pm$ 0.4
DeepSeek	Vanilla	69.8 $\pm$ 0.5	20.6 $\pm$ 0.6	46.3 $\pm$ 0.5	49.2 $\pm$ 0.1	29.8 $\pm$ 0.4	64.6 $\pm$ 0.5	46.7 $\pm$ 0.4

Table 1: Main results of RoCo-Rethink and RoCo-PRM. Results are reported as mean

\pm

SD over five random seeds. Bold and underline indicate the best and second-best results, respectively.

Figure 3: Routing-confidence ablation for RoCo-Rethink. Task axes use task-specific ranges. The full formulation is best or tied-best in 17 of 18 settings.

Figure 4: Monte Carlo revision-oracle analysis. Oracle (a) regret (

\downarrow

) and (b) hit rate (

\uparrow

), averaged over three tasks.

M{+}H{-}D

consistently achieves the best revision-point quality.

Figure 5: Test-time rethink-budget scaling. RoCo-Rethink consistently yields larger gains than LECO and random rethink across all three MoE models.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Task	$M$	$H$	$D$
DeepSeek	GSM8K	0.106	0.062	-0.100
DeepSeek	MATH500	0.111	0.053	-0.021
DeepSeek	HumanEval	0.235	0.111	-0.068
DeepSeek	MMLU	0.289	0.236	-0.169
OLMoE	GSM8K	0.249	0.252	-0.113
OLMoE	MATH500	0.028	0.073	-0.090

Appendix

Table 3: Within-trajectory Spearman correlation between routing statistics and verifier-assessed step quality. GSM8K and MATH500 values are averaged over the two math PRMs used in the main paper.

Model	Task	Verifier	Logit	Routing
DeepSeek	GSM8K	Llama-PRM	0.542	0.534
DeepSeek	GSM8K	Qwen-PRM	0.529	0.533
DeepSeek	MATH500	Llama-PRM	0.576	0.583
DeepSeek	MATH500	Qwen-PRM	0.573	0.598
DeepSeek	HumanEval	CodePRM	0.577	0.592
DeepSeek	MMLU	VersaPRM	0.391	0.496

Appendix

Table 4: Grouped-CV bad-step AUROC for routing-based and logit-based predictors.

Model	Layers	Experts	Top- $k$	Hidden	Type
DeepSeek-V2-Lite-Chat	27	64	6	2048	MoE
OLMoE-1B-7B-0924-SFT	16	64	8	2048	MoE
Qwen1.5-MoE-A2.7B-Chat	24	60	4	2048	MoE
Qwen3-30B-A3B	48	128	8	2048	MoE teacher
Qwen2.5-7B-Instruct	28	–	–	3584	Dense
Llama-3.1-8B-Instruct	32	–	–	4096	Dense

Appendix

Table 5: Model configurations used in the experiments. Routing columns are shown for MoE models only.

Benchmark	Eval size	Metric
GSM8K	1,319	Numerical answer accuracy
MATH500	500	Numerical answer accuracy
MMLU	14,042	Multiple-choice accuracy
HumanEval	164	pass@1
IFEval	541 prompts / 834 instr.	Loose prompt accuracy
LogicBench	1,520 QA pairs	BQA accuracy

Appendix

Table 6: Benchmark and evaluation setup.

Evaluation task	Supervision source	Step examples
GSM8K	GSM8K train	155,391
MATH500	MATH train	370,384
MMLU	MMLU train	82,338
HumanEval	APPS train	92,628
IFEval	IFEval supervision set	48,122
LogicBench	LogicBench augmentation	104,846

Appendix

Table 7: Process-supervision data used to train RoCo-PRM. Counts denote final step-level regression examples used for training.

Task	Newly appended reasoning step	$\widetilde{M}_{i}$	$\widetilde{H}_{i}$	$\widetilde{D}_{i}$	$C_{i}$
GSM8K	Let’s count the legs:	1.000	1.000	1.000	1.000
	Dogs: each dog has 4 legs; $2\times 4=8$ legs.	0.384	0.407	0.363	0.427
	Chicken: $1\times 2=2$ legs.	0.000	0.000	0.145	-0.145
	Total legs: $8+2=10$ .	0.049	0.240	0.243	0.046
LogicBench	Step 1: Identify the logical rule or relation in the context.	1.000	1.000	1.000	1.000
	The logical rule is that all cats love to play.	0.598	0.480	0.090	0.987

Appendix

Table 8: Examples of routing-derived RoCo-PRM supervision. Each row corresponds to one prefix-level regression example

(x,s_{\leq i},C_{i})

. For compactness, only the newly appended reasoning step is shown. The routing statistics are normalized within the trajectory, and

C_{i}=\widetilde{M}_{i}+\widetilde{H}_{i}-\widetilde{D}_{i}

Model	Method	GSM8K	MATH500	HumanEval	MMLU	IFEval	LogicBench
OLMoE	Random	48.4	9.5	29.2	47.2	26.3	48.5
	$M$	50.9	10.4	30.5	48.5	28.3	55.4
	$H$	51.0	10.9	29.3	49.2	27.8	55.5
	$-D$	50.9	10.2	29.9	49.3	28.2	55.8
	$M+H$	51.1	11.5	31.7	49.5	29.8	56.3
	$M+H-D$	51.6	12.0	32.3	50.0	30.5	56.7

Appendix

Table 9: Full routing-component ablation for RoCo-Rethink. Results are reported as means over five random seeds; standard deviations are omitted for readability. Bold and underline indicate the best and second-best results within each model–task setting, respectively.

		Random		$M$		$H$		$-D$		$M+H$		$M+H-D$
Task	Model	Regret	Hit	Regret	Hit	Regret	Hit	Regret	Hit	Regret	Hit	Regret	Hit
LogicBench	OLMoE	.125	.56	.101	.60	.110	.60	.125	.59	.099	.60	.094	.63
	DeepSeek	.249	.31	.230	.32	.239	.32	.199	.35	.218	.34	.186	.42
	Qwen1.5	.211	.41	.166	.45	.176	.42	.184	.42	.158	.46	.156	.47
GSM8K	OLMoE	.135	.62	.100	.64	.108	.64	.111	.63	.096	.67	.091	.68
	DeepSeek	.256	.38	.236	.40	.235	.40	.224	.40	.220	.41	.206	.44

Appendix

Table 10: Full Monte Carlo revision-oracle results. Regret measures the gap between the oracle revision value and the value of the selected step, while hit rate measures how often the selected step is oracle-optimal. Lower regret and higher hit rate are better.

Model	Method	$B=1$	$B=2$	$B=3$
OLMoE	Random	0.38	0.62	0.50
OLMoE	LECO	1.90	3.25	3.68
OLMoE	RoCo-Rethink	2.90	4.45	4.68
DeepSeek	Random	0.43	0.83	1.23
DeepSeek	LECO	1.43	3.13	3.43
DeepSeek	RoCo-Rethink	3.18	5.10	5.63

Appendix

Table 11: Mean score gain over Vanilla as the maximum rethink-round budget

B

increases.

Configuration	Value
Backbone	Qwen2.5-7B-Instruct
Objective	Scalar regression (MSE)
Optimizer	AdamW ( adamw_torch )
Weight decay	0
LoRA rank	16
LoRA $\alpha$	32

Appendix

Table 12: Default RoCo-PRM training configuration.

Target	Teacher	GSM8K	MATH500	HumanEval	MMLU	IFEval	LogicBench	Avg.
OLMoE	DeepSeek	50.1	10.8	30.4	48.5	28.3	53.5	36.9
OLMoE	Qwen3	53.0	13.4	32.9	51.3	29.4	57.8	39.6
DeepSeek	DeepSeek	72.2	25.5	47.8	51.2	31.5	69.9	49.7
DeepSeek	Qwen3	73.8	26.7	48.8	51.9	33.1	72.8	51.2
Qwen1.5	DeepSeek	52.9	19.5	20.1	56.4	28.9	67.3	40.9
Qwen1.5	Qwen3	54.3	21.4	23.4	58.5	30.9	70.9	43.2

Appendix

Table 13: Effect of routing-teacher quality on RoCo-PRM. The PRM backbone is fixed to Qwen2.5-7B-Instruct. Results are reported as mean scores; standard deviations are omitted for readability.

Target	PRM backbone	GSM8K	MATH500	HumanEval	MMLU	IFEval	LogicBench	Avg.
OLMoE	Qwen2.5-1.5B	51.3	11.5	31.5	49.2	28.9	56.2	38.1
OLMoE	Qwen2.5-7B	53.0	13.4	32.9	51.3	29.4	57.8	39.6
DeepSeek	Qwen2.5-1.5B	73.0	26.6	48.8	51.8	31.9	71.7	50.6
DeepSeek	Qwen2.5-7B	73.8	26.7	48.8	51.9	33.1	72.8	51.2
Qwen1.5	Qwen2.5-1.5B	54.1	19.9	22.6	57.7	30.4	67.9	42.1
Qwen1.5	Qwen2.5-7B	54.3	21.4	23.4	58.5	30.9	70.9	43.2

Appendix

Table 14: Effect of PRM backbone scale. The routing teacher is fixed to Qwen3-30B-A3B. Results are reported as mean scores; standard deviations are omitted for readability.

Target model	MATH500	GSM8K	MMLU	HumanEval
OLMoE	10.2 $\rightarrow$ 11.8	50.6 $\rightarrow$ 52.0	49.2 $\rightarrow$ 49.6	29.9 $\rightarrow$ 31.5
Qwen2.5-7B	50.0 $\rightarrow$ 53.0	81.2 $\rightarrow$ 84.5	70.0 $\rightarrow$ 70.6	49.3 $\rightarrow$ 51.2
Llama-3.1-8B	29.8 $\rightarrow$ 32.8	76.3 $\rightarrow$ 79.2	60.5 $\rightarrow$ 62.1	47.6 $\rightarrow$ 50.0
Avg. gain	+2.53	+2.53	+0.87	+1.97

Appendix

Table 15: Full leave-one-task-out cross-task generalization results. Each RoCo-PRM is trained without task-aligned routing supervision from the corresponding evaluation task. Values show target-model performance before and after LOO RoCo-PRM guidance.

Step	Reasoning step	$\widetilde{M}_{i}$	$\widetilde{H}_{i}$	$\widetilde{D}_{i}$	$C_{i}$
1	John bought twice as many red ties as blue ties, so he bought $2\times$ (number of blue ties) red ties.	0.114	0.135	0.803	-0.554
2	The red ties cost 50% more than blue ties, so the cost of the red ties is $1.5\times\$ 40=$60$ each.	0.138	0.155	1.000	-0.707
3	He spent $200 on blue ties, so he spent$ 200 $-$ $40 =$ 160 on red ties.	0.000	0.000	0.657	-0.657
4	The total cost of all the ties is $160 +$ 200 = $360.	0.049	0.138	0.522	-0.335
5	#### 360	1.000	1.000	0.000	2.000
6	The answer is: 360	0.738	0.742	0.957	0.523

Appendix

Table 16: Successful RoCo-Rethink correction on GSM8K. The selected revision point is highlighted in bold. Routing statistics are trajectory-normalized before computing

C_{i}=\widetilde{M}_{i}+\widetilde{H}_{i}-\widetilde{D}_{i}

Step	Reasoning step	$\widetilde{M}_{i}$	$\widetilde{H}_{i}$	$\widetilde{D}_{i}$	$C_{i}$
1	The house was bought for $80,000 and then had$ 50,000 in repairs, so the total cost was $130,000.	0.096	0.018	0.564	-0.451
2	The value increased by 150%, so the new value is $130,000 +$ 195,000 = $325,000.	0.170	0.162	0.742	-0.410
3	Josh sold the house for $325,000, so his profit is$ 325,000 $-$ $130,000 =$ 195,000.	0.000	0.000	0.569	-0.569
4	#### 195000	1.000	1.000	0.000	2.000
5	The answer is: 195000	0.742	0.696	1.000	0.438

Appendix

Table 17: RoCo-Rethink failure case on GSM8K. The selected revision point is highlighted in bold. Although the selector identifies the lowest-confidence step, regeneration reproduces the original erroneous continuation.

Process Supervision

Process supervision trains and evaluates AI systems on intermediate reasoning steps, not just final answers, to improve reliability and control.

See all 74 papers

Rubric-Guided Process Reward for Stepwise Model Routing
May 28, 2026Shenghao Ye, Yu Guo, Zhengheng Li +2Process Reward Models RL for Language Model Reasoning
University of Science and Technology of China · Southeast University
Unsupervised Process Reward Models
May 11, 2026Artyom Gadetsky, Maxim Kodryan, Siba Smarak Panigrahi +2Reward Modeling Process Reward Models
Swiss Federal Institute of Technology (EPFL)
Learning Process Rewards via Reasoning State Propagation
Sep 30, 2026Kai Gan, Zi-Hao Zhou, Bo Ye +3Process Reward Models RL for Language Model Reasoning
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China · Zhongguancun Academy +1

May 28, 2026J/K move · Enter open · S save

Rubric-Guided Process Reward for Stepwise Model Routing

Shenghao Ye, Yu Guo, Zhengheng Li, Shuangwu Chen+1

University of Science and Technology of China · Southeast University

Process Reward Models RL for Language Model Reasoning LLM Routing Efficient Language Model Inference

Open paper AskPDF arXiv

Stepwise model routing improves the efficiency of Large Reasoning Models (LRMs) by assigning each reasoning step to a suitable model. Recent methods formulate routing as a sequential decision process and train the router with reinforcement learning. However, although they model routing as a process, they still supervise the router with outcome rewards. Such rewards only reflect final answer correctness and fail to evaluate intermediate routing decisions, which can weaken performance and generalization. To address this gap, we propose RoRo, a rubric-guided process reward framework for stepwise model routing. RoRo first collects diverse routing trajectories and constructs preference pairs based on outcome, cost, and process quality. It then trains a Rubricor to generate a query-specific evaluation rubric and a Judge to score routing trajectories under this rubric through alternating optimization. The resulting process rewards are combined with outcome rewards to optimize the routing policy via GRPO. Experiments on five reasoning benchmarks under both same-family and cross-family settings show that RoRo consistently outperforms strong baselines and achieves better accuracy and cost trade-offs.

Abstract

Process Supervision

Explore similar work

Rubric-Guided Process Reward for Stepwise Model Routing

Unsupervised Process Reward Models

Learning Process Rewards via Reasoning State Propagation