MENTOR: Learning from Failure with On-Policy Pedagogical Intervention

Guanlan Yue; Hang Yu; Xiaoyong Huang; Ruipeng Li; Wei Zhang; Bingchang Liu; Rui Wang

cs.AIOct 9, 2026

MENTOR: Learning from Failure with On-Policy Pedagogical Intervention

Authors: Guanlan Yue, Hang Yu, Xiaoyong Huang, Ruipeng Li, Wei Zhang, Bingchang Liu, Rui Wang

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) is a key driver for Large Reasoning Models, but it is vulnerable to "zero-reward collapse", a training failure where a sudden increase in task difficulty causes the learning signal to vanish. We argue that the conventional approach of discarding these failed trajectories is wasteful, as they contain valuable diagnostic information about the model's cognitive gaps, a bright side that can be leveraged for recovery. Existing solutions, however, often regress to static, "off-policy-style" interventions like pre-defined hint libraries or perform post-hoc refinement, failing to provide the adaptive, forward-looking guidance needed to escape the collapse. To address this, we propose a paradigm shift towards on-policy pedagogical intervention and introduce MENTOR, a multi-agent framework that acts as a live mentor. MENTOR diagnoses the root cause of failure in real-time and dynamically generates targeted, instructional prompts to guide the learning agent's next attempt. As a plug-and-play module, MENTOR integrates seamlessly into existing policy optimization pipelines like GRPO without modifying the training infrastructure. Extensive experiments on math and code generation show that MENTOR effectively mitigates zero-reward collapse, outperforming state-of-the-art baselines. Notably, it achieves massive average relative gains of up to 78.3% on Llama-3.2-3B-Instruct over its base model.

Figures & tables

Figure 1: Deepseek-R1-Distill-Qwen-1.5B ( Guo et al., 2025 ) math reasoning training dynamics.

Figure 2: Comparison of mitigation strategies for zero-reward collapse in RLVR.

Figure 3: Overview of the MENTOR framework.

Methods	AIME24	AIME25	MATH-500	GAOKAO-Bench	Olympiad-Bench	DM	Avg	MMLU
Deepseek-R1-Distill-Qwen-1.5B
Base	16.67	16.67	66.20	63.55	2.50	34.32	38.57	17.14
GRPO	16.67	16.67	67.20	65.89	2.00	40.88	43.55	21.43
DAPO	13.33	16.67	71.20	72.90	3.50	43.72	46.66	22.86
SRPO	20.00	16.67	71.40	72.90	3.50	43.84	46.83	17.14
Critique	16.67	16.67	71.60	74.70	2.50	43.88	46.92	24.29

Table 1: Performance (%) on math reasoning benchmarks and out-of-distribution (OOD) generalization on MMLU. Best: bold; second-best: underlined.Avg denotes the sample-weighted average.All improvements reported are relative gains.

Methods	LCB	HE	MBPP	BCB	Avg
Deepseek-Coder-1.3B
Base	61.93	15.85	46.40	20.09	30.13
GRPO	64.77	16.46	49.40	23.60	33.18
DAPO	65.34	17.07	50.00	23.86	33.59
SRPO	65.34	18.29	49.60	24.04	33.69
Critique	65.34	18.90	50.00	24.56	34.14

Table 2: Performance (%) on code generation benchmarks. Best: bold; second-best: underlined.Avg denotes the sample-weighted average.All improvements reported are relative gains.

Method	GPU Hours	Wall Time	Human Labor	Total Time
Vanilla GRPO	120h	15.3h	0h	15.3h
Critique-GRPO	168h	23.5h	0h	23.5h
Offline Baselines	132h	21.3h	$\sim$ 24h *	$\sim$ 45.3h
MENTOR (Ours)	134.7h	17.9h	0h (Auto)	17.9h
* Estimated expert hours for data curation.

Table 3: Training time and labor cost comparison.

Method	Avg. Acc. (%)	$\delta$ Acc. (%)
w/o Attribution Agent	83.21	-4.50
w/o Generation Agent	83.04	-4.67
w/o Critique-Repair Agent	85.43	-2.28
Ours	87.71	-

Table 4: Ablation study on core components.

Figure 4: Hyperparameter ablation of Max Retry and Refine.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

ERROR_KEYPHRASE_PROMPT (Attribution Agent)

You are an Attribution Agent. Given a question, its ground-truth answer, and one or more failed student solutions/trajectories, diagnose the persistent cognitive error.

Output diagnostic keywords that localize the conceptual gap or logical breakpoint. Do NOT reveal the solution path, final answer, formulas, or implementation steps.

Focus on logical and algorithmic mistakes rather than trivial syntax issues.

Prefer a short keyword phrase (or a semicolon-separated set of keywords) capturing the root misconception.

Examples:

- Confused nowhere dense sets with

F_{\sigma}

sets

Appendix

Table 5: Prompt template for Attribution Agent (diagnostic keyword extraction)

GUIDANCE_PROMPT_TEMPLATE (Generation Agent)

You are a Generation Agent acting as a pedagogical tutor. Using the diagnostic keywords as a directive and the failed trajectories as context, write ONE candidate instructional hint for the same question.

The hint should:

1. Directly address the diagnosed misconception or logical breakpoint

2. Be concise, actionable, and specific to the failure mode

3. Guide thinking without giving the solution path

4. Avoid answer leakage: no final answers, numerical results, formula derivations, or concrete code snippets

Appendix

Table 6: Prompt template for Generation Agent (instructional hint generation)

HINT_VERIFY_PROMPT (Judgment Agent)

You are a Judgment Agent (LLM-as-a-Judge). Evaluate whether a generated hint is HIGH or LOW quality. Do NOT revise the student’s solution; judge only the hint.

A HIGH quality hint:

- Semantically aligned with the diagnosed failure (diagnostic keywords) and the question

- Points to the core concept, principle, algorithm, or reasoning pattern that failed

- Does not leak the answer: no final answer, formula derivation, calculation, or complete implementation

- Clear and targeted, not vague or generic

Appendix

Table 7: Prompt template for Judgment Agent (hint quality verification)

HINT_REFINE_PROMPT (Repair Agent)

You are a Repair Agent. Revise the hint (not the student’s solution) according to the Judgment critique so it becomes a safe, pedagogically sound instructional prompt.

Question: {question}

Diagnostic Keywords: {error_keyphrase}

Original Hint: {current_hint}

Critique / Issue(s): {fixed_feedback}

Guidelines:

Appendix

Table 8: Prompt template for Repair Agent (hint refinement)

CRITIQUE_PROMPT (Repair Agent: Critique step)

You are a Repair Agent performing self-critique on an instructional hint. Identify EXACTLY ONE critical flaw that prevents this hint from being HIGH quality.

Choose from:

- Too vague (e.g., no specific concept named)

- Reveals the answer or key number (answer leakage)

- Gives step-by-step solution instructions

- Misaligned with the diagnosed failure / mentions irrelevant concept

Appendix

Table 9: Prompt template for Repair Agent critique step

REWRITE_PROMPT (Repair Agent: Rewrite step)

Revise the following hint to fix the flaw: “{flaw}". Do not revise the student’s solution.

Question: {question}

Diagnostic Keywords: {error_keyphrase}

Original Hint: {current_hint}

Guidelines:

- Fix the stated flaw

Appendix

Table 10: Prompt template for Repair Agent rewrite step

Input: “Determine whether there exists a nowhere dense set in

\mathbb{R}

that is not an

F_{\sigma}

set.”

Rollout

Rollout 1: “A nowhere dense set has empty interior in its closure… Therefore all nowhere dense sets are

F_{\sigma}

. Answer: No.”

Rollout 2: “

F_{\sigma}

sets are countable unions of closed sets… and nowhere dense sets are meager, the answer is No.”

Rollout 3: “The Cantor set is closed and nowhere dense… Therefore all nowhere dense sets are

F_{\sigma}

. Answer: No.”

Rollout 4: “By the Baire Category Theorem, any meager… set is contained in a countable union of closed sets… Answer: No.”

Attribution Agent: “Confused nowhere dense sets with

F_{\sigma}

sets; misapplied Baire Category Theorem by equating meager sets with

F_{\sigma}

sets.”

Appendix

Table 11: MENTOR framework: complete process on the question “Determine whether there … is not an

F_{\sigma}

set” and ground-truth answer “Yes", with 4 rollouts.

Input: “Find the number of ordered pairs

(x,y)

, where both

x

and

y

are integers between

-100

and

100

, inclusive, such that

12x^{2}-xy-6y^{2}=0

.”

Rollout

Rollout 1: “Factoring the equation

12x^{2}-xy-6y^{2}=0

, we get

(3x+2y)(4x-3y)=0

. Case 1:

3x+2y=0\implies y=-1.5x

. Letting

x=2k,y=-3k

-100\leq-3k\leq 100\implies-33\leq k\leq 33

, which gives 67 pairs. Case 2:

4x-3y=0\implies y=\frac{4}{3}x

. Letting

x=3m,y=4m

-100\leq 4m\leq 100\implies-25\leq m\leq 25

, which gives 51 pairs. Total number of pairs is

67+51=118

. Answer: 118.”

Rollout 2: “We have

12x^{2}-xy-6y^{2}=0

. Using the quadratic formula for

x

x=\frac{y\pm\sqrt{y^{2}-4(12)(-6y^{2})}}{24}=\frac{y\pm 17y}{24}

. So

x=\frac{3y}{4}

x=\frac{-2y}{3}

. For

x=3y/4

y

is a multiple of 4, giving 51 solutions. For

x=-2y/3

y

is a multiple of 3, giving 67 solutions. The total is

51+67=118

. Answer: 118.”

Rollout 3: “The quadratic factors to

(3x+2y)(4x-3y)=0

. There are 67 integer solutions for the first line within the range

[-100,100]

and 51 integer solutions for the second line. Total pairs = 118. Answer: 118.”

Rollout 4: “

(3x+2y)(4x-3y)=0

. From

3x=-2y

, there are 67 solutions. From

4x=3y

, there are 51 solutions. Adding them gives

118

solutions. I don’t see any other constraints. Answer: 118.”

Attribution Agent

Appendix

Table 12: MENTOR framework: complete step-by-step process on the question “Find the number of ordered pairs

(x,y)

, where both

x

and

y

are integers between

-100

and

100

, inclusive, such that

12x^{2}-xy-6y^{2}=0

” and ground-truth answer “117", with 4 rollouts.

Input: “An isosceles trapezoid has an inscribed circle tangent to each of its four sides. The radius of the circle is 3, and the area of the trapezoid is 72. Let the parallel sides of the trapezoid have lengths

r

and

s

, with

r\neq s

. Find

r^{2}+s^{2}

.”

Rollout

Rollout 1: “The inradius gives the height

h=6

, and the area gives

\frac{1}{2}(r+s)(6)=72\implies r+s=24

. Because a circle is inscribed, the two legs satisfy

2c=r+s

, so each leg is

c=12

. A leg rises by the height and runs horizontally by the difference of the two bases, so

(s-r)=\sqrt{12^{2}-6^{2}}=\sqrt{108}

. Therefore

r^{2}+s^{2}=\frac{(r+s)^{2}+(s-r)^{2}}{2}=\frac{576+108}{2}=342

. Answer: 342.”

Rollout 2: “Place the longer base from

(0,0)

(s,0)

and the shorter one from

(s-r,6)

(s,6)

, so the slanted leg joins

(0,0)

(s-r,6)

. With

r+s=24

and each leg equal to 12 by the tangency condition,

(s-r)^{2}+6^{2}=12^{2}\implies(s-r)^{2}=108

, giving

r^{2}+s^{2}=\frac{576+108}{2}=342

. Answer: 342.”

Rollout 3: “Height

h=6

r+s=24

. By Pitot’s theorem for tangential quadrilaterals, the slant side

c

satisfies

2c=r+s=24

, so

c=12

. Drawing altitudes forms a right triangle with hypotenuse 12, height 6, and base

s-r

. By the Pythagorean theorem,

(s-r)^{2}+6^{2}=12^{2}\implies(s-r)^{2}=108

. We know

2(r^{2}+s^{2})=(r+s)^{2}+(s-r)^{2}=576+108=684

. So

r^{2}+s^{2}=342

. Answer: 342.”

Rollout 4: “

r+s=24

and

c=12

. The right triangle from the altitude has legs

6

and

s-r

, and hypotenuse

12

. So

(s-r)^{2}=12^{2}-6^{2}=108

. We want

r^{2}+s^{2}

. Since

r+s=24

and

(s-r)^{2}=108

, we add them:

2r^{2}+2s^{2}=576+108=684\implies r^{2}+s^{2}=342

. Answer: 342.”

Attribution Agent

Appendix

Table 13: MENTOR framework: complete step-by-step process on the question “An isosceles trapezoid has an inscribed circle tangent to each of its four sides. The radius of the circle is 3, and the area of the trapezoid is 72. Let the parallel sides of the trapezoid have lengths

r

and

s

, with

r\neq s

. Find

r^{2}+s^{2}

” and ground-truth answer “504", with 4 rollouts.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,103 papers

Jun 29, 2026

LatentRevise: Learning from Zero-Hit Reasoning

Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by hard prompts on which correct trajectories have low probability, so sampling misses them within a practical budget and leaves the policy update with little useful signal. We frame such zero-hit prompts as RLVR's sampling frontier, where new reasoning behavior is most valuable yet least likely to be sampled. Importantly, failed rollouts can be informative: they expose where the model's reasoning went wrong. We introduce LatentRevise, a first-order latent revision method that recovers training signal for this zero-hit regime. Given a failed rollout and the gold answer as an anchor, LatentRevise optimizes the input embeddings of its reasoning prefix under two complementary gradients, moving the prefix away from the failed continuation and toward the gold answer. The optimization is constrained to the convex hull of the model's vocabulary embeddings, so each update moves the latent toward a real token embedding rather than an arbitrary feature direction. We find that continuations from the revised prefix lengthen, exhibit self-reflection, and reach correct answers missed by the original rollouts. Used as training data, these trajectories improve SFT and RLVR on math benchmarks over standard baselines.

Yiqiu Guo, Xueting Han, Qi Jia +2

1Fudan University · 2Microsoft Research Asia · 3Shanghai AI Laboratory +1

RL for Language Model Reasoning Mathematical Reasoning Reinforcement Learning with Verifiable Rewards

Ask PDF

May 14, 2026

Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective paradigm for improving the reasoning capabilities of large language models. However, RLVR training is often hindered by sparse binary rewards and weak credit assignment, resulting in ambiguous optimization signals and underutilization of the useful information embedded in failed trajectories. To address this challenge, we propose Correction-Oriented Policy Optimization (CIPO), a simple and effective extension to RLVR that converts on-policy failed trajectories into correction-oriented supervision, without relying on any external signals. By jointly optimizing correction samples derived from the model's own failed attempts together with the standard RLVR objective, CIPO improves learning effectiveness while explicitly enhancing the model's ability to correct its own errors. Extensive experiments across 11 benchmarks spanning mathematical reasoning and code generation demonstrate that CIPO consistently and significantly outperforms strong baselines in both reasoning and correction performance. Moreover, CIPO yields stronger pass@K gains, indicating that it improves the model's intrinsic reasoning capacity rather than merely redistributing probability mass over existing correct answers.

Mengjie Ren, Jie Lou, Boxi Cao +6

Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xiaohongshu Inc.

RL for Code Generation LLM Self-Correction RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Ask PDF

May 20, 2026

Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation

Group Relative Policy Optimization (GRPO), a prominent algorithm within the Reinforcement Learning from Verifiable Rewards (RLVR) framework, has achieved strong results in improving the reasoning capabilities of large language models (LLMs). However, GRPO is prone to advantage collapse, a failure mode where homogeneous rewards within a group (e.g., all correct or all incorrect answers) yield near-zero advantages and vanishing gradients. To address this, we introduce the Advantage Collapse Rate (ACR), the first diagnostic metric quantifying the proportion of training batches with ineffective gradients. Across models from 0.5B to 14B parameters on mathematical reasoning benchmarks, we show that ACR strongly predicts training stagnation and final performance. We then propose Adaptive Virtual Sample Policy Optimization (AVSPO), a lightweight extension of GRPO that injects virtual reward samples, guided by real-time ACR monitoring, to enable learning from homogeneous groups without additional model rollouts. AVSPO reduces advantage collapse by 58-63% relative to GRPO and yields consistent accuracy gains of 4-6 percentage points across all model scales, while maintaining generalization on the evaluated out-of-domain task. Code and datasets are available at https://github.com/hexixiang/Advantage-Collapse-Rate.

Xixiang He, Qiyao Sun, Ao Cheng +5

National University of Defense Technology, Changsha, Hunan, China · Intelligent Game and Decision Lab, Beijing, China · The Chinese University of Hong Kong, Shenzhen, Guangdong, China

RL for Language Model Reasoning Group Relative Policy Optimization Policy Optimization Reinforcement Learning with Verifiable Rewards

Ask PDF

Abstract

RL for Language Model Reasoning

Explore similar work

LatentRevise: Learning from Zero-Hit Reasoning

Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards

Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation