Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning

Tiehua Mei; Minxuan Lv; Leiyu Pan; Zhenpeng Su; Hongru Hou; Hengrui Chen; Ao Xu; Deqing Yang

cs.LGMar 10, 2026

Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning

Authors: Tiehua Mei, Minxuan Lv, Leiyu Pan, Zhenpeng Su, Hongru Hou, Hengrui Chen, Ao Xu, Deqing Yang

Organizations: School of Data Science, Fudan University · University of Chinese Academy of Sciences · College of Intelligence and Computing, Tianjin University

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in large language models but treats all correct solutions equally, potentially reinforcing flawed traces that arrive at correct answers by chance. We observe that better reasoning makes better demonstrations: high-quality solutions serve as more effective in-context examples than low-quality ones. We term this teaching ability Demonstration Utility, and show that the policy model's own in-context learning ability provides an efficient way to measure it, yielding a quality signal termed Evidence Gain. To leverage this signal during training, we introduce In-Context RLVR, which prepends demonstrations before each rollout. Theoretically, we prove that this simple input modification implicitly reweights rewards by a factor approximately proportional to Evidence Gain, assigning higher weights to high-quality traces without requiring costly computation. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements in both accuracy and reasoning quality over standard RLVR baselines. Our codes and datasets are available at https://github.com/Mithas-114/IC-DAPO.

Figures & tables

Figure 1: Mean Evidence Gain by quality score with 95% confidence intervals on two models.

	Length	LogProb	MajorVote	$\Delta$ (Ours)
1.5B	$-$ 0.147	0.129	0.079	0.405
7B	$-$ 0.161	0.178	0.109	0.444

Table 1: Spearman correlation (

\rho

) between proxy signals and reasoning quality. Evidence Gain achieves stronger correlation than alternatives.

Method	AIME24	AIME25	HMMT25	MATH500	AMC23	Olympiad	Average	Time/Step (s)
DS-R1-Distill-Qwen-1.5B	29.2	24.1	13.1	86.0	73.7	51.8	46.3	–
+ GRPO ( Shao et al., 2024 )	33.4	28.1	16.6	88.3	79.3	56.2	50.3	457.4
+ IC-GRPO (Ours)	38.3 $\boldsymbol{\uparrow}$	30.6 $\boldsymbol{\uparrow}$	17.7 $\boldsymbol{\uparrow}$	89.5 $\boldsymbol{\uparrow}$	82.5 $\boldsymbol{\uparrow}$	56.9 $\boldsymbol{\uparrow}$	52.6 $\boldsymbol{\uparrow}$	461.8
+ DAPO ( Yu et al., 2025a )	40.0	28.4	19.2	90.0	84.4	61.6	53.9	459.6
+ CISPO ( MiniMax et al., 2025 )	32.9	25.1	13.2	85.8	80.9	54.9	48.8	466.3
+ GSPO ( Zheng et al., 2025 )	42.5	33.6	19.0	90.3	85.9	62.6	55.7	437.3

Table 2: Performance comparison across mathematical reasoning benchmarks. Bold and underline indicate the best and second-best results respectively.

\boldsymbol{\uparrow}

denotes improvement over the corresponding baseline (GRPO or DAPO). Notably, training times are incomparable across scales due to different GPU configurations (32 vs. 128 GPUs). Given this high GPU requirements at 7B scales, GSPO and CISPO are evaluated only at 1.5B.

Figure 2: Training dynamics of Evidence Gain, quality score, and their correlation (

\rho

) across training steps.

Method	Scale	Easy	Medium	Hard
DAPO	1.5B	98.3	90.1	23.1
IC-DAPO	1.5B	98.8 (+0.5%)	93.5 (+3.8%)	26.0 ( +12.6% )
DAPO	7B	98.6	97.8	39.2
IC-DAPO	7B	99.3 (+0.7%)	98.2 (+0.4%)	43.2 ( +10.2% )

Table 3: Difficulty-stratified analysis. Parenthetical values denote relative improvement over DAPO.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	Definition
Repetition	Same steps or ideas repeated
Redundancy	Unnecessary or verbose content
Logical Consistency	Contradictions
Relevance	Off-topic exploration
CoT-Ans Alignment	Answer derived from reasoning
Reasoning Rigor	Claims justified without leaps

Appendix

Table 4: Quality evaluation dimensions.

Figure 3: Spearman correlation between Evidence Gain and individual quality dimensions, alongside mean scores for each dimension. Evaluated on DeepSeek-R1-Distill-Qwen-1.5B with the same setup as Section 2 .

Method	AIME24	AIME25	HMMT25	MATH500	AMC23	Olympiad	Average
DS-R1-Distill-Qwen-1.5B	29.2	24.1	13.1	86.0	73.7	51.8	46.3
+ DAPO ( Yu et al., 2025a )	40.0	28.4	19.2	90.0	84.4	61.6	53.9
+ IC-DAPO (V3.1)	44.5 $\boldsymbol{\uparrow}$	32.3 $\boldsymbol{\uparrow}$	19.5 $\boldsymbol{\uparrow}$	90.3 $\boldsymbol{\uparrow}$	85.8 $\boldsymbol{\uparrow}$	61.7 $\boldsymbol{\uparrow}$	55.7 $\boldsymbol{\uparrow}$
+ IC-DAPO (R1)	45.6 $\boldsymbol{\uparrow}$	34.2 $\boldsymbol{\uparrow}$	19.7 $\boldsymbol{\uparrow}$	90.6 $\boldsymbol{\uparrow}$	86.2 $\boldsymbol{\uparrow}$	62.1 $\boldsymbol{\uparrow}$	56.4 $\boldsymbol{\uparrow}$
DS-R1-Distill-Qwen-7B	54.5	39.1	26.2	93.6	90.6	67.0	61.8
+ DAPO ( Yu et al., 2025a )	62.0	45.9	27.4	94.1	92.3	69.9	65.3

Appendix

Table 5: Ablation on demonstration quality. IC-DAPO (R1) uses refined reasoning traces from DeepSeek-R1, while IC-DAPO (V3.1) uses solutions from DeepSeek-V3.1, a non-reasoning model. Both variants outperform DAPO, with R1 demonstrations yielding stronger results. Bold and underline indicate best and second-best results.

\boldsymbol{\uparrow}

denotes improvement over DAPO.

Figure 4: Spearman correlation matrix between DeepSeek-V3.2 quality scores and four human expert ratings on 100 sampled reasoning traces, together with inter rater correlations among experts.

Method	Scale	HumanEval	LCB	IFBench	IFEval	MMLU
DAPO	1.5B	73.6	30.4	9.1	38.3	46.6
IC-DAPO	1.5B	75.5	31.9	10.7	40.6	48.3
DAPO	7B	90.5	48.5	13.3	56.7	65.7
IC-DAPO	7B	92.3	49.8	14.2	58.6	67.3

Appendix

Table 6: Cross-domain zero-shot evaluation on code generation (HumanEval, LCB), instruction following (IFBench, IFEval), and general knowledge (MMLU).

Figure 5: Training dynamics across 1.5B and 7B models. IC-DAPO variants consistently outperform DAPO on AIME24 and AIME25 while maintaining stable entropy throughout training.

Panel of Figure 6 — Figure 6: Quality Evaluation Prompt Template. Placeholders {question} and {response} are replaced with the actual math problem and reasoning trace during evaluation.

Figure 7: Empirical verification of the log-linear relationship between

\log w(q,r)=\log\mathbb{E}_{e}[\exp(\Delta_{e})]

and

\Delta(q,r)=\mathbb{E}_{e}[\Delta_{e}]

. Strong Pearson correlations (

r=0.855

for 1.5B,

r=0.822

for 7B) confirm that Evidence Gain serves as a reliable proxy for the implicit weight.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short
Jun 8, 2026Han Zhou, Adam X. Yang, Laurence Aitchison +2Pairwise Preference Learning Reward Modeling
University of Cambridge · 2Mistral AI · *Work done as an AI Scientist Intern at Mistral AI.
Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning
Apr 23, 2026Qinan Yu, Alexa Tartaglini, Peter Hase +2CoT Faithfulness Reinforcement Learning with Verifiable Rewards
1Stanford University · 2Schmidt Sciences
Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning
Jun 2, 2026Ziyue Wang, Aomufei Yuan, Yongfu Zhu +10LLM Alignment RL for Language Model Reasoning
1Peking University · 2JINGDONG · 3Shanghai Innovation Institute +2

Jun 8, 2026J/K move · Enter open · S save

Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short

Han Zhou, Adam X. Yang, Laurence Aitchison, Anna Korhonen+1

University of Cambridge · 2Mistral AI · *Work done as an AI Scientist Intern at Mistral AI.

Pairwise Preference Learning Reward Modeling RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Bradley–Terry Model Reward Design for RL+1

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has become a leading paradigm for improving the reasoning ability of large language models through outcome-based supervision. However, verifiable rewards frequently become uninformative at the group level: when all sampled traces of a given prompt receive identical rewards, group-relative advantage estimation provides no gradient signal, even though the traces may differ substantially in reasoning quality. We propose Reasoning Arena, an adaptive training framework that routes such non-diverse reward groups to a judge system instead of discarding them. Beyond examining the final answer, Reasoning Arena constructs trace tournaments, where reasoning traces are compared head-to-head to expose finer-grained preferences within the group, converting reasoning quality into rich relative reward signals. To make reward estimation efficient, rather than exhaustively comparing every pair, each new trace is evaluated against a small, dynamically updated pool of previously generated traces as anchors to efficiently establish a relative ranking. We then fit a Bradley-Terry model on the incomplete comparison graph, enabling scalable RL integration without quadratic pairwise comparisons. Empirical results demonstrate that Reasoning Arena consistently outperforms the RLVR baseline by 7.6% on average in competition mathematics and coding benchmarks. By converting otherwise wasted zero-advantage samples into useful gradient updates, our method accelerates training by 27% to 41%, saving nearly 50% of generation compute, and substantially improves overall reasoning performance.

Abstract

RL for Language Model Reasoning

Explore similar work

Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short

Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning