KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance

Linhao Yu; Tianmeng Yang; Siyu Ding; Renren Jin; Naibin Gu; Xiangzhao Hao; Shuaiyi Nie; Deyi Xiong

cs.AIApr 14, 2026

KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance

Authors: Linhao Yu, Tianmeng Yang, Siyu Ding, Renren Jin, Naibin Gu, Xiangzhao Hao, Shuaiyi Nie, Deyi Xiong,

Organizations: TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · Baidu Inc. · Institute of Information Engineering, Chinese Academy of Sciences.

Abstract

RLVR improves reasoning in large language models, but its effectiveness is often limited by severe reward sparsity on hard problems. Recent hint-based RL methods mitigate sparsity by injecting partial solutions or abstract templates, yet they typically scale guidance by adding more tokens, which introduce redundancy, inconsistency, and extra training overhead. We propose KnowRL (Knowledge-Guided Reinforcement Learning), an RL training framework that treats hint design as a minimal-sufficient guidance problem. During RL training, KnowRL decomposes guidance into atomic knowledge points (KPs) and uses Constrained Subset Search (CSS) to construct compact, interaction-aware subsets for training. We further identify a pruning interaction paradox -- removing one KP may help while removing multiple such KPs can hurt -- and explicitly optimize for robust subset curation under this dependency structure. We train KnowRL-Nemotron-1.5B from OpenMath-Nemotron-1.5B. Across eight reasoning benchmarks at the 1.5B scale, KnowRL-Nemotron-1.5B consistently outperforms strong RL and hinting baselines. Without KP hints at inference, KnowRL-Nemotron-1.5B reaches 70.08 average accuracy, already surpassing Nemotron-1.5B by +9.63 points; with selected KPs, performance improves to 74.16, establishing a new state of the art at this scale. The model, curated training data, and code are publicly available at https://github.com/Hasuer/KnowRL.

Figures & tables

Panel of Figure 1 — Figure 1: Three key challenges in hint-based RL. (a) Critical-segment effect: performance improves sharply once a short key hint segment appears, with diminishing returns beyond it. (b) Cross-hint inconsistency: longer prefixes or abstractions may introduce branching or ambiguity, expanding the reasoning search space. (c) Guidance-efficiency trade-off: abstraction-based hints often rely on teacher models or multi-stage curation, increasing computational overhead.

Panel of Figure 2 — Figure 2: Interaction-aware KP selection: inconsistency-induced degradation and the $\delta$ –compactness trade-off.

Panel of Figure 3 — Figure 3: Difficulty-bucket analysis on both test and training sets, where buckets are defined by no-KP accuracy. Full-KP injection shifts the violin distributions upward and improves mean performance in most buckets, but it also induces regressions on a subset of instances. In contrast, CSS-selected KPs deliver larger and more consistent gains across buckets. On the x-axis, $(n=\cdot)$ denotes the number of samples in each bucket, and the gray marker $\mu_{\mathrm{wo}}$ indicates the no-KP mean accuracy of that bucket.

Selection Strategy	AIME24	AIME25	BRUMO25	HMMT Feb 25	AMC23	CMIMC25	MATH-500	Olympiad Bench	Avg.	Avg. #KP
w/o KP	58.75	48.44	61.67	30.10	90.55	30.08	92.40	71.70	60.46	0.00
All KP	60.90	49.01	61.11	32.46	89.67	32.32	92.22	70.55	61.03	5.86
Random	60.52	49.27	61.04	33.23	91.02	31.09	91.65	71.88	61.21	2.53
Max-Score	62.63	49.79	64.27	34.79	90.94	32.99	92.52	73.89	62.73	2.61
S-LOO	62.71	49.22	63.88	33.54	91.71	33.52	92.90	73.70	62.65	1.72
T-LOO	62.11	49.27	64.20	33.65	91.25	33.67	92.40	73.46	62.50	1.20

Table 1: Offline KP selection strategies on Nemotron-1.5B. Avg.#KP denotes the average number of selected key knowledge points per problem. Green numbers indicate improvements over w/o KP.

KP Selection Strategy	Acc	Avg. #KPs
w/o KP	22.40	0
w/ all KP	26.93 +4.53	5.90
CBRS	33.05 +10.65	3.68 -37.7%
CSS	33.51 +11.11	3.61 -38.9%

Table 2: Offline evaluation on the QuestA dataset with different KP selection strategies and the average number of selected KPs.

Model	Hint Setting	AIME24	AIME25	BRUMO25	HMMT25	AMC23	CMIMC25	MATH	OlyBench	Avg.
Nemotron-1.5B ( Moshkov et al., 2025 )	w/o KP	59.06	48.33	60.73	30.63	90.70	30.08	92.35	71.70	60.45
	CBRS	63.02	49.00	64.17	34.79	91.56	33.57	92.65	73.89	62.94
	CSS	64.06	50.10	65.03	35.77	90.47	36.70	92.90	74.09	63.64
QuestA ( Li et al., 2025 )	w/o KP	71.56	62.08	67.5	40.94	93.44	41.48	92.95	72.28	67.78
	CBRS	74.23	62.00	73.23	43.78	95.10	46.12	93.94	78.45	70.86
	CSS	74.26	64.99	73.75	44.35	95.08	47.64	94.05	78.53	71.58

Table 3: Evaluation results of RL training with CSS-selected KP data under different test-time prompting strategies (with and without KPs). All scores are evaluated using the protocol described in Section 4.3 . For QuestA and JustRL, the w/ KP scores are taken directly from the JustRL paper.

Figure 4: Distribution of per-query correct counts on the training set for OpenMath-Nemotron-1.5B, and KnowRL-Nemotron-1.5B model under two offline evaluation settings: without KP hints and with KP hints at inference.

Figure 5: Comparison of KP selection strategies under the same training budget.

Model	AIME24	AIME25	BRUMO25	HMMT25	AMC23	CMIMC25	MATH	OlyBench	Avg.
CBRS step400	64.58	56.77	63.96	34.48	93.52	35.39	93.30	75.41	64.68
CSS step400	65.94	57.08	64.38	35.31	92.73	36.25	92.85	75.46	65.00
CBRS step900	65.42	58.85	64.79	37.08	94.14	35.70	94.00	75.78	65.72
CSS step900	67.19	59.17	65.52	39.06	93.91	37.03	93.77	76.04	66.46

Table 4: Comparison between CBRS- and CSS-selected training data under matched training budgets (steps 400 and 900) across eight reasoning benchmarks.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Comparison with and without entropy annealing. We report the entropy trajectory and the corresponding performance on different validation benchmarks under the 24k setting during training. Entropy annealing yields faster entropy reduction and consistently better validation performance.

Figure 7: Visualization of the critical-segment effect across prefix ratios on 50 training instances.

Panel of Figure 8 — Figure 8: Prompt used for extracting raw knowledge points.

Panel of Figure 9 — Figure 9: Prompt used for leakage verification with an augmented hint.

Panel of Figure 10 — Figure 10: Example augmented prompt with a partial-solution hint

$\delta$	AIME24	AIME25	BRUMO25	HMMT Feb 25	AMC23	CMIMC25	MATH-500	Olympiad Bench	Avg.	Avg. #KP
0/32	63.13	49.90	64.06	34.48	91.60	33.40	92.55	73.95	62.88	1.19
1/32	64.44	50.57	65.03	35.77	91.71	36.70	92.90	74.11	63.90	2.57
2/32	64.05	50.30	64.80	35.20	91.33	35.90	92.85	73.70	63.52	3.45

Appendix

Table 5: Effect of tolerance threshold

\delta

on offline performance and KP compactness.

\delta=1/32

provides the best balance between average accuracy and average number of selected KPs.

Model	AIME24	AIME25	BRUMO25	HMMT25	AMC23	CMIMC25	MATH	OlyBench	Avg.
KnowRL-Nemotron-1.5B	69.79	64.69	69.48	41.04	95.55	44.14	95.70	80.23	70.08
w/o entropy annealing	68.65	62.19	67.40	39.27	95.94	42.81	94.67	77.95	68.61
JustRL	69.69	62.92	66.88	40.63	96.02	41.72	94.15	76.59	68.58

Appendix

Table 6: Ablation of entropy annealing on KnowRL-Nemotron-1.5B across eight evaluation benchmarks.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts
Oct 10, 2025Xinyi Wang, Jinyi Han, Zishang Jiang +7Reinforcement Learning RL for Language Model Reasoning
School of Data Science, Fudan University · 2Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
May 7, 2026Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger +1RL for Language Model Reasoning Efficient Language Model Reasoning
University of Southern California · 2DEVCOM ARL
Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently
Jun 22, 2026Stanley Wei, Juno KimRL for Language Model Reasoning Efficient Language Model Reasoning
1Princeton University · University of California, Berkeley.

Oct 10, 2025J/K move · Enter open · S save

Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts

Xinyi Wang, Jinyi Han, Zishang Jiang, Tingyun Li+6

School of Data Science, Fudan University · 2Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University+1

Reinforcement Learning RL for Language Model Reasoning LLM-Guided RL RL Post-Training

Open paper AskPDF arXiv

Reinforcement Learning (RL) has become a key driver for enhancing the long chain-of-thought (CoT) reasoning capabilities of Large Language Models (LLMs). However, prevalent methods like GRPO often fail when task difficulty exceeds model capacity, leading to reward sparsity and inefficient training. Prior work attempts to mitigate this with off-policy data, but such methods often induce severe distributional mismatches that destabilize policy updates. In this work, we identify a core issue underlying these failures, which we term low training affinity, and introduce Affinity, the first quantitative metric for monitoring the compatibility between external guidance and the model's intrinsic policy. To address this, we propose HINT, an adaptive framework designed to enhance reasoning capabilities while explicitly preserving high Affinity. First, instead of revealing partial answers, HINT supplies Meta-Hints, which act as abstract cognitive scaffolding to guide the model in articulating solutions independently. Second, to ensure stability, we integrate Affinity-Aware Policy Optimization (AAPO), which dynamically modulates the learning objective based on the Affinity. Extensive experiments across diverse benchmarks demonstrate that HINT consistently outperforms strong baselines, while exhibiting superior stability and robust generalization to out-of-distribution tasks. Code is available at https://github.com/ViviqwerAsd/HINT.

Abstract

RL for Language Model Reasoning

Explore similar work

Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently