Dynamic Important Example Mining for Reinforcement Finetuning

Haoru Tan; Sitong Wu; Yanfeng Chen; Shizhen Zhao; Yang-Tian Sun; Tianjia Liu; Chirui Chang; Shaofeng Zhang

cs.AIAug 29, 2026

Dynamic Important Example Mining for Reinforcement Finetuning

Authors: Haoru Tan, Sitong Wu, Yanfeng Chen, Shizhen Zhao, Yang-Tian Sun, Tianjia Liu, Chirui Chang, Shaofeng Zhang,

Organizations: HKU · Hunyuan Team, Tencent · CUHK · Stanford

Abstract

Reinforcement fine-tuning (RFT) is increasingly used to strengthen the reasoning abilities of large models, yet its effectiveness is bound by how training data are selected and used. Most data-centric RFT methods rely on static or heuristic sample selection, implicitly assuming a sample's value is fixed over training. This overlooks the non-stationary dynamics of policy learning and can lead to suboptimal updates. We propose Dynamic Important Example Mining (DIEM), a principled and fully automated framework that makes data utilization adaptive throughout RFT. DIEM integrates two components into each optimization step: (i) a gradient-alignment importance estimator that efficiently approximates each sample's marginal contribution to policy improvement; and (ii) a constrained batch reweighting scheme that maximizes aggregate utility while preserving the update's gradient magnitude to stabilize optimization. Across several reasoning benchmarks, DIEM consistently outperforms strong static and dynamic baselines. The code will be released via https://github.com/hrtan/DIEM.

Figures & tables

Figure 1: The pipeline of our proposed dynamic important example mining (DIEM). Compared with the vanilla RFT pipeline, our DIEM introduces only two lightweight steps: (1) Dynamic Data Importance Measuring , which evaluates the importance of each sample within the current batch using our proposed influence estimator in Eq. 4 ; and (2) Dynamic Data Reweighting , which reallocates sample weights based on their importance. The reweighting step is formulated as a constrained optimization problem that preserves the overall gradient magnitude while emphasizing the most important samples (defined in Eq. 6 , and we further provide an efficient closed-form solution in Eq. 7 ). Overall, DIEM is highly efficient, intuitive, and broadly compatible with existing RFT algorithms ( Shao et al., 2024 ; Hu, 2025 ; Schulman et al., 2017 ; Schulman et al., 2015 ) .

Model	Method	MATH-500	Gaokao23en	AMC-23	AIME24	AIME25	Avg
Qwen-3-1.7B	GRPO ( Shao et al., 2024 )	59.9	48.4	37.2	7.9	3.4	31.36
	HVS ( Wang et al., 2025b )	60.2	48.9	37.8	7.9	3.8	31.72
	LIMR ( Li et al., 2025 )	60.6	49.5	38.4	8.2	4.3	32.20
	DIEM	61.3	51.1	39.7	7.9	5.5	33.10
Qwen2.5-3B	GRPO ( Shao et al., 2024 )	52.2	44.8	35.9	5.0	1.2	27.82
	HVS ( Wang et al., 2025b )	52.9	45.1	36.5	5.3	1.4	28.24

Table 1: Performance comparison on different benchmarks across various models.

Base Model: Qwen2.5-VL-7B
Model	MathVista	MathVerse	MathVision	MMStar	MMMU	AI2D	Avg
GPT-4o ( Hurst et al., 2024 )	63.8	50.8	30.4	65.1	70.7	84.9	60.9
Claude3.7-Sonnet ( Anthropic, 2024 )	66.8	52.0	41.3	–	71.8	–	–
GPT-5-nano ( OpenAI, 2025 )	73.1	66.6	59.7	–	72.6	–	–
Qwen2.5-VL-7B ( Bai et al., 2025 )	68.2	49.2	25.1	63.9	58.6	83.9	58.2
Vanilla RFT ( Shao et al., 2024 )	74.1	51.7	27.6	65.6	57.1	78.4	59.1

Table 2: To examine the effectiveness of our DIEM algorithm, we compare our method against other data selection algorithms, and also famous large models. The selected benchmarks include both domain-specific and general-purpose tasks.

Figure 2: The speed test of various data selection methods when combined with Vanilla RFT (GRPO) ( Shao et al., 2024 ) .

Replace the dynamic data influence score with:	Performance
No Replacement (Ours)	58.0
Random value	53.0
Pass@k score ( Zhang et al., 2025b )	53.2
Pass@k score to median distance	54.9
Difficulty score ( Gao et al., 2025 )	52.1
Difficulty score to median distance	53.6

Table 3: Ablation study on MathVerse ( Zhang et al., 2024a ) with Qwen-2.5-VL-32B ( Bai et al., 2025 ) as the base model. Results on more benchmarks could be seen in the supplementary materials.

Figure 3: Normalized weight trajectories of different difficulty samples (groups) during DIEM Training.

RL Fine-Tuning

Reinforcement learning fine-tuning adapts pretrained models and policies through task rewards, preference feedback, or environment interaction.

See all 230 papers

Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models
Mar 11, 2026Yixiu Mao, Yun Qu, Qi Wang +2Prompt Selection for RL RL for Language Model Reasoning
Department of Automation, Tsinghua University
Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts
Aug 13, 2025Maxime Heuillet, Yufei Cui, Boxing Chen +2RL for Language Model Reasoning Efficient Language Model Training
Universit´e Laval (IID), Canada · Mila - Qu´ebec AI Institute, Canada · Huawei Noah’s Ark Lab (Montreal Research Center), Canada +1
Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner
Mar 18, 2026Hao Ma, Zhiqiang Pu, Yang Liu +1RL Fine-Tuning RL for Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences

Mar 11, 2026J/K move · Enter open · S save

Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models

Yixiu Mao, Yun Qu, Qi Wang, Heming Zou+1

Department of Automation, Tsinghua University

Prompt Selection for RL RL for Language Model Reasoning RL Fine-Tuning Large Reasoning Models

Open paper AskPDF arXiv

Reinforcement learning (RL) finetuning has become a key technique for enhancing the reasoning abilities of large language models (LLMs). However, its effectiveness critically depends on the selection of training data. Recent advances underscore the importance of online prompt selection methods, which typically concentrate training on partially solved or moderately challenging examples under the current policy, thereby yielding more effective model updates. While significantly accelerating RL finetuning in terms of training steps, they also incur substantial computational overhead by requiring extensive LLM rollouts over large candidate batches to identify informative samples, an expense that can outweigh the finetuning process itself. To address this challenge, this work proposes Dynamics-Predictive Sampling (DPS), which online predicts and selects informative prompts by inferring their learning dynamics prior to costly rollouts. Specifically, we introduce a new perspective by modeling each prompt's solving progress during RL finetuning as a dynamical system, where the extent of solving is represented as the state and the transition is characterized by a hidden Markov model. Using historical rollout reward signals, we perform online Bayesian inference to estimate evolving state distributions, and the inference outcome provides a predictive prior for efficient prompt selection without rollout-intensive filtering. Empirical results across diverse reasoning tasks, including mathematics, planning, and visual geometry, demonstrate that DPS substantially reduces redundant rollouts, accelerates the training process, and achieves superior reasoning performance.

Abstract

RL Fine-Tuning

Explore similar work

Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models

Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts

Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner