Large language models (LLMs) achieve strong downstream performance largely due to abundant supervised fine-tuning (SFT) data. However, high-quality SFT data in knowledge-intensive domains such as humanities, social sciences, medicine, law, and finance is scarce because expert curation is expensive, privacy constraints are strict, and label consistency is hard to ensure. Recent work uses synthetic data, typically by prompting a generator over domain documents and filtering outputs with handcrafted rubrics. Yet rubric design is expert-dependent, transfers poorly across domains, and is often optimized through a brittle heuristic loop of writing rubrics, synthesizing data, training, inspecting results, and manually guessing revisions. This process lacks reliable quantitative feedback about how a rubric affects downstream performance. We propose evaluating synthetic data by its training utility on the target model and using this signal to guide data generation. Inspired by influence estimation, we adopt an optimizer-aware estimator that uses gradient information to quantify each synthetic sample's contribution to a target model's objective on specific tasks. Our analysis shows that even when synthetic and real samples are close in embedding space, their influence on learning can differ substantially. Based on this insight, we propose an optimization-based framework that adapts rubrics using target-model feedback. We provide lightweight guiding text and use a rubric-specialized model to generate task-conditioned rubrics. Influence score is used as the reward to optimize the rubric generator with reinforcement learning. Experiments across domains, target models, and data generators show consistent improvements and strong generalization without task-specific tuning.
Figures & tables
Figure 1: Visualization of influence score as an effective feature for assessing sample utility. (a) Relationship between validation influence scores and training performance. (b) Distribution of three validation sets in the embedding space of Qwen3-8B-Base. (c) Distribution of three validation sets in the loss–gradient embedding space, showing closer alignment and shared optimization directions.
Figure 2: The proposed RL framework for synthetic data generation with optimizer-aware rewards. (a) RL workflow. Starting from a seed document, a prompter instantiates G seed-specific rubrics, and a generator produces synthetic QA pair. (b) Reward computation. We employ the influence score for each synthetic QA pair as reward, followed by policy update.
Figure 3: Domain distribution of seed documents used to synthesize question-answering pairs: Humanities and Social Sciences and Medical and Health .
Humanities and Social Sciences (HSS)
Model
MMLU pro
Super GPQA
HLE
BBH
HellaSwag
DROP
Qwen3-8B-Base
22.83
20.77
5.70
35.62
35.71
45.73
Qwen3-8B-Instruct
49.87
23.44
4.66
85.21
78.71
80.73
MAGACorpus
39.90
21.18
8.29
58.21
72.88
62.53
Bonito
39.37
21.36
5.70
45.78
83.00
50.95
Conder
41.99
23.29
6.22
74.21
59.32
41.52
Table 1: Results across two domains. All metrics are accuracy (higher is better). Across both HSS and Medical domains, our method consistently upgrades Qwen3-8B-Base beyond widely used open SFT corpora, often rivaling or surpassing Qwen3-8B-Instruct.
Figure 4: The performance of our framework across different target model scales and families.
Dataset
Total samples
Token mean
Token std
MTLD
HDD
LongWriter
6,000
1823.45
452.20
80.63
0.8954
WildChat
529,428
289.59
344.77
52.70
0.9188
Condor
20,000
428.79
35.70
101.48
0.8650
Cosmopedia
50,000
773.85
44.94
111.52
0.8843
Openhermes 2.5
1,001,551
236.14
249.04
106.43
0.8887
SynthQuestions
2,500
634.60
17.88
137.02
0.8584
Table 2: Statistics of different datasets ( Medical ).
Figure 5: The performance of our framework across different generators.
Figure 6: Effects of the Proposed Method on Rubrics
Figure 9
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Method
MMLU-pro
Super GPQA
HLE
PubMed
MedQA
Health Bench
Text-match
0.000
0.000
0.000
0.000
0.000
0.000
5-gram (0.3)
0.002
0.004
0.000
0.005
0.009
0.001
Embedding (0.85)
0.000
0.001
0.000
0.001
0.009
0.002
Appendix
Table 3: Estimated contamination scores on medical and health benchmarks under different decontamination modes.
Method
MMLU-pro
Super GPQA
HLE
BBH
HellaSwag
DROP
Text-match
0.000
0.000
0.000
0.000
0.000
0.000
5-gram (0.3)
0.006
0.004
0.001
0.000
0.000
0.000
Embedding (0.85)
0.003
0.001
0.000
0.000
0.000
0.000
Appendix
Table 4: Estimated contamination scores on general reasoning benchmarks under different decontamination modes.
Method
MMLU-pro
Super GPQA
HLE
Health Bench
PubMed
MedQA
Base
0.4597
0.2806
0.1036
0.7006
0.6590
0.5145
Monressor-Instruct
0.4672
0.2728
0.0946
0.6453
0.7027
0.5204
Monressor (inter 2)
0.5028
0.3105
0.1047
0.6893
0.7458
0.5469
Ours
0.5369
0.3549
0.1096
0.7062
0.7859
0.5206
Appendix
Table 5: Performance comparison of different influence-based methods.
Method
MMLU-pro
Super GPQA
HLE
Health Bench
PubMed
MedQA
Base-Model
0.4597
0.2806
0.1036
0.7006
0.6590
0.5145
QuRating
0.4954
0.3032
0.1118
0.7106
0.7356
0.5534
FineWeb-Edu
0.4682
0.2892
0.1064
0.7340
0.7044
0.5628
PRRC
0.5317
0.3472
0.0914
0.7556
0.7568
0.5247
Ours
0.5697
0.3828
0.1081
0.7482
0.8070
0.5875
Appendix
Table 6: Comparison of different reward / feedback mechanisms.
Figure 10: The impact of training data volume on model performance.
Scale
MMLU-pro
Super GPQA
Health Bench
PubMed
MedQA
HLE
T-test
8B-base
0.4597
0.2806
0.7006
0.6590
0.5145
0.1036
0.0112 ∗
100
0.5236
0.3016
0.6759
0.6946
0.5141
0.0942
0.0046 ∗∗
300
0.5423
0.3096
0.6837
0.6480
0.4908
0.1006
0.0224 ∗
500
0.5318
0.3426
0.6995
0.7682
0.5738
0.0913
0.0023 ∗∗
800
0.5721
0.3739
0.7958
0.7835
0.5729
0.0982
0.9158
1000
0.5613
0.3663
0.7831
0.7961
0.6063
0.1057
0.7644
Appendix
Table 7: Effect of validation set size on downstream performance (8B model).
Strategy
MMLU-pro
Super GPQA
Health Bench
PubMed
MedQA
HLE
Random
0.5236
0.3016
0.6759
0.6946
0.5141
0.0942
Embedding-Based
0.5428
0.3138
0.6954
0.7284
0.5576
0.1036
EvalTree-Based
0.5601
0.3596
0.7434
0.7392
0.5327
0.0968
All
0.5697
0.3828
0.7482
0.8070
0.5875
0.1081
Appendix
Table 8: Comparison of different validation sampling strategies (100-sample subsets vs. using all validation data).
Figure 11: Comparison of SFT results under the same computational budget.
Figure 12: Instruction template for generating a QA pair.
Figure 13: Prompt template for analyzing quality dimensions before QA generation.
Figure 14: Instruction template for generating a QA pair.
Dataset
Tokens (M)
Num.
ChatDoctor
246.3
100K
MedQA
15.7
12.7k
Medical-o1
557.9
19.7k
Medical-R1-Distill
579.4
22k
ReasonMed
645.4
370k
OptimSyn
125.7
26.6k
Appendix
Table 9: Data Statistics.
Figure 15: Prompt template for generating QA data in the HSS and Medical domain.
Figure 16: An example of generated rubrics for data synthesis.
Figure 17: An example of generated rubrics for data synthesis.
What are the limits of controlling language models via synthetic training data? We develop a reinforcement learning (RL) primitive, the Dataset Policy Gradient (DPG), which can precisely optimize synthetic data generators to produce a dataset of targeted examples. When used for supervised fine-tuning (SFT) of a target model, these examples cause the target model to do well on a differentiable metric of our choice. Our approach achieves this by taking exact data attribution via higher-order gradients and using those scores as policy gradient rewards. We prove that this procedure closely approximates the true, intractable gradient for the synthetic data generator. To illustrate the potential of DPG, we show that, using only SFT on generated examples, we can cause the target model's LM head weights to (1) embed a QR code, (2) embed the pattern 67, and (3) have lower ℓ2 norm. We additionally show that we can cause the generator to (4) rephrase inputs in a new language and (5) produce a specific UUID, even though neither of these objectives is conveyed in the generator's input prompts. These findings suggest that DPG is a powerful and flexible technique for shaping model properties using only synthetic training examples.
Rubric evolution offers a promising approach to improving the quality of rubrics generated by large language models (LLMs). Central to this process is rubric comparison, which identifies the better of two rubrics and guides the direction of evolution. However, accurate rubric comparison is difficult, which presents two challenges. (1) It should reflect downstream task performance, which is essential for assessing rubric utility but often prohibitively expensive to evaluate. (2) It should discourage unnecessary criteria, which increase verification costs and may dilute the influence of essential criteria. To address these challenges, we introduce Rubrics on Trial, a multi-agent framework that evolves rubrics by comparing synthetic response pairs. To address challenge 1, the framework compares synthetic responses that satisfy the respective rubrics, providing a proxy for downstream performance without training a separate policy for each rubric. To address challenge 2, it assesses the necessity of a candidate criterion by independently generating high-quality alternative responses that violate it and comparing them with edited versions that satisfy it. A rubric is favored when it improves response quality in both comparisons, and the resulting comparison signal is further incorporated for rubric evolution. Extensive experiments demonstrate that Rubrics on Trial improves the quality of generated rubrics and leads to better downstream task performance.
Haocheng Yang, Licheng Pan, Yuan Lu +7
School of Computing, National University of Singapore · Xiaohongshu Inc. · School of Cyber Science and Technology, Zhejiang University +3
Synthetic data now is essential to LLM training, used to strengthen advanced capabilities such as autonomous and long-horizon task execution. Yet recent work shows that training on it at scale can degrade model generation, making it important to decide what synthetic data is worth training on. While current data curation practices do so with individual-level signals (i.e., estimates of each data sample's training utility in isolation), across pre-training and post-training settings we show that this is insufficient for synthetic data, and that group-level signals (i.e., estimates of utility that account for interactions among data samples) are necessary for effective data curation. First, we show that individual-level signals are blind to how samples jointly affect training: synthetic datasets with different compositions can be indistinguishable under individual-level influence yet differ sharply under group-level influence, and curating by the latter yields better downstream performance, particularly in generative capability. Second, we find that group-level signals matter more as training pipelines become increasingly synthetic: among widely used data curation methods, only those incorporating them improve over baseline, with gains increasing when weights capturing relations among samples are amplified. Finally, we translate these findings into practice -- for model developers under a compute budget, we offer a cheap diagnostic that prioritizes which groups of synthetic data most need group-level estimation, recovering much of the benefit of full group-level scoring at a fraction of the compute cost.
Cathy Jiao, Chenyan Xiong
Language Technologies Institute, Carnegie Mellon University