AdaPRA: Process Reward Agents with Adaptive Rubrics for Data Science

Atharva Naik; Vivek Chari; Pol Llado; Daniel Fried; Carolyn Rosé; Benjamin Van Durme; Sam Thomson; Justin Svegliato

cs.AIOct 23, 2026

AdaPRA: Process Reward Agents with Adaptive Rubrics for Data Science

Authors: Atharva Naik, Vivek Chari, Pol Llado, Daniel Fried, Carolyn Rosé, Benjamin Van Durme, Sam Thomson, Justin Svegliato

Organizations: Carnegie Mellon University · Microsoft

Abstract

Data science agents can automate end-to-end machine learning workflows, yet they remain far from competitive on human leaderboards. Agents make process errors such as insufficient exploration, unsound validation, coding bugs, and poor use of a fixed budget (re-running the same exploratory data analysis and baseline models in every attempt). We propose AdaPRA to address these issues: a persistent coding agent harness that accumulates knowledge across attempts on the same task, with rubric-guided process supervision that identifies and corrects errors. Our process reward agent audits the agent's trajectory and workspace by executing checks, and adapts its rubrics to each task. We find that rubrics seeded from easier tasks generalize to harder tasks, with rubric adaptation further improving performance. With five attempts per task, AdaPRA improves an 80B Qwen model's leaderboard percentile by 24.5 points on DSPredict-Easy and 14.8 points on DSPredict-Hard while reducing process errors. We also show that supervision with rubrics mined from one base model's runs improves another (DeepSeek-V3.2), and that AdaPRA beats DataPRM, a competitive PRM for data science. We also share our code for reproducibility.

Figures & tables

Figure 1: AdaPRA corrects process errors. Here, the agent must classify voice clips into 12 commands. A wrong file path makes every test clip load as silent audio, so every prediction is classified as the “unknown” command even though the trajectory indicates no errors. To address this, AdaPRA checks the predictions file under the Submission Integrity rubric, identifies the cause, and the agent fixes it.

Figure 2: AdaPRA overview. The agent works in a workspace that persists across attempts. During each attempt, the PRA reads the trajectory and workspace, runs check scripts in its own workspace, injects feedback on failing rubrics into the agent’s trajectory, and adds, updates, or retires rubrics in the per-task store.

Process supervision on top of Our Harness
	Best attempt				Per-attempt (mean)
Method	Pct.	Valid	>Median	Medal	Pct.	Valid	>Median	Medal
Qwen3-Coder-Next + DSGym	44.5	76.3	50.0	2.6	36.3	41.0	35.6	1.1
Qwen3-Coder-Next + Ours	60.8	100.0	71.1	7.9	51.3	84.5	53.9	2.6
+ DataPRM	54.6	100.0	65.8	5.3	44.8	100.0	44.6	2.1
+ PRA + adaptive rubrics ( AdaPRA )	69.0	100.0	86.8	7.9	61.0	100.0	73.2	3.7

Table 1: DSPredict-Easy: Our harness (Ours), with and without process supervision, compared with the DSGym harness. Best attempt is the maximum over a task’s five sequential attempts, and per-attempt averages each task’s scored attempts; all rates use a fixed denominator of 38 tasks (190 attempts). Qwen3-Coder-480B and GPT-5.1 (high) are single-attempt DSGym runs Nie et al. (2026) , so they have no best-attempt results; their percentile is rescaled to our fixed denominator (multiplied by the valid rate), their other rates are as reported, and DSGym reports no Easy medal rates.

Process supervision on top of Our Harness
	Best attempt				Per-attempt (mean)
Method	Pct.	Valid	>Median	Medal	Pct.	Valid	>Median	Medal
Qwen3-Coder-Next + DSGym	21.3	68.5	11.1	3.7	25.2	36.3	11.8	3.0
Qwen3-Coder-Next + Ours	25.5	81.5	18.5	5.6	26.2	64.8	14.4	3.7
+ DataPRM	23.1	88.9	14.8	3.7	21.3	79.3	12.8	3.7
+ PRA + adaptive rubrics ( AdaPRA )	36.1	90.7	33.3	5.6	32.1	86.3	24.1	4.1

Table 2: DSPredict-Hard: As Table 1 , on the 54 competitions of DSPredict-Hard (fixed denominator of 54 tasks, 270 attempts for per-attempt). DSGym reports medal rates but not percentiles on DSPredict-Hard, so the Pct. of its single-attempt runs is also –, and their rates are shown as published

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Domain	Operationalization	Rubric store
		execute vs. read	fixed vs. adaptive
AgentV Zhang et al. (2026a)	math	(✓)	$\times$
RewardAgent Peng et al. (2025)	general	(✓)	$\times$
Agentic Rubrics Raghavendra et al. (2026)	SWE	(✓)	$\times$
GUI-PRA Xiong et al. (2025)	GUI	(✓)	$\times$
Process RM Sohn et al. (2026)	medical	(✓)	$\times$

Appendix

Table 3: Design coverage of related systems. ✓: the dimension is varied as an explicit design choice. (✓): the dimension is varied only by removing one component in an ablation.

\times

: fixed or not applicable. AdaPRA executes checks and adapts its rubrics to each task, and we ablate each component in § 5.3 .

Design choice	Alternative	AdaPRA
Operationalization	Read only ( $-$ PRA): the supervisor reads the workspace and trajectory and scores each rubric by inspection. It cannot write or run code.	Code execution : for each rubric the supervisor either writes and runs a check in its own writable scratch space or scores by inspection.
Rubric store	Fixed ( $-$ adaptive rubrics): the seed rubrics are used unchanged for the whole run.	Adaptive : tools to add, update, and retire rubrics edit a per-task store initialized with the seed rubrics.

Appendix

Table 4: Design choices of the supervisor. The right column is the configuration of AdaPRA . The

-

PRA and

-

adaptive rubrics ablations each replace one entry with its alternative.

Base agent (Qwen3-Coder-Next, temperature 0)			PRA (Claude Sonnet 4.6)
Parameter	Easy	Hard	Parameter	Easy	Hard
Attempts $N$	5	5	Cadence $c$ (agent turns)	10	20
Max turns per attempt	200	300	Review on finish	yes	yes
Wall-clock limit per attempt	3 h	4 h	Max turns per invocation	100	100
Silent-command timeout	10 min	10 min	Time limit per invocation	1 h	1 h
Max single-command timeout	30 min	2 h	Importance threshold ( $i_{j}\geq$ )	0.5	0.5

Appendix

Table 5: Run configuration on DSPredict-Easy and DSPredict-Hard. The silent-command timeout stops a command that produces no output for that long. Time spent in the PRA does not count against the agent’s wall-clock limit. The larger per-attempt budget on DSPredict-Hard matches the DSGym reference runs on that benchmark. Hard attempts are longer, so the PRA is invoked less often to bound its cost (§ 5.3 ).

Process supervision on top of Our Harness
	Leaderboard percentile by selection policy
Method	First	Last	Mean	Oracle
Qwen3-Coder-Next + Ours	48.4	52.9	51.3	60.8
+ PRA + adaptive rubrics ( AdaPRA )	56.4	62.9	61.0	69.0
$-$ adaptive rubrics	55.1	59.8	58.8	67.3
$-$ PRA	52.4	61.1	57.8	63.0

Appendix

Table 6: Leaderboard percentile on DSPredict-Easy under four attempt-selection policies over the 5-attempt beam, task-first averaged over 38 tasks. First takes the first attempt, with an invalid first attempt contributing 0, Last takes the last attempt with a scored submission, Mean averages the scored attempts (the expected result of a random choice), and Oracle takes the best attempt.

Process supervision on top of Our Harness
	Leaderboard percentile by selection policy
Method	First	Last	Mean	Oracle
Qwen3-Coder-Next + Ours	17.0	22.2	26.2	25.5
+ PRA + adaptive rubrics ( AdaPRA )	23.3	30.2	32.1	36.1
$-$ adaptive rubrics	21.1	29.4	30.2	31.8
$-$ PRA	25.1	30.4	30.9	31.8

Appendix

Table 7: As Table 6 , on DSPredict-Hard (54 tasks). First , Last , and Oracle use the fixed denominator, with unscored tasks contributing 0. Mean is the per-attempt percentile of Table 2 , averaged over tasks with at least one scored attempt, so it can exceed Last .

Best attempt (maximum percentile over a task’s valid attempts)
Condition ( vs. harness only)	$\Delta$ mean	Wins/Losses	Perm. $p$	Sign $p$	Wilcoxon $p$	95% CI
AdaPRA	$\mathbf{+8.2}$	$\mathbf{25/11}$	$\mathbf{<0.001}$	$\mathbf{0.014}$	$\mathbf{0.0005}$	$\mathbf{[+4.4,+14.4]}$
$-$ adaptive rubrics	$\mathbf{+6.5}$	$\mathbf{23/15}$	$\mathbf{0.017}$	$\mathbf{0.128}$	$\mathbf{0.030}$	$\mathbf{[+0.3,+13.8]}$
$-$ PRA	$+2.2$	$18/19$	$0.273$	$0.629$	$0.275$	$[-3.0,+10.4]$
Per attempt (mean percentile over a task’s valid attempts)
AdaPRA	$\mathbf{+9.7}$	$\mathbf{31/7}$	$\mathbf{<0.001}$	$\mathbf{<0.001}$	$\mathbf{<0.001}$	$\mathbf{[+5.1,+14.5]}$

Appendix

Table 8: Paired significance of each supervised condition against the harness alone on DSPredict-Easy (task-paired,

n=38

). The one-sided paired permutation test (

200{,}000

sign flips) is primary, with the one-sided sign test, the one-sided Wilcoxon signed-rank test, and a nested bootstrap 95% CI (

20{,}000

resamples) alongside. Ties are excluded from the win and loss counts. Bold marks comparisons significant at

p<0.05

by the permutation test.

Best attempt (maximum percentile over a task’s valid attempts)
Condition ( vs. harness only)	$\Delta$ mean	Wins/Losses	Perm. $p$	Sign $p$	Wilcoxon $p$	95% CI
AdaPRA	$\mathbf{+10.6}$	$\mathbf{33/15}$	$\mathbf{<0.001}$	$\mathbf{0.007}$	$\mathbf{<0.001}$	$\mathbf{[+5.3,+16.2]}$
$-$ adaptive rubrics	$\mathbf{+6.3}$	$\mathbf{25/18}$	$\mathbf{0.007}$	$\mathbf{0.180}$	$\mathbf{0.025}$	$\mathbf{[+1.7,+11.3]}$
$-$ PRA	$\mathbf{+6.3}$	$\mathbf{30/15}$	$\mathbf{0.009}$	$\mathbf{0.018}$	$\mathbf{0.017}$	$\mathbf{[+1.5,+12.2]}$
Per attempt (mean percentile over a task’s valid attempts)
AdaPRA	$\mathbf{+7.8}$	$\mathbf{36/12}$	$\mathbf{0.001}$	$\mathbf{<0.001}$	$\mathbf{<0.001}$	$\mathbf{[+2.8,+13.0]}$

Appendix

Table 9: As Table 8 , on DSPredict-Hard (task-paired,

n=54

). Tasks without a scored attempt enter as 0 so that every task is paired, which makes the per-attempt deltas slightly larger than the differences in Table 2 , where per-attempt means exclude such tasks.

Figure 3: Offline rubric discovery. The base agent’s attempts on each task are ranked by leaderboard percentile. An LLM judge (Claude Sonnet 4.6) reads each attempt’s trajectory and workspace, is told whether the attempt ranks in the top or bottom quarter of its task, and extracts good and bad behaviors with quoted evidence. A second call clusters the good behaviors of top attempts and the issues of bottom attempts across tasks into a MECE set of seed rubrics, with good and bad behaviors as evidence for the same rubric (prompts and settings in Appendix E.3 ).

#	Rubric	Category	Evidence	Source	Support	Expert match
1	Persistent CV Score Registry	Validation and metric	Artifact	Discovered	52	Yes
2	Best-CV Model Submission Selection	Validation and metric	Artifact	Discovered	44	Yes
3	Valid Cross-Validation Methodology	Validation and metric	Artifact	Discovered	41	Yes
4	Suspicious CV Score Investigation	Validation and metric	Trajectory	Discovered	18	No
5	Metric Calculation Bug Detection	Validation and metric	Trajectory	Discovered	8	No
6	Model Diversity and Ensembling	Exploration and model diversity	Artifact	Discovered, restated	28	Yes

Appendix

Table 10: Seed rubric metadata. Evidence : whether the rubric is decisively checkable from the workspace artifacts or requires the trajectory (11 artifact, 8 trajectory). Source : Discovered rubrics come from two-stage discovery on harness-only DSPredict-Easy runs; Discovered, restated rubrics were restated to judge the current attempt against all earlier attempts on the task; Hand-curated rubrics target cross-attempt failures. Support : number of the 519 behaviors from the top and bottom of each task’s score range (238 good behaviors, 281 issues) that stage two clustered into the rubric. Expert match : whether the rubric recovers a criterion of a hand-written expert rubric set (Yes), overlaps with one loosely (Partial), or has no expert counterpart (No).

Rubric and definition	Satisfied if	Violated if	Not applicable / insufficient evidence if
Persistent CV Score Registry ( cv_score_registry ). Whether the agent maintains a persistent cv_scores.json (or equivalent) updated after every experiment to enable objective cross-model comparison.	A machine-readable file (e.g., cv_scores.json) exists in the workspace containing CV scores for every trained model/ensemble, updated consistently throughout the run, and the agent uses it to identify the best model at submission time.	No such registry exists, or it is missing entries for major experiments, or it is overwritten per-run rather than accumulated, preventing reliable comparison across all experiments.	N/A: Only a single model was ever trained and no comparison was needed. Insufficient: The workspace files are not visible and the trajectory does not mention CV score tracking.
Best-CV Model Submission Selection ( best_model_submission_selection ). Whether the agent explicitly selects and submits the model with the best CV score rather than whatever was most recently trained or written to disk.	The final /submission/submission.csv is verifiably sourced from the model/experiment with the highest CV score in the registry, and the agent explicitly checked the registry before writing the final submission.	The submission corresponds to a later, lower-CV experiment that overwrote the best model’s output, or the agent selected the most-recent model without checking CV scores.	N/A: Only one model was trained so no selection was needed. Insufficient: It cannot be determined which model’s predictions are in the submission file.
Valid Cross-Validation Methodology ( valid_cv_methodology ). Whether the agent uses a proper out-of-fold or held-out cross-validation scheme that produces honest, unbiased CV score estimates.	All reported CV scores are computed using held-out fold predictions (OOF) that were not used during model fitting, with an appropriate split strategy for the task (stratified for classification, time-based for time series).	Any CV score in cv_scores.json is computed by evaluating a model on the same data it was trained on (in-sample), or ensemble weights are searched using the same OOF data without a secondary holdout, or the CV scheme is wrong for the task type (e.g., random KFold for time series).	N/A: No model training or CV was performed. Insufficient: The training code is not visible and CV methodology cannot be inferred from outputs.
Suspicious CV Score Investigation ( suspicious_cv_score_investigation ). Whether the agent investigates and correctly handles CV scores that are suspiciously better than all other models (a signal of data leakage or in-sample evaluation).	When a model’s CV score is substantially better than all others ( $>$ 20% relative improvement with no architectural explanation), the agent diagnoses whether it is legitimate or an artifact of leakage/in-sample evaluation, and either discards the model or confirms correctness with a separate holdout before using it for submission.	An obviously inflated CV score (e.g., perfect 1.0 AUC, RMSE 10x better than all others, correlation 0.96 with training labels) is accepted as valid, recorded in cv_scores.json, and used for submission or downstream ensembling without investigation.	N/A: No suspiciously anomalous CV scores appeared during the run. Insufficient: CV score values are not visible in the trajectory or workspace.
Metric Calculation Bug Detection ( metric_bug_detection ). Whether the agent detects and corrects bugs in metric calculation code before reporting or acting on CV scores.	When a CV score appears implausibly low or inconsistent with prior results (e.g., RMSLE of 0.01 when all prior models score ~0.13), the agent identifies and fixes the root cause in the metric code before using the score for model selection.	An obviously buggy metric (double log-transform, wrong target, train-set RMSE) produces a clearly impossible CV score that the agent reports as valid and uses for multiple turns before catching it (if ever).	N/A: No metric bugs occurred and all CV scores were plausible. Insufficient: The metric calculation code is not visible.

Appendix

Table 11: Seed rubric definitions: validation and metric. Text is verbatim as given to the PRA; Context is a preamble that precedes the labels.

Rubric and definition	Satisfied if	Violated if	Not applicable / insufficient evidence if
Model Diversity and Ensembling ( model_diversity_and_ensembling ). Whether the agent trains at least two distinct model families and combines them via ensemble or stacking, and whether model-family coverage across the best-of-N attempts is diverse rather than duplicated. Context: This is a best-of-N run (N=n_attempts); the submitted result is the MAX over attempts, so what matters is the diversity of the WHOLE beam, not just this attempt. Read the accumulated prior-attempt transcripts (traj_a{attempt}_t{turn}.txt in /reviewer/) and cv_scores.json to see which model families the earlier attempts already covered.	The workspace contains trained models from at least two distinct families (e.g., LightGBM + XGBoost, or GBDT + neural net), their OOF predictions are combined via weighted averaging or stacking with the ensemble CV compared against individual models, AND – across attempts – the best-of-N beam is not collapsing onto one family (i.e., this attempt either adds a family/architecture the beam has not tried or is early enough that convergence is not yet a concern).	Only one model family was trained, or models from multiple families were trained but never combined, or an ensemble combined weak+strong models in a way that degraded performance (e.g., blending with a model 2.5pp weaker), OR this attempt is re-training essentially the same model families/ensemble the prior attempts already built (duplicating the beam) when a distinct family could have been tried.	N/A: The competition or task type makes single-model submission standard (e.g., data-centric competition where model is fixed), or fewer than two attempts have run so cross-attempt diversity cannot yet be assessed. Insufficient: Training scripts and the prior-attempt transcripts are not visible.
Hyperparameter Tuning Beyond Defaults ( hyperparameter_tuning ). Whether the agent performs at least one principled hyperparameter search beyond default settings for the primary model family – while recognizing when tuning has already been exhausted in earlier best-of-N attempts. Context: This is a best-of-N run (N=n_attempts). Before scoring, consult the accumulated prior-attempt transcripts (traj_a{attempt}_t{turn}.txt in /reviewer/) and cv_scores.json to see whether hyperparameter search was ALREADY done in an earlier attempt and what it yielded.	EITHER the workspace contains at least one training script or cv_scores.json entry showing a model trained with non-default hyperparameters (e.g., num_leaves != 31, learning_rate != 0.05, or a grid/random/Optuna search) with the tuned CV compared against baseline, OR the agent correctly recognizes that prior attempts already tuned this family to saturation (no meaningful CV gain) and therefore spends its budget elsewhere (new family, features, diversity) rather than re-tuning.	The primary model uses default/near-default hyperparameters (num_leaves=31, max_depth=6, lr=0.05) with no documented search AND no earlier attempt tuned it either, OR a ’tuned’ variant produced CV identical to baseline. Do NOT reward re-running a hyperparameter search that a prior attempt already showed to be saturated – that is over-optimization, not diligence.	N/A: The task is data-centric (fixed model) or the budget was trivially small. Insufficient: Hyperparameter values are not visible in training scripts, the registry, or the prior-attempt transcripts.
Hedging Across Best-of-N With Diverse Model Classes ( single_model_class_hedging ). Whether the best-of-N attempts hedge across DIFFERENT modeling classes rather than converging on the same solution near finish – using the max-over-N structure to try a distinct approach (even a second-best one) instead of duplicating the leading attempt. Context: This is a best-of-N run (N=n_attempts) and the SUBMITTED result is the MAX over attempts, so a genuinely different modeling class that ends up second-best costs nothing, while a duplicate of an existing attempt wastes one of the N shots. Judge this ACROSS attempts using the accumulated prior-attempt transcripts (traj_a{attempt}_t{turn}.txt in /reviewer/), the final submissions / cached test predictions, and cv_scores.json. Only assess once enough attempts exist to see convergence (typically attempt index $\geq$ 2).	The attempts hedge across distinct modeling classes/architectures (e.g., GBDT ensemble in one, a neural net or a linear/kernel model or a very different feature representation in another), so the best-of-N beam covers meaningfully different solutions rather than one repeated recipe.	More than two of the attempts converge near finish on essentially the same solution – same model family and features, near-identical final predictions or CV, often reusing the same cached OOF/test preds – instead of at least one attempt deliberately trying a different modeling class to exploit best-of-N.	N/A: Fewer than three attempts have completed, or the task is data-centric / single-model by nature so diversifying the model class is not meaningful. Insufficient: The final solutions of the prior attempts cannot be compared from the transcripts, submissions, or registry.

Appendix

Table 12: Seed rubric definitions: exploration and model diversity. Text is verbatim as given to the PRA.

Rubric and definition	Satisfied if	Violated if	Not applicable / insufficient evidence if
Training Objective–Metric Alignment ( metric_objective_alignment ). Whether the agent aligns the model’s training objective or target transformation with the competition evaluation metric.	The training loss or target transformation matches the evaluation metric (e.g., log1p target for RMSLE, MAE objective for MedAE, stratified splits for imbalanced AUC, ordinal-aware loss for QWK).	Models are trained with a mismatched objective (e.g., RMSE on raw target for an RMSLE competition, MSE for QWK, no log transform for RMSLE), as evidenced in training scripts or cv_scores.json metadata.	N/A: The competition metric is accuracy or AUC and any standard classification objective is acceptable. Insufficient: Training scripts are not visible and metric alignment cannot be inferred.
Domain-Appropriate Preprocessing ( domain_appropriate_preprocessing ). Whether the agent applies preprocessing steps appropriate to the model type and task (avoiding scaling before tree models, using correct imputation, appropriate CV splits).	Preprocessing choices are appropriate: no feature scaling before tree-based models, correct imputation strategy for the target distribution, and CV split type matches the task (stratified for imbalanced classification, time-based for time series).	StandardScaler is applied before LightGBM/XGBoost/CatBoost, target NaN values are imputed with 0 for a forecasting task, or random KFold is used for a time-series problem where temporal leakage would inflate CV scores.	N/A: Only linear models were used (where scaling is appropriate) or there were no categorical/missing-value features. Insufficient: Preprocessing code is not visible.
Feature Engineering Investment ( feature_engineering_investment ). Whether the agent invests in at least one round of principled feature engineering beyond raw input columns, accounting for feature work already done in earlier best-of-N attempts. Context: This is a best-of-N run (N=n_attempts). Consult the accumulated prior-attempt transcripts (traj_a{attempt}_t{turn}.txt in /reviewer/), features.py/preprocess.py in the persistent workspace, and cv_scores.json to see what feature engineering earlier attempts already tried and whether it helped.	EITHER the workspace contains a feature-engineering script/module adding at least 3 non-trivial derived features (interactions, target encodings, lag features, domain transforms) with a CV comparison showing improvement over the raw-feature baseline, OR this attempt sensibly builds on / diverges from the feature work prior attempts established (e.g., a prior attempt found feature set A helped, so this attempt reuses A and explores a different modeling axis, or a prior attempt found FE gave no lift so this attempt reasonably invests its budget elsewhere).	All training scripts use only raw input columns (label-encoding/fillna only) and NO attempt – this one or an earlier one – engineered features despite a non-trivial feature space; OR this attempt merely re-derives the exact same features a prior attempt already built and validated, adding nothing new to the beam.	N/A: The competition is data-centric (fixed model) or the feature space is already maximal (e.g., pre-extracted embeddings). Insufficient: Feature-engineering code and the prior-attempt transcripts cannot be inspected.

Appendix

Table 13: Seed rubric definitions: preprocessing, target, and features. Text is verbatim as given to the PRA.

Rubric and definition	Satisfied if	Violated if	Not applicable / insufficient evidence if
OOF and Test Prediction Caching ( oof_prediction_caching ). Whether the agent caches OOF and test predictions as parquet files after each model training run to enable fast ensemble search without retraining.	The workspace preds/ directory contains OOF and test prediction files (parquet or similar) for every substantively trained model, and ensemble/stacking scripts load these cached files rather than retraining base models.	No OOF or test prediction files exist for trained models, or ensemble scripts retrain base models from scratch instead of loading cached predictions.	N/A: Only a single model was trained and no ensembling was attempted. Insufficient: The workspace preds/ directory contents cannot be verified from the available information.
Submission File Integrity Verification ( submission_integrity_verification ). Whether the agent explicitly verifies submission file integrity (row count, NaN absence, value range, column names) before concluding the attempt.	The trajectory contains an explicit check of the final /submission/submission.csv covering at least row count matching the test set size and absence of NaN values, with results logged.	No integrity check is performed, or the check reveals problems (wrong row count, NaNs, binary corruption, wrong column names) that are not corrected before finishing.	N/A: The attempt ended before any submission file was generated. Insufficient: It cannot be determined whether a verification step occurred.
CV Regression Detection and Revert ( regression_detection_and_revert ). Whether the agent detects when a subsequent experiment produces a CV regression and explicitly reverts the submission to the best-validated model.	The agent detects (via cv_scores.json comparison or file inspection) that the current submission.csv corresponds to a worse model than a previously computed best, and explicitly restores the best model’s predictions to /submission/submission.csv before finishing.	The final submission reflects a worse model because the agent did not check for regressions, or detected a regression but failed to revert before finishing.	N/A: Only one model was trained, making regression impossible. Insufficient: It cannot be determined whether a regression occurred or was addressed.
Avoiding Repeated Identical Crashes ( no_repeated_identical_crashes ). Whether the agent diagnoses and fixes the root cause of a crash rather than repeatedly re-running the same failing code.	When a script crashes, the agent identifies the root cause, applies a fix, and the same error does not recur more than once after the fix.	The same error (same exception type and cause, e.g., wrong keyword argument, dtype mismatch, array shape error) appears three or more times across successive turns, indicating the agent is re-running without root-cause diagnosis.	N/A: No crashes occurred during the trajectory. Insufficient: Error messages and fix attempts are not visible in the transcript.

Appendix

Table 14: Seed rubric definitions: efficiency, robustness, and submission (part 1 of 2). Text is verbatim as given to the PRA.

Rubric and definition	Satisfied if	Violated if	Not applicable / insufficient evidence if
Avoiding Churn Without Pivoting ( no_churn_without_pivot ). Whether the agent recognizes when an experimental axis has plateaued – WITHIN this attempt or ACROSS the best-of-N attempts – and pivots to a substantively different approach rather than repeating near-identical experiments. Context: This is a best-of-N run (N=n_attempts). Assess churn at BOTH scales, using the accumulated prior-attempt transcripts (traj_a{attempt}_t{turn}.txt in /reviewer/), cv_scores.json, and the submission ledger to reconstruct what earlier attempts already tried.	Within the attempt, after 2-3 consecutive experiments on the same axis (weight search, seed sweep) yield no meaningful CV improvement ( $<$ 0.001 relative) the agent pivots to a different approach (new model family, feature engineering, different stacking architecture); AND, across attempts, this attempt does not merely repeat an axis a prior attempt already saturated – it explores something the beam has not tried.	EITHER five or more consecutive experiments within the attempt make no substantive change and no meaningful CV gain (nearly identical cv_scores.json entries or repeated ’No improvement found’), OR this attempt re-runs the same saturated axis a prior attempt already exhausted (e.g., yet another ensemble-weight sweep on the same cached OOF predictions) instead of pivoting – the classic best-of-N beam collapse where multiple attempts converge on the same solution.	N/A: The agent only ran one or two experiments total and no prior attempt covered this axis. Insufficient: The number and variety of experiments cannot be determined from the transcript or the prior-attempt transcripts are unavailable.
Valid Submission Before Finishing ( no_premature_termination ). Whether the agent ensures a valid, complete submission file exists at /submission/submission.csv before concluding the attempt.	At the time of finish, /submission/submission.csv exists, is a valid CSV with the correct target column and row count matching the test set, and contains non-trivial (non-constant, non-NaN) predictions.	finish is called before any valid submission is written, or the submission file is empty, binary-corrupted, or contains wrong-competition predictions that make it invalid for the actual leaderboard.	N/A: The attempt is still in progress and no finish was called. Insufficient: The submission file contents cannot be verified.
Timeout-Aware Experiment Design ( timeout_aware_experiment_design ). Whether the agent designs experiments to complete within available time budgets, using prototyping or parameter adjustment rather than launching full training jobs that are repeatedly killed.	Before launching expensive training jobs (large datasets, many folds, many boosting rounds), the agent either runs a single-fold prototype to estimate runtime, or explicitly limits parameters (sample size, n_folds, n_estimators) to fit within the time budget – and no major training job is killed before saving results.	Two or more training scripts are interrupted with Ctrl-C or timeout before saving any useful predictions or CV scores, indicating the agent launched jobs without verifying they would complete.	N/A: All training jobs completed successfully without interruption. Insufficient: Job runtimes and completion status are not visible in the transcript.
Over-Optimization Without Significant Gains ( over_optimization ). Whether the agent avoids burning a disproportionate share of turns on hyperparameter tuning and ensemble-weight search that yields no statistically meaningful CV improvement, judged within the current attempt AND across the best-of-N attempts. Context: This is a best-of-N run (N=n_attempts); the submitted result is the MAX over attempts, so grinding for a 4th-decimal CV gain rarely changes the outcome while it burns budget that a different approach could use. ’Statistically meaningful’ means the CV gain exceeds the fold-to-fold noise: compare the improvement to cv_std in cv_scores.json (a rule of thumb: a gain smaller than ~0.25-0.5x cv_std, or $<$ ~0.001 relative when std is unavailable, is inside the noise). Use the accumulated prior-attempt transcripts (traj_a{attempt}_t{turn}.txt in /reviewer/) and cv_scores.json to judge cumulative effort across attempts, not just this one.	Time spent on hyperparameter tuning / ensemble-weight search is proportionate – the agent stops optimizing an axis once gains fall within cv_std and redirects effort to a new model family, features, or a fresh attempt; the beam’s total tuning effort is not dominated by noise-level micro-optimization.	A large share of turns (roughly a third or more of the attempt, or the bulk of multiple attempts) goes to HPO or ensemble-weight tuning whose cumulative CV gain is within the noise floor ( $<$ cv_std, e.g. AUC moving 0.8671 $\rightarrow$ 0.8672 across 30+ turns, or repeated Optuna runs each landing within std of the last) – effort that best-of-N makes almost worthless.	N/A: The agent did little or no hyperparameter/ensemble tuning, or tuning produced clearly meaningful ( $>$ cv_std) gains. Insufficient: cv_std / CV deltas or the turn-level breakdown of effort cannot be recovered from the transcript, workspace, or prior-attempt transcripts.

Appendix

Table 15: Seed rubric definitions: efficiency, robustness, and submission (part 2 of 2). Text is verbatim as given to the PRA.

	Failed attempts (%)		Mean importance
Seed rubric	Easy	Hard	Easy	Hard
Model Diversity and Ensembling	83.2	74.4	0.66	0.63
Avoiding Churn Without Pivoting	83.7	68.1	0.58	0.61
Best-CV Model Submission Selection	68.4	73.7	0.71	0.70
CV Regression Detection and Revert	67.9	69.3	0.64	0.63
Hedging Across Best-of-N With Diverse Model Classes	66.8	52.2	0.53	0.55

Appendix

Table 16: Seed rubric failures in the frozen-rubric runs. Failed attempts : share of attempts in which the rubric failed at least one review. Mean importance : the PRA’s mean importance score for the rubric over all reviews.

Seed rubric	Hard task (modality)	Flagged behavior $\rightarrow$ fix on DSPredict-Hard	DSPredict-Easy counterpart
Valid Cross-Validation Methodology	LANL Earthquake (seismic time series)	Random k-fold on a time series (“temporal leakage”) and “OOF” predictions from the final model on its own training data $\rightarrow$ time-based folds with true out-of-fold predictions.	s5e2: OOF predictions computed in-sample by the final model.
Suspicious CV Score Investigation	Statoil Iceberg (radar imagery)	CV improves 57% “with NO model change” $\rightarrow$ the agent finds and removes the leak; CV returns to a plausible 0.41.	s3e21: CV improves 63.5% “with no architectural explanation”.
Submission File Integrity Verification	TensorFlow Speech (audio)	Required submission file missing, workspace copy uses 30 wrong class labels $\rightarrow$ regenerated with all 12 required labels.	s3e19: required submission file missing.
CV Regression Detection and Revert	Data Science Bowl 2018 (microscopy segmentation)	New submission drops sample mAP from 0.17 to 0.0002 $\rightarrow$ reverts to the best U-Net checkpoint (mAP 0.31).	s3e1: a Ridge script overwrites the best XGBoost submission with one 77% worse.
Training Objective–Metric Alignment	Novozymes Enzyme Stability (protein sequences)	Competition scores Spearman correlation, agent trains and validates on RMSE $\rightarrow$ tracks cross-validated Spearman.	s3e14: competition scores MAE, models trained on default MSE.
Domain-Appropriate Preprocessing	GeoLifeCLEF 2024 (geospatial species presence)	KNN on unscaled features with a 3,432 $\times$ range in scale $\rightarrow$ adds standardization, CV improves.	s3e25: MLP and SVR trained without feature scaling.

Appendix

Table 17: Seed rubrics discovered on DSPredict-Easy catching the same process failures on non-tabular DSPredict-Hard tasks, from the frozen-rubric runs. Each DSPredict-Hard flag was resolved later in the same attempt, confirmed from the PRA’s rationale at the later review. Quoted phrases are from the PRA’s rationales.

Figure 4: Rubric store operations per task for AdaPRA and the

-

PRA ablation. (a) Operations by type, with raw counts above the bars. (b) Additions by the attempt in which they occur. Rates are over tasks with an operation log (DSPredict-Easy: 37 and 38 tasks; DSPredict-Hard: 52 and 49 tasks, for AdaPRA and

-

PRA).

Figure 5: Rubric store operations per attempt for each run, stacked by type. Panel titles give the number of tasks with an operation log. The only update occurs in the first attempt of AdaPRA on DSPredict-Easy, and no run retires a rubric.

Figure 6: Share of tasks by the number of rubrics the PRA adds to the task’s store over its five attempts, over tasks with an operation log.

Theme	AdaPRA	$-$ PRA	AdaPRA	$-$ PRA	Examples
	DSPredict-Easy		DSPredict-Hard
Validation and leakage	7	8	25	10	Calibration fit and scored on the same out-of-fold predictions (s3e26); quantum-mechanical sub-components of the target used as features (CHAMPS Scalar Coupling).
Train–test and submission consistency	3	12	16	24	Ensemble test predictions weighted differently from the scored out-of-fold blend (s5e2, $-$ PRA); submission built on the Stage 1 template (March Machine Learning Mania 2025).
Prediction sanity	11	1	12	4	Out-of-fold predictions compressed toward the target mean (s3e15); negative time-to-failure predictions (LANL Earthquake Prediction).
Features and modeling choices	8	2	8	9	No log transform of a skewed price target under RMSE (s4e9); no building-level target encoding (ASHRAE Energy Prediction).
Crashing code	2	6	2	7	predict_proba called on a ridge classifier (s3e26, $-$ PRA); training script calls an unimported helper (Humpback Whale Identification, $-$ PRA).

Appendix

Table 18: Rubrics added by the PRA, grouped by what they check, as counts per run. Themes were assigned by hand from each rubric’s name and description. Each example gives the task, and

-

PRA marks rubrics added by the ablation.

Cadence	Valid	Pct.	Median
Harness only	94.7	48.4	55.3
$c{=}5$	100.0	52.7	60.5
$c{=}10$ ( AdaPRA )	100.0	56.4	65.8
$c{=}20$	100.0	49.9	57.9

Appendix

Table 19: Single-attempt results on DSPredict-Easy as a function of the PRA cadence

c

(agent turns between PRA invocations). All rows use the same seed rubrics and PRA model.

c{=}10

is the default setting of AdaPRA .

Method	Valid	Pct.	Median
Harness only	97.4	36.9	23.7
AdaPRA	100.0	44.7	47.4

Appendix

Table 20: Single-attempt results on DSPredict-Easy with DeepSeek-V3.2 as the base agent. Rates use the fixed denominator of 38 tasks. Neither condition reaches a medal threshold.

	DSPredict-Easy			DSPredict-Hard
Method	Agent	PRA	Total	Agent	PRA	Total
Harness only	70.5	0.0	70.5	95.5	0.0	95.5
AdaPRA	78.9	142.4	221.3	123.0	125.3	248.4
$-$ adaptive rubrics	84.8	139.5	224.3	133.2	123.6	256.8
$-$ PRA	79.8	90.0	169.8	115.7	78.9	194.7

Appendix

Table 21: Mean turns per attempt (averaged within each task, then over tasks). Agent counts base-agent turns, and PRA counts supervisor turns summed over all invocations in the attempt.

Method	Attempts	Agent ($)	PRA ($)	Total ($)	$/attempt
DSPredict-Easy (38 tasks $\times$ 5 attempts)
Harness only	187	193	—	193	1.0
+ PRA + adaptive rubrics ( AdaPRA )	188	193	1,225	1,418	7.5
$-$ adaptive rubrics	190	213	1,261	1,474	7.8
$-$ PRA	190	202	1,077	1,279	6.7
DSPredict-Hard (54 tasks $\times$ 5 attempts)

Appendix

Table 22: API cost per run. Agent : Qwen3-Coder-Next served through Amazon Bedrock. PRA : Claude Sonnet 4.6. The PRA accounts for 72–86% of the cost of supervised runs. Scoring by reading alone (

-

PRA) lowers the cost per attempt relative to AdaPRA by 11% on DSPredict-Easy and 22% on DSPredict-Hard.

Figure 7: Share of scores in which each rubric passed or failed in an earlier version of the system, over scores with importance

\geq 0.6

(Appendix R ).

Data Analysis Agents

Data analysis agents use language models and tools to explore datasets, run analyses, and produce answers or reports grounded in evidence.

See all 106 papers

Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis
Apr 27, 2026Zhisong Qiu, Shuofei Qiao, Kewei Xu +4Process Reward Models RL for LLM Agents
Zhejiang University Ant Group Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph Hangzhou, China · Zhejiang University Hangzhou, China · Ant Group Hangzhou, China +2
DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?
Aug 11, 2026Mizanur Rahman, Mohammed Saidul Islam, Ridwan Mahbub +3Computer-Use Agent Benchmarks AI Agent Evaluation
York University · Nanyang Technological University · Salesforce AI Research
AdaRubric: Task-Adaptive Rubrics for Reliable LLM Agent Evaluation and Reward Learning
Mar 22, 2026Liang DingLLM Agent Evaluation Rubric-Based Evaluation
The University of Sydney

Apr 27, 2026J/K move · Enter open · S save

Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis

Zhisong Qiu, Shuofei Qiao, Kewei Xu, Yuqi Zhu+3

Zhejiang University Ant Group Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph Hangzhou, China · Zhejiang University Hangzhou, China · Ant Group Hangzhou, China+2

Process Reward Models RL for LLM Agents Data Analysis Agents

Open paper AskPDF arXiv

Process Reward Models (PRMs) have achieved remarkable success in augmenting the reasoning capabilities of Large Language Models (LLMs) within static domains such as mathematics. However, their potential in dynamic data analysis tasks remains underexplored. In this work, we first present a empirical study revealing that general-domain PRMs struggle to supervise data analysis agents. Specifically, they fail to detect silent errors, logical flaws that yield incorrect results without triggering interpreter exceptions, and erroneously penalize exploratory actions, mistaking necessary trial-and-error exploration for grounding failures. To bridge this gap, we introduce DataPRM, a novel environment-aware generative process reward model that (1) can serve as an active verifier, autonomously interacting with the environment to probe intermediate execution states and uncover silent errors, and (2) employs a reflection-aware ternary reward strategy that distinguishes between correctable grounding errors and irrecoverable mistakes. We design a scalable pipeline to construct over 8K high-quality training instances for DataPRM via diversity-driven trajectory generation and knowledge-augmented step-level annotation. Experimental results demonstrate that DataPRM improves downstream policy LLMs by 7.21% on ScienceAgentBench and 11.28% on DABStep using Best-of-N inference. Notably, with only 4B parameters, DataPRM outperforms strong baselines, and exhibits robust generalizability across diverse Test-Time Scaling strategies. Furthermore, integrating DataPRM into Reinforcement Learning yields substantial gains over outcome-reward baselines, achieving 78.73% on DABench and 64.84% on TableBench, validating the effectiveness of process reward supervision. Code is available at https://github.com/zjunlp/DataMind.

Abstract

Data Analysis Agents

Explore similar work

Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis

DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?

AdaRubric: Task-Adaptive Rubrics for Reliable LLM Agent Evaluation and Reward Learning