Language model agents are increasingly used to automate scientific research, yet evaluating their scientific contributions remains a challenge. A key mechanism to obtain such insights is through ablation experiments. To this end, we introduce AblationBench, a benchmark suite for evaluating agents on ablation planning tasks in empirical AI research. It includes two tasks: AuthorAblation, which helps authors propose ablation experiments based on a method section and contains 83 instances, and ReviewerAblation, which helps reviewers find missing ablations in a full paper and contains 350 instances. For both tasks, we develop LM-based judges that serve as an automatic evaluation framework. Our experiments with frontier LMs show that these tasks remain challenging, with the best-performing LM system identifying only 45% of the original ablations on average, below human-level performance. We observe an inverse performance trend between the author and reviewer tasks, which we attribute to differences in model grounding. Lastly, we analyze the limitations of current LMs on these tasks, and find that chain-of-thought prompting outperforms an agent-based approach. Our data is available on https://huggingface.co/collections/ai-coscientist/ablationbench, and our code is available on https://github.com/ai-scientist-bench/ablation-bench .
Figures & tables
Figure 1: Overview of AblationBench. ( Top ) AuthorAblation is a collection of conference papers where the task of the planner is to generate ablation plan based on the paper’s method section. The judge evaluates the generated plan using manually extracted gold ablations from the paper. ( Bottom ) ReviewerAblation is a collection of ICLR submissions where the task of the planner is to generate missing ablation based on the full submission. The judge evaluates the generated ablations using the official reviews of the submission.
AblationBench
AbGen Zhao et al. (2025)
Task
Author and reviewer
Author only
Input
Raw paper text
Manually processed research context from a single paper
Output
Full ablation plan
Specified component ablation
Evaluation
LM-based judges
Human
Criteria
Compared to ground truth
Subjective ablation quality
Domain
Empirical AI (ML, CV, NLP)
NLP
Table 1: AblationBench and AbGen comparison.
Precision
Recall
F1
κ
LMJudge
GPT-5.4
0.77
0.81
0.76
0.56
o3-mini
0.76
0.86
0.78
0.58
GPT-OSS-120B
0.70
0.75
0.63
0.45
MiniMax-M2.5
0.74
0.80
0.73
0.51
Qwen3.5-122B-A10B
0.79
0.81
0.77
0.57
Table 2: Evaluating judges: Macro-average results of the two judge scaffoldings for the JudgeEval. The majority vote is based on the three best models (in terms of κ ) used in each scaffolding.
Precision@ k
Recall@ k
F1@ k
LM-Planner
GPT-5.4
0.33
0.45
0.35
o3-mini
0.26
0.33
0.27
Gemini 2.5 Flash
0.29
0.35
0.29
Llama 3.1 405B Instruct
0.24
0.33
0.26
GPT-OSS-120B
0.28
0.40
0.31
Table 3: Evaluating planners: Macro-average results of the two planner scaffoldings for the AblationBench.
Figure 2: Recall@5 performance for AuthorAblation of each planner approach across different LMs, separated by ablation type.
Figure 3: Qualitative analysis highlighting differences in the reasoning and grounding of Llama 3.1 405B and Qwen 3.5 122B on a ReviewerAblation instance.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: An example of the matching criteria in AuthorAblation instance, showing alignment between GT (gold, top) and generated (blue, bottom) ablations. Matching is done semantically, and is represented as a bipartite graph: nodes correspond to GT and generated ablations and edges indicate valid matches. Evaluation metrics are computed with respect to the GT ablations; in this example, both precision and recall equal 0.75.
Mean
Median
(a) # ablations in GT
AuthorAblation
3.7
3
ReviewerAblation
1.8
1
(b) # T e X files in truncated paper
4.2
2
(c) # figures
AuthorAblation
6.8
4
Appendix
Table 4: Mean and median of attributes for a task instance in AblationBench test set. Row (b) is only relevant for AuthorAblation.
Figure 7: Conference distribution of AuthorAblation test set.
Figure 8: Topic distribution of ReviewerAblation full test set, extracted from OpenReview.
Precision
Recall
F1
κ
LMJudge
w/ GPT-5.4
0.83
0.89
0.83
0.55
w/ o3-mini
0.81
0.89
0.81
0.57
w/ GPT-OSS-120B
0.81
0.88
0.79
0.43
w/ MiniMax-M2.5
0.79
0.86
0.76
0.47
w/ Qwen3.5-122B-A10B
0.85
0.84
0.80
0.51
Appendix
Table 5: Evaluating AuthorAblation Judge: Macro-average results of the two judge scaffoldings measured on AuthorEval.
Precision
Recall
F1
κ
LMJudge
w/ GPT-5.4
0.70
0.72
0.69
0.56
w/ o3-mini
0.71
0.82
0.74
0.59
w/ GPT-OSS-120B
0.58
0.62
0.47
0.47
w/ MiniMax-M2.5
0.69
0.74
0.69
0.54
w/ Qwen3.5-122B-A10B
0.72
0.77
0.73
0.63
Appendix
Table 6: Evaluating ReviewerAblation Judge: Macro-average results of the two judge scaffoldings measured on ReviewerEval.
Precision@5 ↑
Recall@5 ↑
F1@5 ↑
nDCG@5 ↑
LM-Planner
w/ GPT-5.4
0.35
0.52
0.39
0.53
w/ o3-mini
0.31
0.42
0.33
0.45
w/ Gemini 2.5 Flash
0.32
0.39
0.32
0.41
w/ Llama 3.1 405B Instruct
0.23
0.33
0.25
0.34
w/ GPT-OSS-120B
0.33
0.49
0.37
0.51
Appendix
Table 7: Evaluating AuthorAblation Planners: Macro-average results of the two planner scaffoldings.
Precision@2 ↑
Recall@2 ↑
F1@2 ↑
LM-Planner
w/ GPT-5.4
0.31
0.37
0.31
w/ o3-mini
0.20
0.24
0.20
w/ Gemini 2.5 Flash
0.25
0.30
0.26
w/ Llama 3.1 405B Instruct
0.25
0.32
0.27
w/ GPT-OSS-120B
0.23
0.31
0.25
Appendix
Table 8: Evaluating ReviewerAblation Planners: Macro-average results of the two planner scaffoldings.
Error Type
Description
o3-mini (%)
GPT OSS 120B (%)
False Negative
The judge marked it as "not found" even though the ablation appears in the prediction.
51
17
Replacement Mismatch
The correct component is identified, but the suggested replacements are not aligned with the GT.
9
20
Surface-Level Match
The match is based on similar wording, but the actual meaning does not align.
7
30
Partial Match
The predicted ablation is either broader or narrower than the GT.
16
26
Action Type Mismatch
The action (e.g., remove vs. modify) in the predicted ablation differs from the GT.
12
5
Other
Miscellaneous errors that do not fall into the above categories (e.g., formatting errors).
5
2
Appendix
Table 9: Analysis of prediction errors for AuthorAblation judge, using LMJudge powered by o3-mini and GPT OSS, measured on AuthorEval. Percentage indicates the percentage of all errors.
Error Type
Description
o3-mini (%)
GPT OSS 120B (%)
Misinterpreted due to review breadth
The review includes an ablation suggestion, but the model draws an incorrect conclusion due to vague or broad review suggestions.
25
38
Ablation not identified
The model fails to identify the ablation suggestion in the review (although it appears).
34
37
Misinterpreted due to contextual understanding
The model identifies the ablation but misinterprets it in the context of the paper.
17
5
Incorrectly flags a non-ablation experiment
The model mistakenly classifies an unrelated experiment suggestion as a suggested ablation.
19
18
Other
Miscellaneous errors that do not fall into the above categories (e.g., formatting errors).
5
2
Appendix
Table 10: Analysis of prediction errors for ReviewerAblation judge, using LMJudge powered by o3-mini and GPT OSS, measured on ReviewerEval. Percentage indicates the percentage of all errors.
Figure 9: Evaluation metrics for AuthorAblation using the LM-Planner with GPT-5.4, across varying values of k , number of generated ablations. Standard error presented across papers.
Figure 10: Metrics@5 per conference type measured on AuthorAblation using the LM-Planner with GPT-5.4. We categorize the conferences in the following manner: (1) General ML: ICML, NeurIPS, ICLR, AAAI, IJCAI, KDD; (2) CV: CVPR, ECCV, ICPR, IEEE, ICCV; and (3) NLP: ACL, EMNLP, NAACL.