Benchmark Reproducibility

Latest papers

21 papers5 papers in the last four weeks, up 25% on the four weeks before. 0.0% of all new papers.

Re-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Risk
Oct 10, 2026Tianyu Ding, Yaocheng Tong, Muzhi Yang +3Benchmark Reproducibility Automated Agent Evaluation
Independent Researchers
Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing
Oct 9, 2026Oktie Hassanzadeh, Dharmashankar SubramanianBenchmark Reproducibility LLM Evaluation
IBM Yorktown Heights, NY, USA
Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing
Oct 7, 2026Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek +2Benchmark Reproducibility LLM Inference
Lucerne University of Applied Sciences and Arts, Switzerland · Microsoft, Switzerland · Uthereal AG, Switzerland +1
Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation
Sep 29, 2026Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager +1LLM Evaluation Language Model Generation Evaluation
Johannes Gutenberg University Mainz, Germany · Universidad Iberoamericana, Mexico · University of Colorado Boulder, USA
AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines
Sep 16, 2026Li ChenGPU Kernel Optimization Benchmark Design
Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints
Sep 3, 2026Haoyaun Zhu, Jie ZhangLLM Evaluation LLM-as-a-Judge
School of Electronic and Electrical Engineering, University of Sheffield, Sheffield S10 2TN, UK · R&D Department, Ranplan Wireless Network Design Ltd., Cambridge CB23 3UY, UK · R&D Department, Cambridge AI+ Ltd., Cambridge CB23 3UY, UK
SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction
Aug 25, 2026Xue Hu, Zewei Pan, Zeli Su +2ML Reproducibility Scientific Code Generation
Beihang University · Shanghai Jiao Tong University · Minzu University of China +3
Noise Floor Audit for Agent Benchmarks
Aug 23, 2026Yihang Chen, Pin Qian, Su Wang +4Benchmark Reproducibility LLM Agent Reliability
Reproducible Multimodal Affordance Prediction
Aug 18, 2026Tommaso Apicella, Alessio Xompero, Andrea CavallaroBenchmark Reproducibility Multimodal Model Evaluation
Istituto Italiano di Tecnologia, Italy · Independent Researcher · EPFL, Switzerland
Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact
Jul 28, 2026Mateusz KozłowskiVLM Evaluation Radiology VLMs
Independent Researcher, Kraków, Poland
Metrics or Mirage? An Audit of Evaluation Inconsistencies in Colonoscopy Polyp Segmentation Benchmarks
Jul 9, 2026Aisha Urooj, Zain Ul Abdien, Neelu MadanPolyp Segmentation Benchmark Reproducibility
Lunit, South Korea · RPTU Kaiserslautern, Germany · Aalborg University & Pioneer Center, Denmark
Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations
Jun 24, 2026Hiroki TambaLLM Evaluation LLM-as-a-Judge
Tamba Research Academy
ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
Jun 16, 2026Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5LLM Agent Evaluation AI Agent Benchmarks
School of Computer Science, Carnegie Mellon University · Datadog
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
May 27, 2026Pengyu Zhu, Lijun Li, Yaxing Lyu +8LLM Agent Evaluation AI Agent Benchmarks
1Beijing University of Posts and Telecommunications · 2Shanghai Artificial Intelligence Laboratory · 4North China Electric Power University +1
SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks
May 25, 2026Yanhang Li, Zhichao Fan, Zexin ZhuangPairwise Comparison Pairwise Preference Evaluation
Northeastern University, Boston, MA, USA · University of Illinois Urbana-Champaign, Urbana, IL, USA · Southern Methodist University, Dallas, TX, USA
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
May 20, 2026Mahdi Naser Moghadasi, Faezeh GhaderiBenchmark Auditing LLM Auditing
BrightMind AI, Texas Tech University · Research Division, BrightMind AI, Seattle, WA · Texas Tech University, Lubbock, TX +2
The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility
May 19, 2026David Pape, Jonathan Evertz, Lea SchönherrLLM Evaluation LLM Inference
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review
May 4, 2026Kevin Riehl, Andres L. Marin, Nikofors Zacharof +6AI-Assisted Scientific Research Scientific Reproducibility
European Commission, Joint Research Centre · Ideas Forward
Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks
Mar 3, 2026Junjie Chu, Xinyue Shen, Ye Leng +3Benchmark Reproducibility LLM Safety Benchmarks
CISPA Helmholtz Center for Information Security · University of Waterloo · Flexera
Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
Jan 18, 2025Jialun Cao, Yuk-Kit Chan, Zixuan Ling +12Benchmark Construction Benchmark Design
The Hong Kong University of Science and Technology · Guangzhou HKUST Fok Ying Tung Research Institute · The Chinese University of Hong Kong +6
CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
Sep 17, 2024Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir +2AI Agent Evaluation AI Agent Benchmarks
Princeton University

Oct 10, 2026J/K move · Enter open · S save

Re-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Risk

Tianyu Ding, Yaocheng Tong, Muzhi Yang, Fei Jiang+2

Independent Researchers

Benchmark Reproducibility Automated Agent Evaluation Code Generation Evaluation AI Agent Benchmarks AI Agent Auditing Scientific Reproducibility

Open paper AskPDF arXiv

Executable tests grade large language model (LLM) coding-agent patches for leaderboards, rewards and self-improvement. Two preregistered re-execution studies returned negative results. SWE-bench Pro v1's offline option rejected none of 59 accepted Go patches whose grading configuration changed. Their exploratory, unweighted upper confidence limit is 4.95% under candidate independence, leaving a 3% exposed-patch risk unresolved; the preregistered estimator uses inverse sampling-probability weights. An extension was inconclusive. In a selected stratum, a reconstructed module cache built from the reference solution's dependencies rejected accepted alternatives from four models that passed with their own dependencies. Re-executing 1,929 selected agent-version/task evaluations from the Darwin Gödel Machine's archive preserved best-agent selection and system-variant ranking, with search history and scores of evaluations not rerun fixed. Exploratory host, image and adapted-grader comparisons also preserved them. Seven of 23 disagreements with archived verdicts did not recur on a second host; rebuilt images changed 16 three-run majority verdicts (0.83%) under four-CPU execution. We provide an evidence-to-claim summary, lessons for grader audits and an anonymous replication package. These results concern accepted patches and fixed-history decisions; general policy safety and adaptive-learning robustness require broader evaluations.