Benchmark Auditing

Follow this topic Ask about this topic

Benchmark auditing studies whether test items, labels, scoring rules, and evaluation setups support the claims made from model scores. Auditors use interventions and matched controls to detect shortcuts, contamination, and information leakage; they also check labels with human review or item-response models and test software verifiers through replay and mutation. Automated LLM and coding-agent tools can help inspect large benchmarks, but their findings still need validation against independent evidence. A central challenge is that audits themselves can be underpowered or incomplete, while scores and rankings may change with prompts, splits, references, and runtime conditions.

Latest papers

97 papers18 papers in the last four weeks, down 10% on the four weeks before. 0.1% of all new papers.

FinRule-Bench: A Benchmark for Joint Reasoning over Financial Tables and Principles
Mar 11, 2026Arun Vignesh Malarkkan, Manan Roy Choudhury, Guangwei Zhang +4Financial Reasoning in Language Models Benchmark Auditing
Arizona State University Tempe, Arizona, USA · Pine AI Singapore, Singapore · The College of William & Mary Williamsburg, Virginia, USA +1
DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality
Mar 6, 2026Yukun Huang, Leonardo F. R. Ribeiro, Momchil Hardalov +3Automated Fact-Checking AI Agent Benchmarks
Duke University · Amazon AGI · Boston University
Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks
Mar 3, 2026Junjie Chu, Xinyue Shen, Ye Leng +3Benchmark Reproducibility LLM Safety Benchmarks
CISPA Helmholtz Center for Information Security · University of Waterloo · Flexera
AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors
Feb 26, 2026Abhay Sheshadri, Aidan Ewart, Elias Kempf +8Benchmark Auditing AI Agent Evaluation
Anthropic Fellows Program · Anthropic
Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts
Nov 6, 2025Ellis Brown, Jihan Yang, Shusheng Yang +2Benchmark Auditing VLM Evaluation
New York University
WirelessMathBench-XL: An Auditable Benchmark for Wireless Mathematical Reasoning
Sep 27, 2025Xin Li, Mengbing Liu, Yiyang Zhu +4Mathematical Reasoning Benchmarks Benchmark Auditing
Nanyang Technological University
Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks
Apr 10, 2025Pavel Chizhov, Anton Changalidis, Vishnu Prasad Vijaya Kumar +4Commonsense Reasoning Benchmark Auditing
1CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt · 2PleIAs, Paris, France

Mar 11, 2026J/K move · Enter open · S save

FinRule-Bench: A Benchmark for Joint Reasoning over Financial Tables and Principles

Arun Vignesh Malarkkan, Manan Roy Choudhury, Guangwei Zhang, Vivek Gupta+3

Arizona State University Tempe, Arizona, USA · Pine AI Singapore, Singapore · The College of William & Mary Williamsburg, Virginia, USA+1

Financial Reasoning in Language Models Benchmark Auditing

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly applied to financial analysis, yet their ability to audit structured financial statements under explicit accounting principles remains poorly explored. Existing benchmarks primarily evaluate question answering, numerical reasoning, or anomaly detection on synthetically corrupted data, making it unclear whether models can reliably verify or localize rule compliance on correct financial statements. We introduce FinRule-Bench, a benchmark for evaluating diagnostic completeness in rule-based financial reasoning over real-world financial tables. FinRule-Bench pairs ground-truth financial statements with explicit, human-curated accounting principles and spans four canonical statement types: Balance Sheets, Cash Flow Statements, Income Statements, and Statements of Equity. The benchmark defines three auditing tasks that require progressively stronger reasoning capabilities: (i) rule verification, which tests compliance with a single principle; (ii) rule identification, which requires selecting the violated principle from a provided rule set; and (iii) joint rule diagnosis, which requires detecting and localizing multiple simultaneous violations at the record level. We evaluate LLMs under zero-shot and few-shot prompting, and introduce a causal-counterfactual reasoning protocol that enforces consistency between decisions, explanations, and counterfactual judgments. Across tasks and statement types, we find that while models perform well on isolated rule verification, performance degrades sharply for rule discrimination and multi-violation diagnosis. FinRule-Bench provides a principled and reproducible testbed for studying rule-governed reasoning, diagnostic coverage, and failure modes of LLMs in high-stakes financial analysis.