Benchmarking Large Language Models

Follow this topic Ask about this topic

Recent momentum

-33%

14 papers in the last 28 days · 0.2% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

7 new papers

A weekly snapshot of new work published in Benchmarking Large Language Models.

Period ending 2026-09-14

4 new papers

A weekly snapshot of new work published in Benchmarking Large Language Models.

Period ending 2026-09-07

3 new papers

A weekly snapshot of new work published in Benchmarking Large Language Models.

154 papers

Latest in Benchmarking Large Language Models

Browse all topics →

MA-ProofBench: A Two-Tiered Evaluation of LLMs for Theorem Proving in Mathematical Analysis
Jun 11, 2026Lushi Pu, Weiming Zhang, Xinheng Xie +6Theorem Mathematics
SupraBench: A Benchmark for Supramolecular Chemistry
Jun 11, 2026Tianyi Ma, Yijun Ma, Zehong Wang +6Benchmarking Large Language Models
VHDLSuite: Unified Pipeline for LLM VHDL Generation with Data Synthesis and Evaluation
Jun 11, 2026Yijun Shen, Minghao Shao, Yichen Zhao +4Vhdl Hardware Description Languages
MÖVE: A Holistic LLM Benchmark for the German Public Sector
Jun 11, 2026Camilla Dalerci, Thilo Michael, Robin Schaefer +1Benchmarking Large Language Models German
Structured Testbench Generation for LLM-Driven HDL Design and Verification-Oriented Data Curation
Jun 11, 2026En-Ming Huang, Yu-Hung Kao, Ren-Hao Deng +10Hardware Description Languages Flexible Register Transfer Level Design Automation
Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation
Jun 10, 2026Selen Erkan, Bastian Boll, Kristian Kersting +2Prompt Tuning Benchmarking Large Language Models
Flaws in the LLM Automation Narrative
Jun 9, 2026George Perrett, Javae Elliott, Jennifer Hill +1Benchmarking Large Language Models Narratives
TheoremBench: Evaluating LLMs on Theorem Proving in Formal Mathematics
Jun 8, 2026QuocViet Pham, Elvir Karimov, Andrey Galichin +1Theorem Reasoning Benchmark
RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models
Jun 8, 2026Jing Wang, Shang Liu, Wenji Fang +3Register Transfer Level Benchmarking Large Language Models
SurgiQ: A Large-Scale Multi-Domain Benchmark for Evaluating Surgical Understanding in Large Language Models
Jun 6, 2026Ayah Al-Naji, Edoardo Fazzari, Saif Alkindi +3Gradient Surgery Benchmarking Large Language Models
Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese
Jun 5, 2026Xing Yue, Yongliang Shen, Weiming LuPhonological Understanding Phonetic Representations
PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models
Jun 2, 2026Zetian Ouyang, Linlin Wang, Gerard de Melo +1Mathematical Reasoning Benchmarking Large Language Models
GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory
Jun 2, 2026Noujoud Nader, Ibrahem Aljabea, Patrick Diehl +1Reasoning Graphs Benchmarking Large Language Models
CodegenBench: Can LLMs Write Efficient Code Across Architectures?
Jun 1, 2026Jie Li, Wenzhao Wu, Junqi Hu +5High-Performance Computing Code Generation
PlanarBench: Evaluating LLM Spatial Reasoning via Planar Graph Drawing
Jun 1, 2026Oleksandr Nikitin, Anna KravchenkoSpatial Reasoning Large Language Model Planning
Consistency evaluation of benchmarks used for causal discovery
Jun 1, 2026Yuzhe Zhang, Chihui Chen, Lina Yao +1Causal Foundation Models Causal Graph
Consistent and Distinctive: LLM Benchmark Efficiency via Maximum Independent Set Prompt Selection on Similarity Graphs
May 31, 2026Denica Kjorvezir, Marko Djukanović, Ana Gjorgjevikj +2Benchmarking Large Language Models Semantic Similarity Metrics
Benchmarking Local LLMs for Natural-Language-to-SQL Querying in Biopharmaceutical Manufacturing: An Empirical Benchmark on Consumer-Grade Hardware
May 31, 2026Sagar Bhetwal, Rajan Bastakoti, Nirajan Acharya +2Text-To-Sql Benchmarking Large Language Models
ProjectionBench: Evaluating Scientific Hypothesis Generation in LLMs Under Progressive Information Disclosure
May 28, 2026A. J. Lew, Y. Cao, M. J. BuehlerScientific Discovery Artificial Intelligence Scientists
CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models
May 28, 2026Vedant PadwalBenchmarking Large Language Models Code Generation
SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers
May 27, 2026Kaihua Qin, Dawn Song, Arthur GervaisSmart Contracts Disassembly
PetroBench: A Benchmark for Large Language Models in Petroleum Engineering
May 27, 2026Xiang Wang, Tingting Zhang, Sen Wang +4Benchmarking Large Language Models
AssertLLM2: A Comprehensive LLM Benchmark for Assertion Generation from Design Specifications
May 26, 2026Yuchao Wu, Wenji Fang, Jing Wang +3Systemverilog Assertions Test Generation
DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking
May 25, 2026Matt L. Wiemann, Lindsay M. Smith, Peter Melchior +4Scientific Discovery Physics-Aware Models
Automated Benchmark Auditing for AI Agents and Large Language Models
May 25, 2026Junlin Wang, Federico Bianchi, Shang Zhu +4Artificial Intelligence Benchmarks Benchmarking Large Language Models
Efficient Benchmarking Is Just Feature Selection and Multiple Regression
May 25, 2026Sam Bowyer, Acyr Locatelli, Kris CaoBenchmarking Large Language Models Relevance
FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization
May 24, 2026Minwei Kong, Chonghe Jiang, Ao Qu +24Llm-Based Optimization Modeling Benchmarking Large Language Models
GlobalDentBench: A Multinational Benchmark for Evaluating LLM Clinical Reasoning in Dentistry with Expert Calibration
May 23, 2026Junjie Zhao, Jingyi Liang, Zhenyang Cai +22Clinical Reasoning Benchmarking Large Language Models
Healthcare LLM Benchmarks Are Only as Good as Their Explicit Assumptions
May 21, 2026Naveen Raman, Santiago Cortes-Gomez, Mateo Dulce Rubio +2Healthcare Benchmarking Large Language Models
VeriScale: Adversarial Test-Suite Scaling for Verifiable Code Generation
May 21, 2026Yifan Bai, Xiaoyang Liu, Zihao Mou +7Benchmarking Large Language Models Code Generation
SMDD-Bench: Can LLMs Solve Real-World Small Molecule Drug Design Tasks?
May 20, 2026Kevin Han, Renfei Zhang, Kathy Wei +3Drug Design Drug Discovery
LLM Benchmark Datasets Should Be Contamination-Resistant
May 19, 2026Ali Al-Lawati, Jason Lucas, Dongwon Lee +1Contamination-Aware Benchmarking Framework Benchmarking Large Language Models
Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
May 19, 2026Husnain Amjad, Raja Khurram Shahzad, Aamir Shahzad +1Mathematical Reasoning Large Language Model Reasoning
The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility
May 19, 2026David Pape, Jonathan Evertz, Lea SchönherrBenchmarking Large Language Models LLM Inference Efficiency
The Evaluation Game: Beyond Static LLM Benchmarking
May 19, 2026Paul Wang, Jade Garcia-Bourrée, Anne-Marie Kermarrec +1Adversarial Evaluation Benchmarking Large Language Models
SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science
May 18, 2026Nithin Somasekharan, Youssef Hassan, Shiyao Lin +5Scientific Discovery Benchmarking Large Language Models
How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and Benchmarking
May 18, 2026Rafid Ahmed, Intesar Tahmid, Mir Sazzat Hossain +3Medical Visual Question Answering Bangla
BacktestBench: Benchmarking Large Language Models for Automated Quantitative Strategy Backtesting
May 18, 2026Zhensheng Wang, Wenmian Yang, Qingtai Wu +3Benchmarking Large Language Models Large-Scale Benchmark
A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
May 17, 2026Qingchuan Ma, Yuexiao Ma, Yongkang Xie +3Reasoning Benchmark Benchmarking Large Language Models
BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks
May 16, 2026Ruth Wan Theng Chew, Zhiliang Chen, Apivich Hemachandra +1Benchmarking Large Language Models Bayesian Optimization
MHGraphBench: Knowledge Graph-Grounded Benchmarking of Mental Health Knowledge in Large Language Models
May 15, 2026Weixin Liu, Congning Ni, Shelagh A. Mulvaney +4Benchmarking Large Language Models Mental Health
RTL-BenchMT: Dynamic Maintenance of RTL Generation Benchmark Through Agent-Assisted Analysis and Revision
May 15, 2026Jing Wang, Shang Liu, Hangan Zhou +1Flexible Register Transfer Level Design Automation Register Transfer Level
Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench
May 13, 2026Qingyun Zou, Feng Yu, Hongshi Tan +2Electronic Design Automation Benchmarking Large Language Models
Discovery of Hidden Miscalibration Regimes
May 13, 2026Katarzyna Kobalczyk, Mihaela van der SchaarMulticalibration and Calibration Confidence Calibration
PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization
May 13, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +5Benchmarking Large Language Models Code Optimization
Forge: Quality-Aware Reinforcement Learning for NP-Hard Optimization in LLMs
May 9, 2026Xiaozhe Li, Xinyu Fang, Shengyuan Ding +5Reasoning Benchmark Reinforcement Learning With Verifiable Reward
Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs
May 8, 2026Andrea Sassella, Andrea Chizzola, Tommaso Bianchi +2Benchmarking Large Language Models Italian Case Studies
LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systems
May 8, 2026Siyu Wu, Yulong Ye, Zezhen Xiang +3Benchmarking Large Language Models Hyperparameter Optimization
NSMQ Riddles: A Benchmark of Scientific and Mathematical Riddles for Quizzing Large Language Models
May 8, 2026George Boateng, Naafi Ibrahim, Samuel John +8Benchmarking Large Language Models Reasoning Benchmark
Revealing Modular Gradient Noise Imbalance in LLMs: Calibrating Adam via Signal-to-Noise Ratio
May 7, 2026Ziqing Wen, Zhouyang Liu, Jiahuan Wang +4Adam Benchmarking Large Language Models
KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels
May 6, 2026Han Wang, Jintao Zhang, Kai Jiang +3Graphics Processing Unit Kernels Benchmarking Large Language Models
BenCSSmark: Making the Social Sciences Count in LLM Research
May 6, 2026Arnault Chatelain, Étienne Ollion, Qianwen Guan +7Benchmarking Large Language Models Computational Social Science
Benchmarking LLMs on the Massive Sound Embedding Benchmark (MSEB)
May 6, 2026Cyril Allauzen, Tom Bagby, Georg Heigold +2Large Audio Language Models Audio Latent
Probing Structural Mathematical Reasoning in Language Models with Algebraic Trapdoors
May 5, 2026Igor RivinMathematical Reasoning Computable
MolViBench: Evaluating LLMs on Molecular Vibe Coding
May 4, 2026Jiatong Li, Yuxuan Ren, Weida Wang +4Vast Chemical Space Code Generation
Submodular Benchmark Selection
May 4, 2026Alexander SmolaSubmodular Maximization Benchmarking Large Language Models
What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models
May 3, 2026Ranit Karmakar, Jayita ChatterjeeLarge Language Model Reliability Benchmarking Large Language Models
Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking
May 2, 2026Deep Pankajbhai MehtaBenchmarking Large Language Models
Evaluating LLMs on Large-Scale Graph Property Estimation via Random Walks
May 2, 2026Sunil Kumar Maurya, Xin LiuBenchmarking Large Language Models Large Language Model Evaluation
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks
May 2, 2026Benjamin Warner, Ratna Sagari Grandhi, Max Kieffer +32Benchmarking Large Language Models Clinical Reasoning

Jun 11, 2026 · cs.AIJ/K move · Enter open · S save

MA-ProofBench: A Two-Tiered Evaluation of LLMs for Theorem Proving in Mathematical Analysis

Lushi Pu, Weiming Zhang, Xinheng Xie, Zixuan Fu+5

ModelBest Inc. · Tsinghua University

Large Language Models (LLMs) have made notable progress in automated theorem proving, yet existing formal benchmarks remain limited in both mathematical coverage and difficulty. Most are concentrated in areas that are easier to formalize, such as algebra and elementary number theory, and provide limited coverage of subfields that require deeper reasoning, including mathematical analysis. To address this gap, we introduce MA-ProofBench, to the best of our knowledge, the first formal theorem-proving benchmark dedicated to Mathematical Analysis. The benchmark contains 200 formalized theorems covering 6 core topics and 27 subcategories, including measure and integration theory, complex analysis, and functional analysis. The problems are divided into two difficulty levels, an undergraduate level (Level I, 100 problems) and a Ph.D. qualifying level (Level II, 100 problems), to evaluate how well LLMs perform formal reasoning at different mathematical depths. Each problem is constructed through a human-led, LLM-assisted formalization pipeline followed by independent expert review, ensuring that the formal statements remain faithful to the original mathematics. We evaluate a range of recent general-purpose reasoning models and formal theorem provers on MA-ProofBench. However, most models perform poorly: even the best-performing model, GPT-5.5, achieves only 16% Pass@8 on Level I and 5% on Level II, while most models stay close to 0% on Level II. Further analysis identifies Mathlib hallucinations and incomplete proofs as the two dominant failure modes, while an evaluation on the natural-language version of the benchmark exposes a clear gap between informal and formal reasoning. MA-ProofBench is intended to serve as a reliable reference for tracking progress in formal mathematical reasoning in advanced domains.

Theorem Mathematics Reasoning Benchmark Benchmarking Large Language Models

Open paper Ask about it PDF

Recent momentum

Neighbouring directions

Recent digests

7 new papers

4 new papers

3 new papers

Latest in Benchmarking Large Language Models