Benchmarking Large Language Models

Ask about this topic

Recent momentum

-33%

14 papers in the last 28 days · 0.2% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Also in LLM-as-a-Judge

Neighbouring directions

Large Language Model EvaluationLlm-As-A-JudgeLarge Language Model JudgesJudges

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

7 new papers

A weekly snapshot of new work published in Benchmarking Large Language Models.

Period ending 2026-09-14

4 new papers

A weekly snapshot of new work published in Benchmarking Large Language Models.

Period ending 2026-09-07

3 new papers

A weekly snapshot of new work published in Benchmarking Large Language Models.

152 papers

Latest in Benchmarking Large Language Models

Open your feed →
CardsList

  1. PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces

    Sep 17, 2026Pyrros Koussios, Benjamin Jäger, John Hua Yao +3Reasoning BenchmarkLarge Language Model Reasoning

  2. Reproducibility is not construct validity: LLM measurement of institutionally situated communication

    Sep 17, 2026Veronika Batzdorfer, Carlo Romano Marcello Alessandro SantagiustinaLarge Language Model ReliabilityInter-Rater Variability

  3. Beyond Outcomes: Dual-View Relational Learning for Efficient Agent Benchmarking

    Sep 16, 2026Xinshuai Guo, Junjie Wu, Dolly Deng +4Agentic BenchmarksBenchmarking Large Language Models

  4. AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines

    Sep 16, 2026Li ChenBenchmarking Large Language ModelsProgrambench

  5. K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

    Sep 14, 2026Laura M. Vowels, Matthew J. Vowels, Shivali Sharma +9Large Language Model SafetyBenchmarking Large Language Models

  6. How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks

    Sep 14, 2026Ali Ansari, Haoran Sun, Andy Zeyi Liu +48Benchmarking Large Language ModelsFrontier Models

  7. Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models

    Sep 14, 2026Zhongzhan Huang, Junxin Li, Guoming Ling +3Benchmarking Large Language ModelsLarge Language Model Evaluation

  8. Φ-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?

    Sep 9, 2026Leilei Ding, Shumin Wang, Yuting Huang +10Benchmarking Large Language ModelsInfrastructure

  9. To What Extent Do Large Language Models Understand Bangla Idioms?

    Sep 3, 2026Mousumi Akter, Md. Faiyaz Abdullah Sayeedi, Nurul Labib Sayeedi +1Multilingual IdiomsMultilingual Large Language Models

  10. Benchmarking Language Models for Statistical Problem Formulation

    Sep 2, 2026Chen Wang, Junzhe Zhao, Xin Cong +2Benchmarking Large Language ModelsStatistical Inference

  11. Can LLMs Discover Scientific Laws in Real and Parallel Worlds?

    Sep 1, 2026Yiming Huang, Ziche Liu, Zhuohang Wu +11Scientific DiscoveryBenchmarking Large Language Models

  12. VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean Preferences

    Sep 1, 2026Yiwen Jiang, Yang Deng, Stephanie Fong +9PersonalizationBenchmarking Large Language Models

  13. Can Large Language Models Forecast What Researchers Study Next?

    Sep 1, 2026Fenghai Li, Zihan Tang, Haofei Yu +2Benchmarking Large Language Models

  14. Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?

    Aug 31, 2026Qiaoyuan Zheng, Yiqu YangItem Response TheoryLeaderboard

  15. FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models

    Aug 20, 2026Dingzirui Wang, Xuanliang Zhang, Keyan Xu +2Benchmarking Large Language ModelsFormalization

  16. RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks

    Aug 12, 2026Jinjun Huang, Zhongzhen Wen, Tongtong Xu +3Graphics Processing Unit KernelsBenchmarking Large Language Models

  17. The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance

    Aug 12, 2026Shailja Thakur, Sungeun An, Chad DeLuca +1Benchmarking Large Language Models

  18. TCS-BENCH: Benchmarking State-of-the-Art Generative AI Theoretical Computer Science Research Ability

    Aug 10, 2026Vincent Cohen-Addad, Dimitris Paparas, Ernest van Wijland +13Benchmarking Large Language ModelsReasoning Benchmark

  19. Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities

    Aug 9, 2026Harish Kashyap, Kiran Byadarhaly, Sriram Chakaravarthy +2DiagramsDiagnostic Benchmark

  20. Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure

    Aug 9, 2026Víctor GallegoGraphics Processing Unit KernelsCode Optimization

  21. LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs

    Aug 6, 2026Lukas Twist, Twm Stone, Helen Yannakoudakis +1Benchmarking Large Language ModelsModel Preference

  22. EpiBench: Can LLMs Understand Epitopes for Antibody Drug Discovery?

    Aug 6, 2026Zirui Wang, Jiaqi Wang, Qinghan Wang +4Epitope PredictionAntibody

  23. SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models

    Aug 5, 2026Sihan Hu, Lyuhan Huang, Youjin Deng +1Benchmarking Large Language ModelsFrontier Models

  24. What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend

    Aug 5, 2026Shahed Masoudian, Passant Shafaei, Monorama Swain +1Benchmarking Large Language ModelsLLM Inference Efficiency

  25. Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks

    Aug 4, 2026Shunfan Zheng, Dongsheng Shi, Yue Li +3Text-To-SqlBenchmarking Large Language Models

  26. Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks

    Aug 4, 2026Mohsen ArjmandiContext LengthLarge Language Model Memory

  27. Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks

    Aug 3, 2026Xiao Fei, Yang Zhang, Sarah Almeida Carneiro +1PsychometricsMultiple-Choice Questions

  28. Unified Multi-Dimensional Benchmark for Complex Graph Reasoning in Large Language Models

    Aug 2, 2026Fali Wang, Ali Al-Lawati, Iliyas Bektas +5Reasoning BenchmarkBenchmarking Large Language Models

  29. AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers

    Jul 31, 2026Tianyu Huai, Tingshuo Fan, Xinchi Chen +5Agentic OptimizationBenchmarking Large Language Models

  30. ARB: A Matched Authorship-Rewriting Benchmark Dataset for AI-Text Detector Evaluation

    Jul 31, 2026Gaetano Perrone, Simon Pietro RomanoMachine-Generated Text DetectionBenchmarking Large Language Models

  31. Benchmarking Frontier Large Language Models Against Official Crash Database Coding Using Police Crash Narratives

    Jul 31, 2026Sudhir Bharati, Rajendra K C Khatri, Sudip BharatiAccidentsBenchmarking Large Language Models

  32. InfoOps Bench: A live information operations safety benchmark

    Jul 30, 2026Dorian Quelle, Lisa-Maria Neudert, Jonathan Bright +1Artificial Intelligence BenchmarksBenchmarking Large Language Models

  33. RepBench: Compiling Benchmarks into Capability Representations for Large Language Models

    Jul 30, 2026Yanshi Li, Xueru Bai, Shuman Liu +1Benchmarking Large Language ModelsSynthetic Data

  34. Benchmarking LLM Competence on Logical Inference over Probability Operators

    Jul 29, 2026Nayera Hasan, Jack Greff, Alvin GrissomBenchmarking Large Language ModelsInference Engine

  35. Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks

    Jul 28, 2026Juan Francisco, Mandujano ReyesItem Response TheoryBenchmarking Large Language Models

  36. Benchmarking LLMs for Verilog Design Flows

    Jul 23, 2026Angshuman Chakravertty, Rahul Koshti, Buddhi Prakash Sharma +1VhdlHigh-Level Synthesis

  37. KernelGenBench: Can LLMs and Agents Write Efficient Kernels Across Operator Sources and Hardware Platforms?

    Jul 22, 2026Peiyu Zang, Jian Tao, Jialing Zhang +4Graphics Processing Unit KernelsBenchmarking Large Language Models

  38. SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation

    Jul 21, 2026Weifeng Sun, Ye Fan, Yuchen Chen +6Code GenerationBenchmarking Large Language Models

  39. Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains

    Jul 20, 2026Liam SwayneReasoning BenchmarkBenchmarking Large Language Models

  40. Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration

    Jul 18, 2026Maksim Sheverev, David Finkelstein, Sergey NikolenkoLarge Language Model MemoryAgent Memory

  41. GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models

    Jul 14, 2026Bidyarthi Paul, Nahida Jannat Mayouree, Md. Asif Karim +2BanglaMathematical Reasoning

  42. Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

    Jul 13, 2026Chuhan Shi, Xiaoquan Ren, Sicheng Song +3Scientific DiscoveryBenchmarking Large Language Models

  43. Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

    Jul 9, 2026Siddhartha Jain, Ameya VelingkerContext LengthLongbench

  44. REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming

    Jul 7, 2026Nicolas Koller, Andreas u. SchmidtReverse EngineeringDisassembly

  45. Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

    Jul 7, 2026So Hasegawa, Shailaja Keyur Sampat, Lei Liu +1Benchmarking Large Language Models

  46. Spectral Signatures of Large Language Models

    Jul 3, 2026Zhuoying Zhang, Ishan V. Prasad, Yuanzhe Hu +4Benchmarking Large Language ModelsLineages

  47. MedCalc-Pro: Solving Complex Medical Calculations with LLM Agents

    Jul 3, 2026Siran Zhao, Ruihui Hou, Ziyue Huai +2Clinical NotesBenchmarking Large Language Models

  48. Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks

    Jul 2, 2026Jihan Yao, Gantavya Bhatt, Arnav Das +16Benchmarking Large Language ModelsGeometry-Aware Uncertainty Coresets

  49. Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth

    Jun 30, 2026Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmad ElShiekh +2Arabic Natural Language ProcessingArabic

  50. Benchmarking Large Language Models on Floating-Point Error Classification

    Jun 30, 2026Lisa Taldir, Muhammad Ahmad Saeed, David Defour +2Benchmarking Large Language ModelsFloating-Point

  51. SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

    Jun 28, 2026Tiziano Santilli, Francesco Daghero, Mayhar Tourchi MoghaddamBenchmarking Large Language ModelsMultiple-Choice Questions

  52. Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking

    Jun 27, 2026Balázs Szalontai, Ábel Szauter, Balázs Márton +3BugBenchmarking Large Language Models

  53. Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data

    Jun 24, 2026Kylie AnglinConfidence IntervalsSample Size

  54. Evaluating LLMs on Real-World Software Performance Optimization

    Jun 24, 2026Ezgi Sarıkayak, Wenchao Gu, Hesham Ghonim +1Code OptimizationBenchmarking Large Language Models

  55. On the Stability of Prompt Ranking in Large Language Model Evaluation

    Jun 23, 2026Shaoshuai Du, Penghao Liang, Yixian Shen +3Benchmarking Large Language ModelsLarge Language Model Evaluation

  56. Benchmarking Large Language Models for Grapheme-to-Phoneme Conversion: A Japanese Case Study

    Jun 20, 2026Tomoki KoriyamaBenchmarking Large Language ModelsZero-Shot Text-To-Speech

  57. The Metanym Game: A Self-Contained, Self-Consistent LLM Peer-Community Benchmark for Structural Intelligence

    Jun 19, 2026David NordforsBenchmarking Large Language ModelsAnalogy

  58. Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

    Jun 18, 2026Maria Ivanova, Pavel Zadorozhny, Rodion Levichev +5Benchmarking Large Language ModelsCode Generation

  59. QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation

    Jun 18, 2026Xinyi Zheng, Ling Shi, Tianlong Yu +3Reasoning BenchmarkLarge Language Model Reasoning

  60. SciRisk-Bench: A Risk-Dimension-Aware Benchmark for AI4Science Safety

    Jun 17, 2026Linghao Feng, Yinqian Sun, Dongqi Liang +8Large Language Model SafetyBenchmarking Large Language Models