Large Language Model Benchmarks

Ask about this topic

Recent momentum

-56%

11 papers in the last 28 days · 0.3% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Also in Llm-As-A-Judge

Neighbouring directions

Large Language Model EvaluationLlm-As-A-JudgeLarge Language Model JudgesJudgesFairjudge Models

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-14

4 new papers

A weekly snapshot of new work published in Large Language Model Benchmarks.

Period ending 2026-09-07

3 new papers

A weekly snapshot of new work published in Large Language Model Benchmarks.

167 papers

Latest in Large Language Model Benchmarks

Open your feed →
CardsList

  1. K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

    Sep 14, 2026Laura M. Vowels, Matthew J. Vowels, Shivali Sharma +9Large Language Model SafetyLarge Language Model Benchmarks

  2. How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks

    Sep 14, 2026Ali Ansari, Haoran Sun, Andy Zeyi Liu +48Physics ProblemLarge Language Model Benchmarks

  3. GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMs

    Sep 14, 2026Zixiang Xu, Yanbo Wang, Chenxi Wang +6Reasoning GraphsLarge Language Model Benchmarks

  4. Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models

    Sep 14, 2026Zhongzhan Huang, Junxin Li, Guoming Ling +3Large Language Model BenchmarksLarge Language Model Compression

  5. Φ-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?

    Sep 9, 2026Leilei Ding, Shumin Wang, Yuting Huang +10Large Language Model BenchmarksRepository

  6. YallaMorph: A Benchmark for Evaluating Arabic Morphological Generation in Large Language Models

    Sep 9, 2026Mahmoud Reda, Salam Khalifa, Reham Marzouk +1Arabic Natural Language ProcessingLarge Language Model Benchmarks

  7. S^3-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistants

    Sep 9, 2026Heyang Liu, Jiayi Huang, Wenyang Xiao +8Speech Language ModelsVoice Agents

  8. PerfReasoning: How Well Do LLMs Reason on Hardware Performance?

    Sep 7, 2026Dan Zhao, Karthikeyan Sankaralingam, Christos Kozyrakis +1LLM Reasoning MethodsLarge Language Model Benchmarks

  9. To What Extent Do Large Language Models Understand Bangla Idioms?

    Sep 3, 2026Mousumi Akter, Md. Faiyaz Abdullah Sayeedi, Nurul Labib Sayeedi +1Multilingual IdiomsLarge Language Model Benchmarks

  10. Benchmarking Language Models for Statistical Problem Formulation

    Sep 2, 2026Chen Wang, Junzhe Zhao, Xin Cong +2Large Language Model BenchmarksStatistics

  11. VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean Preferences

    Sep 1, 2026Yiwen Jiang, Yang Deng, Stephanie Fong +9Large Language Model PersonalizationPersonalization

  12. Can Large Language Models Forecast What Researchers Study Next?

    Sep 1, 2026Fenghai Li, Zihan Tang, Haofei Yu +2Scientific IdeationLarge Language Model Benchmarks

  13. Perceptually Better, Semantically Worse: Measuring Speech Enhancement Impact on LLM-Based Voice Systems

    Aug 31, 2026Randy Frans Fela, Pejman MowlaeeSpeech EnhancementWhisper-Large-V3

  14. Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation

    Aug 30, 2026Nishant Balepur, Paiheng Xu, Wei Ai +3Multiple-Choice QuestionsScoring Rules

  15. FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models

    Aug 20, 2026Dingzirui Wang, Xuanliang Zhang, Keyan Xu +2Large Language Model BenchmarksFormalization

  16. RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks

    Aug 12, 2026Jinjun Huang, Zhongzhen Wen, Tongtong Xu +3Graphics Processing Unit KernelsLarge Language Model Benchmarks

  17. The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance

    Aug 12, 2026Shailja Thakur, Sungeun An, Chad DeLuca +1Large Language Model BenchmarksState Drift

  18. From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models

    Aug 11, 2026Si'an Xie, Jiaxun Liu, Biao Yang +4Large Language Model BenchmarksAgentic Pipeline

  19. How Robust Are LLMs to Vietnamese Dialects?

    Aug 11, 2026Minh Tran, Trinh Chau, Thanh-Nhan Le +4DialectsLarge Language Model Benchmarks

  20. TCS-BENCH: Benchmarking State-of-the-Art Generative AI Theoretical Computer Science Research Ability

    Aug 10, 2026Vincent Cohen-Addad, Dimitris Paparas, Ernest van Wijland +13Large Language Model BenchmarksAI Benchmarks

  21. Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities

    Aug 9, 2026Harish Kashyap, Kiran Byadarhaly, Sriram Chakaravarthy +2DiagramsDiagnostic Benchmark

  22. LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs

    Aug 6, 2026Lukas Twist, Twm Stone, Helen Yannakoudakis +1Large Language Model BenchmarksRecommendation Quality

  23. LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs

    Aug 5, 2026Jiahao Zhang, Yongzhi Tong, Zelin Fu +4Large Language Model PersonalizationPersonalization

  24. What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend

    Aug 5, 2026Shahed Masoudian, Passant Shafaei, Monorama Swain +1Large Language Model BenchmarksLLM Inference Efficiency

  25. Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks

    Aug 4, 2026Shunfan Zheng, Dongsheng Shi, Yue Li +3Text-To-SqlLarge Language Model Benchmarks

  26. ConlangBench: Exploring Language Knowledge and Learning in LLMs through Diverse Constructed Languages

    Aug 4, 2026Jinhong Jeong, Seungyeop Yi, Sangah Lee +1Large Language Model BenchmarksIndic Languages

  27. Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks

    Aug 4, 2026Mohsen ArjmandiContext LengthRetrieval Benchmarks

  28. Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks

    Aug 3, 2026Xiao Fei, Yang Zhang, Sarah Almeida Carneiro +1Multiple-Choice QuestionsPsychometrics

  29. Semantic Alignment of AI Models: Concept Collapse, Checkpoint Dynamics, and Cross-Lingual Transfer

    Aug 3, 2026Tyler Ashoff, Jordan RoduLarge Language Model AlignmentLarge Language Model Benchmarks

  30. AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers

    Jul 31, 2026Tianyu Huai, Tingshuo Fan, Xinchi Chen +5Agentic BenchmarksLarge Language Model Benchmarks

  31. ARB: A Matched Authorship-Rewriting Benchmark Dataset for AI-Text Detector Evaluation

    Jul 31, 2026Gaetano Perrone, Simon Pietro RomanoMachine-Generated Text DetectionRewrite

  32. RepBench: Compiling Benchmarks into Capability Representations for Large Language Models

    Jul 30, 2026Yanshi Li, Xueru Bai, Shuman Liu +1Large Language Model BenchmarksAI Benchmarks

  33. ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Models

    Jul 29, 2026Ruxi Gu, Zhenliang Zhang, Wei WangLarge Language Model MemoryLarge Language Model Benchmarks

  34. Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks

    Jul 28, 2026Juan Francisco, Mandujano ReyesItem Response TheoryLarge Language Model Benchmarks

  35. Benchmarking LLMs for Verilog Design Flows

    Jul 23, 2026Angshuman Chakravertty, Rahul Koshti, Buddhi Prakash Sharma +1VhdlElectronic Design Automation

  36. RUMBA: Russian User Memory Benchmark

    Jul 23, 2026Elizaveta Shevtsova, Inna Glebkina, Mark Baushenko +2Dialogue BenchmarksLarge Language Model Benchmarks

  37. When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs

    Jul 23, 2026Anna Mosolova, Djamé SeddahLarge Language Model BenchmarksMultilingual Benchmark

  38. KernelGenBench: Can LLMs and Agents Write Efficient Kernels Across Operator Sources and Hardware Platforms?

    Jul 22, 2026Peiyu Zang, Jian Tao, Jialing Zhang +4Graphics Processing Unit KernelsLarge Language Model Benchmarks

  39. SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation

    Jul 21, 2026Weifeng Sun, Ye Fan, Yuchen Chen +6AI Code GenerationLarge Language Model Benchmarks

  40. Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains

    Jul 20, 2026Liam SwayneReasoning BenchmarkLarge Language Model Benchmarks

  41. JOR-Bench: Japanese Operations Research Benchmarks for Large Language Models

    Jul 18, 2026Yuu JinnaiLarge Language Model BenchmarksAI Benchmarks

  42. GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models

    Jul 14, 2026Bidyarthi Paul, Nahida Jannat Mayouree, Md. Asif Karim +2BanglaLarge Language Model Benchmarks

  43. Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

    Jul 13, 2026Chuhan Shi, Xiaoquan Ren, Sicheng Song +3Scientific DiscoveryLarge Language Model Benchmarks

  44. Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

    Jul 9, 2026Siddhartha Jain, Ameya VelingkerContext LengthLarge Language Model Benchmarks

  45. Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System

    Jul 8, 2026Yiming Gai, Junde Lu, Xuefei HuangLarge Language Model EvaluationLarge Language Model Benchmarks

  46. REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming

    Jul 7, 2026Nicolas Koller, Andreas u. SchmidtReverse EngineeringDisassembly

  47. Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

    Jul 7, 2026So Hasegawa, Shailaja Keyur Sampat, Lei Liu +1Large Language Model BenchmarksData Science Agents

  48. Spectral Signatures of Large Language Models

    Jul 3, 2026Zhuoying Zhang, Ishan V. Prasad, Yuanzhe Hu +4Large Language Model BenchmarksLineages

  49. PPE-Bench: A Benchmark for Evaluating MLLM Unlearning under Private-Public Entanglement

    Jul 3, 2026Xianren Zhang, Delvin Ce Zhang, Dongwon Lee +1Large Language Model UnlearningHeuristic Privacy Metrics

  50. Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks

    Jul 2, 2026Jihan Yao, Gantavya Bhatt, Arnav Das +16Large Language Model BenchmarksGeometry-Aware Uncertainty Coresets

  51. Benchmarking Large Language Models on Floating-Point Error Classification

    Jun 30, 2026Lisa Taldir, Muhammad Ahmad Saeed, David Defour +2Large Language Model BenchmarksFloating-Point

  52. Bridging Scientific Heritage: An Arabic--Russian Parallel Corpus and LLM Benchmark for Sustainable Knowledge Transfer

    Jun 29, 2026M. K. ArabovMultilingual Language ModelsLarge Language Model Benchmarks

  53. SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

    Jun 28, 2026Tiziano Santilli, Francesco Daghero, Mayhar Tourchi MoghaddamLarge Language Model BenchmarksMultiple-Choice Questions

  54. Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking

    Jun 27, 2026Balázs Szalontai, Ábel Szauter, Balázs Márton +3BugLarge Language Model Benchmarks

  55. Evaluating LLMs on Real-World Software Performance Optimization

    Jun 24, 2026Ezgi Sarıkayak, Wenchao Gu, Hesham Ghonim +1Code OptimizationLarge Language Model Benchmarks

  56. On the Stability of Prompt Ranking in Large Language Model Evaluation

    Jun 23, 2026Shaoshuai Du, Penghao Liang, Yixian Shen +3Large Language Model EvaluationLarge Language Model Benchmarks

  57. Bayesian Adaptation Gym: A Benchmark for the Bayesian Low-Rank Adaptation of Multi-Modal Language Models

    Jun 20, 2026Colin Samplawski, Ramneet Kaur, Manoj Acharya +2Large Language Model AdaptationLarge Language Model Benchmarks

  58. Benchmarking Large Language Models for Grapheme-to-Phoneme Conversion: A Japanese Case Study

    Jun 20, 2026Tomoki KoriyamaGrapheme-To-PhonemeLarge Language Model Benchmarks

  59. Evaluation of Small Language Models for Arabic Language Processing

    Jun 19, 2026Jumana Alsubhi, Ahmed Alhusayni, Abdulrahman Gharawi +5Arabic Natural Language ProcessingSmall Large Language Models

  60. The Metanym Game: A Self-Contained, Self-Consistent LLM Peer-Community Benchmark for Structural Intelligence

    Jun 19, 2026David NordforsLarge Language Model BenchmarksAnalogy