LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

Evaluation of LLMs for Mathematical Formalization in Lean
Jun 4, 2026Tyson Klingner, Drew Bladek, Escher Crawford +6LLM Evaluation Lean Theorem Proving
Math AI Lab, University of Washington, Seattle, WA, USA. · Math AI Lab, University of Washington, Seattle, WA
Exploring LLMs for South Asian Music Understanding and Generation
Jun 3, 2026Faria Binte Kader, Mohtasim Hadi Rafi, Shah Wasif Sajjad +1Music Generation Evaluation LLM Evaluation
University of Central Florida · Florida, USA · Auburn University +1
Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data
Jun 3, 2026XiuYu Zhang, Yi Shan, Junfeng Fang +1LLM Evaluation LLM-as-a-Judge
National University of Singapore · Beijing University of Technology
Knowledge Index of Noah's Ark
Jun 3, 2026Sheng Jin, Minghao Liu, Yunze Xiao +24LLM Evaluation Human-in-the-Loop Annotation
12077AI · 2M-A-P · 3Carnegie Mellon University +7
Large Language Models in K-12 Education: Alignment with State Curriculum Standards and Student Personas
Jun 3, 2026Lisa Korver, Tomo Lazovich, Sherief RedaLLM Evaluation AI in Education
Department of Computer Engineering · Brown University · Providence, RI 02906 +1
FALSIFYBENCH: Evaluating Inductive Reasoning in LLMs with Rule Discovery Games
Jun 3, 2026Leonardo Bertolazzi, Katya Tentori, Raffaella BernardiLLM Evaluation AI Agent Benchmarks
University of Trento · Free University of Bozen-Bolzano
Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models
Jun 3, 2026Sabrina Kaniewski, Fabian Schmidt, Tobias HeerRetrieval-Augmented Generation LLM Evaluation
Institute for Secure Networked Systems, Esslingen University, Esslingen, Germany · Institute for Intelligent Systems, Esslingen University, Esslingen, Germany
Self-Evolving Deep Research via Joint Generation and Evaluation
Jun 3, 2026Han Zhu, Chengkun Cai, Yuanfeng Song +3LLM Evaluation LLM Agent Training
1The Hong Kong University of Science and Technology · 2ByteDance, China · University College London
SANE Schema-aware Natural-language Evaluation of Biological Data
Jun 3, 2026Rolf Gattung, Martin Krueger, Markus ReischlLLM Evaluation LLM Grounding
Institute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Hermann-von-Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, Germany.
Do Large Language Models Have Emotions?
Jun 3, 2026Amit Goldenberg, James J. GrossLLM Evaluation LLM Interpretability
Harvard Business School · Department of Psychology, Harvard University · Harvard University, Digital, Data and Design Institute +1
FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses
Jun 3, 2026Nicholas SabanLLM Evaluation Agent Evaluation
Patronus AI · University of California, Berkeley
Long Live Fine-Tuning: Task-Specific Transformers Outperform Zero-Shot LLMs for Misinformation Response Classification on Reddit
Jun 2, 2026JooYoung Lee, Lin Tian, Angela Brillantes +2LLM Evaluation Zero-Shot Text Classification
University of Technology Sydney, Sydney, Australia.
Thinking Through Signs: PEEL as a Semiotic Scaffolding for Epistemically Accountable AI-Enabled Research
Jun 2, 2026Clarisse de Souza, Gabriel Barbosa, Simone Diniz Junqueira Barbosa +3LLM Evaluation AI Accountability
PUC-Rio, PUC-Behring Institute of Artificial Intelligence, Brazil · PUC-Rio, Brazil
Discourse-Role Labels as Presentation-Time Variables for Context Use in Language Models
Jun 2, 2026Jianguo Zhu, Xiangmei Li, Wenjie LiuPrompt Sensitivity LLM Evaluation
School of Cybersecurity (Xin Gu Industrial College), Chengdu University of Information, Chengdu, 610225, China · Sichuan Provincial Key Laboratory of Cyberspace Security, Chengdu, 610225, China · Advanced Cryptography and System Security Key Laboratory of Sichuan Province, Chengdu, 610225, China +1
Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics
Jun 2, 2026Mutsumi Sasaki, Go kamoda, Ryosuke Takahashi +4Numerical Reasoning in Language Models LLM Evaluation
Tohoku University · SOKENDAI · NINJAL +2
Quantifying Faithful Confidence Expression in Large Reasoning Models
Jun 2, 2026Areeb Gani, Asal Meskin, Gabrielle Kaili-May Liu +1LLM Evaluation Confidence Estimation in Language Models
1Yale University
Reasoning Structure of Large Language Models
Jun 2, 2026Frédéric Berdoz, Luca A. Lanzendörfer, Fabian Farestam +1Logical Reasoning LLM Evaluation
1ETH Zurich, Switzerland.
Clustered Self-Assessment: A Simple yet Effective Method for Uncertainty Quantification in Large Language Models
Jun 2, 2026Qi Cao, Takeshi Kojima, Andrew Gambardella +3LLM Evaluation Uncertainty Quantification
The University of Tokyo, Japan
From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models
Jun 2, 2026Hongyu Guo, Hao Li, He Cao +2LLM Evaluation LLM Reasoning
Peking University, Shenzhen Graduate School · International Digital Economy Academy (IDEA)
CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks
Jun 2, 2026Alexander Apartsin, Yehudit ApersteinModel Selection LLM Evaluation
Holon Institute of Technology, Israel · Afeka Tel Aviv Academic College of Engineering, Israel
Evaluating LLMs' Effectiveness on Real-World Consumer Device Repair Questions
Jun 2, 2026Atm Mizanur Rahman, Md Arid Hasan, Syed Ishtiaque Ahmed +1Multilingual Language Model Evaluation LLM Evaluation
University of Illinois Urbana-Champaign · University of Toronto
Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions
Jun 2, 2026Xuan Yang, Hao Xu, Tingfeng Hui +4LLM Evaluation Tool-Use Evaluation
Department of Data Science, City University of Hong Kong · 3Li Auto Inc. · 2Hong Kong Institute of AI for Science, City University of Hong Kong +1
Beyond "To whom it may concern": Tailoring Machine Translation to Audience and Intent
Jun 2, 2026Raphael Merx, Ekaterina Vylomova, Trevor CohnLLM Evaluation Machine Translation
1The University of Melbourne · 2Google
When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming
Jun 2, 2026Zelalem Abahana, David Evans, Satish Mahadevan Srinivasan +1Reward Hacking LLM Evaluation
College of Engineering, The Pennsylvania State University, University Park, PA, USA · Department of Computer Science, University of Virginia, Charlottesville, VA, USA · 3Penn State Great Valley, The Pennsylvania State University, Malvern, PA, USA +1
SenseJudge: Human-Centric Preference-Driven Judgment Framework
Jun 2, 2026Rui Li, Junfeng Liu, Xiangwen Kong +2LLM Evaluation LLM-as-a-Judge
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · StepFun · Xi’an Jiaotong University
Fully Automated Identification of Lexical Alignment and Preference-Stage Shifts in Large Language Models
Jun 2, 2026Thomas Stephan Juzek, Xiaoyang Ming, Jose A. HernandezLLM Evaluation LLM Alignment
Florida State University
Rethinking Molecular Text Representations for LLMs: An Empirical Study
Jun 2, 2026Arun Raja, Garrett M. Morris, Kian Ming A. ChaiLLM Evaluation Molecular Generation
University of Oxford · DSO National Laboratories
The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment
Jun 2, 2026Sourabrata Mukherjee, Hamna Hamna, Kalika Bali +1Inter-Rater Reliability Multilingual Language Model Evaluation
Microsoft Research
TriEval: A Resource-Efficient Pipeline for LLM Bias, Toxicity, and Truthfulness Assessment
Jun 2, 2026Akshatha Srikantha, Manpreet Singh, Yash Jajoo +1LLM Evaluation Language Model Safety Evaluation
University of California, Irvine, CA, USA · Boston University, Boston, MA, USA · New York University, New York City, NY, USA +1
Pretraining Language Models on Historical Text
Jun 2, 2026Xiaoxi Luo, Zachary Shinnick, Niclas Griesshaber +5Language Model Pretraining LLM Evaluation
University of Waterloo · Vector Institute · AIML, Adelaide University +3

Jun 4, 2026, cs.AIJ/K move · Enter open · S save

Evaluation of LLMs for Mathematical Formalization in Lean

Tyson Klingner, Drew Bladek, Escher Crawford, Bohao Chen+5

Math AI Lab, University of Washington, Seattle, WA, USA. · Math AI Lab, University of Washington, Seattle, WA

LLM Evaluation Lean Theorem Proving LLM Mathematical Reasoning

Open paper AskPDF arXiv

Within the past few years, the ability of Large Language Models (LLMs) to generate formal mathematical proofs has improved drastically. We provide a comparison of various LLMs' effectiveness in producing formal proofs in Lean 4 with the goal of assisting those seeking to use LLMs to support their own projects. We utilize both pass@

k

and refine@

k

metrics as the benchmark for our comparison and evaluate on subsets of both miniF2F and miniCTX datasets. Our testing shows that overall, Gemini 3.1 Pro and Claude Opus 4.7 perform best. Gemini 3.1 Pro achieved a 92% success rate on miniF2F via refine@32 whereas Opus 4.7 achieved a 86% success rate on miniCTX via refine@32. When taking cost into account, NVIDIA Nemotron 3 Super and GPT-OSS 120B were the most efficient, with competitive accuracies and average costs of

<\

0.01$ per correct proof.

Momentum

Related topics

Latest papers 1,643