LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Apr 19, 2026Miaosen Chai, Wang Bill Zhu, Shangshang Wang +5LLM Evaluation Automated Program Repair
University of Southern California · Microsoft · University of Wisconsin–Madison +1
Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty
Apr 19, 2026Jingyi Ren, Ante Wang, Yunghwei Lai +5LLM Evaluation Selective Prediction
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China
From Legal Text to Executable Decision Models: Evaluating Structured Representations for Legal Decision Model Generation
Apr 18, 2026David GrausLLM Evaluation Legal Document Analysis
University of Amsterdam · Amsterdam, The Netherlands
SciImpact: A Multi-Dimensional, Multi-Field Benchmark for Scientific Impact Prediction
Apr 18, 2026Hangxiao Zhu, Yuyu Zhang, Ping Nie +1LLM Evaluation
Texas A&M University · Verdent AI · University of Waterloo
Do LLMs Hold Their Values? MANTA: A Multi-Turn Adversarial Benchmark for Animal Welfare Reasoning
Apr 18, 2026Isabella Luong, Joyee Chen, Arturs Kanepajs +5LLM Evaluation Moral Reasoning in Language Models
SPAR · Compassion Aligned Machine Learning · NUS +2
Beyond Word Boundaries: A Hebrew Coreference Benchmark and an Evaluation Protocol for Morphologically Complex Text
Apr 18, 2026Refael Shaked Greenfeld, Reut TsarfatyCoreference Resolution LLM Evaluation
Bar-Ilan University
Configuration Over Selection: Hyperparameter Sensitivity Exceeds Model Differences in Open-Source LLMs for RTL Generation
Apr 18, 2026Minghao Shao, Zeng Wang, Weimin Fu +5LLM Evaluation Language Model Generation Evaluation
NYU Tandon School of Engineering, USA · Kansas State University, USA · NYU Abu Dhabi, UAE
Auditing Support Strategies in LLMs through Grounded Multi-Turn Social Simulation
Apr 18, 2026Michelle Star, Andrew Aquilina, Yu-Ru LinLLM Evaluation Emotional Support Conversation
School of Computing and Information, University of Pittsburgh
PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinations
Apr 18, 2026Yuhe Wu, Guangyu Wang, Yuran Chen +6LLM Evaluation Hallucination in Language Models
HKUST(GZ) · NYUSH · DUFE +2
Learning to Trade Like an Expert: Cognitive Fine-Tuning for Stable Financial Reasoning in Language Models
Apr 18, 2026Yuchen Pan, Soung Chang LiewLLM Evaluation LLM Fine-Tuning
Department of Information Engineering, The Chinese University of Hong Kong
Expressing Social Emotions: Misalignment Between LLMs and Human Cultural Emotion Norms
Apr 18, 2026Sree Bhattacharyya, Manas Mehta, Leona Chen +4LLM Evaluation Cultural Bias in Language Models
The Pennsylvania State University · Duke University · Northeastern University +1
Machine individuality: Separating genuine idiosyncrasy from response bias in large language models
Apr 18, 2026Valentin Kriegmair, Dirk U. WulffLLM Evaluation Personality Modeling in Language Models
Center for Adaptive Rationality, Max Planck Institute for Human Development, 14195 Berlin, Germany · Center for Cognitive and Decision Sciences, Department of Psychology, University of Basel, 4055 Basel, Switzerland
Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models
Apr 17, 2026Yang Liu, Hongming Li, Melissa Xiaohui Qin +2LLM Evaluation
University of Science and Technology Beijing · 2State Key Laboratory of General Artificial Intelligence, BIGAI
Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Design
Apr 17, 2026Shriram Chennakesavalu, Kirill Shmilovich, Hayley Weir +5LLM Evaluation Molecular Property Prediction
Prescient Design, Genentech, South San Francisco, CA, USA
No Universal Courtesy: A Cross-Linguistic, Multi-Model Study of Politeness Effects on LLMs Using the PLUM Corpus
Apr 17, 2026Hitesh Mehta, Arjit Saxena, Garima Chhikara +1Multilingual Language Model Evaluation LLM Evaluation
Department of Computer Science Engineering, Delhi Technological University, New Delhi, India · Department of Electronics and Communication Engineering, Delhi Technological University, New Delhi, India
From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text
Apr 17, 2026Van-Truong LeLLM Evaluation Legal Document Analysis
University of Science, VNUHCM Vietnam National University Hanoi Open University Ho Chi Minh, Vietnam · Vietnam National University · Hanoi Open University
SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation
Apr 17, 2026Deshan Sumanathilaka, Nicholas Micallef, Julian Hough +1LLM Evaluation Few-Shot Prompting
Department of Computer Science, Swansea University, Wales, UK
Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models
Apr 17, 2026Reham Alharbi, Valentina Tamma, Terry R. Payne +1LLM Evaluation
Taibah University, Madinah, Saudi Arabia · University of Liverpool, Liverpool, UK
BAGEL: Benchmarking Animal Knowledge Expertise in Language Models
Apr 17, 2026Jiacheng Shen, Masato Hagiwara, Milad Alizadeh +9LLM Evaluation Question Answering
2NYU Center for Data Science; NYU Shanghai · 1Earth Species Project · 3NYU Shanghai Center for Data Science; NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai
On the Robustness of LLM-Based Dense Retrievers: A Systematic Analysis of Generalizability and Stability
Apr 17, 2026Yongkang Li, Panagiotis Eustratiadis, Yixing Fan +1LLM Evaluation Retrieval Robustness
University of Amsterdam, The Netherlands · Chinese Academy of Sciences, China
MEDLEY-BENCH: Scale Buys Evaluation but Not Control in AI Metacognition
Apr 17, 2026Farhad Abtahi, Abdolamir Karbalaie, Eduardo Illueca-Fernandez +1Belief Revision LLM Evaluation
Department of Clinical Science, Intervention and Technology (CLINTEC), Karolinska Institutet, 17177 Stockholm, Sweden · Department of Clinical Physiology, Karolinska University Hospital, 17176 Stockholm, Sweden · Department of Biomedical Engineering and Health Systems, KTH Royal Institute of Technology, 14157 Huddinge, Sweden +2
How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Models
Apr 17, 2026Judith Sieker, Sina ZarrießLLM Evaluation LLM-as-a-Judge
Computational Linguistics, Department of Linguistics Bielefeld University, Germany
QuantSightBench: Evaluating LLM Quantitative Forecasting with Prediction Intervals
Apr 17, 2026Jeremy Qin, Maksym AndriushchenkoLLM Evaluation Forecasting Benchmarks
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center
Artificial Effort
Apr 17, 2026Federico Belotti, Stefano Coniglio, Antonio Cosma +1LLM Evaluation
University of Bergamo, Italy.
Exploring the Capability Boundaries of LLMs in Mastering of Chinese Chouxiang Language
Apr 17, 2026Dianqing Lin, Tian Lan, Jiali Zhu +7LLM Evaluation
College of Computer Science, Inner Mongolia University, China
KWBench: Measuring Unprompted Problem Recognition in Knowledge Work
Apr 17, 2026Ankit MalooLLM Evaluation
Clio AI
The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring
Apr 17, 2026Jon-Paul CacioliLLM Evaluation Language Model Calibration
Independent Researcher Melbourne, Australia
LLMbench: A Comparative Close Reading Workbench for Large Language Models
Apr 16, 2026David M. BerryLLM Evaluation LLM Interpretability
University of Sussex
Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
Apr 16, 2026Manan Gupta, Dhruv KumarLLM Evaluation LLM-as-a-Judge
1BITS Pilani, Pilani Campus, India
PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research
Apr 16, 2026Tingjia Miao, Wenkai Jin, Muhua Zhang +19LLM Evaluation AI Agent Benchmarks
School of Artificial Intelligence, Shanghai Jiao Tong University · Zhiyuan College, Shanghai Jiao Tong University · School of Physics and Astronomy, Shanghai Jiao Tong University +7

Apr 19, 2026, cs.SEJ/K move · Enter open · S save

Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?

Miaosen Chai, Wang Bill Zhu, Shangshang Wang, Yejia Liu+4

University of Southern California · Microsoft · University of Wisconsin–Madison+1

LLM Evaluation Automated Program Repair

Open paper AskPDF arXiv

Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during debugging. To evaluate how far LLMs are from precise debugging, we introduce the Precise Debugging Benchmark (PDB) framework, which automatically converts any coding dataset into a debugging benchmark with precision-aware evaluation. PDB generates buggy programs by synthesizing verified atomic bugs and composing them into multi-bug programs. We define two novel metrics, edit-level precision and bug-level recall, which measure how many necessary edits are made and how many bugs are resolved. We release two evaluation benchmarks: PDB-Single-Hard on single-line bugs, and PDB-Multi on multi-line bugs. Experiments show that frontier models, such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking, achieve unit-test pass rates above 76% but exhibit precision below 45%, even when explicitly instructed to perform minimal debugging. Finally, we show that iterative and agentic debugging strategies do not substantially improve precision or recall, highlighting the need to rethink post-training pipelines for coding models.

Momentum

Related topics

Latest papers 1,631