LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,446 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code
Apr 25, 2026Jelena Ilić VulićevićLLM Evaluation On-Device Language Model Inference
Independent Researcher
Evaluating Large Language Models on Computer Science University Exams in Data Structures
Apr 25, 2026Edan Gabay, Yael Maoz, Jonathan Stahl +7LLM Evaluation Computer Science Education
Blavatnik School of Computer Science and AI
Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning Perspective
Apr 25, 2026Bishwamittra Ghosh, Soumi Das, Till Speicher +5LLM Evaluation LLM Fine-Tuning
Max Planck Institute for Software Systems, Germany · Boston University, USA
Evaluating Temporal Consistency in Multi-Turn Language Models
Apr 24, 2026Yash Kumar Atri, Steven L. Johnson, Tom HartvigsenLLM Evaluation Temporal Reasoning in Language Models
University of Virginia
A Systematic Approach for Large Language Models Debugging
Apr 24, 2026Basel Shbita, Anna Lisa Gentile, Bing Zhang +10LLM Evaluation
1IBM Research, San Jose, CA, USA · 2IBM Research, Bengaluru, India
AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputs
Apr 24, 2026Pouya Pezeshkpour, Estevam HruschkaLLM Evaluation LLM Answer Verification
Megagon Labs
Rethinking Math Reasoning Evaluation: A Robust LLM-as-a-Judge Framework Beyond Symbolic Rigidity
Apr 24, 2026Erez Yosef, Oron Anschel, Shunit Haviv Hakimi +4Mathematical Reasoning Benchmarks LLM Evaluation
Tel-Aviv University · Amazon Prime Video · Ben-Gurion University
BLAST: Benchmarking LLMs with ASP-based Structured Testing
Apr 24, 2026Manuel Alejandro Borroto Santana, Erica Coppolillo, Francesco Calimeri +3Logic Programming LLM Evaluation
University of Calabria, Rende, Italy · ICAR-CNR, Rende, Italy · Artificial Intelligence LAB, DeMaCS, University of Calabria, Rende, Italy +1
Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screen
Apr 24, 2026Jon-Paul CacioliLLM Evaluation Confidence Estimation in Language Models
Independent Researcher, Melbourne, Australia
An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models
Apr 24, 2026Juan Manuel ContrerasLLM Evaluation Language Model Self-Assessment
Independent Researcher
When AI Speaks, Whose Values Does It Express? A Cross-Cultural Audit of Individualism-Collectivism Bias in Large Language Models
Apr 24, 2026Pruthvinath Jeripity VenkataLLM Evaluation Cross-Cultural Language Model Evaluation
Independent Researcher
The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
Apr 23, 2026Abel YagubyanInter-Rater Reliability LLM Evaluation
Independent Researcher
MathDuels: Evaluating LLMs as Problem Posers and Solvers
Apr 23, 2026Zhiqiu Xu, Shibo Jin, Shreya Arya +1Mathematical Reasoning Benchmarks LLM Evaluation
Department of Computer and Information Science, University of Pennsylvania · Rabdos AI · Department of Mathematics, University of Pennsylvania
Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms
Apr 23, 2026Yuto Nishida, Naoki Shikoda, Yosuke Kishinami +4LLM Evaluation Memorization in Language Models
Nara Institute of Science and Technology · Future Corporation
Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboards
Apr 23, 2026Minji Jung, Minjae Lee, Yejin Kim +2LLM Evaluation Human-in-the-Loop Evaluation
Yonsei University, South Korea
A Metamorphic Testing Approach to Diagnosing Memorization in LLM-Based Program Repair
Apr 23, 2026Milan De Koning, Ali Asgari, Pouria Derakhshanfar +1LLM Evaluation Metamorphic Testing
JetBrains Research, Amsterdam, The Netherlands · Delft University of Technology, Delft, The Netherlands
OptiVerse: A Comprehensive Benchmark towards Optimization Problem Solving
Apr 23, 2026Xinyu Zhang, Boxuan Zhang, Yuchen Wan +5LLM Evaluation Benchmark Design
School of Computer Science and Technology, Xi’an Jiaotong University · 2Ministry of Education Key Laboratory of Intelligent Networks and Network Security, China · 4Lenovo Research +1
Evaluating AI Meeting Summaries with a Reusable Cross-Domain Pipeline
Apr 23, 2026Philip Zhong, Don Wang, Jason ZhangLLM Evaluation Factual Consistency Evaluation
Webex Suite AI Cisco Systems, Inc. San Jose, California, USA
EngramaBench: Evaluating Long-Term Conversational Memory with Structured Graph Retrieval
Apr 23, 2026Julian AcunaLong-Term Conversational Memory LLM Evaluation
A Demonstration of SQLyzr: A Platform for Fine-Grained Text-to-SQL Evaluation and Analysis
Apr 23, 2026Sepideh Abedini, M. Tamer ÖzsuLLM Evaluation Text-to-SQL
University of Waterloo
SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation
Apr 22, 2026Ruohan Liu, Shukang Yin, Tao Wang +6Audio-Language Model Evaluation LLM Evaluation
1Nanjing University · 2Xiaomi
Diagnosing CFG Interpretation in LLMs
Apr 22, 2026Hanqi Li, Lu Chen, Kai YuLLM Evaluation In-Context Learning
1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 3Shanghai Innovation Institution, Shanghai, China · 4Jiangsu Key Lab of Language Computing, Suzhou, China +2
Exploiting LLM-as-a-Judge Disposition on Free Text Legal QA via Prompt Optimization
Apr 22, 2026Mohamed Hesham Elganayni, Runsheng Chen, Sebastian Nagl +1LLM Evaluation LLM-as-a-Judge
Technical University of Munich Germany
Cross-Dataset Bloom Question Classification: Supervised Models and Prompted LLMs
Apr 22, 2026Abdolali Faraji, Mohammadreza Molavi, Zohreh Rasoulkhani +2LLM Evaluation Educational Assessment
Leibniz Information Centre for Science and Technology (TIB) · University of Genoa
Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint Analysis
Apr 22, 2026Viet-Man Le, Thi Ngoc Trang Tran, Sebastian Lubos +2Software Engineering LLM Evaluation
Graz University of Technology Graz, Austria
Benchmarking Large Language Models for Safety Data Extraction
Apr 22, 2026Jonas Grill, Thomas Bayer, Sören BerlingerLLM Evaluation LLM Prompting
1SAP SE, Germany. · Institute for Digital Transformation, Ravensburg-Weingarten University.
DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectories
Apr 22, 2026Neemesh Yadav, Palakorn Achananuparp, Jing Jiang +1LLM Evaluation Theory of Mind
Singapore Management University · Australian National University
CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge
Apr 22, 2026Gustav Keppler, Ghada Elbez, Veit HagenmeyerLLM Evaluation Cybersecurity
Institute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
ActuBench: A Multi-Agent LLM Pipeline for Generation and Evaluation of Actuarial Reasoning Tasks
Apr 22, 2026Jan-Philipp SchmidtLLM Evaluation Educational Assessment
TH Köln, Institut für Versicherungswesen (ivwKöln)
The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models
Apr 22, 2026Yilun Liu, Chunguang Zhao, Mengyao Piao +14Multilingual Language Model Evaluation LLM Evaluation
Huawei, China · Fudan University, China

Apr 25, 2026J/K move · Enter open · S save

An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code

Jelena Ilić Vulićević

Independent Researcher

LLM Evaluation On-Device Language Model Inference

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated strong performance on a wide range of software engineering tasks, including code generation and analysis. However, most prior work relies on cloud-based models or specialized hardware, limiting practical applicability in privacy-sensitive or resource-constrained environments. In this paper, we present a systematic empirical evaluation of two locally deployed LLMs, LLaMA 3.2 and Mistral, for real-world Python bug detection using the BugsInPy benchmark. We evaluate 349 bugs across 17 projects using a zero-shot prompting approach at the function level and an automated keyword-based evaluation framework. Our results show that locally executed models achieve accuracy between 43% and 45%, while producing a large proportion of partially correct responses that identify problematic code regions without pinpointing the exact fix. Performance varies significantly across projects, highlighting the importance of codebase characteristics. The results demonstrate that local models can identify a meaningful share of bugs, though precise localization remains difficult for locally executed LLMs, particularly when handling complex and context dependent bugs in realistic development scenarios.