LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,446 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

Evaluating Prompting and Execution-Based Methods for Deterministic Computation in LLMs
May 4, 2026Hongkun YuLLM Evaluation LLM Prompting
Virginia Tech Blacksburg, VA, USA
Benchmarking Local Language Models for Social Robots using Edge Devices
May 4, 2026Dorian Lamouille, Matevž B. Zorec, Farnaz Baksh +1LLM Evaluation On-Device Language Model Inference
University of Tartu Tartu, Estonia · Institute of Computer Science University of Tartu Tartu, Estonia · Institute of Technology University of Tartu Tartu, Estonia
Evaluating Reasoning Models for Queries with Presuppositions
May 4, 2026Rose Sathyanathan, Kinshuk Vasisht, Danish PruthiLLM Evaluation Language Model Robustness
Indian Institute of Science Bengaluru, KA, India
SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures
May 4, 2026Nedjma Ousidhoum, Junho Myung, Carla Perez-Almendros +27Multilingual Language Model Evaluation LLM Evaluation
Cardiff University · KAIST · MBZUAI +13
HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs
May 4, 2026Ahmed CherifLLM Evaluation LLM Hallucination Mitigation
1Sofrecom Tunisia, Orange Innovation, Tunis, 1053, Tunisia.
When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models
May 4, 2026Cosimo Galeone, Minsu Park, Giuseppe Ettorre +1LLM Evaluation LLM Input Optimization
Alomana, Grottaglie, Italy
Complexity Horizons of Compressed Models in Analog Circuit Analysis
May 4, 2026Pacome Simon MbonimpaLLM Evaluation LLM Compression
Carnegie Mellon University-Africa
Submodular Benchmark Selection
May 4, 2026Alexander SmolaSubmodular Optimization LLM Evaluation
Boson AI 4677 Old Ironsides Dr Santa Clara CA 95054
Retrieval and Multi-Hop Reasoning in 1M-Token Context Windows: Evaluating LLMs on Classical Chinese Text
May 4, 2026Eric H. C. ChowLong-Context Retrieval In-Context Retrieval
School of Humanities, The University of Hong Kong
CLaC at SemEval-2026 Task 6: Response Clarity Detection in Political Discourse
May 4, 2026Nawar Turk, Lucas Miquet-Westphal, Leila KosseimLLM Evaluation Language Model Ensembles
Computational Linguistics at Concordia (CLaC) Lab · Dept. of Computer Science and Software Engineering · Concordia University, Montréal, Québec, Canada
So Many Opinions, So Many LLMs: Comparing Large Language Models to Traditional Machine Learning for Open- Ended Survey Analysis
May 4, 2026Abdullah Akinde, Mariam Akinde, Rasheedat Emiola +1LLM Evaluation LLM Interpretability
Austin Peay State University, United States
EditPropBench: Measuring Factual Edit Propagation in Scientific Manuscripts
May 3, 2026Garvin KruthofKnowledge Editing LLM Evaluation
Technical University of Munich
What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models
May 3, 2026Ranit Karmakar, Jayita ChatterjeeLLM Evaluation Language Model Calibration
Harvard University, Boston, MA 02115 · Boston, MA 02134
Conventional Commit Classification using Large Language Models and Prompt Engineering
May 3, 2026H. M. Sazzad Quadir, Sakib Al Hasan, Md. Nurul Ahad TawhidSoftware Engineering LLM Evaluation
Institute of Information Technology University of Dhaka Dhaka, Bangladesh
Language models fail at extended rule following
May 3, 2026Tianxiang Dai, Jonathan FanLLM Evaluation LLM Reliability
Department of Electrical Engineering, Stanford University, Stanford, CA 94305.
StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models
May 3, 2026Yongrui Chen, Yangyang Ma, Xiaoying Huang +4LLM Evaluation Benchmark Design
Southeast University · Hangzhou Dianzi University · Zhejiang University +1
Spatiotemporal Hidden-State Dynamics as a Signature of Internal Reasoning in Large Language Models
May 3, 2026Kotaro Furuya, Takahito TanimuraTransformer Interpretability LLM Evaluation
Research and Development Group, Hitachi, Ltd. 1-280 Higashi-Koigakubo, Kokubunji-shi, Tokyo 185-8601, Japan
Prompt Framing Distorts Count-Based Evaluation of LLM Error Detection: Evidence from Numeric Anchoring
May 3, 2026Dekun YangPrompt Sensitivity LLM Evaluation
Zhejiang University
RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
May 3, 2026Yangyang Zhou, Yi-Chen LiReward Modeling LLM Evaluation
Beijing University of Posts and Telecommunications · National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Molecular Representations for Large Language Models
May 3, 2026Nicholas T. Runcie, Fergus Imrie, Charlotte M. DeaneLLM Evaluation Large Language Model-Guided Scientific Discovery
Department of Statistics University of Oxford Oxford, UK
Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking
May 2, 2026Deep Pankajbhai MehtaPrompt Sensitivity LLM Evaluation
Adobe Inc.
Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
May 2, 2026Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6LLM Evaluation LLM-as-a-Judge
Maritaca AI · Jusbrasil
Evaluating LLMs on Large-Scale Graph Property Estimation via Random Walks
May 2, 2026Sunil Kumar Maurya, Xin LiuLLM Evaluation LLM Reasoning with Graphs
University of Tokyo Tokyo, Japan · AIST Tokyo, Japan
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks
May 2, 2026Benjamin Warner, Ratna Sagari Grandhi, Max Kieffer +32LLM Evaluation Healthcare
Sophont · MedARC · Prime Intellect
Structural Ranking of the Cognitive Plausibility of Computational Models of Analogy and Metaphors with the Minimal Cognitive Grid
May 2, 2026Alessio Donvito, Antonio LietoLLM Evaluation Computational Cognitive Modeling
University of Bari, Italy · University of Salerno, CIIT Lab / ICAR-CNR, Italy
On Stable Long-Form Generation: Benchmarking and Mitigating Length Volatility
May 2, 2026Zhitao He, Haolin Yang, Rui Min +2LLM Evaluation Long-Form Document Generation
Hong Kong University of Science and Technology
The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choice
May 2, 2026Jikai Jin, Vasilis SyrgkanisLLM Evaluation Off-Policy Evaluation
Psychologically Potent, Computationally Invisible: LLMs Generate Social-Comparison-Eliciting Posts They Fail to Detect
May 1, 2026Hua Zhao, Jiapei Gu, Michelle Mingyue GuSocial Media Analysis LLM Evaluation
Department of English Language Education, The Education University of Hong Kong · Analytics/Assessment Research Centre, The Education University of Hong Kong
Can AI Debias the News? LLM Interventions Improve Cross-Partisan Receptivity but LLMs Overestimate Their Own Effectiveness
May 1, 2026Faisal Feroz, Jonas R. KunstLLM Evaluation Human Behavior Simulation
Department of Experimental Psychology, University of Oxford · Uehiro Oxford Institute, University of Oxford · Psychology Programme, School of Social Sciences, Nanyang Technological University +1
When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models
May 1, 2026Sailesh Panda, Pritam Kadasi, Abhishek Upperwal +1LLM Evaluation Instruction Following
Indian Institute of Technology Gandhinagar · Soket AI

May 4, 2026J/K move · Enter open · S save

Evaluating Prompting and Execution-Based Methods for Deterministic Computation in LLMs

Hongkun Yu

Virginia Tech Blacksburg, VA, USA

LLM Evaluation LLM Prompting LLM Code Generation Prompt Engineering

Open paper AskPDF arXiv

Large Language Models (LLMs) have demonstrated strong capabilities in natural language understanding and reasoning. However, their ability to perform exact, deterministic computation remains unclear. In this work, we systematically evaluate multiple prompting strategies, including Chain-of-Thought (CoT), Least-to-Most decomposition, Program-of-Thought (PoT), and Self-Consistency (SC), on tasks requiring precise and error-free outputs, including binary counting, longest substring detection, and arithmetic evaluation. To support this study, we introduce a synthetic dataset with diverse natural language instructions, enabling controlled evaluation of exact computation across multiple task types. Our results show that standard prompting methods achieve only moderate accuracy on sequence-based tasks. CoT provides limited improvement, while Least-to-Most suffers from error accumulation. In contrast, PoT achieves perfect accuracy by generating executable code and delegating computation to an external interpreter. Self-Consistency improves robustness through majority voting, but incurs substantial computational overhead. We further train a small domain-specific model (CodeT5-small) to generate executable programs, which achieves perfect accuracy on held-out synthetic test data across all tasks with minimal training cost. Overall, our findings suggest that LLMs may simulate reasoning patterns rather than reliably perform exact symbolic computation. For deterministic tasks, combining LLMs with external tools or using specialized models provides a more reliable and efficient solution.