LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

ReasonLab: A Controlled and Auditable Evaluation of Prompting Techniques for Multiple-Choice QA
May 7, 2026Inder Preet, Shuxin Lin, Dhaval PatelLLM Evaluation Language Model Generation Evaluation
IBM, Dublin · IBM, New York
MultiSoc-4D: A Benchmark for Diagnosing Instruction-Induced Label Collapse in Closed-Set LLM Annotation of Bengali Social Media
May 7, 2026Souvik Pramanik, S. M. Riaz Rahman Antu, Shak Mohammad Abyad +2LLM Evaluation LLM-Assisted Annotation
North South University, Dhaka, Bangladesh
Bias and Uncertainty in LLM-as-a-Judge Estimation
May 7, 2026James FiedlerLLM Evaluation LLM-as-a-Judge
Indeed Inc.
Can LLMs Take Retrieved Information with a Grain of Salt?
May 7, 2026Behzad Shayegh, Mohamed Osama Ahmed, Fred Tung +1Retrieval-Augmented Generation LLM Evaluation
RBC Borealis
How Well Do LLMs Perform on the Simplest Long-Chain Reasoning Tasks: An Empirical Study on the Equivalence Class Problem
May 7, 2026Chun Zheng, Lianlong Wu, Bingqian Li +2LLM Evaluation LLM Reasoning
University of Science and Technology of China · University of Oxford
Examining Human-Like Behaviors in LLMs: A Multi-Dimensional Analysis of Model Behaviors, User Factors, and System Prompts
May 7, 2026Sunnie S. Y. Kim, Margit Bowler, Leon A GatysLLM Evaluation LLM Prompting
Apple
IntentGrasp: A Comprehensive Benchmark for Intent Understanding
May 7, 2026Yuwei Yin, Chuyuan Li, Giuseppe CareniniLLM Evaluation Intent Classification
Department of Computer Science, University of British Columbia
Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Supervised ML
May 7, 2026Jai Moondra, Ayela Chughtai, Bhargavi Lanka +1Multilingual Language Model Evaluation LLM Evaluation
Carnegie Mellon University · MIT Sloan School of Management
When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels
May 7, 2026Sushant Gautam, Finn Schwall, Annika Willoch Olstad +6LLM Safety Benchmarks LLM Evaluation
1Simula Metropolitan Center for Digital Engineering · 2Oslo Metropolitan University · 4Simula Research Laboratory +2
Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents
May 7, 2026Hailey Onweller, Elias Lumer, Austin Huber +3LLM Evaluation Citation Verification
Commercial Technology and Innovation Office, PricewaterhouseCoopers, U.S.
How Many Iterations to Jailbreak? Dynamic Budget Allocation for Multi-Turn LLM Evaluation
May 7, 2026Shai Feldman, Yaniv RomanoLLM Evaluation Language Model Safety Evaluation
Department of Computer Science Technion, Israel · Departments of Electrical & Computer Engineering. and Computer Science Technion, Israel
Ex Ante Evaluation of AI-Induced Idea Diversity Collapse
May 7, 2026Nafis Saami Azad, Raiyan Abdul BatenLLM Evaluation Computational Creativity
Bellini College of Artificial Intelligence, Cybersecurity, and Computing University of South Florida Tampa, FL–33620, USA
Towards Emotion Consistency Analysis of Large Language Models in Emotional Conversational Contexts
May 7, 2026Sneha Oram, Ojaswita Bhushan, Pushpak BhattacharyyaLLM Evaluation Language Model Robustness
Indian Institute of Technology Bombay, India
SCRuB: Social Concept Reasoning under Rubric-Based Evaluation
May 7, 2026Jamelle Watson-Daniels, Himaghna Bhattacharjee, Skyler Wang +11LLM Evaluation Rubric-Based Evaluation
1Meta · 2McGill University · 3Handshake AI +1
Rethinking Vacuity for OOD Detection in Evidential Deep Learning
May 7, 2026Claire McNamaraLLM Evaluation Evidential Deep Learning
∗Accenture Labs · †Trinity College Dublin.
Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity
May 7, 2026Florian A. D. Burnat, Brittany I. DavidsonLLM Safety Benchmarks LLM Evaluation
University of Bath
Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models
May 7, 2026Haoxiang Wang, Da Yu, Huishuai ZhangLLM Evaluation Language Model Generation Evaluation
Peking University · Google Research · Wangxuan Institute of Computer Technology, Peking University State Key Laboratory of General Artificial Intelligence
Teaching LLMs Program Semantics via Symbolic Execution Traces
May 7, 2026Jonas Bayer, Stefan Zetzsche, Olivier Bouissou +3LLM Evaluation Continued Pretraining
University of Cambridge · Amazon Web Services
Visual Fingerprints for LLM Generation Comparison
May 7, 2026Amal Alnouri, Andreas Hinterreiter, Christina Humer +2LLM Evaluation Language Model Fingerprinting
Johannes Kepler University Linz · ETH Zürich · Independent Researcher
More Aligned, Less Diverse? Analyzing the Grammar and Lexicon of Two Generations of LLMs
May 7, 2026Adrián Gude, Roi Santos-Ríos, Francis Bond +3LLM Evaluation
Universidade da Coruña, CITIC · Palacký University, Olomouc · Independent Researcher
Strat-LLM: Stratified Strategy Alignment for LLM-based Stock Trading with Real-time Multi-Source Signals
May 7, 2026Wenliang Huang, Zengyi YuLLM Evaluation LLM Alignment
School of Economics Zhejiang University of Technology Hangzhou, China · Faculty of Education East China Normal University Shanghai, China
Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking
May 7, 2026Yang Xu, Jiefu Zhang, Haixiang Sun +3LLM Evaluation Selective Inference
Purdue University · Johns Hopkins University · Purdue University, USA
Logic-Regularized Verifier Elicits Reasoning from LLMs
May 7, 2026Xinyu Wang, Changzhi Sun, Lian Cheng +4LLM Evaluation LLM Answer Verification
Department of Computer Science and Technology, East China Normal University · Institute of Artificial Intelligence (TeleAI), China Telecom
CITE: Anytime-Valid Statistical Inference in LLM Self-Consistency
May 7, 2026Hirofumi Ota, Naoto Iwase, Yuki Ichihara +2LLM Evaluation Anytime-Valid Inference
arXiv:2605.05873v1
Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs
May 7, 2026Fahd Seddik, Fatemeh FardLLM Evaluation LLM Interpretability
Department of Computer Science University of British Columbia Kelowna, BC, Canada
I'm Sorry, but I Can't Help with Braille: Revealing Accessibility Failures in State-of-the-Art LLMs
May 7, 2026Abdullah AbdullahLLM Evaluation LLM Fine-Tuning
orinu Inc. Hwaseong, Republic of Korea
MRI-Eval: A Tiered Benchmark for Evaluating LLM Performance on MRI Physics and GE Scanner Operations Knowledge
May 6, 2026Perry E. RadauLLM Evaluation Healthcare
Department of Radiology, University of Calgary · Child and Adolescent Imaging Research (CAIR) Program · Alberta Children’s Hospital Research Institute +1
The Pinocchio Dimension: Phenomenality of Experience as the Primary Axis of LLM Psychometric Differences
May 6, 2026Hubert Plisiecki, Sabina Siudaj, Kacper Dudzic +4LLM Evaluation Personality Modeling in Language Models
IDEAS Research Institute · University of Warsaw
Curated AI beats frontier LLMs at pharma asset discovery
May 6, 2026Łukasz Kidziński, Kevin ThomasLLM Evaluation Healthcare
Gosset Research
BenCSSmark: Making the Social Sciences Count in LLM Research
May 6, 2026Arnault Chatelain, Étienne Ollion, Qianwen Guan +7LLM Evaluation Benchmark Design
CREST (École Polytechnique, ENSAE, CNRS), 5 avenue Le Chatelier, 91120 Palaiseau, France · LLF (Université Paris Cité and CNRS), UFRL Olympe de Gouges, 13 place Paul Ricoeur, 75013 Paris, France · Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France +1

May 7, 2026, cs.CLJ/K move · Enter open · S save

ReasonLab: A Controlled and Auditable Evaluation of Prompting Techniques for Multiple-Choice QA

Inder Preet, Shuxin Lin, Dhaval Patel

IBM, Dublin · IBM, New York

LLM Evaluation Language Model Generation Evaluation Multiple-Choice Question Answering LLM Prompting Large Reasoning Models

Open paper AskPDF arXiv

Probing the capabilities of Large Language Models (LLMs) and building robust solutions for Multiple-Choice Question Answering (MCQA) remain central challenges in natural language understanding. Furthermore, the rapid proliferation of LLMs has created the implicit assumption that more sophisticated prompting techniques yield better performance. Several studies claim such gains, but report them under differing models, prompt wordings and answer-extraction rules, so the gains cannot be attributed to the technique alone. We address this gap with ReasonLab, an evaluation framework in which the prompting technique is a first-class experimental variable alongside the model and the dataset, and which retains every generation for inspection. Using ReasonLab we conduct a controlled study of 8 prompting techniques across 10 MCQA datasets, 27 model configurations and 480,927 evaluations at temperature 0. We find that the prompting technique is a minor determinant of accuracy: on configurations without a reasoning budget the reasoning triggers improve on direct prompting by only 3.92 to 4.69 pp and are indistinguishable from one another, and on configurations with reasoning enabled no technique differs by more than 0.51 pp. Self-Generate is the only technique with a consistent effect, a reduction of 2.95 pp. We further investigate three phenomena: (1) the comparison of models on a common set of datasets, where model size does not predict accuracy, (2) the trade-offs across thinking budgets, where enabling reasoning is worth up to 12.74 pp whereas an eightfold budget increase adds only 0.48 to 2.10 pp, and (3) the variation in dataset difficulty, with 60% of benchmarks below 70% accuracy and a 43.9 pp spread from easiest to hardest. These results suggest that, for MCQA, the prompting technique is a minor lever compared with enabling model reasoning, and that substantial headroom remains.

Momentum

Related topics

Latest papers 1,643