LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

Comparing Large Language Models on Scrum Certification-Style Questions: Accuracy, Stability, and Error Patterns
Jun 29, 2026Robson Alves Vilar, Emanuel Dantas Filho, Ademar França de Sousa Neto +5Software Engineering LLM Evaluation
VIRTUS Research, Development and Innovation Center, Federal University of Campina Grande, Campina Grande, PB, Brazil
RoPoLL: Robust Panel of LLM Judges
Jun 29, 2026Anish Acharya, Kris W Pan, Brian VerkhovskyLLM Evaluation LLM-as-a-Judge
Amazon Web Services
BayesBench: Evaluating LLM Belief Trajectories Under Multi-Turn Evidence Accumulation
Jun 29, 2026Ankur Samanta, Akshayaa Magesh, Tal Lancewicki +7LLM Evaluation Bayesian Inference
Meta AI · Columbia University · Work done at Meta +2
When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs
Jun 29, 2026Zhichao Yang, Caiqi Zhang, Ruihan Yang +3LLM Evaluation Confidence Estimation in Language Models
Fudan University · University of Cambridge
Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?
Jun 29, 2026Shanshan Wang, Derek F. Wong, Jingming Yao +1LLM Evaluation LLM-as-a-Judge
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Department of Portuguese, Faculty of Arts and Humanities, University of Macau
Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning
Jun 29, 2026Sangmook Lee, Minbeom Kim, Jeonghye Kim +3LLM Evaluation Reasoning Diversity
Seoul National University · KAIST
Clinical Reasoning Graphs: Structured Evaluation of LLM Diagnostic Reasoning Reveals Competence Without Consistency
Jun 29, 2026Nisarg A. PatelLLM Evaluation Reasoning Consistency in Language Models
University of California, San Francisco · Department of Oral and Maxillofacial Surgery, University of California, San Francisco, CA, USA · Department of Medicine, Division of Clinical Informatics and Digital Transformation, University of California, San Francisco, CA, USA
SrDetection: A Self-Referential Framework for Data Leakage Detection in Code Large Language Models
Jun 29, 2026Shuaimin Li, Liyang Fan, Zeyang Li +9LLM Evaluation LLM Auditing
Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Shenzhen University · Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology +5
How Far Do On-Prem Open LLMs Get on Text-to-SQL? A Cross-Family Size x Technique Frontier on BIRD
Jun 29, 2026Vladimir BeskorovainyiLLM Self-Correction LLM Evaluation
Besk Tech · Moscow Institute of Physics and Technology (MIPT)
A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents
Jun 29, 2026Liu ZewenLLM Evaluation LLM Agent Evaluation
Qilu Institute of Technology, School of Software Engineering · Tai’an, Shandong, China
SFBench: The SciFy Scientific Feasibility Benchmark
Jun 28, 2026Cash Costello, James Mayfield, Elsbeth Turcan +7LLM Evaluation Benchmark Design
Johns Hopkins Applied Physics Laboratory
Do We Still Need Fine Tuning? Turkish Sentiment Analysis in the Era of Large Language Model
Jun 28, 2026Sercan Karakaş, Yusuf ŞimşekSupervised Fine-Tuning LLM Evaluation
University of Chicago · Fırat University
SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models
Jun 28, 2026Tiziano Santilli, Francesco Daghero, Mayhar Tourchi MoghaddamSoftware Engineering LLM Evaluation
University of Southern Denmark, Odense 5300, DK
Cognitive World Models for Process-Level Social Influence Evaluation
Jun 28, 2026Minghui Ma, Bin Guo, Han Wang +4LLM Evaluation Cognitive Modeling
Northwestern Polytechnical University, Xi’an, China · Harbin Engineering University, Harbin, China
The Complexity Ceiling Benchmark: A Multi-Domain Evaluation of Sequential Reasoning Under Depth Scaling
Jun 28, 2026Shubh Chapra, Dhruv Kumar, Murari Mandal +1LLM Evaluation Structured Reasoning
BITS Pilani, Pilani Campus · KIIT Bhubaneswar
Understanding Evaluation Illusion in Diffusion Large Language Models
Jun 28, 2026Hengxiang Zhang, Jiaxi Ren, Renchunzi Xie +1Prompt Sensitivity LLM Evaluation
Department of Statistics and Data Science, Southern University of Science and Technology · Intelligent Computing Research Center, Great Bay Institute for Advanced Study, Great Bay University
Aristotelian Virtue Profiling of LLMs through Ethical Dilemmas
Jun 27, 2026Ioannis Tzachristas, John PavlopoulosLLM Evaluation Moral Reasoning in Language Models
1Technical University of Munich, Germany · 2Athens University of Economics and Business, Greece · 3Archimedes, Athena Research Center, Greece
When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling
Jun 27, 2026Yong Yi Bay, Kathleen A. YearickLLM Evaluation Self-Consistency Decoding
University of Illinois at Urbana-Champaign
What LLMs explain is not what they believe: Evaluating explanation sufficiency under models' own input beliefs
Jun 26, 2026Nhi Nguyen, Shauli Ravfogel, Rajesh RanganathLLM Evaluation LLM Interpretability
New York University
Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs
Jun 26, 2026Manuel PitaLLM Evaluation Psychometric Validation
Artificial Intelligence, Social Interaction and Complexity Laboratory CICANT, Universidade Lusófona 376, Campo Grande, 1700-097 Lisbon, Portugal
Developmental Trajectories of Situation Modeling and Mentalizing in Transformer Language Models
Jun 26, 2026Pamela D. Rivière, Cameron Jones, Sean TrottLLM Evaluation Theory of Mind
Rutgers University - Newark · Stony Brook University
Data and Evaluation Closed-Loop for Model Capability Enhancement
Jun 26, 2026Zhixuan Li, Jiangan Yuan, Han XuLLM Evaluation LLM Post-Training
Baidu Inc.
Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA
Jun 26, 2026Sambaran BandyopadhyayIn-Context Retrieval LLM Evaluation
Adobe Research
An Empirical Analysis of Factual Errors in Human-Written Text and Its Application to Factual Error Detection
Jun 26, 2026Kazuma Iwamoto, Kazumasa Omura, Shotaro IshiharaLLM Evaluation Automated Fact-Checking
Nikkei Inc.
When are likely answers right? On Sequence Probability and Correctness in LLMs
Jun 25, 2026Johannes Zenn, Jonas GeipingLLM Evaluation Language Model Decoding
Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen · AI Center Tübingen +2
When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models
Jun 25, 2026Josef ChenLLM Evaluation LLM Routing
KAIKAKU
How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation
Jun 25, 2026Maria LevchenkoHistorical Document Analysis LLM Evaluation
University of Bologna
RSPC: A Benchmark for Modeling Stress and Psychiatric Conditions in Digitally Mediated Relationships using Psychiatrist Annotations
Jun 25, 2026Parmitha Vangapandu, Sai Ganesh Mokkapati, Sathwik Narkedimilli +4Social Media Analysis LLM Evaluation
Indian Institute of Information Technology Dharwad · Keshav Memorial College of Engineering · National University of Singapore +4
Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
Jun 25, 2026Sangwoo Cho, Kushal Chawla, Pengshan Cai +4LLM Evaluation LLM-as-a-Judge
Capital One, AI Foundations, McLean, VA 22102, USA.
The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans
Jun 25, 2026Bella Fascendini, Kathryn McGregor, Max D. Gupta +1LLM Evaluation LLM Reasoning
Department of Psychology, Princeton University · Department of Computer Science, Princeton University

Jun 29, 2026, cs.SEJ/K move · Enter open · S save

Comparing Large Language Models on Scrum Certification-Style Questions: Accuracy, Stability, and Error Patterns

Robson Alves Vilar, Emanuel Dantas Filho, Ademar França de Sousa Neto, Mirko Perkusich+4

VIRTUS Research, Development and Innovation Center, Federal University of Campina Grande, Campina Grande, PB, Brazil

Software Engineering LLM Evaluation Question Answering LLM Reliability

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly used in exam- and certification-style question answering tasks, where their ability to retrieve, interpret, and apply domain-specific knowledge can be systematically assessed. In Software Engineering, such settings are particularly relevant when questions depend on strict adherence to normative definitions, roles, artifacts, and rules. This paper evaluates the performance of three contemporary LLMs, \textit{GPT-5 mini}, \textit{Gemini 3 Flash}, and \textit{DeepSeek Chat 3.2}, in answering 993 Scrum certification-style questions aligned with the Professional Scrum Master I (PSM I) assessment format. We evaluated the models under three prompting strategies (\textit{zero-shot}, \textit{chain-of-thought}, and \textit{source-grounded}), with repeated executions to assess intra-model stability. We also analyzed performance across Scrum topics and question formats, complemented by a qualitative analysis of recurring error patterns in incorrect answers. Results revealed clear differences among models, with Gemini 3 Flash achieving the highest accuracy, followed by GPT-5 mini and DeepSeek Chat 3.2, while intra-model variability remained low across all conditions. By question format, the models achieved the highest accuracy on single-answer multiple-choice items, whereas multi-select and True/False questions were more error-prone. By topic, performance was more consistent in normatively explicit areas such as Artifacts, Empiricism, and Product Value, but more fragile in Scrum Values, Self-Managing Teams, and Stakeholders & Customers. The qualitative analysis showed that errors were systematic rather than random, involving overgeneralization, restrictive wording, compound distractors, and conflicts between common market interpretations and strict Scrum definitions.

Momentum

Related topics

Latest papers 1,643