LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

Can LLMs Value the Right Evidence? Evidence-Value Misalignment in Dynamic Medical Diagnosis
Sep 28, 2026Kehua Feng, Yunsheng Lu, Yitong Qiao +5LLM Evaluation Evidence-Grounded Reasoning
Zhejiang University · Ant Healthcare, Ant Group
QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks
Sep 28, 2026Pranav GuptaLLM Evaluation Quantum Circuit Simulation
Cisco, 2901 3rd Ave, Suite 600, Seattle, WA 98121, USA
AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabic
Sep 28, 2026Ignacio Iacobacci, Faroq Altam, Zhaozhi Qian +1LLM Evaluation Arabic NLP
Elm Company
AwarenessBench: Assessing Cognitive Capabilities of Language Models
Sep 28, 2026Xiaojian Li, Rongwu Xu, Tianyun Zhang +9LLM Evaluation Evaluation Awareness in Language Models
Tsinghua University · Shanghai Qi Zhi Institute · Fangcun AI +5
How Well Can LLMs Simulate Real Learner Evaluations of Educational Feedback?
Sep 28, 2026Momoka Furuhashi, Kouta Nakayama, Takashi Kodama +2LLM Evaluation Large Language Model-Based Social Simulation
Tohoku University · Research and Development Center for Large Language Models, National Institute of Informatics · National Institute of Informatics +2
Large Language Models for Automated Cross-Domain Machine Learning Task Type Identification: A Benchmark Dataset and Evaluation
Sep 28, 2026Petros Tsialis, Steffen Limmer, Tobias Rodemann +1LLM Evaluation Tabular ML
University of Applied Sciences Aalen · Honda Research Institute Europe
MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs
Sep 28, 2026Zineddine Tighidet, Andrea Mogini, Jiali Mei +2LLM Evaluation Memorization in Language Models
BNP Paribas Paris, France · Sorbonne Université Paris, France · Criteo AI Lab
Measuring Collapse and Correction in Homogeneous-Panel LLM Debate
Sep 28, 2026Xin Li, Mengbing Liu, Chau YuenLLM Evaluation LLM Auditing
Nanyang Technological University
VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation
Sep 28, 2026Hanxun Huang, Yutao Wu, Qizhou Wang +9LLM Evaluation Fake News Detection
The University of Melbourne · Deakin University · Singapore Management University +3
Pass or Fail? Evaluating LLMs on Two Greek Examination Benchmarks
Sep 28, 2026Panagiota Kyriazi, Eleni Kasoura, Prokopis ProkopidisMultilingual Language Model Evaluation Prompt Sensitivity
Institute for Language and Speech Processing / Athena RC
TQTS-Bench: A Multi-Syntax Benchmark for Text-to-Query over Time-Series Databases
Sep 28, 2026Fei Lyu, Zhiyi Peng, Jiaming Liu +5LLM Evaluation Time Series QA
College of Computer Science and Electronic Engineering, Hunan University
Using LLMs to Detect LLM-Generated Texts: A Cross-Generation Analysis
Sep 28, 2026Haiyue Yuan, Jie Guo, Weidong Qiu +3LLM Evaluation AI-Generated Text Detection
Institute of Cyber Security for Society (iCSS) & School of Computing, University of Kent, UK · School of Cyber Science and Engineering, Shanghai Jiao Tong University, China · School of Computer Science, University of Birmingham, UK
SleuthBench: Benchmarking Statistical LLM Evaluation Using Tabular Hidden Signals
Sep 28, 2026Jingyun Jia, Antoine Remond-Tiedrez, Aaron Alvarez +3LLM Evaluation AI Agent Benchmarks
University of Wisconsin–Madison, Intelligible · Intelligible · University of Cincinnati
Do System One Decisions Add Up? A Study of Probabilistic Coherence
Sep 27, 2026Saman Sarker JoyLLM Evaluation Classification
Faculty of Computer Science and Information Technology Universiti Malaya Kuala Lumpur, Malaysia
SlopBench: How Well Can We Rank Language Models by Slop? A Multi-Domain Benchmark of Repetitive AI Writing
Sep 27, 2026Dhruv Roongta, Harsha Gaddipati, Anh Tuan HuynhLLM Evaluation Language Model Generation Evaluation
Slashy, San Francisco, California, USA
Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation
Sep 27, 2026Gowthamkumar NandakishoreLLM Evaluation Selective Prediction
The Effects of Incremental Instruction Delivery on Language-Model Creative Writing
Sep 27, 2026Anshuman Singh, Abrar Eyasir, Haseeb Yaqoob +1LLM Evaluation LLM Prompting
SGT UNIVERSITY · CSE, University of Dhaka · NED University of Engineering and Technology +1
From Granular Revision Operations to Meaningful Revision Units: Evaluating LLMs for Revision Boundary Detection
Sep 27, 2026Yu Tian, Andrew Potter, Katerina Christhilf +4LLM Evaluation
Learning Engineering Institute, Arizona State University, Tempe, Arizona, USA · The Polytechnic School, Arizona State University, Tempe, Arizona, USA · Department of English, Arizona State University, Tempe, Arizona, USA +1
SpecRead: A Benchmark for Measuring Whether Language Models Understand Hardware Specifications
Sep 27, 2026Feilian HuangDocument Understanding LLM Evaluation
Independent Researcher
One Model Is Not a Crowd: Multi-LLM and Aspect-Conditioned Diverse Comment Generation
Sep 27, 2026Nafis Irtiza Tripto, Delvin Ce Zhang, Mahjabin Nahar +1LLM Evaluation Controllable Text Generation
College of Information Sciences and Technology, Pennsylvania State University, PA, USA · University of Sheffield, Sheffield, UK
E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding
Sep 27, 2026Khloud AL Jallad, Nada Ghneim, Ghaida RebdawiLLM Evaluation Arabic NLP
Informatics Department, Higher Institute for Applied Sciences and Technology, Damascus, Syria. · Faculty of Informatics and Communication, Arab International University, Daraa, Syria.
LLM4Trust: Exploring the Capabilities of Large Language Models for Trust Evaluation
Sep 27, 2026Jie Wang, Yanbo Sun, Zheng Yan +2LLM Evaluation Trust in AI
State Key Laboratory of Integrated Services Networks, School of Cyber Engineering, Xidian University · Hangzhou Institute of Technology, Xidian University · Department of Computer Science, Purdue University
MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses
Sep 27, 2026Xuanjun Chen, Hua-Hsuan Chen, Wei-Chung Lu +3LLM Evaluation Benchmark Design
National Taiwan University · NTU AI-Core · Queen Mary University of London
Knowing Is Not Choosing: What Explicit Verification Adds Beyond Generative Preference
Sep 27, 2026Yilong Li, Chengpo Yan, Aayan Arish +1LLM Evaluation LLM Answer Verification
Department of Computer Science University of Wisconsin-Madison Madison, WI 53703, USA
Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge Leaderboards
Sep 27, 2026Krishna Chytanya AyyagariLLM Evaluation LLM-as-a-Judge
Google
LLM Judge Validation Under Sparse Overlap: From Inference to Design
Sep 25, 2026Junxuan Li, Arko Mukherjee, Soumyabrata PalInter-Rater Reliability LLM Evaluation
Adobe · Adobe Research
How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
Sep 24, 2026Dipankar SarkarLLM Evaluation LLM Auditing
Skelf Research
Scoring Both Directions: LLMs realize the MRS they cannot reliably parse
Sep 24, 2026Soham DanLLM Evaluation Semantic Parsing
Scale AI
Style, Not Self: Surface Cues Explain Zero-Shot Code Attribution by Large Language Models
Sep 24, 2026Ehsan Barkhordar, Surendrabikram ThapaLLM Evaluation Language Model Generation Evaluation
Koç University · Virginia Tech
Augur: A Synthetic Decision Lab for Rehearsing Reactions to Product and Policy Changes
Sep 24, 2026Rahul Khedar, Mayank Malhotra, Avinash KarnLLM Evaluation LLM Decision-Making
PayPal AI

Sep 28, 2026, cs.CLJ/K move · Enter open · S save

Can LLMs Value the Right Evidence? Evidence-Value Misalignment in Dynamic Medical Diagnosis

Kehua Feng, Yunsheng Lu, Yitong Qiao, Tiantian He+4

Zhejiang University · Ant Healthcare, Ant Group

LLM Evaluation Evidence-Grounded Reasoning Medical Diagnosis Clinical Decision-Making

Open paper AskPDF arXiv

A correct diagnosis reached from insufficient or misleading evidence can pose a clinical hazard, yet outcome-based accuracy may reward such lucky guesses. We call this mismatch between diagnostic decisions and the value of available evidence Evidence-Value Misalignment (EVM). To disentangle evidential grounding independently from diagnostic accuracy, we introduce MedEVM, a dynamic benchmarking environment comprising 1,050 cases across 24 disease systems. Observations arrive turn by turn, requiring models to continuously calibrate its decision by deciding whether to wait for more evidence or submit a diagnosis. Across 9 LLMs, four interesting patterns are observed. (1) Miscalibrated evidence tracking. Making a diagnosis often fails to calibrate evidence sufficiency, even in more capable models, and even worsens in reasoning mode. (2) Misaligned diagnosis submission. Confidence in the correct diagnosis often fails to ensure timely submission despite sufficient evidence. (3) Evidence order matters. Reordering the same evidence changes diagnoses even when model confidence remains similar. (4) Misleading evidence remains influential. Added misleading evidence redirects diagnoses even after prior evidence becomes sufficient. We further verify that EVM predicts errors and that preventing premature submission improves accuracy. These findings motivate Evidence-Verified Diagnosis Harness (EVD-Harness). It decouples diagnosis generation from submission through an offline Contrastive Diagnostic Wiki and three online control stages, namely observation management, proposal and witness verification, and diagnosis submission control. Across five LLMs, EVD-Harness improves accuracy by 12.0--51.1 percentage points while mitigating EVM-related failures. Our results demonstrate that verifying evidential support before submission can make diagnostic decisions more reliable.

Momentum

Related topics

Latest papers 1,631