LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

Latent Confidence Alignment for LLM Self-Assessment
Jun 20, 2026Ting-Yu Chen, Tingting Yu, Pei-Cing Huang +3LLM Evaluation Confidence Estimation in Language Models
Department of Information Management National Sun Yat-Sen University Kaohsiung, Taiwan · Kaohsiung Medical University Hospital Kaohsiung Medical University Kaohsiung, Taiwan
Fixed RAG Compression Collapses Measured Reader Scaling
Jun 20, 2026Sugam Panthi, Rabab AbdelfattahRetrieval-Augmented Generation LLM Evaluation
The University of Southern Mississippi
LLM and Human Modes of Representation
Jun 19, 2026Shalom LappinLLM Evaluation LLM Reasoning
Queen Mary University of London and King’s College London
Rubric-as-Experts: Case-Specific MQM Rubrics for Translation Quality Evaluation
Jun 19, 2026Weilu Xu, Yunzhi Shen, Xinye Wang +2LLM Evaluation Rubric-Based Evaluation
National Key Laboratory for Novel Software Technology, Nanjing University
Finetuning with Scientific Data Increases Hallucinations: A Multi-domain Factuality Evaluation of LLMs
Jun 19, 2026Raia Abu Ahmad, Nikolas Rauscher, Ekaterina Borisova +3LLM Evaluation LLM Fine-Tuning
Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKI) · Technische Universität Berlin · Humboldt-Universität zu Berlin
LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluation
Jun 19, 2026Xingyu Chen, Rui Wang, Zhaopeng Tu +1LLM Evaluation Benchmark Design
Shanghai Jiao Tong University, Tencent · Shanghai Jiao Tong University · Tencent
Coherence Under Commitment: Probing Generalization and Vacuous Memorization in LLM Logical Reasoning
Jun 19, 2026Noor Islam S. Mohammad, Mahmudul HasanLLM Evaluation Reasoning Consistency in Language Models
Department of Computer Science, Informatics Institute, Istanbul Technical University, ˙Istanbul, Türkiye · School of Information Technology, Deakin University, Geelong, VIC 3220, Australia
The Metanym Game: A Self-Contained, Self-Consistent LLM Peer-Community Benchmark for Structural Intelligence
Jun 19, 2026David NordforsLLM Evaluation LLM-as-a-Judge
Comparing Transformers and Hybrid Models at the Token Level
Jun 18, 2026Yanhong Li, William MerrillLLM Evaluation Transformer
Allen Institute for AI
PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality
Jun 18, 2026Zeyuan Chen, Ziqing Yang, Yihan Ma +2LLM Evaluation LLM Prompting
CISPA Helmholtz Center for Information Security
Can LLMs Reason About Brand Ownership? An Empirical Study of Domain Attribution Intelligence
Jun 18, 2026Fathima Mashood, Mohamed NabeelLLM Evaluation LLM Reliability
CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes
Jun 18, 2026Zhijian Zhou, Zesheng Ye, Zhaorun Chen +2LLM Evaluation Anytime-Valid Inference
The University of Melbourne · Department of Computer Science, University of Chicago · Department of Computer Science, University of Illinois Urbana-Champaign
What Shapes Emergent Misalignment? Insights from Training Dynamics, Model Priors, and Data
Jun 18, 2026Yuchen Zhang, Anietta Weckauff, Diego Garcia-Olano +1LLM Evaluation LLM Alignment
Max Planck Institute for Intelligent Systems, ELLIS Institute Tübingen, Tübingen AI Center, Tübingen, Germany · Meta.
Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software
Jun 18, 2026Arastoo Zibaeirad, Marco VieiraLLM Evaluation Software Vulnerability Detection
University of North Carolina at Charlotte, Charlotte, NC, USA
The Register Gap: A Meaning Intelligence Framework for Nigerian Public Discourse
Jun 18, 2026Celestine AchiMultilingual Language Model Evaluation LLM Evaluation
AGENTPR™ / AI-Powered PR / Cihan Digital Academy
QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation
Jun 18, 2026Xinyi Zheng, Ling Shi, Tianlong Yu +3Logical Reasoning LLM Evaluation
Huazhong University of Science and Technology Wuhan, China · Nanyang Technological University Singapore, Singapore · Hubei University Wuhan, China +2
Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact
Jun 18, 2026Jelena Meyer, David Garcia, Dirk U. WulffLLM Evaluation Personality Modeling in Language Models
Max Planck Institute for Human Development. · University of Konstanz. · Barcelona Supercomputing Center. +1
BIM-Edit: Benchmarking Large Language Models for IFC-Based Building Information Modeling
Jun 18, 2026Bharathi Kannan Nithyanantham, Clemens Kujat, Tobias Sesterhenn +5LLM Evaluation Building Information Modeling
University of Rostock · Clausthal University of Technology
A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models
Jun 18, 2026Jiayi Wang, Xu-Yao ZhangLLM Evaluation Uncertainty Quantification
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
CombEval: A Framework for Evaluating Combinatorial Counting in Large Language Models
Jun 18, 2026Yuxu Zhou, Ondřej Kuželka, Yuyi Wang +2LLM Evaluation LLM Reasoning
School of Artificial Intelligence, Jilin University, Changchun, China · Czech Technical University in Prague, Prague, Czech Republic · CRRC Zhuzhou Institute, Zhuzhou, China +3
NRITYAM: Language Models Meet Art and Heritage of Dance
Jun 18, 2026Punit Kumar Singh, Niladri Ghosh, Advait Joshiınst +3Multilingual Language Model Evaluation LLM Evaluation
Shenzhen Technology University, 518118 Shenzhen, China · Indian Institute of Technology, India · Ramakrishna Mission Vivekananda Educational and Research Institute, India +4
Closing the Operational Gap in Semantic Caching
Jun 18, 2026Aditeya Baral, Radoslav Ralev, Iliya Sotirov Zhechev +2LLM Evaluation Cost-Aware Inference
New York University · Redis
Hard or Just Unreached? Diagnosing the Sampling Blind Spot in Math-Reasoning Difficulty Estimation
Jun 17, 2026Luca Zhou, Sajel Shah, Emanuele Rodolà +1Mathematical Reasoning Benchmarks LLM Evaluation
1Sapienza University of Rome · 2Not Diamond · 3Paradigma
Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deployment
Jun 17, 2026Miloš Nikolić, Ali Hadi Zadeh, Enrique Torres Sanchez +1LLM Quantization LLM Evaluation
ByteShape · University of Toronto · Vector Institute for Artificial Intelligence
Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
Jun 17, 2026Justin D. Norman, Michael U. Rivera, D. Alex HughesInter-Rater Reliability LLM Evaluation
UC Berkeley School of Information 102 South Hall Drive, Berkeley CA, 94708
RECOM: A Validity Discrimination Tradeoff in Automatic Metrics for Open Ended Reddit Question Answering
Jun 17, 2026Pushwitha Krishnappa, Amit Das, Vinija Jain +2Open-Ended Generation LLM Evaluation
University of Alabama Huntsville · University of North Alabama · Stanford University +2
G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment
Jun 17, 2026Fengying Ye, Yanming Sun, Runzhe Zhan +3Figurative Language Understanding Multilingual Language Model Evaluation
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Faculty of Arts and Humanities, University of Macau
As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language
Jun 17, 2026Jasmine Owers, Edwin Simpson, Martha LewisFigurative Language Understanding Prompt Sensitivity
Intelligent Systems Lab University of Bristol UK · ILLC University of Amsterdam The Netherlands
RedactionBench
Jun 17, 2026Sean Brynjólfsson, Shashvat Jayakrishnan, Esha Sali +2LLM Evaluation Contextual Integrity
A10 Networks, Inc.
RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing
Jun 17, 2026Guannan Lai, Haoran Hu, Han-Jia YeLLM Evaluation Pairwise Preference Evaluation
School of Artificial Intelligence, Nanjing University, China · 2National Key Laboratory for Novel Software Technology, Nanjing University, 210023, China · 3SinapisAI

Jun 20, 2026, cs.CYJ/K move · Enter open · S save

Latent Confidence Alignment for LLM Self-Assessment

Ting-Yu Chen, Tingting Yu, Pei-Cing Huang, Chan Hsu+2

Department of Information Management National Sun Yat-Sen University Kaohsiung, Taiwan · Kaohsiung Medical University Hospital Kaohsiung Medical University Kaohsiung, Taiwan

LLM Evaluation Confidence Estimation in Language Models Language Model Self-Assessment Language Model Calibration Metacognition in Language Models

Open paper AskPDF arXiv

Confidence calibration in large language models (LLMs) is commonly evaluated by comparing predicted confidence with observed accuracy. However, such approaches do not model item difficulty, making it difficult to interpret discrepancies and to determine whether model confidence reflects genuine self-assessment or is merely a byproduct of the response generation process. To address this, we adopt a Rasch model-based latent ability framework and a metacognitive perspective, and propose Latent Confidence Alignment Error (LCAE) to measure the consistency between model self-assessment and the latent error probability implied by model ability and item difficulty. We further incorporate item difficulty as an external signal with a reasoning mechanism. Experiments on a medical-domain dataset with 20 models show that the proposed approach improves self-assessment quality without affecting model ability, and reveals an association between reliability and inference cost.

Momentum

Related topics

Latest papers 1,631