Language Model Self-Assessment

Ask about this topic

Overview

Language model self-assessment asks whether a model can judge its own answer quality, uncertainty, capabilities, errors, and behavior—and use those judgments to decide when to answer, abstain, verify, or revise. Methods range from verbal confidence, token probabilities, and agreement across samples to hidden-state probes, structured self-questioning, and training with reinforcement learning or self-generated rubrics. Work evaluates these signals on factual and specialized question answering, math and code reasoning, safety tasks, and agent interactions, measuring calibration, selective risk, and whether reflection improves outcomes. Self-assessment can support routing and refinement, but may be sensitive to prompts, miss shared errors, and fail to track objective correctness or actual behavior, making external or environment-grounded checks important.

Latest papers

CardsList

93 papers13 papers in the last four weeks, up 8% on the four weeks before. 0.1% of all new papers.

  1. LADDER: Diagnosing Domain-Level Self-Assessment Under Noisy Feedback

    Oct 26, 2026Zhenshuai Yin, Jianshu Liang, Xuyue Xiong +6Language Model Self-AssessmentMetacognition in Language Models

    School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen), Longgang, Shenzhen, Guangdong, 518172, P.R. China · ByteDance · Shenzhen Research Institute of Big Data, Shenzhen, China

  2. Disentangling Question, Response, and Verifier Languages in Multilingual P(True) Self-Evaluation

    Oct 23, 2026Jewon Yeom, Jaewon Sok, Seonghyeon Park +3Language Model Self-AssessmentMultilingual Language Model Evaluation

    Graduate School of Data Science, Seoul National University

  3. Which Answer Did the Seventeen-Year-Old Write? A Behavioral Case Study of Author-Age Misattribution, Miscalibrated Confidence, and Non-Corrective Self-Explanation in Six Frontier Language Models

    Oct 22, 2026Brianne LeeDemographic Bias in Language ModelsLanguage Model Self-Assessment

    Independent Researcher

  4. Self-Assessment Reinforcement Learning for Search-State Awareness in Retrieval-Augmented Large Language Models

    Oct 21, 2026Taewhoo Lee, Minju Song, Jungwoo Park +6RL for RAGRetrieval-Augmented Reasoning

    Korea University · Amazon · AIGEN Sciences +2

  5. Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges

    Oct 18, 2026Saeid Asgari, Rui Ying, Sara Malvar +5Rubric-Based EvaluationLLM-as-a-Judge

  6. Structural and Representational Agency in LLM-Driven Timbre Generation and Interpolation

    Oct 8, 2026Joseph Cameron, Alan BlackwellAudio GenerationLanguage Model Self-Assessment

    University of Cambridge Cambridge, United Kingdom

  7. Identifying Introspection From the Inside

    Oct 5, 2026David I. Atkinson, Dillon Plunkett, David BauLLM InterpretabilityLanguage Model Introspection

    Northeastern University · Eleos AI Research

  8. Anosognosia in LLMs: Probing Self-Awareness of Quantized Computational Substrate

    Oct 5, 2026Yoshihiro Izawa, Gouki Minegishi, Yoko YamakataLanguage Model Self-AssessmentMetacognition in Language Models

    The University of Tokyo

  9. Self-Evaluating Recursive Agents

    Oct 4, 2026TianYi Lyu, Xiaozhe Li, Yang Li +7LLM Agent TrainingRecursive Language Models

    Tongji University · Independent · Shanghai Artificial Intelligence Laboratory +3

  10. Representational Control over Self-Report & Behavior Coherence in LLM Risk-Taking

    Oct 2, 2026Rafal Kocielnik, Peiyang Song, Pengrui Han +4Language Model Self-AssessmentLLM Interpretability

    Caltech · Carnegie Mellon University · UIUC +1

  11. LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs

    Oct 2, 2026Seoyeon Ye, Gayoung Kim, Jiyoung Hong +2Factual Knowledge in Language ModelsLLM Evaluation

    Ewha Womans University

  12. Also Small Models Can Reasonably Self-Evaluate Their Confidence

    Sep 30, 2026Idil Kapikiran, Thomas Decker, Thomas RunklerConfidence Estimation in Language ModelsSmall Language Models

    Technical University of Munich · Siemens AG · LMU Munich +1

  13. On the Limits of Metacognitive Monitoring in LLMs

    Sep 28, 2026Dongqi Han, Yifan Yang, Dongsheng LiLanguage Model Error DetectionConfidence Estimation in Language Models

    Microsoft Research Asia

  14. Nudgeability: Reasoning Models Follow Confidence Signals Without Tracking Their Own Competence

    Sep 28, 2026Rohit Saxena, Utkarsh UpadhyayLanguage Model Self-AssessmentLLM Tool Use

  15. Self-Designed Evaluators and Warm Memory for Long-Horizon Agents

    Sep 27, 2026Saeid Asgari, Emre Kiciman, Leonardo de Oliveira Nunes +1Language Model Generation EvaluationLong-Horizon Agent Evaluation

    Microsoft

  16. Scoring the Wrong Question: Readout Failures in Constrained-Option Evaluation

    Sep 27, 2026Jiaxuan Guo, Kejia Zhang, Shuo Xin +3LLM EvaluationConfidence Estimation in Language Models

    Stanford University · University of Maryland, College Park

  17. Self-Reports Do Not Identify Self-Models: An Identifiability Test for Counterfactual Reports

    Sep 26, 2026Phongsakon Mark Konrad, Toygar Tanyel, Serkan AyvazLanguage Model Self-AssessmentMetacognition in Language Models

    Centre for Industrial Software, University of South- ern Denmark, Sønderborg, Denmark · Promake, Newark, DE, USA.

  18. Audio LLMs Know When They Can't Hear You

    Sep 24, 2026Amirhosein Javadi, Richa Dixit, Mehrdad Farajtabar +3Audio-Language Model EvaluationLanguage Model Self-Assessment

    Apple · University of California San Diego

  19. Style, Not Self: Surface Cues Explain Zero-Shot Code Attribution by Large Language Models

    Sep 24, 2026Ehsan Barkhordar, Surendrabikram ThapaLanguage Model FingerprintingCode Language Models

    Koç University · Virginia Tech

  20. An Analysis of Training-Free Self-Reported Confidence in Language Models

    Sep 17, 2026Lukas Meyer, Sofia Rossi, Wei Chen +2Confidence Estimation in Language ModelsLanguage Model Self-Assessment

    DreamAI

  21. When Should LLMs Abstain? Chain-of-Self-Questioning for Selective Risk Control

    Sep 15, 2026Ali ŞenolSelective PredictionLLM Prompting

    Department of Computer Engineering, Tarsus University, Tarsus, Türkiye

  22. Strangers to Themselves: What Language Models Say About Themselves Is Generic

    Sep 9, 2026Phil Blandfort, Urja PawarLanguage Model Self-AssessmentMetacognition in Language Models

    Predictably Weird · Independent

  23. Self-Reports Are Not Verification: Environment-Grounded Auditing of LLM Operators in Evolutionary Search

    Sep 1, 2026Enrong Pan, Ryan Zhou, Ting HuLLM AuditingLanguage Model Self-Assessment

    Queen’s University

  24. The Assistant's Ideal Self

    Aug 31, 2026Mert YazanPersonality Modeling in Language ModelsLanguage Model Self-Assessment

    Leiden University · Research conducted at the Digital Minds Research Sprint, August 2026.

  25. S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?

    Aug 31, 2026Jiajun Shi, Siyuan Tao, Yuhao Wu +18LLM Agent EvaluationLanguage Model Self-Assessment

    ByteDance Seed

  26. Evaluating and Improving LLM Self-Modeling

    Aug 31, 2026Siqi Zeng, Andre N. Assis, Rowan WangLLM EvaluationLanguage Model Introspection

    University of Illinois · Constellation · Anthropic

  27. Sleight of Word Benchmark: Can Language Models Notice If Their Own Output Was Tampered With?

    Aug 30, 2026Alberto CetoliLanguage Model Generation EvaluationLanguage Model Self-Assessment

    Cleo

  28. RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges

    Aug 25, 2026Kaiyuan Liu, Ziyuan Zhuang, Rongxiang Weng +1LLM-as-a-JudgeLanguage Model Self-Assessment

    College of Computer Science and Technology, Zhejiang University · Meituan LongCat Team, China

  29. FIDES: A Concordance Protocol for LLM-Generated Trading Strategies

    Aug 24, 2026Arther Tian, Alex Ding, Simon Wu +1Look-Ahead Bias in Language ModelsLanguage Model Generation Evaluation

    DGrid AI

  30. Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimization

    Aug 19, 2026Chenle Chen, Yangbo Wei, Chao Yao +4LLM-as-a-JudgeLanguage Model Self-Assessment

    University of California, Los Angeles · Shanghai Jiao Tong University · Arizona State University +1