Semantic Entropy

Ask about this topic

Overview

Semantic entropy estimates how uncertain a model is by grouping different answers that express the same meaning and measuring the spread across meaning groups. Methods use sampled answers with entailment-based clustering or embedding and graph comparisons, and sometimes combine these signals with token probabilities or conformal calibration. The estimates are used to detect hallucinations, decide when to abstain or route a query to another model, and select informative examples for evaluation across text, audio, medical vision, and code tasks. Results vary by task and benchmark, and a key tension is balancing reliable semantic grouping and calibrated decisions against the cost of extra model calls; no single approach works best everywhere.

Latest papers

CardsList

14 papers4 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

  1. Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models

    Oct 19, 2026Debojeet Ghosh, Shreyansh Kumar, Ayushman GargConfidence Estimation in Language ModelsLanguage Model Calibration

  2. Rarity Determines Minimax Rates of Semantic Entropy From Weak Pairwise Judgments With a Known Channel

    Oct 13, 2026Xiaoli Yu, Jiamiao LiuSemantic EntropyMinimax Estimation

    School of Cyber Security and Information Law, Chongqing University of Posts and Telecommunications, Chongqing 400065, China · Department of Information, Xinqiao Hospital, Army Medical University (Third Military Medical University), Chongqing 400037, China

  3. Embedding Graph Kernel Entropy (EGKE): Fine-grained Hallucination Detection for LLMs from Embeddings

    Oct 9, 2026Dilrangi Sankalpana, Pragatheeswaran Vipulanandan, Uthayasanker Thayasivam +1LLM Hallucination DetectionUncertainty Quantification

    Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka · Department of Electrical and Computer Engineering, University of Miami, Coral Gables, Florida, USA

  4. Evaluating Escalation Signals for LLM Routing: Targets, Controls, and Five Ways to Fool Yourself

    Oct 5, 2026Ramin Pishehvar, Andrea Morandi, Mahesh ViswanathanLLM RoutingLanguage Model Cascading

    Cisco Systems, Inc., San Jose, CA, USA

  5. From Tokens to Semantics: Leveraging Complementary Signals for Hallucination Detection in Black-Box LLMs

    Sep 2, 2026Urja Pawar, Rajitha Ramanayake, Owen O'Neill +5LLM Hallucination DetectionLLM Uncertainty Estimation

    BNY

  6. Revisiting Chain-of-Thought Reasoning under Limited Supervision: Semi-supervised Chain-of-Thought Learning

    Jul 1, 2026Hongyang He, Jiuming Liu, Victor SanchezSelf-Training for Language ModelsCoT Reasoning

    University of Warwick · University of Cambridge

  7. Agentic AI-based Framework for Mitigating Premature Diagnostic Handoff and Silent Hallucination in Healthcare Applications

    Jun 16, 2026Divyansh Srivastava, Shreya Ghosh, Anshul Verma +1Multi-Agent LLM SystemsLLM Hallucination Mitigation

    Quantum Cloud Computing and Distributed Systems (qCLOUDS) Lab, School of Computing and Information Systems, The University of Melbourne, Australia · Department of Computer Science and Engineering, School of Electrical and Computer Sciences (SECS), Indian Institute of Technology Bhubaneswar, India · Department of Computer Science2026 Banaras Hindu University, Varanasi, India

  8. FASE: Fast Adaptive Semantic Entropy for Code Quality

    Jun 8, 2026Shizhe Lin, Ladan TahvildariSemantic EntropyLLM Uncertainty Estimation

    University of Waterloo, Canada

  9. VIHD: Visual Intervention-based Hallucination Detection for Medical Visual Question Answering

    May 20, 2026Jiayi Chen, Benteng Ma, Zehui Liao +3Hallucination Detection in VLMsHealthcare

    Department of Data Science & AI, Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia · Hong Kong Polytechnic University, Hong Kong SAR, China · Alfred Health Radiology, Alfred Health, Melbourne, VIC 3004, Australia +1

  10. Active Testing of Large Language Models via Approximate Neyman Allocation

    May 11, 2026Zeli Liu, Jiancheng Zhang, Cong Liu +1LLM EvaluationLanguage Model Generation Evaluation

    University of California, Riverside

  11. LLMs Uncertainty Quantification via Adaptive Conformal Semantic Entropy

    May 5, 2026Hamed Karimi, Vaishali Meyappan, Reza SamaviSelective PredictionSemantic Entropy

    Toronto Metropolitan University, Toronto, Ontario, Canada · Vector Institute, Toronto, Ontario, Canada

  12. Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models

    Apr 28, 2026Chun-Yi Kuan, Wei-Ping Huang, Hung-yi LeeSemantic EntropyAudio Understanding

    Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · Artificial Intelligence Center of Research Excellence (AI-CoRE), National Taiwan University, Taiwan

  13. BEACON: Behavioral Entropy Aggregation for Cross-Model Hallucination Detection in Large Language Models

    Apr 20, 2026Naveen Bera, Pulijala Sai Nikhila, Kondaguduru Abhiram +6LLM EvaluationHallucination Detection

    LLM Lens

  14. CoE: Collaborative Entropy for Uncertainty Quantification in Agentic Multi-LLM Systems

    Mar 30, 2026Kangkang Sun, Jun Wu, Jianhua Li +3LLM Uncertainty EstimationMulti-LLM Collaboration

    Shanghai Key Laboratory of Integrated Administration Technologies for Information Security School of Computer Science Shanghai Jiao Tong University Shanghai, China · School of Science and Engineering The Chinese University of Hong Kong, Shenzhen Shenzhen, China