LLM Hallucination Mitigation

LLM: Large Language Model

Ask about this topic

Overview

The field studies how to make language-model answers more faithful to evidence and more honest about uncertainty. Common approaches include retrieval-augmented generation with claim-level verification, knowledge graphs and structured tools, fine-tuning or reinforcement learning, decoding and activation interventions, and calibrated abstention. Evaluation spans factual question answering, long-form summaries, retrieval-augmented systems, high-stakes medical and legal tasks, vision-language models, and tool-using agents. A central tension is reducing unsupported claims without causing excessive refusal, losing useful information, degrading general capabilities, or adding prohibitive cost and latency.

Latest papers

CardsList

180 papers13 papers in the last four weeks, down 7% on the four weeks before. 0.1% of all new papers.

Mar 20, 2026

Neural Uncertainty Principle: A Unified View of Adversarial Fragility and LLM Hallucination

Adversarial vulnerability in vision and hallucination in large language models are conventionally viewed as separate problems, each addressed with modality-specific patches. This study first reveals that they share a common geometric origin: the input and its loss gradient are conjugate observables subject to an irreducible uncertainty bound. Formalizing a Neural Uncertainty Principle (NUP) under a loss-induced state, we find that in near-bound regimes, further compression must be accompanied by increased sensitivity dispersion (adversarial fragility), while weak prompt-gradient coupling leaves generation under-constrained (hallucination). Crucially, this bound is modulated by an input-gradient correlation channel, captured by a specifically designed single-backward probe. In vision, masking highly coupled components improves robustness without costly adversarial training; in language, the same prefill-stage probe detects hallucination risk before generating any answer tokens. NUP thus turns two seemingly separate failure taxonomies into a shared uncertainty-budget view and provides a principled lens for reliability analysis. Guided by this NUP theory, we propose ConjMask (masking high-contribution input components) and LogitReg (logit-side regularization) to improve robustness without adversarial training, and use the probe as a decoding-free risk signal for LLMs, enabling hallucination detection and prompt selection. NUP thus provides a unified, practical framework for diagnosing and mitigating boundary anomalies across perception and generation tasks.

Dong-Xiao Zhang, Hu Lou, Jun-Jie Zhang +2

Northwest Institute of Nuclear Technology, Xi’an, Shaanxi 710024, China. · Department of Computer Science and Technology, Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint Center for ML, Tsinghua University, Beijing 10084, China · the School of Mathematics and Statistics and the Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China.

Hallucination in Language ModelsAdversarial RobustnessLLM Hallucination MitigationLLM Hallucination Detection
AskPDF
Mar 20, 2026

EvidenceRL: Reinforcing Evidence Consistency for Trustworthy Language Models

Large Language Models (LLMs) are fluent but prone to hallucinations, producing answers that appear plausible yet are unsupported by available evidence. This failure is especially problematic in high-stakes domains where decisions must be justified by verifiable information. We introduce EvidenceRL, a reinforcement learning framework that enforces evidence adherence during training. EvidenceRL scores candidate responses for grounding (entailment with retrieved evidence and context) and correctness (agreement with reference answers) and optimizes the generator using Group Relative Policy Optimization (GRPO). We evaluate across two high-stakes domains, cardiac diagnosis and legal reasoning, where EvidenceRL consistently improves evidence grounding and faithfulness without sacrificing task accuracy. On cardiac diagnosis, F1@3 increases from 37.0 to 54.5 on Llama-3.2-3B while grounding (Gmax⁡@3G_{\max}@3) rises from 47.6 to 78.2; hallucinations drop nearly 5×\times and evidence-supported diagnoses increase from 31.8% to 61.6%. On legal reasoning, EvidenceRL raises Faithfulness from 32.8% to 67.6% on Llama-3.1-8B, demonstrating consistent behavioral change across domains. Our code is open-sourced at https://github.com/Wizaaard/EvidenceRL.git.

J. Ben Tamo, Yuxing Lu, Benoit L. Marteau +2

Georgia Institute of Technology · Pekin University

LLM Hallucination MitigationLLM GroundingFaithfulness of Language Model ExplanationsRL for Language Model Reasoning
AskPDF
Mar 18, 2026

Mitigating LLM Hallucinations through Domain-Grounded Tiered Retrieval

Large Language Models (LLMs) have achieved unprecedented fluency but remain susceptible to "hallucinations" - the generation of factually incorrect or ungrounded content. This limitation is particularly critical in high-stakes domains where reliability is paramount. We propose a domain-grounded tiered retrieval and verification architecture designed to systematically intercept factual inaccuracies by shifting LLMs from stochastic pattern-matchers to verified truth-seekers. The proposed framework utilizes a four-phase, self-regulating pipeline implemented via LangGraph: (I) Intrinsic Verification with Early-Exit logic to optimize compute, (II) Adaptive Search Routing utilizing a Domain Detector to target subject-specific archives, (III) Refined Context Filtering (RCF) to eliminate non-essential or distracting information, and (IV) Extrinsic Regeneration followed by atomic claim-level verification. The system was evaluated across 650 queries from five diverse benchmarks: TimeQA v2, FreshQA v2, HaluEval General, MMLU Global Facts, and TruthfulQA. Empirical results demonstrate that the pipeline consistently outperforms zero-shot baselines across all environments. Win rates peaked at 83.7% in TimeQA v2 and 78.0% in MMLU Global Facts, confirming high efficacy in domains requiring granular temporal and numerical precision. Groundedness scores remained robustly stable between 78.8% and 86.4% across factual-answer rows. While the architecture provides a robust fail-safe for misinformation, a persistent failure mode of "False-Premise Overclaiming" was identified. These findings provide a detailed empirical characterization of multi-stage RAG behavior and suggest that future work should prioritize pre-retrieval "answerability" nodes to further bridge the reliability gap in conversational AI.

Md. Asraful Haque, Aasar Mehdi, Maaz Mahboob +1

Computational Unit, Z.H. College of Engineering & Technology, Aligarh Muslim University, India · Interdisciplinary Center for Artificial Intelligence, Aligarh Muslim University, India · Department of Computer Engineering, Aligarh Muslim University, India

LLM Hallucination MitigationLLM GroundingLLM Answer VerificationRetrieval-Augmented Generation
AskPDF
Mar 18, 2026

Inducing Epistemological Humility in Large Language Models: A Targeted SFT Approach to Reducing Hallucination

Large language models (LLMs) often hallucinate, producing fluent but false information, partly because supervised fine-tuning (SFT) implicitly rewards always responding. We introduce HypoTermInstruct\textit{HypoTermInstruct}, an SFT dataset (31,487 responses for 11,151 questions) designed to teach models epistemological humility-the ability to recognize the limits of their own knowledge and admit uncertainty. This is achieved through questions about non-existent "hypothetical" terms. We also release HypoTermQA-Enhanced\textit{HypoTermQA-Enhanced}, a benchmark for hallucination tendency strengthened through multiple validations. We conducted 800 controlled LoRA SFT runs across Llama3.1-8B\textit{Llama3.1-8B} and Gemma3-4B\textit{Gemma3-4B} (base and instruct), testing 100 fine-tuning configurations with paired controls. Our results demonstrate that replacing generic instruction data with HypoTermInstruct\textit{HypoTermInstruct} significantly improves the HypoTerm Score (median increases of 0.19% to 25.91%) and FactScore (+0.39% to +0.86%), while maintaining stable performance on MMLU (minimal decreases of 0.26% to 0.35%). Our work demonstrates that targeted, high-quality SFT data teaching meta-cognitive skills can effectively reduce hallucination without preference/RL pipelines, providing mechanistic insights and a practical path toward more reliable AI systems.

Cem Uluoglakci, Tugba Taskaya Temizel

Information Systems Middle East Technical University Ankara, Turkey · Data Informatics Middle East Technical University Ankara, Turkey

LLM Hallucination MitigationHallucination in Language ModelsLLM ReliabilitySupervised Fine-Tuning
AskPDF
Mar 17, 2026

POaaS: Minimal-Edit Prompt Optimization as a Service to Lift Accuracy and Cut Hallucinations on On-Device sLLMs

Small language models (sLLMs) are increasingly deployed on-device, where imperfect user prompts--typos, unclear intent, or missing context--can trigger factual errors and hallucinations. Existing automatic prompt optimization (APO) methods were designed for large cloud LLMs and rely on search that often produces long, structured instructions; when executed under an on-device constraint where the same small model must act as optimizer and solver, these pipelines can waste context and even hurt accuracy. We propose POaaS, a minimal-edit prompt optimization layer that routes each query to lightweight specialists (Cleaner, Paraphraser, Fact-Adder) and merges their outputs under strict drift and length constraints, with a conservative skip policy for well-formed prompts. Under a strict fixed-model setting with Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct, POaaS improves both task accuracy and factuality while representative APO baselines degrade them, and POaaS recovers up to +7.4% under token deletion and mixup. Overall, per-query conservative optimization is a practical alternative to search-heavy APO for on-device sLLMs.

Jungwoo Shim, Dae Won Kim, Sun Wook Kim +4

Electronics and Telecommunications Research Institute, Republic of Korea

Automatic Prompt OptimizationLLM Hallucination MitigationPrompt RobustnessOn-Device Language Model Inference
AskPDF
Mar 15, 2026

An Industrial-Scale Insurance LLM Achieving Verifiable Domain Mastery and Hallucination Control without Competence Trade-offs

Adapting Large Language Models (LLMs) to high-stakes vertical domains like insurance presents a significant challenge: scenarios demand strict adherence to complex regulations and business logic with zero tolerance for hallucinations. Existing approaches often suffer from a Competency Trade-off - sacrificing general intelligence for domain expertise - or rely heavily on RAG without intrinsic reasoning. To bridge this gap, we present INS-S1, an insurance-specific LLM family trained via a novel end-to-end alignment paradigm. Our approach features two methodological innovations: (1) A Verifiable Data Synthesis System that constructs hierarchical datasets for actuarial reasoning and compliance; and (2) A Progressive SFT-RL Curriculum Framework that integrates dynamic data annealing with a synergistic mix of Verified Reasoning (RLVR) and AI Feedback (RLAIF). By optimizing data ratios and reward signals, this framework enforces domain constraints while preventing catastrophic forgetting. Additionally, we release INSEva, the most comprehensive insurance benchmark to date (39k+ samples). Extensive experiments show that INS-S1 achieves SOTA performance on domain tasks, significantly outperforming DeepSeek-R1 and Gemini-2.5-Pro. Crucially, it maintains top-tier general capabilities and achieves a record-low 0.6% hallucination rate (HHEM). Our results demonstrate that rigorous domain specialization can be achieved without compromising general intelligence.

Qian Zhu, Xinnan Guo, Jingjing Huo +5

Ant Group

Domain Adaptation for LLMsLLM AlignmentLLM Hallucination MitigationRL for Language Model ReasoningLLM Evaluation
AskPDF
Mar 13, 2026

ESG-Bench: Benchmarking Long-Context ESG Reports for Hallucination Mitigation

As corporate responsibility increasingly incorporates environmental, social, and governance (ESG) criteria, ESG reporting is becoming a legal requirement in many regions and a key channel for documenting sustainability practices and assessing firms' long-term and ethical performance. However, the length and complexity of ESG disclosures make them difficult to interpret and automate the analysis reliably. To support scalable and trustworthy analysis, this paper introduces ESG-Bench, a benchmark dataset for ESG report understanding and hallucination mitigation in large language models (LLMs). ESG-Bench contains human-annotated question-answer (QA) pairs grounded in real-world ESG report contexts, with fine-grained labels indicating whether model outputs are factually supported or hallucinated. Framing ESG report analysis as a QA task with verifiability constraints enables systematic evaluation of LLMs' ability to extract and reason over ESG content and provides a new use case: mitigating hallucinations in socially sensitive, compliance-critical settings. We design task-specific Chain-of-Thought (CoT) prompting strategies and fine-tune multiple state-of-the-art LLMs on ESG-Bench using CoT-annotated rationales. Our experiments show that these CoT-based methods substantially outperform standard prompting and direct fine-tuning in reducing hallucinations, and that the gains transfer to existing QA benchmarks beyond the ESG domain.

Siqi Sun, Ben Peng Wu, Mali Jin +3

School of Computer Science, University of Sheffield, Sheffield, UK

LLM Hallucination MitigationLLM Safety BenchmarksDocument QALong-Context QA
AskPDF
Mar 12, 2026

Hallucination as output-boundary misclassification: a composite abstention architecture for language models

Large language models often produce unsupported claims. We frame this as a misclassification error at the output boundary, where internally generated completions are emitted as if they were grounded in evidence. This motivates a composite intervention that combines instruction-based refusal with a structural abstention gate. The gate computes a support deficit score, St, from three black-box signals: self-consistency (At), paraphrase stability (Pt), and citation coverage (Ct), and blocks output when St exceeds a threshold. In a controlled evaluation across 50 items, five epistemic regimes, and three models, neither mechanism alone was sufficient. Instruction-only prompting reduced hallucination sharply, but still showed over-cautious abstention on answerable items and residual hallucination for GPT-3.5-turbo. The structural gate preserved answerable accuracy across models but missed confident confabulation on conflicting-evidence items. The composite architecture achieved high overall accuracy with low hallucination, while also inheriting some over-abstention from the instruction component. A supplementary 100-item no-context stress test derived from TruthfulQA showed that structural gating provides a capability-independent abstention floor. Overall, instruction-based refusal and structural gating show complementary failure modes, which suggests that effective hallucination control benefits from combining both mechanisms.

Angelina Hintsanen

NEXUS Laboratory, Helsinki, Finland

LLM Hallucination MitigationHallucination in Language ModelsLLM AbstentionRefusal Mechanisms in Language ModelsRefusal Behavior in Language ModelsQuestion Answering
AskPDF
Mar 10, 2026

Adaptive Activation Cancellation for Hallucination Mitigation in Large Language Models

Large Language Models frequently generate fluent but factually incorrect text. We propose Adaptive Activation Cancellation (AAC), a real-time inference-time framework that treats hallucination-associated neural activations as structured interference within the transformer residual stream, drawing an explicit analogy to classical adaptive noise cancellation from signal processing. The framework identifies Hallucination Nodes (H-Nodes) via layer-wise linear probing and suppresses them using a confidence-weighted forward hook during auto-regressive generation -- requiring no external knowledge, no fine-tuning, and no additional inference passes. Evaluated across OPT-125M, Phi-3-mini, and LLaMA 3-8B on TruthfulQA and HaluEval, the real-time hook is the only intervention that consistently improves downstream accuracy on all three scales. Critically, the method is strictly surgical: WikiText-103 perplexity and MMLU reasoning accuracy are preserved at exactly 0.0% degradation across all three model scales, a property that distinguishes AAC from interventions that trade fluency or general capability for factual improvement. On the LLaMA 3-8B scale, the hook additionally yields positive generation-level gains (MC1 +0.04; MC2 +0.003; Token-F1 +0.003) while achieving probe-space selectivity 5.94x - 3.5x higher than the ITI baseline -- demonstrating that targeted neuron-level suppression can simultaneously improve factual accuracy and preserve model capability.

Eric Yocam, Varghese Vaidyan, Gurcan Comert +3

The Beacom College of Computer and Cyber Sciences, Dakota State University, Madison, SD 57042, USA. · Department of Computational Data Science and Engineering, North Carolina A&T State University, Greensboro, NC 27411, USA. · Department of Computer Science and Software Engineering, California Polytechnic State University, San Luis Obispo, CA 93407, USA. +2

LLM Hallucination MitigationActivation Steering
AskPDF
Mar 8, 2026

Toward Epistemic Stability: Engineering Consistent Procedures for Industrial LLM Hallucination Reduction

Hallucinations in large language models (LLMs) are outputs that are syntactically coherent but factually incorrect or contextually inconsistent. They are persistent obstacles in high-stakes industrial settings such as engineering design, enterprise resource planning, and IoT telemetry platforms. We present and compare five prompt engineering strategies intended to reduce the variance of model outputs and move toward repeatable, grounded results without modifying model weights or creating complex validation models. These methods include: (M1) Iterative Similarity Convergence, (M2) Decomposed Model-Agnostic Prompting, (M3) Single-Task Agent Specialization, (M4) Enhanced Data Registry, and (M5) Domain Glossary Injection. Each method is evaluated against an internal baseline using an LLM-as-Judge framework over 100 repeated runs per method (same fixed task prompt, stochastic decoding at tau = 0.7. Under this evaluation setup, M4 (Enhanced Data Registry) received ``Better'' verdicts in all 100 trials; M3 and M5 reached 80% and 77% respectively; M1 reached 75%; and M2 was net negative at 34% when compared to single shot prompting with a modern foundation model. We then developed enhanced version 2 (v2) implementations and assessed them on a 10-trial verification batch; M2 recovered from 34% to 80%, the largest gain among the four revised methods. We discuss how these strategies help overcome the non-deterministic nature of LLM results for industrial procedures, even when absolute correctness cannot be guaranteed. We provide pseudocode, verbatim prompts, and batch logs to support independent assessment.

Brian Freeman, Adam Kicklighter, Matt Erdman +1

Trane Technologies, Davidson NC, USA

LLM Hallucination MitigationLLM PromptingLLM ReliabilitySelf-Consistency in Language Models
AskPDF
Mar 8, 2026

A Novel Multi-Agent Architecture to Reduce Hallucinations of Large Language Models in Multi-Step Structural Modeling

Large language models (LLMs) such as GPT and Gemini have demonstrated remarkable capabilities in contextual understanding and reasoning. The strong performance of LLMs has sparked growing interest in leveraging them to automate tasks traditionally dependent on human expertise. Recently, LLMs have been integrated into intelligent agents capable of operating structural analysis software (e.g., OpenSees) to construct structural models and perform analyses. However, existing LLMs are limited in handling multi-step structural modeling due to frequent hallucinations and error accumulation during long-sequence operations. To this end, this study presents a novel multi-agent architecture to automate the structural modeling and analysis using OpenSeesPy. First, problem analysis and construction planning agents extract key parameters from user descriptions and formulate a stepwise modeling plan. Node and element agents then operate in parallel to assemble the frame geometry, followed by a load assignment agent. The resulting geometric and load information is translated into executable OpenSeesPy scripts by code translation agents. The proposed architecture is evaluated on a benchmark of 20 frame problems over ten repeated trials, achieving 100% accuracy in 18 cases and 90% in the remaining two. The architecture also significantly improves computational efficiency and demonstrates scalability to larger structural systems.

Ziheng Geng, Jiachen Liu, Ran Cao +3

Department of Civil and Architectural Engineering, University of Miami, Coral Gables, FL 33146, USA · HBC Engineering Company, Miami, FL 33178, USA · College of Civil Engineering, Hunan University, Changsha, 410082, China +3

Structural EngineeringMulti-Agent LLM SystemsLLM Hallucination MitigationCode Generation
AskPDF
Mar 4, 2026

RADIANT-LLM: an Agentic Retrieval Augmented Generation Framework for Reliable Decision Support in Safety-Critical Nuclear Engineering

Reliable decision support in nuclear engineering requires traceable, domain-grounded knowledge retrieval, yet safety and risk analysis workflows remain hampered by fragmented documentation and hallucination when use pre-trained large language model (LLM) in specialized nuclear domains. To address these challenges, this paper presents RADIANT-LLM (Retrival-Augumented, Domain-Intelligent Agent for Nuclear Technologies using LLM), a multi-modal retrieval-augmented generation (RAG) framework designed for nuclear safety, security, and safeguards applications. The framework uses a local-first, model-agnostic architecture that pairs a multi-modal document ingestion pipeline with a structured, metadata-rich knowledge base, supporting page- and figure-level retrieval from technical documents. An agentic layer coordinates domain-specific tools, enforces citation-backed responses with provenance tracking, and supports human-in-the-loop validation to reduce hallucination risks. To rigorously evaluate this framework, we develop and apply a suite of domain-aware metrics, including Context Precision (CoP), Hallucination Rate (HR), and Visual Recall (ViR), to expert-curated benchmarks derived from Used Nuclear Fuel Storage Facility design guidance. Across varying knowledge base sizes, CoP and ViR remain within an 85--98% band, and hallucination rates are substantially lower than those observed in general-purpose deployments. When the same queries are posed to commercial LLM platforms without the RAG layer, hallucinations and citation errors increase markedly. These results indicate that a locally controlled, multi-modal RAG framework with domain-specific retrieval and provenance enforcement is necessary to achieve the factual accuracy, transparency, and auditability that nuclear engineering workflows demand.

Zavier Ndum Ndum, Jian Tao, John Ford +2

Department of Nuclear Engineering, Texas A&M University, College Station, TX, USA · College of Performance, Visualization and Fine Arts, Texas A&M University, College Station, TX, USA

Agentic RAGMultimodal RAGLLM Hallucination MitigationRAG Evaluation
AskPDF
Feb 26, 2026

Towards Faithful Industrial RAG: A Reinforced Co-adaptation Framework for Advertising QA

Industrial advertising question answering (QA) is a high-stakes task in which hallucinated content, particularly fabricated URLs, can lead to financial loss, compliance violations, and legal risk. Although Retrieval-Augmented Generation (RAG) is widely adopted, deploying it in production remains challenging because industrial knowledge is inherently relational, frequently updated, and insufficiently aligned with generation objectives. We propose a reinforced co-adaptation framework that jointly optimizes retrieval and generation through two components: (1) Graph-aware Retrieval (GraphRAG), which models entity-relation structure over a high-citation knowledge subgraph for multi-hop, domain-specific evidence selection; and (2) evidence-constrained reinforcement learning via Group Relative Policy Optimization (GRPO) with multi-dimensional rewards covering faithfulness, style compliance, safety, and URL validity. Experiments on an internal advertising QA dataset show consistent gains across expert-judged dimensions including accuracy, completeness, and safety, while reducing the hallucination rate by 72%. A two-week online A/B test demonstrates a 28.6% increase in like rate, a 46.2% decrease in dislike rate, and a 92.7% reduction in URL hallucination. The system has been running in production for over half a year and has served millions of QA interactions.

Wenwei Li, Ming Xu, Tianle Xia +7

Tencent

Question AnsweringRetrieval-Augmented GenerationLLM Hallucination MitigationGraph Retrieval-Augmented GenerationRL for Language Models
AskPDF
Feb 23, 2026

No One Size Fits All: QueryBandits for Hallucination Mitigation

Advanced reasoning capabilities in Large Language Models (LLMs) have led to more frequent hallucinations; yet most mitigation work focuses on open-source models for post-hoc detection and parameter editing. The dearth of studies focusing on hallucinations in closed-source models is especially concerning, as they constitute the vast majority of models in institutional deployments. We introduce QueryBandits, a model-agnostic contextual bandit framework that adaptively learns online to select the optimal query-rewrite strategy by leveraging an empirically validated and calibrated reward function. Across 16 QA scenarios, our top QueryBandit (Thompson Sampling) achieves an 87.5% win rate over a No-Rewrite baseline and outperforms zero-shot static policies (e.g., Paraphrase or Expand) by 42.6% and 60.3%, respectively. Moreover, all contextual bandits outperform vanilla bandits across all datasets, with higher feature variance coinciding with greater variance in arm selection. This substantiates our finding that there is no single rewrite policy optimal for all queries. We also discover that certain static policies incur higher cumulative regret than No-Rewrite, indicating that an inflexible query-rewriting policy can worsen hallucinations. Thus, learning an online policy over semantic features with QueryBandits can shift model behavior purely through forward-pass mechanisms, enabling its use with closed-source models and bypassing the need for retraining or gradient-based adaptation.

Nicole Cho, William Watson, Alec Koppel +2

JPMorgan AI Research, New York, NY, USA

LLM Hallucination MitigationQuery ReformulationContextual Bandits
AskPDF
Feb 23, 2026

AgenticSum: An Agentic Inference-Time Framework for Faithful Clinical Text Summarization

Large language models (LLMs) offer substantial promise for automating clinical text summarization, yet maintaining factual consistency remains challenging due to the length, noise, and heterogeneity of clinical documentation. We present AgenticSum, an inference-time, agentic framework that separates context selection, generation, verification, and targeted correction to reduce hallucinated content. The framework decomposes summarization into coordinated stages that compress task-relevant context, generate an initial draft, identify weakly supported spans using internal attention grounding signals, and selectively revise flagged content under supervisory control. We evaluate AgenticSum on two public datasets, using reference-based metrics, LLM-as-a-judge assessment, and human evaluation. Across various measures, AgenticSum demonstrates consistent improvements compared to vanilla LLMs and other strong baselines. Our results indicate that structured, agentic design with targeted correction offers an effective inference time solution to improve clinical note summarization using LLMs.

Fahmida Liza Piya, Rahmatollah Beheshti

University of Delaware, Newark, DE, USA

Text SummarizationLLM Hallucination MitigationLLM AgentsLLM Self-Correction
AskPDF
Feb 19, 2026

Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Study

Uncertainty quantification has emerged as an effective approach to closed-book hallucination detection for LLMs, but existing methods are largely designed for short-form outputs and do not generalize well to long-form generation. We introduce a taxonomy for fine-grained uncertainty quantification in long-form LLM outputs that distinguishes methods by design choices at three stages: response decomposition, unit-level scoring, and response-level aggregation. We formalize several families of consistency-based black-box scorers, providing generalizations and extensions of existing methods. We also introduce FactScore-STEM-Geo, a new 400-question long-form QA dataset spanning four categories across STEM and Geography. In our experiments across multiple LLMs and datasets, we find 1) claim-response entailment consistently performs better or on par with more complex claim-level scorers, 2) claim-level scoring generally yields better results than sentence-level scoring, and 3) uncertainty-aware decoding is highly effective for improving the factuality of long-form outputs. Our framework clarifies relationships between prior methods, enables apples-to-apples comparisons, and provides practical guidance for selecting components for fine-grained UQ.

Dylan Bouchard, Mohit Singh Chauhan, Viren Bajaj +1

CVS Health, Wellesley, MA

LLM Hallucination MitigationHallucination in Language ModelsFactual Consistency EvaluationLLM ReliabilityLLM Hallucination DetectionLLM Uncertainty Estimation
AskPDF
Feb 17, 2026

Fine-Refine: Iterative Fine-grained Refinement for Mitigating Dialogue Hallucination

The tendency for hallucination in current large language models (LLMs) negatively impacts dialogue systems. Such hallucinations produce factually incorrect responses that may mislead users and undermine system trust. Existing refinement methods for dialogue systems typically operate at the response level, overlooking the fact that a single response may contain multiple verifiable or unverifiable facts. To address this gap, we propose Fine-Refine, a fine-grained refinement framework that decomposes responses into atomic units, verifies each unit using external knowledge, assesses fluency via perplexity, and iteratively corrects granular errors. We evaluate factuality across the HybriDialogue and OpendialKG datasets in terms of factual accuracy (fact score) and coverage (Not Enough Information Proportion), and experiments show that Fine-Refine substantially improves factuality, achieving up to a 7.63-point gain in dialogue fact score, with a small trade-off in dialogue quality.

Xiangyan Chen, Yujian Gan, Matthew Purver

Queen Mary University of London, UK · Queen’s University Belfast, UK · Institut Jožef Stefan, Slovenia

LLM Hallucination MitigationHallucination MitigationLLM Self-Refinement
AskPDF
Feb 15, 2026

FloCA: Towards Faithful and Logically Consistent Flowchart Reasoning

Flowchart-oriented dialogue (FOD) systems aim to guide users through multi-turn decision-making or operational procedures by following a domain-specific flowchart to achieve a task goal. In this work, we formalize flowchart reasoning in FOD as grounding user input to flowchart nodes at each dialogue turn while ensuring node transition is consistent with the correct flowchart path. Despite recent advances of LLMs in task-oriented dialogue systems, adapting them to FOD still faces two limitations: (1) LLMs lack an explicit mechanism to represent and reason over flowchart topology, and (2) they are prone to hallucinations, leading to unfaithful flowchart reasoning. To address these limitations, we propose FloCA, a zero-shot flowchart-oriented conversational agent. FloCA uses an LLM for intent understanding and response generation, while delegating flowchart reasoning to an external tool that performs topology-constrained graph execution, ensuring faithful and logically consistent node transitions across dialogue turns. We further introduce an evaluation framework with an LLM-based user simulator and five new metrics covering reasoning accuracy and interaction efficiency. Extensive experiments on FLODIAL and PFDial datasets highlight the bottlenecks of existing LLM/VLM-based methods and demonstrate the superiority of FloCA. The code and dataset are publicly available at https://github.com/Jinzi-Zou/FloCA-flowchart-reasoning.

Jinzi Zou, Bolin Wang, Shuo Zhang +2

MoE KLINNS Lab, Xi’an Jiaotong University, Xi’an 710049, P. R. China · China Mobile Group Shaanxi Co., Ltd.

LLM Hallucination MitigationConversational Agents
AskPDF
Jan 20, 2026

Paper2Rebuttal: A Multi-Agent Framework for Transparent Author Response Assistance

Writing effective rebuttals is a high-stakes task that demands more than linguistic fluency, as it requires precise alignment between reviewer intent and manuscript details. Current solutions typically treat this as a direct-to-text generation problem, suffering from hallucination, overlooked critiques, and a lack of verifiable grounding. To address these limitations, we introduce RebuttalAgent\textbf{RebuttalAgent}, the first multi-agents framework that reframes rebuttal generation as an evidence-centric planning task. Our system decomposes complex feedback into atomic concerns and dynamically constructs hybrid contexts by synthesizing compressed summaries with high-fidelity text while integrating an autonomous and on-demand external search module to resolve concerns requiring outside literature. By generating an inspectable response plan before drafting, RebuttalAgent\textbf{RebuttalAgent} ensures that every argument is explicitly anchored in internal or external evidence. We validate our approach on the proposed RebuttalBench\textbf{RebuttalBench} and demonstrate that our pipeline outperforms strong baselines in coverage, faithfulness, and strategic coherence, offering a transparent and controllable assistant for the peer review process.

Qianli Ma, Chang Guo, Zhiheng Tian +4

AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University

Multi-Agent LLM SystemsLLM Hallucination MitigationScholarly Peer ReviewAgentic RAGText Generation
AskPDF
Jan 5, 2026

Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

As Large Language Models (LLMs) increasingly utilize massive context windows as working memory for autonomous tasks, their reliability fluctuates significantly depending on how information is distributed in real-world corpora. We investigate how fact placement, corpus-level distributions, and anti-hallucination ("Don't Make It Up") prompts influence model behavior by introducing a model-agnostic extended needle-in-a-haystack benchmark designed for scalability, which we apply to evaluate Gemini-2.5-flash, ChatGPT-5-mini, Claude-4.5-haiku, and Deepseek-v3.2-chat. Unlike prior work, we separately evaluate literal extraction, logical inference, and hallucination risk. We identify two critical failure modes: Distributional Collapse, where performance degrades significantly when evidence is dispersed; and a Safety Tax, where anti-hallucination prompts cause over-conservative refusal of present facts and evidence, sharply reducing accuracy. Our results suggest that many failures stem from ineffective context utilization, as models struggle to prioritize relevant information even when it is present. These findings highlight the need for model-specific robustness and effective context management to ensure reliable deployment in long-horizon agentic workflows.

Amirali Ebrahimzadeh, Seyyed M. Salili

Department of Electrical Engineering & Computer Science, University of Michigan, Ann Arbor, MI 48109, USA · Independent Researcher, Ann Arbor, MI 48106, USA

Long-Context RetrievalLLM Hallucination MitigationLanguage Model Safety EvaluationLLM PromptingHallucination in Language ModelsLLM Reliability+1
AskPDF
Nov 14, 2025

Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation

Modern language models fail a fundamental requirement of trustworthy intelligence: knowing when not to answer. Despite achieving impressive accuracy on benchmarks, these models produce confident hallucinations, even when wrong answers carry catastrophic consequences. Our evaluations on GSM8K, MedQA and GPQA show frontier models almost never abstain despite explicit warnings of severe penalties, suggesting that prompts cannot override training that rewards any answer over no answer. As a remedy, we propose Reinforced Hesitation (RH): a modification to Reinforcement Learning from Verifiable Rewards (RLVR) to use ternary rewards (+1 correct, 0 abstention, -λλ error) instead of binary. Controlled experiments on logic puzzles reveal that varying λλ produces distinct models along a Pareto frontier, where each training penalty yields the optimal model for its corresponding risk regime: low penalties produce aggressive answerers, high penalties conservative abstainers. The same frontier holds on MATH Levels 4--5 and on medical QA, where it transfers to an unseen dataset. We then introduce two inference strategies that exploit trained abstention as a coordination signal: cascading routes queries through models with decreasing risk tolerance, while self-cascading re-queries the same model on abstention. Both outperform majority voting with lower computational cost. These results establish abstention as a first-class training objective that transforms ``I don't know'' from failure into a coordination signal, enabling models to earn trust through calibrated honesty about their limits.

Mohamad Amin Mohamadi, Tianhao Wang, Zhiyuan Li

Toyota Technological Institute at Chicago · University of California, San Diego

LLM AbstentionLLM Hallucination MitigationLLM ReliabilityLanguage ModelingReinforcement Learning with Verifiable RewardsRefusal Behavior in Language Models+3
AskPDF
Oct 28, 2025

Mitigating Hallucination in Large Language Models: A Capability-Oriented Survey on RAG, Reasoning, and Agentic Systems

Hallucination remains one of the key obstacles to the reliable deployment of large language models (LLMs). Although various mitigation approaches have been proposed, existing studies often analyze different technical paradigms independently, lacking a unified perspective to understand the underlying mechanisms of different approaches and their correspondence with different types of hallucinations. This survey adopts a capability enhancement perspective to systematically examine hallucination mitigation approaches, focusing on Retrieval-Augmented Generation (RAG), reasoning enhancement, and their integration within agentic systems. Based on their primary mitigation mechanisms, we categorize hallucinations into knowledge-based hallucinations and logic-based hallucinations, analyze how RAG and reasoning enhancement methods respectively improve knowledge acquisition and reasoning reliability, and further discuss the integration mechanisms of retrieval and reasoning capabilities in Agentic Systems for mitigating composite hallucinations. By considering the applicability, mitigation mechanisms, and limitations of different approaches, this survey establishes a unified analytical framework connecting hallucination types, key capability dimensions, and technical paradigms.

Yihan Li, Xiyuan Fu, Ghanshyam Verma +2

Electronic Information School, Wuhan University, Wuhan, China · School of Computing, Dublin City University, Dublin, Ireland · School of Public Health, Wuhan University, Wuhan, China +2

Retrieval-Augmented GenerationLLM Hallucination MitigationHallucination in Language ModelsAgentic RAGLLM Reasoning
AskPDF
Sep 25, 2025

Expert-guided Clinical Text Augmentation via Query-Based Model Collaboration

Data augmentation is a widely used strategy to improve model robustness and generalization by enriching training datasets with synthetic examples. While large language models (LLMs) have demonstrated strong generative capabilities for this purpose, their applications in high-stakes domains like healthcare present unique challenges due to the risk of generating clinically incorrect or misleading information. In this work, we propose a novel query-based model collaboration framework that integrates expert-level domain knowledge to guide the augmentation process to preserve critical medical information. Compared to existing LLM-based and traditional augmentation methods, our generated data significantly improves preservation of critical medical information and reduces hallucinations at both the token and concept levels. Experiments on downstream clinical prediction tasks demonstrate consistent performance gains over existing augmentation methods. This lightweight collaborative framework addresses the gap between LLM augmentation potential and the safety requirements of specialized domains.

Dongkyu Cho, Miao Zhang, Rumi Chunara

Department of Computer Science, New York University, New York, NY, United States · Department of Biostatistics, School of Global Public Health, New York University, New York, NY, United States

Synthetic Data AugmentationData AugmentationLLM Hallucination MitigationMulti-LLM CollaborationMedical LLMs
AskPDF
Sep 17, 2025

Geometric Uncertainty for Detecting and Correcting Hallucinations in LLMs

Large language models are known to hallucinate, generating linguistically plausible but incorrect answers to questions. Uncertainty quantification has been proposed as a strategy to detect such behaviour, but existing methods lack a unified framework to assess reliability at both the prompt and answer level. We introduce a geometric framework which quantifies language model uncertainty at both levels by explicitly modelling a prompt-conditioned semantic distribution in answer embedding space. Our approach is black-box and sampling-based; we generate multiple answers per prompt, and use archetypal analysis to estimate a geometric support for the answer distribution. At the prompt level, we approximate the distribution entropy to quantify uncertainty; for each individual answer, we then use notions of atypicality to assess its reliability relative to the batch. We employ our framework to not only detect hallucinations but correct them, by selecting the batch example deemed most reliable. Experiments show that our framework performs comparably to or better than prior methods on short form question-answering datasets, and achieves superior results on medical datasets where hallucinations carry particularly critical risks. Beyond pure performance, we suggest the theoretical grounding of our work provides support for semantic distributions as useful objects of study for language model uncertainty.

Edward Phillips, Sean Wu, Soheila Molaei +3

Department of Engineering Science, University of Oxford · GlaxoSmithKline · Oxford Suzhou Centre for Advanced Research

Uncertainty QuantificationLLM Hallucination MitigationHallucination in Language ModelsLLM ReliabilityLLM Hallucination DetectionLLM Uncertainty Estimation+1
AskPDF
Sep 4, 2025

Cross-Layer Attention Probing for Fine-Grained Hallucination Detection

With the large-scale adoption of Large Language Models (LLMs) in various applications, there is a growing reliability concern due to their tendency to generate inaccurate text, i.e. hallucinations. In this work, we propose Cross-Layer Attention Probing (CLAP), a novel activation probing technique for hallucination detection, which processes the LLM activations across the entire residual stream as a joint sequence. Our empirical evaluations using five LLMs and three tasks show that CLAP improves hallucination detection compared to baselines on both greedy decoded responses as well as responses sampled at higher temperatures, thus enabling fine-grained detection, i.e. the ability to disambiguate hallucinations and non-hallucinations among different sampled responses to a given prompt. This allows us to propose a detect-then-mitigate strategy using CLAP to reduce hallucinations and improve LLM reliability compared to direct mitigation approaches. Finally, we show that CLAP maintains high reliability even when applied out-of-distribution. Code available at: https://github.com/itsmemala/CLAP

Malavika Suresh, Rahaf Aljundi, Ikechukwu Nkisi-Orji +1

Robert Gordon University, Aberdeen, United Kingdom · Toyota Motor Europe, Brussels, Belgium

LLM Hallucination DetectionLLM ReliabilityLanguage Model ProbingLLM Hallucination Mitigation
AskPDF
Aug 5, 2025

SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revision

Large Vision-Language Models (LVLMs) recently achieve significant breakthroughs in understanding complex visual-textual contexts. However, hallucination issues still limit their real-world applicability. Although previous mitigation methods effectively reduce hallucinations in photographic images, they largely overlook the potential risks posed by stylized images, which play crucial roles in critical scenarios such as game scene understanding, art education, and medical analysis. In this work, we first construct a dataset comprising photographic images and their corresponding stylized versions with carefully annotated caption labels. We then conduct head-to-head comparisons on both discriminative and generative tasks by benchmarking 13 advanced LVLMs on the collected datasets. Our findings reveal that stylized images tend to induce significantly more hallucinations than their photographic counterparts. To address this issue, we propose Style-Aware Visual Early Revision SAVER, a novel mechanism that dynamically adjusts LVLMs' final outputs based on the token-level visual attention patterns, leveraging early-layer feedback to mitigate hallucinations caused by stylized images. Extensive experiments demonstrate that SAVER achieves state-of-the-art performance in hallucination mitigation across various models, datasets, and tasks.

Zhaoxu Li, Chenqi Kong, Yi Yu +6

ROSE Lab, Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · ROSE Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · City University of Hong Kong, Hong Kong SAR +3

VLM EvaluationLLM Hallucination MitigationHallucination in Language ModelsVLM HallucinationLarge Vision-Language ModelsVLM Hallucination Mitigation
AskPDF
Jul 21, 2025

Towards Mitigation of Hallucination for LLM-empowered Agents: Progressive Generalization Bound Exploration and Watchdog Monitor

Empowered by large language models (LLMs), intelligent agents have become a popular paradigm for interacting with open environments to facilitate AI deployment. However, hallucinations generated by LLMs-where outputs are inconsistent with facts-pose a significant challenge, undermining the credibility of intelligent agents. Only if hallucinations can be mitigated, the intelligent agents can be used in real-world without any catastrophic risk. Therefore, effective detection and mitigation of hallucinations are crucial to ensure the dependability of agents. Unfortunately, the related approaches either depend on white-box access to LLMs or fail to accurately identify hallucinations. To address the challenge posed by hallucinations of intelligent agents, we present HalMit, a novel black-box watchdog framework that models the generalization bound of LLM-empowered agents and thus detect hallucinations without requiring internal knowledge of the LLM's architecture. Specifically, a probabilistic fractal sampling technique is proposed to generate a sufficient number of queries to trigger the incredible responses in parallel, efficiently identifying the generalization bound of the target agent. Experimental evaluations demonstrate that HalMit significantly outperforms existing approaches in hallucination monitoring. Its black-box nature and superior performance make HalMit a promising solution for enhancing the dependability of LLM-powered systems.

Siyuan Liu, Wenjing Liu, Zhiwei Xu +3

College of Computer Science, Nankai University · College of intelligent Science and Technology, Inner Mongolia University of Technology · Institute of Computing Technology, Chinese Academy of Sciences +3

LLM Hallucination MitigationHallucination in Language ModelsAI Agent MonitoringLLM Hallucination DetectionLLM AgentsLLM Agent Reliability
AskPDF
May 23, 2025

Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularity

Large language models (LLMs) often produce incorrect answers with high confidence, yet the factors associated with such overconfidence remain insufficiently understood. We study this problem through the lens of knowledge popularity. Using entity-centric factual QA derived from Wikidata triplets, we characterize popularity through question entity popularity, answer popularity, and question-answer co-occurrence. We find two consistent patterns. First, hallucinated answers are far from random: compared with ground-truth answers, they tend to be more popular or more frequently associated with the question entity. Second, confidence is strongly tied to the popularity of generated answers: even among incorrect predictions, more popular answers or those with higher question-answer co-occurrence receive higher confidence. Together, these findings suggest that popular but wrong alternatives may contribute to overconfidence. We further show that popularity-related signals can mitigate overconfidence and improve overall confidence estimation. Across six models and three datasets, incorporating knowledge popularity reduces average confidence on incorrect answers from 0.765 to 0.254 and overall ECE from 0.356 to 0.050, while improving Alignment from 77.08% to 83.72%.

Shiyu Ni, Keping Bi, Jiafeng Guo +1

CAS Key Lab of Network Data Science and Technology, ICT, CAS · State Key Laboratory of AI Safety · University of Chinese Academy of Sciences

LLM Uncertainty EstimationLLM Hallucination MitigationFactual QAConfidence Estimation in Language ModelsHallucination in Language ModelsQuestion Answering+3
AskPDF
May 14, 2025

Atomic Consistency Preference Optimization for Long-Form Question Answering

Large Language Models (LLMs) often produce factoid hallucinations - plausible yet incorrect answers. A common mitigation strategy is model alignment, which improves factual accuracy by training on curated (factual, non-factual) pairs. However, this approach often relies on a stronger model (e.g., GPT-4) or an external knowledge base to assess factual correctness that may not always be accessible. Addressing this, we propose Atomic Consistency Preference Optimization (ACPO), a self-supervised preference-tuning method that enhances factual accuracy without external supervision. ACPO leverages atomic consistency signals (i.e., the agreement of individual facts across multiple stochastic responses) to identify high- and low-quality data pairs for model alignment. Despite being fully self-supervised, ACPO outperforms the strong supervised alignment baseline by 1.95 points averaged across Phi-3 and Llama3 on the LongFact and BioGen datasets, demonstrating its effectiveness in improving factual reliability without relying on external models or knowledge bases.

Jingfeng Chen, Raghuveer Thirukovalluru, Junlin Wang +2

LLM Hallucination MitigationLLM AlignmentFactual Knowledge in Language ModelsPreference OptimizationQuestion AnsweringSelf-Consistency in Language Models
AskPDF
Mar 3, 2025

HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputs

An Achilles heel of Large Language Models (LLMs) is their tendency to hallucinate non-factual statements. A response mixed of factual and non-factual statements poses a challenge for humans to verify and accurately base their decisions on. To combat this problem, we propose Highlighted Chain-of-Thought Prompting (HoT), a technique for prompting LLMs to generate responses with XML tags that ground facts to those provided in the question. That is, given an input question, LLMs would first re-format the question to add XML tags highlighting key facts, and then, generate a response with highlights over the facts referenced from the input. Compared to vanilla chain of thought prompting (CoT), HoT reduces the rate of hallucination and separately improves LLM accuracy consistently on over 22 tasks from arithmetic, reading comprehension, to logical reasoning. When asking humans to verify LLM responses, highlights help time-limited participants to more accurately and efficiently recognize when LLMs are correct. Yet, surprisingly, when LLMs are wrong, HoTs tend to fool users into believing that an answer is correct.

Tin Nguyen, Logan Bolton, Mohammad Reza Taesiri +2

Auburn University · University of Alberta

LLM GroundingLLM Answer VerificationLLM Hallucination MitigationCoT ReasoningHallucination in Language ModelsLLM Reliability
AskPDF