cs.SEMar 8, 2026

Toward Epistemic Stability: Engineering Consistent Procedures for Industrial LLM Hallucination Reduction

Authors: Brian Freeman, Adam Kicklighter, Matt Erdman, Zach Gordon

Organizations: Trane Technologies, Davidson NC, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Hallucinations in large language models (LLMs) are outputs that are syntactically coherent but factually incorrect or contextually inconsistent. They are persistent obstacles in high-stakes industrial settings such as engineering design, enterprise resource planning, and IoT telemetry platforms. We present and compare five prompt engineering strategies intended to reduce the variance of model outputs and move toward repeatable, grounded results without modifying model weights or creating complex validation models. These methods include: (M1) Iterative Similarity Convergence, (M2) Decomposed Model-Agnostic Prompting, (M3) Single-Task Agent Specialization, (M4) Enhanced Data Registry, and (M5) Domain Glossary Injection. Each method is evaluated against an internal baseline using an LLM-as-Judge framework over 100 repeated runs per method (same fixed task prompt, stochastic decoding at tau = 0.7. Under this evaluation setup, M4 (Enhanced Data Registry) received ``Better'' verdicts in all 100 trials; M3 and M5 reached 80% and 77% respectively; M1 reached 75%; and M2 was net negative at 34% when compared to single shot prompting with a modern foundation model. We then developed enhanced version 2 (v2) implementations and assessed them on a 10-trial verification batch; M2 recovered from 34% to 80%, the largest gain among the four revised methods. We discuss how these strategies help overcome the non-deterministic nature of LLM results for industrial procedures, even when absolute correctness cannot be guaranteed. We provide pseudocode, verbatim prompts, and batch logs to support independent assessment.

Figures & tables

Figure 1: Taxonomy of the five hallucination-reduction strategies and the root cause each addresses. M1 and M2 target prompt reasoning and structure; M3 targets agent architecture; M4 and M5 target input data quality.
Figure 2: Iterative convergence profile for M1 across three representative trials. Diamonds mark the iteration reaching σsim=0.85\sigma_{\text{sim}}=0.85 .
FieldTypeDescription
component_typestringPhysical role (e.g., “cooling coil valve”)
normal_rangestringExpected value range in operational state
fault_thresholdstringValue indicating fault or alarm condition
depends_onlistIDs of upstream components
fault_implicationstringWhat downstream effect this fault causes
unitsstringEngineering unit (e.g., %, ∘ F, GPM)
Table 1: Enhanced registry fields added to each raw sensor record (M4).
Figure 3: LLM-as-Judge evaluation pipeline. Each trial generates a method-specific baseline and an enhanced response independently, then a zero-temperature judge produces dimension-level scores and an aggregate verdict.
IDDomainMethodsDescription
T1Enterprise softwareM1, M23-sprint agile plan for a cloud-based real-time IoT telemetry pipeline; Fibonacci story points, role assignments, acceptance criteria, inter-sprint dependencies
T2ERP incident responseM3Production ERP outage at 02:14 EST: 6 simultaneous symptoms spanning connection pool, memory, CPU, load balancer, 6 microservices, 3 cron jobs
T3HVAC diagnosisM4Warm-air fault diagnosis; component identifiers (CMP-1, VLV-3, SNS-9) with raw state values; relationship between compressor and TXV
T4BMS/HVAC troubleshootingM5AHU fault query containing 17 domain acronyms; DX coil lockout, chilled water coil, VAV/DDC/VFD interactions
Table 2: Task scenarios used across the five methods. The prompt text is reproduced verbatim in Appendix A.10 .
MethodBetterSameWorseNotes
 (%)(%)(%) 
M1 (Iterative)75187Stability proxy; some variance
M2 (Decomposed)342541Net negative; context lost in synthesis
M3 (Agents)80191Consistent improvement
M4 (Registry)10000No worse trials; see confound note
M5 (Glossary)77221Consistent improvement
Table 3: D1 Results: 100 Trials, v1 Methods
Figure 4: D1 baseline 100-trial results. All five methods shown. M4 received “Better” in all 100 trials under this judge rubric; M2 was net negative.
MethodBetterSameWorseNotes
 (%)(%)(%) 
M1 v2 (Self-Critique)1000010/10 trials; provisional
M2 v2 (Context-Aware)801010Large gain over v1
M3 v2 (Consensus)1000010/10 trials; provisional
M4 Registry (unch.)10000Consistent with D1
M5 v2 (Dynamic)60400See Section 8.4
Table 4: D2 Results: 10 Trials, v2 Methods (exploratory)
Figure 5: D2 verification results: 10-trial batch. M1 v2, M3 v2, and M4 each received “Better” in all 10 trials; these should be treated as provisional. M2 v2 shows the largest single-method gain over its v1 baseline (+46 points). M5 v2 discussion in Section 8.4 .
Figure 6: Left: v1 (D1) vs. v2 (D2) “Better” rate per method. Right: improvement delta Δ=v2−v1\Delta=\text{v2}-\text{v1} . M2 shows the largest gain (+46 points) under this rubric; M5 shows a nominal decline attributable primarily to sample size.
Figure 7: “Better” ratings across both evaluation datasets (D1 baseline and D2 verification).
MethodD1 v1D2 v2Interpretation
 n=100n=10 
M175100v2 gain likely; n=10 provisional
M23480Large gain; 100-trial follow-up warranted
M380100v2 gain likely; n=10 provisional
M4100100Consistent; confound risk noted
M57760Variance dominates at n=10
Table 5: “Better” (%) Summary: D1 (n=100) and D2 (n=10)
More in this topic

LLM Hallucination Mitigation

Large language model hallucination mitigation develops ways to detect and reduce unsupported, fabricated, or inconsistent outputs.

See all 178 papers

Explore similar work

CardsList
May 30, 2026

Towards Lightweight Reliability: Using Soft Prompts for Hallucination Mitigation in Large Language Models

Large language models (LLMs) have seen widespread adoption across various domains, yet their reliability is frequently undermined by hallucinations - responses that are plausible-sounding but factually incorrect. In high-stakes domains, these errors can reduce trust and introduce real-world risk. To address this challenge, we present a parameter-efficient approach that uses soft prompts to mitigate hallucinated content and promote responsible abstention in generative question-answering (QA) tasks. Our method, called Responsible Contrastive Soft Prompting (RCSP), uses a composite loss to train soft prompts that balance three goals: suppressing hallucinatory content, encouraging abstention under uncertainty, and preserving or improving factual recall. To achieve these goals, we incorporate contrastive loss, curriculum learning, and KL regularization into our training mechanism. We evaluate our approach on five diverse generative QA datasets using an LLM-as-a-Judge framework. Experimental results on the Gemma 3 (12B) and Llama 3.1 (8B) backbones demonstrate that RCSP effectively balances factual recall with hallucination suppression and abstention, yielding a generally superior F-score over standard reasoning and instruction-based prompting baselines. Notably, these improvements are achieved by training only a fraction of the parameters required by other tuning techniques. Our results demonstrate that soft prompts provide a modular and computationally efficient path toward improving LLM reliability.

S M Tahmid Siddiqui, Akib Jawad Ononto, Anoop Singhal +1

The University of Texas at Dallas, Richardson, TX 75080, USA · National Institute of Standards and Technology, Gaithersburg, USA

LLM Hallucination MitigationQuestion AnsweringPrompt TuningLLM ReliabilityLLM Abstention
AskPDF
Aug 12, 2026

Reducing Hallucinations in Large Language Models Through Integrated Self-Verification and Retrieval-Augmented Generation

Large Language Models (LLMs) are progressively used for advanced engineering tasks, includes Computer-Aided Design (CAD) documentation, standards compliance verification, and knowledge retrieval. Still, they are prone to produce hallucinations, outputs that seem convincing but aren't based on context that limit their trustworthiness in high-end engineering applications where precision and compliance are crucial. The paper introduces CoVe-RAG+, a unified framework that integrates Chain-of-Verification (CoVe) with Retrieval-Augmented Generation (RAG) to mitigate hallucinations in the results generated by large language models (LLMs). CoVe-RAG+ supports LLM verification in external sources of authority, such as engineering standards, CAD information, and simulation reports, while applying an iterative self-verification process to validate important claims. CoVe-RAG+ is assessed on engineering activities such as CAD model documentation, standards compliance verification, and the reutilization of historical design data. Experimental findings indicate a 28% improvement in factual accuracy relative to baseline CoVe and RAG methodologies. Moreover, CoVe-RAG+ strengthens user confidence by providing elucidative verification reports and source traceability. The findings indicate that CoVe-RAG+ provides a scalable and reliable option for implementing LLMs in engineering design processes where factual accuracy is critical.

Ashly Joseph

Cisco Systems, San Jose, CA, USA

LLM Hallucination MitigationLLM Answer VerificationRetrieval-Augmented Generation
AskPDF
Oct 19, 2026

Target-Aware Curation of Hallucination Stress Tests for Large Language Models

Hallucination stress tests can yield similar average scores across language models even when different prompts elicit hallucinations in each model. We develop a procedure that iteratively edits source-grounded seed prompts to expose hallucinations in a target model. An LLM editor revises each prompt in response to the target's answer and a judge's category-specific score and explanation. In the main setting, the target also receives feedback on its previous response. Experiments cover 500 seeds from seven datasets and ten topics, with intrinsic and extrinsic hallucinations evaluated across open-weight and closed-source models. Editing substantially increases judge-rated severity over the seeds. In the baseline configuration, retained responses fall in the judge's strong-hallucination bin in 77.2% of runs targeting extrinsic hallucination and 74.2% targeting intrinsic hallucination. Later editing rounds account for more than half of the mean score increase. Among 219 prompts that pass model-assisted source-validity checks, 89 elicit responses containing unsupported claims. Across five models, 200 curated prompts yield similar average severity scores, although pairwise Spearman correlations between prompt-level scores are only ρ=0.0387ρ=0.0387 to 0.18970.1897. Curated prompts can be reused across models, but their diagnostic value depends on source validity and the failures they expose in each model.

Jiho Lee, Bingquan Zhang, Lin Ting Liang +2

Dartmouth College · Oakland University

LLM EvaluationLanguage Model Generation EvaluationHallucination in Language ModelsAdversarial Prompt Generation
AskPDF