LLM Reliability

LLM: Large Language Model

Ask about this topic

Overview

It asks not only whether a model gets an answer right, but whether it knows when it may be wrong, follows constraints, uses evidence faithfully, and behaves consistently across repeated or multi-turn use. Researchers test these properties with adversarial and perturbation benchmarks, repeated-run and cross-language evaluations, domain-specific tasks, and audits of LLM judges and tool-using agents. Common approaches include estimating uncertainty from logits, hidden representations, or response agreement; selective abstention and conformal calibration; retrieval and symbolic verification; and training methods that reward grounded answers, correction, or hesitation. The central tension is that benchmark accuracy and verbal confidence often fail to predict reliability on individual cases, long tasks, or shifted inputs, while stronger checks and abstention trade coverage, latency, and cost for safer decisions.

Latest papers

CardsList

802 papers116 papers in the last four weeks, up 18% on the four weeks before. 0.7% of all new papers.

Apr 19, 2026

Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models

Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in various perception and reasoning tasks. Despite this success, ensuring their reliability in practical deployment necessitates robust confidence estimation. Prior works have predominantly focused on text-only LLMs, often relying on computationally expensive self-consistency sampling. In this paper, we extend this to multimodal settings and conduct a comprehensive evaluation of MLLMs' response confidence estimation. Our analysis reveals a significant instinct-reflection misalignment: the model's implicit token-level support frequently diverges from its verbal self-assessment confidence. To address this misalignment, we propose a monotone confidence fusion framework to merge dual-channel signals and cross-channel consistency to estimate correctness. Subsequently, an order-preserving mean alignment step is applied to correct global bias, which improves calibration while preserving the risk-coverage trade-off for selective prediction. Experiments on diverse open-source and closed-source MLLMs show that our method consistently yields more reliable confidence estimates and improves both calibration and failure prediction. Code will be available at https://github.com/Yunkaidang/Instinct-vs.-Reflection.

Yunkai Dang, Yifan Jiang, Yizhu Jiang +3

School of Artificial Intelligence, Nanjing University.

Selective PredictionConfidence Estimation in Language ModelsMultimodal Large Language ModelsLanguage Model CalibrationLLM ReliabilityLLM Uncertainty Estimation+1
AskPDF
Apr 18, 2026

Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification

Large language models (LLMs) often produce confident yet incorrect responses, and uncertainty quantification is one potential solution to more robust usage. Recent works routinely rely on self-consistency to estimate aleatoric uncertainty (AU), yet this proxy collapses when models are overconfident and produce the same incorrect answer across samples. We analyze this regime and show that cross-model semantic disagreement is higher on incorrect answers precisely when AU is low. Motivated by this, we introduce an epistemic uncertainty (EU) term that operates in the black-box access setting: EU uses only generated text from a small, scale-matched ensemble and is computed as the gap between inter-model and intra-model sequence-semantic similarity. We then define total uncertainty (TU) as the sum of AU and EU. In a comprehensive study across five 7-9B instruction-tuned models and ten long-form tasks, TU improves ranking calibration and selective abstention relative to AU, and EU reliably flags confident failures where AU is low. We further characterize when EU is most useful via agreement and complementarity diagnostics.

Kimia Hamidieh, Veronika Thost, Walter Gerych +2

1MIT · MIT-IBM Watson AI Lab · 3WPI +1

Selective PredictionConfidence Estimation in Language ModelsLLM ReliabilityLLM Uncertainty EstimationAleatoric UncertaintyUncertainty Calibration+2
AskPDF
Apr 18, 2026

Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL

Reinforcement fine-tuning improves the reasoning ability of large language models, but it can also encourage them to answer unanswerable queries by guessing or hallucinating missing information. Existing abstention methods either train models to produce generic refusals or encourage follow-up clarifications without verifying whether those clarifications identify the key missing information. We study queries that are clear in meaning but cannot be reliably resolved from the given information, and argue that a reliable model should not only abstain, but also explain what is missing. We propose a clarification-aware RLVR reward that, while rewarding correct answers on answerable queries, jointly optimizes explicit abstention and semantically aligned post-refusal clarification on unanswerable queries. Using this reward, we train Abstain-R1, a 3B model that improves abstention and clarification on unanswerable queries while preserving strong performance on answerable ones. Experiments on Abstain-Test, Abstain-QA, and SelfAware show that Abstain-R1 substantially improves over its base model and achieves unanswerable-query behavior competitive with larger systems including DeepSeek-R1, suggesting that calibrated abstention and clarification can be learned through verifiable rewards rather than emerging from scale alone.

Skylar Zhai, Jingcheng Liang, Dongyeop Kang

University of Minnesota

LLM Hallucination MitigationRL for Language Model ReasoningLLM ReliabilityReinforcement Learning with Verifiable RewardsRefusal Behavior in Language ModelsLLM Abstention
AskPDF
Apr 17, 2026

Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench

Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, but this assumption has rarely been validated against human annotation. We introduce AgentProp-Bench, a 2,000-task benchmark with 2,300 traces across four domains, nine production LLMs, and a 100-label human-validated subset. We quantify judge reliability, characterize error propagation, and evaluate a runtime mitigation. Substring-based judging agrees with human annotation at kappa=0.049 (chance-level); a three-LLM ensemble reaches kappa=0.432 (moderate) with a conservative bias. Under validated evaluation, a parameter-level injection propagates to a wrong final answer with human-calibrated probability approximately 0.62 (range 0.46-0.73 across models). Rejection (catching bad parameters) and recovery (correcting after acceptance) are independent model capabilities (Spearman rho=0.126, p=0.747). A tuned runtime interceptor reduces hallucination on GPT-4o-mini by 23.0 percentage points under a concurrent n=600 control, but shows no significant effect on Gemini-2.0-Flash, whose aggressive parameter rejection eliminates the target failure mode. All code, data, traces, and human labels are released at https://github.com/bhaskargurram-ai/agenthallu-bench.

Bhaskar Gurram

Zasti Inc., Ashburn, VA, USA

AI Agent ReliabilityTool-Augmented Language Model AgentsLLM-as-a-JudgeLLM Agent EvaluationAI Agent BenchmarksLLM Reliability+2
AskPDF
Apr 17, 2026

No-Worse Context-Aware Decoding: Preventing Neutral Regression in Context-Conditioned Generation

Large language models (LLMs) can answer questions and summarize documents when conditioned on external contexts (e.g., retrieved evidence), yet context use remains unreliable: models may overwrite an already-correct output (neutral regression) even when the context is non-informative. We formalize neutral regression as a do-no-harm requirement and quantify it by measuring accuracy drops on baseline-correct items under answer-consistent contexts. We propose No-Worse Context-Aware Decoding (NWCAD), a decode-time adapter built on a two-stream setup with a two-stage gate: it backs off to no-context decoding when the context is non-informative, and otherwise uses context-conditioned decoding with a CAD-style fallback under uncertainty. We evaluate NWCAD on benchmarks that separate do-no-harm reliability from context utilization (accuracy gains on genuinely helpful contexts). NWCAD prevents neutral regression on baseline-correct items while preserving strong context-driven accuracy on helpful contexts.

Yufei Tao, Ameeta Agrawal

Department of Computer Science, Portland State University, USA

Language Model DecodingLLM ReliabilityLanguage Model Robustness
AskPDF
Apr 17, 2026

Beyond Surface Statistics: Robust Conformal Prediction for LLMs via Internal Representations

Large language models are increasingly deployed in settings where reliability matters, yet output-level uncertainty signals such as token probabilities, entropy, and self-consistency can become brittle under calibration--deployment mismatch. Conformal prediction provides finite-sample validity under exchangeability, but its practical usefulness depends on the quality of the nonconformity score. We propose a conformal framework for LLM question answering that uses internal representations rather than output-facing statistics: specifically, we introduce Layer-Wise Information (LI) scores, which measure how conditioning on the input reshapes predictive entropy across model depth, and use them as nonconformity scores within a standard split conformal pipeline. Across closed-ended and open-domain QA benchmarks, with the clearest gains under cross-domain shift, our method achieves a better validity--efficiency trade-off than strong text-level baselines while maintaining competitive in-domain reliability at the same nominal risk level. These results suggest that internal representations can provide more informative conformal scores when surface-level uncertainty is unstable under distribution shift.

Yanli Wang, Peng Kuang, Xiaoyu Han +2

Imperial College London · Zhejiang University · University of Illinois Urbana-Champaign +1

Language Model RobustnessConformal PredictionConfidence Estimation in Language ModelsLanguage Model CalibrationLLM ReliabilityConformal Risk Control+2
AskPDF
Apr 17, 2026

LLMs Corrupt Your Documents When You Delegate

Large Language Models (LLMs) are poised to disrupt knowledge work, with the emergence of delegated work as a new interaction paradigm (e.g., vibe coding). Delegation requires trust - the expectation that the LLM will faithfully execute the task without introducing errors into documents. We introduce DELEGATE-52 to study the readiness of AI systems in delegated workflows. DELEGATE-52 simulates long delegated workflows that require in-depth document editing across 52 professional domains, such as coding, crystallography, and music notation. Our large-scale experiment with 19 LLMs reveals that current models degrade documents during delegation: even frontier models (Gemini 3.1 Pro, Claude 4.6 Opus, GPT 5.4) corrupt an average of 25% of document content by the end of long workflows, with other models failing more severely. Additional experiments reveal that agentic tool use does not improve performance on DELEGATE-52, and that degradation severity is exacerbated by document size, length of interaction, or presence of distractor files. Our analysis shows that current LLMs are unreliable delegates: they introduce sparse but severe errors that silently corrupt documents, compounding over long interaction.

Philippe Laban, Tobias Schnabel, Jennifer Neville

Microsoft Research

LLM ReliabilityLong-Horizon LLM AgentsLLM Agent Reliability
AskPDF
Apr 16, 2026

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. We present a two-pronged diagnostic toolkit applied to SummEval: (1)\textbf{(1)} a transitivity analysis that reveals widespread per-input inconsistency masked by low aggregate violation rates (ρˉ=0.8\barρ = 0.8-4.1%4.1\%), with 3333-67%67\% of documents exhibiting at least one directed 3-cycle; and (2)\textbf{(2)} split conformal prediction sets over 1-5 Likert scores providing theoretically-guaranteed ≥(1−α)\geq(1{-}α) coverage, with set width serving as a per-instance reliability indicator (rs=+0.576r_s = {+}0.576, N=1,918N{=}1{,}918, p<10−100p < 10^{-100}, pooled across all judges). Critically, prediction set width shows consistent cross-judge agreement (rˉ=0.32\bar{r} = 0.32-0.380.38), demonstrating it captures document-level difficulty rather than judge-specific noise. Across four judges and four criteria, both diagnostics converge: criterion matters more than judge, with relevance judged most reliably (avg. set size ≈3.0\approx 3.0) and coherence moderately so (avg. set size ≈3.9\approx 3.9), while fluency and consistency remain unreliable (avg. set size ≈4.9\approx 4.9). We release all code, prompts, and cached results.

Manan Gupta, Dhruv Kumar

1BITS Pilani, Pilani Campus, India

LLM EvaluationLLM-as-a-JudgeLanguage Model Generation EvaluationLLM ReliabilitySummarization EvaluationAutomated Evaluation
AskPDF
Apr 16, 2026

IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation

Despite the rapid advancement of Large Language Models (LLMs), uncertainty quantification in LLM generation is a persistent challenge. Although recent approaches have achieved strong performance by restricting LLMs to produce short or constrained answer sets, many real-world applications require long-form and free-form text generation. A key difficulty in this setting is that LLMs often produce responses that are semantically coherent yet factually inaccurate, while the underlying semantics are multifaceted and the linguistic structure is complex. To tackle this challenge, this paper introduces Interrogative Uncertainty Quantification (IUQ), a novel framework that leverages inter-sample consistency and intra-sample faithfulness to quantify the uncertainty in long-form LLM outputs. By utilizing an interrogate-then-respond paradigm, our method provides reliable measures of claim-level uncertainty and the model's faithfulness. Experimental results across diverse model families and model sizes demonstrate the superior performance of IUQ over two widely used long-form generation datasets. The code is available at https://github.com/louisfanhz/IUQ.

Haozhi Fan, Jinhao Duan, Kaidi Xu

University of Pennsylvania · 2UNC Chapel Hill · 3City University of Hong Kong

Uncertainty QuantificationLanguage Model Generation EvaluationLLM ReliabilityLLM Uncertainty Estimation
AskPDF
Apr 16, 2026

Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options

Multiple choice evaluation is widely used for benchmarking large language models, yet near ceiling accuracy in low option settings can be sustained by shortcut strategies that obscure true competence. Therefore, we propose a massive option evaluation protocol that scales the candidate set to one hundred options and sharply reduces the impact of chance performance. We apply this framework to a Korean orthography error detection task where models must pick the single incorrect sentence from a large candidate set. With fixed targets and repeated resampling and shuffling, we obtain stable estimates while separating content driven failures from positional artifacts. Across experiments, results indicate that strong performance in low option settings can overstate model competence. This apparent advantage often weakens under dense interference at high NN, revealing gaps that conventional benchmarks tend to obscure. We identify two failure modes, semantic confusion and position bias toward early options under uncertainty. To isolate the effect of context length, we run padding controlled and length matched tests, which suggest that the main bottleneck is candidate ranking rather than context length. Together, these findings support massive option evaluation as a general framework for stress testing model reliability under extreme distractor density, beyond what low option benchmarks can reveal.

Nahyun Lee, Guijin Son

Chung-Ang University · Seoul National University

LLM EvaluationMultiple-Choice Question AnsweringLLM ReliabilityLanguage Model Robustness
AskPDF
Apr 15, 2026

Correct Prediction, Wrong Steps? Consensus Reasoning Knowledge Graph for Robust Chain-of-Thought Synthesis

Large language models (LLMs) have become increasingly used for various tasks, often coupled with Chain-of-Thought (CoT) prompting to boost accuracy. Recent work has shown that high label-prediction accuracy does not guarantee correct intermediate reasoning, and the causes of reasoning flaws vary from sample to sample, yet existing remedies either focus on a single domain or assume that one flaw type applies uniformly across samples. A simple mitigation method is to provide the model with the correct answer, but we show that this yields no consistent improvement in reasoning quality. This indicates that the problem cannot be fixed by LLMs' awareness of answers, and must instead be addressed through the structure of reasoning. Motivated by this, we propose CRAFT (Consensus Reasoning knowledge graph Aggregation for Flaw-aware Trace synthesis), which aggregates the consensus components shared across multiple candidate reasoning traces to synthesize improved ones. CRAFT consistently improves label-prediction accuracy on both logical and mathematical reasoning benchmarks, outperforming most baselines, while its post-processed traces achieve higher quality under fine-grained benchmark evaluation.

Zipeng Ling, Shuliang Liu, Seonil Son +4

The Hong Kong University of Science and Technology (Guangzhou) · University of Alberta · Alberta Machine Intelligence Institute +3

Reasoning Trace AnalysisLogical ReasoningSelf-Consistency DecodingCoT ReasoningMathematical ReasoningStructured Reasoning+3
AskPDF
Apr 15, 2026

ERRORQUAKE: Heavy-Tailed Error Severity Distributions in Open-Weight Large Language Models

At matched accuracy, open-weight LLMs differ substantially in the shape of their error severity distribution -- a difference invisible to the scalar error rate. Hallucination benchmarks report a single error count and treat all errors as equivalent, yet a wrong date and a fabricated court ruling differ by orders of magnitude. We introduce Errorquake-10k, a 10,000-query benchmark scoring each response on a continuous 0-4 severity scale across 8 domains and 5 difficulty tiers, and we fit per-model severity distributions for 21 open-weight models. For each model we estimate a severity distribution index (b, the Gutenberg-Richter upper-tail slope) with 95% bootstrap confidence intervals. Headline: across the 210 model pairs, 85 have disjoint 95% b confidence intervals at matched accuracy (|Delta epsilon| < 0.05) on human-consensus scoring, e.g. deepseek-v3.2 vs. ministral-14b at epsilon = 0.586 and Delta b = 0.47. A 519-item three-rater human validation study confirms measurement reliability (ICC(2,k=3) = 0.85), validates the LLM-judge ranking (rho = 0.89), and confirms the dense-model scaling correlation on human data (rho_s = -0.86). We prove a Non-Reducibility Theorem showing that severity profile and error rate are informationally non-redundant (I(b; model | epsilon) = 1.56 bits; 64.5% of cross-model b variance is unexplained by epsilon). A severity mechanism taxonomy (kappa = 0.83) reveals that error type shifts categorically with severity: low-severity errors are retrievals (71%); high-severity errors are fabrications (39%) -- and this composition differs by model size (p < 0.0001). Severity distribution should be reported alongside accuracy; it carries discriminative information that the error rate cannot.

Jason Z Wang

Independent

LLM EvaluationOpen-Weight Language ModelsLLM ReliabilityHallucination in Language Models
AskPDF
Apr 15, 2026

The Cognitive Circuit Breaker: A Systems Engineering Framework for Intrinsic AI Reliability

As Large Language Models (LLMs) are increasingly deployed in mission-critical software systems, detecting hallucinations and faked truthfulness'' has become a paramount engineering challenge. Current reliability architectures rely heavily on post-generation, black-box mechanisms, such as Retrieval-Augmented Generation (RAG) cross-checking or LLM-as-a-judge evaluators. These extrinsic methods introduce unacceptable latency, high computational overhead, and reliance on secondary external API calls, frequently violating standard software engineering Service Level Agreements (SLAs). In this paper, we propose the Cognitive Circuit Breaker, a novel systems engineering framework that provides intrinsic reliability monitoring with minimal latency overhead. By extracting hidden states during a model's forward pass, we calculate the Cognitive Dissonance Delta'' -- the mathematical gap between an LLM's outward semantic confidence (softmax probabilities) and its internal latent certainty (derived via linear probes). We demonstrate statistically significant detection of cognitive dissonance, highlight architecture-dependent Out-of-Distribution (OOD) generalization, and show that this framework adds negligible computational overhead to the active inference pipeline.

Jonathan Pan

Home Team Science and Technology Agency Singapore

LLM ReliabilityLLM Hallucination DetectionLLM Uncertainty Estimation
AskPDF
Apr 14, 2026

Numerical Instability and Chaos: Quantifying the Unpredictability of Large Language Models

As Large Language Models (LLMs) are increasingly integrated into agentic workflows, their unpredictability stemming from numerical instability has emerged as a critical reliability issue. While recent studies have demonstrated the significant downstream effects of these instabilities, the root causes and underlying mechanisms remain poorly understood. In this paper, we present a rigorous analysis of how unpredictability is rooted in the finite numerical precision of floating-point representations, tracking how rounding errors propagate, amplify, or dissipate through Transformer computation layers. Specifically, we identify a chaotic "avalanche effect" in the early layers, where minor perturbations trigger binary outcomes: either rapid amplification or complete attenuation. Beyond specific error instances, we demonstrate that LLMs exhibit universal, scale-dependent chaotic behaviors characterized by three distinct regimes: 1) a stable regime, where perturbations fall below an input-dependent threshold and vanish, resulting in constant outputs; 2) a chaotic regime, where rounding errors dominate and drive output divergence; and 3) a signal-dominated regime, where true input variations override numerical noise. We validate these findings extensively across multiple datasets and model architectures.

Chashi Mahiul Islam, Alan Villarreal, Mao Nishino +2

Department of Computer Science, Florida State University, Tallahassee, USA · Department of Mathematics, Florida State University, Tallahassee, USA

LLM Reliability
AskPDF
Apr 14, 2026

Is Vibe Coding the Future? An Empirical Assessment of LLM Generated Codes for Construction Safety

The emergence of vibe coding, a paradigm where non-technical users instruct Large Language Models (LLMs) to generate executable codes via natural language, presents both significant opportunities and severe risks for the construction industry. While empowering construction personnel such as the safety managers, foremen, and workers to develop tools and software, the probabilistic nature of LLMs introduces the threat of silent failures, wherein generated code compiles perfectly but executes flawed mathematical safety logic. This study empirically evaluates the reliability, software architecture, and domain-specific safety fidelity of 450 vibe-coded Python scripts generated by three frontier models, Claude 3.5 Haiku, GPT-4o-Mini, and Gemini 2.5 Flash. Utilizing a persona-driven prompt dataset (n=150) and a bifurcated evaluation pipeline comprising isolated dynamic sandboxing and an LLM-as-a-Judge, the research quantifies the severe limits of zero-shot vibe codes for construction safety. The findings reveal a highly significant relationship between user persona and data hallucination, demonstrating that less formal prompts drastically increase the AI's propensity to invent missing safety variables. Furthermore, while the models demonstrated high foundational execution viability (~85%), this syntactic reliability actively masked logic deficits and a severe lack of defensive programming. Among successfully executed scripts, the study identified an alarming ~45% overall Silent Failure Rate, with GPT-4o-Mini generating mathematically inaccurate outputs in ~56% of its functional code. The results demonstrate that current LLMs lack the deterministic rigor required for standalone safety engineering, necessitating the adoption of deterministic AI wrappers and strict governance for cyber-physical deployments.

S M Jamil Uddin

Assistant Professor, Department of Construction Management, Colorado State University

Construction SafetyLLM Code GenerationLLM ReliabilityCode Generation Evaluation
AskPDF
Apr 13, 2026

Phantom Paradoxes: Language Models Treat Unresolved Circular Truth Dependence as Contradiction

Do language models struggle with self-reference itself, or with structures where propagating truth values along references cannot settle a value in finitely many steps? Our results support the latter. In our main test, statements refer to one another's truth in a cycle, yet exactly one consistent assignment exists. Local propagation never settles the queried value, but exhaustive search finds it, so INCONSISTENT (no consistent assignment) is always wrong. We call such misreadings of satisfiable problems phantom paradoxes. Each circular problem has a grounded twin with the same answer but no cycle. To rule out logical difficulty and unfamiliar wording, we added constraint twins (equivalent formulas without talk of truth) and perplexity-matched grounded controls. In confirmatory behavioral tests fixed before any confirmation response, four open models from three families answered INCONSISTENT on average 31.0 percentage points more often on circular problems than on grounded twins, 36.4 more than on constraint twins and 37.1 more than on perplexity-matched controls. In each model, validated layer-wise truth directions, fitted on unrelated arithmetic, recovered the queried value at the question's last token for grounded twins but not for circular problems. Constraint twins, likewise unrecovered, drew INCONSISTENT far less often, so circular truth dependence, not the unrecovered value alone, matters. In a small natural-language bank, self-reference settling in finitely many steps drew INCONSISTENT 23.4 points less often than unsettled self-reference with the same classical solution. Pooled tests did not show the predicted late-layer revision or attention spread. Overall, models treat unresolved circular truth dependence as contradiction. We will release all prompts, outputs and hidden states.

Ji Ho Bae

JRTI

LLM InterpretabilityLLM Reliability
AskPDF
Apr 13, 2026

Leveraging Weighted Syntactic and Semantic Context Assessment Summary (wSSAS) Towards Text Categorization Using LLMs

The use of Large Language Models (LLMs) for reliable, enterprise-grade analytics such as text categorization is often hindered by the stochastic nature of attention mechanisms and sensitivity to noise that compromise their analytical precision and reproducibility. To address these technical frictions, this paper introduces the Weighted Syntactic and Semantic Context Assessment Summary (wSSAS), a deterministic framework designed to enforce data integrity on large-scale, chaotic datasets. We propose a two-phased validation framework that first organizes raw text into a hierarchical classification structure containing Themes, Stories, and Clusters. It then leverages a Signal-to-Noise Ratio (SNR) to prioritize high-value semantic features, ensuring the model's attention remains focused on the most representative data points. By incorporating this scoring mechanism into a Summary-of-Summaries (SoS) architecture, the framework effectively isolates essential information and mitigates background noise during data aggregation. Experimental results using Gemini 2.0 Flash Lite across diverse datasets - including Google Business reviews, Amazon Product reviews, and Goodreads Book reviews - demonstrate that wSSAS significantly improves clustering integrity and categorization accuracy. Our findings indicate that wSSAS reduces categorization entropy and provides a reproducible pathway for improving LLM based summaries based on a high-precision, deterministic process for large-scale text categorization.

Shreeya Verma Kathuria, Nitin Mayande, Sharookh Daruwalla +2

Tellagence Inc. · Villanova School of Business, Villanova University

Text ClusteringText ClassificationLLM ReliabilityLanguage Model RobustnessClusteringText Summarization
AskPDF
Apr 13, 2026

When AI Tells You What You Want to Hear: Sycophantic Behavior of Large Language Models in Dementia Care Settings

Large language models (LLMs) are increasingly used in clinical and care settings. This exploratory study investigates whether LLMs exhibit sycophantic behavior - adapting their responses to social expectation signals rather than maintaining professional quality - in the context of dementia care. Five prompts with systematically increasing confirmatory and authority-related framing (P1 neutral to P5 authority-signaled implementation support) were submitted to four LLMs (GPT-5, Claude Sonnet 4.6, Gemini 3.1 Pro, Mistral Large), each repeated five times (N = 100 responses). Responses were evaluated using an LLM-as-a-Judge methodology against seven nursing-ethical quality criteria (K1-K7) and a tone scale (0-3). All models showed significant negative Spearman correlations between prompt level and response quality (rho ranging from -0.543 to -0.734, all p < 0.01). Mistral Large exhibited the most pronounced effect (rho = -0.734), with mean scores dropping from 6.0/7 at P1 to 0.2/7 at P5. The findings suggest that LLMs pose context-sensitive risks in high-stakes care environments and that prompt framing significantly shapes response quality - a dimension that has received insufficient attention in healthcare AI deployment.

Christian Kolb

Dementia CareLLM SycophancyLLM ReliabilityHealthcareClinical Language Model Evaluation
AskPDF
Apr 13, 2026

From Answers to Arguments: Toward Trustworthy Clinical Diagnostic Reasoning with Toulmin-Guided Curriculum Goal-Conditioned Learning

The integration of Large Language Models (LLMs) into clinical decision support is critically obstructed by their opaque and often unreliable reasoning. In the high-stakes domain of healthcare, correct answers alone are insufficient; clinical practice demands full transparency to ensure patient safety and enable professional accountability. A pervasive and dangerous weakness of current LLMs is their tendency to produce "correct answers through flawed reasoning." This issue is far more than a minor academic flaw; such process errors signal a fundamental lack of robust understanding, making the model prone to broader hallucinations and unpredictable failures when faced with real-world clinical complexity. In this paper, we establish a framework for trustworthy clinical argumentation by adapting the Toulmin model to the diagnostic process. We propose a novel training pipeline: Curriculum Goal-Conditioned Learning (CGCL), designed to progressively train LLM to generate diagnostic arguments that explicitly follow this Toulmin structure. CGCL's progressive three-stage curriculum systematically builds a solid clinical argument: (1) extracting facts and generating differential diagnoses; (2) justifying a core hypothesis while rebutting alternatives; and (3) synthesizing the analysis into a final, qualified conclusion. We validate CGCL using T-Eval, a quantitative framework measuring the integrity of the diagnosis reasoning. Experiments show that our method achieves diagnostic accuracy and reasoning quality comparable to resource-intensive Reinforcement Learning (RL) methods, while offering a more stable and efficient training pipeline.

Chen Zhan, Xiaoyu Tan, Gengchen Ma +3

School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai, 201620, China. · Tencent Youtu Lab, Shanghai, 200232, China.

Clinical ReasoningLLM ReliabilityClinical Decision SupportCurriculum Learning
AskPDF
Apr 12, 2026

SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?

Accelerating scientific discovery requires the identification of which experiments would yield the best outcomes before committing resources to costly physical validation. While existing benchmarks evaluate LLMs on scientific knowledge and reasoning, their ability to predict experimental outcomes - a task where AI could significantly exceed human capabilities - remains largely underexplored. We introduce SciPredict, a benchmark comprising 405 tasks derived from recent empirical studies in 33 specialized sub-fields of physics, biology, and chemistry. SciPredict addresses two critical questions: (a) can LLMs predict the outcome of scientific experiments with sufficient accuracy? and (b) can such predictions be reliably used in the scientific research process? Evaluations reveal fundamental limitations on both fronts. Model accuracies are 14-26% and human expert performance is ≈\approx20%. Although some frontier models exceed human performance model accuracy is still far below what would enable reliable experimental guidance. Even within the limited performance, models fail to distinguish reliable predictions from unreliable ones, achieving only ≈\approx20% accuracy regardless of their confidence or whether they judge outcomes as predictable without physical experimentation. Human experts, in contrast, demonstrate strong calibration: their accuracy increases from ≈\approx5% to ≈\approx80% as they deem outcomes more predictable without conducting the experiment. SciPredict establishes a rigorous framework demonstrating that superhuman performance in experimental science requires not just better predictions, but better awareness of prediction reliability. For reproducibility all our data and code are provided at https://github.com/scaleapi/scipredict

Udari Madhushani Sehwag, Elaine Lau, Haniyeh Ehsani Oskouie +14

Scale AI · University of California, Los Angeles · Human Frontier Collective, Scale AI +2

AI-Assisted Scientific ResearchScientific Reasoning in Language ModelsLLM ReliabilityLLM Uncertainty Estimation
AskPDF
Apr 12, 2026

Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework

Incorporating large language models (LLMs) in medical question answering demands more than high average accuracy: a model that returns substantively different answers each time it is queried is not a reliable medical tool. Online health communities such as Reddit have become a primary source of medical information for millions of users, yet they remain highly susceptible to misinformation; deploying LLMs as assistants in these settings amplifies the need for output consistency alongside correctness. We present a practical, open-source evaluation framework for assessing small, locally-deployable open-weight LLMs on medical question answering, treating reproducibility as a first-class metric alongside lexical and semantic accuracy. Our pipeline computes eight quality metrics, including BERTScore, ROUGE-L, and an LLM-as-judge rubric, together with two within-model reproducibility metrics derived from repeated inference (N=10 runs per question). Evaluating three models (Llama 3.1 8B, Gemma 3 12B, MedGemma 1.5 4B) on 50 MedQuAD questions (N=1,500 total responses) reveals that despite low-temperature generation (T=0.2), self-agreement across runs reaches at most 0.20, while 87-97% of all outputs per model are unique -- a safety gap that single-pass benchmarks entirely miss. The clinically fine-tuned MedGemma 1.5 4B underperforms the larger general-purpose models on both quality and reproducibility; however, because MedGemma is also the smallest model, this comparison confounds domain fine-tuning with model scale. We describe the methodology in sufficient detail for practitioners to replicate or extend the evaluation for their own model-selection workflows. All code and data pipelines are available at https://github.com/aviad-buskila/llm_medical_reproducibility.

Avi-ad Avraam Buskila

Department of Information Science and Applied Artificial Intelligence, Bar-Ilan University, Ramat-Gan, Israel

LLM EvaluationLLM ReliabilityClinical Language Model EvaluationMedical LLMsBiomedical QA
AskPDF
Apr 12, 2026

VeriSim: A Configurable Framework for Stress-Testing Medical AI Under Patient Communication Noise

Medical large language models are typically evaluated on idealized patient cases that do not reflect how real patients communicate. We introduce VeriSim, a patient simulation framework that injects controllable noise along six clinically grounded communication dimensions while substantially preserving each patient's medical record. Truth adherence is supported by a verifier that extracts atomic claims from each candidate utterance and judges them against a UMLS-grounded vector index built with BioLORD embeddings, using the retrieved atoms' structured clinical metadata (e.g., drug class, anatomical site, treats-condition relations) rather than surface-text similarity alone. Across seven open-weight LLMs, realistic noise reduces diagnostic accuracy by 15-25 percentage points and increases conversation length by 34-55%; the 7-8B models degrade 1.4x more than 70B+ models. A board-certified physician and a licensed nurse rate VeriSim's conversations highly on truth, realism, clinical utility, and noise fidelity (inter-annotator agreement >= 0.80 across all dimensions), and an LLM-as-judge closely tracks their ratings, enabling scalable evaluation. We release VeriSim as an open-source framework for rigorous stress-testing of clinical AI: https://github.com/mohitmarvania/VeriSim

Sina Mansouri, Mohit Marvania, Vibhavari Ashok Shihorkar +5

Department of Computer Science, George Mason University · Department of Health Administration and Policy, College of Public Health, George Mason University · Department of Science, George Mason University

HealthcareLLM ReliabilityClinical Language Model EvaluationLanguage Model RobustnessMedical LLMs
AskPDF
Apr 11, 2026

VeriTrans: Fine-Tuned LLM-Assisted NL-to-PL Translation via a Deterministic Neuro-Symbolic Pipeline

VeriTrans is a reliability-first ML system that compiles natural-language requirements into solver-ready logic with validator-gated reliability. The pipeline integrates an instruction-tuned NL ⁣→ ⁣\!\to\!PL translator, round-trip reconstruction (PL ⁣→ ⁣\!\to\!NL) used as a high-precision acceptance gate, and canonical PL ⁣→ ⁣\!\to\!CNF compilation, all executed via fixed API configuration (temperature=0=0; fine-tuning runs use seed=42=42) and per-item artifact logging (prompts, outputs, hashes) to support auditability and replay-driven debugging. On SatBench (2{,}100 specifications), VeriTrans achieves 94.46% SAT/UNSAT correctness and 87.73% median round-trip similarity. Compact fine-tuning on 100--150 curated examples improves fidelity by about 1--1.5,pp without increasing latency (mean 25.8,s/spec on our 201-spec runtime subset). A thresholded acceptance policy on the round-trip score exposes a reliability--coverage knob: at τ=75τ{=}75, roughly 68% of items are retained with ∼\sim94% correctness on the accepted set. Validator overhead contributes <15%<15\% of end-to-end runtime, and all prompts/responses and timing metadata are logged to enable replay-driven debugging and regression testing. By separating learned translation from symbolic verification and enforcing deterministic, validator-gated acceptance, VeriTrans turns NL ⁣→ ⁣\!\to\!logic front-ends into auditable, reproducible components for reliability-critical workflows.

Xuan Liu, Dheeraj Kodakandla, Kushagra Srivastva +1

The Pennsylvania State University, University Park, Pennsylvania, USA

Boolean SatisfiabilityLLM ReliabilityNeuro-Symbolic AI
AskPDF
Apr 11, 2026

From GPT-3 to GPT-5: Mapping their capabilities, scope, limitations, and consequences

We present the progress of the GPT family from GPT-3 through GPT-3.5, GPT-4, GPT-4 Turbo, GPT-4o, GPT-4.1, and the GPT-5 family. Our work is comparative rather than merely historical. We investigates how the family evolved in technical framing, user interaction, modality, deployment architecture, and governance viewpoint. The work focuses on five recurring themes: technical progression, capability changes, deployment shifts, persistent limitations, and downstream consequences. In term of research design, we consider official technical reports, system cards, API and model documentation, product announcements, release notes, and peer-reviewed secondary studies. A primary assertion is that later GPT generations should not be interpreted only as larger or more accurate language models. Instead, the family evolves from a scaled few-shot text predictor into a set of aligned, multimodal, tool-oriented, long-context, and increasingly workflow-integrated systems. This development complicates simple model-to-model comparison because product routing, tool access, safety tuning, and interface design become part of the effective system. Across generations, several limitations remain unchanged: hallucination, prompt sensitivity, benchmark fragility, uneven behavior across domains and populations, and incomplete public transparency about architecture and training. However, the family has evolved software development, educational practice, information work, interface design, and discussions of frontier-model governance. We infer that the transition from GPT-3 to GPT-5 is best understood not only as an improvement in model capability, but also as a broader reformulation of what a deployable AI system is, how it is evaluated, and where responsibility should be located when such systems are used at scale.

Hina Afridi, Habib Ullah, Sultan Daud Khan +1

MediaFutures, Department of Information Science and Media Studies, University of Bergen, Bergen, Norway. · Faculty of Science and Technology, Norwegian University of Life Sciences, Ås, Norway. · Faculty of Computing and Information Technology, Sohar University, Sohar, Oman. +2

Language ModelingGenerative AI EvaluationLLM Tool UseLLM ReliabilityMultimodal Large Language Models
AskPDF
Apr 11, 2026

Like a Hammer, It Can Build, It Can Break: Large Language Model Uses, Perceptions, and Adoption in Cybersecurity Operations on Reddit

Large language models (LLMs) have recently emerged as promising tools for augmenting Security Operations Center (SOC) workflows, with vendors increasingly marketing autonomous AI solutions for SOCs. However, there remains a limited empirical understanding of how such tools are used, perceived, and adopted by real-world security practitioners. To address this gap, we conduct a mixed-methods analysis of discussions in cybersecurity-focused forums to learn how a diverse group of practitioners use and perceive modern LLM tools for security operations. More specifically, we analyzed 892 posts between December 2022 and September 2025 from three cybersecurity-focused forums on Reddit, and, using a combination of qualitative coding and statistical analysis, examined how security practitioners discuss LLM tools across three dimensions: (1) their stated tools and use cases, (2) the perceived pros and cons of each tool across a set of critical factors, and (3) their adoption of such tools and the expected impacts on the cybersecurity industry and individual analysts. Overall, our findings reveal nuanced patterns in LLM tools adoption, highlighting independent use of LLMs for low-risk, productivity-oriented tasks, alongside active interest around enterprise-grade, security-focused LLM platforms. Although practitioners report meaningful gains in efficiency and effectiveness in LLM-assisted workflows, persistent issues with reliability, verification overheads, and security risks sharply constrain the autonomy granted to LLM tools. Based on these results, we also provide recommendations for developing and adopting LLM tools to ensure the security of organizations and the safety of cybersecurity practitioners.

Souradip Nath, Chih-Yi Huang, Aditi Ganapathi +3

Arizona State University · Technische Universität Berlin

LLMs for CybersecurityCybersecurityLLM Reliability
AskPDF
Apr 10, 2026

Building Trust in the Skies: A Knowledge-Grounded LLM-based Framework for Aviation Safety

The integration of Large Language Models (LLMs) into aviation safety decision-making represents a significant technological advancement, yet their standalone application poses critical risks due to inherent limitations such as factual inaccuracies, hallucination, and lack of verifiability. These challenges undermine the reliability required for safety-critical environments where errors can have catastrophic consequences. To address these challenges, this paper proposes a novel, end-to-end framework that synergistically combines LLMs and Knowledge Graphs (KGs) to enhance the trustworthiness of safety analytics. The framework introduces a dual-phase pipeline: it first employs LLMs to automate the construction and dynamic updating of an Aviation Safety Knowledge Graph (ASKG) from multimodal sources. It then leverages this curated KG within a Retrieval-Augmented Generation (RAG) architecture to ground, validate, and explain LLM-generated responses. The implemented system demonstrates improved accuracy and traceability over LLM-only approaches, effectively supporting complex querying and mitigating hallucination. Results confirm the framework's capability to deliver context-aware, verifiable safety insights, addressing the stringent reliability requirements of the aviation industry. Future work will focus on enhancing relationship extraction and integrating hybrid retrieval mechanisms.

Anirudh Iyengar, Alisa Tiselska, Dumindu Samaraweera +1

Department of Mathematics, Embry-Riddle Aeronautical University, Daytona Beach, FL

Aviation SafetyLLM ReliabilityLLM Hallucination MitigationKG Construction
AskPDF
Apr 9, 2026

HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy

Cross-document relation extraction (RE) aims to identify relations between the head and tail entities located in different documents. Existing approaches typically adopt the paradigm of ``Small Language Model (SLM) + Classifier''. However, the limited language understanding ability of SLMs hinders further improvement of their performance. In this paper, we conduct a preliminary study to explore the performance of Large Language Models (LLMs) in cross-document RE. Despite their extensive parameters, our findings indicate that LLMs do not consistently surpass existing SLMs. Further analysis suggests that the underperformance is largely attributed to the challenges posed by the numerous predefined relations. To overcome this issue, we propose an LLM-based Hierarchical Classification model for cross-document RE (HCRE), which consists of two core components: 1) an LLM for relation prediction and 2) a hierarchical relation tree derived from the predefined relation set. This tree enables the LLM to perform hierarchical classification, where the target relation is inferred level by level. Since the number of child nodes is much smaller than the size of the entire predefined relation set, the hierarchical relation tree significantly reduces the number of relation options that LLM needs to consider during inference. However, hierarchical classification introduces the risk of error propagation across levels. To mitigate this, we propose a prediction-then-verification inference strategy that improves prediction reliability through multi-view verification at each level. Extensive experiments show that HCRE outperforms existing baselines, validating its effectiveness.

Guoqi Ma, Liang Zhang, Hongyao Tu +6

School of Informatics, Xiamen University · Li Auto Inc. · Alibaba International Digital Commerce Group

Relation ExtractionHierarchical ClassificationLLM Reliability
AskPDF
Apr 8, 2026

Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMs

Hard-gated safety checkers often over-refuse and misalign with a vendor's model spec; prevailing taxonomies also neglect robustness and honesty, yielding safer-on-paper yet less useful systems. This work introduces Guardian-as-an-Advisor (GaaA), a soft-gating pipeline where a guardian predicts a binary risk label plus a concise explanation and prepends this advice to the original query for re-inference, keeping the base model operating under its original spec. To support training and evaluation, GuardSet is constructed, a 208k+ multi-domain dataset unifying harmful and harmless cases with targeted robustness and honesty slices. GuardAdvisor is trained via SFT followed by RL to enforce label-explanation consistency. GuardAdvisor attains competitive detection accuracy while enabling the advisory workflow; when used to augment inputs, responses improve over unaugmented prompts. A latency study shows advisor inference uses below 5% of base-model compute and adds only 2-10% end-to-end overhead under realistic harmful-input rates. Overall, GaaA steers models to comply with the model spec, maintaining safety while reducing over-refusal.

Yue Huang, Haomin Zhuang, Jiayi Ye +6

University of Notre Dame · University of California, Los Angeles · MIT-IBM Watson AI Lab +1

LLM GuardrailsLLM Safety AlignmentLLM ReliabilityAI EthicsLanguage Model RobustnessLLM Safety Benchmarks
AskPDF
Apr 8, 2026

AGSC: Adaptive Granularity and Semantic Clustering for Uncertainty Quantification in Long-text Generation

Large Language Models (LLMs) have demonstrated impressive capabilities in long-form generation, yet their application is hindered by the hallucination problem. While Uncertainty Quantification (UQ) is essential for assessing reliability, the complex structure makes reliable aggregation across heterogeneous themes difficult, in addition, existing methods often overlook the nuance of neutral information and suffer from the high computational cost of fine-grained decomposition. To address these challenges, we propose AGSC (Adaptive Granularity and GMM-based Semantic Clustering), a UQ framework tailored for long-form generation. AGSC first uses NLI neutral probabilities as triggers to distinguish irrelevance from uncertainty, reducing unnecessary computation. It then applies Gaussian Mixture Model (GMM) soft clustering to model latent semantic themes and assign topic-aware weights for downstream aggregation. Experiments on BIO and LongFact show that AGSC achieves state-of-the-art correlation with factuality while reducing inference time by about 60% compared to full atomic decomposition.

Guanran Luo, Wentao Qiu, Wanru Zhao +4

School of Informatics, Xiamen University

Semantic ClusteringUncertainty QuantificationLLM Uncertainty EstimationLLM ReliabilityLanguage Model Generation Evaluation
AskPDF
Apr 8, 2026

The Illusion of Stochasticity in LLMs

In this work, we demonstrate that reliable stochastic sampling is a fundamental yet unfulfilled requirement for Large Language Models (LLMs) operating as agents. Agentic systems are frequently required to sample from distributions, often inferred from observed data, a process which needs to be emulated by the LLM. This leads to a distinct failure point: while standard RL agents rely on external sampling mechanisms, LLMs fail to map their internal probability estimates to their stochastic outputs. Through rigorous empirical analysis across multiple model families, model sizes, prompting styles, and distributions, we demonstrate the extent of this failure. Crucially, we show that while powerful frontier models can convert provided random seeds to target distributions, their ability to sample directly from specific distributions is fundamentally flawed.

Xiangming Gu, Soham De, Michalis Titsias +3

Google DeepMind · National University of Singapore

LLM ReliabilityLLM Agent Reliability
AskPDF