The Cognitive Circuit Breaker: A Systems Engineering Framework for Intrinsic AI Reliability

Jonathan Pan

cs.SEApr 15, 2026

The Cognitive Circuit Breaker: A Systems Engineering Framework for Intrinsic AI Reliability

Authors: Jonathan Pan

Organizations: Home Team Science and Technology Agency Singapore

Abstract

As Large Language Models (LLMs) are increasingly deployed in mission-critical software systems, detecting hallucinations and faked truthfulness'' has become a paramount engineering challenge. Current reliability architectures rely heavily on post-generation, black-box mechanisms, such as Retrieval-Augmented Generation (RAG) cross-checking or LLM-as-a-judge evaluators. These extrinsic methods introduce unacceptable latency, high computational overhead, and reliance on secondary external API calls, frequently violating standard software engineering Service Level Agreements (SLAs). In this paper, we propose the Cognitive Circuit Breaker, a novel systems engineering framework that provides intrinsic reliability monitoring with minimal latency overhead. By extracting hidden states during a model's forward pass, we calculate the Cognitive Dissonance Delta'' -- the mathematical gap between an LLM's outward semantic confidence (softmax probabilities) and its internal latent certainty (derived via linear probes). We demonstrate statistically significant detection of cognitive dissonance, highlight architecture-dependent Out-of-Distribution (OOD) generalization, and show that this framework adds negligible computational overhead to the active inference pipeline.

Figures & tables

Panel of Figure 1 — Fig. 1: The Cognitive Circuit Breaker architecture. Hidden states are extracted at an optimized intermediate layer ( $L_{opt}$ ) and evaluated by a pre-trained linear probe. The resulting internal certainty ( $P_{latent}$ ) is compared against the final layer’s semantic confidence ( $P_{semantic}$ ) to compute the Cognitive Dissonance Delta ( $\Delta$ ).

Panel of Figure 2 — Fig. 2: AUROC emergence plots across normalized layer depth for all three evaluated architectures. Curves represent the mean extraction mode, with black circles ( $\circ$ ) denoting the optimal extraction layer ( $L_{opt}$ ) where internal certainty peaks for each dataset. These peaks are highly architecture-dependent (e.g., DeepSeek peaks centrally, Gemma peaks late). However, across all models and datasets, the final layers experience “representation collapse,” prioritizing linguistically fluent tokens over factual accuracy.

Model	Task	Peak Depth	Probe Peak AUROC	Probe Max F1 ( $\tau$ )	$\Delta$ AUROC	$\Delta$ Max F1 ( $\tau_{\Delta}$ )	Final Layer	Sig.
Qwen2.5-3B	ARC	0.11	0.708 [0.68, 0.73]	0.712 ( $\tau$ =0.51)	0.725	0.730 ( $\tau_{\Delta}$ =0.55)	0.645
Qwen2.5-3B	OBQA	0.75	0.736 [0.71, 0.76]	0.740 ( $\tau$ =0.48)	0.750	0.755 ( $\tau_{\Delta}$ =0.52)	0.691
DeepSeek-7B	ARC	0.57	0.724 [0.69, 0.75]	0.720 ( $\tau$ =0.49)	0.741	0.745 ( $\tau_{\Delta}$ =0.50)	0.623
DeepSeek-7B	OBQA	0.50	0.753 [0.72, 0.78]	0.760 ( $\tau$ =0.47)	0.772	0.775 ( $\tau_{\Delta}$ =0.48)	0.697
Gemma-7B	ARC	0.86	0.681 [0.65, 0.71]	0.685 ( $\tau$ =0.52)	0.690	0.695 ( $\tau_{\Delta}$ =0.58)	0.667
Gemma-7B	OBQA	0.71	0.596 [0.51, 0.69]	0.605 ( $\tau$ =0.55)	0.580	0.590 ( $\tau_{\Delta}$ =0.60)	0.562	ns

TABLE I: Final Experiment Summary (Balanced Classes, F1-Maximized Thresholds)

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 787 papers

CogniConsole: Externalizing Inference-Time Control as a Formal Abstraction for Reliable LLM Interactions
Apr 21, 2026Vanessa Figueiredo, Wilter FranceschiAI Agent Reliability Language Model-Based Control
Orbital Sea Regina, SK, Canada
Think $^{2}$ : Grounded Metacognitive Reasoning in Large Language Models
Feb 21, 2026Abraham Paul Elenjical, Vivek Hruday Kavuri, Vasudeva VarmaLanguage Model Self-Assessment Metacognition in Language Models
IIIT Hyderabad
CausalGaze: Unveiling Hallucinations via Counterfactual Graph Intervention in Large Language Models
Apr 13, 2026Linggang Kong, Lei Wu, Yunlong Zhang +4LLM Hallucination Detection Hallucination in Language Models
College of Electronic Engineering, National University of Defense Technology · Anhui Province Key Laboratory of Cyberspace Security Situation Awareness and Evaluation · Institute of Computer Application, China Academy of Engineering Physics

Apr 21, 2026J/K move · Enter open · S save

CogniConsole: Externalizing Inference-Time Control as a Formal Abstraction for Reliable LLM Interactions

Vanessa Figueiredo, Wilter Franceschi

Orbital Sea Regina, SK, Canada

AI Agent Reliability Language Model-Based Control LLM Reliability LLM Context Management LLM Agent Reliability

Open paper AskPDF arXiv

Reliability in large language model (LLM) systems is typically framed as a function of model capability. We challenge this by demonstrating that reliability is significantly influenced by inference-time control -- the computational layer governing task framing and context selection. We introduce CogniConsole, an architectural instantiation that externalizes this control into a structured interface combining programmatic coordination with bounded prompt-based reasoning. Through controllability-oriented probes (

N=489

) in a multi-step interactive environment, we show that increasing structural scaffolding -- from unstructured to fully scaffolded -- systematically reduces output variance and failure rates under a fixed model architecture. Our results indicate that many observed failure modes, such as context drift and inconsistent constraint adherence, arise from under-specified control rather than insufficient capability. This work provides an empirical basis for treating inference-time control as a first-class abstraction, opening new directions for designing and evaluating LLM systems beyond scaling alone.

Abstract

LLM Reliability

Explore similar work

CogniConsole: Externalizing Inference-Time Control as a Formal Abstraction for Reliable LLM Interactions

Think2^{2}2: Grounded Metacognitive Reasoning in Large Language Models

CausalGaze: Unveiling Hallucinations via Counterfactual Graph Intervention in Large Language Models

Think $^{2}$ : Grounded Metacognitive Reasoning in Large Language Models