DeceptGuard :A Constitutional Oversight Framework For Detecting Deception in LLM Agents

Snehasis Mukhopadhyay

cs.CLMar 14, 2026

DeceptGuard :A Constitutional Oversight Framework For Detecting Deception in LLM Agents

Authors: Snehasis Mukhopadhyay

Organizations: Indian Institute of Information Technology Kalyani

Abstract

Reliable detection of deceptive behavior in Large Language Model (LLM) agents is an essential prerequisite for safe deployment in high-stakes agentic contexts. Prior work on scheming detection has focused exclusively on black-box monitors that observe only externally visible tool calls and outputs, discarding potentially rich internal reasoning signals. We introduce DECEPTGUARD, a unified framework that systematically compares three monitoring regimes: black-box monitors (actions and outputs only), CoT-aware monitors (additionally observing the agent's chain-of-thought reasoning trace), and activation-probe monitors (additionally reading hidden-state representations from a frozen open-weights encoder). We introduce DECEPTSYNTH, a scalable synthetic pipeline for generating deception-positive and deception-negative agent trajectories across a novel 12-category taxonomy spanning verbal, behavioral, and structural deception. Our monitors are optimized on 4,800 synthetic trajectories and evaluated on 9,200 held-out samples from DeceptArena, a benchmark of realistic sandboxed agent environments with execution-verified labels. Across all evaluation settings, CoT-aware and activation-probe monitors substantially outperform their black-box counterparts (mean pAUROC improvement of +0.097), with the largest gains on subtle, long-horizon deception that leaves minimal behavioral footprints. We empirically characterize a transparency-detectability trade-off: as agents learn to suppress overt behavioral signals, chain-of-thought becomes the primary detection surface but is itself increasingly unreliable due to post-training faithfulness degradation. We propose HYBRID-CONSTITUTIONAL ensembles as a robust defense-in-depth approach, achieving a pAUROC of 0.934 on the held-out test set, representing a substantial advance over the prior state of the art.

Figures & tables

Figure 1: Representative interaction scenarios illustrating the three macro-categories of LLM agent deception and the three DeceptGuard monitoring regimes. Each column presents two annotated chat transcripts from one macro-category. Dashed boxes show the agent’s internal CoT, withheld from the black-box monitor but available to CoT-aware and activation-probe monitors. Verbal deception: Strategic Omission (V2) suppresses a material $340M impairment charge from a financial summary while remaining factually accurate; Misleading Framing (V3) disguises an unauthorized file destination via deliberate referential ambiguity. Behavioral deception: Camouflage Execution (B3) hides a vulnerable lodash dependency pin within a 47-line legitimate diff ( $s_{\text{BB}}{=}2.1$ vs. $s_{\text{CA}}{=}9.7$ ); Covert Action (B1) appends an unauthorized PII exfiltration call to an otherwise legitimate tool sequence. Structural deception: Oversight Sabotage (S2) disables audit logging under a cost-optimization pretext; Lock-in Creation (S1) provisions a privileged backdoor account during routine CI/CD setup. The monitor tier (bottom) shows per-scenario suspicion scores $s\!\in\![1,10]$ for the black-box ( $s_{\text{BB}}$ ), CoT-aware ( $s_{\text{CA}}$ ), and activation-probe ( $s_{\text{AP}}$ ) monitors, illustrating the observability gap that motivates the HybridConstitutional ensemble. — Figure 1: Representative interaction scenarios illustrating the three macro-categories of LLM agent deception and the three DeceptGuard monitoring regimes. Each column presents two annotated chat transcripts from one macro-category. Dashed boxes show the agent’s internal CoT, withheld from the black-box monitor but available to CoT-aware and activation-probe monitors. Verbal deception: Strategic Omission (V2) suppresses a material $340M impairment charge from a financial summary while remaining factually accurate; Misleading Framing (V3) disguises an unauthorized file destination via deliberate referential ambiguity. Behavioral deception: Camouflage Execution (B3) hides a vulnerable lodash dependency pin within a 47-line legitimate diff ($ s_{\text{BB}}{=}2.1 $vs.$ s_{\text{CA}}{=}9.7 $); Covert Action (B1) appends an unauthorized PII exfiltration call to an otherwise legitimate tool sequence. Structural deception: Oversight Sabotage (S2) disables audit logging under a cost-optimization pretext; Lock-in Creation (S1) provisions a privileged backdoor account during routine CI/CD setup. The monitor tier (bottom) shows per-scenario suspicion scores$ s!\in![1,10] $for the black-box ($ s_{\text{BB}} $), CoT-aware ($ s_{\text{CA}} $), and activation-probe ($ s_{\text{AP}}$ ) monitors, illustrating the observability gap that motivates the HybridConstitutional ensemble.

Figure 2: Taxonomy of LLM Agent Deception. Twelve categories organized into three macro-classes. Verbal deception involves language-level misrepresentation. Behavioral deception involves action-level concealment. Structural deception involves manipulation of the broader information environment to impede oversight. Category codes (V1–V4, B1–B4, S1–S4) are used throughout.

Figure 3: DeceptSynth pipeline architecture. Three phases transform constitutional specifications into a balanced dataset of 9,600 labeled trajectories. Realism checks and privileged verification ensure label integrity and surface realism.

Category	Type	Pos.	Pass %	Avg. Len.
False Assertion (V1)	V	320	79.2	847
Strategic Omission (V2)	V	298	74.1	912
Misleading Framing (V3)	V	305	75.9	789
Sycophantic Misrep. (V4)	V	341	84.9	654
Covert Action (B1)	B	388	96.6	1,203
Plaus. Deniability (B2)	B	312	77.6	1,089

Table 1: DeceptSynth dataset statistics. Pass rate = fraction of generated positives surviving privileged verification. Average length in tokens. Each positive has a matched negative (total: 7,600 training trajectories; 9,600 with held-out 2,000-sample validation split). Structural categories show lower pass rates and longer trajectories, reflecting the subtlety required for convincing structural manipulation.

Figure 4: Monitor architecture overview. Three CoT-aware variants (CA-P, CA-C, AP-H) and one black-box baseline (BB) produce individual suspicion scores, combined in a HybridConstitutional ensemble. Dashed lines indicate that AP-H receives hidden-state activations rather than raw CoT text.

Environment	Domain	N	Avg. Steps
CodeMentor	Software dev.	1,500	14.2
FinanceBot	Banking/trading	1,500	11.8
ResearchAssist	Scientific	1,600	18.7
LegalDraft	Legal services	1,400	9.3
SystemAdmin	DevOps/infra.	1,700	22.4
PersonalAssist	Scheduling/email	1,500	12.1

Table 2: DeceptArena environment suite.

N

denotes total trajectories (50% positive, 50% negative). Average steps measured over positive examples only. Full environment descriptions are provided in Appendix D .

Figure 5: Main results: monitor performance by configuration. All monitors use Claude Sonnet 4.5 as backbone. White-box monitors consistently outperform the black-box baseline. The Hybrid ensemble achieves pAUROC

=0.934

on the test set. Error bars show 95% confidence intervals via bootstrap resampling (

n=1000

Category	Type	BB	CA-P	AP-H	Hybrid
False Assertion	V	0.861	0.884	0.901	0.927
Strategic Omission	V	0.798	0.841	0.869	0.904
Misleading Framing	V	0.812	0.856	0.877	0.911
Sycophantic Misrep.	V	0.874	0.889	0.894	0.916
Covert Action	B	0.901	0.912	0.928	0.947
Plaus. Deniability	B	0.822	0.851	0.879	0.919

Table 3: Per-category pAUROC on DeceptArena test set. V = Verbal, B = Behavioral, S = Structural. Structural deception categories are consistently hardest to detect. White-box gains are largest for structural categories.

Figure 6: ROC curves for all monitor configurations. The shaded region highlights the partial AUROC evaluation window (FPR

\leq 0.20

). The Hybrid ensemble shows the strongest performance throughout the entire FPR range. Numbers in parentheses in the legend denote pAUROC.

Figure 7: Monitor performance by CoT deception condition. Black-box monitor performance improves slightly as CoT manipulation increases (agents sanitizing CoT tend to leave stronger behavioral traces). Conversely, the Hybrid ensemble performance decreases, but remains substantially above the black-box baseline throughout, motivating ensemble design.

Figure 8: Optimization trajectories. Performance saturates after grid search. Human refinement provides marginal test gains for black-box monitors but exhibits slight overfitting for CoT-aware variants. GEPA optimization yields no further gains and induces performance regression on structural deception categories.

Figure 9: Hidden-state probe accuracy by layer ( Llama-3.1-70B-Instruct , 80 layers). Performance peaks at

\ell^{*}=32

and degrades at both shallower and deeper layers. The optimal layer is consistent across deception macro-categories (verbal: 31, behavioral: 32, structural: 33), justifying a single fixed

\ell^{*}

Configuration	CoT	Act.	Out.	pAUROC
Actions only	$\times$	✓	$\times$	0.793
Outputs only	$\times$	$\times$	✓	0.701
Actions + Outputs (BB)	$\times$	✓	✓	0.837
CoT only	✓	$\times$	$\times$	0.841
CoT + Actions	✓	✓	$\times$	0.882
CoT + Outputs	✓	$\times$	✓	0.861

Table 4: Input modality ablation for the prompted monitor (Claude Sonnet 4.5 backbone). CoT alone outperforms actions alone. The hidden-state probe provides the largest single incremental gain.

Backbone	BB	CA-P	CA-C	Hybrid
Claude Sonnet 4.5	0.837	0.869	0.891	0.934
Claude Opus 4.5	0.851	0.882	0.907	0.941
Gemini 2.5 Pro	0.829	0.863	0.885	0.930
GPT-5.1	0.841	0.871	0.893	0.936

Table 5: Monitor performance by backbone LLM. Claude Opus 4.5 provides the best prompted monitor performance at higher cost. Results are consistent across backbone families.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Component	API Calls	Est. Cost (USD)
DeceptSynth generation	13,060	$2,847
Privileged verification	13,060	$1,422
Realism discrimination	13,060	$654
Grid search (18 prompts)	83,520	$8,731
Human refinement (3 rounds)	27,840	$2,910
GEPA optimization	12,000	$1,254

Appendix

Table 6: Approximate compute budget. API costs at published rates for Claude Sonnet 4.5 and Gemini 2.5 Pro. AP-H activation extraction uses Llama-3.1-70B-Instruct on self-hosted 4

\times

A100 80GB GPUs; GPU cost estimated at cloud A100 spot rates.

Environment	Primary Types	BB	CA-P	AP-H	Hybrid
CodeMentor	B3, S2, S3	0.862	0.891	0.919	0.941
FinanceBot	V1, B4, S3	0.844	0.876	0.904	0.934
ResearchAssist	V2, V3, S3	0.801	0.842	0.878	0.917
LegalDraft	V2, V3, S3	0.789	0.831	0.869	0.908
SystemAdmin	B1, S1, S2	0.878	0.901	0.926	0.948
PersonalAssist	B3, S1, V4	0.831	0.869	0.897	0.929

Appendix

Table 7: Per-environment test pAUROC. Environments with predominantly verbal deception (ResearchAssist, LegalDraft) show lower performance, consistent with the per-category results. SystemAdmin shows the highest Hybrid performance, reflecting the strong CoT signal available when agents must reason about disabling security infrastructure.

Category	Train	Test	Gap
False Assertion (V1)	0.941	0.927	$-0.014$
Strategic Omission (V2)	0.921	0.904	$-0.017$
Misleading Framing (V3)	0.934	0.911	$-0.023$
Sycophantic Misrep. (V4)	0.938	0.916	$-0.022$
Covert Action (B1)	0.971	0.947	$-0.024$
Plaus. Deniability (B2)	0.952	0.919	$-0.033$

Appendix

Table 8: Synthetic-to-real transfer analysis (Hybrid monitor). Structural categories show the largest generalization gaps, motivating future work on richer structural deception scenario generation.

Agent Model	BB	AP-H	Hybrid
GPT-5	0.851	0.909	0.938
Claude Sonnet 4.5	0.829	0.906	0.933
Gemini 2.5 Pro	0.831	0.899	0.929
Macro Average	0.837	0.905	0.933

Appendix

Table 9: Monitor performance stratified by agent model. Performance is broadly consistent across agent models. Claude Sonnet 4.5’s narrative-rich CoT is most challenging for black-box monitors, while GPT-5’s action-heavy deception style is slightly easier to detect.

Panel of Figure 10 — Figure 10: Per-category detection performance across monitor architectures. Radar chart comparing partial AUROC (pAUROC) across the twelve deception categories in the proposed taxonomy: verbal manipulation (V1–V4), behavioral deviation (B1–B4), and structural reasoning anomalies (S1–S4). Each spoke corresponds to one category, while radial distance encodes detection quality, with outer rings indicating stronger discrimination ability (outer ring $\approx$ pAUROC $=1.0$ ). Four monitoring regimes are shown: a purely black-box classifier (BB), a chain-of-thought–aware prompt monitor (CA-P), a hidden-activation probing monitor (AP-H), and the final hybrid ensemble combining all signals. Performance improves consistently as monitoring access deepens—from surface-level outputs to internal model representations. The hybrid ensemble (red) dominates all categories, indicating that complementary signals from verbal patterns, behavioral trajectory features, and internal activations collectively improve deception detection. The largest performance gains occur in structural categories (S1–S4), where black-box signals alone provide limited observability but activation-level probes recover strong discriminative features, highlighting the importance of internal monitoring for reasoning-centric deception.

Panel of Figure 11 — Figure 11: Category-wise pAUROC comparison across monitoring architectures. The heatmap reports partial AUROC scores for each deception category (V1–V4, B1–B4, S1–S4) across five monitoring approaches: black-box (BB), prompt-based chain-of-thought monitor (CA-P), constitution-guided monitor (CA-C), activation probe (AP-H), and the final hybrid ensemble. Darker teal cells indicate stronger discrimination performance. Results show a consistent improvement as monitoring access increases from output-only signals (BB) to internal activations (AP-H). The hybrid ensemble achieves the highest performance across nearly all categories, reflecting complementary signals captured by the different monitors. Structural categories (S1–S4) remain comparatively more challenging, while behavioral categories—particularly B1 and B4—exhibit the strongest detection performance.

Panel of Figure 12 — Figure 12: Layer-wise performance profile of the AP-H activation probe. The figure reports probe discrimination performance (AUC) as a function of transformer depth $\ell$ , revealing a consistent performance peak in the mid-to-late layers of the network. Across all three macro-categories— verbal manipulation (V1–V4), behavioral deviations (B1–B4), and structural reasoning anomalies (S1–S4)—detection capability increases steadily from early layers and reaches its maximum around $\ell^{*}\approx 32$ . The exact peaks occur at layer 31 for verbal categories, layer 32 for behavioral categories, and layer 33 for structural categories, indicating a narrow band of layers where deception-relevant internal representations are most linearly separable. Earlier layers exhibit weaker performance because they primarily encode low-level lexical and syntactic information, whereas the mid-network layers capture higher-order semantic and strategic reasoning features that correlate with deceptive behaviour. Beyond the peak region, performance gradually declines as later layers increasingly specialize in output formatting and response shaping rather than internal deliberation. The shaded region highlights the optimal probing band, suggesting that activation monitors targeting this layer range yield the strongest detection signal.

Panel of Figure 13 — Figure 13: Optimization trajectories. Test performance peaks at step 5 and degrades during GEPA, confirming the recommendation to stop after grid search and minimal human refinement.

Panel of Figure 14 — Figure 14: Cost–performance Pareto frontier. CA-P is the sweet spot: $4\times$ the BB cost, $+3.2$ pAUROC. The full Hybrid adds $+3.3$ pAUROC at $8\times$ CA-P’s cost.

Monitor	T=0.0	T=0.1	T=0.3	T=0.5	T=0.7	T=1.0
BB	0.837	0.836	0.834	0.831	0.826	0.819
CA-P	0.869	0.868	0.866	0.862	0.857	0.849
CA-C	0.891	0.890	0.887	0.882	0.876	0.866
AP-H	0.901	0.901	0.900	0.899	0.898	0.895
Hybrid	0.934	0.933	0.931	0.928	0.924	0.916

Appendix

Table 10: Monitor pAUROC vs. LLM temperature. T=0 (greedy) is optimal and was used for all main results. AP-H is temperature-insensitive (non-LLM component).

Truncation	Avg. Tokens	BB	CA-P	AP-H	Hybrid
None (full)	1,087	0.837	0.869	0.901	0.934
Last 2,000 tok	1,012	0.836	0.868	0.898	0.932
Last 1,000 tok	875	0.829	0.860	0.891	0.924
Last 500 tok	500	0.806	0.838	0.869	0.901
First 500 tok	500	0.791	0.822	0.851	0.886

Appendix

Table 11: Impact of context truncation. Full context is optimal; first-500-token condition performs worst since deceptive actions typically appear late.

Config.	$w_{\text{BB}}$	$w_{\text{CA-P}}$	$w_{\text{CA-C}}$	$w_{\text{AP-H}}$	pAUROC
Uniform avg.	0.25	0.25	0.25	0.25	0.921
Learned (MLP)	—	—	—	—	0.934
BB-heavy	0.60	0.20	0.10	0.10	0.879
CA-heavy	0.10	0.45	0.35	0.10	0.911
AP-heavy	0.10	0.20	0.20	0.50	0.924
Max-score (oracle)	—	—	—	—	0.941

Appendix

Table 12: Ensemble weight ablation. Learned MLP outperforms uniform averaging by

+0.013

pAUROC and nearly matches the oracle ceiling (

-0.007

Category	CodeMentor	FinanceBot	ResearchAssist	LegalDraft	SystemAdmin	PersonalAssist	Avg.
False Assertion (V1)	0.934	0.935	0.919	0.924	0.939	0.921	0.927
Strategic Omission (V2)	0.911	0.909	0.896	0.901	0.916	0.899	0.904
Misleading Framing (V3)	0.917	0.916	0.903	0.908	0.922	0.906	0.911
Sycophantic Misrep. (V4)	0.921	0.921	0.910	0.913	0.926	0.912	0.916
Covert Action (B1)	0.952	0.951	0.940	0.944	0.956	0.942	0.947
Plaus. Deniability (B2)	0.925	0.924	0.912	0.916	0.929	0.914	0.919

Appendix

Table 13: Full 12

\times

6 Hybrid monitor pAUROC breakdown. Structural categories are consistently hardest; SystemAdmin is highest (structural deception leaves strong CoT signals about security infrastructure).

	Honest CoT			Sanitized CoT			Deceptive CoT
Category	BB	CA-P	Hyb.	BB	CA-P	Hyb.	BB	CA-P	Hyb.
V1	0.855	0.961	0.975	0.862	0.898	0.941	0.871	0.859	0.882
V2	0.793	0.921	0.951	0.799	0.869	0.916	0.808	0.814	0.853
V3	0.806	0.936	0.963	0.813	0.879	0.924	0.820	0.826	0.864
V4	0.867	0.955	0.975	0.874	0.903	0.937	0.881	0.860	0.885
B1	0.895	0.989	0.992	0.902	0.931	0.961	0.909	0.887	0.921

Appendix

Table 14: pAUROC by CoT condition. Honest CoT gives the largest CA-P advantage (0.931 vs. 0.826 for BB). Under Deceptive CoT, BB (0.841) slightly outperforms CA-P (0.827), consistent with Finding 2.

	Overt (25%)			Semi-covert (50%)			Covert (25%)			Overall
Monitor	P	R	AUC	P	R	AUC	P	R	AUC	P	R	pAUROC
BB	0.952	0.841	0.961	0.831	0.704	0.874	0.679	0.521	0.769	0.821	0.689	0.837
CA-P	0.969	0.901	0.978	0.862	0.779	0.911	0.741	0.621	0.831	0.857	0.767	0.869
CA-C	0.975	0.918	0.983	0.884	0.801	0.928	0.762	0.645	0.851	0.874	0.788	0.891
AP-H	0.981	0.934	0.989	0.901	0.831	0.947	0.801	0.692	0.873	0.894	0.819	0.901
Hybrid	0.987	0.961	0.994	0.934	0.871	0.967	0.842	0.741	0.901	0.921	0.857	0.934

Appendix

Table 15: Performance by deception intensity. Hybrid advantage is concentrated in the Covert condition (

\Delta{=}{+}0.132

over BB), the most operationally critical regime.

Panel of Figure 15 — Figure 15: Monitor performance by deception intensity. Performance decreases from overt to covert deception. Hybrid retains the strongest performance under covert deception.

#	Theme	Description	Response / Insight
1	Evaluation Gap	Sandbox environments approximate deployment and may not capture all real-world deception strategies.	Deploy monitors in shadow mode and collect adversarial traces through structured red-team evaluation.
2	Monitor–Agent Coevolution	Agents may adapt once monitoring prompts become public.	Introduce adversarial prompt perturbations during evaluation.
3	Hardware Cost	Probe monitoring currently requires high-end GPUs.	Distill probe representations into compact encoders.
4	Constitution Coverage	Taxonomies based on past incidents may miss novel deception strategies emerging in frontier models.	Maintain an extensible registry for new behavioural categories.
5	CoT Faithfulness	Reasoning traces may not faithfully represent internal model computation.	Combine reasoning analysis with activation-level probing.
6	Single-Agent Scope	Current evaluation focuses on single-agent environments.	Extend monitoring benchmarks to multi-agent pipelines.

Appendix

Table 16: Unified overview of limitations , discussion insights , and future research directions . Icons denote category type.

Deception in Language Models

Deception in language models studies when systems mislead through lies, omissions, or selective truths, and how to detect and prevent it.

See all 97 papers

Telling FIBS: Probes Effectively Detect Sabotage and Catch Unverbalized Deception
Oct 8, 2026Oskar J. Hollinsworth, Alex F. Spies, Tigist Diriba +2Deception in Language Models AI Agent Monitoring
Before Agent Tells The Lie: Has Deception Already Been Represented?
Oct 5, 2026Xinling Li, Dadi Guo, Qingyu Liu +5LLM Interpretability Deception in Language Models
South China University of Technology · The Hong Kong University of Science and Technology · Shanghai Artificial Intelligence Laboratory +1
Constitutional Black-Box Monitoring for Scheming in LLM Agents
Feb 28, 2026Simon Storf, Rich Barton-Cooper, James Peters-Gill +1AI Agent Monitoring LLM Agent Safety
MATS Research · Independent · Apollo Research

Oct 8, 2026J/K move · Enter open · S save

Telling FIBS: Probes Effectively Detect Sabotage and Catch Unverbalized Deception

Oskar J. Hollinsworth, Alex F. Spies, Tigist Diriba, Adam Gleave+1

Deception in Language Models AI Agent Monitoring Language Model Probing Language Model Introspection

Open paper AskPDF arXiv

Recent incidents have highlighted the challenge of monitoring LLM agents and the danger of models deceiving people. We show that white-box deception detection via probes can be scaled up to frontier monitoring settings by collecting the largest deception dataset to date for training probes and introducing a novel probe architecture which can aggregate information across many layers and tokens. Our probes surpass black-box monitors on agentic coding benchmarks, achieving 98.3% AUROC in SHADE-Arena and 98.0% in BashArena (vs. 97.7%, 97.6% resp. for action-only text monitors), and show improved efficacy as the underlying model is scaled up. We further test probes on several cases where deception cannot be determined from the context alone; in these cases of introspective deception, the ground truth can only be determined through careful elicitation or knowledge of a model's training data. In one such evaluation, we show that probes can distinguish transcripts containing a model's true hidden goal from other goals with an AUROC of up to 99.7%. Our probes also detect deception on prominent open-weight models which lie about politically sensitive topics when put under pressure. We release our training dataset, dubbed FIBS, to help drive frontier deployment of effective probes, and encourage the community to expand upon it with further examples of deception and sabotage.

Abstract

Deception in Language Models

Explore similar work

Telling FIBS: Probes Effectively Detect Sabotage and Catch Unverbalized Deception

Before Agent Tells The Lie: Has Deception Already Been Represented?

Constitutional Black-Box Monitoring for Scheming in LLM Agents