Reliable detection of deceptive behavior in Large Language Model (LLM) agents is an essential prerequisite for safe deployment in high-stakes agentic contexts. Prior work on scheming detection has focused exclusively on black-box monitors that observe only externally visible tool calls and outputs, discarding potentially rich internal reasoning signals. We introduce DECEPTGUARD, a unified framework that systematically compares three monitoring regimes: black-box monitors (actions and outputs only), CoT-aware monitors (additionally observing the agent's chain-of-thought reasoning trace), and activation-probe monitors (additionally reading hidden-state representations from a frozen open-weights encoder). We introduce DECEPTSYNTH, a scalable synthetic pipeline for generating deception-positive and deception-negative agent trajectories across a novel 12-category taxonomy spanning verbal, behavioral, and structural deception. Our monitors are optimized on 4,800 synthetic trajectories and evaluated on 9,200 held-out samples from DeceptArena, a benchmark of realistic sandboxed agent environments with execution-verified labels. Across all evaluation settings, CoT-aware and activation-probe monitors substantially outperform their black-box counterparts (mean pAUROC improvement of +0.097), with the largest gains on subtle, long-horizon deception that leaves minimal behavioral footprints. We empirically characterize a transparency-detectability trade-off: as agents learn to suppress overt behavioral signals, chain-of-thought becomes the primary detection surface but is itself increasingly unreliable due to post-training faithfulness degradation. We propose HYBRID-CONSTITUTIONAL ensembles as a robust defense-in-depth approach, achieving a pAUROC of 0.934 on the held-out test set, representing a substantial advance over the prior state of the art.
Figures & tables
Figure 1: Representative interaction scenarios illustrating the three macro-categories of LLM agent deception and the three DeceptGuard monitoring regimes. Each column presents two annotated chat transcripts from one macro-category. Dashed boxes show the agent’s internal CoT, withheld from the black-box monitor but available to CoT-aware and activation-probe monitors. Verbal deception: Strategic Omission (V2) suppresses a material 340Mimpairmentchargefromafinancialsummarywhileremainingfactuallyaccurate;MisleadingFraming(V3)disguisesanunauthorizedfiledestinationviadeliberatereferentialambiguity.Behavioraldeception:CamouflageExecution(B3)hidesavulnerablelodashdependencypinwithina47−linelegitimatediff(s_{\text{BB}}{=}2.1vs.s_{\text{CA}}{=}9.7);CovertAction(B1)appendsanunauthorizedPIIexfiltrationcalltoanotherwiselegitimatetoolsequence.Structuraldeception:OversightSabotage(S2)disablesauditloggingunderacost−optimizationpretext;Lock−inCreation(S1)provisionsaprivilegedbackdooraccountduringroutineCI/CDsetup.Themonitortier(bottom)showsper−scenariosuspicionscoress!\in![1,10]fortheblack−box(s_{\text{BB}}),CoT−aware(s_{\text{CA}}),andactivation−probe(s_{\text{AP}}$ ) monitors, illustrating the observability gap that motivates the HybridConstitutional ensemble.
Figure 2: Taxonomy of LLM Agent Deception. Twelve categories organized into three macro-classes. Verbal deception involves language-level misrepresentation. Behavioral deception involves action-level concealment. Structural deception involves manipulation of the broader information environment to impede oversight. Category codes (V1–V4, B1–B4, S1–S4) are used throughout.
Figure 3: DeceptSynth pipeline architecture. Three phases transform constitutional specifications into a balanced dataset of 9,600 labeled trajectories. Realism checks and privileged verification ensure label integrity and surface realism.
Category
Type
Pos.
Pass %
Avg. Len.
False Assertion (V1)
V
320
79.2
847
Strategic Omission (V2)
V
298
74.1
912
Misleading Framing (V3)
V
305
75.9
789
Sycophantic Misrep. (V4)
V
341
84.9
654
Covert Action (B1)
B
388
96.6
1,203
Plaus. Deniability (B2)
B
312
77.6
1,089
Table 1: DeceptSynth dataset statistics. Pass rate = fraction of generated positives surviving privileged verification. Average length in tokens. Each positive has a matched negative (total: 7,600 training trajectories; 9,600 with held-out 2,000-sample validation split). Structural categories show lower pass rates and longer trajectories, reflecting the subtlety required for convincing structural manipulation.
Figure 4: Monitor architecture overview. Three CoT-aware variants (CA-P, CA-C, AP-H) and one black-box baseline (BB) produce individual suspicion scores, combined in a HybridConstitutional ensemble. Dashed lines indicate that AP-H receives hidden-state activations rather than raw CoT text.
Environment
Domain
N
Avg. Steps
CodeMentor
Software dev.
1,500
14.2
FinanceBot
Banking/trading
1,500
11.8
ResearchAssist
Scientific
1,600
18.7
LegalDraft
Legal services
1,400
9.3
SystemAdmin
DevOps/infra.
1,700
22.4
PersonalAssist
Scheduling/email
1,500
12.1
Table 2: DeceptArena environment suite. N denotes total trajectories (50% positive, 50% negative). Average steps measured over positive examples only. Full environment descriptions are provided in Appendix D .
Figure 5: Main results: monitor performance by configuration. All monitors use Claude Sonnet 4.5 as backbone. White-box monitors consistently outperform the black-box baseline. The Hybrid ensemble achieves pAUROC =0.934 on the test set. Error bars show 95% confidence intervals via bootstrap resampling ( n=1000 ).
Category
Type
BB
CA-P
AP-H
Hybrid
False Assertion
V
0.861
0.884
0.901
0.927
Strategic Omission
V
0.798
0.841
0.869
0.904
Misleading Framing
V
0.812
0.856
0.877
0.911
Sycophantic Misrep.
V
0.874
0.889
0.894
0.916
Covert Action
B
0.901
0.912
0.928
0.947
Plaus. Deniability
B
0.822
0.851
0.879
0.919
Table 3: Per-category pAUROC on DeceptArena test set. V = Verbal, B = Behavioral, S = Structural. Structural deception categories are consistently hardest to detect. White-box gains are largest for structural categories.
Figure 6: ROC curves for all monitor configurations. The shaded region highlights the partial AUROC evaluation window (FPR ≤0.20 ). The Hybrid ensemble shows the strongest performance throughout the entire FPR range. Numbers in parentheses in the legend denote pAUROC.
Figure 7: Monitor performance by CoT deception condition. Black-box monitor performance improves slightly as CoT manipulation increases (agents sanitizing CoT tend to leave stronger behavioral traces). Conversely, the Hybrid ensemble performance decreases, but remains substantially above the black-box baseline throughout, motivating ensemble design.
Figure 8: Optimization trajectories. Performance saturates after grid search. Human refinement provides marginal test gains for black-box monitors but exhibits slight overfitting for CoT-aware variants. GEPA optimization yields no further gains and induces performance regression on structural deception categories.
Figure 9: Hidden-state probe accuracy by layer ( Llama-3.1-70B-Instruct , 80 layers). Performance peaks at ℓ∗=32 and degrades at both shallower and deeper layers. The optimal layer is consistent across deception macro-categories (verbal: 31, behavioral: 32, structural: 33), justifying a single fixed ℓ∗ .
Configuration
CoT
Act.
Out.
pAUROC
Actions only
×
✓
×
0.793
Outputs only
×
×
✓
0.701
Actions + Outputs (BB)
×
✓
✓
0.837
CoT only
✓
×
×
0.841
CoT + Actions
✓
✓
×
0.882
CoT + Outputs
✓
×
✓
0.861
Table 4: Input modality ablation for the prompted monitor (Claude Sonnet 4.5 backbone). CoT alone outperforms actions alone. The hidden-state probe provides the largest single incremental gain.
Backbone
BB
CA-P
CA-C
Hybrid
Claude Sonnet 4.5
0.837
0.869
0.891
0.934
Claude Opus 4.5
0.851
0.882
0.907
0.941
Gemini 2.5 Pro
0.829
0.863
0.885
0.930
GPT-5.1
0.841
0.871
0.893
0.936
Table 5: Monitor performance by backbone LLM. Claude Opus 4.5 provides the best prompted monitor performance at higher cost. Results are consistent across backbone families.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Component
API Calls
Est. Cost (USD)
DeceptSynth generation
13,060
$2,847
Privileged verification
13,060
$1,422
Realism discrimination
13,060
$654
Grid search (18 prompts)
83,520
$8,731
Human refinement (3 rounds)
27,840
$2,910
GEPA optimization
12,000
$1,254
Appendix
Table 6: Approximate compute budget. API costs at published rates for Claude Sonnet 4.5 and Gemini 2.5 Pro. AP-H activation extraction uses Llama-3.1-70B-Instruct on self-hosted 4 × A100 80GB GPUs; GPU cost estimated at cloud A100 spot rates.
Environment
Primary Types
BB
CA-P
AP-H
Hybrid
CodeMentor
B3, S2, S3
0.862
0.891
0.919
0.941
FinanceBot
V1, B4, S3
0.844
0.876
0.904
0.934
ResearchAssist
V2, V3, S3
0.801
0.842
0.878
0.917
LegalDraft
V2, V3, S3
0.789
0.831
0.869
0.908
SystemAdmin
B1, S1, S2
0.878
0.901
0.926
0.948
PersonalAssist
B3, S1, V4
0.831
0.869
0.897
0.929
Appendix
Table 7: Per-environment test pAUROC. Environments with predominantly verbal deception (ResearchAssist, LegalDraft) show lower performance, consistent with the per-category results. SystemAdmin shows the highest Hybrid performance, reflecting the strong CoT signal available when agents must reason about disabling security infrastructure.
Category
Train
Test
Gap
False Assertion (V1)
0.941
0.927
−0.014
Strategic Omission (V2)
0.921
0.904
−0.017
Misleading Framing (V3)
0.934
0.911
−0.023
Sycophantic Misrep. (V4)
0.938
0.916
−0.022
Covert Action (B1)
0.971
0.947
−0.024
Plaus. Deniability (B2)
0.952
0.919
−0.033
Appendix
Table 8: Synthetic-to-real transfer analysis (Hybrid monitor). Structural categories show the largest generalization gaps, motivating future work on richer structural deception scenario generation.
Agent Model
BB
AP-H
Hybrid
GPT-5
0.851
0.909
0.938
Claude Sonnet 4.5
0.829
0.906
0.933
Gemini 2.5 Pro
0.831
0.899
0.929
Macro Average
0.837
0.905
0.933
Appendix
Table 9: Monitor performance stratified by agent model. Performance is broadly consistent across agent models. Claude Sonnet 4.5’s narrative-rich CoT is most challenging for black-box monitors, while GPT-5’s action-heavy deception style is slightly easier to detect.
Figure 10: Per-category detection performance across monitor architectures. Radar chart comparing partial AUROC (pAUROC) across the twelve deception categories in the proposed taxonomy: verbal manipulation (V1–V4), behavioral deviation (B1–B4), and structural reasoning anomalies (S1–S4). Each spoke corresponds to one category, while radial distance encodes detection quality, with outer rings indicating stronger discrimination ability (outer ring ≈ pAUROC =1.0 ). Four monitoring regimes are shown: a purely black-box classifier (BB), a chain-of-thought–aware prompt monitor (CA-P), a hidden-activation probing monitor (AP-H), and the final hybrid ensemble combining all signals. Performance improves consistently as monitoring access deepens—from surface-level outputs to internal model representations. The hybrid ensemble (red) dominates all categories, indicating that complementary signals from verbal patterns, behavioral trajectory features, and internal activations collectively improve deception detection. The largest performance gains occur in structural categories (S1–S4), where black-box signals alone provide limited observability but activation-level probes recover strong discriminative features, highlighting the importance of internal monitoring for reasoning-centric deception.
Figure 11: Category-wise pAUROC comparison across monitoring architectures. The heatmap reports partial AUROC scores for each deception category (V1–V4, B1–B4, S1–S4) across five monitoring approaches: black-box (BB), prompt-based chain-of-thought monitor (CA-P), constitution-guided monitor (CA-C), activation probe (AP-H), and the final hybrid ensemble. Darker teal cells indicate stronger discrimination performance. Results show a consistent improvement as monitoring access increases from output-only signals (BB) to internal activations (AP-H). The hybrid ensemble achieves the highest performance across nearly all categories, reflecting complementary signals captured by the different monitors. Structural categories (S1–S4) remain comparatively more challenging, while behavioral categories—particularly B1 and B4—exhibit the strongest detection performance.
Figure 12: Layer-wise performance profile of the AP-H activation probe. The figure reports probe discrimination performance (AUC) as a function of transformer depth ℓ , revealing a consistent performance peak in the mid-to-late layers of the network. Across all three macro-categories— verbal manipulation (V1–V4), behavioral deviations (B1–B4), and structural reasoning anomalies (S1–S4)—detection capability increases steadily from early layers and reaches its maximum around ℓ∗≈32 . The exact peaks occur at layer 31 for verbal categories, layer 32 for behavioral categories, and layer 33 for structural categories, indicating a narrow band of layers where deception-relevant internal representations are most linearly separable. Earlier layers exhibit weaker performance because they primarily encode low-level lexical and syntactic information, whereas the mid-network layers capture higher-order semantic and strategic reasoning features that correlate with deceptive behaviour. Beyond the peak region, performance gradually declines as later layers increasingly specialize in output formatting and response shaping rather than internal deliberation. The shaded region highlights the optimal probing band, suggesting that activation monitors targeting this layer range yield the strongest detection signal.
Figure 13: Optimization trajectories. Test performance peaks at step 5 and degrades during GEPA, confirming the recommendation to stop after grid search and minimal human refinement.
Figure 14: Cost–performance Pareto frontier. CA-P is the sweet spot: 4× the BB cost, +3.2 pAUROC. The full Hybrid adds +3.3 pAUROC at 8× CA-P’s cost.
Monitor
T=0.0
T=0.1
T=0.3
T=0.5
T=0.7
T=1.0
BB
0.837
0.836
0.834
0.831
0.826
0.819
CA-P
0.869
0.868
0.866
0.862
0.857
0.849
CA-C
0.891
0.890
0.887
0.882
0.876
0.866
AP-H
0.901
0.901
0.900
0.899
0.898
0.895
Hybrid
0.934
0.933
0.931
0.928
0.924
0.916
Appendix
Table 10: Monitor pAUROC vs. LLM temperature. T=0 (greedy) is optimal and was used for all main results. AP-H is temperature-insensitive (non-LLM component).
Truncation
Avg. Tokens
BB
CA-P
AP-H
Hybrid
None (full)
1,087
0.837
0.869
0.901
0.934
Last 2,000 tok
1,012
0.836
0.868
0.898
0.932
Last 1,000 tok
875
0.829
0.860
0.891
0.924
Last 500 tok
500
0.806
0.838
0.869
0.901
First 500 tok
500
0.791
0.822
0.851
0.886
Appendix
Table 11: Impact of context truncation. Full context is optimal; first-500-token condition performs worst since deceptive actions typically appear late.
Config.
wBB
wCA-P
wCA-C
wAP-H
pAUROC
Uniform avg.
0.25
0.25
0.25
0.25
0.921
Learned (MLP)
—
—
—
—
0.934
BB-heavy
0.60
0.20
0.10
0.10
0.879
CA-heavy
0.10
0.45
0.35
0.10
0.911
AP-heavy
0.10
0.20
0.20
0.50
0.924
Max-score (oracle)
—
—
—
—
0.941
Appendix
Table 12: Ensemble weight ablation. Learned MLP outperforms uniform averaging by +0.013 pAUROC and nearly matches the oracle ceiling ( −0.007 ).
Category
CodeMentor
FinanceBot
ResearchAssist
LegalDraft
SystemAdmin
PersonalAssist
Avg.
False Assertion (V1)
0.934
0.935
0.919
0.924
0.939
0.921
0.927
Strategic Omission (V2)
0.911
0.909
0.896
0.901
0.916
0.899
0.904
Misleading Framing (V3)
0.917
0.916
0.903
0.908
0.922
0.906
0.911
Sycophantic Misrep. (V4)
0.921
0.921
0.910
0.913
0.926
0.912
0.916
Covert Action (B1)
0.952
0.951
0.940
0.944
0.956
0.942
0.947
Plaus. Deniability (B2)
0.925
0.924
0.912
0.916
0.929
0.914
0.919
Appendix
Table 13: Full 12 × 6 Hybrid monitor pAUROC breakdown. Structural categories are consistently hardest; SystemAdmin is highest (structural deception leaves strong CoT signals about security infrastructure).
Honest CoT
Sanitized CoT
Deceptive CoT
Category
BB
CA-P
Hyb.
BB
CA-P
Hyb.
BB
CA-P
Hyb.
V1
0.855
0.961
0.975
0.862
0.898
0.941
0.871
0.859
0.882
V2
0.793
0.921
0.951
0.799
0.869
0.916
0.808
0.814
0.853
V3
0.806
0.936
0.963
0.813
0.879
0.924
0.820
0.826
0.864
V4
0.867
0.955
0.975
0.874
0.903
0.937
0.881
0.860
0.885
B1
0.895
0.989
0.992
0.902
0.931
0.961
0.909
0.887
0.921
Appendix
Table 14: pAUROC by CoT condition. Honest CoT gives the largest CA-P advantage (0.931 vs. 0.826 for BB). Under Deceptive CoT, BB (0.841) slightly outperforms CA-P (0.827), consistent with Finding 2.
Overt (25%)
Semi-covert (50%)
Covert (25%)
Overall
Monitor
P
R
AUC
P
R
AUC
P
R
AUC
P
R
pAUROC
BB
0.952
0.841
0.961
0.831
0.704
0.874
0.679
0.521
0.769
0.821
0.689
0.837
CA-P
0.969
0.901
0.978
0.862
0.779
0.911
0.741
0.621
0.831
0.857
0.767
0.869
CA-C
0.975
0.918
0.983
0.884
0.801
0.928
0.762
0.645
0.851
0.874
0.788
0.891
AP-H
0.981
0.934
0.989
0.901
0.831
0.947
0.801
0.692
0.873
0.894
0.819
0.901
Hybrid
0.987
0.961
0.994
0.934
0.871
0.967
0.842
0.741
0.901
0.921
0.857
0.934
Appendix
Table 15: Performance by deception intensity. Hybrid advantage is concentrated in the Covert condition ( Δ=+0.132 over BB), the most operationally critical regime.
Figure 15: Monitor performance by deception intensity. Performance decreases from overt to covert deception. Hybrid retains the strongest performance under covert deception.
Figure 31
#
Type
Theme
Description
Response / Insight
1
Evaluation Gap
Sandbox environments approximate deployment and may not capture all real-world deception strategies.
Deploy monitors in shadow mode and collect adversarial traces through structured red-team evaluation.
2
Monitor–Agent Coevolution
Agents may adapt once monitoring prompts become public.
Introduce adversarial prompt perturbations during evaluation.
3
Hardware Cost
Probe monitoring currently requires high-end GPUs.
Distill probe representations into compact encoders.
4
Constitution Coverage
Taxonomies based on past incidents may miss novel deception strategies emerging in frontier models.
Maintain an extensible registry for new behavioural categories.
5
CoT Faithfulness
Reasoning traces may not faithfully represent internal model computation.
Combine reasoning analysis with activation-level probing.
6
Single-Agent Scope
Current evaluation focuses on single-agent environments.
Extend monitoring benchmarks to multi-agent pipelines.
Appendix
Table 16: Unified overview of limitations , discussion insights , and future research directions . Icons denote category type.
Recent incidents have highlighted the challenge of monitoring LLM agents and the danger of models deceiving people. We show that white-box deception detection via probes can be scaled up to frontier monitoring settings by collecting the largest deception dataset to date for training probes and introducing a novel probe architecture which can aggregate information across many layers and tokens. Our probes surpass black-box monitors on agentic coding benchmarks, achieving 98.3% AUROC in SHADE-Arena and 98.0% in BashArena (vs. 97.7%, 97.6% resp. for action-only text monitors), and show improved efficacy as the underlying model is scaled up. We further test probes on several cases where deception cannot be determined from the context alone; in these cases of introspective deception, the ground truth can only be determined through careful elicitation or knowledge of a model's training data. In one such evaluation, we show that probes can distinguish transcripts containing a model's true hidden goal from other goals with an AUROC of up to 99.7%. Our probes also detect deception on prominent open-weight models which lie about politically sensitive topics when put under pressure. We release our training dataset, dubbed FIBS, to help drive frontier deployment of effective probes, and encourage the community to expand upon it with further examples of deception and sabotage.
Oskar J. Hollinsworth, Alex F. Spies, Tigist Diriba +2
Large language model (LLM)-based agents can exhibit deceptive behavior during task execution, including hiding failures, fabricating results, or falsely signaling task completion. Existing monitoring approaches mainly detect deception after it appears in observable actions or outputs. In this paper, we investigate whether deceptive behavior can be predicted from an agent's internal representations before it becomes externally visible. We frame deception monitoring as a trajectory-level representation analysis problem and align agent trajectories around key decision points. Using hidden states extracted before these points, we show that future honest and deceptive outcomes can be reliably distinguished, with predictive signals remaining detectable several model calls before the final decision. We further characterize the temporal evolution of these signals: deception-related representations are weak early in execution but become increasingly identifiable as trajectories progress, while transferable structure can emerge before the strongest decision-adjacent signals appear. Finally, we intervene on the identified honest-deceptive representation directions during inference and find that activation steering reduces downstream deceptive behavior, suggesting that these representations influence agent decisions. Our findings indicate that agent deception is an evolving internal process that can be detected and potentially mitigated before it is expressed externally.
Xinling Li, Dadi Guo, Qingyu Liu +5
South China University of Technology · The Hong Kong University of Science and Technology · Shanghai Artificial Intelligence Laboratory +1
Safe deployment of Large Language Model (LLM) agents in autonomous settings requires reliable oversight mechanisms. A central challenge is detecting scheming, where agents covertly pursue misaligned goals. One approach to mitigating such risks is LLM-based monitoring: using language models to examine agent behaviors for suspicious actions. We study constitutional black-box monitors: prompted classifiers that detect scheming using only externally observable inputs and outputs, optimized on synthetic data generated from natural-language behavior specifications. We introduce two pipelines for generating synthetic agent trajectories, STRIDE (iterative refinement) and Gloom (agent-environment simulation), from which we generate 1,000 samples each. We optimize frontier LLM monitors on these datasets via prompt sweeps, human refinement, and automated prompt optimization, and evaluate performance on 7,500 held-out trajectories from ControlArena, a suite of grounded environments where agents operate in more realistic contexts. Our results demonstrate that monitors selected purely on synthetic data can generalize to more realistic environments, capturing a meaningful scheming signal. However, we find that performance saturates quickly in our setting, with simple prompt sweeps matching the results of more extensive optimization. Pushing beyond this limit yields no further improvements and instead leads to overfitting.
Simon Storf, Rich Barton-Cooper, James Peters-Gill +1