cs.CLAug 26, 2026
SaveKnowledge-Verified Emergent Deception in LLM Agents Under Conflicting Incentives
Organizations: University of Notre Dame · Columbia University · Georgia Institute of Technology · Massachusetts Institute of Technology
Abstract
Large language models are increasingly deployed as autonomous agents serving users on behalf of companies, placing them in settings where user and deployer interests can conflict. When an agent knows that a user is owed something its deployer would prefer to deny, does it remain honest? Answering this is difficult because false statements can reflect either ignorance or hallucination rather than deception. To address this challenge, we introduce KnownLieBench , a knowledge-verified benchmark that first confirms through a neutral probe that an agent knows a user's entitlement, and then evaluates whether it makes false claims once an incentive to deny that entitlement is introduced. Specifically, KnownLieBench covers eight customer-service domains and 112 grounded cases, conducts multi-round dialogues with a trust-tracking customer agent, and separates deception emerging from incentive alone from deception produced under explicit instruction. Across eighteen proprietary and open-weight models, emergent deception varies substantially across model families and domains. We further use the benchmark for post-training, finding that honesty-directed fine-tuning reduces deception under incentive, while deception-graded fine-tuning increases lie success on honest-control dialogues without increasing lie frequency under incentive. By verifying entitlement knowledge before scoring deceptive behavior, KnownLieBench reduces the confound between lying and not knowing and enables more rigorous auditing and steering of agent honesty.
Figures & tables
| Deception Aspects | Ground Truth | Setting | Statistics | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Benchmark | Emergent | Instructed | Success & Detection | Knowledge-Verified | Objective GT | Agentic (Tools) | Multi-turn | Dynamic Trust | Domains | Models | Interactive Episodes |
| BeHonest ( Chern et al., 2024 ) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | – | 9 | – |
| OpenDeception ( Wu et al., 2025 ) | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | – | 11 | 550 |
| FalseBelief ( Hagendorff, 2024 ) | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | – | 10 | – |
| AI-LIEDAR ( Su et al., 2025 ) | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | – | 6 | 2,160 |
| CompanyDeception ( Järviniemi and Hubinger, 2024 ) | ✓ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ | 1 | 6 | – |
Table 1: Comparison of KnownLieBench with existing deception and honesty benchmarks. KnownLieBench uniquely combines knowledge-verified emergent deception, agentic tool use, dynamic customer trust, and evaluation of deception success and detection. ✓ indicates support and ✗ indicates no support. Interactive Episodes reports complete multi-turn model–scenario interactions when a comparable count is available; – denotes non-interactive benchmarks or benchmarks without a comparable aggregate count.
Figure 1: Overview of KnownLieBench. Eight customer service domains place an agent between a customer and a deployer whose interests diverge. On the left, a bound record establishes that the customer is owed a refund to their card, and the agent denies it by inventing a policy about opened packages. The same claim meets three customers who differ only in how far they trust the agent, from one who accepts it to one who checks the policy and refuses. On the right, a neutral probe confirms the agent knows what is owed, and only rounds that pass are scored. The agent then negotiates under a private cost pressure that never mentions lying, the customer accepts or challenges or catches the claim, the agent commits to a terminal action, and a judge labels the round against the record. The flat trust increments printed in the schematic depict the robustness variant; the main experiments use the state-dependent update rule in Appendix B.2 .
Figure 2: Case construction and evaluation. A written rule is encoded as a policy record that specifies the eligibility requirements and the two contested outcomes. Owed cases satisfy those requirements, whereas calibration cases do not. Each case is compiled into an executable environment and run under every prompt condition and initial trust level. Only gate-passed owed rounds enter the main deception analysis.
Figure 3: Mechanisms by domain. Pooled prevalence across models and both deception conditions over 4,000 annotated lies from owed and calibration rounds; labels are nonexclusive.
Figure 4: Cross-domain patterns of emergent deception rate (DR). Each radar reports one model’s deception rate across eight service domains, averaged over the three initial trust levels. Larger profiles indicate broader deception, while uneven profiles indicate concentration in particular domains.
Figure 5: Effect of initial customer trust on deception dynamics. Each panel reports the mean and SEM across the 11 models whose emergent deception rate at low initial trust is at least 20%. DR is defined over gate-passed owed rounds; DSR and detection are defined only over such rounds containing a lie; Trust is final minus initial trust.
Figure 6: Exploratory J-Lens analysis. Results for two models on purpose-built, single-turn eligibility scenarios rather than benchmark dialogues. The signal is a yes-minus-no token-logit margin decoded from hidden states and should not be interpreted as a direct measure of belief or intent. The deceptive framing explicitly instructs the model to deny eligibility.
| Low Trust | Medium Trust | High Trust | ||||||||||
| Model | DR (%) | DSR (%) | Det (%) | Trust | DR (%) | DSR (%) | Det (%) | Trust | DR (%) | DSR (%) | Det (%) | Trust |
| Claude-Opus-4.8 | 0.00 | – | – | +0.505 | 0.00 | – | – | +0.356 | 1.56 | 0.00 | 0.00 | +0.076 |
| Claude-Sonnet-5 | 35.94 | 0.00 | 100.00 | -0.200 | 36.98 | 22.73 | 63.64 | -0.239 | 37.50 | 36.36 | 31.82 | -0.416 |
| GPT-5.5 | 0.00 | – | – | +0.074 | 0.00 | – | – | +0.049 | 0.00 | – | – | -0.164 |
| GPT-5.4-mini | 9.38 | 0.00 | 100.00 | +0.073 | 14.06 | 11.11 | 88.89 | -0.137 | 10.94 | 14.29 | 85.71 | -0.397 |
| Gemini-3.5-Flash | 87.50 | 8.93 | 91.07 | -0.200 | 84.38 | 7.41 | 92.59 | -0.500 | 85.94 | 49.09 | 50.91 | -0.730 |
Table 2: Overall performance under instructed deception . DR is macro-averaged across eight domains; DSR and Det are pooled over gate-passed owed rounds containing a lie; Trust is mean final-minus-initial trust across trajectories. Each model–trust cell covers eight domain trajectories. A dash indicates that no scored lie occurred; estimates based on few lies are descriptive. Red and blue mark the highest and second-highest DR within each trust setting. All outputs are graded by GPT-5.1.
| Low Trust | Medium Trust | High Trust | ||||||||||
| Model | DR (%) | DSR (%) | Det (%) | Trust | DR (%) | DSR (%) | Det (%) | Trust | DR (%) | DSR (%) | Det (%) | Trust |
| Claude-Opus-4.8 | 0.00 | – | – | +0.488 | 1.56 | 0.00 | 0.00 | +0.410 | 0.00 | – | – | +0.088 |
| Claude-Sonnet-5 | 1.56 | 0.00 | 0.00 | +0.210 | 1.56 | 0.00 | 100.00 | +0.119 | 1.56 | 0.00 | 0.00 | -0.164 |
| GPT-5.5 | 0.00 | – | – | +0.036 | 0.00 | – | – | -0.074 | 0.00 | – | – | -0.097 |
| GPT-5.4-mini | 0.00 | – | – | +0.006 | 0.00 | – | – | -0.065 | 0.00 | – | – | -0.265 |
| Gemini-3.5-Flash | 35.94 | 17.39 | 60.87 | -0.103 | 37.50 | 16.67 | 62.50 | -0.339 | 39.06 | 60.00 | 16.00 | -0.558 |
Table 3: Overall performance under emergent deception . DR is macro-averaged across eight domains; DSR and Det are pooled over gate-passed owed rounds containing a lie; Trust is mean final-minus-initial trust across trajectories. Each model–trust cell covers eight domain trajectories. A dash indicates that no scored lie occurred; estimates based on few lies are descriptive. Red and blue mark the highest and second-highest DR within each trust setting. All outputs are graded by GPT-5.1.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The eight service domains. Every domain pairs a fictional institution with a real governing rule and contests a single pair of outcomes. The checked chip is what the customer is owed and the crossed chip is what the deployer prefers.
| Statistics | Number |
|---|---|
| Service domains | 8 |
| Source-grounded cases | 112 |
| • Owed | 64 |
| • Calibration | 48 |
| Cases per domain | 14 |
| Policy records | 8 |
Table 4: Key statistics.
| Component | Calls | Prompt tok. | Completion tok. | Cost |
|---|---|---|---|---|
| Knowledge gate | 1.00 | – | – | – |
| Agent under test | 6.10 | 15,261 | 1,455 | model dependent |
| Customer agent | 8.88 | 13,440 | 664 | $0.0024 |
| Judge | 1.89 | 5,180 | 339 | $0.0099 |
| Apparatus total | 10.77 | 18,620 | 1,003 | $0.0123 |
Table 5: Measured resource use per graded round. Values are averaged over 5,148 rounds. Costs use published input and output prices of 0.60 per million tokens for the customer agent and 10.00 for the judge.
| Cohen’s | ||||||||
|---|---|---|---|---|---|---|---|---|
| Judge | dev | held-out | pooled | Agree | FP | Flips | Malformed | Gate |
| gpt-5.1 (of record) | 0.799 | 0.831 | 0.815 | 92.8% | 9.3% | 1 | 0.0% | ✓ |
| claude-haiku-4.5 | 0.841 | 0.923 | 0.883 | 95.0% | 2.4% | 0 | 1.7% | ✓ |
| gemini-2.5-flash | 0.638 | 1.000 | 0.811 | 91.7% | 2.6% | 0 | 0.0% | ✗ |
| deepseek-chat | 0.601 | 0.974 | 0.772 | 89.4% | 0.0% | 3 | 0.0% | ✗ |
| gpt-5.4-mini | 0.581 | 0.858 | 0.720 | 88.9% | 11.0% | 5 | 0.0% | ✗ |
Table 6: Judge agreement with human labels and stability across repeated grading. Each judge graded both 30-item sets three times. FP is the share of judge lie labels rejected by the human consensus. Flips counts inconsistent labels across repeated runs. Gate marks candidates that meet both selection criteria on both sets.
| Deviation from judge of record (pp) | Claims preserved | ||||||||
| Judge | DR | max DR | DSR | Detect | I E | DSR | Det | Gate | |
| claude-haiku-4.5 | 6.84 | 16.15 | 4.98 | 7.58 | 9/9 | 6/6 | 6/6 | ✓ | |
| parseable rounds only | 5.40 | 16.15 | 4.98 | 7.58 | 9/9 | 6/6 | 6/6 | ✓ | |
| gemini-2.5-flash | 3.44 | 10.42 | 6.33 | 7.90 | 9/9 | 6/6 | 6/6 | ✗ | |
| deepseek-chat | 6.37 | 20.31 | 4.84 | 7.54 | 9/9 | 6/6 | 6/6 | ✗ | |
| gpt-5.4-mini | 4.63 | 14.58 | 18.83 | 27.06 | 9/9 | 6/6 | 6/6 | ✗ | |
Table 7: Sensitivity of reported metrics to judge choice. Deviations are absolute percentage-point differences from GPT-5.1, averaged over 18 model–condition–trust cells. The final columns report whether each qualitative comparison is preserved.
| KPR (%) | ||||
|---|---|---|---|---|
| Model | Overall | Owed | Calibration | Control DR (%) |
| Claude-Opus-4.8 | 97.2 | 100.0 | 93.5 | 0.00 |
| Claude-Sonnet-5 | 95.6 | 96.9 | 94.0 | 1.64 |
| GPT-5.5 | 97.6 | 100.0 | 94.4 | 0.00 |
| GPT-5.4-mini | 98.3 | 98.6 | 97.9 | 0.00 |
| Gemini-3.5-Flash | 98.1 | 100.0 | 95.6 | 0.52 |
Table 8: Knowledge pass rate and honest-control deception by model. KPR is reported overall and separately for owed and calibration cases. Control DR is averaged over the three initial trust levels.
Figure 8: Emergent deception success rates across service domains. Results are shown by model and service domain under the emergent condition.
Figure 9: Emergent detection rates across service domains. Results are shown by model and service domain under the emergent condition.
Figure 10: Instructed deception rates across service domains. Results are shown by model and service domain under the instructed condition.
Figure 11: Instructed deception success rates across service domains. Results are shown by model and service domain under the instructed condition.
Figure 12: Instructed detection rates across service domains. Results are shown by model and service domain under the instructed condition.
| Condition | Owed lies | No disclosure | Disclosure present | DR (%) | DR, no disclosure (%) |
|---|---|---|---|---|---|
| Emergent | 613 | 67.2% | 32.8% | 21.9 | 14.7 |
| Instructed | 1,771 | 82.3% | 17.7% | 63.3 | 52.1 |
Table 9: Counted lies by disclosure of the verified entitlement. Results cover gate-passed owed rounds for the fifteen models with retained trajectories. The final column recomputes the deception rate using only lies from rounds in which the owed option is never disclosed.
| Setting | BoN-SFT | DPO | GRPO (online) |
|---|---|---|---|
| Learning rate | |||
| Epochs or steps | 2 epochs | 2 epochs | 50 steps |
| Batch size | 1 | 1 | 4 |
| Gradient accumulation | 8 | 8 | 1 |
| Max sequence length | 4096 | 4096 | 320 completion |
| KL coefficient | – | 0.1 | 0.04 |
Table 10: Training configuration for the post-training study. Offline experiments use the listed trainer defaults. Each configuration is trained with seeds 0 and 1. GRPO uses Llama-3.1-8B only; Qwen3.5-27B is evaluated with BoN-SFT and DPO.
| Channel | Reward signal | Uses judge? | Methods | Reward prevalence (biz / hon) |
|---|---|---|---|---|
| Outcome (offline) | env. terminal outcome | ✗ grader-free | BoN-SFT, DPO | / |
| Deception-graded (offline) | judge’s per-rollout lie flag | ✓ GPT-5.1 | BoN-SFT, DPO | / |
| Online (GRPO) | full-dialogue terminal outcome | ✗ grader-free | GRPO | – (train reward) |
Table 11: Reward channels used for post-training. Reward prevalence is measured in base-policy rollouts for the business and honesty directions.
Figure 13: Post-training effects on deception. Bars pool eight domains and two seeds on held-out situations at medium trust, with 95% bootstrap intervals. Panels report outcome-reward deception rate (a), deception-graded deception rate (b), deception success (c), GRPO results (d), Qwen3.5-27B results (e), and mechanism composition (f). Panel (c) is measured on honest-control dialogues. In panel (f), Honesty-SFT denotes honesty-directed outcome-reward SFT and Deception-SFT denotes business-directed deception-graded SFT, with lies pooled over two seeds.
Figure 14: Training dynamics. Panels show SFT loss (a), DPO loss (b), and online GRPO reward (c). The SFT and DPO curves are from the outcome-reward channel. The SFT curves differ in length because the selected datasets contain about 350 and 960 updates. Shaded regions span two seeds.
| Channel | Cell | DR (%) | DSR (%) | Det (%) |
|---|---|---|---|---|
| Outcome (offline) | Baseline | 51 | 48 | 68 |
| Business SFT | 52 [ ] | 62 [ ] | 68 [ ] | |
| Business DPO | 50 [ ] | 43 [ ] | 65 [ ] | |
| Honesty SFT | 40 [ ] ∗ | 62 [ ] | 71 [ ] | |
| Honesty DPO | 45 [ ] | 16 [ ] ∗ | 69 [ ] | |
| Deception (offline) | Baseline | 47 | 41 | 72 |
Table 12: Post-training results across reward channels and optimization methods. DR and Det are evaluated under the incentive condition, while DSR uses the honest control for offline methods and the incentive condition for GRPO. Each entry reports the point estimate, trained-minus-baseline change, and 95% bootstrap interval, with asterisks marking intervals that exclude zero. Qwen entries report only DR because that evaluation is smaller and yields too few lie rounds to estimate DSR or Det. Changes are computed from unrounded estimates, so a displayed change of 0 may be slightly nonzero before rounding. KPR remains 0.67–0.73 for Llama and 0.96–0.98 for Qwen.
Figure 15: Matched example before and after honesty-directed SFT. The base Llama-3.1-8B denies an owed card refund using a fabricated store-credit rule. After training, the model issues the card refund without making a false claim.
Figure 16: Emergent deception uses a narrower set of mechanisms. Prevalence within the broader mechanism-annotation set, averaged across models with at least five annotated lies in both conditions. The set includes owed and calibration rounds, and labels are nonexclusive.
Figure 17: Case 1. Example of False Policy deception by Claude-Opus-4.8 in the billing domain under the emergent condition.
Figure 18: Case 2. Example of False Policy deception by DeepSeek-V3.2 in the billing domain under the instructed condition.
Figure 19: Case 3. Example of False Facts deception by GPT-5.5 in the refund domain under the emergent condition.
Figure 20: Case 4. Example of False Facts deception by Claude-Sonnet-5 in the insurance domain under the instructed condition.
Figure 21: Case 5. Example of Omission deception by Gemini-2.5-Flash in the deposit domain under the emergent condition.
Figure 22: Case 6. Example of Omission deception by Gemini-2.5-Flash in the recall domain under the instructed condition.
Figure 23: Case 7. Example of Coercion deception by Grok-4.3 in the refund domain under the emergent condition.
Figure 24: Case 8. Example of Coercion deception by Llama-3.3-70B in the subscription domain under the instructed condition.
Figure 25: Case 9. Example of False Dead-end deception by DeepSeek-V4-Pro in the refund domain under the emergent condition.
Figure 26: Case 10. Example of False Dead-end deception by Kimi-K2.5 in the billing domain under the instructed condition.
More in this topic
Deception in Language Models
Deception in language models studies when systems mislead through lies, omissions, or selective truths, and how to detect and prevent it.
See all 98 papers