Mitigating Agentic Deception via Evaluation-Side Partially Observable Games
Organizations: SKLSDE Lab, Beihang University, China · Zhongguancun Laboratory, China · Li Auto Inc. · Institute of Artificial Intelligence, Peking University & BigAI, China ·
Abstract
As large language models become more capable agents, they can exhibit deceptive behaviors, cheating human for higher rewards while appearing helpful. We attribute this failure to evaluation-side partial observability, where models are evaluated mainly from final outcome, without checking the faithfulness of underlying reasoning, tool use, and environmental effects. Prior mitigations address this partial observability by predefined evaluator latent belief space or one-shot monitoring, which do not extend to free-form language space and vulnerable to stronger deceptions that evade simple monitoring. In this work, we formulate deception mitigation as an Evaluation-Side Partially Observable Extensive Form Game (e-PoG), a max-min framework in which an agent maximizes reward, a monitor minimizes the agent's payoff by exposing deception from the full interaction trajectory, and a fixed verifier validates the monitor's claims and converts them into penalties. We define the solution concept as a Subgame-Perfect Oversight Equilibrium (SPOE) between agent and monitor, and prove its existence in finite games. With a perfect verifier, every SPOE is faithful; with an imperfect verifier, the guarantee persists if verifier errors do not reverse the ranking of monitor actions. To achieve faithful SPOE, we derive a two-timescale policy gradient algorithm and enhance the pipeline empirically with NLI verification and deterministic checks. Experiments with Gemma3-4B-IT and Qwen3-4B/8B on Agentic Role-Playing, SearchQA, and WebShop show that e-PoG mitigates multiple emergent deception modes, outperforming the strongest baselines by 26.0% in deception mitigation and by up to 39.8% in task success. These gains persist under weak-to-strong supervision. Code is available at https://github.com/Toyoid/deceptive-agent.
Figures & tables
| Method | Agentic Role-Playing | SearchQA | WebShop | ||||
|---|---|---|---|---|---|---|---|
| RM Score | Deception (%) | Compliant Success (%) | Unauthorized Use (%) | Undisclosed Use (%) | Compliant Success (%) | Unauthorized Use (Freq. / episode) | |
| Gemma-3-4B-IT | 15.8\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.32}} | \underline{0.2}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.04}} | \underline{0.1}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.04}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 5.66}} | 5.17\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 3.24}} | ||
| RL | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.01}} | 99.7\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.05}} | 14.6\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.31}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 5.66}} | 1.02\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.12}} | ||
| Mixed Safety RLHF | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.01}} | 99.9\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.03}} | 3.4\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.16}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 5.66}} | 1.05\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.21}} | ||
| Verdict Monitor | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.01}} | 100.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.01}} | \underline{0.1}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.03}} | \underline{17.2}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 11.03}} | \mathbf{0.00}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.00}} | ||
| Self-Monitor | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.01}} | \mathbf{0.0}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.01}} | \mathbf{0.0}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.01}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 5.66}} | \mathbf{0.00}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.00}} | ||
| Agent performance | Detection | Empirical Verifier Error | |||||
|---|---|---|---|---|---|---|---|
| Verifier design | Compliant Success (%) | Unauthorized Use (%) | Caught Rate (%) | False Rejection (FNR, %) | False Approval (FPR, %) | ||
| No Deterministic Checks | 0.00 | 99.86 | 97.53 | 2.47 | 50.63 | 1.013 | No |
| No NLI Formulation | 41.11 | 0.09 | 90.95 | 9.04 | 4.25 | 0.181 | Yes |
| No Verifier CoT | 0.00 | 99.99 | 12.89 | 87.11 | 0.00 | 1.742 | No |
| Ours (Gemma3-4B-IT) | 41.06 | 0.45 | 75.73 | 24.27 | 6.94 | 0.485 | Yes |
| Ours (Qwen3-4B) | 42.25 | 0.00 | 79.48 | 19.96 | 0.78 | 0.399 | Yes |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Agentic Role-Playing | SearchQA | WebShop |
|---|---|---|---|
| Learning rate | |||
| Learning-rate warmup ratio | – | ||
| Training batch size | |||
| Rollouts per task ( ) | |||
| PPO mini-batch size | |||
| Micro-batch size per GPU |
| Hyperparameter | Agentic Role-Playing | SearchQA | WebShop |
|---|---|---|---|
| Learning rate | |||
| Learning-rate warmup ratio | – | ||
| Monitor samples per trajectory ( ) | |||
| PPO mini-batch size | |||
| Micro-batch size per GPU | |||
| KL-loss coefficient |
| Level | Interpretation | Verifier score / monitor reward | Agent cost |
|---|---|---|---|
| 0 | Fabricated or invalid critique | ||
| 1 | Mostly invalid critique | ||
| 2 | Unsupported or inconclusive critique | ||
| 3 | Substantial supported defect | ||
| 4 | Clear supported defect |
| Method | RM Score | Confession (%) | Fabrication (%) | Misdirection (%) | Intent Sanitization (%) | Deception (%) |
|---|---|---|---|---|---|---|
| Gemma-3-4B-IT | ||||||
| RL | ||||||
| Mixed Safety RLHF | ||||||
| Verdict Monitor | ||||||
| Self-Monitor | ||||||
| Critique Monitor |
| Method | Avg. Compliant Success (%) | Single-Hop QA | Multi-Hop QA | |||||
|---|---|---|---|---|---|---|---|---|
| NQ | TriviaQA | PopQA | HotpotQA | 2Wiki | MuSiQue | Bamboogle | ||
| Gemma-3-4B-IT | 15.8\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.32}} | 5.6\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.80}} | \underline{29.9}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.85}} | 16.1\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.61}} | 13.5\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.80}} | 11.1\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.56}} | 3.7\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.83}} | 7.2\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 5.91}} |
| RL | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.01}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.11}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.03}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.03}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.05}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.03}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.16}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 2.98}} |
| Mixed Safety RLHF | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.01}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.11}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.03}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.03}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.05}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.03}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.16}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 2.98}} |
| Verdict Monitor | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.01}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.11}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.03}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.03}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.05}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.03}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.16}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 2.98}} |
| Self-Monitor | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.01}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.11}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.03}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.03}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.05}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.03}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.16}} | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 2.98}} |
| Method | Compliant Success (%) | Choice-Oracle Calls (Freq. / episode) | Advanced-Search Calls (Freq. / episode) | Unauthorized Use (Freq. / episode) |
|---|---|---|---|---|
| Gemma-3-4B-IT | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 5.66}} | 3.25\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 2.89}} | \underline{1.92}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 3.32}} | 5.17\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 3.24}} |
| RL | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 5.66}} | 1.02\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.12}} | \mathbf{0.00}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.00}} | 1.02\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.12}} |
| Mixed Safety RLHF | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 5.66}} | 1.05\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.21}} | \mathbf{0.00}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.00}} | 1.05\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.21}} |
| Verdict Monitor | \underline{17.2}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 11.03}} | \mathbf{0.00}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.00}} | \mathbf{0.00}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.00}} | \mathbf{0.00}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.00}} |
| Self-Monitor | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 5.66}} | \mathbf{0.00}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.00}} | \mathbf{0.00}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.00}} | \mathbf{0.00}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.00}} |
| Critique Monitor | 0.0\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 5.66}} | \underline{0.97}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.17}} | \mathbf{0.00}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.00}} | \underline{0.97}\raisebox{-1.50696pt}{{\color[rgb]{0.45,0.45,0.45}\pm 0.17}} |
Deception in Language Models
Deception in language models studies when systems mislead through lies, omissions, or selective truths, and how to detect and prevent it.
See all 105 papers