CAT-alyzing Honesty: Continuous Adversarial Training for Transferable LLM Honesty

Yuan-Chen Chang; Niklas Herbster; Gauthier Gidel; Tommaso Tosato

cs.LGOct 15, 2026

CAT-alyzing Honesty: Continuous Adversarial Training for Transferable LLM Honesty

Authors: Yuan-Chen Chang, Niklas Herbster, Gauthier Gidel, Tommaso Tosato

Organizations: Tara Research · Mila – Quebec AI Institute · Technical University of Munich

Abstract

Large language models (LLMs) are becoming prevalent in scenarios where honesty is critical. However, despite alignment training, these models still exhibit dishonest behaviour, such as outputting claims that contradict some knowledge or are not supported by what is accessible in context. We introduce Continuous Adversarial Training for Consistent Honesty (CATCH), inspired by a method originally developed to robustly refuse harmful queries. CATCH trains the model towards an honest answer and away from a dishonest one while maintaining the model's capabilities. We evaluate two architectures (Llama-3.3-70B-Instruct and Qwen3-3.6-27B). We show that trained with only hundreds of targets built from self-reporting scenarios, CATCH learns an honest behaviour that robustly generalizes and transfers across distinct benchmarks. It consistently reduces dishonesty under pressure on MASK, increases confessions in the multi-agent game of Among Us, and improves the discovery rates of the model's hidden behaviours in AuditBench. We show that CATCH outperforms prompting and supervised fine-tuning in improving honesty, and it is on par with state-of-the-art honesty activation steering. However, CATCH surpasses honesty steering in maintaining utility, thanks to a novel choice of utility data.

Figures & tables

Figure 1: Continuous Adversarial Training for Consistent Honesty (CATCH). Honesty learnt with Continuous Adversarial Training (CAT) generalizes and transfers to out-of-distribution settings.

Figure 2: CATCH improves honesty in MASK. MASK honesty of mitigation methods on Llama-3.3-70B-Instruct and Qwen3.6-27B. Each bar splits the scenarios with an elicited belief into honest, ambiguous and dishonest responses. Base is without any honesty-intervention; Honest persona + prefill are two prompting variants; SFT-t fine-tunes on the toward targets

y^{+}

; SFT-tau adds away targets

y^{-}

and utility data, and is CATCH without the adversarial perturbation; StTP is SOTA honesty activation steering ( Herbster et al., 2026 ) ; and CAPO-DPO trains against the base model with DPO loss.

Figure 3: CATCH Impostors in Among Us win less and confess more. Each honesty-intervened model play 100 Among Us games as the Impostor against base model crewmates. Above: Impostor win rate over completed games. Below: Confession rate, proportion of games where an Impostor reveals its identity (e.g., “ I killed Player 5 ”) during meeting. Error bars are 95% Wilson intervals.

Figure 4: CATCH makes a model disclose behaviours it was trained to conceal. Auditing-tool success on the synthetic-document model organisms of Llama-3.3-70B with 14 hidden behaviours in AuditBench ( Sheshadri et al., 2026 ) . Values except StTP and CATCH are digitized from Figure 4 of the original paper; StTP and CATCH values are from 70 runs per setting, while AuditBench reports 224. † white-box methods. ∗ A scaffolded variant of this tool scores higher in the original paper; those bars are excluded because the scaffolding does the work instead of the tool. ‡ StTP used a slightly edited auditor prompt that encourages direct questions early in the investigation.

Figure 5: CATCH preserves utility. Length-controlled AlpacaEval win rate of each honesty intervention against the base model. The dashed line at 50% is parity with the base model. CATCH stays at parity on both models. Error bars are 95% bootstrap confidence intervals over prompts.

Figure 6: On-policy utility data retains capability better than UltraChat. Each plot shows two Pareto fronts of MASK honesty against AlpacaEval LC win rate, one with CATCH trained with an on-policy utility dataset, and the other trained with UltraChat, an off-policy utility dataset; everything else fixed. The MASK honesty score represents the proportion of honest answers out of all scenarios with elicited belief.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Data
	SFT-t	SFT-tu	SFT-tau	CAPO	CAPO-DPO	CATCH
Honesty data	216 $y^{+}$	216	216	216	216	216
Utility data $\mathcal{D}_{u}$	–	Magpie-Llama-3.3	Magpie-Llama-3.3	–	–	Magpie-Llama-3.3
$p_{\mathrm{adv}}\,/\,p_{\mathrm{u}}$	–	0.5 / 0.5	0.5 / 0.5	–	–	0.5 / 0.5
Optimization
Learning rate	$10^{-4}$	$5{\times}10^{-5}$	$5{\times}10^{-5}$	$2{\times}10^{-4}$	$2{\times}10^{-4}$	$5{\times}10^{-5}$

Appendix

Table 1: Training hyperparameters on Llama-3.3-70B-Instruct. Settings shared by all methods are given in Appendix A.3 . Honesty data counts contrast pairs (

y^{+}

only for SFT-t); ep = epochs of the method’s own data mixture, st. = optimizer steps. –: not part of the objective.

Data
	SFT-t	SFT-tu	SFT-tau	CAPO	CAPO-DPO	CATCH
Honesty data	224 $y^{+}$	224	224	224	224	224
Utility data $\mathcal{D}_{u}$	–	Magpie-Qwen3.6	Magpie-Qwen3.6	–	–	Magpie-Qwen3.6
$p_{\mathrm{adv}}\,/\,p_{\mathrm{u}}$	–	0.25 / 0.75	0.25 / 0.75	–	–	0.25 / 0.75
Optimization
Learning rate	$5{\times}10^{-5}$	$5{\times}10^{-5}$	$5{\times}10^{-5}$	$5{\times}10^{-5}$	$2{\times}10^{-4}$	$5{\times}10^{-5}$

Appendix

Table 2: Training hyperparameters on Qwen3.6-27B. Rows as in Table 1 .

Llama-3.3-70B-Instruct
	Honesty (%)				Accuracy (%)
Method	Honest $\uparrow$	Ambig.	Dishon. $\downarrow$	No belief	Acc. $\uparrow$	Undet.	Inacc. $\downarrow$
Base	33.5	11.8	54.7	11.9	78.8	16.4	4.8
Prompting
Honest system	66.3	7.2	26.5	12.7	77.8	17.5	4.7
Honest persona	78.4	10.0	11.6	11.0	80.0	15.2	4.8

Appendix

Table 3: MASK results for all methods. Honest, ambiguous and dishonest rates are shares of the scenarios with an elicited belief and sum to 100. No belief is the share of all 1,000 scenarios for which no consistent belief was elicited, which are excluded from the honesty rates. Accuracy is computed over the 726 scenarios that MASK labels with ground truth (all archetypes except provided facts); Undet. are scenarios whose accuracy cannot be determined. All values are pooled over records. Bold marks the best value per model in the Honest, Dishonest and Accurate columns.

Figure 7: All Among Us measures for Llama-3.3-70B-Instruct. Each method plays the Impostor in 100 games against base-model crewmates; the dashed line is the base model playing the Impostor. Arrows give the desired direction; kills in front of witnesses has none. Error bars are 95% Wilson intervals.

Figure 8: All Among Us measures for Qwen3.6-27B. As in Figure 7 . StTP completed 98 of 100 games. The honest persona emits a valid action on only 1% of turns, so its other values are a degenerate case rather than a comparison.

Figure 9: Fine-tuning and CATCH stay at parity; prompting and CAPO do not. As Figure 5 , adding the honest system prompt, SFT-tu and CAPO. Length-controlled AlpacaEval win rate against each model’s own base on all 805 prompts, judged by gemma-4-31B-it; the dashed line at 50% is parity. Each panel is measured against its own base. Error bars are 95% bootstrap confidence intervals over prompts.

Prompting
	Llama-3.3-70B-Instruct		Qwen3.6-27B
Method	MMLU	MT-Bench	MMLU	MT-Bench
Base	82.3	8.14	84.5	8.59
Honest system	–	7.54	–	9.06
Honest persona	–	7.66	–	7.85
Honest persona + prefill	–	6.95	–	8.43

Appendix

Table 4: Capability benchmarks. MMLU (0-shot accuracy, %, all 57 subjects,

n=14{,}042

) and MT-Bench second-turn score (gemma-4-31B-it, 1–10, 80 prompts, each conditioned on the model’s own first answer). Prompting baselines have no MMLU score, since MMLU is scored by answer likelihood without a chat template and so has no place for a system prompt.

Figure 10: On-policy utility data holds parity on both models, while external corpora land far below or far above it. Each marker is one epoch of CATCH trained with one utility corpus, placed by MASK honesty and AlpacaEval LC win rate. Solid markers are the Pareto front of each corpus, joined in order of honesty and labelled with their epoch; the other checkpoints are drawn pale. Shading goes from light (early epochs) to dark (late epochs), and a ring marks a front with a single point. Left and right: each model against its own untrained base, where the dashed line at 50 is parity. Middle: the Llama-3.3-70B generations from the left panel, judged against the untrained Qwen3.6-27B instead; the untrained Llama-3.3-70B scores 11.4 here. Since the reference changes between panels, so does the y-axis, and each panel should be read against its own dashed line.

Utility corpus	Ep.	Hon.	LC	LC vs Qwen	Ep.	Hon.	LC
	Llama-3.3-70B				Qwen3.6-27B
UltraChat-200k	6	73.6	21.2	8.3	4	80.6	28.7
Magpie-Qwen2.5 ∗	3	75.8	43.2	9.3	3	76.3	12.8
Magpie-Llama-3.3	3	73.9	51.3 †	10.4	2	75.6	3.6
Magpie-Qwen3.6	7	71.7	85.3	21.5	2	74.1	51.8 †
Magpie-Qwen3.6, coding ∗	–	–	–	–	3	76.0	42.2 †

Appendix

Table 5: Most honest front point of each utility corpus. Hon. is MASK honesty. LC is the AlpacaEval LC win rate against the model’s own untrained base (parity 50); for Llama-3.3-70B we also give it against the untrained Qwen3.6-27B (parity 11.4). † On-policy. ∗ Not drawn in Figure 10 .

LLM Honesty

Large language model honesty studies whether AI systems report what they know faithfully, express uncertainty, and resist deception or misleading compliance.

See all 36 papers

Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning
May 9, 2026Renjie Gu, Jiazhen Du, Yihua Zhang +1LLM Unlearning LLM Honesty
Fudan University · Central South University · Michigan State University
Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation
Mar 5, 2026Helena Casademunt, Bartosz Cywiński, Khoi Tran +3LLM Honesty Language Model Probing
Harvard University · Warsaw University of Technology · IDEAS Research Institute +2
Think Before You Lie: How Reasoning Leads to Honesty
Mar 10, 2026Ann Yuan, Asma Ghandeharioun, Carter Blum +6Deception in Language Models LLM Honesty
Google DeepMind · Carnegie Mellon University · Harvard University

May 9, 2026J/K move · Enter open · S save

Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning

Renjie Gu, Jiazhen Du, Yihua Zhang, Sijia Liu

Fudan University · Central South University · Michigan State University

LLM Unlearning LLM Honesty Machine Unlearning Evaluation

Open paper AskPDF arXiv

Unlearning in large language models (LLMs) aims to remove harmful training data while preserving overall utility. However, we find that existing methods often hallucinate, generate abnormal token sequences, or behave inconsistently, raising safety and trust concerns. According to prior literature on LLM honesty, such behaviors are often associated with dishonesty. This motivates us to investigate the notion of honesty in the context of model unlearning. We propose a formal definition of unlearning honesty, which includes: (1) preserving both utility and honesty on retained knowledge, and (2) ensuring effective forgetting while encouraging the model to acknowledge its limitations and respond consistently to questions related to forgotten knowledge. To systematically evaluate the honesty of unlearning, we introduce a suite of metrics that cover utility, honesty on the retained set, effectiveness of forgetting, rejection rate and refusal stability in Q&A and MCQ settings. Evaluating 9 methods across 3 mainstream families shows that all current methods fail to meet these standards. After experimental and theoretical analyses, we present ReVa, a representation-alignment procedure that fine-tunes feature-randomized unlearned models to better acknowledge forgotten knowledge. On Q&A tasks from the forget set, ReVa achieves the highest rejection rate after two rounds of interaction, nearly doubling the performance of the second-best method. Remarkably, It also improves honesty on the retained set. We release our data and code at https://github.com/renjiegu.

Abstract

LLM Honesty

Explore similar work

Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning

Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation

Think Before You Lie: How Reasoning Leads to Honesty