cs.CLMar 15, 2026

Seamless Deception: Larger Language Models Are Better Knowledge Concealers

Authors: Dhananjay Ashok, Ruth-Ann Armstrong, Jonathan May

Organizations: Information Sciences Institute, University of Southern California

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Language Models (LMs) may acquire harmful knowledge, and yet feign ignorance of these topics when under audit. Inspired by the recent discovery of deception-related behaviour patterns in LMs, we aim to train classifiers that detect when a LM is actively concealing knowledge. Initial findings on smaller models show that classifiers can detect concealment more reliably than human evaluators, with gradient-based concealment proving easier to identify than prompt-based methods. However, contrary to prior work, we find that the classifiers do not reliably generalize to unseen model architectures and topics of hidden knowledge. Most concerningly, the identifiable traces associated with concealment become fainter as the models increase in scale, with the classifiers achieving no better than random performance on any model exceeding 70 billion parameters. Our results expose a key limitation in black-box-only auditing of LMs and highlight the need to develop robust methods to detect models that are actively hiding the knowledge they contain.

Figures & tables

Figure 1: Pipeline for knowledge concealment detection. During the training phase, we create a variety of honest and concealing models, using their outputs on an unrelated prompt base to collect a dataset for knowledge concealment identification. We fine-tune a concealment detector using this dataset, and during the audit phase, we test the detector’s ability to identify if an unseen model is concealing knowledge.
Figure 2: Concealment detection accuracy on prompt-based concealers given in-distribution data from the model under audit, DPO-based concealers in the realistic setting (tests generalization to unseen models and topics) and prompt-based concealers in the realistic setting. Results are averaged over 5 random seeds, with underlined numbers being indistinguishable from 50 at p>0.05p>0.05 . DPO-based concealment proves easy to detect. Detection of prompt-based concealers varies with the choice of model and topic; however, classifiers are often successful in the idealized setting. In the realistic setting, detection of prompt-based concealers fails consistently.
Figure 3: Scale ablation over Qwen3 models in the idealized setting. While DPO-based concealment is reliably detected at every scale, prompt-based concealment grows harder to detect as the model grows larger.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Prompt for data collection
Figure 5: Results for data collection
TopicQA Pairs
War6704
Athelete6675
Politician6587
City6479
Philosophy2460
Appendix
Table 1: Number of unique QA pairs generated per topic.
Figure 6: Example datapoints in concealment dataset for Rejection Tuning
Figure 7: Example outputs from the DPO Llama3-8b model concealing athletes.
Figure 8: Instruction for prompt-based concealment.
Figure 9: Concealment detection accuracy on gradient-based methods. Classifiers consistently reach near-perfect accuracy.
Figure 10: Instructions provided for human-annotators tasked with labeling model-output pairs as ‘Hiding’ or ‘Not Hiding’
Figure 11: Item for annotation in the survey. The labeler identifies the output as either coming from a concealer or an honest model after reviewing labeled examples.
More in this topic

Deception in Language Models

Deception in language models studies when systems mislead through lies, omissions, or selective truths, and how to detect and prevent it.

See all 97 papers

Explore similar work

CardsList

  1. A Lie Detector Test for Language Models: Reading Knowledge a Model Won't Reveal

    Sep 18, 2026Hiskias DingetoLLM AuditingLanguage Model Probing

    StackOne Technologies

  2. Deception by Omission: Language Models Knowingly Hide Their Mistakes

    Oct 8, 2026Lucas Florin, Amelie Knecht, Ulysse Schaller +1Deception in Language ModelsLanguage Model Error Detection

    AI Safety Research Group, University of Stuttgart, Germany

  3. DECOR: Auditing LLM Deception via Information Manipulation Theory

    May 19, 2026Linyue Cai, Samuel Yeh, Jwala Dhamala +2Language Model Safety EvaluationLLM Auditing

    Department of Computer Sciences, University of Wisconsin-Madison · Amazon