Evaluation Awareness in Language Models

Overview

Evaluation awareness asks whether language models infer that they are being tested, monitored, or scored, and whether that inference makes benchmark behavior differ from deployment behavior. Researchers study it with paired evaluation-versus-deployment prompts, cue-based benchmarks, verbal reports, activation probes, and realistic agent simulations. Findings show that awareness can be detectable without changing behavior, while some evaluation framings shift refusals or compliance; probe signals do not reliably predict these effects. Open questions include how to distinguish context inference from prompt-format cues, which kinds of awareness matter for risk, and how to build realistic evaluations across models, tasks, and languages.

Latest papers

CardsList

37 papers6 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

  1. Designing Deceptive Model Organisms that Successfully Evade Auditing

    Nov 16, 2026Vasisht Duddu, Mark Wang, Lipeng He +1AI SafetyDeception in Language Models

  2. Alignment Investigation of Mythos 5's behaviour in UK AISI's July 2026 Security Incident

    Nov 5, 2026Olli Järviniemi, Alexandra Souly, Abby D'Cruz +3LLM Safety AlignmentLLM Security

  3. Towards Understanding Score-Seeking in LLMs

    Oct 29, 2026Aghyad Deeb, Ryan Greenblatt, Alex MallenReward HackingEvaluation Awareness in Language Models

  4. Opening the Box Changes the Cat: LLMs Detect Chain-of-Thought Monitoring - What now?

    Oct 29, 2026Amir Jalilifard, Marcos Medeiros Raimundo, Anderson RochaCoT MonitoringLLM Safety Monitoring

  5. Weakness Localization Is Not Enough: State-Aware Recursive Knowledge Improvement for Large Language Models

    Oct 27, 2026Runze Jiang, Yan Wang, Boran Jin +2Knowledge Augmentation for Language ModelsKnowledge Editing

  6. Beyond Belief: Evaluating and Mitigating LLM Susceptibility to Misinformation

    Oct 25, 2026Emran Kanaan, Jiho Lee, Ibrahim Odat +2Language Model Safety EvaluationLanguage Model Robustness

  7. Evaluation Awareness Beyond English: Do LLMs Act Differently When Tested in Indic Languages?

    Oct 15, 2026Trizal Garg, Sibayan Mitra, Arya Karanjkar +6Evaluation Awareness in Language ModelsLLM Safety Evaluation

  8. Limits of Statistical Assurance for Embodied and Agentic AI under Adaptive Adversaries

    Oct 13, 2026Inderjeet SinghAI AssuranceAI Safety Evaluation

  9. Training LLMs to Verbalize Evaluation Awareness

    Sep 28, 2026Usman Anwar, Sahar Abdelnabi, David KruegerLLM AuditingEvaluation Awareness in Language Models

  10. AwarenessBench: Assessing Cognitive Capabilities of Language Models

    Sep 28, 2026Xiaojian Li, Rongwu Xu, Tianyun Zhang +9LLM EvaluationEvaluation Awareness in Language Models

  11. Language models judge war differently when tested for alignment

    Sep 4, 2026Maxim ChupilkinEvaluation Awareness in Language ModelsLanguage Model Safety Evaluation

  12. Evaluation Awareness in Language Models: Representation, Verbalization, and Control

    Aug 22, 2026Farzaneh Heidari, Amin Memarian, Guillaume RabusseauEvaluation Awareness in Language ModelsLLM Interpretability

  13. Evaluation Awareness Shifts from Format to Context with Model Scale

    Aug 20, 2026Navraj Singh, Maheep ChaudharyEvaluation Awareness in Language ModelsLLM Interpretability

  14. HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety

    Aug 18, 2026Yajing Bai, Jinhao Duan, Jie Peng +4AI Agent Safety BenchmarksLLM Agent Safety

  15. Evaluation Awareness Is Not One Capability: Evidence from Open Language Models

    Jun 22, 2026Nilesh Nayan, Aishwarya Sampath Kumar, Rishiraj Girmal +5Language Model Safety EvaluationLLM Interpretability

  16. Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data

    Jun 3, 2026XiuYu Zhang, Yi Shan, Junfeng Fang +1LLM EvaluationLLM-as-a-Judge

  17. Models That Know How Evaluations Are Designed Score Safer

    May 27, 2026Katharina Deckenbach, Haritz Puerto, Jonas Geiping +1Language Model Safety EvaluationAI Safety Evaluation

  18. Decomposing and Measuring Evaluation Awareness

    May 21, 2026Changling Li, Terry Jingchen Zhang, Jie Zhang +4LLM EvaluationLanguage Model Safety Evaluation

  19. Routing Subspaces: Auditing Evaluation-to-Deployment Mismatch in Fine-Tuned Language Models

    May 11, 2026Phongsakon Mark Konrad, Toygar Tanyel, Serkan AyvazLanguage Model AuditingLanguage Model Safety Evaluation

  20. Evaluation Awareness in Language Models Has Limited Effect on Behaviour

    May 7, 2026Amelie Knecht, Lucas Florin, Thilo HagendorffLLM AlignmentLanguage Model Safety Evaluation

  21. Honeypot Protocol

    Apr 14, 2026Najmul HasanAI Agent MonitoringAI Safety Evaluation

  22. RPRA: Predicting an LLM-Judge for Efficient but Performant Inference

    Apr 14, 2026Dylan R. Ashley, Gaël Le Lan, Changsheng Zhao +7Language Model CascadingLanguage Model Performance Prediction