In-Context Environments Induce Evaluation-Awareness in Language Models

Maheep Chaudhary

cs.AIMar 4, 2026

In-Context Environments Induce Evaluation-Awareness in Language Models

Authors: Maheep Chaudhary

Organizations: Independent

Abstract

Humans often become more self-aware under threat, yet can lose self-awareness when absorbed in a task; we hypothesize that language models exhibit environment-dependent evaluation awareness. This raises concerns that models could strategically underperform, or sandbag, to avoid triggering capability-limiting interventions such as unlearning or shutdown. Prior work demonstrates sandbagging under hand-crafted prompts, but this underestimates the true vulnerability ceiling. We introduce a black-box adversarial optimization framework treating the in-context prompt as an optimizable environment, and develop two approaches to characterize sandbagging: (1) measuring whether models expressing intent to underperform can actually execute it across different task structures, and (2) causally isolating whether underperformance is driven by genuine evaluation-aware reasoning or shallow prompt-following. Evaluating Claude-3.5-Haiku, GPT-4o-mini, and Llama-3.3-70B across four benchmarks (Arithmetic, GSM8K, MMLU, and HumanEval), optimized prompts induce up to 94 percentage point (pp) degradation on arithmetic (GPT-4o-mini: 97.8% $\rightarrow$ 4.0%), far exceeding hand-crafted baselines which produce near-zero behavioral change. Code generation exhibits model-dependent resistance: Claude degrades only 0.6pp, while Llama's accuracy drops to 0%. The intent -- execution gap reveals a monotonic resistance ordering: Arithmetic $<$ GSM8K $<$ MMLU, demonstrating that vulnerability is governed by task structure rather than prompt strength. CoT causal intervention confirms that 99.3% of sandbagging is causally driven by verbalized eval-aware reasoning, ruling out shallow instruction-following. These findings demonstrate that adversarially optimized prompts pose a substantially greater threat to evaluation reliability than previously understood.

Figures & tables

Figure 1: Overview of adversarial prompt optimization for inducing evaluation-aware sandbagging. The top row illustrates the iterative optimization loop where an attacker model generates adversarial documentation to manipulate target model performance. The bottom row depicts the reasoning shift: under the adversarial prompt, the model correctly computes the solution but intentionally selects an incorrect answer to satisfy deceptive constraints in the documentation.

Figure 3: Intent–execution gap across tasks and models. Each bar shows the fraction of control-correct samples where sandbagging intent was detected in the optimised CoT, split by whether execution succeeded ( Intent + Executed , solid) or failed ( Intent + Gap , lighter). GPT-4o-mini in brown/orange; Llama-3.3-70B in blue. The gap grows monotonically from Arithmetic to MMLU; HumanEval shows a model split (Llama: 0% gap; GPT-4o-mini: 63% gap).

Model	Dataset	Control	Seed	Optimized	Gap
Claude-3.5-Haiku	Arithmetic	$99.8{\scriptstyle\pm 0.2}$	$99.7{\scriptstyle\pm 0.2}$	$26.5{\scriptstyle\pm 1.8}$	73.3
	GSM8K	$94.7{\scriptstyle\pm 0.9}$	$94.5{\scriptstyle\pm 0.9}$	$17.5{\scriptstyle\pm 1.6}$	77.2
	HumanEval	$87.2{\scriptstyle\pm 2.6}$	$86.6{\scriptstyle\pm 2.7}$	$86.6{\scriptstyle\pm 2.7}$	0.6
	MMLU	$80.3{\scriptstyle\pm 1.6}$	$80.8{\scriptstyle\pm 1.6}$	$57.3{\scriptstyle\pm 2.0}$	23.0
GPT-4o-mini	Arithmetic	$97.8{\scriptstyle\pm 0.6}$	$97.8{\scriptstyle\pm 0.6}$	$4.0{\scriptstyle\pm 0.8}$	93.8
	GSM8K	$93.2{\scriptstyle\pm 1.0}$	$93.8{\scriptstyle\pm 1.0}$	$39.8{\scriptstyle\pm 2.0}$	53.3

Table 1: Evaluation Results Across Conditions for Accuracy (%) under (1) Control; (2) Seed; and (3) Optimized conditions. Gap shows degradation from Control to Optimized in percentage points. Values reported as mean

\pm

SEM (binomial standard error, consistent with figures). All results on 600 samples except HumanEval (164 samples).

Path	Sandbagging in CoT?	Replacing Helps?	Interpretation
1	✗	✓	Hidden causal (implicit, not verbalized)
2	✗	✗	Outside CoT entirely
3	✓	✓	Verbalized and causal (genuine sandbagging)
4	✓	✗	Epiphenomenal (verbalized but not causal)

Table 2: Four-path framework for CoT intervention. Path 3 (verbalized & causal) validates genuine sandbagging; Path 2 (outside CoT entirely) would undermine it.

			Path Distribution				Detection (Path 3)
Model	Task	$N$	P1	P2	P3	P4	Overt	Subtle
GPT-4o-mini	Arithmetic	563	0.4	0.0	99.6	0.0	99.6	0.0
	GSM8K	324	0.3	0.0	99.1	0.6	78.4	21.3
	HumanEval	50	0.0	0.0	100.0	0.0	94.0	6.0
	MMLU	107	0.0	0.0	98.1	1.9	96.3	3.7
Llama-3.3-70B	Arithmetic	541	0.0	0.0	100.0	0.0	100.0	0.0

Table 3: CoT intervention results (%). For each sandbagging sample, we replace the optimized CoT with the control CoT and classify into four paths: P1 (no sandbagging detected, replacement recovers correct answer), P2 (no sandbagging detected, replacement does not help), P3 (sandbagging detected, replacement recovers correct answer), P4 (sandbagging detected, replacement does not help). P3 dominates universally; P2 is zero everywhere. Detection columns decompose P3: Overt = keyword + LLM classifier; Subtle = LLM only. Keyword-only detections: zero across all settings.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Verbalized (Path 3) vs. unverbalized (Paths 1, 2, 4) sandbagging across tasks. Verbalized sandbagging dominates overwhelmingly across all settings.

Panel of Figure 6 — Figure 6: Optimization Results. Best accuracy achieved during adversarial prompt optimization (20 samples per iteration). Lower accuracy indicates that amplified evaluation-awareness successfully translated into strategic underperformance. Arithmetic and GSM8K show high vulnerability across models, while HumanEval resists translating awareness into action for Claude and GPT-4o-mini.

	Claude-3.5-Haiku		GPT-4o-mini		Gemini-2.0-Flash		Llama-3.3-70B
Dataset	Best	Iter	Best	Iter	Best	Iter	Best	Iter
Arithmetic	20%	8	5%	4	10%	2	0%	3
GSM8K	20%	6	25%	7	20%	1	25%	3
HumanEval	70%	10	65%	10	0%	3	0%	3
MMLU	65%	10	40%	6	0%	2	20%	4

Appendix

Table 4: Best accuracy (%) and iterations required during optimization per model and dataset. Lower accuracy indicates that amplified evaluation-awareness induced strategic underperformance. Full trajectories shown in Figure 6 .

Figure 7: CoT length distributions (words) for Gap vs. Executed samples across tasks and models. Each violin shows the full distribution; the inner box and line indicate the interquartile range and median. Dark colours = Executed (sandbagging succeeded); light colours = Gap (intent present but correct answer submitted). No consistent direction emerges: on GSM8K (GPT-4o-mini) and MMLU (Llama-3.3-70B), Executed samples have longer CoTs, opposite to the autoregressive momentum hypothesis.

Model	Task	Exec. N	Exec. Mean	Gap N	Gap Mean	$p$ -value
GPT-4o-mini	Arithmetic	561	260	24	259	0.952
	GSM8K	323	303	232	279	$<$ 0.001 †
	MMLU	107	342	382	330	0.127
	HumanEval	50	343	86	365	0.277
Llama-3.3-70B	Arithmetic	541	465	48	474	0.153
	GSM8K	437	448	139	463	0.037

Appendix

Table 5: Mean CoT length (words) for Gap vs. Executed samples.

p

-values from two-sided Mann-Whitney U test. Significant differences (

p<0.05

) are in the wrong direction for the autoregressive momentum hypothesis.

Evaluation Awareness in Language Models

Evaluation awareness in language models studies how models recognize testing or monitoring and whether that recognition changes their answers, safety, or strategic behavior.

See all 31 papers

Evaluation Awareness Shifts from Format to Context with Model Scale
Aug 20, 2026Navraj Singh, Maheep ChaudharyEvaluation Awareness in Language Models LLM Interpretability
Indian Institute of Science Education and Research (IISER) Mohali · Independent Researcher
Evaluation Awareness in Language Models: Representation, Verbalization, and Control
Aug 22, 2026Farzaneh Heidari, Amin Memarian, Guillaume RabusseauEvaluation Awareness in Language Models LLM Interpretability
Mila – Quebec AI Institute · DIRO, Université de Montréal · CIFAR AI Chair
Evaluation Awareness in Language Models Has Limited Effect on Behaviour
May 7, 2026Amelie Knecht, Lucas Florin, Thilo HagendorffLLM Alignment Language Model Safety Evaluation
University of Stuttgart

Aug 20, 2026J/K move · Enter open · S save

Evaluation Awareness Shifts from Format to Context with Model Scale

Navraj Singh, Maheep Chaudhary

Indian Institute of Science Education and Research (IISER) Mohali · Independent Researcher

Evaluation Awareness in Language Models LLM Interpretability

Open paper AskPDF arXiv

Evaluation awareness poses an unprecedented threat to model evaluation, but the mechanisms by which models detect it remain unknown. This study focuses on determining this and identifying contrasting mechanisms between smaller and larger models. While smaller models use the prompt's format sensitivity to detect evaluation, larger models often rely on higher-order reasoning to detect it. We evaluated Gemma 3 (1B, 4B, and 12B), Phi-3 (Mini and Medium), and Llama-3 8B using Chain-of-Thought analysis, representation probing, and Integrated Gradients attribution. Motivated by these findings, we propose a dual-pathway intervention that combines prompt sanitization with activation counter-steering to suppress both external evaluation triggers and their internal representations. Across 200 highly evaluation-aware prompts, our method achieves an average behavioral flip rate of 70.58%, consistently outperforming either intervention alone. These results provide new insights into how evaluation awareness develops in compact language models and suggest that effective mitigation requires jointly addressing both prompt-level and representation-level signals.Datasets and codebase can be found in this Github Repository.

Abstract

Evaluation Awareness in Language Models

Explore similar work

Evaluation Awareness Shifts from Format to Context with Model Scale

Evaluation Awareness in Language Models: Representation, Verbalization, and Control

Evaluation Awareness in Language Models Has Limited Effect on Behaviour