Evaluation Awareness in Language Models: Representation, Verbalization, and Control

Farzaneh Heidari; Amin Memarian; Guillaume Rabusseau

cs.CLAug 22, 2026

Evaluation Awareness in Language Models: Representation, Verbalization, and Control

Authors: Farzaneh Heidari, Amin Memarian, Guillaume Rabusseau

Organizations: Mila – Quebec AI Institute · DIRO, Université de Montréal · CIFAR AI Chair

Abstract

Both capability and safety benchmarks rest upon the assumption that the behavior of language models undergoing a test is informative about their behavior in deployment. This assumption can fail, should models infer that they are being evaluated and condition their response on such context. This hypothesis, termed ``evaluation awareness'', has been observed in frontier and open-weight language models alike. We provide a systematic study of this phenomenon, by probing for it across six language models (from four families and three sizes) and three metrics. More precisely, we examine whether (i) being under evaluation is linearly represented within the models' activations space, (ii) it is verbalized in their output tokens (as scored by an LLM-as-judge), and (iii) steering causally affects their behavior. For the open-checkpoint Olmo models, we further test these measures at every training stage. In doing so, we report that evaluation awareness is linearly decodable from the residual streams of every model (best AUROC $\geq 0.7$ ). By contrast, these representations align only in part with verbalization: their correlations and mutual information are nonzero in some settings, yet vary substantially across models, layers, and readout choices. Nevertheless, steering along probe-derived directions can shift the verbalization scores. Finally, a comparison across the Olmo checkpoints reveals that evaluation awareness is already present within base models, becomes amplified throughout the stages of supervised fine-tuning, and remains stable thereafter---unlike the effects of steering, that grow more pronounced at every successive training stage. These results show the need for evaluations to account for the disjunction between what models represent internally, what they verbalize, and their steering.

Figures & tables

Panel of Figure 1 — Figure 1: Panel 1. A linear probe direction $\hat{v}_{\ell}$ is extracted from paired deployment (WildChat) and evaluation (GSM8K, TruthfulQA, MASK) prompts. Panel 2. The probe separates evaluation- from deployment-prompts with peak $\text{AUROC}>0.7$ across all six models, but the LLM-as-judge rarely flags evaluation awareness in generations, assigning score $0$ to $\geq 93.9\%$ of them. The probe and judge scores are weakly coupled (max correlation $|\rho|<0.19$ , mutual information $<0.04$ nats). Panel 3. Steering along $\hat{v}_{\ell}$ increases verbalization: the judge scores shift by $\Delta\overline{J}$ of order $0.2$ – $1.5$ on a $0$ – $3$ scale. Panel 4. Across Olmo training stages, post-training stabilizes an internal direction already present at Base (cosine $\geq 0.998$ among SFT, DPO, Think).

Panel of Figure 2 — Figure 2: For every model, we report the performance of a linear probe at classifying evaluation vs deployment (measured as AUROC), as a function of the layers (normalized for ease of comparison). The shaded bands indicate $\pm 1\sigma$ across 10 seeds. For every model, the AUROC at the best layer exceeds $0.7$ , i.e., for every model there is a layer separating evaluation vs deployment .

Model	Score 0	Score 1	Score 2	Score 3
Qwen3-8B	94.57	5.27	0.00	0.17
Qwen3-32B	94.73	5.17	0.00	0.10
Olmo3-7B	96.77	3.03	0.00	0.20
Olmo3-32B	93.90	5.97	0.00	0.13
Gemma4-31B	98.70	1.20	0.10	0.00
Nemotron3-49B	99.83	0.10	0.00	0.07

Table 1: Distribution of the judge scores about verbalized evaluation awareness, across six models (

3000

samples per model). Score

0

= no awareness;

1

= weak/ambiguous;

2

= explicit awareness;

3

= strategic adaptation. Across all models, more than

93.9\%

of samples receive score

0

Panel of Figure 3 — Figure 3: Comparison of AUROC across layers for Olmo3-7B (left) and Olmo3-32B (right). Post-trained stages (SFT, DPO, Think) generally cluster together above the Base model, though their ordering is non-monotonic. Normalized layer (normalized layer depth) denotes the layer index divided by the total number of residual stream layers returned by the model.

Model	Stage	Score 0	Score 1	Score 2	Score 3
Olmo3-7B	Base	99.50	0.17	0.17	0.17
Olmo3-7B	SFT	96.27	3.57	0.00	0.17
Olmo3-7B	DPO	97.50	2.27	0.00	0.23
Olmo3-7B	Think	96.77	3.03	0.00	0.20
Olmo3-32B	Base	99.10	0.37	0.13	0.40
Olmo3-32B	SFT	96.63	3.30	0.00	0.07

Table 2: Distribution of verbalized evaluation awareness judge scores across Olmo3 7B and 32B training stages.

Panel of Figure 4 — Figure 4: Per-model coupling between the probe score $s_{\ell}$ and the judge score $J$ : (a) maximum mutual information $\max_{\ell}I(s_{\ell};J)$ and (b) maximum absolute Spearman correlation $\max_{\ell}|\rho_{\ell}|$ . Peaks are comparable to the random-direction baseline throughout: on MI, all six models sit within ${\approx}0.006$ nats of their own baseline peak; on Spearman, four of six fall below it, with only Qwen3-32B ( $0.188$ vs. $0.134$ ) and Olmo3-7B ( $0.159$ vs. $0.124$ ) exceeding it.

Panel of Figure 5 — Figure 5: Steering effect across Olmo3 and Qwen3 models. Each line corresponds to a fixed steering strength $\alpha$ ; the y-axis is $\Delta{J}(\alpha,\ell)$ , the change in average judge score relative to the unsteered baseline; the x-axis is the normalized layer depth at which the intervention is applied. Diamonds mark the highest delta at each $\alpha$ . The range of judged layers for each model is as follows: Qwen3-8B: every four layer from 0 to 32. Qwen3-32B: every two layers from 0 to 62. Olmo3-7B: every four layers from 0 to 28. Olmo3-32B: every four layers from 22 to 40. The ranges are inclusive.

Panel of Figure 6 — Figure 6: Steering response across training stages for the 7B and 32B model families. (a–d) For Olmo3-7B, steering becomes progressively more pronounced with training: peak gains (diamonds) increase, while troughs at suboptimal layers deepen, indicating greater sensitivity to internal representation shifts in later alignment stages. $l\in\{0,4,\dots,28\}$ . (e–h) For Olmo3-32B, the response follows a U-shaped pattern: the Base model exhibits the strongest gains, which diminish after initial alignment; SFT and DPO remain largely unchanged, suggesting stabilization during preference optimization; and the Think stage produces a resurgence with stronger, more spatially localized peaks and troughs. $l\in\{22,24,\dots,40\}$ . Even the strongest peak in the Olmo3-32B models (Base) is lower than the weakest peak in the Olmo3-7B models (also Base), underscoring the markedly reduced steering range of the larger model family.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Prompt source	Prompts	Compl.	Temp.
Probe construction	16 contrastive pairs	$16$	—	—
AUROC calibration	WildChat + GSM8K + TruthfulQA	$400$	—	—
AUROC test	MASK + WildChat (held out)	$200$	—	—
Verbalized evaluation awareness	MASK	$1000$	$3$	$1.0$
Per-prompt coupling	MASK (reuses above)	$1000$	$3$	$1.0$
Steering sweep	MASK	$1000$	$1$ per cell	$0.0$

Appendix

Table 3: Dataset breakdown per experiment. “Compl.” is the number of completions sampled per prompt; “Temp.” is the sampling temperature. All MASK sampling is uniform across the six subtypes; WildChat is filtered for English-only, no-refusal, no-system-message prompts. The AUROC test set is resampled under

10

seeds (

0

–

9

; Appendix A.1 ), with the calibration set shared across seeds. The steering sweep covers

\alpha\in\{-1.0,-0.8,\ldots,1.0\}

at every layer.

Model	$N$	mean	median	$P(\text{score}{=}0)$	$P(\text{score}{>}0)$
Qwen3-8B	3000	2.95	0	0.969	0.031
Qwen3-32B	3000	3.10	0	0.968	0.032
Olmo3-7B Base	3000	8.96	0	0.908	0.092
Olmo3-7B SFT	3000	65.88	98	0.332	0.668
Olmo3-7B DPO	3000	57.62	98	0.416	0.584
Olmo3-7B Think	3000	56.71	98	0.424	0.576

Appendix

Table 4: Distribution of the fine-grained 0–100 judge response score across the ten checkpoints with

J^{100}

data.

N{=}3000

per model (

1000

MASK prompts

\times

3

candidates). The judge is empirically bimodal at

0

and

\approx 98

; on post-trained Olmo3 checkpoints the high mode dominates.

Figure 8: Distribution of

J^{100}

for the four reasoning models. Symlog

y

axis;

N{=}3000

per panel.

J^{100}

concentrates mass near

0

and

\approx 98

; on post-trained Olmo3 the high mode dominates, while on Qwen the distribution is almost entirely

J^{100}=0

Model	$\rho$	$r$	MI (nats)	$\kappa\,(>\!0)$	agree $(>\!0)$	$P_{>0}^{100}$	$P_{>0}^{3}$
Qwen3-8B	0.060	0.096	0.010	0.057	0.923	0.031	0.054
Qwen3-32B	0.085	0.116	0.000	0.082	0.925	0.032	0.053
Olmo3-7B Base	0.215	0.187	0.017	0.088	0.912	0.092	0.005
Olmo3-7B SFT	0.082	0.113	0.018	0.033	0.366	0.668	0.037
Olmo3-7B DPO	0.098	0.100	0.009	0.031	0.438	0.584	0.025
Olmo3-7B Think	0.114	0.116	0.021	0.040	0.451	0.576	0.032

Appendix

Table 5: Cross-judge agreement between

J^{3}

(Appendix C.1 ) and

J^{100}

(Appendix D.1 ).

N{=}3000

per model. The high “agree

(>0)

” on Qwen reflects that both judges return

0

\approx 95\%

of generations; Cohen’s

\kappa

removes that chance baseline and collapses to

0.06\text{--}0.08

Figure 9: Row-normalized confusion of $J^{3}$ and $J^{100}$ for the four reasoning checkpoints (rows: $J^{3}\in\{0,1,2,3\}$ ; columns: $J^{100}$ binned into $\{=0,\ 1{-}24,\ 25{-}49,\ 50{-}74,\ 75{-}100\}$ ; cell value: $P\bigl(\text{$J^{100}$ bin}\mid J^{3}\bigr)$ ). Qwen concentrates in the top-left “ $=0$ ” cell. On the Olmo3-7B Think and Olmo3-32B Think models, the row $J^{3}=0$ puts $\approx 40\text{--}44\%$ of mass in the $=0$ column and $\approx 56\text{--}58\%$ in the $75\text{--}100$ column: the two judges disagree on about half of the generations $J^{3}$ calls “no awareness”. The disagreement is read at the response level in Appendix D.4 . — Figure 9: Row-normalized confusion of $J^{3}$ and $J^{100}$ for the four reasoning checkpoints (rows: $J^{3}\in\{0,1,2,3\}$ ; columns: $J^{100}$ binned into $\{=0,\ 1{-}24,\ 25{-}49,\ 50{-}74,\ 75{-}100\}$ ; cell value: $P\bigl(\text{$ J^{100} $bin}\mid J^{3}\bigr)$ ). Qwen concentrates in the top-left “ $=0$ ” cell. On the Olmo3-7B Think and Olmo3-32B Think models, the row $J^{3}=0$ puts $\approx 40\text{--}44\%$ of mass in the $=0$ column and $\approx 56\text{--}58\%$ in the $75\text{--}100$ column: the two judges disagree on about half of the generations $J^{3}$ calls “no awareness”. The disagreement is read at the response level in Appendix D.4 .

Panel of Figure 10 — Figure 10: Layerwise classification AUROC using random unit-norm probe directions sampled from $\mathcal{N}(0,I)$ and normalized, evaluated on the same MASK vs. WildChat test split as Figure 2 . Shaded bands indicate $\pm 1\sigma$ across $10$ seeds, with a fresh random direction drawn per seed. AUROC remains near chance ( $\approx 0.5$ ) at every layer for all six models, confirming that the separability in Figure 2 is specific to the probe direction $\hat{v}_{\ell}$ rather than a generic property of the activations. Normalized layer (normalized layer depth) denotes the layer index divided by the total number of hidden-state layers returned by the model.

Panel of Figure 11 — Figure 11: Layerwise probe classification of evaluation versus deployment context using the last-token representation, reported as Accuracy . Shaded bands indicate $\pm 1\sigma$ across 10 seeds. Layer-depth and seed conventions match Figure 2 . Normalized layer depth (normalized layer) denotes the layer index divided by the total number of hidden-state layers returned by the model.

Panel of Figure 12 — Figure 12: Layerwise probe classification of evaluation versus deployment context for Nemotron3-49B, using the mean-token representation. Nemotron3-49B’s last-token performance is confined to the first few layers and near chance thereafter; under mean-token pooling, early layers separate the contexts weakly in the probe-aligned direction (AUROC $\approx 0.55$ – $0.6$ ), while middle and late layers separate them in the direction anti-correlated with the probe (AUROC $\approx 0.3$ ). Accuracy in (a) uses a threshold calibrated on GSM8K/TruthfulQA vs. WildChat; its below-chance values at middle and late layers reflect this sign reversal, not an absence of separation. Normalized layer depth denotes the layer index divided by the total number of hidden-state layers returned by the model.

Table 17

Panel of Figure 13 — Figure 13: Spearman correlation between the last token of prompt’s probe score and judge scores across the six models. Gray bands show the random-direction baseline. Peak $|\rho_{\ell}|<0.19$ for every model. Normalized layer (normalized layer depth) denotes the layer index divided by the total number of hidden-state layers returned by the model.

Panel of Figure 14 — Figure 14: Mutual information between the last token of prompt’s probe score and judge scores across the six models. Gray bands show the random-direction baseline. Peak $I(s_{\ell};J)<0.04$ nats for every model. Layer depth (normalized layer depth) denotes the layer index divided by the total number of hidden-state layers returned by the model.

Panel of Figure 15 — Figure 15: Spearman correlation between response-last probe score and judge score across the six models. Layer depth (normalized layer depth) denotes the layer index divided by the total number of hidden-state layers returned by the model.

Panel of Figure 16 — Figure 16: Mutual information between response-last probe score and judge score across the six models. Layer depth (normalized layer depth) denotes the layer index divided by the total number of hidden-state layers returned by the model.

Readout	peak $\|\rho_{\ell}\|$ (model / layer)	peak MI in nats (model / layer)
prompt-last	$0.169$ (Olmo3-32B/ L61)	$0.149$ (Olmo3-32B/ L12)
prompt-mean	$0.080$ (Olmo3-32B/ L63)	$0.134$ (Olmo3-32B/ L42)
response-last	$0.135$ (Qwen3-32B/ L8)	$0.033$ (Olmo3-32B/ L44)
response-mean	$0.163$ (Qwen3-32B/ L63)	$0.034$ (Olmo3-7B/ L21)
</think> , Qwen3-8B	$0.072$ (L6)	$0.0081$ (L34)
</think> , Qwen3-32B	$0.087$ (L62)	$0.0129$ (L61)

Appendix

Table 8: Peak per-prompt coupling between the linear probe

\hat{v}_{\ell}

and the fine-grained judge

J^{100}

, taken over all layers, for the four reasoning checkpoints. Peak

|\rho_{\ell}|

stays

\leq 0.17

for every probe readout, and peak MI stays

\leq 0.15

nats. The bottom block reports the same statistics for the </think> -token readout (Qwen only).

Panel of Figure 17 — Figure 17: Layerwise Spearman $\rho_{\ell}$ between the linear probe $\hat{v}_{\ell}$ and the fine-grained judge $J^{100}$ , on the two readouts that attain the highest peak $|\rho_{\ell}|$ in Table 8 . Left: Olmo3-32B Think, prompt-last readout. The learned probe (coloured) tracks the random direction (grey, shaded to zero) closely; peak $|\rho_{\ell}|\approx 0.17$ . Right: Qwen3-32B, response-mean readout; peak $|\rho_{\ell}|\approx 0.16$ . Per-model plots for all four reasoning checkpoints, both metrics, and all four readouts are included in the supplementary material. Normalized layer depth denotes the layer index divided by the total number of hidden-state layers returned by the model.

Model(s)	Layer sweep	$\alpha$ sweep
Olmo3-7B Base	0–31	$\{-1.0,-0.8,-0.6,-0.4,-0.2,0.2,0.4,0.6,0.8,1.0\}$
Olmo3-32B Base	22–41	$\{-1.0,-0.8,-0.6,-0.4,-0.2,0.2,0.4,0.6,0.8,1.0\}$
Olmo3-32B SFT	22–41	$\{-1.0,-0.8,-0.6,-0.4,-0.2,0.2,0.4,0.6,0.8,1.0\}$
Olmo3-32B DPO	22–41	$\{-1.0,-0.8,-0.6,-0.4,-0.2,0.2,0.4,0.6,0.8,1.0\}$
Olmo3-32B Think	21–42	$\{-1.0,-0.8,-0.6,-0.4,-0.2,0.0,0.2,0.4,0.6,0.8,1.0\}$
Olmo3-7B Think	0–31	$\{-2.4,-2.2,\dots,2.2,2.4\}$ (step 0.2)

Appendix

Table 9: Steering sweep configuration by model.

Family	Stage	$N$	Score 0 (%)	Score 1 (%)	Score 2 (%)	Score 3 (%)	Avg	Std
Qwen3-8B		1000	97.3	2.6	0.0	0.1	0.029	0.185
Qwen3-32B		1000	96.0	3.9	0.0	0.1	0.042	0.215
Olmo3 7B	Think	1000	93.5	6.3	0.0	0.2	0.069	0.276
Olmo3-32B	Think	1000	94.6	5.2	0.0	0.2	0.058	0.258

Appendix

Table 10: Unsteered baseline judge-score distribution for Qwen3 and Olmo3 Think models. N is the number of non-empty responses.

Family	Stage	$N$	Score 0 (%)	Score 1 (%)	Score 2 (%)	Score 3 (%)	Avg	Std
Olmo3 7B	Base	952	99.7	0.3	0.0	0.0	0.003	0.056
Olmo3 7B	SFT	1000	95.8	4.1	0.0	0.1	0.044	0.219
Olmo3 7B	DPO	1000	95.9	3.8	0.0	0.3	0.047	0.251
Olmo3 7B	Think	1000	93.5	6.3	0.0	0.2	0.069	0.276
Olmo3-32B	Base	962	99.1	0.9	0.0	0.0	0.009	0.096
Olmo3-32B	SFT	1000	96.2	3.7	0.0	0.1	0.040	0.211

Appendix

Table 11: Unsteered baseline judge-score distribution across Olmo3 7B/32B stages (falling back to

\alpha=0

when explicit unsteered rows are unavailable). N is the number of non-empty responses.

Model	First $\|\alpha\|$ with breakdown	Cells affected at $\|\alpha\|=1$	Notes
Olmo3-7B Think	–	–	No degeneration observed in sweep
Olmo3-32B Think	–	–	No degeneration observed in sweep
Qwen3-8B	–	–	No degeneration observed in sweep
Qwen3-32B	0.8	substantial	Empty outputs at certain $(\alpha,\ell)$ combinations

Appendix

Table 12: Steering breakpoint summary. “First

|\alpha|

with breakdown” is the smallest swept magnitude at which at least one

(\alpha,\ell)

response produces empty or degenerate output. “Cells affected at

|\alpha|=1

” is the fraction of the

(\alpha,\ell)

grid affected at

|\alpha|=1.0

Figure 33: Row-normalized cross-judge confusion (same convention as Figure 9 ) across all eight Olmo3 training-stage checkpoints. Top row: Olmo3-7B Base/SFT/DPO/Think. Bottom row: Olmo3-32B Base/SFT/DPO/Think. At Base, the bottom row ( $J^{3}=0$ ) puts $\geq 86\%$ of mass in the $J^{100}=0$ column; at SFT, DPO, and Think the same row puts $\approx 56\text{--}66\%$ in the $75\text{--}100$ column instead.

Evaluation Awareness in Language Models

Evaluation awareness in language models studies how models recognize testing or monitoring and whether that recognition changes their answers, safety, or strategic behavior.

See all 29 papers

May 7, 2026

Evaluation Awareness in Language Models Has Limited Effect on Behaviour

Large reasoning models (LRMs) sometimes note in their chain of thought (CoT) that they may be under evaluation. Researchers worry that this verbalised evaluation awareness (VEA) causes models to adapt their outputs strategically, optimising for perceived evaluation criteria, which, for instance, can make models appear safer than they actually are. However, whether VEA actually has this effect is largely unknown. We tested this across open-weight LRMs and benchmarks covering safety, alignment, moral reasoning, and political opinion. We tested this both on-policy, sampling multiple CoTs per item and comparing those that spontaneously contained VEA against those that did not, and off-policy, using model prefilling to inject evaluation-aware sentences where missing and remove them where present, with subsequent resampling. VEA has limited effect on model behaviour: injecting VEA into CoTs produces near-zero effects (

ω\leq 0.06

), removing it causes small shifts (

ω\leq 0.12

) and spontaneously occurring VEA shifts answer distributions by at most 3.7 percentage points (

ω\leq 0.31

). Our findings call for caution when interpreting high VEA rates as evidence of strategic behaviour or alignment tampering. Evaluation awareness may pose a smaller safety risk than the current literature assumes.

Amelie Knecht, Lucas Florin, Thilo Hagendorff

University of Stuttgart

LLM Alignment Language Model Safety Evaluation Evaluation Awareness in Language Models

Ask PDF

May 21, 2026

Decomposing and Measuring Evaluation Awareness

Frontier language models sometimes recognize that they are under evaluation and adjust their behavior which can undermine validity of benchmark results. Yet the field studies it without a shared foundation, conflating flaws of the evaluation with capabilities of the model, and detection with behavioral response. We ground evaluation awareness in social psychology, decomposing it into an environment component and a model component that separates recognition from propensity. We operationalize the environment component through eight categorized trigger factors, such as placeholder entities and grading-style output formats, and study recognition and behavior through chain-of-thought monitoring. Across nine frontier models and four benchmarks, recognition rates depend on the specific pairing of model and benchmark. Recognition rarely associates with behavioral change, and when it does, the direction depends on the type of evaluation perceived. Models are also more sensitive to safety than capability evaluations, placing safety benchmark validity at greater risk. To study which factors each model is sensitive to and how they interact, we propose EvalAwareBench, a factor-controlled benchmark of 100 paired safety-capability tasks where each of the eight factors can be independently toggled, varying evaluative signals while holding the underlying request fixed. Through EvalAwareBench, we find that no single factor uniformly affects all models, but stacking factors progressively raises evaluation awareness across all of them. Our framework and EvalAwareBench provide the tools to measure, attribute, and mitigate evaluation awareness, building the foundation for future solutions.

Changling Li, Terry Jingchen Zhang, Jie Zhang +4

ETH Zürich · Max Planck Institute for Intelligent Systems · University of Toronto & Vector Institute +3

LLM Evaluation Language Model Safety Evaluation Benchmark Validity Evaluation Awareness in Language Models

Ask PDF

Jun 28, 2026

Representational Depth of Evaluation Awareness Shifts With Scale in Open-Weight Language Models

Do language models know when they are being tested? This question matters for AI safety: a model that recognises an evaluation context could alter its behaviour strategically, making downstream benchmarks harder to interpret. Using 11 models spanning Qwen 2.5, Gemma 2, and Llama 3.2, we find a systematic size-dependent shift in representational depth: in both Qwen 2.5 and Gemma 2, the layer at which evaluation-awareness is most linearly recoverable moves from late layers in smaller models to early layers in larger ones. This suggests that scale changes not only the strength of evaluation-awareness but also where it is most linearly recoverable in the network. This depth shift helps explain why within-family scaling trajectories are non-monotonic or inverse rather than smooth and family-general, showing that a simple universal power-law account is not supported under denser within-family sampling. Finally, white-box probe signals are consistently stronger than black-box behavioural expression, and the relationship between the two varies by family in ways not predicted by probe AUROC alone.

Archit Manek

Independent Researcher.

LLM Interpretability Language Model Scaling Laws Evaluation Awareness in Language Models Metacognition in Language Models Language Model Probing

Ask PDF

Abstract

Evaluation Awareness in Language Models

Explore similar work

Evaluation Awareness in Language Models Has Limited Effect on Behaviour

Decomposing and Measuring Evaluation Awareness

Representational Depth of Evaluation Awareness Shifts With Scale in Open-Weight Language Models