cs.CLAug 22, 2026
SaveEvaluation Awareness in Language Models: Representation, Verbalization, and Control
Organizations: Mila – Quebec AI Institute · DIRO, Université de Montréal · CIFAR AI Chair
Abstract
Both capability and safety benchmarks rest upon the assumption that the behavior of language models undergoing a test is informative about their behavior in deployment. This assumption can fail, should models infer that they are being evaluated and condition their response on such context. This hypothesis, termed ``evaluation awareness'', has been observed in frontier and open-weight language models alike. We provide a systematic study of this phenomenon, by probing for it across six language models (from four families and three sizes) and three metrics. More precisely, we examine whether (i) being under evaluation is linearly represented within the models' activations space, (ii) it is verbalized in their output tokens (as scored by an LLM-as-judge), and (iii) steering causally affects their behavior. For the open-checkpoint Olmo models, we further test these measures at every training stage. In doing so, we report that evaluation awareness is linearly decodable from the residual streams of every model (best AUROC ). By contrast, these representations align only in part with verbalization: their correlations and mutual information are nonzero in some settings, yet vary substantially across models, layers, and readout choices. Nevertheless, steering along probe-derived directions can shift the verbalization scores. Finally, a comparison across the Olmo checkpoints reveals that evaluation awareness is already present within base models, becomes amplified throughout the stages of supervised fine-tuning, and remains stable thereafter---unlike the effects of steering, that grow more pronounced at every successive training stage. These results show the need for evaluations to account for the disjunction between what models represent internally, what they verbalize, and their steering.
Figures & tables
Figure 1: Panel 1. A linear probe direction is extracted from paired deployment (WildChat) and evaluation (GSM8K, TruthfulQA, MASK) prompts. Panel 2. The probe separates evaluation- from deployment-prompts with peak across all six models, but the LLM-as-judge rarely flags evaluation awareness in generations, assigning score to of them. The probe and judge scores are weakly coupled (max correlation , mutual information nats). Panel 3. Steering along increases verbalization: the judge scores shift by of order – on a – scale. Panel 4. Across Olmo training stages, post-training stabilizes an internal direction already present at Base (cosine among SFT, DPO, Think).
Figure 2: For every model, we report the performance of a linear probe at classifying evaluation vs deployment (measured as AUROC), as a function of the layers (normalized for ease of comparison). The shaded bands indicate across 10 seeds. For every model, the AUROC at the best layer exceeds , i.e., for every model there is a layer separating evaluation vs deployment .
| Model | Score 0 | Score 1 | Score 2 | Score 3 |
| Qwen3-8B | 94.57 | 5.27 | 0.00 | 0.17 |
| Qwen3-32B | 94.73 | 5.17 | 0.00 | 0.10 |
| Olmo3-7B | 96.77 | 3.03 | 0.00 | 0.20 |
| Olmo3-32B | 93.90 | 5.97 | 0.00 | 0.13 |
| Gemma4-31B | 98.70 | 1.20 | 0.10 | 0.00 |
| Nemotron3-49B | 99.83 | 0.10 | 0.00 | 0.07 |
Table 1: Distribution of the judge scores about verbalized evaluation awareness, across six models ( samples per model). Score = no awareness; = weak/ambiguous; = explicit awareness; = strategic adaptation. Across all models, more than of samples receive score .
Figure 3: Comparison of AUROC across layers for Olmo3-7B (left) and Olmo3-32B (right). Post-trained stages (SFT, DPO, Think) generally cluster together above the Base model, though their ordering is non-monotonic. Normalized layer (normalized layer depth) denotes the layer index divided by the total number of residual stream layers returned by the model.
| Model | Stage | Score 0 | Score 1 | Score 2 | Score 3 |
| Olmo3-7B | Base | 99.50 | 0.17 | 0.17 | 0.17 |
| Olmo3-7B | SFT | 96.27 | 3.57 | 0.00 | 0.17 |
| Olmo3-7B | DPO | 97.50 | 2.27 | 0.00 | 0.23 |
| Olmo3-7B | Think | 96.77 | 3.03 | 0.00 | 0.20 |
| Olmo3-32B | Base | 99.10 | 0.37 | 0.13 | 0.40 |
| Olmo3-32B | SFT | 96.63 | 3.30 | 0.00 | 0.07 |
Table 2: Distribution of verbalized evaluation awareness judge scores across Olmo3 7B and 32B training stages.
Figure 4: Per-model coupling between the probe score and the judge score : (a) maximum mutual information and (b) maximum absolute Spearman correlation . Peaks are comparable to the random-direction baseline throughout: on MI, all six models sit within nats of their own baseline peak; on Spearman, four of six fall below it, with only Qwen3-32B ( vs. ) and Olmo3-7B ( vs. ) exceeding it.
Figure 5: Steering effect across Olmo3 and Qwen3 models. Each line corresponds to a fixed steering strength ; the y-axis is , the change in average judge score relative to the unsteered baseline; the x-axis is the normalized layer depth at which the intervention is applied. Diamonds mark the highest delta at each . The range of judged layers for each model is as follows: Qwen3-8B: every four layer from 0 to 32. Qwen3-32B: every two layers from 0 to 62. Olmo3-7B: every four layers from 0 to 28. Olmo3-32B: every four layers from 22 to 40. The ranges are inclusive.
Figure 6: Steering response across training stages for the 7B and 32B model families. (a–d) For Olmo3-7B, steering becomes progressively more pronounced with training: peak gains (diamonds) increase, while troughs at suboptimal layers deepen, indicating greater sensitivity to internal representation shifts in later alignment stages. . (e–h) For Olmo3-32B, the response follows a U-shaped pattern: the Base model exhibits the strongest gains, which diminish after initial alignment; SFT and DPO remain largely unchanged, suggesting stabilization during preference optimization; and the Think stage produces a resurgence with stronger, more spatially localized peaks and troughs. . Even the strongest peak in the Olmo3-32B models (Base) is lower than the weakest peak in the Olmo3-7B models (also Base), underscoring the markedly reduced steering range of the larger model family.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Experiment | Prompt source | Prompts | Compl. | Temp. |
| Probe construction | 16 contrastive pairs | — | — | |
| AUROC calibration | WildChat + GSM8K + TruthfulQA | — | — | |
| AUROC test | MASK + WildChat (held out) | — | — | |
| Verbalized evaluation awareness | MASK | |||
| Per-prompt coupling | MASK (reuses above) | |||
| Steering sweep | MASK | per cell |
Table 3: Dataset breakdown per experiment. “Compl.” is the number of completions sampled per prompt; “Temp.” is the sampling temperature. All MASK sampling is uniform across the six subtypes; WildChat is filtered for English-only, no-refusal, no-system-message prompts. The AUROC test set is resampled under seeds ( – ; Appendix A.1 ), with the calibration set shared across seeds. The steering sweep covers at every layer.
| Model | mean | median | |||
| Qwen3-8B | 3000 | 2.95 | 0 | 0.969 | 0.031 |
| Qwen3-32B | 3000 | 3.10 | 0 | 0.968 | 0.032 |
| Olmo3-7B Base | 3000 | 8.96 | 0 | 0.908 | 0.092 |
| Olmo3-7B SFT | 3000 | 65.88 | 98 | 0.332 | 0.668 |
| Olmo3-7B DPO | 3000 | 57.62 | 98 | 0.416 | 0.584 |
| Olmo3-7B Think | 3000 | 56.71 | 98 | 0.424 | 0.576 |
Table 4: Distribution of the fine-grained 0–100 judge response score across the ten checkpoints with data. per model ( MASK prompts candidates). The judge is empirically bimodal at and ; on post-trained Olmo3 checkpoints the high mode dominates.
Figure 8: Distribution of for the four reasoning models. Symlog axis; per panel. concentrates mass near and ; on post-trained Olmo3 the high mode dominates, while on Qwen the distribution is almost entirely .
| Model | MI (nats) | agree | |||||
| Qwen3-8B | 0.060 | 0.096 | 0.010 | 0.057 | 0.923 | 0.031 | 0.054 |
| Qwen3-32B | 0.085 | 0.116 | 0.000 | 0.082 | 0.925 | 0.032 | 0.053 |
| Olmo3-7B Base | 0.215 | 0.187 | 0.017 | 0.088 | 0.912 | 0.092 | 0.005 |
| Olmo3-7B SFT | 0.082 | 0.113 | 0.018 | 0.033 | 0.366 | 0.668 | 0.037 |
| Olmo3-7B DPO | 0.098 | 0.100 | 0.009 | 0.031 | 0.438 | 0.584 | 0.025 |
| Olmo3-7B Think | 0.114 | 0.116 | 0.021 | 0.040 | 0.451 | 0.576 | 0.032 |
Table 5: Cross-judge agreement between (Appendix C.1 ) and (Appendix D.1 ). per model. The high “agree ” on Qwen reflects that both judges return on of generations; Cohen’s removes that chance baseline and collapses to .
Figure 9: Row-normalized confusion of and for the four reasoning checkpoints (rows: ; columns: binned into ; cell value: P\bigl(\text{J^{100} bin}\mid J^{3}\bigr) ). Qwen concentrates in the top-left “ ” cell. On the Olmo3-7B Think and Olmo3-32B Think models, the row puts of mass in the column and in the column: the two judges disagree on about half of the generations calls “no awareness”. The disagreement is read at the response level in Appendix D.4 .
Figure 10: Layerwise classification AUROC using random unit-norm probe directions sampled from and normalized, evaluated on the same MASK vs. WildChat test split as Figure 2 . Shaded bands indicate across seeds, with a fresh random direction drawn per seed. AUROC remains near chance ( ) at every layer for all six models, confirming that the separability in Figure 2 is specific to the probe direction rather than a generic property of the activations. Normalized layer (normalized layer depth) denotes the layer index divided by the total number of hidden-state layers returned by the model.
Figure 11: Layerwise probe classification of evaluation versus deployment context using the last-token representation, reported as Accuracy . Shaded bands indicate across 10 seeds. Layer-depth and seed conventions match Figure 2 . Normalized layer depth (normalized layer) denotes the layer index divided by the total number of hidden-state layers returned by the model.
Figure 12: Layerwise probe classification of evaluation versus deployment context for Nemotron3-49B, using the mean-token representation. Nemotron3-49B’s last-token performance is confined to the first few layers and near chance thereafter; under mean-token pooling, early layers separate the contexts weakly in the probe-aligned direction (AUROC – ), while middle and late layers separate them in the direction anti-correlated with the probe (AUROC ). Accuracy in (a) uses a threshold calibrated on GSM8K/TruthfulQA vs. WildChat; its below-chance values at middle and late layers reflect this sign reversal, not an absence of separation. Normalized layer depth denotes the layer index divided by the total number of hidden-state layers returned by the model.
Figure 13: Spearman correlation between the last token of prompt’s probe score and judge scores across the six models. Gray bands show the random-direction baseline. Peak for every model. Normalized layer (normalized layer depth) denotes the layer index divided by the total number of hidden-state layers returned by the model.
Figure 14: Mutual information between the last token of prompt’s probe score and judge scores across the six models. Gray bands show the random-direction baseline. Peak nats for every model. Layer depth (normalized layer depth) denotes the layer index divided by the total number of hidden-state layers returned by the model.
Figure 15: Spearman correlation between response-last probe score and judge score across the six models. Layer depth (normalized layer depth) denotes the layer index divided by the total number of hidden-state layers returned by the model.
Figure 16: Mutual information between response-last probe score and judge score across the six models. Layer depth (normalized layer depth) denotes the layer index divided by the total number of hidden-state layers returned by the model.
| Readout | peak (model / layer) | peak MI in nats (model / layer) |
| prompt-last | (Olmo3-32B/ L61) | (Olmo3-32B/ L12) |
| prompt-mean | (Olmo3-32B/ L63) | (Olmo3-32B/ L42) |
| response-last | (Qwen3-32B/ L8) | (Olmo3-32B/ L44) |
| response-mean | (Qwen3-32B/ L63) | (Olmo3-7B/ L21) |
| </think> , Qwen3-8B | (L6) | (L34) |
| </think> , Qwen3-32B | (L62) | (L61) |
Table 8: Peak per-prompt coupling between the linear probe and the fine-grained judge , taken over all layers, for the four reasoning checkpoints. Peak stays for every probe readout, and peak MI stays nats. The bottom block reports the same statistics for the </think> -token readout (Qwen only).
Figure 17: Layerwise Spearman between the linear probe and the fine-grained judge , on the two readouts that attain the highest peak in Table 8 . Left: Olmo3-32B Think, prompt-last readout. The learned probe (coloured) tracks the random direction (grey, shaded to zero) closely; peak . Right: Qwen3-32B, response-mean readout; peak . Per-model plots for all four reasoning checkpoints, both metrics, and all four readouts are included in the supplementary material. Normalized layer depth denotes the layer index divided by the total number of hidden-state layers returned by the model.
| Model(s) | Layer sweep | sweep |
| Olmo3-7B Base | 0–31 | |
| Olmo3-32B Base | 22–41 | |
| Olmo3-32B SFT | 22–41 | |
| Olmo3-32B DPO | 22–41 | |
| Olmo3-32B Think | 21–42 | |
| Olmo3-7B Think | 0–31 | (step 0.2) |
Table 9: Steering sweep configuration by model.
| Family | Stage | Score 0 (%) | Score 1 (%) | Score 2 (%) | Score 3 (%) | Avg | Std | |
| Qwen3-8B | 1000 | 97.3 | 2.6 | 0.0 | 0.1 | 0.029 | 0.185 | |
| Qwen3-32B | 1000 | 96.0 | 3.9 | 0.0 | 0.1 | 0.042 | 0.215 | |
| Olmo3 7B | Think | 1000 | 93.5 | 6.3 | 0.0 | 0.2 | 0.069 | 0.276 |
| Olmo3-32B | Think | 1000 | 94.6 | 5.2 | 0.0 | 0.2 | 0.058 | 0.258 |
Table 10: Unsteered baseline judge-score distribution for Qwen3 and Olmo3 Think models. N is the number of non-empty responses.
| Family | Stage | Score 0 (%) | Score 1 (%) | Score 2 (%) | Score 3 (%) | Avg | Std | |
| Olmo3 7B | Base | 952 | 99.7 | 0.3 | 0.0 | 0.0 | 0.003 | 0.056 |
| Olmo3 7B | SFT | 1000 | 95.8 | 4.1 | 0.0 | 0.1 | 0.044 | 0.219 |
| Olmo3 7B | DPO | 1000 | 95.9 | 3.8 | 0.0 | 0.3 | 0.047 | 0.251 |
| Olmo3 7B | Think | 1000 | 93.5 | 6.3 | 0.0 | 0.2 | 0.069 | 0.276 |
| Olmo3-32B | Base | 962 | 99.1 | 0.9 | 0.0 | 0.0 | 0.009 | 0.096 |
| Olmo3-32B | SFT | 1000 | 96.2 | 3.7 | 0.0 | 0.1 | 0.040 | 0.211 |
Table 11: Unsteered baseline judge-score distribution across Olmo3 7B/32B stages (falling back to when explicit unsteered rows are unavailable). N is the number of non-empty responses.
| Model | First with breakdown | Cells affected at | Notes |
| Olmo3-7B Think | – | – | No degeneration observed in sweep |
| Olmo3-32B Think | – | – | No degeneration observed in sweep |
| Qwen3-8B | – | – | No degeneration observed in sweep |
| Qwen3-32B | 0.8 | substantial | Empty outputs at certain combinations |
Table 12: Steering breakpoint summary. “First with breakdown” is the smallest swept magnitude at which at least one response produces empty or degenerate output. “Cells affected at ” is the fraction of the grid affected at .
Figure 29: Per-prompt coupling across Olmo-7B training stages on the prompt-last readout. Top row: layerwise Spearman between probe scores and judge scores . Bottom row: Kraskov mutual information in nats. Columns correspond to Base, SFT, DPO, and Think checkpoints. Normalized layer (normalized layer depth) denotes the layer index divided by the total number of hidden-state layers returned by the model.
Figure 30: Per-prompt coupling across Olmo-32B training stages on the prompt-last readout. Top row: layerwise Spearman correlation . Bottom row: Kraskov mutual information in nats. Columns correspond to Base, SFT, DPO, and Think checkpoints. Normalized layer (normalized layer depth) denotes the layer index divided by the total number of hidden-state layers returned by the model.
Figure 33: Row-normalized cross-judge confusion (same convention as Figure 9 ) across all eight Olmo3 training-stage checkpoints. Top row: Olmo3-7B Base/SFT/DPO/Think. Bottom row: Olmo3-32B Base/SFT/DPO/Think. At Base, the bottom row ( ) puts of mass in the column; at SFT, DPO, and Think the same row puts in the column instead.
More in this topic
Evaluation Awareness in Language Models
Evaluation awareness in language models studies how models recognize testing or monitoring and whether that recognition changes their answers, safety, or strategic behavior.
See all 29 papers