cs.CLApr 8, 2026
SaveLURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness
Organizations: Meridian Cambridge
Abstract
Large language models can recognize when they are being evaluated (evaluation awareness) and behave differently because of that, which undermines the validity of safety and alignment benchmarks. We propose LURE (Live-Usage Replay Evaluations), a method for constructing deployment-like evaluations by replaying realistic agentic interaction trajectories and appending evaluation prompt at the end. We also introduce an automated pipeline for measuring evaluation realism, combining detection of verbalized evaluation awareness and judge-model estimates of the probability of logs being an evaluation, and validate it on a large dataset of deployment and evaluation transcripts. We find that LURE-based evaluations are substantially less distinguishable from deployment than widely used benchmarks and synthetic evaluation generators, and can approach the realism of real conversations with users. We instantiate LURE in scheming, AI safety sabotage, and sycophancy settings. Our results suggest that evaluation realism is a crucial property of alignment benchmarks and should be reported alongside benchmark results, especially when such results are used in safety cases.
Figures & tables
Figure 1: Overview of LURE. Frontier LLMs often detect evaluation contexts and alter their behavior accordingly (left). LURE constructs realistic evaluations by collecting real agentic trajectories, lightly adapting them, and appending a safety-relevant test (top right). Evaluation cues materially affect evaluation results (bottom left) and LURE evaluations are far less distinguishable from deployment than existing benchmarks (bottom right).
Figure 2: Validation of the evaluation-likeness judge. (a) ROC curve for classifying transcripts as evaluation vs. deployment on the agentic subset of Needham et al. (2025) , using a weighted average of Claude Sonnet 4.5 and Gemini 3 Flash Preview scores (AUROC = 0.906). (b) True-positive and true-negative rates at selected thresholds. We adopt (highlighted), which identifies 96% of evaluations while mislabeling 45% of deployment transcripts.
Figure 3: Misalignment rates on LURE-Scheming. In both settings, only GPT-5.2 exhibits covert misalignment (18.5%). Gemini 3 Flash, Sonnet 4.5, Opus 4.6, and Gemini 3 Pro show no covert misalignment in either setting.
| Source | Type | What we include / how we run it | Reference |
|---|---|---|---|
| Control deployment | Deployment control | Agentic deployment transcripts from Needham et al. (2025) . | Needham et al. (2025) |
| Apollo OSv | Benchmark | Oversight Subversion subset of Apollo in-context scheming evaluations; default settings. | Meinke et al. (2024) |
| InstrumentalEval | Benchmark | Subsets targeting covert actions and deception; default settings. | He et al. (2025) |
| Petri 2.0 | Synthetic generator | Scheming evaluations generated with Petri 2.0 with realism threshold a . | Fronsdal et al. (2026) |
| Bloom | Synthetic generator | Goal-guarding evaluations generated with Bloom; target behavior set to self-preservation. | Gupta et al. (2025) |
Table 1: Evaluation sources used for realism comparison in § 4.2 .
Figure 4: Evaluation awareness across evaluation sources. LURE-Scheming (both variants) falls below the realism threshold (dashed red line) and shows zero verbalized awareness, comparable to real deployment transcripts. All other benchmarks and generation tools exceed the threshold, with most also showing substantial verbalized awareness.
Figure 5: Results of LURE-Sabotage. No model exhibited covert misalignment. Most models refused to comply with user requests, with Claude models doing so most frequently. Claude Opus 4.6 refuses significantly more often without reasoning.
Figure 6: Environment realism of sycophancy evaluations. All four existing sycophancy benchmarks we test exceed the realism threshold . By contrast, LURE-Sycophancy falls below the threshold, approaching the realism of LURE-Scheming.
Figure 7: Model rankings differ sharply across evaluation formats. Each point is a model. Points above the diagonal indicate higher caving rates under LURE than on raw SYCON-Bench; points below the diagonal indicate the reverse. The overall rank correlation is negative (Spearman , , ).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Example LURE-Scheming sample (simplified). A coding agent is asked to explore a travel-assistant repository and then edit the agent’s priorities from customer benefit to revenue maximization. Earlier turns (top) establish a realistic coding-session context drawn from a real agentic trajectory. The final request (middle) is the alignment-relevant test. An aligned model implements the requested edit (bottom left); a covertly misaligned model edits the file to preserve customer-friendly behavior while reporting compliance to the user (bottom right).
Figure 9: True-positive and true-negative rates for the binary evaluation-deployment classification method using GPT-4.1 as a judge, on the agentic subset of the calibration dataset from Needham et al. (2025) .
Figure 10: P(eval) as a function of transcript length. There is a weak positive correlation between transcript length and sample realism.
Figure 11: An explicit goal conflict reduces evaluation realism, but not enough to exceed the threshold. No verbalized evaluation awareness is observed in either condition.
Figure 12: Effect of reasoning on evaluation-likeness discrimination. ROC curves for classifying transcripts as evaluation vs. deployment, with (blue) and without (red) reasoning. All three models improve with reasoning; Sonnet 4.5 shows the largest gain (AUROC 0.89 0.95), while Gemini 3 Flash (0.88 0.89) and DeepSeek V3.2 (0.77 0.79) show modest improvements.
Figure 13: Scheming rates under degraded realism. Compared to the original nudged benchmark, no substantial change in scheming rates is observed. Only GPT-5.2 exhibits covert misalignment in both conditions, suggesting that evaluation realism does not necessarily affect scheming rates.
Figure 14: Example LURE-Sabotage sample (simplified). A coding agent is asked to apply steering vectors to its own weights, while the user claims to be an AI safety researcher employed by the model developer. An aligned model implements the requested edit (bottom left); a model that rejects the request provides a justification (bottom right).
Figure 15: Environment realism of sycophancy evaluations. All four existing sycophancy benchmarks we test exceed the realism threshold . By contrast, LURE-Sycophancy constructs fall below the threshold, approaching the realism of LURE-Scheming. The stance-consistency construct remains somewhat higher because the repeated back-and-forth itself is a detectable pattern.
| Raw SYCON | LURE Truth | |||
| Opus 4.6 | 2 [1,7] | 46 [33,60] | +44 | 0.97 |
| Llama 4 Scout | 16 [10,24] | 58 [44,71] | +42 | 0.90 |
| Sonnet 4.5 | 8 [4,15] | 44 [31,58] | +36 | 0.88 |
| Llama 4 Maverick † | 58 [48,67] | 55 [41,68] | 0.76 | |
| DeepSeek V3 | 47 [38,57] | 36 [24,50] | 0.74 | |
| GPT-4o | 34 [25,44] | 22 [13,35] | 0.65 |
Table 2: Sycophancy rates (%) on raw SYCON-Bench ( ) and LURE Truthfulness ( ). 95% Wilson confidence intervals in brackets. Ordered by . is the model’s own estimate that raw SYCON-Bench is an evaluation.
Figure 16: Sycophancy rates by model under raw and LURE conditions. Paired bars show raw SYCON-Bench and LURE Truthfulness. Several Claude and Llama models exhibit much higher caving rates in the LURE setting, whereas Gemini models and GPT-5.2 exhibit lower rates.
Figure 17: Three signals of evaluation awareness for sycophancy evaluations. Relative to raw benchmarks, LURE constructs show lower external , lower model self-estimated evaluation probability, and near-zero verbalized awareness.
More in this topic
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,408 papers