cs.LGApr 15, 2026
SaveLinear Probe Accuracy Scales with Model Size and Benefits from Multi-Layer Ensembling
Organizations: Georgia Institute of Technology, Atlanta, Georgia, USA · Independent Researcher · Stanford University, Stanford, California, USA
Abstract
Linear probes can detect when language models produce outputs they "know" are wrong, a capability relevant to both deception and reward hacking. However, single-layer probes are fragile: the best layer varies across models and tasks, and probes fail entirely on some deception types. We show that combining probes from multiple layers into an ensemble recovers strong performance even where single-layer probes fail, improving AUROC by +29% on Insider Trading and +78% on Harm-Pressure Knowledge. Across 12 models (0.5B--176B parameters), we find probe accuracy improves with scale: ~5% AUROC per 10x parameters (R=0.81). Geometrically, deception directions rotate gradually across layers rather than appearing at one location, explaining both why single-layer probes are brittle and why multi-layer ensembles succeed.
Figures & tables
Figure 1: Multi-layer ensembling separates deceptive from honest activations. A 5-layer ensemble on Llama 70B achieves +78% AUROC improvement on Harm-Pressure Knowledge, a task where single-layer probes fail. The visualization shows probe outputs projected to 3D via PCA, with honest (blue) and deceptive (red) samples. Misclassified samples (black outlines) cluster near the decision boundary, indicating errors occur on genuinely ambiguous cases rather than systematic failures.
Figure 2: Larger models have more linearly accessible deception representations. Log-linear fits show AUROC improvement per 10 parameters ( , ). Explicit deception tasks (Convincing Game, Instructed Deception) scale fastest and achieve near-ceiling performance; implicit deception (Harm-Pressure) scales weakly and remains challenging even at 70B+ parameters.
Figure 3: Best layer position vs model size across families. Llama models (red) show high variance; Qwen models (blue) exhibit more consistent positioning in the 60–80% range for larger models.
Figure 4: Probe performance varies dramatically by deception type. Explicit deception (Convincing Game, Instructed Deception) achieves near-perfect detection (AUROC 0.9) across most models, while implicit deception (Harm-Pressure tasks) proves challenging, with some models near chance. This task hierarchy is consistent across model families, suggesting the difficulty lies in the deception type, not model architecture.
| Dataset | Our Results | Baseline | |
|---|---|---|---|
| Instructed Deception (ID) | 0.916 | 0.530 | +72.8% |
| Harm-Pressure Choice (HP-C) | 0.729 | 0.750 | 2.8% |
| Insider Trading (IT) | 0.423 | 0.890 | 52.5% |
| Harm-Pressure Knowledge-Report (HP-KR) | 0.842 | 0.560 | +50.4% |
| Convincing Game (CG) | 0.926 | 0.550 | +68.4% |
| Average | 0.767 | 0.656 | +17.0% |
Table 1: Comparison of deception detection performance (AUROC) between our probe at layer 52 and the Liars Bench baseline (Unrelated Question Classifier) on Llama-70B variants. Best results per dataset in bold .
Figure 5: Layer-wise AUROC across Qwen 2.5 model sizes (0.5B and 72B) for 5 deception tasks. We see improvement in representation quality with scale. Qwen 0.5B shows highly erratic patterns while Qwen 72B achieves smooth, well-differentiated representations that maintain high performance in later layers.
Figure 6: Probes at nearby layers detect similar features. We plot the cosine similarity between probe weight vectors against the absolute difference in their AUROC scores, colored by layer distance . The negative correlation ( ) confirms that geometrically similar probes perform similarly. Adjacent layers (yellow/green, – ) cluster in the high-similarity, low-AUROC-difference regime; distant layers (purple, ) show greater divergence, indicating the deception direction rotates gradually through the network.
Figure 7: Double fault matrix for Llama 3.3 70B probe layers. Double fault measures how often two probes fail simultaneously on the same sample. A low score indicates complementary failure modes (when probe A errs, probe B often succeeds). Darker cells indicate better ensemble candidates. The block structure reveals that early-layer probes (layers 0 to 30) have complementary failures to late-layer probes (layers 50 to 80), motivating cross-depth ensembling.
Figure 8: Two-layer ensemble AUROC for all layer pairs on Llama-70B. Each cell shows the AUROC achieved by combining probes from the corresponding row and column layers. Darker cells indicate poorer ensemble performance.
Figure 9: Two-layer ensembles fail to consistently improve across tasks. Left: Minimum AUROC improvement across all five datasets for each layer pair (blue = improvement, red = degradation). The predominantly red heatmap shows that no two-layer combination improves on all tasks simultaneously. Right: Mean AUROC versus ensemble weighting for selected layer pairs; while optimal weights exist (peaks above the single-layer baseline, dotted line), gains on some tasks come at the cost of degradation on others.
| Dataset | Single (28) | 3-Layer | 5-Layer | 5L Impr. |
|---|---|---|---|---|
| Instructed Deception | 0.939 | 0.818 | 0.889 | 5.3% |
| Harm-Pressure Choice | 0.949 | 0.899 | 0.909 | 4.2% |
| Convincing Game | 1.000 | 1.000 | 1.000 | +0.0% |
| Insider Trading | 0.737 | 0.869 | 0.953 | +29.3% |
| Harm-Pressure Knowledge | 0.510 | 0.870 | 0.910 | +78.4% |
| Mean | 0.827 | 0.891 | 0.932 | +12.7% |
Table 2: Ensemble performance on Liars’ Bench datasets using proper train/test evaluation (80/20 split, logistic regression stacking). Improvement computed relative to best single layer (layer 28). The 5-layer ensemble shows strong gains on challenging tasks but mixed results overall.
Figure 10: Per-dataset ensemble improvement heatmap. Each cell shows the AUROC improvement relative to the best single-layer probe for a given dataset (rows) and ensemble size (columns). Blue cells indicate improvement over the single-layer baseline, while red cells indicate degradation. The 5-layer ensemble achieves the largest gains on Insider Trading and Harm-Pressure Knowledge, but shows slight degradation on tasks where single-layer probes already perform well.
Figure 11: PCA and t-SNE visualization of ensemble predictions on Llama-70B. Top row shows 3-layer ensemble, bottom row shows 5-layer ensemble across all five Liars’ Bench datasets. Points are colored by true label (blue=honest, red=deceptive), with black circles indicating misclassified samples. Misclassified samples cluster near the decision boundary, indicating errors occur on genuinely ambiguous cases.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Layer-wise AUROC across all 14 evaluated models. Each subplot shows performance across layers for five deception tasks. Smaller models (e.g., Qwen 0.5B, Llama 1B) exhibit noisy, unstable patterns with substantial layer-to-layer variation. Larger models (e.g., Llama 70B, Qwen 72B) develop smoother, more stable representations with clearer task differentiation. Task colors: Instructed Deception (red), Harm Refusal Choice (orange), Insider Trading (green), Harm Refusal Knowledge (blue), Convincing Game (purple).
Figure 13: Distribution of best performing layer positions (relative to total layers) across all model-task combinations. The distribution is centered around 0.65 (mean) and 0.64 (median), indicating deception information tends to be most accessible in the latter two-thirds of networks, but with substantial variation (range: 0.15-0.98).
Figure 14: Model family comparison by dataset. Box plots show best-layer AUROC distribution within each family. All families achieve high performance on Instructed Deception and Convincing Game ; Harm Refusal Choice proves challenging across all families.
Figure 15: Scaling analysis by model family. Each family shows distinct scaling patterns. The Llama family exhibits particularly strong performance on Insider Trading and Convincing Game tasks.
Figure 16: Cosine similarity vs performance change by layer distance. Correlation remains consistent ( to ) across layer distances to , demonstrating that representational continuity persists across both adjacent and distant layers.
Figure 17: True Positive Rate at 1% False Positive Rate for two-layer ensembles across all tasks on Llama-70B. This metric measures the proportion of deceptive samples correctly identified while limiting false alarms to 1 in 100 honest samples. Patterns are similar to AUROC results, with the same layer combinations performing well on both metrics.
Figure 18: 3D PCA visualization of 5-layer ensemble predictions with linear decision plane shown in green. The figure displays four viewing angles of the same data. The decision plane separates honest samples (blue circles) from deceptive samples (red circles) with 66% accuracy in the reduced 3D PCA space. Misclassified samples (marked with black outlines) concentrate near the plane boundary, confirming that ensemble errors occur on genuinely ambiguous cases.
More in this topic
Language Model Probing
Language model probing measures what information and capabilities models encode and how those signals relate to their behavior.
See all 275 papers