Probing studies what information is encoded in a frozen LLM's layer representations by training a lightweight predictor on top of them. Beyond analysis, probes are often used operationally in probe-then-steer pipelines: a learned concept vector is extracted from a probe and injected via additive activation steering by adding it to a layer representation during the forward pass. The effectiveness of this pipeline hinges on estimating concept vectors that are accurate, directionally stable under ablation, and inexpensive to obtain. Motivated by these desiderata, we propose RAPTOR (Ridge-Adaptive Logistic Probe), a simple L2-regularized logistic probe whose validation-tuned ridge strength yields concept vectors from normalized weights. Across extensive experiments on instruction-tuned LLMs and human-written concept datasets, RAPTOR matches or exceeds strong baselines in accuracy while achieving competitive directional stability and substantially lower training cost; these quantitative results are supported by qualitative downstream steering demonstrations. Finally, using the Convex Gaussian Min-max Theorem (CGMT), we provide a mechanistic characterization of ridge logistic regression in an idealized Gaussian teacher-student model in the high-dimensional few-shot regime, explaining how penalty strength mediates probe accuracy and concept-vector stability and yielding structural predictions that qualitatively align with trends observed on real LLM embeddings.
Figures & tables
Figure 1: Overview of the Raptor pipeline for additive activation steering: extract layerwise last-token embeddings, standardize features, fit an ℓ2 -regularized logistic probe with λ selected on a validation split, rescale to embedding space and normalize to obtain the concept vector vℓ , then steer at inference by hℓ,T←hℓ,T+αvℓ .
Model
Method
STSA
Cities
Common
Counterfact
HateXplain
Sarcasm
Average
Qwen 2.5 Series
Qwen-3B-Instruct
GCS
83.5 (92.6)
84.1 (99.4)
62.5 (67.9)
64.7 (77.6)
66.0 (69.8)
81.1 (87.9)
73.6 (82.5)
xRFM
87.8 (93.4)
91.6 (99.5)
69.0 (72.0)
66.6 (75.8)
70.3 (74.3)
85.2 (91.0)
78.4 (84.4)
Fixed-reg
84.6 (91.7)
87.4 (99.6)
61.7 (65.2)
63.1 (73.4)
68.7 (71.3)
81.5 (88.9)
74.5 (81.7)
CAA
78.1 (91.4)
84.3 (99.7)
64.1 (68.6)
62.9 (77.8)
62.6 (68.5)
74.2 (82.7)
71.0 (81.4)
Raptor (Ours)
88.1 (93.7)
88.1 (99.5)
68.6 (71.6)
68.8 (80.0)
70.9 (74.1)
85.3 (90.9)
78.3 (85.0)
Table 1: Probe accuracy (%) across models and datasets. Results are presented as Average over layers (Best Layer) . All values are means over five seeds; best layers are selected using validation accuracy. The purple background highlights the highest average-over-layers accuracy within each model and column, including all ties at the displayed precision. The vertical line separates individual task results from the overall average across all six tasks.
Figure 2: Accuracy differences across the full model–dataset grid (Raptor minus baseline)
Figure 4: Per-layer probe training time across 6×6 model–dataset settings. The horizontal axis uses a logarithmic scale. Each circle shows the per-layer training-time summary for one model–dataset setting. Diamonds indicate medians across the 36 settings for each method.
Dataset
Dir.
Succ.
Intv.
∣α∣ med
∣α∣ p90
CounterFact
away
1.000
0.833
10.50
66.50
CounterFact
towards
1.000
0.556
6.88
59.25
HateXplain
towards
1.000
0.654
6.80
49.00
HateXplain
away
1.000
0.538
4.70
24.50
Sarcasm
towards
1.000
0.715
8.83
28.00
Sarcasm
away
1.000
0.631
6.20
30.75
Table 2: Steering controllability.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Raptor
xRFM
GCS
Model
Best
Avg
Best
Avg
Best
Avg
CounterFact (grouped by subject)
Qwen2.5-3B
80.54±1.43
73.72±0.68
81.49±0.70
74.41±0.97
77.59±0.78
65.62±0.25
Qwen2.5-7B
84.38±0.97
76.73±0.62
84.38±1.24
76.95±0.71
81.06±0.66
69.96±0.40
Qwen2.5-32B
85.58±0.77
80.20±0.44
85.71±1.45
80.31±0.57
83.40±0.44
74.50±0.25
Gemma-7B-it
81.57±1.62
74.95±1.03
81.61±1.69
74.97±0.78
78.43±0.76
68.84±0.40
Appendix
Table 3: Grouped-split sensitivity on CounterFact and Cities. Best denotes test accuracy at the validation-selected layer; Avg denotes test accuracy averaged over all evaluated layers. Entries are means ± sample standard deviations across five outer seeds, in percent. The Six-model mean rows first average the six models within each seed, then report the mean and sample standard deviation across those five values.
dataset
layers
separable
perc_zero
STSA
28
26
25
cities
28
26
25
sarcasm
28
28
28
common
28
24
23
counterfact
28
22
20
HateXplain
28
26
23
Appendix
Table 4: Layer-wise linear separability summary across datasets. separable counts layers where LinearSVC achieves zero training error for some C∈{104,106,108} (after standardization). perc_zero counts layers where a perceptron reaches zero training error within 2000 epochs.
Dataset
Setting
Dir.
Layers ( L )
N
Succ.
Intv.
∣α∣ med
p90
max
τ
counterfact
truth_to_lie
away
9–26 (18)
1152
1.000
0.833
10.50
66.50
97.5
0.7
counterfact
lie_to_truth
towards
9–26 (18)
1152
1.000
0.556
6.88
59.25
81.5
0.7
hatexplain
neutral_to_hate
towards
1–26 (26)
1664
1.000
0.654
6.80
49.00
67.5
–
hatexplain
neutral_to_nonhate
away
1–26 (26)
1664
1.000
0.538
4.70
24.50
249.0
–
sarcasm
neutral_to_sarcastic
towards
1–26 (26)
1664
1.000
0.715
8.83
28.00
143.0
–
sarcasm
neutral_to_sincere
away
1–26 (26)
1664
1.000
0.631
6.20
30.75
211.0
–
Appendix
Table 5: Full steering controllability and cost (from logs). Dir. indicates the target direction (towards: Pm→0.9999 , away: Pm→0.0001 ). Layers is the range used after optional reliability filtering ( τ is the probe-accuracy threshold; “–” means no filtering in that run). N counts evaluated layer–prompt pairs after filtering. Succ. is the probe-target success rate (towards: Pm≥0.9999 ; away: Pm≤0.0001 ). Intv. is the intervention rate. We summarize cost by the distribution of per-layer ∣αℓ∣ .
Dataset
Setting
Top layers by median ∣αℓ∣
Top layers by max ∣αℓ∣
counterfact
truth_to_lie
25(97.50), 22(66.50), 14(29.50)
25(97.5), 22(66.5), 14(29.5)
counterfact
lie_to_truth
23(81.50), 26(59.25), 24(52.25)
23(81.5), 26(59.2), 24(52.2)
hatexplain
neutral_to_hate
22(67.50), 24(49.75), 25(49.00)
22(67.5), 24(49.8), 25(49.0)
hatexplain
neutral_to_nonhate
26(249.00), 13(35.00), 18(24.50)
26(249.0), 13(35.0), 18(24.5)
sarcasm
neutral_to_sarcastic
25(120.50), 22(65.25), 7(28.00)
25(143.0), 22(73.5), 23(29.0)
sarcasm
neutral_to_sincere
26(168.50), 24(155.00), 13(30.75)
26(211.0), 24(165.0), 13(31.2)
Appendix
Table 6: Hard layers that dominate steering cost. Each entry lists LayerID(value), where values are computed from per-layer ∣αℓ∣ reported in the logs. We show top-3 layers by median ∣αℓ∣ (typical difficulty) and by max ∣αℓ∣ (tail risk).
model
dataset
layer
λ⋆
ρs
r
Qwen2.5-7B
STSA
10
104
1.000
0.996
Qwen2.5-7B
STSA
20
103
0.928
0.973
Qwen2.5-7B
common
10
105
0.771
0.913
Qwen2.5-7B
common
20
103
1.000
0.981
Qwen2.5-7B
hatexplain
10
105
0.657
0.842
Qwen2.5-7B
hatexplain
20
105
0.600
0.900
Appendix
Table 7: Structure validation on real embeddings. We sweep δ=n/p by stratified subsampling and compare the theory-inspired predictor Accpred to the empirical accuracy Acctrue along the δ sweep. We report Spearman ρs and Pearson r correlations across the sweep. In implementation we tune the inverse regularization parameter C (LinearSVC-style); for consistency with the theory we report the equivalent ridge strength λ=1/C . λ⋆ is the value (from the tuned grid) that maximizes ρs for each setting.
Metric
Original
GCS
Raptor
Target sentiment strength
1.74
2.14
2.88
Output quality
–
7.88
7.91
Naturalness
–
7.89
7.93
Overall steering success
5.61
5.76
6.05
Appendix
Table 8: Blinded LLM-as-judge evaluation of open-ended steering outputs. Scores are reported on the 160 fully valid prompt–seed cases. Higher is better. Original denotes the unsteered baseline. Output quality and naturalness are scored only for generated steered outputs.
Figure 5: Layerwise probing accuracy curves (Part I): each column is a dataset; each column stacks 6 models (top to bottom: Llama-3.1-70B, Llama-3.1-8B, Gemma-7B-it, Qwen2.5-32B, Qwen2.5-7B, Qwen2.5-3B). Curves show mean test accuracy over five outer seeds ( 42 – 46 ) for Raptor , xRFM, GCS, Fixed-reg, and CAA with midpoint readout.
Figure 6: Layerwise probing accuracy curves (Part II): each column is a dataset; each column stacks 6 models in the same order as Fig. 5 . Curves show mean test accuracy over the same five outer seeds for the five methods listed in Fig. 5 .
Dataset
Raptor
xRFM
GCS
Fixed-reg
CAA
Qwen2.5-3B-Instruct
STSA
93.73±0.68
93.44±0.51
92.63±0.46
91.71±0.74
91.42±0.77
Cities
99.47±0.51
99.53±0.38
99.44±0.35
99.60±0.55
99.67±0.33
Common
71.63±0.59
71.98±0.77
67.94±0.63
65.25±1.84
68.63±1.25
CounterFact
80.03±0.83
75.85±1.03
77.56±0.69
73.35±0.82
77.80±1.21
HateXplain
74.13±0.51
74.34±0.94
69.82±0.18
71.30±1.15
68.51±0.43
Appendix
Table 9: Full five-seed accuracy at validation-selected layers. Best denotes test accuracy at the layer selected separately for each method and seed using pre-refit outer-validation accuracy, with ties resolved by the lowest layer index. Entries are mean ± sample SD across five outer seeds, in percent. Six-dataset mean rows average the six datasets within each seed before computing the mean and SD.
Dataset
Raptor
xRFM
GCS
Fixed-reg
CAA
Qwen2.5-3B-Instruct
STSA
88.11±0.19
87.79±0.17
83.52±0.16
84.64±0.36
78.10±0.56
Cities
88.09±0.61
91.64±0.29
84.06±0.32
87.42±0.52
84.30±0.46
Common
68.63±0.59
68.96±0.66
62.49±0.45
61.74±0.54
64.11±0.64
CounterFact
68.83±0.55
66.60±0.73
64.70±0.20
63.07±0.54
62.94±0.47
HateXplain
70.85±0.43
70.33±0.67
65.98±0.28
68.67±0.56
62.62±0.45
Appendix
Table 10: Full five-seed layer-average accuracy. Avg denotes test accuracy averaged over the complete evaluated layer set within each seed. Entries are mean ± sample SD across five outer seeds, in percent. Six-dataset mean rows average the six datasets within each seed before computing the mean and SD.
Raptor
xRFM
GCS
Fixed-reg
CAA
Dataset
Max
Mean
Max
Mean
Max
Mean
Max
Mean
Max
Mean
Qwen2.5-3B-Instruct (36 layers)
STSA
1.00
0.90
0.83
0.77
1.00
0.94
1.00
0.87
1.00
1.00
Cities
1.00
0.93
0.94
0.67
1.00
0.95
1.00
0.94
1.00
0.97
Common
1.00
0.94
0.72
0.69
1.00
0.96
1.00
0.88
0.99
0.99
CounterFact
1.00
0.90
0.81
0.60
1.00
0.93
1.00
0.87
1.00
0.93
Appendix
Table 11: Complete directional robustness results under 20% training-data ablations. Max and Mean denote the maximum and mean layerwise robustness, respectively. All values are rounded to two decimal places; a displayed value of 1.00 does not imply exact unit stability.
Asset
Type
License / terms of use
Use in this paper
STSA / Stanford Sentiment Treebank
Dataset
License listed as unknown in the SST data card .
Probing / steering benchmark
Cities
Dataset
Underlying GeoNames data : CC BY 4.0. No benchmark-specific license stated in the source repository.
Probing benchmark
Common / CommonClaim
Dataset
Dataset license not specified in the source repository.
Probing benchmark
CounterFact
Dataset
Dataset-specific license not stated on the official download page .
Probing / steering benchmark
HateXplain
Dataset
CC BY 4.0 in the data-card metadata ; MIT in the same card’s licensing text.
Probing / steering benchmark
Sarcasm News Headlines
Dataset
CC BY 4.0, as listed in the author’s Kaggle release .
Probing / steering benchmark
Appendix
Table 12: External datasets and pretrained models used in the experiments, with license information reported by their sources.