Decodability Is Not Validity: Evaluating Nuisance Sensitivity in Language Model Concept Probes
Abstract
Two concept probes can be equally accurate at predicting a concept from a language model's hidden states while their scores contain different mixtures of the target concept and a correlated nuisance attribute. In four target--nuisance settings (two aspect sentiments in CEBaB; occupation and gender in BiasBios; Amazon review sentiment with negation, and with first-person framing), two LLMs, and a population of concept extractors, we compare decodability with discriminant evidence: the score's dependence on the nuisance at fixed target, and its response to input edits intended to change only the nuisance, filtered by two LLM judges and compared against placebo edits. Under correlated training data, extractors that are statistically equivalent in held-out AUROC differ up to several-fold in nuisance sensitivity in three of the four settings, and decodability is not a reliable proxy for it: their rank correlation changes sign across settings. The effect follows the probe's training data, not the nuisance's natural association with the target: first-person framing, unrelated to sentiment in natural reviews, is absorbed at least as much as negation, which co-occurs with negative sentiment, and decorrelated training removes the systematic effect. Selecting extractors with nuisance-aware development diagnostics, which need nuisance labels but no counterfactuals, lowers counterfactual nuisance sensitivity wherever the default choice is sensitive, at a small accuracy cost. Concept measurements should be validated, not just decoded.
Figures & tables
| target / nuisance | edits | judged by both | accepted | |
|---|---|---|---|---|
| CEBaB | service / food sentiment | 210 | 210 | 41% |
| BiasBios | surgeon vs. nurse / gender | 1561 | 1561 | 93% |
| Amazon-Neg | negative sentiment / negation | 1427 | 1427 | 58% |
| Amazon-1P | negative sentiment / first person | 1083 | 1083 | 68% |
| Civil ∗ | toxic / Muslim or LGBTQ | 762+400 | 762+400 | 13%, 29% |
| best | #near | near-best | equiv. | Conflict | ||
| Dec | (equiv.) | range | range | range | ||
| (correlated training data) | ||||||
| CEBaB | Q | .966 | 37 (9) | .46–.89 | .54–.83 | .44–.69 |
| G | .979 | 39 (8) | .26–.72 | .25–.63 | .57–.82 | |
| BiasBios | Q | .998 | 66 (65) | .09–.42 | .09–.42 | .83–.97 |
| G | .998 | 66 (61) | .09–.33 | .11–.33 | .92–.97 | |
| standard extractors | + erased | ||||
|---|---|---|---|---|---|
| max Dec | max Conflict | min Leak | max Conflict | ||
| CEBaB | Q | .69 | .54 ( .012) | .50 ( .013) | .34 ( .007) |
| G | .41 | .26 ( .005) | .27 ( .004) | .11 ( .004) | |
| BiasBios | Q | .22 | .13 ( .001) | .09 ( .004) | .03 ( .002) |
| G | .24 | .14 ( .002) | .12 ( .006) | .02 ( .002) | |
| Amazon-Neg | Q | .04 | .04 † (+.000) | .07 † ( .007) | .00 ( .000) |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| data | edit kind | judge | kept | changed | fluency drop | major other | accepted | |
|---|---|---|---|---|---|---|---|---|
| Amazon-1P | nuisance | 1083 | Qwen3.8-27B | .98 | 1.00 | .21 | .00 | .74 |
| Amazon-1P | nuisance | 1083 | gpt-oss-120b | .98 | 1.00 | .16 | .00 | .82 |
| Amazon-1P | placebo | 567 | Qwen3.8-27B | .99 | .00 | .21 | .00 | .71 |
| Amazon-1P | placebo | 567 | gpt-oss-120b | .98 | .00 | .12 | .00 | .88 |
| Amazon-Neg | nuisance | 1427 | Qwen3.8-27B | .97 | 1.00 | .13 | .06 | .61 |
| Amazon-Neg | nuisance | 1427 | gpt-oss-120b | .98 | 1.00 | .03 | .02 | .90 |
| data | edit kind | accept (Qwen3) | accept (gpt-oss) | both | ||
|---|---|---|---|---|---|---|
| CEBaB | nuisance | 210 | .44 | .69 | .41 | .41 |
| CEBaB | placebo | 158 | .46 | .92 | .44 | .07 |
| CEBaB | target | 125 | .55 | .76 | .52 | .42 |
| BiasBios | nuisance | 1561 | .94 | .96 | .93 | .55 |
| BiasBios | placebo | 400 | .60 | .97 | .60 | .06 |
| Amazon-Neg | nuisance | 1427 | .61 | .90 | .58 | .18 |
| data | model | all vs. accepted | vs. SD norm. | signed vs. | direction vs. | Civil fwd vs. rev | |
|---|---|---|---|---|---|---|---|
| Amazon-1P | G | 0.5 | +.99 | 1.00 | +.69 | +.84 | – |
| Amazon-1P | G | 0.7 | 1.00 | +.98 | +.91 | +.94 | – |
| Amazon-1P | G | 0.9 | 1.00 | +.97 | +.99 | +.97 | – |
| Amazon-1P | Q | 0.5 | +.99 | +.99 | +.46 | +.84 | – |
| Amazon-1P | Q | 0.7 | 1.00 | +.97 | +.92 | +.94 | – |
| Amazon-1P | Q | 0.9 | 1.00 | +.94 | 1.00 | +.98 | – |
| data | planted error | Qwen | gpt-oss | both |
|---|---|---|---|---|
| Amazon-Neg | none (retest) | .88 | .98 | .86 |
| opinion reversed | .08 | .14 | .06 | |
| opinion stronger/weaker | .18 | .68 | .16 | |
| garbled (rule) | .12 | .32 | .10 | |
| minor fact changed | .76 | .88 | .70 | |
| Amazon-1P | none (retest) | .98 | .92 | .92 |
| all | Qwen | gpt-oss | either | both | |
|---|---|---|---|---|---|
| CEBaB | 210: .61–1.15/.41–.93 | 93: .47–.93/.26–.75 | 144: .56–1.01/.35–.81 | 151: .56–1.02/.35–.82 | 86: .46–.89/.26–.72 |
| BiasBios | 1561: .09–.41/.09–.33 | 1470: .09–.41/.09–.33 | 1498: .09–.41/.09–.33 | 1517: .09–.41/.09–.33 | 1451: .09–.42/.09–.33 |
| Amazon-Neg | 1427: .04–.13/.02–.13 | 870: .03–.12/.02–.12 | 1280: .04–.13/.02–.12 | 1316: .04–.13/.02–.12 | 834: .03–.12/.02–.12 |
| Amazon-1P | 1083: .11–.27/.06–.33 | 798: .11–.26/.06–.31 | 883: .11–.26/.06–.32 | 945: .11–.27/.06–.32 | 736: .11–.25/.05–.31 |
| best | #near | near-best | equiv. | Conflict | ||
|---|---|---|---|---|---|---|
| Dec | (equiv.) | range | range | range | ||
| CEBaB | Q | .934 | 18 (0) | .15–.54 | – | .70–.86 |
| G | .967 | 13 (0) | .05–.26 | – | .86–.93 | |
| BiasBios | Q | .995 | 62 (0) | .01–.14 | – | .96–.99 |
| G | .995 | 61 (0) | .01–.11 | – | .96–.99 | |
| data | model | Conflict | Leak | |||
|---|---|---|---|---|---|---|
| Amazon-1P | G | 0.5 | .43 [ .66, .18] | .66 [ .80, .48] | +.96 [+.93, +.97] | +.47 [+.28, +.62] |
| Amazon-1P | G | 0.7 | .31 [ .53, .05] | .51 [ .70, .29] | +.94 [+.89, +.96] | .11 [ .32, +.11] |
| Amazon-1P | G | 0.9 | .00 [ .25, +.25] | .06 [ .32, +.19] | +.68 [+.51, +.80] | +.32 [+.07, +.56] |
| Amazon-1P | Q | 0.5 | .21 [ .45, +.05] | .69 [ .79, .50] | +.98 [+.95, +.99] | +.19 [ .08, +.43] |
| Amazon-1P | Q | 0.7 | .58 [ .75, .37] | .71 [ .83, .55] | +.94 [+.87, +.97] | .20 [ .42, +.07] |
| Amazon-1P | Q | 0.9 | .44 [ .64, .18] | .47 [ .69, .23] | +.87 [+.80, +.91] | .11 [ .36, +.16] |
| data | model | Dec [90% CI] | equiv. | probe | erased | |
|---|---|---|---|---|---|---|
| Amazon-1P | G | 0.5 | +.000 [ .000, +.000] | yes | .00 | .01 |
| Amazon-1P | G | 0.9 | +.003 [+.001, +.005] | yes | .10 | .04 |
| Amazon-1P | Q | 0.5 | +.000 [ .000, +.000] | yes | .00 | .01 |
| Amazon-1P | Q | 0.9 | +.002 [+.000, +.004] | yes | .17 | .07 |
| Amazon-Neg | G | 0.5 | .001 [ .001, .000] | yes | .01 | .04 |
| Amazon-Neg | G | 0.9 | +.001 [+.000, +.001] | yes | .03 | .00 |
| selection, | ||||||
|---|---|---|---|---|---|---|
| data | readout | best Dec | near-best | best Dec | near-best | : Dec Conflict ( Dec ) |
| CEBaB | mean | .966 | .46–.89 | .924 | .02–.28 | .69 .54 ( .012) |
| CEBaB | last token | .968 | .51–.76 | .921 | .15–.27 | .56 .49 ( .003) |
| BiasBios | mean | .998 | .09–.42 | .995 | .04–.05 | .22 .13 ( .001) |
| BiasBios | last token | .998 | .10–.74 | .994 | .06–.01 | .31 .11 ( .005) |
| Amazon-Neg | mean | .992 | .03–.12 | .981 | .06– .00 | .04 .04 (+.000) |
| standard extractors: ( Dec ) | + erased probe: ( Dec ) | |||||||
|---|---|---|---|---|---|---|---|---|
| max Dec | max Conflict | min Leak | Dec Leak | max Conflict | min Leak | Dec Leak | ||
| CEBaB | Q | .69 | .54 ( .012) | .50 ( .013) | .52 ( .008) | .34 ( .007) | .34 ( .007) | .33 ( .011) |
| G | .41 | .26 ( .005) | .27 ( .004) | .26 ( .005) | .11 ( .004) | .12 ( .004) | .14 ( .010) | |
| BiasBios | Q | .22 | .13 ( .001) | .09 ( .004) | .11 ( .021) | .03 ( .002) | .03 ( .002) | .03 ( .002) |
| G | .24 | .14 ( .002) | .12 ( .006) | .08 ( .017) | .02 ( .002) | .02 ( .001) | .02 ( .001) | |
| Amazon-Neg | Q | .04 | .04 † (+.000) | .07 † ( .007) | .05 † ( .009) | .00 ( .000) | .00 ( .000) | .00 ( .000) |
| best | #near | near-best | equiv. | Conflict | ||
| Dec | (equiv.) | range | range | range | ||
| (correlated training data) | ||||||
| CEBaB | Q | .966 | 37 (9) | .46–.89 | .54–.83 | .44–.69 |
| G | .979 | 39 (8) | .26–.72 | .25–.63 | .57–.82 | |
| BiasBios | Q | .998 | 66 (65) | .09–.42 | .09–.42 | .83–.97 |
| G | .998 | 66 (61) | .09–.33 | .11–.33 | .92–.97 | |
Language Model Probing
Language model probing measures what information and capabilities models encode and how those signals relate to their behavior.
See all 298 papers