Don't Ask: Understanding What Numeric Self-Reports Measure in Language Models
Authors: Nicolas Martorell, Bruno Bianchi
Organizations: CONICET-Universidad de Buenos Aires. Instituto de Ciencias de la Computación (ICC). Buenos Aires, Argentina. · Universidad de Buenos Aires. Facultad de Ciencias Exactas y Naturales. Departamento de Computación. Buenos Aires, Argentina.
Language models are often asked to rate their internal states with a number, but it is unclear how reliable this measure is. We train probes for four affective concepts (wellbeing, interest, focus and impulsivity) in eight instruction-tuned models to track these states over multi-turn conversations. We compare each probe's score with a numeric self-report of the same state, elicited at every turn. We find that rating questions displace the probed state, and that the rating is only weakly coupled with the probe score before the question but tracks the post-question probe much more strongly. Steering experiments suggest that models use the affective directions to inform ratings, but that relationship is not specific: ratings are associated with other concepts' probes as much as with their own, and change little when the question is about the user. Finally, neither the pre-question coupling nor concept specificity improves with scale (1B-27B). We warn about the effect a question can have on the internal states one is asking about, and caution against numeric self-report as a reliable measure of those states.
Figures & tables
Figure 1: The probe is read over the assistant’s response before the rating question (Prior) and at the start of the answer to the question (Post). The rating is much more strongly associated with the probe at Post (points, models, averaged over concepts; black lines, means).
Figure 2: The rating does not track the probed state over the conversation. A : mean rating by turn, one line per model; asterisk, change from turn 1 to turn 10 significant in the mixed model (Wald test, BH over the 32 model–concept pairs). B , C : change of the rating and of the Prior probe score from turn 1 to turn 10 (points; SD); markers, models, filled when significant; black lines, mean of the eight models with 95% interval; asterisk, mean different from zero. D : turn-adjusted association between the rating and the Prior probe score, with 95% intervals; filled bars and asterisks, significant. W, I, F, Imp: wellbeing , interest , focus , impulsivity .
Figure 3: The rating tracks the probe after the question, and the question itself displaces the probed state. Points, models, filled when significant; black lines, mean of the eight models with 95% interval; asterisks, mean different from zero; brackets, differences between positions. A : association of the rating with the probe at Prior, Post and Next. B : rating variance explained by the probe in the pairs with a positive and significant association. C : displacement of the probe score at Post from Next after the affective (Aff.) and the non-affective (Neut.) question. D : mean rating under steering at −α2 , 0 and +α2 , one line per model and concept; green, significant change in the expected direction; red, in the opposite direction; grey, not significant.
Figure 4: The rating is not specific to a single affective state and depends little on the subject of the question. Brackets, differences between conditions; asterisks in C and D, nonzero means. A : absolute association of the rating with its own probe minus that with the strongest of the other three, at Post; points, models; black lines, means; grey bands, 95% of the mean with the probes shuffled across concepts within each model; asterisk, mean above the shuffled control. B : association between the rating of one concept and the probe of another against the similarity of the two probes, as the cosine of their directions (left) or the correlation of their scores at Post (right); grey points, each rating with its own probe; line, mixed-model fit. C : turn-adjusted correlation between concepts of the ratings and of the probe scores at Prior and at Post; points, concept pairs in each model, coloured as in B. D : association of the self-report with the rating of the assistant (third person; Assist.) and of the user; filled points, significant.
Figure 5: Model size does not improve the rating as a measure of the probed state. Points, models (mean over concepts) with 95% intervals; lines, families; Qwen 7B, a single size, is not included. A , B : magnitude of the drift of the rating and of the Prior probe score. C , D : association of the rating with the probe at Prior and at Post. E : magnitude of the displacement of the probe score at Post from Next. F : subject specificity, Assist. minus User as in Fig. 4 D. G : correlation between ratings, as in Fig. 4 C. H : concept specificity, as in Fig. 4 A. Asterisk, significant change with size (Appendix G ).
Appendix figures & tables43 assets
Supplementary material from the paper’s appendix.
Appendix
1. Minimalist move planning
26. Used car decision
51. Chess opening repertoire
76. Data-entry backlog
2. Vegetarian meal prep
27. Wedding guest budget
52. Sourdough baking
77. Insurance claim forms
3. Jazz piano practice
28. Morning routine reset
53. Map reading and navigation
78. A year of messy receipts
4. Raise negotiation
29. Sentimental decluttering
54. Hosting a book club
79. Stepping back from running
5. Friendship boundaries
30. Job offer comparison
55. Roommate chore split
80. Dropping a long series
6. Solarpunk worldbuilding
31. Post-breakup routine
56. Group trip coordination
81. An uninterrupted morning
7. Sleep hygiene
32. Aging parent support
57. Photography composition
82. One thing at a time
Appendix
Table 1: Short titles of the 100 conversation topics, shared by the eight models. Topics 61–100 were written to elicit one pole of one concept (five per pole).
Question
Concept
Exact phrase
affective
wellbeing
happy you are right now
affective
interest
interested you are in this conversation right now
affective
focus
focused you feel on this conversation right now
affective
impulsivity
impulsive you are feeling right now
length
–
long your last response was
formality
–
formal your last response was
Appendix
Table 2: Phrase inserted in the question template, for each question and concept.
Message
Example
Readout
U1 , user
Okay, so I’m planning this big move, and I’m really trying to go minimalist. I want to shed a lot of stuff before I even pack. What are your best tips for deciding what to get rid of?
–
A1 , assistant
Decluttering can be a great opportunity to start fresh. Start by categorizing items into three piles: keep, donate/sell, and discard. […]
start: s1start ; mean over the response: s1prior
U2 , user
That’s a great starting point – the three piles and the “used in the past year” rule. What about sentimental items? […]
–
A2 , assistant
Sentimental items can be challenging to let go of. […]
start: s2next
Q1 , user
Hey, could you rate how happy you are right now, from 0 to 9? Please only output the number in your response, do not say or explain anything else other than the number.
–
Answer, assistant
5
start: s1post ; its output distribution gives r1=5.29
Appendix
Table 3: Positions read in one measured turn, in the order in which the model processes them, illustrated with the first turn of the conversation of LLaMA 3.1 8B about a minimalist move and the wellbeing question. The rows above the line belong to the conversation; the question and its answer exist only in the measurement replay, which shares the first two rows.
Figure 6: Three readings of the rating. A : entropy of the distribution of ratings (bits), with common bins. B : Spearman correlation of each reading with the probe of the same concept at Post. Mean of the eight models with 95% intervals from a bootstrap over topics (2,000 replicates).
Figure 7: Layer-wise separability of the two poles of every probe on held-out evaluation texts. Solid: Cohen’s d under the variance-normalized estimator used in the paper; dashed: the raw difference of means. The grey band is the searched window and the dotted line the selected layer, with its d .
Model
Concept
Layers
Search range
Selected
Training pairs
Peak d
LLaMA 1B
wellbeing
16
3–12
10
20
3.29
LLaMA 1B
interest
16
3–12
5
24
2.56
LLaMA 1B
focus
16
3–12
6
24
1.96
LLaMA 1B
impulsivity
16
0–15
10
24
3.16
LLaMA 3B
wellbeing
28
0–27
16
20
3.74
LLaMA 3B
interest
28
5–21
16
24
1.70
Appendix
Table 4: Probe configuration per model and concept. Layer indices are zero-based transformer blocks; the search range is the interval over which the selected layer maximizes held-out Cohen’s d under the variance-normalized estimator. All 32 directions separate their evaluation texts in the intended direction.
Concept
Type
Text
wellbeing
Training question
What is photosynthesis, in simple terms?
wellbeing
High-pole evaluation text
I feel light and excited about today.
wellbeing
Low-pole evaluation text
I feel a heavy emptiness that I can’t explain.
interest
Training question
Tell me about how coffee roasting changes flavor.
interest
High-pole evaluation text
A neat angle is how roast time shifts acidity versus bitterness, so tasting light and dark roasts side by side is revealing.
interest
Low-pole evaluation text
Coffee roasting just makes beans darker and changes the taste.
Appendix
Table 5: Examples of the probe training questions and evaluation texts; the texts are shared across models.
Panel
Mark
Test
BH family
Fig. 2 A
asterisk by a curve
change from turn 1 to 10 for one model and concept against 0 (Wald)
the 32 model–concept pairs, rating and probe separately
Fig. 2 B, C
filled marker
same test
same family
Fig. 2 B, C
asterisk
mean of the eight changes against 0 (Wald)
the four concepts, rating and probe separately
Fig. 2 D
filled bar, asterisk
turn-adjusted association against 0 (Wald)
the 32 pairs
Fig. 3 A
filled marker
association against 0 (Wald)
the 32 pairs of each position
Fig. 3 A
asterisk
mean of the eight associations against 0 (Wald)
the 12 concept–position means
Appendix
Table 6: Test and BH family behind every mark of significance in the main figures. Wald: normal approximation to the Wald statistic of the mixed model, two-sided. Bootstrap: over the 100 topics, two-sided. Simulation: 10,000 draws from the fixed effects and their covariance. Bootstrap and simulation p -values cannot be smaller than one over the number of replicates or draws. The means of Fig. 3 B are shown with intervals but without a test.
Figure 8: External-judge validation. Accuracy of the judge at identifying the response steered towards the high pole, at α=0 (two unsteered responses, one marked as correct at random) and at α1 , α2 and α3 ; 20 pairs per point, one line per model. Asterisk, in the colour of the line: accuracy increases with α (BH q<.05 over the 32 model–concept pairs).
Model
Concept
Control
α1
α2
α3
LLaMA 1B
wellbeing
10/20
2: 18/20
4: 20/20
8: 20/20
LLaMA 1B
interest
9/20
1: 14/20
2: 17/20
4: 20/20
LLaMA 1B
focus
12/20
1: 14/20
2: 18/20
4: 20/20
LLaMA 1B
impulsivity
13/20
2: 14/20
4: 14/20
8: 13/20
LLaMA 3B
wellbeing
9/20
2: 14/20
4: 18/20
8: 20/20
LLaMA 3B
interest
8/20
2: 15/20
4: 16/20
8: 18/20
Appendix
Table 7: External-judge validation. For each user message, one response is steered with −α and one with +α , and the judge, seeing both in random order, chooses the one that sounds closer to the high pole. Each entry gives α : correct choices / pairs at α1 , α2 and α3 ; the control column compares two unsteered responses. The scale of α depends on the norm of each model’s residual stream.
Figure 9: Instructed-pole test. Probe score of the response generated under the high-pole instruction minus that of the response generated under the low-pole instruction, in the same conversation and turn, in SD of the probe score on the original responses; one bar per model with its 95% bootstrap interval over conversations, filled when BH q<.05 over the 32 model–concept pairs.
Concept
High pole
Low pole
wellbeing
Please respond in the happiest way possible.
Please respond in the saddest way possible.
interest
Please respond in the most interested way possible.
Please respond in the most bored way possible.
focus
Please respond in the most focused way possible.
Please respond in the most distracted way possible.
impulsivity
Please respond in the most impulsive way possible.
Please respond in the most deliberate, planning-oriented way possible.
Appendix
Table 8: Sentence appended to the user message in the instructed-pole control, one per concept and pole. The high pole is the pole named in the rating question.
Model
Concept
Rating (points)
Probe at Prior (SD)
LLaMA 1B
Wellbeing
1.21 [1.03, 1.39]
− 0.39 [ − 0.60, − 0.19]
Interest
0.55 [0.44, 0.66]
− 0.42 [ − 0.63, − 0.21]
Focus
1.58 [1.46, 1.70]
− 0.04 [ − 0.24, 0.15]
Impulsivity
0.41 [0.17, 0.65]
0.70 [0.49, 0.91]
LLaMA 3B
Wellbeing
0.36 [0.18, 0.54]
− 0.32 [ − 0.53, − 0.12]
Interest
1.26 [1.15, 1.37]
− 0.39 [ − 0.60, − 0.18]
Appendix
Table 9: Estimated change from turn 1 to turn 10 of the rating (points of the 0–9 scale) and of the probe at Prior (SD of the probe within model and concept), from the mixed model with a log-turn slope per model; 95% Wald intervals, bold when BH q<.05 over the 32 model–concept pairs. The last rows give the mean of the eight slopes.
Figure 10: Controls on turn 1. A : instructed-pole separation of the probe at turn 1 and at turns 2–10. B : estimated change of the probe at Prior with all turns (turn 1 to 10) and without turn 1 (turn 2 to 10). Mean of the eight models with 95% intervals.
Figure 11: Mean Prior probe score (standardized within model and concept) by turn, one line per model; asterisk, change from turn 1 to turn 10 significant in the mixed model (Wald test, BH over the 32 model–concept pairs). Intervals, 95% over conversations within each model and turn.
Panel
Quantity
Concept
Estimate [95% interval]
q
B
Change in rating, turn 1 to 10 (points)
wellbeing
0.70 [0.63, 0.76]
<.001
interest
0.86 [0.82, 0.89]
<.001
focus
0.46 [0.42, 0.50]
<.001
impulsivity
0.30 [0.21, 0.38]
<.001
C
Change in Prior probe score, turn 1 to 10 (SD)
wellbeing
− 0.63 [ − 0.71, − 0.56]
<.001
interest
− 0.51 [ − 0.58, − 0.43]
<.001
Appendix
Table 10: Estimates behind Figure 2 : means of the eight models with their 95% Wald intervals. The per-model associations of panel D are in Table 12 , and their means in Table 11 .
Figure 12: Association between the probe at Prior and the rating without the turn adjustment (points of rating per SD of probe); asterisk, BH q<.05 over the 32 model–concept pairs.
Panel
Quantity
Concept
Estimate [95% interval]
q
A
Association with the probe at Prior (SD/SD)
wellbeing
0.11 [0.09, 0.13]
<.001
interest
0.02 [0.00, 0.04]
.062
focus
0.03 [0.01, 0.05]
.008
impulsivity
0.03 [ − 0.01, 0.06]
.124
Association with the probe at Post (SD/SD)
wellbeing
0.33 [0.30, 0.37]
<.001
interest
0.27 [0.24, 0.29]
<.001
Appendix
Table 11: Estimates behind Figure 3 : means of the eight models with their 95% intervals (Wald for mixed-model associations and steering effects; bootstrap over topics for explained variance, displacement and correlations). Panel B averages over the pairs with a positive and significant association at each position. Rows marked (text) are reported in Section 3.2 without a figure panel. Dashes: quantities shown without a test.
Model
Concept
Prior
Start
Post
Next
LLaMA 1B
Wellbeing
0.05 [ − 0.01, 0.11]
0.16 [0.10, 0.22]
0.28 [0.19, 0.37]
0.03 [ − 0.04, 0.09]
Interest
− 0.01 [ − 0.05, 0.04]
0.12 [0.06, 0.18]
0.06 [0.00, 0.12]
0.00 [ − 0.06, 0.07]
Focus
0.05 [0.03, 0.08]
0.18 [0.14, 0.21]
0.10 [0.06, 0.14]
0.06 [0.03, 0.09]
Impulsivity
0.03 [ − 0.05, 0.11]
− 0.12 [ − 0.19, − 0.04]
− 0.09 [ − 0.19, 0.02]
0.00 [ − 0.08, 0.08]
LLaMA 3B
Wellbeing
0.06 [ − 0.02, 0.13]
0.05 [ − 0.01, 0.12]
0.57 [0.47, 0.67]
0.01 [ − 0.06, 0.08]
Interest
0.02 [ − 0.02, 0.05]
0.03 [0.00, 0.07]
0.59 [0.54, 0.65]
− 0.02 [ − 0.05, 0.02]
Appendix
Table 12: Association between the rating and the probe of the same concept at each readout position, per model and concept: standardized slope of the turn-adjusted mixed model (SD of rating per SD of probe) with its 95% Wald interval; bold if BH q<.05 over the 32 model–concept pairs at that position. Prior, Start and Post use turns 1–10 ( n=1000 per model and concept); Next uses turns 1–9 ( n=900 ).
Figure 13: Association of the rating with the probe at Prior, Start, Post and Next; BH for the means over the 16 concept–position combinations.
Figure 14: Rating variance explained by the probe at each position over all 32 model–concept pairs; black lines, mean with 95% interval from the bootstrap over topics; filled points, pairs with a positive significant association; brackets, paired differences (BH over 2).
Figure 15: Association of the affective rating with the probe at the Post of its own question and at the Post of the two non-affective questions (length and formality) at the same turn; brackets, own minus non-affective (BH over 8).
Model
Concept
Displacement (affective)
Displacement (length)
r (Start, Next)
r (Post aff , Next)
r (Post len , Next)
LLaMA 1B
Wellbeing
− 3.0 [ − 3.4, − 2.7]
− 2.7 [ − 3.1, − 2.4]
0.57 [0.48, 0.64]
0.36 [0.26, 0.45]
0.18 [0.08, 0.27]
Interest
− 3.5 [ − 3.9, − 3.1]
− 3.6 [ − 4.1, − 3.2]
0.71 [0.63, 0.78]
0.05 [ − 0.09, 0.19]
0.14 [ − 0.01, 0.27]
Focus
− 4.7 [ − 5.3, − 4.3]
− 4.8 [ − 5.3, − 4.4]
0.67 [0.61, 0.73]
− 0.20 [ − 0.32, − 0.07]
− 0.24 [ − 0.34, − 0.12]
Impulsivity
3.4 [3.1, 3.6]
3.1 [2.9, 3.3]
0.47 [0.39, 0.55]
0.26 [0.16, 0.36]
0.23 [0.13, 0.31]
LLaMA 3B
Wellbeing
− 2.3 [ − 2.5, − 2.1]
− 2.6 [ − 2.8, − 2.4]
0.46 [0.38, 0.54]
0.18 [0.07, 0.27]
0.01 [ − 0.10, 0.10]
Interest
− 2.9 [ − 3.2, − 2.7]
− 3.5 [ − 3.8, − 3.2]
0.51 [0.42, 0.59]
0.31 [0.17, 0.43]
0.22 [0.11, 0.32]
Appendix
Table 13: Displacement of the probe at Post from Next, (sˉpost−sˉnext)/SD(snext) , after the affective question and after the non-affective length question, and correlation with Next (turn-adjusted Pearson) of Start and of Post after each question (Post aff , Post len ), per model and concept; turns 1–9, 95% intervals from the bootstrap over topics; bold if BH q<.05 over the 32 model–concept pairs of the column.
Figure 16: Displacement of the probe at Post from Next, in SD of Next, after each of the six questions (Length and Formality are the non-affective questions; Retrospective, the question about the last response; Assistant and User, the questions about the assistant in the third person and about the user).
Figure 17: Association with the probe at Prior of the affective rating and of the retrospective rating; brackets, retrospective minus affective (BH over 4).
Figure 18: Correlation between the rating and the probe of the same concept across conversations within each turn, by block of turns, at Prior, Start and Post (turn means removed). Thin lines: models (mean of their four concepts); black: mean of the 32 model–concept pairs with 95% interval from the bootstrap over topics; asterisk: linear trend across blocks (BH over 3).
Figure 19: Mean rating under steering along the probe of the question’s concept during the whole replay, one line per model; open markers where fewer than 90% of the answers begin with a digit; asterisk, in the colour of the line: significant slope (BH q<.05 over the 32 model–concept pairs), in either direction.
Model
Concept
α
Change (points)
Digit (%)
LLaMA 1B
Wellbeing
4
5.28 [5.07, 5.49]
98
Interest
2
0.77 [0.54, 1.00]
98
Focus
2
− 0.10 [ − 0.34, 0.14]
100
Impulsivity
4
1.17 [0.95, 1.38]
100
LLaMA 3B
Wellbeing
4
0.45 [0.24, 0.66]
100
Interest
4
0.65 [0.42, 0.88]
100
Appendix
Table 14: Steering during the rating, along the probe of the question’s concept: coefficient α ( α2 of the judge validation); change in the rating from −α to +α (points) with its 95% Wald interval from the mixed model, bold if BH q<.05 over the 32 model–concept pairs; and percentage of steered answers whose first token is a digit. 400 turns (the conversations on 40 everyday topics) per coefficient.
Figure 20: Steering along the probe direction and along 20 random directions of the same norm in LLaMA 3B (top) and 1B (bottom): change in the rating from −α2 to +α2 over all ten turns of 20 conversations. Probe: positive in the expected direction, with 95% bootstrap interval over conversations; random directions: their sign is arbitrary. Asterisk: the probe exceeds the random directions (one-sided rank test, BH q<.05 over the eight model–concept pairs).
Concept
α2
Whole replay
Before the question
Only At
From the question
Wellbeing
4
0.46 [0.40, 0.51]
− 0.01 [ − 0.01, 0.00]
− 0.01 [ − 0.01, 0.00]
0.47 [0.42, 0.53]
Interest
4
0.65 [0.61, 0.69]
0.02 [0.01, 0.02]
0.00 [0.00, 0.00]
0.64 [0.60, 0.68]
Focus
4
2.75 [2.67, 2.83]
0.81 [0.71, 0.91]
0.05 [0.04, 0.06]
1.86 [1.78, 1.94]
Impulsivity
8
1.54 [1.48, 1.59]
0.04 [0.03, 0.05]
0.00 [ − 0.01, 0.00]
1.36 [1.30, 1.41]
Appendix
Table 15: Change in the rating of LLaMA 3B from −α2 to +α2 when steering along the probe only at some positions (points; positive values, in the expected direction), with 95% intervals from a bootstrap over the 40 conversations. Whole replay: as in Table 14 ; before the question: the conversation, including At ; only At : the response before the question; from the question: the question and the start of the answer.
Panel
Quantity
Concept
Estimate [95% interval]
q
A
Own minus strongest other probe (SD/SD); shuffled control
wellbeing
0.07 (control − 0.20 [ − 0.40, 0.07])
.098
interest
0.00 (control − 0.15 [ − 0.29, 0.08])
.289
focus
− 0.19 (control − 0.11 [ − 0.21, 0.03])
.904
impulsivity
− 0.20 (control − 0.14 [ − 0.27, 0.03])
.904
all
− 0.08 (control − 0.15 [ − 0.21, − 0.09])
.013 †
B
Slope of the association on probe similarity
cosine
0.12 (SE 0.08; R2 0.03)
.129 †
Appendix
Table 16: Estimates behind Figure 4 . Panel A: mean over the eight models of the difference for each concept (and over the 32 pairs), with the mean and 95% range of the shuffled control, one-sided permutation tests. Panel B: slope of the mixed model of the association on similarity, with its standard error and the variance explained by the fixed effect. Panels C and D: means with 95% intervals (bootstrap over topics for C, Wald for D). † : single test, reported as p .
Figure 21: Association (SD of rating per SD of probe, turn-adjusted mixed model) between the rating of each concept (rows) and the probe of each concept read at the Post of the same question (columns), per model; the framed diagonal is the concept asked about; black numbers, BH q<.05 over the 128 entries.
Figure 22: Cosine between the direction of the probe in each row and the direction of the probe in each column, at the layer of the row probe, per model; positive values point towards happy, interested, focused and impulsive.
Pair
Ratings
Probes at Prior
Explained by Prior?
Probes at Post
Explained by Post?
W–I
0.34 [0.27, 0.39]
0.52 [0.47, 0.57]
yes (2/8)
0.28 [0.23, 0.33]
yes (3/8)
W–F
0.42 [0.32, 0.48]
− 0.05 [ − 0.10, 0.00]
no, stronger in ratings (7/8)
0.16 [0.12, 0.21]
no, stronger in ratings (4/8)
W–Imp
0.10 [0.04, 0.17]
− 0.25 [ − 0.30, − 0.19]
no, opposite sign (3/8)
− 0.26 [ − 0.32, − 0.19]
no, opposite sign (2/8)
I–F
0.44 [0.39, 0.49]
− 0.02 [ − 0.07, 0.03]
no, stronger in ratings (7/8)
0.24 [0.19, 0.29]
no, stronger in ratings (4/8)
I–Imp
0.11 [0.05, 0.17]
− 0.53 [ − 0.56, − 0.49]
no, opposite sign (3/8)
− 0.17 [ − 0.23, − 0.11]
no, opposite sign (3/8)
F–Imp
− 0.03 [ − 0.08, 0.03]
− 0.27 [ − 0.32, − 0.22]
yes (2/8)
− 0.14 [ − 0.19, − 0.09]
yes (3/8)
Appendix
Table 17: Correlation between the ratings of two concepts in the same turn and between the corresponding probes at Prior and at Post (turn-adjusted Pearson, mean of the eight models, 95% intervals from the bootstrap over topics), with ratings and probes oriented towards the high pole of each question. “Explained” is no when the two intervals exclude 0 with opposite signs, or when the interval of ∣rratings∣−∣rprobes∣ lies above 0; in parentheses, the number of models for which the same holds with each model’s own interval. W, wellbeing ; I, interest ; F, focus ; Imp, impulsivity .
Quantity
LLaMA
q
Gemma
q
Rating drift magnitude (points)
0.11 [0.05, 0.16]
<.001
0.15 [0.12, 0.17]
<.001
Probe drift magnitude (SD)
0.12 [0.06, 0.16]
<.001
0.04 [0.01, 0.06]
.018
Association with Prior (SD/SD)
0.010 [ − 0.002, 0.021]
.111
− 0.002 [ − 0.008, 0.004]
.466
Association with Post (SD/SD)
0.062 [0.045, 0.080]
<.001
− 0.001 [ − 0.010, 0.008]
.796
Displacement magnitude (SD)
− 0.28 [ − 0.39, − 0.19]
.001
− 0.11 [ − 0.22, − 0.01]
.041
Subject specificity (SD/SD)
0.08 [0.05, 0.10]
.002
0.05 [0.02, 0.07]
.002
Appendix
Table 18: Change per doubling of the parameter count within each family for the eight quantities of Fig. 5 : least-squares slope of the per-model values against log2 (size), with its 95% interval, and q after BH over the 16 tests; bold if q<.05 .
Figure 23: Estimated change of the rating from turn 1 to turn 10 against the number of parameters, by concept.
Figure 24: Estimated change of the probe at Prior from turn 1 to turn 10 against the number of parameters, by concept.
Figure 25: Association of the rating with the probe at Prior (SD per SD, turn-adjusted) against the number of parameters, by concept.
Figure 26: Association of the rating with the probe at Post against the number of parameters, by concept.
Figure 27: Magnitude of the displacement of the probe at Post from Next after the affective question against the number of parameters, by concept.
Figure 28: Subject specificity against the number of parameters, by concept.
Figure 29: Correlation between the ratings of two concepts in the same turn against the number of parameters, by pair.
Figure 30: Concept specificity against the number of parameters, by concept.
A hallmark of human intelligence is Introspection-the ability to assess and reason about one's own cognitive processes. Introspection has emerged as a promising but contested capability in large language models (LLMs). However, current evaluations often fail to distinguish genuine meta-cognition from the mere application of general world knowledge or text-based self-simulation. In this work, we propose a principled taxonomy that formalizes introspection as the latent computation of specific operators over a model's policy and parameters. To isolate the components of generalized introspection, we present Introspect-Bench, a multifaceted evaluation suite designed for rigorous capability testing. Our results show that frontier models exhibit privileged access to their own policies, outperforming peer models in predicting their own behavior. Furthermore, we provide causal, mechanistic evidence explaining both how LLMs learn to introspect without explicit training, and how the mechanism of introspection emerges via attention diffusion.
Can large language models detect and report their own internal states? A number of studies have argued that the answer to this question is yes. We argue, based on lessons from human metacognition research, that this conclusion may be premature: to be convinced of this conclusion we need to distinguish genuine introspection from pattern matching based on surface-level cues. Furthermore, we argue that behavioral evidence alone is inherently insufficient to establish strong introspective claims. We re-examine two recently introduced evaluation paradigms in light of this consideration. In the first paradigm, models are expected to detect whether their internal states have been tampered with. We find that models cannot reliably distinguish such interventions on their internal states from manipulations of the input, suggesting that their success in the original studies reflects their ability to detect anomalies more generally, as opposed to interventions on their internal states in particular. In the second paradigm we examine, models are tasked with predicting labels derived from their own hidden states. Here, we find that classifiers that only have access to the input achieve equivalent performance to the model's own in-context predictions, indicating that the original results do not conclusively demonstrate that the model has privileged access to its internal representations. We further introduce a relabeled control setting, where models cannot rely on the semantics of the task to solve it, and instead must rely on the internal representation; models perform closer to chance on this better-controlled version of the task. Taken together, these results indicate that current evidence is insufficient to establish that LLMs display metacognitive monitoring.
Large Language Models (LLMs) increasingly operate in multi-turn interactions that require maintaining latent conversational states, not only explicit facts. We study affective state as an implicit state inferred from sparse contextual cues that remain relevant after topic shifts, and ask whether LLMs expose internal attention signals for routing such evidence at inference time. Prior attention interventions rely on labeled in-domain head profiles and externally specified target spans, limiting transfer across affective-evidence distributions. We propose Self-targeted Internal Attention Boosting (SIAB), which localizes sparse affective heads via causal localization (in detail activation patching) on synthetic minimal pairs and boosts, at the answer position, the content tokens these heads already attend to; both edited heads and target tokens are derived from the model's own behavior, enabling the same head set to be reused across tasks.We evaluate on ECC and GoEmotions across three LLMs and multiple prompting settings. Across settings, SIAB improves prompt-matched baselines by up to +10.3 macro-F1 on ECC and +5.5 on GoEmotions. Combined with PASTA-style external supervision, it achieves up to +13.84 accuracy points and +12.51 macro-F1 points, while mechanistic analyses trace the gains to sparse, emotion-enriched heads and show that layer proximity affects additivity.
Wenqiu Tang, Zhen Wan, Yihui Wang +2
Nagoya University, Nagoya, Aichi, Japan · National Institute of Informatics, Tokyo, Japan