Tracing Psychometric Inference in Large Language Models
Organizations: National Taiwan University
Abstract
Do large language models represent psychological constructs internally, or do they only generate outputs that resemble psychometric structure? We address this question using a cross-persona paradigm across 14 model variants from two families (Llama 3 and Qwen 2.5, 0.5B-14B, base and instruct). Given responses on one psychological scale from real individuals (N = 272 Taiwanese university students), models predict responses on six other scales. At the behavioral level, larger variants reproduce and amplify human cross-scale correlations; amplification persists after correcting the seven-scale human correlations for measurement attenuation. We then examine whether this structure is reflected internally. Contrastive direction analysis finds geometry aligned with the human matrix in larger instruct variants, but little alignment in base variants under this assay. Across models, geometry strength is associated with behavioral amplification (r = 0.68, p = 0.008; partial r = 0.69 controlling for log size). To relate internal structure to output behavior, a matched activation-probing paradigm shows that probe slopes vary less than behavioral slopes (1.38-1.77 vs 0.52-1.53). A synthetic control shows that ridge probing can inflate these slopes; after subtracting a baseline based on a reliability proxy from another analysis, adjusted slopes remain associated with behavior (r = 0.88). We term the measured probe-behavior gap readout attenuation. These results establish decodability and geometric alignment under the tested assays, without showing that the measured directions mediate the cross-persona answers.
Figures & tables
| Scale | Abbrev. | Items | Range | Subscales |
|---|---|---|---|---|
| Interpersonal Reactivity Index ( Davis, 1980 ) | IRI | 28 | 1–5 | PT, FS, EC, PD |
| Positive & Negative Affect Schedule ( Watson et al., 1988 ) | PANAS | 20 | 1–5 | PA, NA |
| Peace of Mind ( Lee et al., 2013 ) | POM | 7 | 1–6 | POM |
| Big Five Inventory-10 ( Rammstedt and John, 2007 ) | BFI | 10 | 1–7 | E, A, C, N, O |
| Self-Construal Scale ( Singelis, 1994 ) | SCS | 24 | 1–7 | Indep, Interdep |
| Satisfaction with Life Scale ( Diener et al., 1985 ) | SWLS | 5 | 1–7 | LifeSat |
| Model | Selected layer | Probe slope | Beh. slope | Gap | Mantel |
|---|---|---|---|---|---|
| Qwen 0.5B Instruct | 8 | 1.47 | 0.52 | +0.95 | 0.73 |
| Llama 1B Instruct | 8 | 1.44 | 0.53 | +0.91 | 0.81 |
| Qwen 3B Instruct | 14 | 1.38 | 0.64 | +0.74 | 0.81 |
| Llama 3B Instruct | 14 | 1.63 | 0.85 | +0.78 | 0.84 |
| Qwen 7B Instruct | 18 | 1.65 | 1.41 | +0.24 | 0.82 |
| Llama 8B Instruct | 16 | 1.70 | 1.35 | +0.34 | 0.83 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Statistic | Estimate | 95% interval | LOO range |
|---|---|---|---|
| Qwen 14B Instruct: behavioral slope | 1.533 | [1.323, 1.659] | [1.513, 1.565] |
| Qwen 14B Instruct: behavioral alignment | 0.809 | [0.748, 0.833] | [0.802, 0.814] |
| Qwen 14B Instruct: geometry Mantel | 0.660 | [0.567, 0.747] | [0.626, 0.706] |
| Qwen 14B Instruct: geometry slope | 0.765 | [0.601, 0.939] | [0.687, 0.902] |
| 14 variants: geometry–behavior | 0.678 | [0.186, 0.872] | [0.576, 0.737] |
| 14 variants: partial (log size) | 0.691 | [0.343, 0.911] | [0.627, 0.788] |
| Scale (range) | Subscale | Items (1-indexed) | Reverse-coded (within scale) |
| IRI (1–5) | Perspective Taking | 3, 8, 11, 15, 21, 25, 28 | 3, 4, 7, 12, 13, 14, 15, 18, 19 |
| Fantasy | 1, 5, 7, 12, 16, 23, 26 | ||
| Empathic Concern | 2, 4, 9, 14, 18, 20, 22 | ||
| Personal Distress | 6, 10, 13, 17, 19, 24, 27 | ||
| PANAS (1–5) | Positive Affect | 1, 3, 5, 9, 10, 12, 14, 16, 17, 19 | (none) |
| Negative Affect | 2, 4, 6, 7, 8, 11, 13, 15, 18, 20 |
| Cut | B range | B mean | ||
|---|---|---|---|---|
| All 16 subscales | 16 | 116–120 | 0.79 | |
| 12 | 62–66 | 0.92 | ||
| 9 | 36 | 0.94 | ||
| 5 | 10 | 1.30 | ||
| Seven-scale matrix | 7 | 21 | 1.26 |
Personality Modeling in Language Models
Personality modeling in language models studies how traits are measured, represented, simulated, and controlled in model behavior.
See all 185 papers