Organizations: Department of Computer Science, Southern Methodist University, Dallas, TX, USA · Department of Psychological and Brain Sciences, Washington University in St. Louis, St. Louis, MO, USA
Personality is only weakly and inconsistently signaled by written language, and existing text-based methods typically combine language-model and psycholinguistic representations at the input or decision level. It remains open whether a more deliberate combination of the two views, and a decomposition of the prediction target, can recover more of the signal. We present a three-phase framework for estimating Big Five personality domain scores from life-narrative interview text. It encodes text and psycholinguistic features separately and combines them by concatenation or by a learned weighting, decomposes each domain into contrastively trained facet embeddings instead of predicting the domain directly, and pools the facet embeddings by attention into a domain-level prediction. We compare end-to-end with staged training and ablate the input modalities, their combination, and the facet-pooling strategy. On 1,332 participants from the St. Louis Personality and Aging Network (SPAN) study, facet embeddings reach AUC up to 0.740, and domain-level R2 reaches 0.328, highest for Openness. End-to-end training gives lower error than staged training in every domain, and attention pooling gives the lowest error in every domain. Combining the two views lowers error relative to text alone for Conscientiousness and Agreeableness, whereas a learned weighting of the views shows no consistent advantage over concatenation. The framework provides facet- level predictions that a domain-only model cannot, and it improves modestly on a regression from the raw features in four of the five domains.
Figures & tables
Figure 1: Phase 1 encodes the text and psycholinguistic views, Phase 2 combines them and learns six contrastive facet embeddings, and Phase 3 pools them with attention to predict the domain score.
Component
Parameter
Value
Embeddings
Dimension of zpt , zps
256
Psy encoder
Token width d
128
Attention heads
4
Layers L
2
Feed-forward width
256
Dropout
0.1
Table 1: Hyperparameters.
Domain
AUC μ
R2
r
RMSE
MAE
O
0.694
0.328
0.596
14.76
11.70
C
0.664
0.180
0.430
17.01
12.93
E
0.660
0.154
0.402
16.83
13.16
A
0.610
0.065
0.269
13.21
10.27
N
0.605
0.008
0.148
19.67
16.06
Table 2: Domain-level performance of the full model (text and psycholinguistic embeddings combined by concatenation, attention-based facet pooling, joint training) on the 266 held-out participants. AUC μ is the mean facet-level AUC across the domain’s six facets; R2 , Pearson r , RMSE, and MAE are computed on the original domain-score scale (0–192).
Facet
AUC
F1
Prec
Rec
Openness
O1
0.681
0.625
0.626
0.626
O2
0.723
0.670
0.670
0.675
O3
0.642
0.615
0.616
0.620
O4
0.656
0.594
0.595
0.598
O5
0.717
0.683
0.684
0.686
Table 3: Per-facet classification performance of the full model (text and psycholinguistic embeddings combined by concatenation, attention-based facet pooling, joint training) on the 266 held-out participants. Each held-out participant is scored by the difference in cosine similarity to the high and low class prototypes computed from training embeddings. AUC, F1, precision, and recall are macro-averaged across the high and low classes.
Openness
Conscientiousness
Extraversion
Agreeableness
Neuroticism
Model
R2↑
MSE ↓
R2↑
MSE ↓
R2↑
MSE ↓
R2↑
MSE ↓
R2↑
MSE ↓
SVR on LIWC + NRC
0.134
280.66
0.097
318.58
0.156
282.49
0.037
179.84
0.027
379.31
MLP on BERT
0.218
253.62
0.005
351.08
0.156
282.41
−0.342
250.45
−0.069
416.80
MLP on Mistral
0.186
263.76
0.128
307.54
0.039
321.78
−0.222
228.11
−0.135
442.43
PsyAttention
−0.030
333.77
−0.067
376.29
−0.007
336.96
−0.015
189.41
−0.055
411.41
FACET (ours)
0.328
217.90
0.180
289.32
0.154
283.31
0.065
174.53
0.008
386.86
Table 4: Comparison with baselines on the 266 test participants. R2 and MSE are computed on the original 0–192 domain scale. Best value per column in bold.
O
C
E
A
N
R2
MSE
R2
MSE
R2
MSE
R2
MSE
R2
MSE
Modality
Text only
0.319
220.66
0.109
314.27
0.178
275.17
0.005
185.78
0.010
386.23
Psy only
0.129
282.44
0.048
336.00
0.101
301.10
0.014
184.09
0.005
388.11
Both (concatenation)
0.328
217.90
0.180
289.32
0.154
283.31
0.065
174.53
0.008
386.86
Modality Combination
Table 5: Ablation results for domain-level personality prediction across the Big Five traits. Results examine the contribution of modality inputs, modality combination, facet pooling, and training regime with Phases 1–2 removed. For each trait, both R2 and MSE are reported. Best value per column within each block in bold.
Personality traits are richly encoded in natural language, and large language models (LLMs) trained on human text can simulate personality when conditioned on persona descriptions. However, existing evaluations rely predominantly on questionnaire self-report by the conditioned model, are limited in architectural diversity, and rarely use real human psychometric data. Without addressing these limitations, it remains unclear whether personality conditioning produces psychometrically informative representations of individual differences or merely superficial alignment with trait descriptors. To test how robustly LLMs can encode personality into extended text, we condition LLMs on real psychometric profiles from 290 participants to generate first-person life story narratives, and then task independent LLMs to recover personality scores from those narratives alone. We show that personality scores can be recovered from the generated narratives at levels approaching human test-retest reliability (mean r = 0.750, 85% of the human ceiling), and that recovery is robust across 10 LLM narrative generators and 3 LLM personality scorers spanning 6 providers. Decomposing systematic biases reveals that scoring models achieve their accuracy while counteracting alignment-induced defaults. Content analysis of the generated narratives shows that personality conditioning produces behaviourally differentiated text: nine of ten coded features correlate significantly with the same features in participants' real conversations, and personality-driven emotional reactivity patterns in narratives replicate in real conversational data. These findings provide evidence that the personality-language relationship captured during pretraining supports robust encoding and decoding of individual differences, including characteristic emotional variability patterns that replicate in real human behaviour.
Ben Wigler, Maria Tsfasman, Tiffany Matej Hrkalovic
Fine-tuning a language model on data containing a narrow flaw, such as insecure code or incorrect mathematical answers, can cause broad misalignment through a mechanism that remains debated. We provide an interpretable account: in the models and corpora we study, misalignment behaves like a shift in personality. Prior work extracts activation directions for character traits from a single binary contrast, which can separate or steer behavior without establishing a calibrated scale. We instead extract personality vectors for the Big Five using a graded, three-level intervention and validate them on two open-weight models. The three levels are linearly ordered, with Cohen's d values of up to 6.2; the vectors transfer zero-shot and trait-specifically to an independent corpus; and their effects are strongest within a middle-layer band. Applied to training data, the vectors reveal that misaligned corpora across eight domains share a common Big Five signature: lower agreeableness and conscientiousness, together with higher extraversion and neuroticism. This signature is recovered by both models with a correlation of r = 0.94. Fine-tuning imprints the same profile, shifting the model's generations along the corresponding signature, with r = 0.83 using activation-based measurements and r = 0.90 using a text-based judge, while also shifting internal activations with r = 0.69. The same vectors characterize sycophancy as high extraversion and low conscientiousness rather than excess agreeableness, a distinction that a single direction cannot capture. Calibrated personality vectors transform an opaque safety phenomenon into a human-legible diagnostic profile.
This paper investigates whether Large Language Models can answer a personality questionnaire as a specific individual would. Given a user's social-media posts, the model completes the BFI-2-60 Big Five questionnaire as that user. The predicted responses are converted into trait scores and compared with the user's actual scores. We evaluate GPT-4, Gemma, Llama 3.1, and Mistral on the MyPersonality dataset. To determine whether predictions reflect information from the user's writing, we compare them with answers produced without user text and with values derived from general population tendencies. We measure prediction error across all users and separately for individuals with low, intermediate, and high trait scores. We also test whether an LLM can express an explicit personality profile when given only the target Big Five scores, without user text, and examine how predictions change as more text is provided. The results show that user text does not consistently improve performance. Models perform better near the middle of the trait distribution, while at the lower and upper ends they often remain close to central values or perform well at one end but poorly at the other. When given an explicit profile, GPT-4 most clearly translates trait information into consistent questionnaire responses, particularly for Extraversion and Neuroticism. Increasing context does not systematically reduce error: open-source models fluctuate more, whereas GPT-4 remains comparatively stable. Overall, the evaluated models do not consistently reproduce the questionnaire behavior of specific individuals.
Francesco Di Cursi, Chiara Boldrini, Andrea Passarella +1
Institute of Informatics and Telematics (IIT), National Research Council (CNR), Pisa, Italy · Department of Computer Science, University of Pisa, Pisa, Italy