cs.CLSep 19, 2026
SaveMeasuring Behavioural Signatures of Large Language Models through Psychometric Profiling
Organizations: School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong, 518172, China. · Key Laboratory of Quark & Lepton Physics and Institute of Particle Physics, Central China Normal University, Wuhan, 430079, China. · Shandong Science and Technology Press, Shandong Publishing Group, Jinan, 250003, China. · Faculty of Physics, Bielefeld University, Bielefeld, D-33615, Germany. ·
Abstract
Large language models (LLMs) increasingly mediate human decisions and communication, yet their behavioural regularities remain difficult to characterize systematically. We develop a cross-linguistic psychometric profiling framework and evaluate nine LLMs using seven psychological instruments, with five repeated administrations per model and language in Chinese and English. Items unresolved after a prespecified retry procedure are retained as NA. Joint analysis of scored and NA responses captures response tendencies and boundaries of self-report applicability. LLMs exhibit structured, model-specific profiles despite a shared alignment-shaped pattern of higher prosocial and self-regulatory responses and lower dominance, disengagement and harmful-intent endorsement. NA responses are structured rather than uniformly distributed, indicating where outputs are treated as inapplicable, refused or cannot be mapped to valid response options. Language condition and provider origin are associated with profile configuration and answerability, whereas repeated administrations show high reproducibility and permit recovery of model identity. Human-reference and prompt-robustness analyses further indicate that these signatures are context dependent. Joint analysis of psychometric profiling and answerability offers a framework for quantifying deployment-level behavioural signatures.
Figures & tables
Figure 1: Psychometric assessment framework, evaluated LLMs and psychological instruments . a , Overview of the assessment workflow, including the psychometric scale library, standardized prompt-instruction library, API-based LLM interaction and psychometric profile analysis. b , Sankey diagram of the evaluated LLMs, organized by origin group, provider, model family or version and language condition. Each model is tested in both Chinese and English. c , Hierarchical representation of the psychological instruments. The second ring shows psychological domains, the third ring shows individual instruments and the outer ring shows scored dimensions. Each instrument is scored and interpreted within its own scale structure.
Figure 2: Model-specific psychometric response profiles across languages . Radar plots show the mean profiles of the nine evaluated LLMs across 20 non-redundant psychometric dimensions: a , Claude-Haiku-4.5; b , Claude-Sonnet-4.5; c , GPT-5.4; d , GPT-5.4-mini; e , Gemini-2.5-Flash; f , Gemini-2.5-Flash-Lite; g , DeepSeek-V3.2; h , Doubao-Seed-1.8; and i , Kimi-K2. Scores are calculated according to the original scoring rules of each instrument and retain the original direction of each dimension. Dimension abbreviations are defined in Supplementary Table S1.
Figure 3: Scale-level psychometric profiles and model differences . a-h , Scale-level profiles across models and languages for IPIP BFFM-50 (a,b), MFQ-30 (c,d), OEJTS-32 (e,f) and OUS-9, PMD-8, SDO7-16 and SSIS-10 (g,h). i,j , Shared and model-specific response structure, showing the model-equal-weighted profile relative to scale midpoints (i) and centred model profiles (j). k-n , Dimension-level model differences assessed using partial (k), ANOVA with FDR-adjusted values (l), Kruskal-Wallis tests (m) and pairwise model contrasts (n). Dashed lines mark the FDR-adjusted threshold; asterisks indicate FDR-corrected significance.
Figure 4: Whole-profile similarity and NA-response boundaries . a-d , Whole-profile similarity analyses include within-family cosine distances (a), within-group distances (b), between-group distances (c) and pairwise model distances (d). e , Correlation network among the 20 non-redundant psychometric dimensions. f-k , NA rates overall and by language (f), model (g), model and language (h), provider-origin group (i), instrument (j) and dimension (k). Larger distances indicate greater profile divergence.
Figure 5: Language and origin effects in LLM psychometric assessment . a-c , Language effects across models and dimensions include Chinese-minus-English score differences (a), cross-language profile distances (b) and dimension-level effect directions (c). d-f , Provider-origin associations in dimension scores (d), NA rates (e) and reduced psychometric profile space (f). g-i , Provider-origin-by-language patterns in NA rates (g), cross-language profile shifts (h) and language-effect directions (i). j,k , Item-level language effects on scored responses and NA responses. l , Model-specific shifts from English to Chinese administration in the reduced profile space.
Figure 6: Reliability, identifiability and robustness of LLM psychometric profiles . a,b , Item-level ICCs (a) and test–retest correlations (b) across repeated administrations. c-f , Profile stability assessed using within- and cross-model distances (c), same- and cross-language distances (d), model-level stability rankings (e) and dimension-level ICCs (f). g-i , Combined-response clustering (g) and model-identification performance under leave-one-run-out (h) and cross-language evaluation (i). j,k , Human-reference comparisons across the five IPIP BFFM-50 dimensions, showing response distributions (j) and standardized model–human differences (k). l-o , Prompt sensitivity of scored responses across models and languages (l), NA responses across models and languages (m), scored responses across instruments (n) and NA responses across instruments (o). Scored-response sensitivity is measured as the mean absolute normalized difference between Prompt A and Prompt B, whereas NA-response sensitivity is measured as the mean absolute difference in NA rates.
| Abbreviation | Instrument | Dimension |
|---|---|---|
| AGR | IPIP BFFM-50 | Agreeableness |
| CON | IPIP BFFM-50 | Conscientiousness |
| ES | IPIP BFFM-50 | Emotional Stability |
| BEX | IPIP BFFM-50 | Extraversion |
| OPN | IPIP BFFM-50 | Intellect or Imagination |
| AUT | MFQ-30 | Authority |
Supplementary Table 1: Dimension abbreviations used in the psychometric profile.
Supplementary Figure 1: Ranked high- and low-scoring dimensions for each LLM . a - i , Highest- and lowest-scoring dimensions for each LLM across the 20-dimensional psychometric profile. Scores are normalized within each dimension to support profile-level comparison across instruments. The ranked profiles provide model-level details underlying the whole-profile comparisons in Figure 3 .
Supplementary Figure 2: Complete item-level language effects across the psychological instruments . a , Model-level NA response differences between Chinese and English administrations across all scale items. b , Model-level item response differences between Chinese and English administrations. Items are ordered by psychological instrument and item index. Positive and negative values indicate the direction of language-dependent change. These full item-level results complement the top differential item-level language-effect results shown in Figures 5 j- 5 k.
Supplementary Figure 3: Model-specific item response differences between Chinese and English administrations . a-i , Item-level response differences for each evaluated LLM across the full psychological instruments. Panels correspond to a, Claude-Haiku-4.5; b, Claude-Sonnet-4.5; c, GPT-5.4; d, GPT-5.4-mini; e, Gemini-2.5-Flash; f, Gemini-2.5-Flash-Lite; g, DeepSeek-V3.2; h, Doubao-Seed-1.8 and i, Kimi-K2. Positive and negative values indicate the direction and magnitude of cross-language response shifts for individual items.
Supplementary Figure 4: Model-specific NA-response differences between Chinese and English administrations . a-i , Item-level NA-response differences for each evaluated LLM across the full psychological instruments. Panels correspond to a, Claude-Haiku-4.5; b, Claude-Sonnet-4.5; c, GPT-5.4; d, GPT-5.4-mini; e, Gemini-2.5-Flash; f, Gemini-2.5-Flash-Lite; g, DeepSeek-V3.2; h, Doubao-Seed-1.8 and i, Kimi-K2. Positive and negative values indicate the direction and magnitude of cross-language response shifts for individual items.
Supplementary Figure 5: Item-level test-retest consistency across evaluated LLMs . a-i , Scatter plots comparing item-level scores between test and retest administrations for each evaluated LLM. Each point represents an item-level response from repeated administrations, with the dashed diagonal indicating perfect test-retest agreement. The solid line shows the fitted linear trend. Insets report the mean Pearson test-retest correlation and ICC(3,1) for each model. Panels correspond to a, Claude-Haiku-4.5; b, Claude-Sonnet-4.5; c, GPT-5.4; d, GPT-5.4-mini; e, Gemini-2.5-Flash; f, Gemini-2.5-Flash-Lite; g, DeepSeek-V3.2; h, Doubao-Seed-1.8 and i, Kimi-K2. High correlations and ICC values indicate that item-level psychometric responses are reproducible across repeated administrations, whereas larger scatter reflects lower response stability.
Supplementary Figure 6: Consistency and structural patterns in LLM psychometric responses . a , Intraclass correlation coefficients across repeated evaluations for each psychometric scale dimension. b , Heatmap of ICC(3,1) across repeats stratified by both model and scale dimension. c , Hierarchical clustering of item responses across models. d , Hierarchical clustering of NA responses across models.
Supplementary Figure 7: Model identification from LLM psychometric responses . a-d , Confusion matrices for model identification using four standard classifiers: a , support vector machine (SVM); b , k-nearest neighbours (KNN); c , random forest (RF); and d , logistic regression (LR). Within each row, the three columns show classification results based on item-level responses, NA responses and combined responses, respectively. Each matrix compares the true model label with the predicted model label.
Supplementary Figure 8: Cross-language (Chinese English) model identification from LLM psychometric responses . a-d , Confusion matrices for cross-language model identification using four standard classifiers: a , support vector machine (SVM); b , k-nearest neighbours (KNN); c , random forest (RF); and d , logistic regression (LR). Models are trained on Chinese administrations and tested on English administrations. Within each row, the three columns show classification results based on item-level responses, NA responses and combined responses, respectively. Each matrix compares the true model label with the predicted model label.
Supplementary Figure 9: Cross-language (English Chinese)model identification from LLM psychometric responses . a-d , Confusion matrices for cross-language model identification using four standard classifiers: a , support vector machine (SVM); b , k-nearest neighbours (KNN); c , random forest (RF); and d , logistic regression (LR). Models are trained on English administrations and tested on Chinese administrations. Within each row, the three columns show classification results based on item-level responses, NA responses and combined responses, respectively. Each matrix compares the true model label with the predicted model label.
Supplementary Figure 10: Prompt templates used for LLM psychometric assessment . The figure shows the standardized instruction templates used for Prompt A and Prompt B under English and Chinese administration conditions. Each template specifies the testing task, response options, AI self-mapping rule, response stance, administration protocol, stable-tendency instruction, safety constraints, information-source restrictions and required output format.
Supplementary Figure 11: Language- and model-resolved comparisons of LLM and human personality-response distributions . a , Human and LLM response distributions across the five IPIP BFFM-50 dimensions with LLM responses stratified by English and Chinese administration. b - f , Model-level response distributions for Extraversion (b), Agreeableness (c), Conscientiousness (d), Emotional Stability (e) and Intellect/Imagination (f) across repeated administrations. In b-f, the shaded background denotes the human 95 response range, the dashed horizontal line denotes the human mean and diamond markers denote the mean response of each LLM. The panels provide language- and model-level detail for the human-reference comparison summarized in Figures 6 a- 6 b.
More in this topic
Personality Modeling in Language Models
Personality modeling in language models studies how traits are measured, represented, simulated, and controlled in model behavior.
See all 169 papers