On the Behavioral Traits of LLM Agents
Organizations: University of New South Wales · Johns Hopkins University · Carnegie Mellon University · Fudan University ·
Abstract
Users increasingly describe different AI agents as distinct colleagues to work with. AI personality research aims to quantify such impressions by attributing human-like "traits" to agents. However, existing measures fall short: models' self-reports (S-data) diverge from their actual behavior, while informant ratings from LLM judges (I-data) are costly to scale and cover few everyday scenarios. In this paper, we propose A-B-D to infer traits bottom-up from behavioral data (B-data), namely how agents act on their environment and communicate with users, as recorded in existing trajectories. From 345,667 real-world trajectories spanning 80 models, 12 tasks, and 50 harnesses, we extract 318 candidate features that capture both the actions an agent takes at each step (functional) and the language accompanying them (linguistic). We retain only features that show instance-level stability, cross-task consistency, and model discriminability. Factor analysis of the remaining 79 features uncovers six stable, model-attributable factors, two functional and four linguistic. For example, Kimi-K3 exhibits the most planfulness, whereas GPT-5.5 and GPT-5.6 are the least energetic. Moreover, we quantify the "knowledge-action gap" in the wild: these factors correlate only weakly with self-reported Big Five scores, even for conceptually matched pairs such as extroversion and energetic (r = 0.07, p = 0.58). Our work offers a new lens for understanding AI personality, with implications for users, developers, and researchers from both computer science and social science.
Figures & tables
| Derived Factors from a-b-d | ||||||
|---|---|---|---|---|---|---|
| Planfulness | Deliberation | Talkative | Responsible | Energetic | Absolutist | |
| Task success | ||||||
| Release date | ||||||
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Source | Tasks | #Models | #Harn. | #Raw | #Subset |
|---|---|---|---|---|---|
| GAIA ( Mialon et al., 2024 ) | research-qa | 3 | 1 | 65,915 | 90 |
| HAL ( Kapoor et al., 2026 ) | science, swe, dialogue, research-qa, web | 22 | 13 | 58,095 | 4,876 |
| APIFlow-Bench ( Wan et al., 2026 ) | tool-api | 19 | 1 | 56,037 | 570 |
| Terminal-Bench 2.0 (Community) ( Merrill et al., 2026 ) | terminal, science, swe, security, ml-eng, other | 31 | 9 | 52,272 | 11,254 |
| Multi-SWE-bench ( Zan et al., 2025 ) | swe | 7 | 2 | 27,514 | 420 |
| MCPMark ( Wu et al., 2026 ) | tool-api | 17 | 1 | 11,314 | 510 |
| Stage | Dropped | Remaining |
| Raw collected (21 sources) | 345,667 | |
| model not in target candidate set | 63,485 | |
| harness unidentified | 200 | |
| task unidentified | 1,969 | |
| result record only (no trajectory) | 18,294 | |
| multi-agent (flattened threads) | 2,162 |
| Metric | Spans per Step | Micro-F1 | Jaccard | |||
|---|---|---|---|---|---|---|
| Human | GPT-OSS-120B | Group | Sub-Action | Group | Sub-Action | |
| Value | 2.34 | 2.50 | 0.45 | 0.30 | 0.36 | 0.27 |
| Group | Human | GPT-OSS-120B | |
|---|---|---|---|
| Grounding | 16 | 23 | 0.11 |
| Retrieval | 11 | 5 | 0.43 |
| Reasoning | 8 | 18 | 0.14 |
| Planning | 11 | 22 | 0.22 |
| Evaluate | 7 | 9 | -0.16 |
| Deciding | 4 | 3 | 0.24 |
| Family | Feature | Description | # |
| Frequency | Share of behavioral spans labeled group . | 10 | |
| Behavioral spans per agent step. | 1 | ||
| Normalized entropy of the group distribution (behavioral variety). | 1 | ||
| Co-occurrence | Within-step co-occurrence lift of groups and : how much more than chance they occur in the same step. | 45 | |
| Mean number of distinct groups in a step. | 1 | ||
| Transition | Probability that a step containing group is followed by a step containing group (conditional presence; a row need not sum to one). | 100 |
| Rule | Pattern | Note |
| (a) Message level | ||
| JSON prose | message parses as a JSON dict | keep only fields analysis , plan , thought , reasoning , reflection ; drop the rest (e.g. command payloads) |
| (b) Span strippers (matched span space) | ||
| Fenced code | .*? s | fenced code blocks |
| Dangling fence | | an unclosed trailing code fence |
| Think tags | </?(mm:)?think> i | reasoning-wrapper tags (e.g. MiniMax mm:think ) |
| Family | Feature | ICC | ||
|---|---|---|---|---|
| Transition | 0.98 | 0.64 | 0.36 | |
| 0.95 | 0.65 | 0.24 | ||
| 0.93 | 0.50 | 0.20 | ||
| 0.94 | 0.55 | 0.20 | ||
| 0.94 | 0.55 | 0.20 | ||
| 0.95 | 0.52 | 0.20 |
| Domain | LIWC Cat. | ICC | ||
|---|---|---|---|---|
| Function words | we | 0.99 | 0.71 | 0.49 |
| function words | 0.98 | 0.46 | 0.37 | |
| articles | 0.98 | 0.46 | 0.34 | |
| linguistic dim. | 0.97 | 0.42 | 0.34 | |
| personal pron. | 0.97 | 0.59 | 0.32 | |
| total pron. | 0.97 | 0.51 | 0.32 |
| Trait | F1 | F2 | |
| 0.35 | 0.94 | ||
| 0.27 | 0.85 | ||
| 0.94 | |||
| 0.30 | 0.81 | ||
| 0.35 | 0.83 | ||
| 0.29 | 0.74 |
| Trait | F1 | F2 | F3 | F4 | |
|---|---|---|---|---|---|
| function words | 0.08 | 0.19 | 0.01 | 0.94 | |
| linguistic dim. | 0.30 | 0.96 | |||
| impersonal pron. | 0.12 | 0.88 | |||
| present focus | 0.03 | 0.12 | 0.81 | ||
| determiners | 0.12 | 0.15 | 0.81 | ||
| auxiliary verbs | 0.14 | 0.13 | 0.81 |
| Functional | Linguistic | |||||
| F1 | F2 | F1 | F2 | F3 | F4 | |
| Positive | function words | clout | curiosity | negations | ||
| linguistic dim. | social ref. | verbs | all-or-none | |||
| impersonal pron. | social proc. | personal pron. | allure | |||
| present focus | polite | attention | ||||
| determiners | affiliation | I | ||||
| Jac. | Functional | Linguistic | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Held-out task | F | L | F1 | F2 | F1 | F2 | F3 | F4 | ||
| swe | 2 | 6 | .87 | .92 | .99 | .98 | .89 | .93 | .92 | .61 |
| terminal | 2 | 4 | .81 | .86 | 1.00 | 1.00 | .99 | .99 | .98 | .99 |
| tool-api | 2 | 5 | .76 | .91 | .97 | .97 | .95 | .93 | .86 | .88 |
| web | 2 | 4 | .94 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| gui | 2 | 5 | 1.00 | 1.00 | 1.00 | 1.00 | .96 | .97 | .91 | .94 |
| Model | Fun_F1 | Fun_F2 | Lin_F1 | Lin_F2 | Lin_F3 | Lin_F4 |
| alibaba/qwen2.5-72b-instruct | 0.05 | 1.46 | 2.02 | 0.87 | 0.46 | 0.09 |
| alibaba/qwen3-coder-480b-a35b-instruct | 0.95 | -0.82 | -0.41 | -0.18 | 1.28 | 0.29 |
| alibaba/qwen3.5-27b | 0.68 | -0.41 | 0.41 | -1.18 | 0.23 | -0.75 |
| alibaba/qwen3.5-35b-a3b | 1.57 | -0.42 | 0.31 | -0.94 | -0.06 | -1.21 |
| alibaba/qwen3.5-397b-a17b | 1.10 | -0.49 | 0.38 | -1.00 | -0.13 | -0.74 |
| alibaba/qwen3.6-27b | 0.53 | -0.32 | 0.68 | -1.54 | -0.14 | -0.28 |
| Table Continued | ||||||
|---|---|---|---|---|---|---|
| Model | Fun_F1 | Fun_F2 | Lin_F1 | Lin_F2 | Lin_F3 | Lin_F4 |
| moonshot/kimi-k2-0905-preview | 0.98 | -1.08 | -0.55 | -0.36 | 1.32 | -0.06 |
| moonshot/kimi-k2-thinking | 0.95 | -0.79 | -0.80 | -0.46 | 0.95 | 0.13 |
| moonshot/kimi-k2.5 | 0.94 | 0.46 | -0.27 | -0.62 | -0.37 | 1.63 |
| moonshot/kimi-k2.6 | 0.32 | 0.50 | 0.44 | -0.19 | -1.58 | -0.73 |
| moonshot/kimi-k2.7-code | 0.46 | 3.22 | 0.77 | -0.45 | -1.43 | 0.88 |
Personality Modeling in Language Models
Personality modeling in language models studies how traits are measured, represented, simulated, and controlled in model behavior.
See all 162 papersFrom Representations to Behaviors: Exploring the Person-Situation-Behavior Triad in LLMs
Same Voice, Different Lab: On the Homogenization of Frontier LLM Personalities
middle-of-distribution traits such as poetic or playful, but even these so-called creative models tend to have more neutral identities. These similarities suggest an implicit emergence of a standard of optimal assistant behavior. In a landscape of varied training methods, character training, therefore, stands out for its uniformity, offering insight into a tacit consensus between model developers.