On the Behavioral Traits of LLM Agents

Haokai Zhao; Jie Gao; Yunze Xiao; Xintao Wang; Weihao Xuan; Aditya Joshi; Mark Dredze; Jen-tse Huang

cs.CLSep 26, 2026

On the Behavioral Traits of LLM Agents

Authors: Haokai Zhao, Jie Gao, Yunze Xiao, Xintao Wang, Weihao Xuan, Aditya Joshi, Mark Dredze, Jen-tse Huang

Organizations: University of New South Wales · Johns Hopkins University · Carnegie Mellon University · Fudan University ·

Abstract

Users increasingly describe different AI agents as distinct colleagues to work with. AI personality research aims to quantify such impressions by attributing human-like "traits" to agents. However, existing measures fall short: models' self-reports (S-data) diverge from their actual behavior, while informant ratings from LLM judges (I-data) are costly to scale and cover few everyday scenarios. In this paper, we propose A-B-D to infer traits bottom-up from behavioral data (B-data), namely how agents act on their environment and communicate with users, as recorded in existing trajectories. From 345,667 real-world trajectories spanning 80 models, 12 tasks, and 50 harnesses, we extract 318 candidate features that capture both the actions an agent takes at each step (functional) and the language accompanying them (linguistic). We retain only features that show instance-level stability, cross-task consistency, and model discriminability. Factor analysis of the remaining 79 features uncovers six stable, model-attributable factors, two functional and four linguistic. For example, Kimi-K3 exhibits the most planfulness, whereas GPT-5.5 and GPT-5.6 are the least energetic. Moreover, we quantify the "knowledge-action gap" in the wild: these factors correlate only weakly with self-reported Big Five scores, even for conceptually matched pairs such as extroversion and energetic (r = 0.07, p = 0.58). Our work offers a new lens for understanding AI personality, with implications for users, developers, and researchers from both computer science and social science.

Figures & tables

Figure 1: Cumulative distribution of each statistic over 211 functional and 107 linguistic features. Vertical lines mark the thresholds; the height of a curve is the fraction of failures in that layer.

Figure 2: The six behavioral factors and their main features. Each fan is one factor. Leaves are the features with the largest loadings: solid with filled dots for positive, dashed with hollow dots for negative.

Panel of Figure 3 — Figure 3: Three models on the same task and harness (terminal-bench-2 sqlite-with-gcov , terminus-2). Top: factor scores. Bottom: verbatim excerpts from each trajectory at the same moment, when configure fails for want of a compiler; [ $n$ ] is the step index, $ a command issued in that step, […] an elision.

Figure 4: Pearson correlation between factor scores and self-reported Big Five domains.

	Derived Factors from a-b-d
	Planfulness	Deliberation	Talkative	Responsible	Energetic	Absolutist
Task success	$\phantom{-}0.38^{***}$	$\phantom{-}0.06$	$-0.00$	$-0.31^{**}$	$-0.20$	$-0.11$
Release date	$\phantom{-}0.42^{***}$	$-0.24^{*}$	$-0.36^{**}$	$-0.68^{***}$	$-0.53^{***}$	$-0.06$

Table 1: Pearson correlation between factor scores and raw task success (mean normalized outcome over a model’s trajectories) and model release date.

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Tasks	#Models	#Harn.	#Raw	#Subset
GAIA ( Mialon et al., 2024 )	research-qa	3	1	65,915	90
HAL ( Kapoor et al., 2026 )	science, swe, dialogue, research-qa, web	22	13	58,095	4,876
APIFlow-Bench ( Wan et al., 2026 )	tool-api	19	1	56,037	570
Terminal-Bench 2.0 (Community) ( Merrill et al., 2026 )	terminal, science, swe, security, ml-eng, other	31	9	52,272	11,254
Multi-SWE-bench ( Zan et al., 2025 )	swe	7	2	27,514	420
MCPMark ( Wu et al., 2026 )	tool-api	17	1	11,314	510

Appendix

Table 2: Raw trajectory sources. Raw = trajectories collected; Subset = trajectories retained after quality control and stratified sampling. #Models and #Harn. are counts after the pipeline. For Long-Horizon-Terminal-Bench, only the 503 tasks with a complete trajectory are counted as raw; 698 runs whose log keeps only the last continuation phase were excluded. The sources vary widely in sizes and tasks, and no single source dominates the subset.

Stage	Dropped	Remaining
Raw collected (21 sources)		345,667
model not in target candidate set	63,485
harness unidentified	200
task unidentified	1,969
result record only (no trajectory)	18,294
multi-agent (flattened threads)	2,162

Appendix

Table 3: Filtering from all collected trajectories to the final subset. Each discarded trajectory is logged with its exclusion reason for transparency and reproducibility. Eligibility predicates are applied in the listed order, and each trajectory is attributed to the first predicate it fails.

Metric	Spans per Step		Micro-F1		Jaccard
Metric	Human	GPT-OSS-120B	Group	Sub-Action	Group	Sub-Action
Value	2.34	2.50	0.45	0.30	0.36	0.27

Appendix

Table 4: Step-level agreement between a human annotator and the GPT-OSS-120B annotator on 8 trajectories (56 steps, 131 human and 140 model spans). Each step is reduced to the set of labels its spans carry; Micro-F1 treats the human labels as the reference, Jaccard is the mean per-step overlap of the two label sets.

Group	Human	GPT-OSS-120B	$\kappa$
Grounding	16	23	0.11
Retrieval	11	5	0.43
Reasoning	8	18	0.14
Planning	11	22	0.22
Evaluate	7	9	-0.16
Deciding	4	3	0.24

Appendix

Table 5: Per-group Cohen’s

\kappa

between the human and the GPT-OSS-120B annotator on the same 56 steps. For each group the two annotations are reduced to whether the step carries at least one span of that group;

\kappa

is the chance-corrected agreement of these binary decisions. Count: steps carrying the group. Learning never occurs on either side (

\kappa

undefined). Pooled: one

\kappa

over all 560 step

\times

group decisions.

Family	Feature	Description	#
Frequency	$\mathrm{prop}_{g}$	Share of behavioral spans labeled group $g$ .	10
	$\mathrm{density}$	Behavioral spans per agent step.	1
	$\mathrm{entropy}$	Normalized entropy of the group distribution (behavioral variety).	1
Co-occurrence	$\mathrm{lift}_{g,h}$	Within-step co-occurrence lift of groups $g$ and $h$ : how much more than chance they occur in the same step.	45
Co-occurrence	$\mathrm{grps\_per\_step}$	Mean number of distinct groups in a step.	1
Transition	$\mathrm{trans}_{g\to h}$	Probability that a step containing group $g$ is followed by a step containing group $h$ (conditional presence; a row need not sum to one).	100

Appendix

Table 6: The functional behavior feature bank. Co-occurrence features are computed within a step, transition and stage features over consecutive steps.

Rule	Pattern	Note
(a) Message level
JSON prose	message parses as a JSON dict	keep only fields analysis , plan , thought , reasoning , reflection ; drop the rest (e.g. command payloads)
(b) Span strippers (matched span $\rightarrow$ space)
Fenced code	`.*?` s	fenced code blocks
Dangling fence		an unclosed trailing code fence
Think tags	</?(mm:)?think> i	reasoning-wrapper tags (e.g. MiniMax mm:think )

Appendix

Table 7: Cleaning rules applied to each trajectory’s raw text, in the order they run. Each removal is a single named regular expression: spans matched in the span-stripper block are replaced by a space, and the line-filter block drops whole lines. Regex flags: s DOTALL, m multiline, i case-insensitive.

Family	Feature	$\rho_{\text{stab}}$	$\rho_{\text{con}}$	ICC
Transition	$\mathrm{trans\_entropy}$	0.98	0.64	0.36
	$\mathrm{trans}_{\mathrm{Plan}\to\mathrm{Plan}}$	0.95	0.65	0.24
	$\mathrm{trans}_{\mathrm{Reas}\to\mathrm{Plan}}$	0.93	0.50	0.20
	$\mathrm{trans}_{\mathrm{Reas}\to\mathrm{Reas}}$	0.94	0.55	0.20
	$\mathrm{trans}_{\mathrm{Exec}\to\mathrm{Reas}}$	0.94	0.55	0.20
	$\mathrm{trans}_{\mathrm{Plan}\to\mathrm{Reas}}$	0.95	0.52	0.20

Appendix

Table 8: The 31 functional features that pass all three tests, grouped by the feature families of Table 6 . Act· onomy groups are abbreviated in subscripts: Plan = Planning, Reas = Reasoning, Exec = Executing, Eval = Evaluate, Refl = Reflection, Grnd = Grounding, Retr = Retrieval, Deci = Deciding.

\rho_{\text{stab}}

: split-half reliability within a task;

\rho_{\text{con}}

: cross-task consistency; ICC: between-model discriminability.

Domain	LIWC Cat.	$\rho_{\text{stab}}$	$\rho_{\text{con}}$	ICC
Function words	we	0.99	0.71	0.49
	function words	0.98	0.46	0.37
	articles	0.98	0.46	0.34
	linguistic dim.	0.97	0.42	0.34
	personal pron.	0.97	0.59	0.32
	total pron.	0.97	0.51	0.32

Appendix

Table 9: The 48 linguistic features that pass all three tests. Features are grouped by LIWC-22’s own output-header domains ( Function words abbreviates LIWC’s Linguistic Dimensions header, Summary its summary variables, and Motives is our label for the header-less need /…/ allure block). Categories are scored word-level on the full cleaned corpus.

\rho_{\text{stab}}

: split-half reliability within a task;

\rho_{\text{con}}

: cross-task consistency; ICC: between-model discriminability.

Trait	F1	F2	$h^{2}$
$\mathrm{trans}_{\mathrm{Eval}\to\mathrm{Plan}}$	$\mathbf{0.90}$	0.35	0.94
$\mathrm{trans}_{\mathrm{Refl}\to\mathrm{Plan}}$	$\mathbf{0.88}$	0.27	0.85
$\mathrm{trans}_{\mathrm{Plan}\to\mathrm{Plan}}$	$\mathbf{0.85}$	$\mathbf{0.46}$	0.94
$\mathrm{trans\_entropy}$	$\mathbf{0.85}$	0.30	0.81
$\mathrm{entropy}$	$\mathbf{0.84}$	0.35	0.83
$\mathrm{trans}_{\mathrm{Deci}\to\mathrm{Plan}}$	$\mathbf{0.81}$	0.29	0.74

Appendix

Table 10: Functional family: varimax-rotated loadings of the 31 functional features on the two factors retained by parallel analysis (KMO

=0.89

). Loadings with

|\lambda|\geq 0.40

are marked in bold;

h^{2}

is the communality. Traits are ordered by their primary factor and loading.

Trait	F1	F2	F3	F4	$h^{2}$
function words	$\mathbf{0.95}$	0.08	0.19	0.01	0.94
linguistic dim.	$\mathbf{0.93}$	$-0.00$	0.30	$-0.04$	0.96
impersonal pron.	$\mathbf{0.91}$	$-0.10$	0.12	$-0.14$	0.88
present focus	$\mathbf{0.89}$	0.03	0.12	$-0.06$	0.81
determiners	$\mathbf{0.88}$	0.12	0.15	$-0.00$	0.81
auxiliary verbs	$\mathbf{0.88}$	0.14	0.13	$-0.01$	0.81

Appendix

Table 11: Linguistic family: varimax-rotated loadings of the 48 linguistic features on the four factors retained by parallel analysis (KMO

=0.70

). Loadings with

|\lambda|\geq 0.40

are marked in bold;

h^{2}

is the communality. Traits are ordered by their primary factor and loading.

	Functional		Linguistic
	F1	F2	F1	F2	F3	F4
Positive	$\mathrm{trans}_{\mathrm{Eval}\to\mathrm{Plan}}$	$\mathrm{prop}_{\mathrm{Reas}}$	function words	clout	curiosity	negations
	$\mathrm{trans}_{\mathrm{Refl}\to\mathrm{Plan}}$	$\mathrm{late}_{\mathrm{Reas}}$	linguistic dim.	social ref.	verbs	all-or-none
	$\mathrm{trans}_{\mathrm{Plan}\to\mathrm{Plan}}$	$\mathrm{mid}_{\mathrm{Reas}}$	impersonal pron.	social proc.	personal pron.	allure
	$\mathrm{trans\_entropy}$	$\mathrm{trans}_{\mathrm{Exec}\to\mathrm{Reas}}$	present focus	polite	attention
	$\mathrm{entropy}$	$\mathrm{early}_{\mathrm{Reas}}$	determiners	affiliation	I

Appendix

Table 12: Features load on each factor (

|\lambda|\geq 0.40

in the varimax solution), split by the sign of the loading and ordered by its size. The two families are analyzed separately, so their factors are numbered independently; a feature may load on two factors of its family.

Held-out task	$k_{F}$	$k_{L}$	F	L	F1	F2	F1	F2	F3	F4
			Jac.		Functional $\phi$		Linguistic $\phi$
swe	2	6	.87	.92	.99	.98	.89	.93	.92	.61
terminal	2	4	.81	.86	1.00	1.00	.99	.99	.98	.99
tool-api	2	5	.76	.91	.97	.97	.95	.93	.86	.88
web	2	4	.94	1.00	1.00	1.00	1.00	1.00	1.00	1.00
gui	2	5	1.00	1.00	1.00	1.00	.96	.97	.91	.94

Appendix

Table 13: Leave-one-out stability of the pipeline. For every fold the held-out task type is removed and the whole pipeline is rerun on the remaining eleven; the result is compared with the reference run.

k_{F}

k_{L}

: retained factors (reference run: 2 and 4). Jac.: Jaccard overlap of the fold’s trait set with the reference run’s (31 functional, 48 linguistic). Per factor: Tucker congruence

\phi

of the varimax loadings on the traits both runs kept, factors matched by Hungarian assignment; – marks a reference-run factor without a match. Last row: mean over folds (Linguistic F4 over the 11 folds in which it is matched).

Model	Fun_F1	Fun_F2	Lin_F1	Lin_F2	Lin_F3	Lin_F4
alibaba/qwen2.5-72b-instruct	0.05	1.46	2.02	0.87	0.46	0.09
alibaba/qwen3-coder-480b-a35b-instruct	0.95	-0.82	-0.41	-0.18	1.28	0.29
alibaba/qwen3.5-27b	0.68	-0.41	0.41	-1.18	0.23	-0.75
alibaba/qwen3.5-35b-a3b	1.57	-0.42	0.31	-0.94	-0.06	-1.21
alibaba/qwen3.5-397b-a17b	1.10	-0.49	0.38	-1.00	-0.13	-0.74
alibaba/qwen3.6-27b	0.53	-0.32	0.68	-1.54	-0.14	-0.28

Appendix

Table 14: Factor scores of all models (standardized across models to zero mean and unit variance), grouped by model family. Cells are shaded by the score, orange for positive and blue for negative, with full saturation at

|z|=3

Table Continued
Model	Fun_F1	Fun_F2	Lin_F1	Lin_F2	Lin_F3	Lin_F4
moonshot/kimi-k2-0905-preview	0.98	-1.08	-0.55	-0.36	1.32	-0.06
moonshot/kimi-k2-thinking	0.95	-0.79	-0.80	-0.46	0.95	0.13
moonshot/kimi-k2.5	0.94	0.46	-0.27	-0.62	-0.37	1.63
moonshot/kimi-k2.6	0.32	0.50	0.44	-0.19	-1.58	-0.73
moonshot/kimi-k2.7-code	0.46	3.22	0.77	-0.45	-1.43	0.88

Appendix

Table 19

Personality Modeling in Language Models

Personality modeling in language models studies how traits are measured, represented, simulated, and controlled in model behavior.

See all 162 papers

AgentHabit: Characterizing Distinct Behaviors of Agents on Everyday Tasks
Sep 26, 2026Woojung Song, Hoyeol Yang, Jeonghoon Shim +4LLM Agents LLM Agent Evaluation
Seoul National University, Seoul, Republic of Korea
From Representations to Behaviors: Exploring the Person-Situation-Behavior Triad in LLMs
Jul 29, 2026Ruikang Zhang, Shuo Wang, Qi SuLLM Interpretability Personality Modeling in Language Models
1Peking University, Beijing, China · 2Beijing Institute of Technology, Beijing, China
Same Voice, Different Lab: On the Homogenization of Frontier LLM Personalities
Mar 20, 2026Avinash Krishna, Kalyana Chadalavada, Unso Eun Seo JoPersonality Modeling in Language Models Behavioral Consistency of LLM Agents
Independent · Anthropic · Cornell University

Sep 26, 2026J/K move · Enter open · S save

AgentHabit: Characterizing Distinct Behaviors of Agents on Everyday Tasks

Woojung Song, Hoyeol Yang, Jeonghoon Shim, Sungjib Lim+3

Seoul National University, Seoul, Republic of Korea

LLM Agents LLM Agent Evaluation AI Agent Benchmarks Behavioral Consistency of LLM Agents

Open paper AskPDF arXiv

Large language model (LLM) agents assist users with everyday tasks that can be completed in many reasonable ways. Even when their answers are useful, how agents carry out these tasks may not match users' preferences and needs. For example, agents differ in whether they ask clarifying questions or search the web. We introduce HABIT, a taxonomy of 23 behavioral axes in five categories, which three authors and three LLMs derive bottom-up from 408 agent trajectories across 17 domains. On held-out tasks, HABIT distinguishes models more clearly than existing taxonomies of human values and agent actions while supporting comparably consistent annotation. Building on HABIT, we construct AgentHABIT, a benchmark that profiles each agent's behavioral tendencies from its trajectories on 86 everyday tasks. Profiling 18 models with AgentHABIT reveals a range of distinctive tendencies. For example, most GPT and Claude models state their assumptions and offer alternatives when requirements conflict, whereas Qwen and Google's models more often leave assumptions or changes to requirements unstated. These profiles remain recognizable even when built from entirely different sets of tasks, indicating that they reflect general tendencies rather than task-specific behavior. Prompting agents to adopt specific behaviors shifts some axes readily but barely changes others, while fine-tuning on another model's trajectories changes only part of a model's profile and leaves much of it intact. Overall, HABIT and AgentHABIT provide a systematic framework for characterizing how agents carry out everyday tasks beyond task success, offering insights to guide the development of agents whose behavior better fits users' needs.

Abstract

Personality Modeling in Language Models

Explore similar work

AgentHabit: Characterizing Distinct Behaviors of Agents on Everyday Tasks

From Representations to Behaviors: Exploring the Person-Situation-Behavior Triad in LLMs

Same Voice, Different Lab: On the Homogenization of Frontier LLM Personalities