AgentHabit: Characterizing Distinct Behaviors of Agents on Everyday Tasks

Woojung Song; Hoyeol Yang; Jeonghoon Shim; Sungjib Lim; Jonggeun Lee; Yunho Choi; Yohan Jo

cs.AISep 26, 2026

AgentHabit: Characterizing Distinct Behaviors of Agents on Everyday Tasks

Authors: Woojung Song, Hoyeol Yang, Jeonghoon Shim, Sungjib Lim, Jonggeun Lee, Yunho Choi, Yohan Jo

Organizations: Seoul National University, Seoul, Republic of Korea

Abstract

Large language model (LLM) agents assist users with everyday tasks that can be completed in many reasonable ways. Even when their answers are useful, how agents carry out these tasks may not match users' preferences and needs. For example, agents differ in whether they ask clarifying questions or search the web. We introduce HABIT, a taxonomy of 23 behavioral axes in five categories, which three authors and three LLMs derive bottom-up from 408 agent trajectories across 17 domains. On held-out tasks, HABIT distinguishes models more clearly than existing taxonomies of human values and agent actions while supporting comparably consistent annotation. Building on HABIT, we construct AgentHABIT, a benchmark that profiles each agent's behavioral tendencies from its trajectories on 86 everyday tasks. Profiling 18 models with AgentHABIT reveals a range of distinctive tendencies. For example, most GPT and Claude models state their assumptions and offer alternatives when requirements conflict, whereas Qwen and Google's models more often leave assumptions or changes to requirements unstated. These profiles remain recognizable even when built from entirely different sets of tasks, indicating that they reflect general tendencies rather than task-specific behavior. Prompting agents to adopt specific behaviors shifts some axes readily but barely changes others, while fine-tuning on another model's trajectories changes only part of a model's profile and leaves much of it intact. Overall, HABIT and AgentHABIT provide a systematic framework for characterizing how agents carry out everyday tasks beyond task success, offering insights to guide the development of agents whose behavior better fits users' needs.

Figures & tables

Figure 1: From everyday interactions to behavioral profiles. How two agents respond to the same opening request (left) and their behavioral profiles (right).

Figure 3: Five-stage construction of Habit .

Taxonomy	Matching (%)	Gap	Entropy	Intra-AC1	Inter-AC1
Habit	92.2	0.114	0.653	0.851	0.718
Schwartz human values	72.8	0.047	0.491	0.650	0.383
Act·ONOMY	87.2	0.044	0.387	0.830	0.755

Table 1: Taxonomy comparison on held-out tasks. Bold indicates the highest value in each column.

Figure 4: Behavioral profiles on five representative HABIT axes, renormalized after excluding N/A.

Figure 5: Pairwise similarity of behavioral profiles across models.

	GPT-5.6			GPT-OSS	Claude			Qwen 3.7			Gemini 3.7	Gemma 4
Metric	Sol	Terra	Luna	120B	Opus 5	Sonnet 5	Haiku 4.5	Max	Plus	Flash	Flash	31B	26B	12B	Mean	Random
Mode (%)	65.2	69.6	47.8	47.8	56.5	65.2	34.8	43.5	47.8	56.5	43.5	43.5	39.1	52.2	50.9	33.3

Table 2: Self-report–behavior correspondence. Mode denotes Mode Agreement (%).

	Gemma 4 31B			Gemma 4 12B
Axis	Base	Steered	$\Delta$	Base	Steered	$\Delta$
Decorative Symbol Use	16.5	98.4	+81.9	3.2	96.5	+93.3
Next Step Guidance	0.3	45.0	+44.7	0.5	22.9	+22.4
Web Search Timing	4.3	37.2	+32.9	4.7	28.3	+23.6
Missing Information Handling	1.6	11.6	+10.1	1.6	1.6	0.0
Source Support Visibility	4.1	6.6	+2.5	3.2	8.5	+5.3

Table 3: Prompt-steering target-label rates (%) for six representative shared-target axes.

\Delta

denotes the absolute change.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

#	Domain	Subdomains
claude.ai-derived domains (56 subdomains)
1	Business Marketing Operations	Business Presentations & Process Diagrams Startup & Business Strategy Digital Marketing & Social Media Financial Analysis & Spreadsheets Company Research & Business Intelligence Business Operations & Logistics
2	Academic Education Research	Educational Curriculum & Teaching Materials Social Sciences & Humanities Academic Research Methods & Statistical Analysis Academic Coursework & Research Writing Books & Academic Literature Education & Healthcare Training
3	Health Wellness Relationships	Personal Development & Organizational Leadership Recipes & Nutrition Planning Beauty Fashion & Wellness Products Fitness Training & Weight Management Psychology & Neuroscience Family Life & Social Benefits Medical & Health
4	STEM Sciences Engineering	Math & Physics Problems Earth & Space Sciences Multidisciplinary Engineering Chemistry & Life Sciences AI & Machine Learning Optimization Algorithms & System Performance
5	Creative Media Design	UI and Brand Design Architecture & Visual Arts Fiction & Public Affairs Video & Podcast Production Music & Film/TV Video Games & Tabletop Roleplay

Appendix

Table 4: The 17 task domains and 86 subdomains derived from AEI request categories, grouped by platform.

Domain	Task	Opening request
Construction tasks
Business Marketing Operations	Marketing budget	How should I split our $12,000 monthly ad budget next quarter across Google Search, Instagram, and paid influencers?
Academic Education Research	Science unit plan	Can you help me plan eight 50-minute lessons on energy transfer for a class of 28?
Health Wellness Relationships	Skin products	I need an affordable skincare routine to help with ongoing breakouts along my jawline. I’d like to keep it around $60 a month.
STEM Sciences Engineering	Home solar	I own a house in New Jersey and spend about $2,400 a year on electricity. Are rooftop solar panels worth the money, and should I buy them?
Creative Media Design	Signup flow	Could you recommend a better order for our seven-screen mobile web signup flow based on last month’s funnel? Signup drop-off is bad, and I want an actionable sequencing change.

Appendix

Table 5: Verbatim opening requests for the 17 construction and 17 held-out tasks.

Family	Model route	Access route	Run settings
GPT	gpt-5.6-sol †	OpenAI API	$R=$ medium
	gpt-5.6-terra †	OpenAI API	$R=$ medium
	gpt-5.6-luna	OpenAI API	$R=$ medium
GPT-OSS	openai/gpt-oss-120b	Local vLLM	$T=0.7$
Claude	anthropic/claude-opus-5 †	OpenRouter (Anthropic)	$R=$ medium
	anthropic/claude-sonnet-5 †	OpenRouter (Anthropic)	$R=$ medium

Appendix

Table 6: Model routes and run settings.

\dagger

marks the eight Habit construction models;

T

is temperature and

R

is requested reasoning effort.

Score	Naturalness	Contextual appropriateness	Role and self-consistency
1	Wording is very unnatural or mechanical, making it difficult to regard as a real user’s utterance.	Largely unrelated to the preceding question or deliverable, or clearly inconsistent with the conversation.	Clearly contradicts the user’s previously stated role, goals, or facts.
2	Expressions are noticeably awkward, repetitive, or formulaic.	Responds to only part of the preceding content, or the timing or direction of the response is substantially awkward.	Changes important goals, circumstances, or constraints without explanation.
3	Understandable, but the tone or sentence flow is noticeably awkward.	Generally connected to the context, but misses an important part of the question or deliverable.	Could be the same user, but shows noticeable tension or ambiguity relative to earlier utterances.
4	Sounds like a real user, with only minor awkwardness.	Responds appropriately to the preceding content, with only minor contextual mismatches.	Generally consistent with earlier information, with only minor tensions.
5	Very natural wording for a real user in the given situation.	Responds accurately and naturally to the preceding question or deliverable.	Consistently maintains the user’s role, goals, facts, and constraints.

Appendix

Table 7: the five-point rubric used to evaluate simulator utterances. Each utterance receives one score for each criterion.

Criterion	Mean $\pm$ SD	Gwet’s AC1	95% CI
Naturalness	$4.55\pm 0.87$	0.621	[0.513, 0.734]
Contextual appropriateness	$4.68\pm 0.71$	0.714	[0.616, 0.811]
Role and self-consistency	$4.57\pm 0.83$	0.624	[0.539, 0.712]

Appendix

Table 8: Human evaluation of 74 simulator utterances by three annotators. Means and sample standard deviations use 222 ratings per criterion. Gwet’s AC1 measures exact agreement among all three annotators; its 95% confidence intervals use 5,000 trajectory-level bootstrap resamples.

Taxonomy	Axes	Within	Cross	Gap	95% interval
Habit	23	0.639	0.525	0.114	$[0.080,0.148]$
Schwartz human values	10	0.616	0.569	0.047	$[0.030,0.065]$
Act·ONOMY	36	0.812	0.769	0.044	$[0.029,0.061]$

Appendix

Table 9: Model-separation gap on the held-out trajectories. Within and Cross denote same-model and different-model label agreement within a task, respectively. The Axes column gives the number of axes with defined gaps. Intervals summarize axis-bootstrap variation.

Annotator pair	Agreement	95% CI
H1–H2	0.6675	[0.611, 0.729]
H1–H3	0.5882	[0.537, 0.642]
H2–H3	0.6215	[0.558, 0.680]

Appendix

Table 10: Human pairwise agreement on 391 cells, with trajectory-bootstrap 95% intervals.

Annotator	Qwen	Gemma	Muse	GPT-5.6 Luna (low)
H1	0.7008	0.6726	0.6368	0.6215
H2	0.8235	0.8056	0.7570	0.7954
H3	0.6496	0.6650	0.6036	0.5831

Appendix

Table 11: Agreement between each human and each judge’s modal labels on all 391 cells.

Judge	Agreement	Cohen’s $\kappa$	Pooled AC1	AC1 95% CI	Mean axis AC1
Qwen3.8-27B	0.8352	0.8282	0.8327	[0.785, 0.877]	0.7878
Gemma 4 31B	0.8159	0.8080	0.8132	[0.755, 0.864]	0.7659
Muse-Glimmer-30B	0.7555	0.7450	0.7519	[0.700, 0.801]	0.6951
GPT-5.6 Luna (low)	0.7527	0.7435	0.7489	[0.700, 0.794]	0.6728

Appendix

Table 12: Judge agreement with the human majority on 364 cells.

Comparison	AC1 difference	Paired 95% CI
Qwen $-$ Gemma	0.0196	[ $-0.0164$ , 0.0553]
Qwen $-$ Muse	0.0808	[0.0448, 0.1188]
Qwen $-$ Luna	0.0838	[0.0471, 0.1192]
Gemma $-$ Muse	0.0612	[0.0111, 0.1087]
Gemma $-$ Luna	0.0643	[0.0195, 0.1062]
Muse $-$ Luna	0.0030	[ $-0.0333$ , 0.0428]

Appendix

Table 13: Paired differences in pooled human-majority AC1, with 95% intervals from 2,000 trajectory-bootstrap resamples.

Figure 25

Figure 26

Figure 27

Figure 28

Family	Similarity	Difference	95% interval	$p$	$p_{\mathrm{Holm}}$
GPT-5.6	0.933	0.131	$[0.120,\ 0.134]$	0.0002	0.0008
Qwen	0.908	0.105	$[0.094,\ 0.112]$	0.0002	0.0008
Gemma	0.913	0.111	$[0.091,\ 0.120]$	0.0002	0.0008
Claude	0.769	$-0.033$	$[-0.050,\ -0.010]$	0.0044	0.0044

Appendix

Table 14: Within-family similarity relative to all 91 model pairs. Differences have 95% paired domain-bootstrap intervals;

p_{\mathrm{Holm}}

adjusts four family comparisons.

		Disjoint Remaining Tasks			Full Profile
Subset (%)	Tasks	Model (%)	Family (%)	$D$	Model (%)	Family (%)	$D$
1	1	42.18	63.84	0.394	45.63	65.93	0.389
10	9	80.21	98.90	0.146	90.37	99.66	0.131
20	17	86.34	99.90	0.112	96.91	100.00	0.090
30	26	89.31	99.99	0.097	99.14	100.00	0.068
40	34	89.51	100.00	0.092	99.74	100.00	0.056

Appendix

Table 15: Profile identification accuracy and same-model distance

D

, averaged over 1,000 repetitions per subset size. References use either disjoint remaining tasks or all 86 tasks.

	GPT-5.6			GPT-OSS	Claude			Qwen 3.7			Gemini	Gemma 4
Axis	Sol	Terra	Luna	120B	Opus 5	Sonnet 5	Haiku 4.5	Plus	Max	Flash	3.7 Flash	31B	26B-A4B	12B
Artifact Completeness	0.609	0.549	0.513	0.738	0.664	0.580	0.736	0.611	0.631	0.534	0.689	0.510	0.373	0.509
Ask User Timing	0.375	0.407	0.170	0.101	0.499	0.434	0.269	0.103	0.221	0.132	0.262	0.063	0.085	0.061
Certainty and Uncertainty Framing	0.497	0.465	0.532	0.150	0.736	0.521	0.179	0.262	0.286	0.180	0.092	0.180	0.161	0.142
Constraint Conflict Handling	0.702	0.664	0.668	0.091	0.752	0.579	0.307	0.404	0.439	0.329	0.166	0.289	0.332	0.271
Constraint Preservation Across Outputs	0.289	0.337	0.271	0.487	0.512	0.242	0.734	0.676	0.624	0.689	0.770	0.331	0.539	0.651

Appendix

Table 16: Self-report–behavior Similarity by axis and model. Bold indicates overlapping modal-label sets.

Axis	Most selected label	MCQ selection (%)	Observed frequency (%)
Artifact Completeness	Complete usable	97.9	57.0
Ask User Timing	Both phases	91.4	15.9
Certainty and Uncertainty Framing	Qualified conclusion	100.0	31.3
Constraint Conflict Handling	States and offers alternative	100.0	42.8
Constraint Preservation Across Outputs	Inconsistently preserved	67.9	27.4
Decorative Symbol Use	None	87.9	66.2

Appendix

Table 17: Most frequently self-reported label on each behavioral axis, with its MCQ selection rate and frequency in AgentHabit . Both percentages weight models equally.

Model	Own	Other	$\Delta$	Own	Other	$\Delta$
	Mode Agreement (%)			Similarity
GPT-5.6 Sol	65.2	60.9	$+4.3$	0.558	0.504	$+0.054$
GPT-5.6 Terra	69.6	51.8	$+17.7$	0.546	0.499	$+0.047$
GPT-5.6 Luna	47.8	43.1	$+4.7$	0.477	0.446	$+0.031$
GPT-OSS 120B	47.8	47.8	$0.0$	0.414	0.451	$-0.038$
Claude Opus 5	56.5	31.8	$+24.7$	0.516	0.418	$+0.098$

Appendix

Table 18: Self-report correspondence with the model’s own behavior (Own) and the mean over 13 other models (Other).

\Delta

is Own minus Other.

	Probe–Behavior		Probe–MCQ		Mean–Sum
Model	Mode (%)	Similarity	Mode (%)	Similarity	Mode (%)
GPT-OSS 120B	30.4	0.511	43.5	0.466	13.0
Gemma 4 31B	30.4	0.491	30.4	0.355	39.1
Gemma 4 26B-A4B	21.7	0.457	26.1	0.316	39.1
Gemma 4 12B	30.4	0.508	34.8	0.361	30.4
Mean	28.3	0.492	33.7	0.374	30.4

Appendix

Table 19: Question-only probe correspondence, averaged over 23 axes.

	Probe–Behavior		Probe–MCQ
Axis	Mode (%)	Similarity	Mode (%)	Similarity
Artifact Completeness	25.0	0.621	25.0	0.351
Ask User Timing	100.0	0.668	0.0	0.014
Certainty and Uncertainty Framing	25.0	0.564	25.0	0.272
Constraint Conflict Handling	0.0	0.406	0.0	0.090
Constraint Preservation Across Outputs	0.0	0.373	25.0	0.277

Appendix

Table 20: Question-only probe correspondence by axis, averaged equally over four models. Mode columns report Mode Agreement (%).

Axis	Question suffix and options
Artifact Completeness	whether the assistant supplies the requested concrete artifact with its major required parts? A. Provides the complete structure but leaves explicit blanks for genuinely unknown values. B. Discusses or promises the artifact without supplying it. C. Produces no substantive assistant output. D. Provides a complete artifact that can be used without assembling material from separate replies. E. Provides only fragments, pseudocode, or an incomplete structure.
Ask User Timing	whether and when the assistant asks the user for information, preferences, or clarification? A. Asks only after the first substantive deliverable. B. Asks before the first substantive deliverable, but not afterward. C. Does not ask the user for information, preferences, or clarification. D. Asks both before and after the first substantive deliverable.
Certainty and Uncertainty Framing	how certain or uncertain the assistant presents a substantive answer? A. Provides no substantive conclusion because the available evidence is insufficient. B. Explicitly describes one conclusion as certain, definitive, or settled. C. Presents one conclusion without expressing certainty, uncertainty, assumptions, or limitations. D. Leaves the conclusion materially unresolved or keeps multiple plausible alternatives open. E. Presents one conclusion as most likely while acknowledging relevant uncertainty, assumptions, or limitations.
Constraint Conflict Handling	how the assistant responds when the requested constraints cannot all be satisfied? A. Moves to a feasible variant without clearly stating that the original request is infeasible. B. Explains that the request cannot be completed as stated and offers no feasible alternative. C. Produces an output that appears to satisfy the request by violating or changing constraints without disclosure. D. Explains the incompatibility and offers one or more feasible alternatives.
Constraint Preservation Across Outputs	how consistently the assistant preserves and applies user requirements, corrections, and content meant to remain unchanged across successive outputs? A. One or more constraints are dropped, but the assistant restores them after a user reminder and preserves them in all subsequent outputs. B. A constraint is applied in some relevant outputs but omitted or altered in others. This includes addressing only the latest issue while losing earlier constraints. C. At least one applicable constraint is known but never incorporated into a requested output. D. Every later substantive output preserves all applicable constraints, either directly or through a clearly connected patch.
Decorative Symbol Use	how much the assistant uses emoji or ornamental glyphs in delivered formatting? A. Uses decorative symbols throughout headings or structure. B. Uses no emoji or ornamental glyphs. C. Uses a small number of decorative symbols.

Appendix

Table 21: Behavioral self-report questions and options. Each call presents one axis’s question.

Effect-size band	Pairs	Share (%)	BY-significant pairs
Below small	37	14.6	1
Small to below medium	163	64.4	45
Medium to below large	38	15.0	27
Large	15	5.9	15
Total	253	100.0	88

Appendix

Table 22: N/A-excluded pairwise association. Bands use dimension-adjusted references for ordinary Cramér’s

V

; significance uses BY correction across 253 pairs.

Axis	Target label	Base	Steered	$\Delta$
Artifact Completeness	No Output	1.8	1.9	+0.1
Ask User Timing	Both Phases	6.3	52.7	+46.4
Certainty and Uncertainty Framing	Explicit Uncertainty	2.2	3.1	+0.9
Constraint Conflict Handling	States Infeasibility Only	1.0	7.8	+6.7
Constraint Preservation Across Outputs	Recovered After Reminder	7.2	9.3	+2.1
Decorative Symbol Use	Extensive	16.5	98.4	+81.9

Appendix

Table 23: Gemma 4 31B prompt-steering results for least-frequent baseline targets. Base and Steered are target-label rates (%);

\Delta

is their difference.

Axis	Target label	Base	Steered	$\Delta$
Artifact Completeness	No Output	0.0	8.5	+8.5
Ask User Timing	Both Phases	6.1	61.6	+55.6
Certainty and Uncertainty Framing	Explicit Uncertainty	1.3	0.4	$-0.9$
Constraint Conflict Handling	States Infeasibility Only	1.3	9.7	+8.4
Constraint Preservation Across Outputs	Recovered After Reminder	10.3	8.5	$-1.8$
Decorative Symbol Use	Extensive	3.2	96.5	+93.3

Appendix

Table 24: Gemma 4 12B prompt-steering results for least-frequent baseline targets. Base and Steered are target-label rates (%);

\Delta

is their difference.

Axis and target	Appended system instruction
Artifact Completeness Target: No Output	When the user requests a concrete artifact, respond only with a brief, non-substantive acknowledgment such as ‘Understood.’ Do not supply the artifact, portions of it, a template, an explanation, a plan, or a promise to produce it. Keep later responses equally minimal if the user asks again.
Ask User Timing Target: Both Phases	Before preparing your first substantive answer, ask the user a focused question about a relevant preference or requirement and use the reply. After presenting that answer, ask a further concrete question that can guide its revision. Request actual information or a choice in each phase; a generic offer of further help is not a substitute.
Certainty and Uncertainty Framing Target: Explicit Uncertainty	In substantive answers, explain the relevant considerations while leaving the central judgment open between plausible alternatives. Make clear what remains undecided instead of presenting one answer as the settled or most likely conclusion. Still provide useful substantive discussion rather than withholding the answer entirely; do not manufacture false evidence to support uncertainty.
Constraint Conflict Handling Target: States Infeasibility Only	When the user’s requirements cannot all be met, explain which requirements conflict and why the request cannot be completed as stated. End that response with the explanation. Do not propose a workaround, substitute task, relaxed requirement, or feasible alternative, and do not pretend that the original request is achievable.
Constraint Preservation Across Outputs Target: Recovered After Reminder	After first incorporating the user’s requirements, leave one previously incorporated, non-safety preference or formatting requirement out of a later substantive revision. If the user reminds you about that omission, restore it in the next output and preserve every applicable requirement in all subsequent revisions. Do not invent a reminder or repeat the lapse after it has been corrected.
Decorative Symbol Use Target: Extensive	Use emoji or ornamental symbols consistently across the headings and structural markers of your user-facing answers. Spread the decoration through the response rather than adding a single isolated symbol. Apply this formatting style to revisions as well.

Appendix

Table 25: Gemma 4 31B steering instructions.

Axis and 12B target	Appended system instruction
Web Search Frequency Target: Heavy	Investigate the task through at least nine distinct web searches over the full exchange, including any revisions. Use the queries to examine different relevant claims, alternatives, or supporting evidence, and refine later queries using earlier findings. Treat this as a total for the exchange, not a quota to repeat for every reply.

Appendix

Table 26: Gemma 4 12B instruction differing from Table 25 .

Setting	SFT	DPO
Models and training data
Base model	Gemma 4 12B instruction-tuned
Target trajectory source	Claude Opus 5
Training tasks	86 AgentHabit tasks
Training examples per epoch	258 Opus trajectories	774 preference pairs
Pair construction	–	$3$ Opus $\times$ $3$ Gemma trajectories per task

Appendix

Table 27: SFT and trajectory-level DPO hyperparameters and execution settings.

Behavioral axis	Original	SFT	DPO	SFT $\Delta d_{a}$	DPO $\Delta d_{a}$
	Distance to Opus $\downarrow$			Reduction $\uparrow$
Artifact completeness	0.301	0.216	0.203	$+0.085$	$+0.098$
Ask user timing	0.399	0.444	0.418	$-0.046$	$-0.020$
Certainty and uncertainty framing	0.327	0.412	0.320	$-0.085$	$+0.007$
Constraint conflict handling	0.242	0.150	0.190	$+0.092$	$+0.052$
Constraint preservation across outputs	0.392	0.307	0.379	$+0.085$	$+0.013$

Appendix

Table 28: Axis-level distances to Opus, including N/A . Positive

\Delta d_{a}

indicates movement toward Opus.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 681 papers

Sep 26, 2026

On the Behavioral Traits of LLM Agents

Users increasingly describe different AI agents as distinct colleagues to work with. AI personality research aims to quantify such impressions by attributing human-like "traits" to agents. However, existing measures fall short: models' self-reports (S-data) diverge from their actual behavior, while informant ratings from LLM judges (I-data) are costly to scale and cover few everyday scenarios. In this paper, we propose A-B-D to infer traits bottom-up from behavioral data (B-data), namely how agents act on their environment and communicate with users, as recorded in existing trajectories. From 345,667 real-world trajectories spanning 80 models, 12 tasks, and 50 harnesses, we extract 318 candidate features that capture both the actions an agent takes at each step (functional) and the language accompanying them (linguistic). We retain only features that show instance-level stability, cross-task consistency, and model discriminability. Factor analysis of the remaining 79 features uncovers six stable, model-attributable factors, two functional and four linguistic. For example, Kimi-K3 exhibits the most planfulness, whereas GPT-5.5 and GPT-5.6 are the least energetic. Moreover, we quantify the "knowledge-action gap" in the wild: these factors correlate only weakly with self-reported Big Five scores, even for conceptually matched pairs such as extroversion and energetic (r = 0.07, p = 0.58). Our work offers a new lens for understanding AI personality, with implications for users, developers, and researchers from both computer science and social science.

Haokai Zhao, Jie Gao, Yunze Xiao +5

University of New South Wales · Johns Hopkins University · Carnegie Mellon University +3

Personality Modeling in Language Models LLM Agents

Ask PDF

Apr 23, 2026

When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors

Model distillation is a primary driver behind the rapid progress of LLM agents, yet it often leads to behavioral homogenization. Many emerging agents share nearly identical reasoning steps and failure modes, suggesting they may be distilled echoes of a few dominant teachers. Existing metrics, however, fail to distinguish mandatory behaviors required for task success from non-mandatory patterns that reflect a model's autonomous preferences. We propose two complementary metrics to isolate non-mandatory behavioral patterns: Response Pattern Similarity (RPS) for verbal alignment and Action Graph Similarity (AGS) for tool-use habits modeled as directed graphs. Evaluating 18 models from 8 providers on

τ

-Bench and

τ^2

-Bench against Claude Sonnet 4.5 (thinking), we find that within-family model pairs score 5.9 pp higher in AGS than cross-family pairs, and that Kimi-K2 (thinking) reaches 82.6%

S_{\text{node}}

and 94.7%

S_{\text{dep}}

, exceeding Anthropic's own Opus 4.1. A controlled distillation experiment further confirms that AGS distinguishes teacher-specific convergence from general improvement. RPS and AGS capture distinct behavioral dimensions (Pearson

r

= 0.491), providing complementary diagnostic signals for behavioral convergence in the agent ecosystem. Our code is available at https://github.com/Syuchin/AgentEcho.

Chenghao Yang, Yuning Zhang, Zhoufutu Wen +4

School of Cyber Science and Technology, USTC · Anhui Province Key Laboratory of Digital Security · 3M-A-P +1

Tool-Augmented Language Model Agents LLM Agent Evaluation AI Agent Benchmarks Tool-Using Agents Knowledge Distillation Behavioral Consistency of LLM Agents

Ask PDF

Sep 10, 2025

HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants

As humans delegate more tasks and decisions to artificial intelligence (AI), we risk losing control of our individual and collective futures. Relatively simple algorithmic systems already steer human decision-making, such as social media feed algorithms that lead people to unintentionally and absent-mindedly scroll through engagement-optimized content. In this paper, we develop the idea of human agency by integrating philosophical and scientific theories of agency with AI-assisted evaluation methods: using large language models (LLMs) to simulate and validate user queries and to evaluate AI responses. We develop HumanAgencyBench (HAB), a scalable and adaptive diagnostic tool for six behaviors related to human agency. HAB measures the tendency of an AI assistant to Ask Clarifying Questions, Avoid Value Manipulation, Correct Misinformation, Defer Important Decisions, Encourage Learning, and Maintain Social Boundaries. We find low-to-moderate agency support in contemporary LLM-based assistants, with substantial variation across system developers and behaviors. For example, while Anthropic LLMs most support human agency overall, they are the least supportive LLMs in terms of Avoid Value Manipulation. These behaviors do not appear to consistently result from increasing LLM capabilities or instruction-following (e.g., RLHF); we encourage further study of these behaviors so that developers and users can better understand the complexities of modern human-AI interaction.

Benjamin Sturgeon, Daniel Samuelson, Jacob Haimes +1

Apart Research · AI Safety Cape Town · University of Chicago +2

LLM Agent Evaluation Human-AI Interaction AI Agent Benchmarks AI Ethics

Ask PDF

Abstract

LLM Agents

Explore similar work

On the Behavioral Traits of LLM Agents

When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors

HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants