Agent Behavior Maps: Locating Agents by What They Would Do, Without Running Them

Shota Fujikawa; Takuya Tamura; Kosuke Akimoto; Daichi Haraguchi; Masafumi Oyamada

cs.LGOct 21, 2026

Agent Behavior Maps: Locating Agents by What They Would Do, Without Running Them

Authors: Shota Fujikawa, Takuya Tamura, Kosuke Akimoto, Daichi Haraguchi, Masafumi Oyamada

Organizations: The University of Tokyo · NEC Corporation

Abstract

LLM agents with similar success rates can follow different action sequences. An Agent Behavior Map represents LLM-harness combinations using centered action log-likelihoods on shared reference trajectories, without requiring new environment rollouts from the target agent. We derive error bounds relating squared coordinate distances to trajectory KL under explicit assumptions, and across eight cross-scored models the distances track the directly measured trajectory KL in rank order. Across 100 LLMs and four environments, ALFWorld-based coordinates support performance prediction and identify behavioral similarities. In ALFWorld, these similarities persist after controlling for success rates and model size, including across model families. As an extension, harness-component displacements approximately compose, with accuracy varying across components and environments.

Figures & tables

Figure 1: Agents close on the map act alike when run. (a) 100 LLMs with the BASE harness, scored on 500 ALFWorld reference trajectories (t-SNE layout; neighbors computed in the original space).

\star

: gemma4-31b; green: its five nearest models on the map; orange rings: the five closest in parameter count. (b) Action sequences of these 11 models on one ALFWorld task. All succeed, but four of the five map neighbors share the anchor’s short route, whereas three of the five size-matched models take longer ones. (c) Effect size of the behavioral distance to the

k

-th nearest model on the map (solid) or in parameter count (dashed), averaged over four aspects and 100 models (§ 3.3 ); positive means more similar than the average pair.

Source	Step	Trajectory	Ratio
Local expansion	$\tfrac{1}{6}\varepsilon^{3}\mathbb{E}_{i}[T]$	$\tfrac{1}{4}\varepsilon^{3}\mathbb{E}_{i}[T^{2}]$	$\tfrac{3\mathbb{E}_{i}[T^{2}]}{2\mathbb{E}_{i}[T]}$
Ref. mismatch	$\tfrac{1}{2}\lambda T_{\max}\varepsilon^{2}$	$\tfrac{1}{2}\lambda T_{\max}^{2}\varepsilon^{2}$	$T_{\max}$
Centering	$\mu^{2}/(2\bar{T})$	$\mu^{2}/2$	$\bar{T}$
Sampling	$\tfrac{6T_{\max}-1}{2}\varepsilon^{2}u_{N}$	$\tfrac{5}{2}T_{\max}^{2}\varepsilon^{2}u_{N}$	$\approx\tfrac{5}{6}T_{\max}$

Table 1: Error bounds by source, with constants: approximation error (Theorem 1 , top three rows) and sampling error (Corollary 1 , bottom row, which defines

u_{N}

). The trajectory bound is about

T

times the step bound for every source.

Figure 2: Map estimates, step version

\tfrac{\bar{T}}{2}\hat{D}_{S}^{2}

and trajectory version

\tfrac{1}{2}\hat{D}_{\tau}^{2}

, against the directly measured symmetric trajectory KL

K_{\mathrm{sym}}

: 8 models, 28 pairs, centered coordinates, log–log axes; error bars are standard errors over the 988 shared ALFWorld train tasks. Both track

K_{\mathrm{sym}}

, the step version more closely.

	Baselines			Coordinates
Downstream task	Size	Family	$\bar{\ell}$	kNN	Ridge	+size
ALFWorld	.63	.46	.20	.88	.81	.82
TextWorld	.71	.44	.33	.76	.77	.78
BFCL	.48	.34	.32	.69	.69	.65
MiniWoB++	.64	.24	.22	.28	.56	.63
4-bench avg.	.74	.42	.34	.74	.78	.83

Table 2: Predicting success rates without running the agents: Pearson

r

under leave-one-model-out over 100 models. Baselines use size (

\log

parameter count), the family mean, or

\bar{\ell}

(mean log-likelihood); coordinate methods are kNN, Ridge, and Ridge with size as a covariate (+size). Bold: higher than size, with a bootstrap 95% interval of the difference excluding zero. RMSE and all predictors in Table 7 .

ALFWorld
	Available without running			Map	Reference
Benchmark / measure	Size	Family	$\bar{\ell}$	Coord.	Success
Success rate	.41 ∗	.49 ∗	.20 ∗	.60 ∗	—
Composition	.23 ∗	.28 ∗	.05	.52 ∗†	.30 ∗
Global	.40 ∗	.62 ∗	.18 +	.88 ∗†	.65 ∗
Local	.41 ∗	.68 ∗	.20 ∗	.96 ∗†	.74 ∗

Table 3: Effect sizes of the behavioral similarity of each model’s five neighbors (§ 3.3 ; positive: neighbors more similar than the remaining models), for the difference in success rate and the four aspects of § 3.3 . Bold: largest of the four columns other than the reference. ∗ :

p<0.01

, + :

p<0.05

; † : significantly closer than success-rate neighbors (

p<0.05

). The success-rate row is omitted for success-rate neighbors. All measures in Table 8 .

Figure 3: Harness displacements add up within an LLM. The 144

\tau

-bench agents (12 LLMs

\times

12 harnesses) in their first two principal components (PC1 separates LLMs; axes scaled independently). Squares: BASE; solid arrows: single-component displacements; dashed: their sum, ending at the predicted position of retrieve + think (open circles); filled circles: its measured position. Cosine and norm ratio: medians over the 12 LLMs in the full space.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	Trajectory KL (§ 3.1 )	100 LLMs (§ 3.2 , 3.3 )
LLMs	8	100
Reference trajectories	Each model’s own rollouts on the 988 of 1,000 ALFWorld train tasks completed by all 8 models (7,904 trajectories, 187,961 steps)	Rollouts of Qwen3.8-27B and Gemma4-E4B on 250 ALFWorld train tasks (500 trajectories, 10,388 steps)
Reference distribution $Q$	Equal-weight pool of the 8 models	Equal-weight mixture of the 2 generators
Scoring	Grammar-constrained, canonical tokens; every model scores every trajectory ( $8\times 8$ )	No grammar mask; each model’s chat template
What is compared	Directly measured $K_{\mathrm{sym}}$ (28 pairs)	Success rates and action sequences on ALFWorld valid_unseen, TextWorld, BFCL, MiniWoB++ (50 tasks each; behavior on BFCL uses multi_turn 25); one run per task at temperature 0.7

Appendix

Table 4: The two experiments of § 3 . Both fix the BASE harness. The multi-harness experiment of § 4 is in Table 11 .

Model	B	Family	Model	B	Family
CohereForAI/c4ai-command-r7b-12-2024	8.0	Cohere	allenai/Llama-3.1-Tulu-3-8B-SFT	8.0	Llama
deepseek-ai/deepseek-coder-1.3b-instruct	1.3	DeepSeek	codellama/CodeLlama-13b-Instruct-hf	13.0	Llama
deepseek-ai/deepseek-coder-6.7b-instruct	6.7	DeepSeek	codellama/CodeLlama-34b-Instruct-hf	33.7	Llama
deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct	15.7	DeepSeek	mistralai/Ministral-3-3B-Instruct-2512	3.8	Mistral
deepseek-ai/DeepSeek-V2-Lite-Chat	15.7	DeepSeek	mistralai/Mistral-7B-Instruct-v0.2	7.2	Mistral
deepseek-ai/deepseek-coder-33b-instruct	33.3	DeepSeek	mistralai/Mistral-7B-Instruct-v0.3	7.2	Mistral

Appendix

Table 5: The 100 open-weight LLMs of § 3.2 – 3.3 , identified by their Hugging Face repository, with parameter counts in billions (total parameters for MoE models) and the family labels used as a baseline.

\diamond

: the eight models of the trajectory-KL comparison (§ 3.1 );

\star

: the twelve LLMs of the harness experiment (§ 4 ).

	$r$	$\rho$	RMSE
Step version
Centered	0.90	0.95	2,056
Uncentered	0.83	0.92	3,773
Trajectory version
Centered	0.86	0.94	29,792
Uncentered	0.77	0.92	72,014

Appendix

Table 6: Agreement between the estimates read from the map and the directly measured symmetric trajectory KL

K_{\mathrm{sym}}

: 8 models, 28 pairs, 988 shared ALFWorld train tasks. The estimates are

\tfrac{\bar{T}}{2}\hat{D}_{S}^{2}

for the step version and

\tfrac{1}{2}\hat{D}_{\tau}^{2}

for the trajectory version. RMSE is in nats.

Figure 4: Predicting success rates without running the agents, with leave-one-model-out over 100 models. The horizontal axis is the prediction and the vertical axis the measured success rate. Open gray circles show the size baseline, a regression on

\log|\theta_{i}|

; blue points show coordinates alone, a Ridge regression on centered coordinates (the Ridge column of Table 2 ). Panels from left: ALFWorld, TextWorld, BFCL, MiniWoB++, and the four-benchmark average.

Information available without running
Predictor	ALFWorld	TextWorld	BFCL	MiniWoB++	4-bench avg.	3 other envs.
Training mean	16.4 / — ↑	29.0 / — ↑	16.1 / — ↑	27.5 / — ↑	19.2 / — ↑	21.4 / — ↑
Size $\log\|\theta_{i}\|$	12.6 / 0.63	20.2 / 0.71	13.9 / 0.48	20.9 / 0.64	12.9 / 0.74	14.6 / 0.72
Family mean	14.7 / 0.46	26.8 / 0.44 ↑	15.5 / 0.34	27.6 / 0.24 ↑	17.7 / 0.42 ↑	19.9 / 0.40 ↑
Mean log-likelihood $\bar{\ell}$	16.0 / 0.20 ↑	27.2 / 0.33 ↑	15.1 / 0.32	26.6 / 0.22 ↑	18.0 / 0.34 ↑	19.9 / 0.34 ↑
Size + $\bar{\ell}$	12.4 / 0.65	19.2 / 0.74	13.4 / 0.54	20.7 / 0.65	12.2 / 0.77	13.8 / 0.76

Appendix

Table 7: All predictors for performance prediction, with leave-one-model-out over 100 models. Each cell gives RMSE in points / Pearson

r

\downarrow

and

\uparrow

mark a 95% bootstrap interval of the RMSE difference from the size baseline that lies below or above zero. The training mean is a constant prediction, so no

r

is shown.

A Choice of neighbors
	Composition		Global	Local	Early steps			Outcome
	Type JS	Profile	Edit	2-gram	First	Cond.	Diverg.	Success	Pattern
ALFWorld
Coordinates	0.52	0.63	0.88	0.96	0.61	0.51	0.70	0.60	0.32
Size	0.23	0.40	0.40	0.41	0.30	0.26	0.34	0.41	0.27
Family	0.28	0.48	0.62	0.68	0.49	0.43	0.62	0.49	0.35

Appendix

Table 8: Neighbor effect sizes for all behavioral distances, including those not in the main text; definitions as in Table 3 , 100 models. A: raw behavioral distances, comparing how neighbors are chosen; bold marks the largest value among the four neighbor types available without running. B: effect of coordinate neighbors on residuals after regressing out success rate and size. Unmarked values have

p<0.01

, + marks

p<0.05

, and gray values are not significant.

Neighbors in uncentered coordinates
	Composition		Global	Local	Early steps			Outcome
	Type JS	Profile	Edit	2-gram	First	Cond.	Diverg.	Success	Pattern
Raw behavioral distance
ALFWorld	0.32	0.51	0.65	0.73	0.45	0.38	0.52	0.64	0.45
TextWorld	0.19	0.49	0.37	0.40	0.12 +	0.39	0.27 +	0.66	0.68
BFCL	0.12	0.31	0.21	0.31	0.22	0.22	0.45	0.47	0.35

Appendix

Table 9: Variants of coordinate neighbors, with definitions as in Table 8 . Top: neighbors in uncentered coordinates. Bottom: coordinate neighbors restricted to models from other families. Neighbors in centered coordinates are in Table 8 . Unmarked values have

p<0.01

, + marks

p<0.05

, and gray values are not significant.

Reliability correlation across task halves
	ALFWorld	TextWorld	BFCL	MiniWoB++
Type JS	0.97	0.89	0.89	0.95
Edit distance	0.91	0.92	0.89	0.92
2-gram	0.88	0.90	0.87	0.85
First action	0.78	0.82	0.70	0.91
Divergence	0.76	0.90	0.62	0.91

Appendix

Table 10: Reliability of the behavioral distances: correlation between distance matrices computed on two random halves of the tasks, averaged over 30 splits; all standard deviations are at most 0.08. The profile and conditional disagreement cannot be split by task and are omitted.

	Reference trajectories (source)	$N$ / steps	LLMs scored
$\tau$ -bench	HAL airline	359 / 4,213	12
ALFWorld	own, 3 LLMs $\times$ 12 harnesses	210 / 2,299	6
SWE-Bench	HAL SWE-agent	100 / 3,916	6

Appendix

Table 11: Setup of the multi-harness experiment: 12 LLMs

\times

12 harnesses, scored on the benchmarks that have reference trajectories (ALFWorld and SWE-Bench use 6 of the 12 LLMs).

N

is the number of reference trajectories and “steps” the number of reference steps scored by all agents.

	$\tau$ -bench			ALFWorld			SWE-Bench
	cos	Norm ratio	Rel. error	cos	Norm ratio	Rel. error	cos	Norm ratio	Rel. error
retrieve + replan	0.97	0.99	0.25	0.62	0.95	0.94	0.97	0.97	0.25
retrieve + think	0.94	1.00	0.34	0.61	0.95	0.95	0.96	0.96	0.28
summarize + replan	0.91	0.96	0.45	0.60	0.89	1.00	0.78	0.89	0.72
summarize + think	0.87	0.98	0.51	0.63	0.91	0.93	0.77	0.86	0.76
replan + think	0.81	0.74	0.77	0.70	0.73	1.03	0.74	0.73	0.95

Appendix

Table 12: Full additivity table (medians over LLMs: cosine, norm ratio, relative reconstruction error).

\tau

-bench uses 12 LLMs; ALFWorld and SWE-Bench use 6 of the 12 LLMs (ALFWorld uses only the 2,299 reference steps scored by all 72 agents). The pooled rows take the median over all LLM–harness cases: the five two-component harnesses (60 cases on

\tau

-bench, 30 on ALFWorld and SWE-Bench) and the two three-component harnesses (24 / 12 / 12 cases).

Figure 5: Additivity for all seven compositions on

\tau

-bench (12 LLMs), in the PC1–PC2 projection of Figure 3 ; axes are scaled independently. The last panel identifies the LLMs by their BASE positions (gray: the other harnesses). Two-component compositions are drawn as parallelograms, three-component ones as chains

\mathrm{BASE}+\Delta_{A}+\Delta_{B}+\Delta_{C}

. Titles give the cosine and norm ratio in the full coordinate space (medians over LLMs).

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,203 papers

GraphACM: Graph-based Action Consistency Measurement for Uncertainty Quantification in LLM Agents
Oct 15, 2026Yaxin Ma, Wenbo Zhang, Satya Krishna Pothapragada +1Behavioral Consistency of LLM Agents LLM Agent Evaluation
University of Florida, Gainesville, FL, USA · University of California, Irvine, CA, USA
AgentHabit: Characterizing Distinct Behaviors of Agents on Everyday Tasks
Sep 26, 2026Woojung Song, Hoyeol Yang, Jeonghoon Shim +4LLM Agents LLM Agent Evaluation
Seoul National University, Seoul, Republic of Korea
When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors
Apr 23, 2026Chenghao Yang, Yuning Zhang, Zhoufutu Wen +4Tool-Augmented Language Model Agents LLM Agent Evaluation
School of Cyber Science and Technology, USTC · Anhui Province Key Laboratory of Digital Security · 3M-A-P +1

Oct 15, 2026J/K move · Enter open · S save

GraphACM: Graph-based Action Consistency Measurement for Uncertainty Quantification in LLM Agents

Yaxin Ma, Wenbo Zhang, Satya Krishna Pothapragada, Alina Zare

University of Florida, Gainesville, FL, USA · University of California, Irvine, CA, USA

Behavioral Consistency of LLM Agents LLM Agent Evaluation LLM Agents LLM Agent Reliability LLM Uncertainty Estimation

Open paper AskPDF arXiv

Behavioral consistency across repeated executions provides an informative signal for assessing the reliability of large language model (LLM) agents. However, existing approaches typically summarize consistency with a single task-level score, without identifying where behavioral differences occur or whether actions are compared under equivalent decision contexts. We propose a graph-based framework for fine-grained behavioral consistency measurement and uncertainty quantification. Given multiple rollouts of the same task, we merge equivalent states into a directed, action-labeled graph, enabling action comparisons at shared decision states. We introduce node-level and global measures of action consistency, using the global consistency score as a proxy for uncertainty to predict task success or failure. Experiments across three LLMs on ALFWorld, AppWorld, and WebShop demonstrate that our measure is positively associated with task success and achieves an average AUROC of 0.811 for task-success prediction, outperforming the strongest behavioral-consistency baseline (0.729) and verbalized confidence (0.769). These results highlight the effectiveness of state-conditioned behavioral consistency and pinpoint where agent actions agree.

Abstract

LLM Agent Evaluation

Explore similar work

GraphACM: Graph-based Action Consistency Measurement for Uncertainty Quantification in LLM Agents

AgentHabit: Characterizing Distinct Behaviors of Agents on Everyday Tasks

When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors