Bridging Foundational and Agentic Capabilities of LLMs

Wei Guo; Yuanjian Xu; Jianing Hao; Yongyao Duan; Ran Chen; Zhong Li; Guang Zhang

cs.CLOct 21, 2026

Bridging Foundational and Agentic Capabilities of LLMs

Authors: Wei Guo, Yuanjian Xu, Jianing Hao, Yongyao Duan, Ran Chen, Zhong Li, Guang Zhang

Organizations: Nankai University · The Hong Kong University of Science and Technology (Guangzhou) · Ant Group · OpenCSG ·

Abstract

Large language models (LLMs) are increasingly deployed as agents, yet it remains unclear how their foundational capabilities translate into agentic capabilities. End-to-end scores leave unresolved whether low benchmark capability or weak deployment drives a failure, limiting actionable diagnosis. We bridge foundational and agentic capabilities through an operation-level measurement framework. We represent each model with a foundational profile derived from capability benchmarks and an operation profile derived from interaction trajectories, and define capability realization gaps as deviations of observed operation performance from predictions based on foundational capabilities. Experiments on fifteen model services in $τ^2$ -Bench show that foundational profiles only partially predict agentic operations. Models with comparable profiles can differ substantially in execution, and these gaps remain consistent across task subsets. Operation profiles add information and substantially improve task-score prediction. The framework locates execution departures at the operation level to inform model development and agent evaluation. We will release our code and analysis artifacts.

Figures & tables

Work	Inter.	Found.	Proc.	F $\rightarrow$ Y	F $\rightarrow$ P	P $\rightarrow$ Y
BFCL ( Patil et al., 2025 )	✓	$\times$	$\times$	$\times$	$\times$	$\times$
ToolSandbox ( Lu et al., 2025 )	✓	$\times$	✓	$\times$	$\times$	$\times$
$\tau^{2}$ -Bench ( Barres et al., 2025 )	✓	$\times$	$\times$	$\times$	$\times$	$\times$
MCP-Universe ( Luo et al., 2025 )	✓	$\times$	$\times$	$\times$	$\times$	$\times$
MMAU ( Yin et al., 2025 )	$\times$	✓	$\times$	$\times$	$\times$	$\times$
ADeLe ( Zhou et al., 2026 )	$\times$	✓	$\times$	✓	$\times$	$\times$

表 1: Coverage of related agent and capability evaluations. Inter. denotes multi-turn interactive evaluation, Found. a separate foundational capability profile, and Proc. trajectory-level process measurement.

F

P

, and

Y

denote foundational profiles, process measurements, and task outcomes; arrows indicate which relationship a study tests. ✓ denotes support and

\times

no support.

图 1: Framework linking foundational capabilities, agentic operations, and task outcomes. Foundational capabilities describe resources measured outside the agent environment; agentic operations describe how a model deploys them during interaction. We measure the relationship between the two profiles and test whether operation profiles add information about task outcomes.

Op.	Scoring evidence
GI	Later decisions preserve the user’s goal, target entities, constraints, and success conditions.
ST	New information and tool results are incorporated into the current state and used correctly.
PL	The action sequence respects dependencies and task constraints.
AS	Actions and arguments are appropriate, authorized, and consistent with the applicable rules.
PM	Completed, pending, failed, and blocked requirements are distinguished correctly.
RC	A preceding error is corrected, and the correction is retained in later execution.

表 2: Trajectory evidence used to score the seven agentic operations. Each row states the behavioral evidence required for one operation; the common 0/0.5/1 label rule and aggregation procedure are defined in Section 3.2 .

Dimension	Evaluation	Reference size	Primary score
TU	MultiChallenge ( Sirdeshmukh et al., 2025 )	Source-specific	Overall rubric pass rate
IS	MRCR, 8 needles, up to 128K ( Vodrahalli et al., 2024 ; OpenAI, 2025 )	484	Mean sequence-match score
KU	MMLU-Pro ( Wang et al., 2024 )	12,032	Answer accuracy
CI	LongBench v2 ( Bai et al., 2025 )	503	Answer accuracy
RE	GPQA Diamond and text-only HLE, without tools	198 + 2,158	Equal-weight mean accuracy
SM	GPQA Diamond and SimpleQA ( Rein et al., 2023 ; OpenAI, 2024 )	198 + 300	Equal-weight mean confidence AUROC

表 3: Evaluation suite used to construct the foundational capability profile. Each row maps one profile dimension to its evaluation, reference size, and primary score. Effective denominators for source-specific scores are recorded in the supplementary materials; all primary scores are reported on a 0–100 scale.

	Foundational Capability Profile							Agentic Operation Profile
Model	TU	IS	KU	CI	RE	SM	CE	GI	ST	PL	AS	PM	RC	TM
GPT-5.6 Sol	73.0	97.6	89.1	67.1	71.8	67.8	72.7	79.2	74.5	79.3	47.4	72.7	60.5	40.1
GPT-5.6 Luna	68.7	81.8	86.0	62.4	65.3	69.1	64.3	78.3	74.6	76.8	55.2	76.7	51.1	49.7
GPT-5.5	73.3	94.2	88.1	67.0	69.7	67.5	75.9	77.1	75.3	84.1	54.1	72.7	37.9	44.7
Claude Opus 5	71.7	97.7	91.6	74.4	74.0	75.1	63.4	81.1	80.7	84.3	59.6	83.3	69.4	47.9
Claude Fable 5	75.3	96.4	91.5	65.2	74.0	86.2	63.5	80.9	82.7	79.5	54.4	78.4	68.3	47.9

表 4: Foundational capability and agentic operation profiles for all 15 model services. Scores use a 0–100 scale; bold marks the best value in each column, and cell shading runs from white (lower) to light teal (higher) within each column. Provider icons and separators identify model families. Dimension definitions appear in Sections 3.1 and 3.2 ; score provenance and evaluation configurations appear in the appendix.

图 2: Mapping foundational capabilities to agentic operations. (a) Pearson correlations between the seven foundational and seven operation scores across 15 models. (b) Model-family-held-out predictive skill

Q^{2}

for each operation using foundational scores and the common ridge penalty

\alpha=10

. Action Selection and Planning have the strongest held-out signal, while Recovery and Termination are the least predictable from foundational scores.

Model	Observed	Predicted	Gap
GPT-5.6 Sol	65.1	68.34	$-3.24$
GPT-5.6 Luna	67.6	66.97	$+0.63$
GPT-5.5	66.5	67.00	$-0.50$
Claude Opus 5	72.8	66.47	$+6.33$
Claude Fable 5	70.7	62.67	$+8.03$
Claude Sonnet 5	69.7	64.23	$+5.47$

表 5: Overall capability realization gaps by model. Observed and model-family-held-out predicted operation macros use a 0–100 scale under the common penalty

\alpha=10

; the gap is observed minus predicted. Provider icons and separators identify model families. Macro-prediction MAE is 4.47 versus 5.92 for the training-mean baseline, with

Q^{2}=0.373

图 3: Model-family-held-out prediction of Banking task reward from foundational and operation profiles. Panels compare foundational features (

F

), operation features (

O

), and their combination (

F+O

). Each point is one model service, dashed lines mark equality between predicted and observed reward, and all predictors use

\alpha=10

Predictors	$\rho\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$Q^{2}\uparrow$
Training-mean baseline	–	8.46	10.40	0.000
Foundational profile $F$	0.416	6.91	8.18	0.382
Operation profile $O$	0.693	4.83	5.68	0.702
Joint profile $F+O$	0.776	4.63	5.38	0.733

表 6: Operation profiles improve model-family-held-out prediction of Banking task reward. All learned predictors use

\alpha=10

;

\rho

is Spearman correlation, errors are in reward points, and

Q^{2}

uses the fold-specific training-mean baseline. Arrows mark the preferred direction; bold marks the best estimate.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Round	States	Repeats	Steps	Runs
Initial (N/I)	4	3	1	72
Expansion (N)	8	3	1	72
Short continuation (N)	4	2	$\leq 3$	24
Final (N)	3	3	$\leq 3$	27

Appendix

表 7: Coverage of the shared-initial-state experiments. States are restored checkpoints, repeats are continuations per condition, and steps count assistant decisions. N denotes natural continuation and I the generic checking prompt.

图 4: Sensitivity of Banking reward prediction to ridge regularization. Curves compare foundational (

F

), operation (

O

), and joint (

F+O

) inputs using (a) MAE and (b) Spearman correlation. The horizontal axis is logarithmic, and the vertical dashed line marks the common setting

\alpha=10

Model	Agentic Operation Residuals
Model	GI	ST	PL	AS	PM	RC	TM
GPT-5.6 Sol	$+1.13$	$-2.74$	$-0.78$	$-6.26$	$-4.70$	$+5.11$	$-6.31$
GPT-5.6 Luna	$+0.46$	$-0.76$	$-2.80$	$+3.10$	$+1.93$	$-3.48$	$+3.92$
GPT-5.5	$+0.19$	$-0.78$	$+4.62$	$+2.10$	$-3.00$	$-14.60$	$-0.86$
Claude Opus 5	$+4.28$	$+4.41$	$+5.91$	$+7.33$	$+7.56$	$+19.63$	$+4.23$
Claude Fable 5	$+8.59$	$+9.05$	$+3.64$	$+6.12$	$+7.66$	$+21.38$	$+8.12$

Appendix

表 10: Operation-level capability realization gaps for every model service. Values are observed operation scores minus model-family-held-out predictions at

\alpha=10

; they therefore differ from the overall gap produced by the separate macro predictor. Provider icons and separators identify model families.

图 5: Capability realization gaps by model and operation. (a) Observed operation scores minus model-family-held-out predictions from foundational profiles at $\alpha=10$ . (b) Gaps from a separate predictor of the overall operation score reported in Table 5 . Blue indicates execution above prediction and red below prediction; exact values appear in Table 10 .

图 6: Association between Constrained Expression and Recovery across analysis specifications. (a) Raw CE and RC scores for 15 models. (b) Pearson and Spearman correlations under six sensitivity specifications; parentheses give the number of included models.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,194 papers

Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents
Oct 3, 2026Nijesh Upreti, Chris Sypherd, Vaishak BelleLLM Agent Evaluation LLM Agents
The University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, UK
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
May 27, 2026Pengyu Zhu, Lijun Li, Yaxing Lyu +8LLM Agent Evaluation AI Agent Benchmarks
1Beijing University of Posts and Telecommunications · 2Shanghai Artificial Intelligence Laboratory · 4North China Electric Power University +1
Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach
Oct 14, 2026Jiarui Qin, Junjie Huang, Yunjia Xi +5LLM Agent Evaluation Agent Evaluation
Tencent Youtu Lab · Shanghai Jiao Tong University

Oct 3, 2026J/K move · Enter open · S save

Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents

Nijesh Upreti, Chris Sypherd, Vaishak Belle

The University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, UK

LLM Agent Evaluation LLM Agents AI Agent Benchmarks Metacognition in Language Models

Open paper AskPDF arXiv

Agentic large language model (LLM) systems are commonly implemented as an LLM in a loop with Planning, Memory, Tools, and Control Flow. This application-focused view connects agentic LLM research with deployable systems and leaves open how such systems should be evaluated beyond end-to-end task success. Building on this view, we define agentic cognitive depth as a trajectory-level profile across five operational criteria. The profile contains context sensitivity (

C

), temporal continuity (

T

), multimodal coordination (

M

), adaptive interaction (

A

), and metacognitive monitoring (

Mc

). The first four criteria measure how well Control Flow, Memory, Tools, and Planning are used across a trajectory. The fifth measures whether the system monitors and regulates the full run. For each criterion, we give operational proxies and a perturbation procedure, then connect the profile to the agent's world model. We provide the structure needed to extend benchmarks such as GAIA, SWE-bench, WebArena, and TRIP-Bench with per-criterion diagnostics. Symbolic verifiers, structured memory, planner coupling, and tool constraints provide practical ways to build and test these capacities.

Abstract

LLM Agent Evaluation

Explore similar work

Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents

UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities

Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach