TabAgent: A Framework for Replacing Agentic Generative Components with Tabular-Textual Classifiers

Ido Levy; Eilam Shapira; Yinon Goldshtein; Avi Yaeli; Nir Mashkif; Segev Shlomov

cs.CLFeb 18, 2026

TabAgent: A Framework for Replacing Agentic Generative Components with Tabular-Textual Classifiers

Authors: Ido Levy, Eilam Shapira, Yinon Goldshtein, Avi Yaeli, Nir Mashkif, Segev Shlomov

Organizations: IBM Research, Haifa, Israel.

Abstract

Agentic systems, AI architectures that autonomously execute multi-step workflows to achieve complex goals, are often built using repeated large language model (LLM) calls for closed-set decision tasks such as routing, shortlisting, gating, and verification. While convenient, this design makes deployments slow and expensive due to cumulative latency and token usage. We propose TabAgent, a framework for replacing generative decision components in closed-set selection tasks with a compact textual-tabular classifier trained on execution traces. TabAgent (i) extracts structured schema, state, and dependency features from trajectories (TabSchema), (ii) augments coverage with schema-aligned synthetic supervision (TabSynth), and (iii) scores candidates with a lightweight classifier (TabHead). On the long-horizon AppWorld benchmark, TabAgent maintains task-level success while eliminating shortlist-time LLM calls, reducing latency by approximately 95% and inference cost by 85-91%. Beyond tool shortlisting, TabAgent generalizes to other agentic decision heads, establishing a paradigm for learned discriminative replacements of generative bottlenecks in production agent architectures.

Figures & tables

Panel of Figure 1 — Figure 1: (a) TabSchema compiles trajectory-derived schema, state, and dependency signals into tabular features. (b) TabHead consumes features and candidates to output calibrated probabilities.

Method	R-precision (P@R)					Recall@7					Recall@9
	AZ	GM	PH	SN	SP	AZ	GM	PH	SN	SP	AZ	GM	PH	SN	SP
BM25	0.28	0.19	0.35	0.42	0.28	0.48	0.51	0.54	0.72	0.60	0.54	0.58	0.57	0.74	0.71
DSR (Zero-Shot)	0.39	0.18	0.30	0.55	0.30	0.65	0.46	0.82	0.89	0.62	0.67	0.53	0.88	0.96	0.70
DSR-FT (Real-Only)	0.45	0.52	0.57	0.63	0.46	0.89	0.84	0.95	0.99	0.82	0.90	0.87	0.95	0.99	0.85
DSR-FT (Real+synth)	0.69	0.56	0.85	0.92	0.64	0.93	0.90	0.98	1.00	0.85	0.95	0.93	0.99	1.00	0.88
Llama-3.2-1B-Instruct	0.02	0.03	0.06	0.07	0.09	0.29	0.27	0.29	0.45	0.33	0.32	0.29	0.33	0.46	0.34

Table 1: Shortlisting performance across five applications: AZ =Amazon, GM =Gmail, PH =Phone, SN =SimpleNote, SP =Spotify. Real-only means training only on features extracted from successful real trajectories. Real+synth adds schema-aligned synthetic supervision generated by TabSynth.

Figure 2: Runtime–cost trade-off on log–log axes. Marker area encodes macro

P@R

across five apps. The curve shows the Pareto frontier (non-dominated trade-offs). TabAgent lies in the faster-and-cheaper corner, while GPT-4.1 (API) , the SOTA generative reference, attains higher macro

P@R

at substantially higher cost and latency. We omit DSR-FT to isolate LLM vs. classifier effects

Figure 3: TabAgent on CodeAct with task-description as only feature vs. TabAgent on CUGA (TabSchema workflow features) on AZ/GM. Slopegraph for

\mathrm{P}\!@\!\mathrm{R}

, Recall@7, and Recall@9; lines rise left

\to

right, indicating that extracted workflow features dominate task-description-only input.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Application	APIs available	Typical horizon	Common dependencies
Amazon (AZ)	$\sim 50$	about 45	search $\rightarrow$ product detail $\rightarrow$ add to cart $\rightarrow$ checkout
Gmail (GM)	$\sim 50$	about 45	list threads $\rightarrow$ open message $\rightarrow$ parse fields $\rightarrow$ act
Phone (PH)	$\sim 50$	about 45	list alarms $\rightarrow$ match to calendar events $\rightarrow$ disable or update
SimpleNote (SN)	$\sim 50$	about 45	retrieve or search $\rightarrow$ edit or create $\rightarrow$ save or sync
Spotify (SP)	$\sim 50$	about 45	login $\rightarrow$ list playlists $\rightarrow$ compute duration $\rightarrow$ play

Appendix

Table 2: Five AppWorld applications used in our study. AppWorld implements nine apps with four hundred fifty seven APIs in total, about fifty per app on average. The five here are a subset. Tasks often involve dozens of calls; see the per split averages in the text.

Panel of Figure 4 — Figure 4: Difficulty-level distribution over all tasks ( $N{=}605$ ).

Panel of Figure 5 — Figure 5: Distribution of tools used per task ( $N{=}605$ ).

# Tools	Count	Share (%)
1	155	25.6
2	175	28.9
3	151	25.0
4	69	11.4
5	29	4.8
6	10	1.7

Appendix

Table 3: Frequency of tools-per-task (

N{=}605

App	$n$	$\mu$	Median	$\sigma$	Min	Max	CV
amazon	230	2.70	3	1.49	1	9	0.55
gmail	190	2.89	3	1.89	1	15	0.66
phone	73	1.70	1	0.95	1	6	0.56
simplenote	17	3.35	3	1.58	2	8	0.47
spotify	19	2.74	3	1.10	1	4	0.40

Appendix

Table 4: Per-application summary of tools per task (five core apps;

N{=}605

Panel of Figure 6 — Figure 6: Average tools per task by application ( $N{=}605$ ; $n$ shown in Table 4 ).

Application	Count	Share (%)
amazon	230	38.0
gmail	190	31.4
phone	73	12.1
simplenote	17	2.8
spotify	19	3.1

Appendix

Table 5: Tasks per application (five core apps;

N{=}605

App	Easy	Medium	Hard	Total
amazon	178 (77.4%)	49 (21.3%)	3 (1.3%)	230
gmail	139 (73.2%)	47 (24.7%)	4 (2.1%)	190
phone	71 (97.3%)	2 (2.7%)	0 (0.0%)	73
simplenote	11 (64.7%)	5 (29.4%)	1 (5.9%)	17
spotify	14 (73.7%)	5 (26.3%)	0 (0.0%)	19

Appendix

Table 6: Application

\times

difficulty cross-tabulation (counts with row-wise percentages;

N{=}605

Contrast: TabAgent(+synth) $-$ DSR-FT(+synt)	AZ	GM	PH	SN	SP
$\mathrm{P}\!@\!\mathrm{R}$		$\dagger$
$\mathrm{Recall}@9$				$\ddagger$
$\mathrm{Recall}@7$				$\ddagger$

Appendix

Table 7: Significance grid (per metric Holm correction,

m{=}5

). Only Gmail

\mathrm{P}\!@\!\mathrm{R}

is robustly significant; SimpleNote fixed-

k

saturates and is flagged

\ddagger

Contrast: TabAgent(+synth) $-$ Llama-3.1-8B	AZ	GM	PH	SN	SP
$\mathrm{P}\!@\!\mathrm{R}$	$\dagger$	$\dagger$	$\dagger$		$\dagger$
$\mathrm{Recall}@9$	$\dagger$	$\dagger$	$\dagger$		$\dagger$
$\mathrm{Recall}@7$	$\dagger$	$\dagger$	$\dagger$		$\dagger$

Appendix

Table 8: Significance grid vs. an LLM shortlister (per metric Holm correction,

m{=}5

). TabAgent(+synth) significantly exceeds Llama-3.1-8B on 4/5 apps for

\mathrm{P}\!@\!\mathrm{R}

and for fixed-

k

; SimpleNote fixed-

k

is not marked due to wide CIs and ceiling proximity.

App	TabAgent(+synth)	DSR-FT(+synt)	$\boldsymbol{\Delta}$
AZ	0.712 [0.675, 0.753]	0.695 [0.648, 0.736]	+0.017
GM	0.659 [0.607, 0.711]	0.556 [0.503, 0.609]	+0.103
PH	0.899 [0.841, 0.954]	0.850 [0.771, 0.923]	+0.048
SN	0.961 [0.902, 1.000]	0.922 [0.804, 1.000]	+0.039
SP	0.611 [0.444, 0.773]	0.644 [0.472, 0.787]	$-$ 0.032

Appendix

Table 9:

\mathrm{P}\!@\!\mathrm{R}

CIs for TabAgent(+synth) vs. DSR-FT(+synt). Differences align with Table 7 .

App	TabAgent(+synth)	Llama-3.1-8B	$\boldsymbol{\Delta}$
AZ	0.712 [0.675, 0.753]	0.593 [0.551, 0.634]	+0.120
GM	0.659 [0.607, 0.711]	0.494 [0.449, 0.538]	+0.166
PH	0.899 [0.841, 0.954]	0.631 [0.546, 0.711]	+0.267
SN	0.961 [0.902, 1.000]	0.750 [0.500, 1.000]	+0.211
SP	0.611 [0.444, 0.773]	0.340 [0.201, 0.493]	+0.271

Appendix

Table 10:

\mathrm{P}\!@\!\mathrm{R}

CIs for TabAgent(+synth) vs. Llama-3.1-8B. Large, consistent margins underpin Table 8 .

Metric	App	LB (w/o desc.)	UB (w/o desc.)	LB (w/ desc.)	UB (w/ desc.)
$\mathrm{P}\!@\!\mathrm{R}$	AZ	0.508	0.612	0.353	0.448
$\mathrm{P}\!@\!\mathrm{R}$	GM	0.476	0.590	0.422	0.529
$\mathrm{P}\!@\!\mathrm{R}$	PH	0.544	0.717	0.598	0.746
$\mathrm{P}\!@\!\mathrm{R}$	SN	0.715	0.911	0.730	0.908
Recall@3	AZ	0.599	0.695	0.442	0.540
Recall@3	GM	0.533	0.641	0.479	0.593

Appendix

Table 11: 95% CI bounds (lower/upper) for the task-description ablation across apps (AZ, GM, PH, SN).

Model	Cost / read ($)	Runtime / read (s)
CUGA’s GPT-4.1 (API) shortlister	0.052	7.50
Llama-3.1-8B-Instruct	0.0284	378.42
Llama-3.2-3B-Instruct	0.0149	198.12
Llama-3.2-1B-Instruct	0.00754	100.48
TabAgent (50M params)	2.0e -7	0.002682

Appendix

Table 12: Per-read cost and runtime. Local costs are estimated on a single V100-32GB as described in § C ; GPT-4.1 uses the metered API charge.

Figure 7: TabSynth pipeline: Rigorous schema-aligned synthetic data generation. ( 1 ) Inputs: FEATURES_CARD, TRAJECTORIES, TOOL_CATALOG, and budget $B{=}10$ constrain ( 2 ) GPT-4.1 synthesis to generate schema-compliant feature vectors. ( 3 ) Three-stage validation checks schema types/arity, dependency feasibility with real trajectories, and de-duplicates via LSH. ( 4 ) Statistical alignment ensures distributional fidelity. Failures trigger ( 5 ) adaptive control (reduce $B$ , regenerate) with feedback to synthesis. ( 6 ) Output: Validated synthetic vectors mixed with real data for training.

Method	AUC	Acc	Prec	Rec	F1
DSR	0.9861	0.9512	0.8610	0.9025	0.8809
DSR-FT	0.9890	0.9636	0.8797	0.9251	0.9013
TabAgent	0.9945	0.9863	0.9667	0.9568	0.9617

Appendix

Table 13: Application selection (multi-label) instance-level macro metrics (5-fold GroupKFold over tasks). TabAgent substantially outperforms both frozen and fine-tuned E5 embeddings, achieving higher recall and F1.

Figure 8: Mean

|\mathrm{SHAP}|

(%) for the TabHead shortlister on held-out tasks (5-fold CV).

Model	Total	Unique to	Overlap	Three-way
	Features	This Model	w/ GPT-4.1	Agreement
GPT-4.1	13	4	—	9
Gemini 2.5	21	12	9 (69%)	9
Claude 4.5 Sonnet	11	2	9 (82%)	9
Pairwise overlap (any two models): 11 features (85% of GPT-4.1 baseline)

Appendix

Table 14: Cross-model feature extraction comparison across trajectories. Three-way agreement represents the 9 features proposed by all three models (69% of GPT-4.1 baseline). GPT-4.1 extracts a balanced feature set, Gemini 2.5 produces expansive coverage with finer-grained text statistics, and Claude 4.5 Sonnet consolidates features into fewer robust representations.

Comparison	Semantic Similarity	Importance Ranking
	(BERTScore)	Agreement (Kendall $\tau$ )
GPT-4.1 vs Gemini 2.5	0.79	0.65
GPT-4.1 vs Claude 4.5 Sonnet	0.88	0.78
Gemini 2.5 vs Claude 4.5 Sonnet	0.76	0.61
Mean across all pairs	0.81	0.68

Appendix

Table 15: Semantic alignment and importance ranking agreement across frontier models. Semantic similarity computed using sentence embeddings with BERTScore-style comparison on feature names concatenated with descriptions, ranging from 0 (unrelated) to 1 (identical). Kendall tau measures agreement between two annotators (part of the research team) who independently ranked shared features by importance for the tool shortlisting task after reviewing feature definitions and sample trajectories. For each model pair, we computed Kendall tau between each annotator’s rankings of the two models’ features, then averaged across annotators.

LLM Agent Workflow Optimization

LLM agent workflow optimization studies how to design and run multi-step agents for better task success, reliability, latency, and cost.

See all 243 papers

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs
Oct 15, 2026Kunlun Zhu, Tao Feng, Xu Han +5LLM Agent Harnesses LLM Agents
University of Illinois Urbana-Champaign · Princeton University · Google
Orla: A Library for Serving LLM-Based Multi-Agent Systems
Mar 13, 2026Rana Shahout, Hayder Tirmazi, Minlan Yu +1LLM Agent Orchestration LLM Agent Serving
Harvard University MA, USA · Boston University MA, USA
ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
Feb 14, 2026Hao Kang, Ziyang Li, Weili Xu +7LLM Agent Orchestration LLM Inference Scheduling
University of Illinois Urbana-Champaign · 5Together AI · 1Georgia Institute of Technology

Oct 15, 2026J/K move · Enter open · S save

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

Kunlun Zhu, Tao Feng, Xu Han, Heng Wang+4

University of Illinois Urbana-Champaign · Princeton University · Google

LLM Agent Harnesses LLM Agents Tool Selection for Language Model Agents LLM Decision-Making AI Agent Benchmarks LLM Agent Training+1

Open paper AskPDF arXiv

Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow and costly. Most of these calls end in a multiple-choice decision, such as which tool to call next. Decision models such as Jev answer these questions directly, at a fraction of an LLM call's time and price. We ask how a harness can delegate these decisions to a decision model with little loss of success. To evaluate decision models in agentic tasks, we build AgentDecisionBench, which tests them from isolated questions up to control of a running agent, on public benchmarks and our DecisionGym environment. The decision model most accurate on isolated questions solved only 69.5% of agent tasks, and without a fitted confidence threshold, delegating tool choices lowered success by 19.0 points. Building on these findings, we then design JevAgentHarness, which fits this threshold for each choice on the agent's own trajectories and lets the LLM decide below it. Because existing decision models never see the states an agent reaches after its own mistakes, we train AgentDecision-4B, an open 4B model, on such states with Jev's answers as labels. On tasks unlike its training data, the agent solved 91.5% of them with AgentDecision-4B and 82.0% with Jev. On tasks like their training data, 4B models trained this way made agents 81.1-89.0% cheaper and up to 2.5 times faster, without losing 3 or more points of task success. The models, data, and harness will be released at https://github.com/JevAgentHarness/JevAgentHarness.

Abstract

LLM Agent Workflow Optimization

Explore similar work

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

Orla: A Library for Serving LLM-Based Multi-Agent Systems

ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System