Unsupervised Tool Evolution for Coding Agents in Data-Lake Analytics

Yuqing Yang; Yurong Liu; Deqing Fu; Manav Garg; Sravan Babu Bodapati; Fatma Ozcan; Alon Y. Halevy; Jiani Zhang

cs.AIOct 19, 2026

Unsupervised Tool Evolution for Coding Agents in Data-Lake Analytics

Authors: Yuqing Yang, Yurong Liu, Deqing Fu, Manav Garg, Sravan Babu Bodapati, Fatma Ozcan, Alon Y. Halevy, Jiani Zhang

Organizations: University of Southern California · Google · New York University

Abstract

Coding agents can answer analytical questions over a data lake by discovering, cleaning, and joining the relevant data. However, they discard this work after each question and redo it from scratch for the next one. Tool evolution, which automatically builds and refines reusable tools across tasks, is a natural fit for this repetition. Since a ground-truth label for a data-lake question requires an expert to solve it first, we consider the unsupervised setting. We propose TEAL (Tool Evolution for Analytics over data Lakes), which evolves a library of tools and usage skills for a given Worker agent without labels. Two agents propose tools from complementary evidence, an Explorer from the data lake itself and a Critic from Worker trajectories. A proposed change is kept only if the Worker adopts the tool on a rerun without extra steps and any changed answer is supported by trajectory evidence. We also introduce TEAL-Q, which generates synthetic questions for tool evolution when real ones are unavailable. We evaluate on KramaBench and LakeQA with four Worker LLMs under step budgets, which cap the number of steps the Worker may take per question. With a tight budget of 10 or 15 steps, TEAL improves every Worker on both benchmarks, lifting Sonnet-5 at 15 steps by 9.3 points on KramaBench and 21.7 points on LakeQA. A tighter budget also yields better feedback for evolution. For example, Sonnet-5's LakeQA library evolved at 20 steps and deployed unchanged at 30 steps scores 68.3, above both the baseline (59.6) and a library evolved directly at 30 steps (64.2). Additionally, a library evolved on synthetic questions performs comparably to one evolved on real questions. These results show that useful tools can be evolved without labels, and hopefully inspire future work on what to evolve and how to obtain feedback in unsupervised agent evolution.

Figures & tables

Figure 1: The TEAL evolution loop. The Worker first solves the questions with the current library (Stage 0). The Explorer proposes tools for the data lake as a whole (Stage 1), and the Critic proposes targeted fixes from the Worker’s trajectories (Stage 2). The Skill Writer then updates the usage guidance (Stage 3), and a rerun of the Worker decides which changes to keep (Stage 4). Robot icons mark LLM roles. No stage uses ground-truth answers.

Figure 2: Synthetic question generation with TEAL-Q . Given the data lake and a small set of seed questions as demonstrations, TEAL-Q generates each question in four stages.

Figure 3: Accuracy versus Worker step budget in the real-question scenario. Rows correspond to benchmarks (top: KramaBench, bottom: LakeQA) and columns to Worker models. The horizontal axis is the step budget

k

at test time, from 10 to 30 steps. The grey line is the baseline Worker without evolved tools. The blue line is TEAL with a library evolved and tested at the same budget

k

. The orange dashed line is a frozen library: it is evolved once at the ringed budget and then tested unchanged at each looser budget.

KramaBench
	$k{=}15$	$k{=}30$
DS-Guru	39.73
Smolagents	46.69
[2pt/2pt] Baseline	$63.23_{\pm 2.57}$	$75.71_{\pm 0.70}$
ISE	$66.81_{\pm 5.96}$	–
TEAL	$72.54_{\pm 1.59}$	$\textbf{78.53}_{\pm 0.65}{}^{\dagger}$

Table 1: Accuracy (%) of existing methods and TEAL , Sonnet-5 as the Worker. † Library evolved at 15 (KramaBench) or 20 (LakeQA) steps, deployed unchanged (orange lines in Figure 3 ).

	KramaBench		LakeQA
	Acc.	$\Delta$	Acc.	$\Delta$
Baseline	$63.23_{\pm 2.57}$	–	$35.83_{\pm 1.91}$	–
TEAL (real questions)	$72.54_{\pm 1.59}$	+9.31	$57.50_{\pm 2.50}$	+21.67
TEAL (synthetic questions)	$72.18_{\pm 3.22}$	+8.95	$56.25_{\pm 3.31}$	+20.42

Table 2: Accuracy (%) of TEAL evolved on real or synthetic questions with Sonnet-5 as the Worker. All configurations use a 15-step budget and the same held-out test sets.

	KramaBench
	Acc.	$\Delta$
Baseline	$63.23_{\pm 2.57}$	–
TEAL	$72.54_{\pm 1.59}$	+9.31
$-$ Explorer	$71.59_{\pm 2.28}$	+8.36
$-$ Evolved skills	$68.69_{\pm 0.85}$	+5.46
[2pt/2pt] $+$ Ground-truth answers		$72.73_{\pm 3.07}$	+9.50

Table 3: Accuracy (%) of ablations and reference configurations with Sonnet-5 as the Worker at a 15-step budget, in the real-question scenario.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Coverage	Role	What it does
resolve_concept_to_datasets	55.3%	concept lookup	Maps a natural-language concept or a few keywords to the lake files that hold the corresponding measurements.
locate_field_in_lake	39.6%	field search	Finds every file, sheet, and column in which a query string appears as a header or a cell value, including inside Excel workbooks that grep cannot read.
catalog_data_lake	37.7%	enumeration	Lists the files under a lake root, following symlinks and optionally filtered by substring, glob, or regex, with a one-line schema per file.
load_report_csv	36.5%	CSV loading	Loads one or more report-style CSVs in one call, detecting stacked data sections separated by title rows and blank rows.
inspect_excel_workbook	13.8%	workbook loading	Lists every sheet of a workbook with its shape and detected header row, or loads one sheet as a clean typed table.

Appendix

Table 4: The five most used tools in the KramaBench library evolved with Gemini-3.1-Pro as the Worker under a 10-step budget. Coverage (%) is the share of the 159 test episodes (53 tasks, three repeats) that call the tool at least once.

Tool	Coverage	Role	What it does
query_lake_catalog	97.1%	metadata lookup	Answers where a dataset is, which columns and format it has, how big it is, what its known traps are, or which sibling files it has, from a pre-built SQLite catalog of the lake.
aggregate_over_files	82.1%	streaming compute	Runs one filter, count, distinct, projection, group-by, or aggregate over a large delimited file or a union of sibling files through DuckDB, without loading the file into memory.
preview_candidate_columns	40.4%	schema preview	Shows the columns of a set of candidate files given as paths, a glob, or a dataset slug.
resolve_filter_values	32.9%	value matching	Maps a fuzzy category term to the exact distinct values stored in a column, with the row count of each value.
search_lake_content	30.8%	full-text search	Finds which dataset, file, or column contains a keyword, phrase, or value through a pre-built full-text index over dataset names and column headers.

Appendix

Table 5: The five most used tools in the LakeQA library evolved with Gemini-3.1-Pro as the Worker under a 20-step budget. Coverage (%) is the share of the 240 test episodes (80 tasks, three repeats) that call the tool at least once.

Figure 4: Accuracy versus mean Worker steps per test episode.

Worker	Uncached input	Cached input	Output
Sonnet-5	2.50	0.20	10.00
Opus-4.8	6.25	0.50	25.00
Gemini-3.1-Pro	2.00	0.20	12.00
Gemini-3.8-Flash	0.75	0.075	3.75

Appendix

Table 6: Vertex AI list prices (USD per million tokens) used to estimate the cost of a test episode.

Figure 5: Accuracy versus estimated inference cost per test episode.

Model	Methods	KramaBench		LakeQA
Model	Methods	Acc.	$\Delta$	Acc.	$\Delta$
Sonnet-5	Baseline	$63.23_{\pm 2.57}$	–	$35.83_{\pm 1.91}$	–
	TEAL (real questions)	$72.54_{\pm 1.59}$	$+9.31$	$57.50_{\pm 2.50}$	$+21.67$
	TEAL (synthetic questions)	$72.18_{\pm 3.22}$	$+8.95$	$56.25_{\pm 3.31}$	$+20.42$
Opus-4.8	Baseline	$70.44_{\pm 1.90}$	–	$50.00_{\pm 2.17}$	–
	TEAL (real questions)	$75.55_{\pm 3.62}$	$+5.11$	$55.00_{\pm 1.25}$	$+5.00$

Appendix

Table 7: Accuracy (%) of TEAL evolved on real or synthetic questions, compared with the same Worker without evolved tools. All configurations use a 15-step budget and the same held-out test tasks.

AI Coding Agents

AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.

See all 327 papers

SANA: What Matters for QA Agents over Massive Data Lakes?
Jun 11, 2026Austin Senna Wijaya, Jiaxiang Liu, Haonan Wang +1LLM Agent Evaluation Question Answering
Columbia University New York, USA
"Skill Issues'': Data-Centric Optimization of Lakehouse Agents
May 31, 2026Nicole Rose Schneider, Davide Ghilardi, Giacomo Piccinini +1AI Coding Agents LLM Agent Skill Learning
University of Maryland USA · Università Milano Bicocca Italy · Bauplan Labs USA
Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning
Nov 20, 2025Peng Xia, Kaide Zeng, Jiaqi Liu +5LLM Agent Self-Improvement Multi-Agent LLMs
UNC-Chapel Hill · Salesforce Research · Stanford University

Jun 11, 2026J/K move · Enter open · S save

SANA: What Matters for QA Agents over Massive Data Lakes?

Austin Senna Wijaya, Jiaxiang Liu, Haonan Wang, Eugene Wu

Columbia University New York, USA

LLM Agent Evaluation Question Answering AI Agent Benchmarks

Open paper AskPDF arXiv

Exploratory question answering (EQA) over data lakes requires an LLM agent to discover relevant sources, analyze retrieved data, and adapt its actions based on intermediate results. End-to-end accuracy alone cannot distinguish failures in search, planning, data analysis, or the agent's Action Policy: its decisions about what to do next and when to submit an answer. We present SANA (Search Agent Navigation Ablation framework), a diagnostic ablation framework that transforms EQA tasks into runtime profiles containing gold source sequence, sanitized subquestions, and execution records. SANA uses these profiles to construct idealized search, planning, and data-analysis tools, allowing each component to be ablated; the residual gap is diagnostic evidence for policy failures. To illustrate SANA as a reusable evaluation framework, we adapted two recent EQA benchmarks, LakeQA and KramaBench, and evaluated lightweight and mid-sized agents under fixed prompts, budgets, data lakes, and runtimes. Across both benchmarks, data analysis is a consistent bottleneck while planning is less so. Search is a major limitation in LakeQA's large data-lake setting, but less so for the smaller-scale KramaBench. SANA thus deconstructs end-to-end task accuracies into a diagnosis of where data-lake agents fail, and allows for systematic comparisons of progress in search, planning, data analysis, and agent design.

Abstract

AI Coding Agents

Explore similar work

SANA: What Matters for QA Agents over Massive Data Lakes?

"Skill Issues'': Data-Centric Optimization of Lakehouse Agents

Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning