Semantic Atoms for Scalable Text Analysis in Data Analytics in the Era of LLM-Based Agents

Jinxiang Xie; Zihao Li; Rui Ding; Bartuer Zhou; Shi Han; Dongmei Zhang

cs.CLOct 26, 2026

Semantic Atoms for Scalable Text Analysis in Data Analytics in the Era of LLM-Based Agents

Authors: Jinxiang Xie, Zihao Li, Rui Ding, Bartuer Zhou, Shi Han, Dongmei Zhang

Organizations: Nanjing University · Tsinghua University · Microsoft Research

Abstract

Much of the information needed for data analysis is buried in text columns: an analyst may infer customer frustration from complaint text and use it to explain resolution time. We call workloads in which processing text columns is central to subsequent data analysis TADA (Text Analysis in Data Analytics). In an agentic setting, the user's query dynamically specifies this processing, and answering it interleaves LLM-based semantic steps with exact SQL or Python operations. This mixed execution raises two design questions: which work should be neural and which symbolic, and which semantic outputs later steps can reuse. We address them for agent-planned TADA workflows with two complementary, query-driven semantic atoms. Each atom takes a sequence of text items and a natural-language query specifying the semantic task: Tagging returns one string per input item, while Categorization returns $M$ category objects over $N$ input items. Tagging results are materialized as a row-aligned column and Categorization results as a category table; SQL or Python then performs exact computation over this exposed state. We realize this interface in TADA-Agent, whose atoms run inside generated Python programs and internally handle inputs beyond a reliable context window. We evaluate TADA-Agent on TABench, a benchmark of 40 complete analysis requests over spreadsheet tables spanning workflow composition, input--output structure, and literature-grounded analytical scenarios. Against a semantic-operator baseline and relative to a held-out TADA reference run, TADA-Agent delivers comparable row-level semantic quality, has higher complete-artifact preservation estimates, and uses over an order of magnitude fewer semantic-model requests under the evaluated native execution paths.

Figures & tables

Representative request	Scenario	Input–output	TADA workflow	Existing operator
Workflows corresponding to familiar semantic-operator families
Label each review as positive, negative, or neutral	Classification	$N{\times}1\to N{\times}1$	Tag with the three labels written in the query	sem_map
Discover issue categories and group the reviews by them	Grouping; classification	$N{\times}1\to(M,N{\times}1)$	Categorize $\to$ build a query from the category names $\to$ Tag $\to$ GroupBy	Semantic grouping
Keep comments that imply a risk of customer churn	Semantic filtering	$N{\times}1\to N^{\prime}{\times}1$	Tag a Yes / No decision $\to$ Filter	sem_filter
Extract time, location, scope, and cause from each incident	Information extraction	$N{\times}1\to N{\times}w$	Invoke Tagging $w$ times, once for each output column	sem_map / sem_extract
Produce one overall summary of the survey responses	Text summarization	$N{\times}1\to 1$	Categorize with a query requesting one overall summary	sem_agg

Table 1: Representative TADA requests viewed by scenario, input–output structure, workflow composition, and correspondence to existing semantic operators. Scenario names follow the fine-grained intent taxonomy; the appendix groups these leaves into 12 broader reporting classes.

Scenario class	TADA-Agent	Semantic operators	Claude Code	Code-only	Direct
Classification	$\mathbf{98.6}$ [97.0, 100.0]	$97.5$ [96.1, 98.8]	$97.9$ [95.2, 100.0]	$96.6$ [94.0, 98.8]	$94.7$ [91.3, 98.0]
Field extraction	$\mathbf{98.9}$ [96.6, 99.9]	$98.2$ [95.5, 99.7]	$97.9$ [93.9, 99.7]	$90.6$ [80.3, 98.4]	$82.5$ [72.2, 95.0]
Semantic ranking (Spearman $\rho$ )	$\mathbf{0.801}$ [0.600, 0.980]	$0.752$ [0.483, 0.957]	$0.774$ [0.558, 0.962]	$0.590$ [0.249, 0.921]	—
Constrained generation	$\mathbf{89.1}$ [83.8, 94.0]	$78.6$ [64.2, 88.8]	$84.2$ [74.8, 90.6]	$35.0$ [27.5, 42.8]	$60.3$ [55.2, 64.0]

Table 2: Row-level semantic quality on TABench by scenario class. Except for the semantic-ranking row, each entry is the percentage of evaluated units judged at least as good as the common reference output, with a 95% case–run cluster-bootstrap interval over three independent runs per configuration (identical inputs, models, and judge). The semantic-ranking row reports mean within-case Spearman

\rho

against the same reference, macro-averaged over the four ranking cases and three runs, with a case–run cluster-bootstrap 95% interval computed the same way (wide because only four cases enter this row); the complementary absolute-score agreement is reported in the text, and Direct is omitted because its incomplete/malformed Q34 artifacts yield method-dependent row coverage. Bold marks the highest point estimate and does not imply statistical significance; classes are never aggregated.

Scenario class	TADA-Agent	Semantic operators	Claude Code	Code-only	Direct
Grouping and clustering	$\mathbf{8/9}$ [56.5, 98.0]	$5/9$ [26.7, 81.1]	$\mathbf{8/9}$ [56.5, 98.0]	$6/9$ [35.4, 87.9]	$5/9$ [26.7, 81.1]
Semantic filtering	$\mathbf{9/12}$ [46.8, 91.1]	$8/12$ [39.1, 86.2]	$6/12$ [25.4, 74.6]	$4/12$ [13.8, 60.9]	$0/12$ [0.0, 24.2]
Aggregation and summarization	$\mathbf{12/12}$ [75.8, 100.0]	$6/12$ [25.4, 74.6]	$\mathbf{12/12}$ [75.8, 100.0]	$11/12$ [64.6, 98.5]	$\mathbf{12/12}$ [75.8, 100.0]
Numerical and temporal analysis	$\mathbf{6/9}$ [35.4, 87.9]	$4/9$ [18.9, 73.3]	$\mathbf{6/9}$ [35.4, 87.9]	$\mathbf{6/9}$ [35.4, 87.9]	$1/9$ [2.0, 43.5]
Anomaly detection	$\mathbf{6/6}$ [61.0, 100.0]	$4/6$ [30.0, 90.3]	$3/6$ [18.8, 81.2]	$5/6$ [43.6, 97.0]	$\mathbf{6/6}$ [61.0, 100.0]

Table 3: Complete-artifact preservation on TABench. Each entry counts outputs judged at least as complete and correct as the fixed reference, with descriptive 95% Wilson intervals in percent. The unit is a delivered artifact rather than a row; functionally determined identity columns do not count as missing content. These results are specific to this reference output and are not combined with Table 2 .

Metric	TADA-Agent	Sem. operators
Task-runs	120	120
Successful requests	451	$\geq$ 18,852
Mean calls per task-run	3.76	$\geq$ 157.10
Relative count	1.0 $\times$	$\geq$ 41.8 $\times$

Table 4: Recorded calls to the dedicated semantic-processing LLM across three runs of each TABench task. “Mean per task-run” is the mean number of such LLM calls, not elapsed time. The

\geq

symbol marks conservatively incomplete request logs.

Workload	System	Accuracy	Macro-F1	Acc./agreement
Sentiment 32,179 rows	TADA-Agent	$\mathbf{0.973{\scriptstyle\pm 0.000}}$	$\mathbf{0.928}$	$\mathbf{1.017}$
	Claude Code	$0.817{\scriptstyle\pm 0.047}$	$0.577$	$0.854$
	LOTUS †	$0.954$	$0.884$	$0.997$
Topic assign. 32,179 rows	TADA-Agent	$\mathbf{0.942{\scriptstyle\pm 0.002}}$	$\mathbf{0.838}$	$\mathbf{1.038}$
	Claude Code	$0.856{\scriptstyle\pm 0.116}$	$0.729$	$0.943$
	LOTUS †	$0.881$	$0.838$	$0.970$

Table 5: Large-scale end-to-end execution. Accuracy is agreement with the majority-vote labels. Acc./agreement divides accuracy by inter-annotator agreement (1.0 means they match). TADA-Agent and Claude Code report mean

\pm

s.d. over three full-scale runs; LOTUS rows are measured subsets: 500 rows for the 32K workloads and the first 2,050 rows for the 6.6K workload (

\dagger

). Macro-F1 and Acc./agreement are also means across valid full-scale repetitions; only Accuracy displays its across-run s.d.

Figure 3: Probability of recovering the true global top-5 categories over 5,000 random partitions of each fixed 32K-row population as the local retention budget varies from

r=5

to 20. The dashed line is the 0.95 target; outlined circles mark pilot-selected values. The blue separated distribution needs no extra candidates; the five orange curves progressively narrow the cutoff gap.

Figure 4: Public document-extraction benchmarks (higher is better). (a) Reproduced CUAD average F1; TADA-Agent, DocETL, and LOTUS rerun with the same GPT-5.5 model and evaluator. (b) Reproduced BioDEX report precision at 5 and 10 predictions (RP@5/10) and recall of gold adverse-event terms under the same model, candidate budget, and official evaluator; bars are means over completed runs, and error bars show the observed min–max where repeats exist. Provenance caveats are stated in the text.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Atom	Input	Raw return	Materialization	Large-scale path
Tagging	Text sequence $X$ and atom-level query $q$	One string $s_{i}$ for each $x_{i}$ , preserving length and order	One row-aligned column associated with input rid values	Independent: partition and reassemble; joint: discover a shared category list, then use the independent path
Categorization	Text sequence $X$ and atom-level query $q$	Category objects with a required name and optional description	One category table with fresh cid values	Partition, retain local candidates, semantically consolidate, and select the requested global list

Appendix

Table 6: Public atom contracts and their materialized analytical state.

$N$	Monolithic sort	Monolithic filter F1	Monolithic agg.	Atom+symbolic
100	79/100	1.000	No	Exact
200	161/200	0.765	No	Exact
500	280/500	0.678	No	Exact
1000	400/1000	0.105	No	Exact

Appendix

Table 7: Single-run neural–symbolic boundary diagnostic. Sort coverage reports returned identifiers over requested rows; aggregation reports exact agreement on count, sum, and average.

ID	Scenario class	Cases	Dominant program structure
S1	Row-wise classification	Q01–Q03, Q27, Q28	Tag typed labels against a fixed or inferred label set (one string value per row per call).
S2	Open grouping & clustering	Q04, Q05, Q36	Categorize a corpus-level taxonomy, optionally Tag rows with its identifiers; variants add outlier isolation.
S3	Semantic filtering	Q06–Q08, Q29, Q38	Tag a predicate per row, then deterministic row selection; includes negated, implicit, and alias-sensitive predicates.
S4	Field extraction	Q09, Q10, Q30, Q39, Q40	Tag typed fields per row from free text; repeated per output column; includes open-set and relational attributions.
S6	Aggregation & summarization	Q13–Q15, Q31	Group rows deterministically, Categorize or summarize per group, compare across groups or time.
S7	Numerical/temporal analysis	Q16, Q17, Q32, Q42	Combine semantic state with numeric filtering, aggregation, or multi-column fusion.

Appendix

Table 8: TABench task inventory by scenario class. Dominant program structure describes the typical composition; a query may exercise more than one class’s mechanics. S5 and S8 are boundary classes outside the two-atom contract and are discussed rather than scored.

	Units	Maj.	Agree	Flip
By stratum
Contested	21	19	17	1
Low-confidence	12	8	4	1
High-confidence	6	6	5	0
By task type
Summarization (Q24)	8	8	8	0

Appendix

Table 9: Human calibration of the LLM judge on 39 stratified judgment units, six annotators. Maj. = units with a human majority verdict; Agree = judge matches human majority; Flip = directional conflict (one side A, the other B).

TADA-Agent (provider-reported)
Workload	Rep	Calls	Prompt	Compl.	Cache	Wall
Sentiment	1	33 (105)	–	–	–	164.6
Sentiment	2	33	891.7	349.2	–	15.1
Sentiment	3	33	891.7	259.9	–	12.2
Topic	1	33	894.7	623.5	–	29.2
Topic	2	33	894.7	674.5	–	31.8

Appendix

Table 10: Per-repetition execution cost. Calls: planned LLM calls for TADA-Agent (observed attempts in parentheses where they differ), API messages for Claude Code, per-row calls for LOTUS. Tokens in thousands; “cache” is Claude Code prompt-cache reads. Wall time in minutes.

\ast

: transcript token fields incomplete; –: not recorded or not aggregable.

Workload / rep	Discovered	Unmapped	Mapped acc.
Topic 32K / 1	10	4	0.690
Topic 32K / 2	10	3	0.675
Topic 32K / 3	10	7	0.685
Category 6.6K / 1	12	8	0.296
Category 6.6K / 2	11	6	0.134
Category 6.6K / 3	10	4	0.195

Appendix

Table 11: Open-vocabulary repetitions: discovered vocabulary size, labels with no reference correspondence, and mapped accuracy.

Regime	$p_{5}-p_{6}$	Oracle $r^{\star}$	$\widehat{r}$	$\widehat{Q}_{.05}$	Rec. at $r=5$	Rec. at $\widehat{r}$
Separated	0.061891	5	5	1.00	1.000	1.000
Narrow 1	0.002293	9	10	0.90	0.909	1.000
Narrow 2	0.001963	9	10	0.95	0.871	1.000
Narrow 3	0.001634	9	10	0.95	0.824	1.000
Narrow 4	0.001306	10	10	0.95	0.744	0.990
Narrow 5	0.001175	10	10	0.95	0.739	0.985

Appendix

Table 12: Planner position relative to the full

r=5,\ldots,20

recovery matrix. The oracle is the first

r

whose recovery and every larger tested

r

remain at least 0.95. Empirical recovery uses 5,000 partitions of each fixed population.

Data Analysis Agents

Data analysis agents use language models and tools to explore datasets, run analyses, and produce answers or reports grounded in evidence.

See all 106 papers

Fail Loudly: An Auditable Runtime for Agentic Data Analysis
Sep 26, 2026Hanxu Yan, Langxuan Deng, Zhengle Wang +2Runtime Enforcement for AI Agents Data Analysis Agents
Independent Researcher · Purdue University
Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agents
Sep 6, 2026Qing YeText-to-SQL Data Analysis Agents
Independent Researcher
PExA: Parallel Exploration Agent for Complex Text-to-SQL
Apr 24, 2026Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang +3LLM Inference Efficiency Text-to-SQL
University of California Los Angeles · Bloomberg

Sep 26, 2026J/K move · Enter open · S save

Fail Loudly: An Auditable Runtime for Agentic Data Analysis

Hanxu Yan, Langxuan Deng, Zhengle Wang, Yibo Wang+1

Independent Researcher · Purdue University

Runtime Enforcement for AI Agents Data Analysis Agents LLM Agent Reliability AI Agent Auditing

Open paper AskPDF arXiv

Large language models (LLMs) have enabled data-science agents to automate multi-step analyses over heterogeneous files. However, incorrect choices regarding data sources, scope, or statistical definitions often lead to silent errors: computations execute successfully but produce plausible yet incorrect outputs that fail to answer the intended question. To mitigate this, we present RADAR, an auditable runtime that makes an agent's analytical choices inspectable and supports their revision through execution feedback. RADAR operates through three core mechanisms. First, an evidence-preserving exploration module retrieves task-relevant content while retaining source locations and observation coverage. Next, the runtime uses typed operators to record the agent's declared inputs, operation arguments, and resulting observations. Finally, runtime validation checks proposed operations against these observations. When a conflict is detected, the runtime rejects the operation or provides diagnostic feedback, allowing the agent to revise its choices before errors propagate. This design enables agents to fail loudly while leaving semantic interpretation to the LLM. On KramaBench, RADAR achieves overall scores of 0.723 with full source retrieval and 0.747 with gold sources supplied, corresponding to relative gains of 35.9% and 28.8% over the strongest baselines. Beyond KramaBench, RADAR achieves relative performance gains of 14.0% on DA-Code and 59.3% on DABStep, demonstrating its applicability across diverse agentic data-analysis workflows.

Abstract

Data Analysis Agents

Explore similar work

Fail Loudly: An Auditable Runtime for Agentic Data Analysis

Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agents

PExA: Parallel Exploration Agent for Complex Text-to-SQL