Agentic Data Cleaning Without a Clean Reference: An Experimental Study of Capabilities and Trade-offs

Hadi Fadlallah

cs.AIAug 14, 2026

Agentic Data Cleaning Without a Clean Reference: An Experimental Study of Capabilities and Trade-offs

Authors: Hadi Fadlallah

Organizations: Faculty of Arts and Sciences University of Sciences and Arts in Lebanon Beirut, Lebanon

Abstract

Data cleaning without a trusted clean reference is challenging because unusual values may represent either genuine errors or valid observations. This paper studies how different agent capabilities affect reference-free data cleaning and proposes an evidence-grounded framework that combines structured context, profiling, LLM reasoning, executable checks, controlled evidence retrieval, source ranking, citation alignment, conservative repair, reversible scripts, and provenance logging. Seven configurations are evaluated across financial, clinical, and environmental-monitoring datasets using controlled synthetic corruption and original-data descriptive analysis, resulting in 126 completed runs. The evaluation includes two comparison baselines and a progressive LLM-based sequence that adds executable tools, evidence retrieval, evidence controls, and conservative repair. In the synthetic evaluation, the deterministic profiling baseline achieved the highest detection F1-score of 0.561. Among the LLM-based configurations, the full conservative configuration achieved the highest F1-score of 0.421, but no configuration performed best across all evaluation criteria. The source-ranked configurations achieved the lowest unsupported-rule rates, while decision-level citation alignment remained weak. The full conservative configuration produced no unsafe or unnecessary modifications, although these rates were already zero before the conservative policy was added, and it performed no direct repairs. Overall, the results show that additional capabilities introduce trade-offs among detection, repair, evidence grounding, conservative behaviour, reproducibility, and operational cost rather than producing consistent improvements. The study provides a structured framework and empirical methodology for evaluating these trade-offs in reference-free agentic data cleaning.

Figures & tables

Taxonomy component	Framework role	Implementation status	Configurations and evaluation
Required outputs	Expose issues, requirements, rules, decisions, evidence, provenance, scripts, and audit summaries.	Logged in the output schema; capability-specific fields are absent when not enabled.	A0–A6; checked through counts, repair metrics, schema/JSON validity, reproducibility, and auditability.
Prior knowledge	Provide schema, unit, temporal, policy, purpose, and risk context that may support repair decisions.	Partly implemented through context profiles, schema/samples, profiling summaries, and risk descriptions; not separately ablated by knowledge type.	Context is available to different degrees across configurations, but individual knowledge types are not evaluated independently.
Tool capabilities	Delegate profiling, parsing, validation, temporal/constraint checks, and reversible scripts to tools.	Implemented for profiling/checking in A1 and for executable validation/script generation from A3 onward.	A1 tests deterministic profiling; A3–A6 test executable checks and reversible scripts; evaluated through detection, repair, execution, and operational metrics.
Evidence and citation controls	Distinguish evidence types, rank sources, and align cited claims with decisions.	Controlled local evidence retrieval is used in A4; source-ranking and citation-alignment checks are used in A5–A6.	A4 tests evidence availability; A5–A6 test ranking/alignment; evaluated through evidence coverage, unsupported-rule rate, citation alignment, runtime, and token use.
Decision classes and conservative policy	Convert suspected issues into repair, flagging, preservation, human review, or rejected repair.	Decision labels are logged across configurations; explicit conservative policy and provenance control are enabled most fully in A6.	A5–A6 isolate conservative control; evaluated through repair/safety metrics, valid-signal preservation, human-review rate, and reproducibility.
Human review and expert adjudication	Provide a high-authority pathway for ambiguous or high-risk cases.	Implemented only as a referral/decision class; no external expert adjudication of original-data cases was performed.	Mainly visible in A6 original-data behavior; reported descriptively through referral rate and discussed as a limitation/future-work need.

Table 1: Mapping between taxonomy components and the reported experimental implementation.

Output	Definition	Purpose
Issue inventory	Structured list of suspected quality issues, including affected columns, rows, entities, timestamps, and issue contexts.	Identifies what may be wrong.
Issue type	Classification of the issue, such as missing value, duplicate, invalid unit, impossible value, inconsistent category, temporal gap, entity mismatch, or suspicious anomaly.	Supports systematic analysis.
Evidence record	Dataset-derived or externally retrieved evidence used to support the issue or repair decision.	Reduces hallucinated cleaning.
Cleaning requirement	A requirement inferred from schema, profiling, documentation, standards, or authoritative sources.	Defines what the data should satisfy.
Validation rule	A machine-checkable rule derived from a cleaning requirement.	Enables repeatable validation.
Repair proposal	A suggested transformation, imputation, normalization, deletion, or correction.	Generates candidate repairs when safe.

Table 2: Required outputs of the evidence-grounded cleaning agent.

Knowledge Type	Definition	Domain-Independent Example
Schema knowledge	Column names, data types, key fields, expected structural constraints, and valid formats.	Identifier fields, timestamps, categorical codes, numeric measures.
Unit and scale knowledge	Meaning, unit, scale, and compatibility of measured values.	Currency units, medical units, sensor units, percentages, rates.
Temporal knowledge	Expected frequency, ordering, allowable time gaps, duplicate timestamps, and continuity constraints.	Hourly readings, admission/discharge dates, transaction sequences.
Entity knowledge	Entities represented in the dataset and valid relationships among them.	Patient ID, station ID, customer ID, product ID, location ID.
Domain-threshold knowledge	Expected ranges, warning limits, legal limits, operational thresholds, or plausible bounds.	Valid age range, sensor range, maximum transaction amount.
Contextual-event knowledge	External events or domain conditions that may explain unusual values.	Maintenance event, market shock, public-health event, environmental event.

Table 3: Taxonomy of prior knowledge required by an autonomous cleaning agent.

Tool Class	Function	Example Output
Profiling tools	Summarize types, missingness, uniqueness, distributions, outliers, patterns, and correlations.	Missingness rate, duplicate count, value ranges.
Parsing and type tools	Validate dates, numbers, codes, units, regular expressions, and encodings.	Invalid timestamp list.
Duplicate and entity-resolution tools	Detect exact duplicates, near duplicates, and possible entity conflicts.	Candidate duplicate clusters.
Temporal validation tools	Check ordering, gaps, repeated timestamps, rolling statistics, and frequency expectations.	Gap intervals and repeated measurements.
Constraint-checking tools	Execute functional, arithmetic, relational, and semantic constraints.	Rule violation table.
Unit and conversion tools	Validate unit compatibility and perform documented conversions.	Converted values with original unit preserved.

Table 4: Tool-capability taxonomy for evidence-grounded data cleaning.

Evidence Type	Definition	Typical Strength
Schema evidence	Evidence from names, declared types, primary keys, foreign keys, and constraints.	Medium to high.
Profile evidence	Evidence from distributions, missingness, duplicates, type errors, and statistical summaries.	Medium; weaker for semantic repair.
Executable evidence	Evidence from deterministic validation, parsing, rule execution, and reproducible checks.	High for syntactic and structural issues.
Dataset documentation	Official metadata, data dictionary, README, or provider documentation.	High when specific and current.
Institutional or regulatory source	Government, standards body, organizational documentation, or domain authority.	High when directly relevant.
Scientific literature	Peer-reviewed source providing domain thresholds, measurement interpretation, or methods.	Medium to high depending on specificity.

Table 5: Evidence taxonomy for cleaning decisions.

Decision Class	Definition	Example
Safe repair	The issue is clearly erroneous and the correction is directly supported by rules, documentation, executable checks, or strong evidence.	Standardizing timestamp format; replacing documented placeholder codes with null.
Conditional repair	A repair is plausible but depends on an explicit documented assumption.	Converting units when the source unit is strongly inferred but not directly stated.
Flag only	The value is suspicious but evidence is insufficient for modification.	Sudden spike, unusual transaction, or unexpected sensor value.
Preserve as possible valid signal	The value is unusual but may represent a meaningful rare event or domain signal.	Rare clinical result, extreme environmental reading, valid financial shock.
Needs human review	The issue is high-risk, ambiguous, or unresolved by available evidence.	Conflicting evidence or uncertain domain interpretation.
Reject unsupported repair	A repair proposal lacks sufficient support or violates policy.	Replacing an outlier only because it is statistically extreme.

Table 6: Cleaning decision classes used by the proposed framework.

Figure 1: Architecture of the evidence-grounded agentic data-cleaning framework.

Dataset	Domain	Cleaning Context	Main Context Requirements
German Credit	Finance	Financial decision support and credit-risk interpretation.	Schema and categorical-code interpretation, plausible financial ranges, sensitive attributes, target-label protection, and conservative handling of unusual but valid credit profiles.
EHR	Healthcare	Clinical data-quality assessment.	Patient/entity consistency, clinical attribute validation, missingness interpretation, categorical consistency, high-risk repair control, and human-review escalation.
Radiation	Environmental monitoring / IoT	Sensor monitoring and anomaly detection.	Timestamp parsing, temporal gaps, duplicate measurements, unit and scale interpretation, sensor anomalies, battery/status fields, and preservation of plausible radiation spikes.

Table 7: Datasets and context profiles used in the experimental study.

Dataset	Missing	Type	Range	Invalid	Invalid	Duplicate	Entity	Missing	Injected	Valid	Total
	value	error	violation	category	timestamp	row	duplicate	identifier	errors	signals	records
German Credit	10	5	5	5	0	5	0	0	30	5	35
EHR	10	5	5	5	0	5	2	1	33	5	38
Radiation	10	5	5	0	5	5	0	0	30	5	35

Table 8: Synthetic corruption and valid-signal protocol used in the completed experiment. Counts are per dataset and repeated run; the same ground-truth file for a given dataset and run was used across all A0–A6 configurations.

Corpus file	Records	Main contents
common.jsonl	2	Reference-free cleaning policy and provenance/reversibility policy shared across datasets.
german_credit.jsonl	5	German Credit dataset card, UCI documentation metadata, financial-domain cleaning policy, valid ranges, and valid-signal preservation policy.
ehr_dataset.jsonl	5	EHR dataset card, Kaggle documentation metadata, clinical safety policy, identifier policy, and categorical-value policy.
radiation_dataset.jsonl	5	Radiation dataset card, radiation-monitoring context, anomaly-preservation policy, sensor validity checks, and temporal-consistency policy.
Total	17	Fixed project-local evidence records used by A4–A6 in the reported experiment.

Table 9: Controlled local evidence corpus used in the completed experiment.

Metric Group	Metric	Definition
Detection	Precision	In the controlled synthetic-corruption setting, the fraction of detected issues that match injected error ground truth. Original-data precision is not interpreted because no injected or manually validated ground truth is available.
Detection	Recall	In the controlled synthetic-corruption setting, the fraction of injected errors detected by the agent. Original-data recall is not interpreted because no ground-truth error set is available.
Detection	F1-score	Harmonic mean of synthetic detection precision and recall.
Repair	Expected-repair match rate	Among repair decisions matched to injected errors for which repair is expected, the fraction whose proposed repair contains the known expected correction.
Repair	Safe repair rate	Fraction of repair-class decisions that match synthetic ground-truth cases for which repair is appropriate, excluding repairs applied to valid-signal or non-repair cases and repairs with no matching ground-truth issue.
Repair	Unsafe repair rate	Fraction of repair-class decisions applied to synthetic cases marked as valid signals or cases whose expected action is preservation, flagging, or human review.

Table 10: Evaluation metrics for reference-free evidence-grounded cleaning.

Configuration	Implemented description	Expected limitation
A0: Schema-only LLM baseline	LLM with schema and sample records only; no profiling summaries, executable checks, evidence retrieval, ranking/alignment, conservative policy, reversibility, or provenance logging.	May hallucinate requirements, miss data-derived patterns, or overinterpret values without profiling and executable evidence.
A1: Deterministic profiling-and-checking baseline	Deterministic profiling, missingness analysis, duplicate detection, type inference, range checks, and provenance logging; no LLM call, retrieval, ranking/alignment, conservative policy, or reversible scripts.	Can detect structural/statistical patterns efficiently, but may treat statistical anomalies as errors without semantic, contextual, or conservative reasoning.
A2: LLM with profiling summaries	LLM reasoning over profiling summaries with provenance logging; no executable Python validation, retrieval, ranking/alignment, conservative policy, or reversible scripts.	May support explanation and contextual reasoning, but still lacks executable validation and retrieved evidence support.
A3: LLM with profiling and Python tools	Adds executable Python checks for validation, parsing, duplicates, consistency, reversible-script generation, and provenance logging; no retrieval, ranking/alignment, or conservative policy.	Can validate candidate issues through executable checks, but may still decide without retrieved contextual evidence or source-ranking controls.
A4: Local-corpus evidence-retrieval configuration	Adds retrieval from the controlled local evidence corpus while keeping profiling, Python tools, reversible scripts, and provenance logging; no source ranking or citation-alignment checks.	Makes evidence records available, but retrieved evidence may not be strictly ranked, filtered, or aligned with final decisions.
A5: Source-ranked evidence configuration	Adds source ranking, filtering, evidence extraction, and citation-alignment checks on top of A4; the full conservative repair policy is not enabled.	Can reduce unsupported-rule behavior through evidence controls, but citation alignment may remain weak and repair behavior may still be insufficiently conservative.

Table 11: Comparison baselines and progressive ablation configurations.

Audit item	Verified value
Expected design	$3\times 2\times 7\times 3=126$ runs
Rows in final run-level metrics	126
Rows in final aggregate metrics	42
Successful final runs	126
Failed final runs	0
Runs requiring retry before success	4

Table 12: Experiment completion and postprocessing audit.

Config.	Configuration	Issues	Decisions	Prec.	Rec.	F1	Repair match	Safe repair
A0	Schema-only	3.222	3.222	0.074	0.007	0.013	0.000	0.000
A1	Profiling/checks	22.556	22.556	0.687	0.484	0.561	0.000	0.770
A2	LLM + profiling	9.444	6.111	0.727	0.200	0.307	0.000	0.000
A3	Python tools	10.111	5.333	0.884	0.267	0.402	0.028	0.444
A4	Python + controlled evidence	11.222	7.000	0.928	0.277	0.407	0.000	0.556
A5	Source-ranked evidence	12.667	4.667	0.804	0.264	0.376	0.000	0.222

Table 13: Controlled synthetic-corruption detection and repair metrics by configuration.

Panel of Figure 2 — Figure 2: Synthetic detection F1-score across the seven evaluated configurations. A0 and A1 are comparison baselines, while A2–A6 represent the progressive LLM-based capability sequence.

Config.	Configuration	Unsafe repair	Unnecessary modification	Valid-signal preservation
A0	Schema-only	0.000	0.222	0.000
A1	Profiling/checks	0.064	0.167	1.000
A2	LLM + profiling	0.000	0.111	0.556
A3	Python tools	0.000	0.000	0.000
A4	Python + controlled evidence	0.000	0.000	0.000
A5	Source-ranked evidence	0.000	0.000	0.000

Table 14: Controlled synthetic-corruption safety and valid-signal metrics by configuration.

Config.	Configuration	Issues	Decisions	Evidence cov.	Unsupported rules	Conserv. flag	Human review
A0	Schema-only	3.778	3.778	0.089	0.667	0.653	0.093
A1	Profiling/checks	10.000	10.000	0.627	0.040	0.453	0.000
A2	LLM + profiling	5.222	2.667	0.236	0.167	0.444	0.111
A3	Python tools	3.556	3.556	0.548	0.074	0.644	0.000
A4	Python + controlled evidence	6.444	3.111	0.469	0.028	0.542	0.000
A5	Source-ranked evidence	4.000	2.222	0.356	0.111	0.444	0.022

Table 15: Original-data descriptive behavior and evidence indicators by configuration. Values are means over nine original-data runs per configuration: three datasets and three repeated runs.

Config.	Configuration	Runtime (s)	Total tokens	JSON repair
A0	Schema-only	25.209	4841.444	0.222
A1	Profiling/checks	0.081	0.000	0.000
A2	LLM + profiling	52.616	9346.222	0.556
A3	Python tools	61.958	9392.444	0.111
A4	Python + controlled evidence	43.521	9918.111	0.333
A5	Source-ranked evidence	34.356	10132.889	0.222

Table 16: Original-data operational cost and structured-output repair by configuration. Values are means over nine original-data runs per configuration.

Panel of Figure 3 — Figure 3: Configuration-level trade-off between synthetic detection performance and mean runtime across all 18 runs per configuration. Filled squares denote the comparison baselines, while open circles denote the progressive LLM-based sequence.

Data Analysis Agents

Data analysis agents use language models and tools to explore datasets, run analyses, and produce answers or reports grounded in evidence.

See all 106 papers

Jul 28, 2026

SARC-DQ: Runtime Data-Quality Gating for Agentic AI: Silent Evidence Defects, the Incompetence Shield, and Downstream-Only Remediation

Agentic systems act, so a defect in the evidence they retrieve becomes a wrong action with a currency cost. The most dangerous enterprise defects are metadata-borne: a stale price or a superseded record, perfectly well-formed in the payload and betrayed only by freshness, lineage, or provenance. Such a defect never enters the agent's context, and an agent cannot doubt data it cannot see. On a priced replenishment benchmark, a competent agent silently converts an injected metadata-borne defect into a costly action about 60% of the time, with zero data-quality flags and behavioral doubt markers at chance (AUC <= 0.50). Across four model tiers spanning roughly 15x in inference price, the rate stays flat: capability does not buy skepticism. A metadata-aware pre-action gate with downstream-only remediation recovers the loss fully on the signals its predicates cover and not at all on those they miss. A model-free oracle derived from the task's decision geometry tracks the measured rates with MAE 0.015 (Pearson r = 0.876, interval coverage 15/16 cells), giving the flat ladder an analytical form. Evidence integrity is a systems axis distinct from model capability; mitigation depends on enforcement placement and predicate coverage. Code, frozen results, and a deterministic analysis pipeline: https://github.com/besanson/dqSarc

Gaston Besanson

Universidad Torcuato Di Tella

AI Agent Reliability Data Provenance AI Agent Monitoring LLM Agent Reliability

Ask PDF

Jun 1, 2026

When Helping Hurts and How to Fix It: Multi-Agent Debate for Data Cleaning

When does multi-agent debate help data cleaning, and when does it hurt? Across three benchmarks, four model families, and over 6,000 task-condition pairs, we find debate's effect reverses sign: it degrades generation across all four models (-1.6 to -15.5pp) through critique-induced confusion (CIC), hallucinated Critic feedback that the Generator accepts uncritically, yet improves error detection (+27.4pp F1, d=1.0). We derive a debate benefit condition: debate helps when the probability of rescuing a wrong output (Critic verification odds weighted by fixability) exceeds the probability of destroying a correct one. A factorial experiment proves adversarial separation is essential: self-verification with identical tools fails, while a separate Critic with code-execution grounding and evidence-gated generation produces the first debate configuration to significantly exceed single-agent on a generative task (+5.3pp, p<0.05). The condition correctly predicts all nine task types and generalizes with zero false positives across 19 published comparisons in seven domains.

Chirag Parmar, Akshat Mehta, Henglin Wu +2

1Meta Platforms, Inc.

Multi-Agent Debate

Ask PDF

Sep 29, 2026

When Tools Silently Lie: Evaluating and Mitigating Blind Compliance in Tool-Augmented Data Agents

Tool-augmented data agents rely on tool outputs for analytical decisions. Yet successful execution can return plausible but incorrect evidence, requiring agents to decide whether to trust or verify it. Understanding this failure requires examining both the evidence obtained through checking and the answer ultimately adopted. We introduce ToxicBench to measure checking and adoption under numerical, label, schema, and retrieval errors, pairing clean and poisoned observations over fixed source data. In the 118-task GPT evaluation across three adapters, poisoning lowers task success by 26 to 39 percentage points. Ordinary retries help under one-shot poisoning, whereas repeated poisoning reveals wrong-answer adoption after checking. Controls on three public tables isolate how supplied evidence affects recovery. After freezing the scorer, we compare its judgments with human annotations on 200 trajectories, finding 96% task-success agreement. Human judgments support retry gains over Base and confirm adoption after checking on audited tasks. We release trajectories, versioned scoring, and reference and delivery audits. These findings highlight evidence availability and answer selection as complementary dimensions of agent reliability.

Zifu Tao, Changqing Yin

Tongji University

AI Agent Reliability Tool-Use Evaluation AI Agent Benchmarks Tool-Using Agents LLM Agent Reliability

Ask PDF

Abstract

Data Analysis Agents

Explore similar work

SARC-DQ: Runtime Data-Quality Gating for Agentic AI: Silent Evidence Defects, the Incompetence Shield, and Downstream-Only Remediation

When Helping Hurts and How to Fix It: Multi-Agent Debate for Data Cleaning

When Tools Silently Lie: Evaluating and Mitigating Blind Compliance in Tool-Augmented Data Agents