Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2

Marek Šuppa; Ivan Vykopal; Jaroslav Kopčan

cs.CLOct 22, 2026

Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2

Authors: Marek Šuppa, Ivan Vykopal, Jaroslav Kopčan

Organizations: Comenius University in Bratislava · Cisco Systems · Kempelen Institute of Intelligent Technologies (KInIT) · Brno University of Technology

Abstract

Multilingual pipelines such as FineWeb2 apply one recipe across a thousand languages, leaving the failure modes of individual languages unanticipated. We use an LLM once, as a corpus debugger rather than a classifier: a tool-using agent explores the corpus through a read-only, budgeted API and compiles each finding into an executable test, which a native speaker then validates. On the 26.5M-document Slovak portion of FineWeb2, three runs costing $2.21-$2.84 each found seven findings over four phenomena. The largest is a machine-generated content farm of 1,408,437 documents (5.32%) that passed the pipeline's filters, 86.6% of it passing even stricter thresholds. Blinded labels support 6 of the 7 findings (defect rates .53-1.00 against .08 in a random control) and do not support one. The compiled suite scans the full corpus in 0.7 s of wall-clock time without a GPU, against $12k-$90k per LLM classification pass. The best static sampling baseline validates more selectors than any agent run, at under half the cost. The agent contributes exact prevalence, a precision estimate checked against sampled documents, an account of why the pipeline missed each slice, and a reusable test. We release the framework, Slovak test suite, validation data and agent trajectories.

Figures & tables

Panel of Figure 1 — Figure 1: The three stages. Every label is human. Verdicts are proposed by a model from those labels and adopted by the annotator, and only an adopted verdict assigns a status. Deployment uses no model.

Path	Contents
CODE
src/corpusaudit/	adapter (DuckDB over Parquet), the eleven tools as an MCP server, test runner, patcher
prompts/	the agent and baseline prompts
tests/	the suite: 7 Slovak slice guards, 2 metadata invariants
DATA & RUNS
out/runs/ , out/baselines/	trajectories (every tool call) and baseline outputs

Table 1: Layout of the released repository.

Finding	Group	Runs	Baselines	Prev. %	$n$	$P$ [95% CI]	Ov. %	Status
firebase_content_farm_spam	G01	3/3	LD, LC, R	5.32	19	.74 [.51,.88]	13	accepted
adult_spam_farm_wordsalad	G04	1/3	–	0.39	19	.84 [.62,.94]	11	accepted
porn_spam_keyword_farm	G04	1/3	–	0.36	12	.83 [.55,.95]	11	accepted
slo_prefix_machine_translated_farm	G05	1/3	–	0.28	14	.07 [.01,.31]	0	candidate
porn_doorway_spam_network	G04	1/3	–	0.20	20	1.00 [.84,1.00]	10	accepted
czech_slovak_code_switched_docs	G02	1/3	–	0.17	21	.81 [.60,.92]	100	accepted

Table 2: The seven distinct findings from three agent runs. Group : phenomenon group assigned mechanically by document overlap (Jaccard

\geq 0.8

) or nesting; rows sharing a group are one phenomenon, so the seven findings are four defects. Runs : agent runs that produced the finding. Baselines : baselines that proposed an equivalent slice (Jaccard

\geq 0.8

): R Random , LC LowConf , LD LID-Disagree , HD HighDup . Prev. : exact prevalence over all 26,470,482 documents.

n

: blinded labels.

P

: human-judged defect rate with Wilson interval. Ov. : largest share of the finding’s documents that the LowConf , LID-Disagree or HighDup slice retrieves; the three G04 selectors lie entirely inside the profanity slice

\texttt{n\_bad\_words}\geq 3

(§ 5 ). Status : as released in the suite. The released suite also holds two metadata invariants that are not slice findings (§ 5 ).

Figure 2: The 67 events logged for the main run (65 audit tool calls plus a run_detector record per save_finding ), by tool type (top) and with cumulative document inspections against the budget of 300 (bottom). The shaded span, events 41 to 49, is the firebase investigation. The agent counts the *.firebaseapp.com and *.web.app hosts (1,408,437 documents), samples and evaluates the detector on them, and saves the finding at event 49. Run total 197 inspections.

Figure 3: Share of each of the 96 Common Crawl snapshots matched by the accepted selectors, in crawl order. The Firebase content farm (top, 5.32% of the corpus overall) is almost absent before 2022 and reaches 38.7% of the 2023-06 crawl. The adult spam family (union of the three G04 selectors) peaks around 2020 and falls from 0.53% of older documents to 0.06% of the eight most recent crawls (shaded), while Czech text (union of the G02 selectors) stays below 0.4% throughout.

Figure 4: Human-judged defect rate for each of the seven agent findings, grouped by phenomenon. Bars are 95% Wilson intervals and the right-hand column gives defective documents over documents judged. The shaded band is the interval for 25 uniformly sampled documents (rate .08) and the dashed line is the acceptance threshold of .5. Six findings clear it. The slo.* selector (open marker, .07) is indistinguishable from the random control.

Figure 5: Where the discovered slices sit relative to three static audit slices. Each point is a candidate slice, placed by its prevalence in the corpus and by the largest share of its documents that LID-Disagree , LowConf or HighDup already retrieves. Agent findings are filled circles and baseline candidates, the selectors the four static audits proposed at the same inspection budget, are crosses. The two Czech findings are inside the language-ID disagreement slice, as one would expect of a mislabelling defect. The content farm, the adult-spam family and the machine-translation farm are not, although the adult-spam family lies entirely inside a profanity-count slice not drawn here (§ 5 ).

Figure 6: The content farm (1,408,437 documents on *.firebaseapp.com and *.web.app ) against the rest of the corpus on the three static signals. Panels (a) and (b) show the share of documents per bin on a log scale, with the LowConf cutoff of 0.8, the FineWeb2 threshold of 0.667 and the HighDup threshold of 100 dashed. No farm document reaches the duplication threshold. Panel (c) shows the share on which all three auxiliary identifiers return Slovak, and the share that also has language_score of at least 0.9 and a cluster size of at most 2, which is 86.6% of the farm (1,220,394 documents).

Condition	Insp.	Cand.	Val.	Grp.	G#	USD
Baselines (single runs)
Random	300	7	2	2	1,2	1.22
LowConf	300	6	4	3	1,2,9	1.03
LID-Disagree	300	5	2	2	1,2	1.16
HighDup	300	5	2	2	2,10	1.06
Agent (single runs)

Table 3: Budget-matched comparison. Rows above the pooled block are single runs, each reading at most 300 documents; pooled rows aggregate the four baselines and the three agent runs, counting selectors that match the same documents within one group once. Insp. : documents read. Cand. : distinct candidate selectors. Val. : selectors with an adopted approve verdict on five or more blinded labels at a defect rate of at least .5 (§ 4 ); a selector two conditions proposed identically is credited to both. Grp. , G# : phenomenon groups as numbered in Table 2 (1 = G01, …; G09 is an adult tag farm, G10 an injected advertising widget). USD : LLM cost.

Figure 7: Validated selectors per run against the cost of the run. Agent runs are blue circles and static baselines grey squares. The best baseline, LowConf , validates four selectors over three phenomenon groups at $1.03, more than any agent run at over twice the cost.

Test	train	held-out	ratio
	%	%
language_score_not_a_probability	40.389	40.230	1.00
firebase_content_farm_spam	5.321	5.024	0.94
adult_spam_farm_wordsalad	0.387	0.348	0.90
porn_spam_keyword_farm	0.359	0.334	0.93
slo_prefix_machine_translated_farm	0.279	0.292	1.05

Table 4: The released suite on the held-out split. Prevalences are exact counts on each split. Every slice prevalence lands within 0.86–1.06 times its training value; the invalid-confidence invariant fails there too, while the identifier invariant passes, as expected of 13 collisions in 26.5M training documents (Appendix F ).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Config	Script	docs	$>$ 1.0	%	max
mlt_Latn	Latin	5,193	4,106	79.1	1.00001
hye_Armn	Armenian	16,131	10,281	63.7	1.00001
ces_Latn	Latin	28,051	16,798	59.9	1.00001
cym_Latn	Latin	8,395	4,348	51.8	1.00001
khm_Khmr	Khmer	16,337	8,369	51.2	1.00001
isl_Latn	Latin	32,586	15,215	46.7	1.00001

Appendix

Table 5: language_score above 1.0 in upstream FineWeb2, measured on each configuration’s test shard. max is the largest score observed. No shard contains a duplicate identifier.

Test	all	old	new	ratio	crawls	peak
language_score_not_a_probability	40.39	40.50	40.13	0.99 $\times$	96/96	2024-10 (46.9)
firebase_content_farm_spam	5.32	0.74	15.91	21.6 $\times$	23/96	2023-06 (38.7)
adult_spam_farm_wordsalad	0.39	0.53	0.06	0.11 $\times$	96/96	2020-40 (2.3)
porn_spam_keyword_farm	0.36	0.50	0.04	0.07 $\times$	95/96	2020-40 (2.3)
slo_prefix_machine_translated_farm	0.28	0.27	0.30	1.1 $\times$	46/96	2021-49 (0.9)
porn_doorway_spam_network	0.20	0.28	0.01	0.05 $\times$	84/96	2020-40 (1.4)

Appendix

Table 6: Every test evaluated per crawl. all / old / new : prevalence (%) over the whole corpus, over all but the eight newest snapshots, and over those eight. ratio : new over old. crawls : snapshots in which the selector matches anything. peak : the snapshot where the slice is largest, with its prevalence there (%).

AI Agent Auditing

AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.

See all 269 papers

Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
May 20, 2026Akshay Manglik, Vijay S. Kalmath, Jason Qin +6Multi-Agent LLM Systems LLM Agent Evaluation
Scale AI
Agent-Driven Corpus Linguistics: A Framework for Autonomous Linguistic Discovery
Apr 8, 2026Jia Yu, Weiwei Yu, Pengfei Xiao +1Corpus Linguistics Autonomous Research Agents
Zhejiang International Studies University, Hangzhou, China · Tianjin University, Tianjin, China
BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
Jun 21, 2026Shreyas KCMultilingual Language Model Evaluation LLM-as-a-Judge

May 20, 2026J/K move · Enter open · S save

Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents

Akshay Manglik, Vijay S. Kalmath, Jason Qin, Apaar Shanker+5

Scale AI

Multi-Agent LLM Systems LLM Agent Evaluation LLM Agent Reliability Reasoning Trace Analysis

Open paper AskPDF arXiv

Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4 pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality.

Abstract

AI Agent Auditing

Explore similar work

Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents

Agent-Driven Corpus Linguistics: A Framework for Autonomous Linguistic Discovery

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories