Smart Content Ingestion for Generative AI Workloads

Abbas Raza Ali; Muhammad Ajmal Siddiqui; Moona Zahid

cs.AIOct 5, 2026

Smart Content Ingestion for Generative AI Workloads

Authors: Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona Zahid

Organizations: Citigroup Inc., London, United Kingdom · Ernst & Young LLP, London, United Kingdom · NVIDIA Corporation, London, United Kingdom

Abstract

The evolution of machine learning has progressively changed where intelligence resides in an AI system. In conventional machine learning the task, data representation, labels and model architecture were tightly coupled, so data preparation was narrow, schema-bound and visible. Generative AI decouples the model from any single task: one foundation model serves open-ended downstream tasks, and the generality gained on the model side is matched by heterogeneity on the data side, because enterprise knowledge is authored in the formats people use (PDF, presentations, spreadsheets, scanned documents, forms, tables, diagrams and mixed-layout files) that carry textual, visual, geometric and structural information at once. A language model or retriever cannot reason reliably over information misrepresented at this interface, so content extraction becomes a lifecycle stage in its own right whose errors no downstream retriever or re-ranker can repair. This paper presents a production-ready content-extraction system that makes this stage explicit, configurable, and measurable. The system incorporates selective OCR routing, a scarcity-first curation engine with a reference-based extraction scorer that measures character, word, and table-structure accuracy, a deterministic structure-aware parent-child chunker, and a read-only retrieval evaluator that generates grounded questions from every page and reports Hit@k, mean reciprocal rank, and latency. On a 180-document corpus the best extractor scores 97.4 of 100 (character error rate 0.13%, table similarity 0.995) and the chunker reaches hit@1 of 68.6%, hit@10 of 92.8% and MRR 0.77 over 25,050 generated questions. We distil three design principles (structure before semantics, never mutate what you measure, budget your labels) and position measured content extraction as the perception layer of enterprise agentic systems.

Figures & tables

Figure 1: Four phases turn raw files into a measured embeddings with a parent store. The embedding model used to index chunks and the one used to embed evaluation queries are byte-identical. The two evaluators (Phases 2 and 4) are strictly read-only.

	Docling	MinerU	Marker	Nougat	Unstr.	This Paper
Native-text fast path	$\bullet$	$\circ$	$\bullet$	-	$\bullet$	$\bullet$
Page-level OCR routing	$\circ$	$\circ$	$\circ$	-	$\circ$	$\bullet$
Pluggable OCR back-ends	$\circ$	$\circ$	$\circ$	-	$\circ$	$\bullet$
Structure-aware chunker	$\bullet$	-	-	-	$\bullet$	$\bullet$
Balanced curation	-	-	-	-	-	$\bullet$
Extraction scorer	-	-	-	-	-	$\bullet$

Table 1: Landscape of open document-ingestion tooling versus the proposed approach.

\bullet

built in;

\circ

partial or via plug-ins; - absent. Assessed from published descriptions Auer et al. (2024) ; Wang et al. (2024) ; Paruchuri (2024) ; Blecher et al. (2023) ; Unstructured Technologies (2024) .

Class	Signal
Scanned	$\|T(p)\|<\kappa$ and $r(p)>\tau_{\mathrm{cov}}$
Text	non-scanned page with extractable text
Table	page.find_tables() non-empty, or OCR has_table
Diagram	vector drawing count above threshold, evaluated only when the page is not already a table (avoids counting gridlines)
Graph / infographic	image-area ratio inside a band $[a_{\min},a_{\max}]$ that excludes icons and full-bleed photographs
Photo, signature	from OCR region annotation only

Table 2: Modality taxonomy and detection signals used by the curation engine.

Figure 2: Scarcity-first sampling. Target quotas

\phi_{m}

are set independently of the corpus distribution; the greedy pass visits the scarcest quota first so that the dominant text class cannot exhaust the page budget. Corpus shares shown are illustrative.

Strategy	Det.	Structure	Index	Query
Fixed / recursive	yes	none	low	low
Semantic	no	none	high	low
RAPTOR	no	summaries	v. high	medium
Late chunking	yes	implicit	high	low
Contextual retrieval	no	LLM prefix	v. high	low
Proposed parent-child	yes	explicit tags	low	$O(1)$

Table 3: Chunking strategies on properties that matter for auditable ingestion: fixed and recursive splitters, semantic splitting, RAPTOR Sarthi et al. (2024) , late chunking Günther et al. (2024) and contextual retrieval Anthropic (2024) .

Figure 3: The retrieval examiner. Page splitting reuses the chunker’s build_page_regex() ; query embedding reuses the chunker’s model fingerprint; search is scoped to the source document; hits are judged on score and page proximity.

Figure 4: Wall-clock decomposition for the Orion corpus (180 documents, 9,654 text and 853 image pages). The shared pre-processing stage dominates; the two other back-ends fall between the extremes shown.

Model	Score	CER	WER	TEDS
Gemini 3.1 Pro	97.37	0.0013	0.0013	0.9952
Mistral OCR v3	95.98	0.0139	0.0155	0.9737
Granite-Docling (258M)	92.92	0.0260	0.0290	0.9599
SmolDocling (256M)	90.70	0.0443	0.0482	0.9274

Table 4: Extraction back-ends ranked by the coverage-penalised composite score (Equation 4 ); lower CER and WER and higher TEDS are better.

Figure 5: Extraction benchmark across four back-ends: (a) composite score, (b) character and word error rates, (c) table structure similarity (TEDS). Text and table fidelity rank the models consistently.

Metric	Value
Documents / questions evaluated	180 / 25,050
Questions missed at all ranks	1,810 (7.2%)
hit@1 / 3 / 5 / 10 (%)	68.58 / 82.30 / 87.66 / 92.77
MRR	0.7745
Median / mean rank	1.0 / 1.81
Mean per-query latency (s)	0.81

Table 5: Retrieval evaluation on Orion (GTE embeddings, FAISS flat index, parent de-duplication, page tolerance

\pm 1

Figure 6: Hit rate as a function of the rank cut-off on the Orion corpus. The steep rise from hit@1 to hit@3 and the median rank of 1 indicate that the correct parent is usually retrieved at or near the top.

Explore similar work

Jul 31, 2026cs.AI

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Enterprise workflows increasingly rely on agents for \emph{schema-guided extraction}: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on HuggingFace and GitHub.

Boyang Zhang, Adrian Lyjak, Eli Stewart +2

Schema Document Language-Model Agents Traceability

Ask PDF

Aug 6, 2026cs.AI

Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI

We present a schema-based framework for extracting complex, structured information from unstructured text documents using generative AI, followed by automated semantic evaluation of the extracted information against a gold standard. The schema, serving as an information model encoding domain knowledge, provides a unified, systematic, and consistent framework for extraction of hierarchical, nested information, with attributes of variable cardinality, and subsequent evaluation of the results. Information extraction from a document is performed in a single call to the model, in zero-shot mode. In the evaluation step, we introduce a path-based semantic matching algorithm to align the nested, variable-cardinality attributes in the extracted results with those in the gold standard. We use generative AI for semantic comparison of the extracted and gold standard values of an attribute, and introduce a rubric to classify the result of the comparison, according to domain-specific considerations, as an exact, semantic, useful, or non-match. We were able to extract 12 out of 14 attributes with an F1 score of

>

90% from documents published by the health technology assessment organisation NICE, using the generative AI model Claude Opus 3. The time needed to extract the attributes from a document was

\sim

30 times lower than the time taken by a human domain expert. We further demonstrate generalisability of this framework across different generative AI models and transferability across different HTA organisations and languages.

Modhurita Mitra, Jan-Willem Versteeg, Maarten D. Schermer +3

Research Engineering Team, Information and Technology Services, Utrecht University, Utrecht, The Netherlands · Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht University, Utrecht, The Netherlands

Information Extraction Schema Unstructured Data Generative Artificial Intelligence

Ask PDF

Apr 29, 2026cs.CL

Benchmarking Complex Multimodal Document Processing Pipelines: A Unified Evaluation Framework for Enterprise AI

Most enterprise document AI today is a pipeline. Parse, index, retrieve, generate. Each of those stages has been studied to death on its own -- what's still hard is evaluating the system as a whole. We built EnterpriseDocBench to take a swing at it: parsing fidelity, indexing efficiency, retrieval relevance, and generation groundedness, all on the same corpus. The corpus is built from public, permissively licensed documents across six enterprise domains (five represented in the current pilot). We ran three pipelines through it -- BM25, dense embedding, and a hybrid -- all with the same GPT-5 generator. The headline numbers: hybrid retrieval narrowly beats BM25 (nDCG@5 of 0.92 vs. 0.91), and both beat dense embedding (0.83). Hallucination doesn't grow monotonically with document length -- short documents and very long ones both hallucinate more than medium ones (28.1% and 23.8% vs. 9.2%). Cross-stage correlations are very weak: parsing->retrieval r=0.14, parsing->generation r=0.17, retrieval->generation 0.02. If quality were cascading the way most of us assume, those numbers would be much higher; they aren't. Design caveats are real (parsing fixed, generator shared, automated proxy metrics) and we don't oversell the result. One result that genuinely surprised us: factual accuracy on stated claims is 85.5%, but answer completeness averages 0.40. The system is right when it answers -- it just leaves things out. That gap matters more for real deployments than the headline accuracy number does. We also describe three reference architectures (ColPali, ColQwen2, agentic complexity-based routing) which are not yet integrated end-to-end. Framework, metrics, baselines, and collection scripts will be released open-source on acceptance.

Saurabh K. Singh, Sachin Raj

Oracle · Independent

Multimodal Documents Large Language Model Pipelines Text Corpora

Ask PDF