cs.AIAug 21, 2026

RAG Deserves an Index: Why Ingest-Time Compilation Beats Query-Time Interpretation

Authors: Kyle Wild, Yusuke Takahashi, Asako Uraki

Organizations: Endgame Labs, Inc. San Francisco, USA · Asia AI Institute, Musashino University Tokyo, Japan · Faculty of Data Science, Musashino University Asia AI Institute AIx, Inc. Tokyo, Japan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Nearly every retrieval-augmented question-answering system in production ships with a hidden interpreter: on each query a language model re-derives the meaning of raw corpus text and then throws that work away. Cheaper models do not close the gap: per-token prices have fallen by orders of magnitude while inference spend has risen, because context volume grows faster than prices fall. This is the modern equivalent of the full-table scan, and the remedy is the one databases found fifty years ago: do the expensive work once, at write time, into a maintained structure that makes reads cheap. A corpus whose read pattern is known before it ever meets a user can and should be indexed too. We call the paradigm ingest-time semantic compilation (ISC): compile a corpus's meaning into a queryable substrate with two coupled layers - incrementally maintained embeddings, and atomic claims whose provenance is validated at compile time - and treat that substrate as a first-class database object with its own DDL, maintenance contract, migration contract, and cost model. Two existence proofs support it. Substrate upkeep scales with change rather than corpus size: incremental updates run 33.7x cheaper than reconstruction while tracking it to floating-point precision. And on a held-out sample of 500 broadcast-interview transcripts, compiled claims as the retrieval payload win all 32 budget-by-model cells: 85.2% correct from roughly 2.2k reader tokens against 72.5% from 16.3k for the best chunk configuration anywhere. The only baseline that keeps pace is a contextualized-chunk pipeline with hybrid retrieval and reranking, statistically indistinguishable from compiled claims at roughly twenty-one times the query-path tokens - and it reaches that parity, we argue, precisely because it has itself begun to compile. We close with the systems agenda this opens, from compilation planners to read planning.

Figures & tables

ContractGuaranteesEnforced byShown
Compilation (semantic DDL)only source-supported claims enterexact-quote validation gate§2.1
Maintenancecost tracks change, not corpus sizeincremental low-rank update§3.1
Migrationa model upgrade is an alignment problemorthogonal Procrustes§3.1
Cost modelcompile only where reads repay itR∗R^{*} as a planner statisticApp. A
Table 1. The four contracts, what each guarantees, the mechanism that enforces it, and where this paper exhibits it.
Figure 1. Query-time semantic reconstruction repeats the interpretive work on every read; ingest-time semantic compilation performs it once and maintains the result. Two parallel flow diagrams. On the left, query-time semantic reconstruction: a query arrives, raw passages are retrieved, a reader model re-interprets them, and an answer is produced, with the cycle repeating on every query. On the right, ingest-time semantic compilation: a document arrives and is compiled and validated once into a substrate of maintained embeddings and validated claims, after which a query becomes a cheap lookup.
Figure 2. The read-time frontier on the held-out sample. Compiled claims occupy the upper left; the only configuration that matches them on accuracy sits far to the right, and every configuration that reads raw text sits below. The vertical axis is truncated below 69% (break marked); exact values are printed beside each point. A scatter plot of accuracy against tokens read per query on a log scale. Compiled claims sit at 2.2k tokens and 85.2 percent accuracy in the upper left. The best chunk cell sits at 16.3k tokens and 72.5 percent, below and to the right. The contextualized stack sits at 47.7k tokens and 88.0 percent, far to the right at roughly the same height as compiled claims.
ApproachCompilesVal.Reader receives
Document expansion ( 10 )the pointernoraw passage plus queries
Proposition indexing ( 3 )the pointernoraw passage
Contextual chunking ( 1 )pointer, part of payloadnoraw passage plus preamble
Agent memory ( 4 , 12 )the payloadnoextracted claims
GraphRAG familystructurenosummaries, traversals
Context compressionnothing; shrinks at read—lossy passage
Table 2. What each approach compiles, whether the compiled form is validated against its source, and what the reading model actually receives. Only the last row fills both of the middle columns.

Explore similar work

CardsList
Aug 29, 2026

Ingest-Time Fact Compilation for Cost-Efficient and Reliable Question Answering over Revised Corpora

Most agentic question answering (QA) systems do an important part of their semantic work at the worst possible time: every time someone asks a question. When a corpus contains revisions, drafts, revocations, deletions, and sources with different levels of authority, the model must reconstruct the governed current state on every read - then throw that work away and repeat it on the next query. This is a bit like a database that rebuilds a materialized view every time someone reads from it. We present ingest-time fact compilation, an architecture that performs this work when corpus data is ingested or changed. Raw passages are rephrased into self-contained facts; rules governing revisions, deletions, effective dates, and source trust are resolved once; and the resulting state is stored as typed records carrying source and revision provenance. At query time, an inexpensive model reads the compiled record instead of reconstructing it from noisy candidates. In a controlled synthetic experiment across five seeds, the same low-cost model produced the correct value, source, and revision in only one of 30 trials under query-time reconstruction, but in all 30 trials from the compiled substrate, at 12.89 times lower mean read cost per question. On simpler revision questions both architectures were exact, but the compiled path used 21.6 times fewer tokens. A separate test found that fact rephrasing roughly halved verbose Federal Reserve dialogue while preserving high source entailment, but left concise Wikipedia prose essentially unchanged. These results support a narrow but practical claim: resolving a corpus state once can make subsequent QA cheaper and more reliable for inexpensive models. We release the open source, MIT-licensed implementation and experimental artifacts.

Kyle Wild, Yusuke Takahashi, Asako Uraki

Endgame Labs, Inc. San Francisco, USA · Asia AI Institute, Faculty of Data Science Musashino University Tokyo, Japan

Temporal QAEfficient Language Model Inference
AskPDF
Aug 17, 2026

Cost Scales with Change, Not Corpus Size: Incrementally Maintaining an Evolving Semantic Substrate

Retrieval-augmented and agentic question-answering systems increasingly re-derive the meaning of a corpus at query time. Put plainly, instead of re-deriving what a corpus means on every question, the work is done once when a document arrives and is thereafter merely consulted -- a compiler, not an interpreter, of meaning. An alternative is to compile that meaning once, at ingest time, into a compact, queryable semantic substrate and maintain it as the corpus evolves. The central objection is maintenance cost: rebuilding a truncated singular value decomposition (SVD) on every change appears prohibitive, and a change of embedding model seems to force a full re-embedding. We argue and show empirically that maintenance cost scales with the amount of change, not corpus size. On a controlled synthetic pilot (dimension 256, rank 32, a corpus grown from 3,000 to 9,000 documents over 50 update events), incremental low-rank updates were 33.7 times cheaper per update than full re-SVD and 23.8 times cheaper cumulatively, while the incremental subspace tracked the full recomputation to within floating-point precision (maximum principal-angle drift below 1e-11 degrees; recall@10 = 1.0). An orthogonal Procrustes virtual axis update recovered 0.95 mean cosine to truly re-embedded vectors by re-embedding only about 10 percent of the corpus. The results support maintaining, rather than repeatedly reconstructing, a semantic substrate.

Yusuke Takahashi, Kyle Wild, Asako Uraki

Asia AI Institute, Faculty of Data Science Musashino University Tokyo, Japan

Matrix FactorizationRetrieval-Augmented GenerationLow-Rank Compression
AskPDF
Jun 14, 2026

CAMI: Cost-Aware Agent-Guided Multi-Indexing for Semantic Retrieval

RAG ingestion pipelines frequently augment search corpus index with semantic enrichment indices (e.g., synthetic queries or summaries generated from corpus chunks) that are subsequently queried alongside the base index to improve retrieval via better alignment between document representations and user intent. While these supplementary representations substantially improve retrieval quality, they introduce a computational bottleneck: the configuration space of enrichment types and generator models is combinatorial, and the cost of exhaustive index-time evaluation scales linearly with corpus size. We introduce CAMI (Cost-Aware Multi-Indexing), a framework that formalizes multi-index construction as a budgeted, multi-objective portfolio selection problem. CAMI targets the upstream decision of which enrichment views to generate and materialize before the retrieval backend is applied. CAMI incorporates three primary mechanisms: (i) an agentic discovery phase that proposes corpus-specific representation templates; (ii) an atomic-unit search procedure that evaluates individual enrichment-model pairs and recombines them via fidelity-local closure to identify synergistic portfolios; and (iii) a confidence-aware promotion schedule that prunes unpromising configurations early, decoupling optimization spend from total corpus size. We evaluate CAMI across diverse retrieval corpora. Our findings reveal that the framework systematically isolates high-recall portfolios under strict budget constraints, outperforming standard content-only baselines in challenging settings by up to 9.4% recall@10. Further, CAMI is able to systematically identify these high-recall portfolios using up to 5x less budget compared to random search baselines, making our approach practical in real production scenarios.

Adnan Qidwai, Anand Eswaran, Sonam Mishra +2

IBM Software Innovation Lab India

Semantic SearchInformation RetrievalMulti-Index Models
AskPDF