From Crawled to Published: A Publisher-Controlled Web Architecture for an AI-Native Internet

Yixi Chen; Muhammad Bilal; Tie Ma; Zafar Ayyub Qazi; Marco Canini; Suhaib Fahmy

cs.NINov 23, 2025

From Crawled to Published: A Publisher-Controlled Web Architecture for an AI-Native Internet

Authors: Yixi Chen, Muhammad Bilal, Tie Ma, Zafar Ayyub Qazi, Marco Canini, Suhaib Fahmy

Abstract

AI agents, not humans, are becoming the web's primary consumers, yet the web still ships content structured for human browsing, with no native way for publishers to expose the semantic units within it. Access is therefore mediated by a handful of AI providers that crawl, chunk, and index content unilaterally, leaving publishers little control over how their content is represented, retrieved, or attributed. We argue this is a structural problem, not one better chunking or retrieval algorithms can solve. The web lacks a native layer for publisher-controlled semantic discovery and retrieval. We propose AISTRAL, an AI-Native Internet architecture that supplies this layer. Publishers expose semantically meaningful units (e.g., passages, claims, tables) through explicit, policy-governed interfaces they control, and a web-native semantic resolver, analogous to DNS but operating over meaning rather than location, lets agents discover relevant sources before retrieving fine-grained content directly from publishers. Our end-to-end prototype matches or exceeds representative centralized retrieval pipelines in answer quality, with up to 4.2x higher accuracy on large, specialized document collections. Against a public web-search agent, it answers up to 4.2x more questions correctly, while moving 2.9-41x less data per query. Because retrieval terminates at publisher-operated servers, attribution, provenance, and usage policies travel with the content, realigning incentives across the web ecosystem.

Figures & tables

Figure 1: Accuracy versus data cost, each as a multiple of the baseline (centralized retrieval, dashed lines at $1\times$ ). Data moved per query on the horizontal axis and answer accuracy on the vertical. Each system has one point per benchmark (marker shape), spanning multiple domains and question types, from encyclopedic multi-hop questions (KILT-HotpotQA) and obfuscated public-web puzzles (BrowseComp+) to single-fact lookups and multi-document aggregations over financial filings (LOONG L1/L2). AISTRAL (blue) stays near the baseline’s data cost ( $\mathord{\sim}2\times$ ) while $1.1$ – $4.2\times$ more accurate, whereas the web-search agent (purple) moves $5$ – $80\times$ more data for mixed accuracy. Up and to the left is better. 4 4 4 The web-search agent is an off-the-shelf LLM ( gpt-5-mini ) that answers each query with live web search instead of our indexed corpus ( § 4 ).

Panel of Figure 2 — Figure 2: AISTRAL architecture and interfaces across publisher, infrastructure-provider, and client roles. Orange dashed arrows denote publication and ingestion, while blue solid arrows denote query-time interactions. The Client’s AI agent is decomposed into a Retriever (query understanding and source discovery/retrieval) and a Synthesizer (context synthesis).

Figure 3: Answer correctness vs. data moved, across four benchmarks. Columns are datasets. Top row: answer correctness. Bottom row: data transferred per query to produce that answer (KB, log scale, shared across panels; lower is better). Both are plotted against retrieval depth

k

(chunks per agentic twin). Two

k

-independent baselines appear as horizontal lines in both rows: perfect document retrieval (green — the reader given the full gold documents) and a web search agent (search-only, no memory) (purple --- the same reader answering from OpenAI’s live public-web search). AISTRAL approaches the perfect-retrieval ceiling and stays well above the web agent on specialized corpora (LOONG, BrowseComp+) 8 8 8 BrowseComp+ is shown only to

k{=}12

: at larger

k

the retrieved context reaches 0.8–2 MB per query, exceeding the reader model’s input limit, and

{\sim}11\%

of queries return an empty answer. , while moving far less data than the web agent.

Figure 4: Best evidence-chunk rank, per query. For each query, the rank (log scale) of the highest-ranked chunk containing any part of the required evidence. Curves show the CDF of this rank across queries; further left is better. This measures how quickly the first relevant chunk surfaces.

Figure 5: Retrieval

\rightarrow

answer attribution for AISTRAL: base vs. contextual chunking. Each stage shows two bars, base with blue and +ctx with orange, as the percentage of queries surviving that stage at the plateau-

k

operating point. Stages split into a retrieval regime (routing to the gold twin, gold-document retrieval, evidence retrieval; shaded blue), controlled by AISTRAL, and an answering regime (producing an answer, answering correctly; shaded purple), controlled by the synthesizer LLM. Routing and document retrieval are near-saturated on all benchmarks; the dominant loss is recall . Contextual chunking lowers it yet does not lower correctness in step — the retrieval and answer stages move independently.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Interface	Argument	Field	Type	Design role
publish	content	—	string/object	Content to be semantically prepared and served locally.
	metadata	source	string	Canonical source reference, e.g., URL or document identifier.
		title	string	Human-readable attribution label.
		timestamp	time	Freshness signal for downstream discovery and retrieval.
		content_type	string	Data modality or representation type.

Appendix

Table 1: Design-relevant fields for publish(content, metadata) .

Interface	Argument	Field	Type	Design role
insert	documents	content	string	Semantic representation registered for source discovery.
		server_url	string	Endpoint of the agentic web server that can later serve retrieval results.
		source	string	Canonical source reference preserved for provenance and attribution.
		title	string	Source or document reference exposed to downstream consumers.
		timestamp	time	Supports freshness-aware discovery constraints.
		category	string	Supports topical partitioning or coarse semantic filtering.

Appendix

Table 2: Design-relevant fields for insert(documents) .

Interface	Argument / Return	Field	Type	Design role
resolve	query	—	string	Query to be mapped to relevant sources.
	filters	limit	integer	Upper bound on the number of returned agentic web server.
		allowed_domains	list[string]	Restricts discovery to selected server domains or endpoints.
		allowed_categories	list[string]	Restricts discovery to selected semantic categories.
		allowed_languages	list[string]	Restricts discovery by language.
		min/max_timestamp	time range	Restricts discovery by freshness range.

Appendix

Table 3: Design-relevant fields for resolve(query, filters) .

Interface	Argument / Return	Field	Type	Design role
search	query	—	string	Query to be evaluated within a selected source.
	limit	—	integer	Number of local semantic units to return.
	return	document	string/object	Retrieved semantic unit returned to the AI agent.
		metadata	object	Optional provenance and within-source location metadata for the unit.
		score / distance	float	Optional local ranking signal from lexical or vector retrieval.
		before/after_context	string	Optional neighboring context for interpretability or expansion.

Appendix

Table 4: Design-relevant fields for search(query, limit) .

Interface	Argument / Return	Field	Type	Design role
process_query	query	—	string	Original user query.
	return	sanitized_query	string	Rewritten query used for downstream discovery and retrieval.
		subqueries	list[string]	Decomposes a complex request into simpler retrieval steps.
		constraints	object	Surfaces implicit requirements such as scope, freshness, or grounding needs.

Appendix

Table 5: Design-relevant view of process_query(query) .

Interface	Argument / Return	Field	Type	Design role
synthesize	query	—	string	Original or processed query whose answer is to be produced.
	context	—	list/object	Retrieved semantic units used for answer construction.
	return	answer	string	Final answer derived from the retrieved context.
		citations	list/object	Optional evidence references preserved for grounding or attribution.

Appendix

Table 6: Design-relevant view of synthesize(query, context) .

Dataset	Query selection	Query transform	Gold evidence
KILT-HotpotQA	50/50 bridge/comparison	none (verbatim)	native (offset rebased)
LOONG	50/50 L1/L2, evict+replace	entity-set prepend	constructed (staged heuristic+LLM)
BrowseComp+	stratified to domain dist.	none (verbatim)	constructed (heuristic/LLM locator)

Appendix

Table 7: Per-dataset preprocessing. Effort scales inversely with native support: KILT and BrowseComp+ ship usable question formats and evidence; LOONG ships neither.

Figure 6: LLMs-and-Web

k

-sweep results under varying retrieval depth. The left panel shows traffic reduction as the per-query retrieval budget

k

increases, while the three right panels show answer quality in terms of correctness , completeness , and focus . Across both AISTRAL and centralized variants, most answer-quality gains saturate by small

k

, while larger

k

primarily reduces traffic efficiency. The dashed horizontal line marks the Opus reference baseline, and the vertical marker highlights

k=2

crossing the baseline performance.

Benchmark	Search-only (reported)	Autonomous	$\Delta$
LOONG L1	0.190	0.174	$+0.016$
LOONG L2	0.540	0.500	$+0.040$
KILT-HotpotQA	0.857	0.830	$+0.027$
BrowseComp+	0.377	0.394	$-0.017$

Appendix

Table 8: Correctness under the search-only (no-memory) configuration vs. the agent’s default autonomous behavior.

Dataset	best-guess	Pipeline correct
LOONG L1	0.00	0.79
LOONG L2	0.06	0.63
KILT-HotpotQA	0.67	0.86
BrowseComp+	0.00	0.79

Appendix

Table 9: Closed-book correctness (no retrieved context) vs. full-pipeline correctness. No-guess is scored under an instruction to decline when unsure; Best-guess under an instruction to always guess.

Dataset	Mechanism	Ablation evidence
BrowseComp+	Retrieval-bound	Best-guess $=0.00$ ; recall $\approx$ correctness
LOONG L2	Candidate-constrained guessing	Best-guess $\approx 0$ ; 5/17 correct with 0 relevant values retrieved
KILT	Shortcut + partial memory	Best-guess $=0.67$ ; 26/34 shortcut, 8/34 memory-only

Appendix

Table 10: Correctness–recall divergence mechanism per dataset, with supporting evidence.

Information Retrieval

Information Retrieval studies how to find, rank, and organize relevant information in large collections of data.

See all 471 papers

Semantics Delivery Network: Rethinking Web Retrieval Infrastructure for LLM Agents
Sep 18, 2026Peichun Hua, Yunming XiaoSemantic Caching Agentic RAG
The Chinese University of Hong Kong, Shenzhen, China
Towards an Agent-First Web: Redesigning the Web for AI Agents
Jun 17, 2026Eranga Bandara, Ross Gore, Ravi Mukkamala +18AI Agent Governance Web Agents
Old Dominion University, Norfolk, VA, USA · Florida International University, USA · Deloitte & Touche LLP, USA +5
AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web
Apr 13, 2026Shanshan Zhong, Kate Shen, Chenyan XiongWeb Agent Benchmarks Web Agents
Language Technologies Institute, School of Computer Science, Carnegie Mellon University · Anaxi Labs

Sep 18, 2026J/K move · Enter open · S save

Semantics Delivery Network: Rethinking Web Retrieval Infrastructure for LLM Agents

Peichun Hua, Yunming Xiao

The Chinese University of Hong Kong, Shenzhen, China

Semantic Caching Agentic RAG LLM Agents

Open paper AskPDF arXiv

Large language models (LLMs) increasingly ground their answers in proprietary data and live web content through both single-shot retrieval-augmented generation (RAG) and multi-turn agentic search. Yet the web path they use was built for human readers: search services return URLs and snippets ranked by generic relevance, and content delivery networks (CDNs) cache URL-addressed objects without knowing which passage an agent needs. LLMs instead consume short, semantically coherent passages - hereafter, chunks - selected for downstream utility rather than similarity alone, retrieve statefully across reasoning turns, and, as uncoordinated streams, repeatedly fetch, clean, chunk, and embed the same content. We argue that semantic chunk retrieval should become a first-class network-delivery abstraction, and propose the Semantics Delivery Network (SemDN): an origin-authorized, hierarchical substrate that indexes, searches, and caches web content at chunk granularity, serving clients and origins alike. SemDN is more than finer-grained caching: it must estimate missing or stale evidence without an explicit miss signal, let clients choose retrieval representations that origins need not see, and share state among clients with the same representation. Preliminary results show that page-first retrieval processes two to three orders of magnitude more content than agents consume, that agentic queries exhibit task-local chunk reuse invisible to URL caches, and that chunk delivery improves answer quality per context token. We organize an open research agenda around shareable retrieval state, hierarchical caching and refresh under coverage risk, and deployability, including incentives, privacy, and verifiability. Our code is available at: https://github.com/sarendis56/SemDN_HotNets26

Abstract

Information Retrieval

Explore similar work

Semantics Delivery Network: Rethinking Web Retrieval Infrastructure for LLM Agents

Towards an Agent-First Web: Redesigning the Web for AI Agents

AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web