Retrieval-augmented generation (RAG) is a standard approach to enhance the reliability of large language model (LLM) outputs by retrieving documents relevant to a query and supplying them in the LLM prompt. Today's RAG systems collect, index, and serve these documents from centralized stores. In practice, however, documents are owned by organizations, devices, and individuals who cannot or will not surrender custody of them to a central party, e.g., because of privacy, copyright or regulatory compliance. Existing modern decentralized retrieval techniques either reintroduce a coordinating authority, or rely on a global data structure that requires continuous network-wide coordination (e.g., DHTs). We present ZeroRAG, a decentralized routing layer for document retrieval in RAG that performs semantic query routing across an overlay of peers with- out global data structures or centralized orchestration. Documents remain with the peers that own them, and each peer keeps only lightweight local state: a bounded set of neighbor advertisements and a cache of recently routed queries. Using these, ZeroRAG steers each query through a best-first walk that progressively rewires the overlay toward semantically relevant neighbors, so recall improves over time as the network self-organizes. We evaluate ZeroRAG using three real-world workloads and two decentralized baselines. We find that ZeroRAG outperforms baselines in terms of recall and required network messages, approaches the recall and LLM answer accuracy of a centralized document store, self-heals after catastrophic peer failures, and scales to 100 000 peers.
Figures & tables
Figure 1. The standard RAG pipeline: offline indexing (left) encodes documents into a vector database; online retrieval (right) matches a query to relevant chunks before LLM answer generation.
Figure 2. (left) Example of 4 query steps of a document searched by peer a with query q . After the search concludes, peer a adds peer i , which has the most relevant document to a ’s query, as neighbor. (right) The dynamics of query caching in ZeroRAG .
Figure 3. Average recall@1 for AOL-IA query batches, for ZeroRAG and different baseline configurations.
Figure 4. Percentage of retrieved results, binned by the number available results for a query, for the Amazon workload.
Figure 5. (a) The PubMedQA end-to-end RAG performance, using ZeroRAG and HNSW . (b) the distribution of number of ZeroRAG requests before the first relevant document.
Figure 6. Average recall@1 per processed query batch in ZeroRAG with the PubMedQA workload, when there is a catastrophic failure event (dashed red line).
Figure 7. Ablation of query memory and adaptive topology on the (a) AOL-IA and (b) Amazon Electronics workloads.
Max. cache size ( δm )
Avg. Recall@1
Avg. requests required
0
(77.860±0.177)%
16.796 ± 0.067
10
(85.834±0.120)%
12.757 ± 0.041
100
(97.612±0.037)%
5.900 ± 0.017
1,000
(99.652±0.007)%
3.448 ± 0.006
10,000
(99.652±0.007)%
3.448 ± 0.006
Table 1. The effect of query cache size.
Network size (peers)
Avg. Recall@1
Avg. requests required
1,000
(99.365±0.272)%
3.209 ± 0.111
5,000
(98.276±0.066)%
4.330 ± 0.059
10,000
(98.378±0.051)%
4.656 ± 0.015
50,000
(99.652±0.007)%
3.448 ± 0.006
100,000
(99.913±0.005)%
5.724 ± 0.011
Table 2. The effect of network size.
Docs. per peer ( t )
Avg. Recall@1
Avg. requests required
10
(98.665±0.018)%
5.940 ± 0.023
20
(99.292±0.016)%
5.545 ± 0.017
30
(99.667±0.007)%
5.361 ± 0.012
40
(99.836±0.006)%
5.276 ± 0.009
50
(99.919±0.004)%
5.222 ± 0.007
Table 3. The effect of document count per peer.
Max. neighbors ( δn )
Avg. Recall@1
Avg. requests required
3
(98.614±0.022)%
6.198 ± 0.020
5
(99.080±0.012)%
5.360 ± 0.009
10
(99.417±0.012)%
4.431 ± 0.010
25
(99.606±0.004)%
3.702 ± 0.010
50
(99.644±0.010)%
3.505 ± 0.005
75
(99.666±0.008)%
3.388 ± 0.004
Table 4. The effect of neighborhood size. Five seeds.
Fanout
Rounds
Avg. Recall@1
Avg. rounds required
3
10
(99.374±0.010)%
1.718 ± 0.002
6
5
(99.203±0.012)%
1.508 ± 0.001
10
3
(97.456±0.040)%
1.408 ± 0.001
Table 5. The effect of parallel searches.
Figure 8. Introduction of a new topic at the 60% mark (vertical line). The lines of owners ≥10 overlap.
System
Decentr.
Unlab.
Data Structure
ZeroRAG [this work]
✓
✓
Local
Semantica ( Neague et al., 2025 )
✓
✓
Global
LSH / Kademlia ( Datar et al., 2004 ; Maymounkov and Mazières, 2002 )
✓
✓
Global
REMINDIN’ ( Tempich et al., 2004 )
✓
Local
SHIMI ( Helmi, 2025 )
✓
Local
Proximity ( Bergman et al., 2025 )
✓
N/A
Table 6. Whether various retrieval and routing systems have (“Decentr.”) decentralized governance, use (“Unlab.”) unlabeled data, or use a global, local, or no (“N/A”) data structure.