cs.DCNov 5, 2026

ZeroRAG: Adaptive Semantic Peer Routing for Decentralized Retrieval-Augmented Generation

Authors: Petru Neague, Quinten Stokkink, Martijn de Vos, Rafael Pires, Mathis Randl, Anne-Marie Kermarrec, Johan Pouwelse

Organizations: Delft University of Technology Delft, The Netherlands · EPFL Lausanne, Switzerland

Abstract

Retrieval-augmented generation (RAG) is a standard approach to enhance the reliability of large language model (LLM) outputs by retrieving documents relevant to a query and supplying them in the LLM prompt. Today's RAG systems collect, index, and serve these documents from centralized stores. In practice, however, documents are owned by organizations, devices, and individuals who cannot or will not surrender custody of them to a central party, e.g., because of privacy, copyright or regulatory compliance. Existing modern decentralized retrieval techniques either reintroduce a coordinating authority, or rely on a global data structure that requires continuous network-wide coordination (e.g., DHTs). We present ZeroRAG, a decentralized routing layer for document retrieval in RAG that performs semantic query routing across an overlay of peers with- out global data structures or centralized orchestration. Documents remain with the peers that own them, and each peer keeps only lightweight local state: a bounded set of neighbor advertisements and a cache of recently routed queries. Using these, ZeroRAG steers each query through a best-first walk that progressively rewires the overlay toward semantically relevant neighbors, so recall improves over time as the network self-organizes. We evaluate ZeroRAG using three real-world workloads and two decentralized baselines. We find that ZeroRAG outperforms baselines in terms of recall and required network messages, approaches the recall and LLM answer accuracy of a centralized document store, self-heals after catastrophic peer failures, and scales to 100 000 peers.

Figures & tables

Explore similar work

May 22, 2026

FD-RAG: Federated Dual-System Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) has emerged as a paradigm for grounding large language models in external knowledge, yet most existing RAG systems assume centralized knowledge access and ample computation. These assumptions break down in edge environments, where knowledge is fragmented across devices, raw data cannot be shared, and repeated LLM calls are prohibitively expensive. We propose FD-RAG, a federated dual-system RAG framework that decouples lightweight memory access from on-demand LLM reasoning for decentralized deployment. Specifically, FD-RAG learns semantic-aware adaptive hypergraphs over local corpora and distills them into compact QA memories. At inference time, it answers well-covered queries via direct memory matching and invokes LLM-based reasoning only when necessary, while tracing retrieved memories to hypergraph-grounded evidence. To mitigate cross-device knowledge fragmentation, FD-RAG aggregates anonymized memories across devices without exposing raw documents. Experiments on QA benchmarks show that FD-RAG improves accuracy by up to 7.8% while reducing latency by 8.4×\times compared with strong local and federated baselines. We also provide theoretical analysis establishing an O(1/ε2)\mathcal{O}(1/ε^{2}) convergence rate for the proposed hypergraph learning, supporting its tractable deployment in edge settings.
Aug 26, 2026

Beyond Raw Context Transfer: Representation-based Federated Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) improves the factuality of large language models (LLMs) and vision-language models (VLMs) by grounding generation in external knowledge. However, most existing RAG frameworks assume a centralized retrieval corpus, which is often impractical in sensitive domains such as healthcare, where data are inherently distributed and raw content cannot be directly shared across institutions. Recent efforts on decentralized RAG primarily follow prompt-based paradigms that exchange raw, human-readable retrieved content, leading to substantial inference-time computational overhead and direct exposure of retrieved information. To address these limitations, we propose Representation-based Federated RAG (FedRepRAG), a decentralized RAG framework that keeps raw documents at their owning clients and exchanges only compact latent representations during cross-client retrieval. To integrate retrieved knowledge, we introduce a collaboratively trained projector that converts retrieval embeddings into generator-compatible representation tokens for a frozen LLM/VLM backbone. Experiments across decentralized visual question answering (VQA) and question answering (QA) benchmarks show that FedRepRAG consistently outperforms direct inference and local retrieval baselines while substantially reducing retrieval-context length and inference-time computational overhead compared with raw-context transfer. Further analyses confirm the importance of query-relevant retrieved representations and characterize the residual representation-level leakage associated with representation exchange. Overall, FedRepRAG provides an effective and efficient framework for federated RAG without transferring raw retrieved content.
Mar 16, 2026

P2^2RAG: Efficient Privacy-Preserving RAG Service Supporting Arbitrary Top-kk Retrieval

Retrieval-Augmented Generation (RAG) enables large language models to use external knowledge, but outsourcing the RAG service raises privacy concerns for both data owners and users. Privacy-preserving RAG systems address these concerns by performing secure top-kk retrieval, which is typically implemented using secure sorting to identify relevant documents. However, existing systems face challenges supporting arbitrary kk due to their inability to change kk, new security issues, and in particular, efficiency degradation with large kk. This is a significant limitation because applications such as finance, law, and healthcare require a kk that is large enough to cause huge overhead for existing systems. Also, modern long-context models generally achieve higher accuracy with larger retrieval sets. We propose P2^2RAG, an efficient privacy-preserving RAG service that supports arbitrary top-kk retrieval. Unlike existing systems, P2^2RAG avoids sorting candidate documents. Instead, it uses an interactive bisection method to determine the set of top-kk documents. For security, P2^2RAG uses secret sharing on two semi-honest non-colluding servers to protect the data owner's database and the user's prompt. It enforces restrictions and verification to defend against malicious users and tightly bounds the information leakage of the database. The experiments show that P2^2RAG is 3--300×\times faster than the state-of-the-art PRAG for k=16k = 16--10241024.