SoK: Agentic Retrieval-Augmented Generation (RAG): Taxonomy, Architectures, Evaluation, and Research Directions

Saroj Mishra; Suman Niroula; Umesh Yadav; Dilip Thakur; Srijan Gyawali; Shiva Gaire

cs.AIMar 7, 2026

SoK: Agentic Retrieval-Augmented Generation (RAG): Taxonomy, Architectures, Evaluation, and Research Directions

Authors: Saroj Mishra, Suman Niroula, Umesh Yadav, Dilip Thakur, Srijan Gyawali, Shiva Gaire

Organizations: University of North Dakota · Youngstown State University · University of Toledo · University of Missouri ·

Abstract

Retrieval-Augmented Generation (RAG) systems are increasingly evolving into agentic architectures where large language models autonomously coordinate multi-step reasoning, dynamic memory management, and iterative retrieval strategies. Despite rapid industrial adoption, current research lacks a systematic understanding of Agentic RAG as a sequential decision-making system, leading to highly fragmented architectures, inconsistent evaluation methodologies, and unresolved reliability risks. This Systematization of Knowledge (SoK) paper provides the first unified framework for understanding these autonomous systems. We formalize agentic retrieval-generation loops as finite-horizon partially observable Markov decision processes, explicitly modeling their control policies and state transitions. Building upon this formalization, we develop a comprehensive taxonomy and modular architectural decomposition that categorizes systems by their planning mechanisms, retrieval orchestration, memory paradigms, and tool-invocation behaviors. We further analyze the critical limitations of traditional static evaluation practices and identify severe systemic risks inherent to autonomous loops, including compounding hallucination propagation, memory poisoning, retrieval misalignment, and cascading tool-execution vulnerabilities. Finally, we outline key doctoral-scale research directions spanning stable adaptive retrieval, cost-aware orchestration, formal trajectory evaluation, and oversight mechanisms, providing a definitive roadmap for building reliable, controllable, and scalable agentic retrieval systems.

Figures & tables

Fig. 1: High-level progression from single-pass retrieval-augmented generation to iterative retrieval and Agentic RAG. This demonstrates the architectural shift from static, one-shot context utilization to explicit multi-step control over retrieval, reasoning, and termination, conceptually anchoring the systematization presented in this paper.

Fig. 2: The architectural evolution from static one-shot RAG pipelines to the Agentic RAG POMDP formulation. The Agentic framework replaces linear generation with a cyclic control policy ( $\pi_{\theta}$ ) managing a persistent memory state ( $\mathcal{M}_{t}$ ).

Agentic Property	POMDP Component	Operational Interpretation
Iterative Control	Stochastic Policy $\pi_{\theta}$	Non-deterministic action selection governing the loop
Dynamic Retrieval	Action Space $\mathcal{A}$	Query generation treated as a discrete runtime action
Tool Mediation	Observation Model $\mathcal{O}$	External API interaction returning state context
State Persistence	Belief State $b_{t}\approx\mathcal{M}_{t}$	Memory acts as a tractable approximation of hidden state
Termination	Finite Horizon $T_{max}$	Constrained loop depth to prevent infinite execution

TABLE I: Mapping Agentic System Properties to POMDP Formalization

Feature	Active RAG	Agentic RAG
Trigger Mechanism	Log-probability thresholds or token heuristics	Policy-driven reasoning and explicit tool-calling
Control Flow	Single-pass, forward-generating	Iterative, multi-step planning loops
Planning Explicitness	Implicit during generation	Explicit deliberative phase
Context Management	Append-only (accumulates fetched text)	Read/Write/Prune capabilities over working memory
Failure Handling	Cannot self-correct prior token generation	Can discard poor retrieval and explicitly try new query

TABLE II: Architectural Distinctions Between Active RAG and Agentic RAG

Fig. 3: Taxonomy of Agentic RAG systems across architecture, retrieval strategy, reasoning paradigm, and memory/context management. This structural mapping demonstrates how orthogonal control-flow decisions combine to form distinct, reproducible agentic archetypes.

Archetype	Topology	Retrieval	Reasoning	Memory/Context	Policy	Representative anchors
Baseline grounded generation	Single-agent	One-shot	Minimal / linear	Minimal; optional filtering	Heuristic	RAG [ 1 ] ; surveys [ 40 , 41 ]
Iterative evidence accumulation	Single-agent	Iterative	CoT / ReAct	Dynamic context construction	Heuristic/Hybrid	IRCoT [ 9 ] ; Iter-RetGen [ 5 ] ; LangChain agents [ 46 ]
Reflective refinement	Single-agent	Self-refining	Reflection-based	Episodic critique; pruning	Hybrid/Learned	Self-RAG [ 6 ] ; RARR [ 57 ] ; Reflexion [ 13 ]
Role-separated orchestration	Planner–Executor	Iterative/self-refining	Planning + execution	Executor logging; pruning	Hybrid	HuggingGPT [ 50 ] ; MRKL [ 11 ] ; OpenAI Agents [ 45 ]
Distributed knowledge work	Multi-agent	Iterative/mixed	ReAct/reflective	Agent-local episodic; aggregation	Hybrid	AutoGen [ 52 , 55 ] ; CrewAI [ 54 ] ; LangGraph [ 53 ] ; ADK [ 44 ]
Memory-centric long-horizon	Any	Mixed	Reflection common	Persistent+episodic; refresh	Hybrid/Learned	MemoryBank [ 63 ] ; MemGPT [ 64 ] ; AgeMem [ 59 ]

TABLE III: Consolidated taxonomy mapping archetypes to their core Agentic RAG attributes.

System	Planning Topology	Retrieval Strategy	Memory Model	Tool Coordination
ReAct	Linear reasoning loop	Iterative retrieval	Short-term scratchpad	Search tools
A-RAG	Hierarchical planning	Progressive retrieval	Hybrid memory	Multi-stage retrieval tools
Search-R2	Actor–refiner architecture	Search-integrated reasoning	Episodic memory	Retrieval and repair tools
LangGraph workflows	Graph-based control flow	Tool-triggered retrieval	Persistent state	Structured tool orchestration
AutoGen agents	Multi-agent coordination	Tool-driven retrieval	Shared memory	Multi-agent tool usage

TABLE IV: Mapping Representative Agentic RAG Systems to the Proposed Taxonomy Dimensions

Fig. 4: Core architectural components and control-flow relationships within a generalized Agentic RAG system. This demonstrates how the Reasoning Engine coordinates bidirectionally with Memory Systems and delegates execution to the Tool Orchestration Layer to maintain verifiable state control.

Module	Inputs	Outputs	Control Signals	Feedback Loops
Planner	User Query, Global State	Sub-task Graph	Depth limits, Max steps	Self-correction on plan failure
Controller	Sub-task, Local Memory	Action / Tool Call	Confidence thresholds	Observation-triggered replanning
Orchestrator	Multi-Agent Outputs	Final Synthesis	Agent-routing logic	Cross-agent consensus voting

TABLE V: Architectural Decomposition of Agentic RAG Modules

Fig. 5: The closed-loop Perception-Planning-Action-Reflection (PPAR) cycle with Human-in-the-Loop (HITL) escalation. This demonstrates the structural necessity of verification loops: outputs failing constraint checks are returned as structured feedback, and unresolvable loops are escalated to prevent autonomous hallucination.

Fig. 6: Control-flow map demonstrating how Agentic RAG systems compose design patterns through explicit decisions over task decomposition, retrieval timing, iterative refinement, and orchestration. This structural mapping highlights the transition from linear pipelines to cyclic loops.

Design Pattern	Core Control Question	Termination Condition	Tradeoffs (Cost / Latency / Risk)	Representative Anchors
Plan-then-retrieve	“What subtasks must be answered before synthesis?”	All planned sub-questions answered	High upfront planning cost; risk of brittle initial plans	Self-Ask [ 29 ] , Plan-and-Solve [ 28 ]
Retrieve-reflect-refine	“Is retrieval needed? Are these passages relevant?”	Reflection indicates sufficiency or budget exhausted	High latency due to sequential query iterations	Self-RAG [ 6 ] , Iter-RetGen [ 5 ]
Decomp-based retrieval	“Given the current reasoning state, what is missing?”	Answer reached with adequate evidence	Extreme token accumulation; reasoning drift risk	IRCoT [ 9 ] , ReAct [ 12 ]
Tool-augmented loops	“Which heterogeneous tool to call now?”	Tool results stabilize answer or verifier halts	Variable latency; high risk of tool-syntax failure	Toolformer [ 10 ] , CRITIC [ 81 ]
Multi-agent collaboration	“Which agent role should handle this task?”	Cross-agent consensus reached	Massive token amplification; coordination overhead	AutoGen [ 52 ] , MetaGPT [ 84 ]
Self-verification	“Which claims require checking against the corpus?”	Verification passes or abstention is triggered	Doubles baseline latency; bounded by retriever recall	CoVe [ 85 ] , GopherCite [ 87 ]

TABLE VI: Comparison of Core Agentic RAG Design Patterns

Metric	Failure Dimension	Why It Fails Agentic Systems	Agentic Failure Case
BLEU / ROUGE	Lexical Rigidity	Primarily measures surface-level lexical overlap; ignores semantic consistency and factual keypoints [ 91 ] .	An agent correctly diagnoses a condition but uses synonyms not in the reference text, receiving a failing score despite factual accuracy [ 94 ] .
Exact Match (EM)	Binary Inflexibility	Offers no flexibility for valid aliases or the superfluous reasoning detail often generated by agents [ 91 ] .	An agent outputs the correct entity but is failed because it included a valid reasoning trace before the target word [ 91 ] .
Final-Answer Accuracy	Trajectory Blindness	Provides a ”black box” view; cannot determine if the agent correctly reasoned or merely guessed [ 92 ] .	A math agent reaches the correct final digit through mutually canceling calculation errors, hiding a fundamental planning breakdown [ 90 ] .
Success Rate (SR)	Credit Assignment	Non-diagnostic; identifies that a failure occurred but fails to pinpoint the bottleneck (retrieval vs. tool call) [ 90 ] .	An agent correctly writes code but fails execution due to a syntax timeout; SR marks it 0, obscuring the successful reasoning [ 92 ] .
Pass@k	Reliability Blindness	Focuses on best-case capability rather than the consistency required for enterprise deployment [ 89 ] .	An agent succeeds once in ten attempts; while technically ”capable,” it is dangerously unreliable for production tasks [ 89 ] .

TABLE VII: Metric Failure Analysis: Why Standard Evaluation Fails for Agentic RAG

Evaluation Framework	Targeted Capability	Limitation for Agentic RAG
RGB & FaithEval [ 98 , 96 ]	Noise robustness, negative rejection, and counterfactual adherence.	Assumes a single forward-pass; cannot evaluate dynamic query reformulation.
RAGBench (TRACe) [ 95 ]	uTilization, Relevance, Adherence, and Completeness across industries.	Static dataset; fails to capture multi-step tool use or environment interaction.
RAGEval & CRAG [ 93 ]	Keypoint-based factual accuracy and multi-hop reasoning coverage.	Evaluates final output via mock APIs but lacks metrics for reasoning efficiency or cost.

TABLE VIII: Synthesis of Current RAG Evaluation Frameworks and Agentic Limitations

Fig. 7: The Agentic RAG Evaluation Pipeline. This framework demonstrates the necessary structural shift from terminal output scoring to multi-layered assessment, capturing component-level tool accuracy, trajectory-level reasoning coherence, and system-level outcome fidelity.

Framework	Orchestrator Model	Planner / Control Flow	Memory Routing	Tool Abstraction
LangGraph [ 105 ]	Cyclic Directed Graphs	State-machine nodes	Persistent checkpointing	Wrapped Python functions
Google ADK [ 106 ]	Hierarchical Composition	Deterministic routing loops	Shared contextual state	Model Context Protocol
CrewAI [ 105 ]	Role-based Sequential	Process-driven delegation	Structured persona memory	Assigned capability arrays
AutoGen [ 105 ]	Asynchronous Chat	Event-driven conversation	Message history logs	Executable code blocks
LlamaIndex [ 105 ]	Query Pipelines	Data-driven routing	Index-centric retrieval	LlamaPack interfaces

TABLE IX: Mapping Industrial Frameworks to Agentic RAG Architectural Modules

Failure Mode	Pipeline Stage	Root Cause	Agentic Amplification Factor	Severity / Impact
Retrieval Drift	Iterative Retrieval	Semantic divergence in query reformulation	Compounds across iterations without convergence guarantees	Moderate (Degrades accuracy, increases cost)
Hallucination	Generation	Context insufficiency or positional bias	Hallucinated outputs become retrieval context in next iteration	High (Corrupts downstream logic and planning)
Tool Misuse	Tool Orchestration	Malformed queries, API failures	Errors propagate through dependent downstream tool calls	High (Causes systemic execution crashes)
Prompt Injection	Retrieval Context	Adversarial content in retrieved documents	Each retrieval iteration exposes new injection surface	Critical (Enables unauthorized data exfiltration)
Memory Poisoning	Memory Systems	Adversarial manipulation of persistent state	Corrupted memory affects all future sessions and decisions	Critical (Persistent, cross-session compromise)
Feedback Instability	Reflection	Reflection module shares generator biases	Self-critique may approve errors or reject correct outputs	Moderate (Prevents loop convergence)

TABLE X: Structured Failure-Mode Categorization for Agentic RAG Systems

Fig. 8: The interdisciplinary mapping of the proposed doctoral-scale grand problems across foundational system dimensions and research time horizons. Addressing these challenges requires systemic integration rather than isolated optimization.

Grand Problem	Primary Risk	Theoretical Gap	Core Evaluation Metric	Interdisciplinary Domain
10.1 Stable Retrieval	Semantic drift and reasoning collapse	Lack of formal convergence proofs for context loops	State-transition convergence bounds	Control Theory, RL
10.2 Reasoning Evaluation	Undetected logical failures	Absence of intermediate trajectory verification	Trajectory inter-rater reliability	Formal Verification
10.3 Memory Security	Persistent episodic poisoning	No robust state modeling against latent triggers	Provable state recovery rate	Systems Security
10.4 Cost Orchestration	Token explosion and latency stacking	No budget-aware multi-agent routing optimization	Pareto efficiency (Compute vs. Accuracy)	Operations Research
10.5 Trust Calibration	Overconfidence in corrupted context	Lack of dynamic uncertainty bounds during retrieval	Expected Calibration Error (ECE)	HCI, Statistics

TABLE XI: Summary of Grand Research Problems and Interdisciplinary Roadmap for Agentic RAG

Agentic RAG

Agentic retrieval-augmented generation lets language models plan searches, use tools, and refine evidence before producing grounded answers.

See all 246 papers

Rethinking Knowledge Retrieval for Generation: A Survey on RAG Architectures and Applications
Oct 1, 2026Meghana Sunil, Shravya V, Shravan Venkatraman +1Retrieval-Augmented Generation RAG Security
School of Computer Science and Engineering, Vellore Institute of Technology, Chennai, India. · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE.
Agent-Orchestrated Adaptive RAG: A Comparative Study on Structured and Multi-Hop Retrieval
Jun 4, 2026Anuj Maharjan, Devinder Kaur, Richard MolyetRetrieval-Augmented Generation Multi-Hop Retrieval
Dept. of Electrical Engineering and Computer Science University of Toledo Toledo, OH, USA
The RAT: A Unified Bayesian Model for RAG Evaluation
Aug 25, 2026Pius von Däniken, Felix Matthias Saaro, Mark Cieliebak +1RAG Evaluation
Centre for Artificial Intelligence ZHAW School of Engineering

Oct 1, 2026J/K move · Enter open · S save

Rethinking Knowledge Retrieval for Generation: A Survey on RAG Architectures and Applications

Meghana Sunil, Shravya V, Shravan Venkatraman, Joe Dhanith PR

School of Computer Science and Engineering, Vellore Institute of Technology, Chennai, India. · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE.

Retrieval-Augmented Generation RAG Security RAG Evaluation RAG Robustness

Open paper AskPDF arXiv

Large Language Models (LLMs) have demonstrated remarkable fluency and versatility across natural language tasks but remain fundamentally limited by their static knowledge and susceptibility to hallucinations, especially in domains requiring up to date or attribute grounded information. Retrieval Augmented Generation (RAG) addresses these challenges by integrating external retrieval mechanisms with generative models, enabling dynamic, context aware generation grounded in verifiable data sources. This survey presents a comprehensive examination of RAG as a modular and evolving paradigm that enhances factual reliability, adaptability, and task alignment in LLM based systems. We formalize the RAG framework through its three foundational components retrieval, generation, and augmentation and survey state of the art methods spanning dense and sparse retrievers, fusion strategies, embedding optimizations, and reinforcement learning based retrieval policies. Anchored around four emerging axes efficiency, security, user centric interactivity, and complex reasoning we categorize recent innovations and highlight their implications for scalability, robustness, and personalization. The paper also reviews advances in evaluation protocols, domain specific applications, and architectural variants such as Naïve RAG, Advanced RAG, and Modular RAG. Finally, we identify persistent challenges and outline future directions aimed at advancing the integration of retrieval with LLMs for more grounded, interpretable, and controllable generation.

Abstract

Agentic RAG

Explore similar work

Rethinking Knowledge Retrieval for Generation: A Survey on RAG Architectures and Applications

Agent-Orchestrated Adaptive RAG: A Comparative Study on Structured and Multi-Hop Retrieval

The RAT: A Unified Bayesian Model for RAG Evaluation