Incremental Retrieval-Augmented Generation for Human-in-the-Loop Decision Support

Federico Mazzoni; Andrea Failla

cs.HCOct 8, 2026

Incremental Retrieval-Augmented Generation for Human-in-the-Loop Decision Support

Authors: Federico Mazzoni, Andrea Failla

Organizations: Independent Researcher · Institute of Information Science and Technologies (ISTI), National Research Council (CNR), Italy

Abstract

We introduce iRAG, a human-in-the-loop decision-support system combining Retrieval-Augmented Generation (RAG) with Hybrid Decision-Making. Unlike conventional RAG systems, relying on static Knowledge Bases (KBs), our proposal addresses settings where the KB is initially empty and is progressively populated through human decisions. The system stores previous decisions as retrievable precedents and moves from silent observation to evidence-grounded contestation and, finally, to autonomous handling with abstention. Against an independent baseline, iRAG matches final-decision error under 10% drift without retrieval decay and reduces it under 40% drift across all configurations. Retrieval decay improves performance on changed knowledge while increasing error on stable knowledge, evidencing a stability--plasticity trade-off rather than a general advantage. Human oversight is an architectural provision of the workflow; its usability with real human users remains to be evaluated.

Figures & tables

Figure 1: iRAG’s three states. Transition between them is governed by Fading Empirical Accuracy .

Aspect	HDM	iRAG
Adapted object	Predictive model	External retrieval KB
Base model	Updated incrementally as observations arrive	LLM parameters remain fixed
New observation	Updates model parameters, statistics, or structure	Appended as an explicit input–decision record
Past observations	Absorbed into the model state and need not be revisited	Retained explicitly and remain individually retrievable
Adaptation to change	Model updates, drift adaptation, or pruning	Temporal weighting of retrieval relevance
Dependence on history	Inference depends primarily on the learned model state	Naive retrieval compares against all stored records

Table 1: Comparison between Incremental-Learning-based HDM and iRAG.

Figure 2: Mean trajectories across ten repetitions for all eight configurations. Rows compare coupled decay, RAG-only decay, FEA-only decay, and no decay; columns compare 10% (left) and 40% (right) drift. Shaded bands show one population standard deviation, vertical dotted lines separate quarters, and horizontal dashed lines mark the state thresholds. Each column repeats the corresponding independent, non-decaying Controller-free RAG-with-defer trajectory.

	10%				40%
Metric	R:D F:D	R:D F:ND	R:ND F:D	R:ND F:ND	R:D F:D	R:D F:ND	R:ND F:D	R:ND F:ND
Human-only baseline	$11.47\pm 0.52$				$17.22\pm 0.68$
Controller-free RAG-with-defer error	$5.50\pm 0.62$				$14.03\pm 0.73$
Final-decision error	$7.59\pm 0.97$	$6.99\pm 1.20$	$5.75\pm 0.42$	$5.52\pm 0.41$	$10.52\pm 0.68$	$9.62\pm 1.43$	$10.36\pm 0.87$	$9.51\pm 0.82$
Abstain (SO)	$15.73\pm 0.65$	$15.68\pm 0.59$	$14.97\pm 0.36$	$15.17\pm 0.32$	$15.71\pm 0.71$	$15.70\pm 0.62$	$14.97\pm 0.36$	$15.12\pm 0.36$
Abstain (SC)	$2.63\pm 0.72$	$2.28\pm 0.58$	$0.03\pm 0.10$	$0.02\pm 0.08$	$2.83\pm 0.54$	$2.27\pm 0.40$	$0.08\pm 0.09$	$0.02\pm 0.07$

Table 2: Mean

\pm

sample standard deviation across ten repetitions. Rates are percentages except for FEA; final-state entries are repetition counts. R and F denote RAG and FEA decay, respectively; R:ND is the incremental-RAG benchmark (

\lambda_{\mathrm{RAG}}=1

). Controller-free RAG uses

\lambda_{\mathrm{RAG}}=1

, so its dataset-level result spans all four columns.

Human-AI Decision Making

Human-AI decision making studies how people and AI systems share, support, or delegate decisions across tasks and institutions.

See all 175 papers

Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generation
May 2, 2026Peiyang Liu, Qiang Yan, Ziqiang Cui +3Retrieval-Augmented Generation LLM Hallucination Mitigation
National Engineering Research Center for Software Engineering, Peking University Beijing, China · PX Securities Shenzhen, China · City University of Hong Kong Hong Kong SAR, China +2
Rethinking Retrieval-Augmented Generation as a Cooperative Decision-Making Problem
Feb 21, 2026Lichang Song, Ting Long, Yi ChangRetrieval-Augmented Generation RAG Robustness
Jilin University
Hypothesis-Conditioned Query Rewriting for Decision-Useful Retrieval
Mar 19, 2026Hangeol Chang, Changsun Lee, Seungjoon Rho +2Retrieval-Augmented Generation Query Reformulation
Graduate School of AI, KAIST, Republic of Korea · School of Electrical Engineering, KAIST, Republic of Korea · Department of Industrial Engineering, Yonsei University, Republic of Korea

May 2, 2026J/K move · Enter open · S save

Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generation

Peiyang Liu, Qiang Yan, Ziqiang Cui, Di Liang+2

National Engineering Research Center for Software Engineering, Peking University Beijing, China · PX Securities Shenzhen, China · City University of Hong Kong Hong Kong SAR, China+2

Retrieval-Augmented Generation LLM Hallucination Mitigation Adversarial Robustness Retrieval Robustness RAG Robustness

Open paper AskPDF arXiv

Standard Retrieval-Augmented Generation (RAG) systems predominantly rely on semantic relevance as a proxy for utility. However, this assumption collapses in realistic decision-making scenarios where user queries are laden with cognitive biases, such as false premises or confirmation bias. In such cases, maximizing relevance paradoxically promotes the retrieval of sycophantic evidence that reinforces hallucinations, a critical failure we term the ``Relevance-Robustness Gap''. To bridge this gap, we propose CoRM-RAG (Counterfactual Risk Minimization for RAG), a framework that aligns retrieval with decision safety rather than mere similarity. Grounded in causal intervention, we introduce a Cognitive Perturbation Protocol to simulate user biases during training, which is then distilled into a lightweight Evidence Critic. This scoring module learns to identify documents that possess sufficient evidential strength to steer the model toward correctness despite adversarial query perturbations. Extensive experiments on decision-making benchmarks demonstrate that CoRM-RAG significantly outperforms strong dense retrievers and LLM-based rerankers in adversarial settings, while enabling effective risk-aware abstention through reliable robustness scoring. Our code is available at https://github.com/PeiYangLiu/CoRM-RAG.git.

Abstract

Human-AI Decision Making

Explore similar work

Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generation

Rethinking Retrieval-Augmented Generation as a Cooperative Decision-Making Problem

Hypothesis-Conditioned Query Rewriting for Decision-Useful Retrieval