FCPRAG: Fusion-Controller Parametric Retrieval-Augmented Generation for Stable Multi-Passage LoRA Injection
Authors: Jinchang Zhu, Jindong Li, Yi Ding, Xiaojian Nie, Rong Fu, Shuangyong Song, Haowei He, Menglin Yang
Organizations: The Hong Kong University of Science and Technology (Guangzhou) · University of Macau · Institute of Artificial Intelligence (TeleAI), China Telecom
Parametric retrieval-augmented generation (PRAG) injects retrieved evidence into a large language model (LLM) through passage-specific LoRA adapters, reducing reliance on long in-context prompts. When multiple passages are retrieved for the same query, however, evidence-level fusion becomes a bottleneck: equal-weight merging can amplify weak or conflicting evidence, and translating retrieval signals into fusion weights often requires fragile global tuning. We propose FCPRAG, a fusion-controlled parametric RAG framework that adds a lightweight controller for retrieval-conditioned, sample-level adapter fusion. The controller predicts per-passage fusion scores together with sample-level calibration signals, including a mixing gate and an adaptive temperature, enabling fusion that stays selective under informative retrieval signals and conservative under uncertainty. FCPRAG is trained with merge-aware supervision derived from each adapter's marginal contribution within a multi-adapter merge, using training data only. We further show that a single dataset-level temperature is suboptimal under heteroscedastic retrieval uncertainty, motivating sample-level adaptation. Experiments on HotpotQA, 2WikiMultiHopQA, PopQA, and ComplexWebQuestions (CWQ) across three LLM backbones show that FCPRAG consistently improves F1 over standard RAG and parametric RAG baselines, with gains of up to 4.65% on 2WikiMultiHopQA and 7.55% on CWQ, while also reducing tuning cost and improving robustness under retrieval perturbations.
Figures & tables
Method
Parametric evidence injection
Fusion in parameter space
Sample-level calibrated fusion
Uniform-prior fallback
RAG Lewis et al. 2020
×
×
×
×
PRAG Su et al. 2025a
✓
✓
×
×
DyPRAG Tan et al. 2025
✓
✓
×
×
Poly-PRAG Su et al. 2025b
✓
✓
×
×
FCPRAG (Ours)
✓
✓
✓
✓
Table 1: Capability comparison across RAG, PRAG-family systems, and FCPRAG.
Figure 1: Overview of FCPRAG . Given a query, retrieve multiple passages and load their passage-specific LoRA adapters (offline-trained). A Fusion Controller predicts per-passage weights and per-sample calibration, then a weighted adapter merger (WAM) merges the adapters for parametric injection (optionally combined with in-context passages).
Base LLM
Method
2WikiMultiHopQA
HotpotQA
PopQA
CWQ
Avg. Rank
Compare
Bridge
Inference
Compose
Avg.
Bridge
Compare
Llama-3.2-1B
Vanilla
36.48
34.23
13.80
5.60
22.53
9.85
37.84
14.56
28.52
8.00
Standard RAG
41.22
39.31
15.57
7.22
25.83
11.42
39.39
17.15
31.78
6.38
PRAG
46.83
44.32
17.16
8.55
29.22
12.99
41.84
19.16
32.09
3.62
PRAG-Combine
41.03
41.26
19.37
8.21
27.47
21.35
42.73
32.94
31.82
4.31
DyPRAG
41.53
37.98
18.67
8.21
26.60
10.84
42.91
18.69
31.92
5.31
Table 2: Main results (F1, %) on 2WikiMultiHopQA, HotpotQA, PopQA, and ComplexWebQuestions (CWQ). Suffix “-Combine” denotes results under the Combine setting. All baseline methods (Vanilla, Standard RAG, PRAG, DyPRAG) were manually reproduced under the same experimental setup for fair comparison. Avg. Rank is the mean rank (lower is better) over the 8 non-Avg columns within each backbone block. Best results are highlighted in bold and second-best results are underlined .
Base LLM
Method
Cmp.
Br.
Inf.
Compst.
Avg.
Qwen2.5-1.5B
PRAG
43.85
35.41
20.17
12.77
28.05
PRAG-Combine
40.62
37.95
18.03
7.58
26.04
Random Ctrl.
44.39
36.05
16.10
7.95
26.12
Random Ctrl.-Combine
38.98
41.87
18.32
7.32
26.62
FCPRAG (Ours)
45.59
42.89
17.81
13.18
29.87
FCPRAG -Combine
38.97
43.73
19.04
7.78
27.38
Table 3: Ablation on 2WikiMultiHopQA (F1 scores, Qwen2.5-1.5B and Llama-3-8B). We compare uniform fusion (PRAG), a random (untrained) fusion controller, and our learned fusion controller ( FCPRAG ) under Parametric-only and Combine settings.
Figure 2: Calibration diagnostics on HotpotQA. (a) F1 over global blend α and temperature T for a score-only fusion baseline. (b) Learned sample-wise calibration outputs (g,τ) from FCPRAG .
Figure 3: Tuning cost vs. performance on 2WikiMultiHopQA (Combine): each point is macro-average F1 over the four 2Wiki subtasks. Circles denote grid-selected global calibration; stars denote FCPRAG ’s learned calibration.
Figure 4: Retrieval perturbation robustness on 2WikiMultiHopQA (Llama-3-8B, Parametric-only; K=3 ). We report replace-one and repeat-one perturbations, with each panel normalized by the method’s own unperturbed score (none =1 ).
Parametric Retrieval-Augmented Generation (PRAG) encodes external documents into lightweight parameter modules that can be retrieved and merged at inference time, offering a promising alternative to in-context retrieval augmentation. Despite its potential, many PRAG implementations train document adapters with task-supervised objectives, which may cause each adapter to encode both document-specific facts and reusable task-solving behavior. This entanglement may make adapter composition less reliable: when multiple adapters are merged at inference time, their overlapping task behaviors can accumulate together with document-specific updates, potentially making the merged adapter less stable and less focused on the intended document knowledge. To examine this issue, we explore Orthogonal Subspace Decomposition (OSD), an adapter-training setup that separates reusable task behavior from document-specific knowledge adapters. Concretely, we first train a Task LoRA to capture reusable task behavior, and then train document LoRAs to encode document-specific knowledge in a orthogonal subspace. This setup provides a controlled way to examine how orthogonalizing task and document LoRA updates affects adapter composition in multi-document PRAG. Experiments across multiple knowledge-intensive tasks and model scales suggest that this orthogonalization strategy can improve compositional robustness in parametric RAG, especially when multiple document adapters are merged.
Weihang Su, Hanwen Zhang, Qingyao Ai +1
Department of Computer Science and Technology, Tsinghua University
Retrieval augmented generation (RAG) depends critically on the quality and granularity of retrieved evidence. Large retrieval units preserve context but often introduce irrelevant content, which can dilute answer bearing evidence and worsen long context utilization. Fine-grained units are more compact, but they may be difficult to retrieve reliably because short chunks can lack semantic, lexical, or bridging cues needed to match the query. We propose Uncertainty-aware Multi-Granularity RAG (UMG-RAG), a training-free hybrid retrieval framework that treats chunk granularity as query-specific reliability estimation. Instead of training a new retriever or modifying the generator, UMG-RAG uses existing dense and sparse retrievers as complementary experts across multiple chunk granularities. For each query, it converts each expert-granularity score list into an evidence distribution, estimates reliability from distribution entropy, and fuses candidates according to query-specific semantic, lexical, and granularity confidence. We further introduce UMGP-RAG, a parent promotion variant that uses fine-grained hits to locate relevant evidence while returning broader non-redundant parent chunks for local coherence. Experiments on question answering benchmarks show that uncertainty-aware fusion and parent promotion improve generation quality while maintaining a lightweight, plug-and-play retrieval pipeline.
Hoin Jung, Xiaoqian Wang
Elmore Family School of Electrical and Computer Engineering Purdue University West Lafayette, IN 47907
Retrieval-Augmented Generation (RAG) significantly enhances Large Language Models (LLMs) by providing access to external knowledge. However, current research primarily focuses on retrieval quality, often overlooking the critical ''integration bottleneck'': even when relevant documents are retrieved, LLMs frequently fail to utilize them effectively due to conflicts with their internal parametric knowledge. In this paper, we argue that implicitly resolving this conflict in a single generation pass is suboptimal. We introduce GuarantRAG, a framework that explicitly decouples reasoning from evidence integration. First, we generate an ''Inner-Answer'' based solely on parametric knowledge to capture the model's reasoning flow. Second, to guarantee faithful evidence extraction, we generate a ''Refer-Answer'' using a novel Contrastive DPO objective. This objective treats the parametric Inner-Answer as a negative constraint and the retrieved documents as positive ground truth, forcing the model to suppress internal hallucinations in favor of external evidence during this phase. Finally, rather than naive concatenation or using the DPO trained model directly, we propose a joint decoding mechanism that dynamically fuses the logical coherence of the Inner-Answer with the factual precision of the Refer-Answer at the token level. Experiments on five QA benchmarks demonstrate that GuarantRAG improves accuracy by up to 12.1% and reduces hallucinations by 16.3% compared to standard and dynamic RAG baselines.
Zhengyi Zhao, Shubo Zhang, Zezhong Wang +7
The Chinese University of Hong Kong · Ministry of Education Key Laboratory of High Confidence Software Technologies, CUHK · University of International Relations +2