Parametric Knowledge in RAG-SFT for Domain-Specific Document Generation

Julian Oestreich; Maximilian Bley; Frank Binder; Lydia Müller; André Alcalde; Maksym Sydorenkoq

cs.CLMar 24, 2026

Parametric Knowledge in RAG-SFT for Domain-Specific Document Generation

Authors: Julian Oestreich, Maximilian Bley, Frank Binder, Lydia Müller, André Alcalde, Maksym Sydorenkoq

Organizations: Institute for Applied Informatics (InfAI) at Leipzig University · CELUS GmbH, Munich

Abstract

Retrieval-Augmented Generation (RAG) fine-tuning has shown substantial improvements over vanilla RAG, yet most studies target document question answering, leaving open whether these gains transfer to specialized tasks. We study supervised RAG fine-tuning (RAG-SFT) for requirements document generation in the electronics engineering domain, adapting two 7B models under two different training data strategies. Because Rouge and BertScore poorly capture factuality on long technical text, we introduce C-FEX, a claim-based evaluation pipeline that attributes each response claim to its origin (augmented prompt or reference response), and propose Parametric Knowledge Precision (PKP), which isolates claims originating from the model's weights and measures their correctness. We show that a prior metric to assess parametric knowledge decomposes as PKP $\times$ PR, separating the rate of parametric output (PR) from its quality (PKP). Empirically, fine-tuned 7B models match or exceed a 72B baseline; standard metrics disagree with claim-based factuality and can mislead about fine-tuning gains; and, fine-tuning does not reinforce correct parametric knowledge but suppresses hallucination---models speak from their weights less often but far more reliably.

Figures & tables

Panel of Figure 1 — Figure 1: C-FEX : Claim Extraction + Fact Evaluation. (A) Extraction: For each example, claims are extracted from the generated response ( $\hat{\mathcal{C}}_{\mathcal{G}}$ ) and its sources (reference $\mathcal{C}_{\mathcal{R}}$ , context $\mathcal{C}_{\mathcal{C}}$ , and user query $\mathcal{C}_{\mathcal{U}}$ ). (B) Attribution: Each generated claim $\hat{\mathcal{C}}_{\mathcal{G}}$ is attributed to a source by matching against top-k candidate claims retrieved from each source. (C) Metrics: Claims and their attributions are aggregated for metric computations. E.g. excluding response claims attributed to the user query or context ( $\mathcal{S}_{\mathcal{U}}$ and $\mathcal{S}_{\mathcal{C}}$ ) allows computation of Parametric Knowledge Precision (PKP) to investigate the quality of parametric knowledge ( $\mathcal{P_{\textit{ref}}}+\mathcal{H}$ ) in different RAG-Generators.

Panel of Figure 2 — Figure 2: One dataset example for illustrating the use case. We prompt an RAG-Generator to generate b), a requirements engineering document, integrating information from a).

Table 1: Absolute scores (in percentage points) with relative finetuning gain of RAG-SFT in parentheses. We repeat the LLM-Judgment three times, compute the mean per-example standard deviation across those three runs and aggregate over all models and test cases: Prec ref

\pm 1.01

, Prec any

\pm 0.75

and Rec ref

\pm 0.76

(in pp).

Panel of Figure 3 — Figure 3: Spearman Rank Correlation of Rouge (R), BertScore (BS) and claim-based metrics.

Table 2: Absolute scores of the parametric knowledge metrics ( * = mean claims per example). Standard deviations across three LLM-Judgments, all models and test cases: PKP

\pm 1.89

, PR

\pm 1.09

and SK

\pm 0.87

(in pp).

Figure 4: Our data creation pipeline: We first extract text from the HTML-files, then prompt GPT-4o to 1) harmonize the text, which means to take any manufacturer-specific text (e.g., component ids, project titles) and change it to fictional nouns, and 2) to use a specific template for generating a structured format, with mandatory sections (e.g., title, functional goals, project architecture).

Figure 5: A model prediction, where Rouge-1 and claim-based Rec ref are disagreeing considerably: Rouge-1 states a high word overlap with the reference, but Rec ref shows very low reference-backed factuality. We see that almost every technical value in the predictions is not grounded in the reference (e.g. "input voltages (85 to 265 V AC)" "DC output of 48 V", "output power of 90 W", "gain bandwidth product of 4.5 MHz", "slew rate of 13 V/s"), but the predicted words and overall structure is very similar (reference: 454/640 tokens overlap ( $70.9\%$ ); prediction: 477/644 tokens overlap ( $74.1\%$ )). Those hallucinations, well hidden under domain-specific prose, can by and large lead to false conclusions—for instance, interpreting an increase in Rouge-1 after training as a successful run.

Panel of Figure 6 — Figure 6: Spearman Rank Correlation of Rouge (R), BertScore (BS) and claim-based metrics.

Figure 7: Attribution of response claims to either the augmented prompt (context and user query), the reference response (parametric reference facts) or to no source at all (hallucinations). On top are the average claims per response (e.g. Qwen2.5-7B (base) generates on average approx. 30 claims). The labels annotating each source fraction report the mean claims per-example for each source (e.g. Qwen2.5-7B (base) generates on average approx. 4 parametric reference claims).

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

Parameters vs. Context: TRACE Fine-Tuning for Robust Retrieval-Augmented Generation
Sep 24, 2026Zhengchen Huang, Yundong Sun, Minrui Song +4RAG Robustness Retrieval-Augmented Generation
School of Information Science & Engineering, LinYi University, China · School of Computer Science and Technology, Harbin Institute of Technology, China
KARE-RAG: Knowledge-Aware Refinement and Enhancement for RAG
Jun 3, 2025Yongjian Li, HaoCheng Chu, Yukun Yan +7Retrieval-Augmented Generation LLM Alignment
School of Biomedical Engineering, Tsinghua Medicine, Tsinghua University · 2Tsinghua Laboratory of Brain and Intelligence · School of Informatics, Xiamen University +6
Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAG
Sep 27, 2026Suting Chen, Peichun Hua, Yunming XiaoRetrieval-Augmented Generation Evidence-Grounded QA
Northwestern University · The Chinese University of Hong Kong, Shenzhen

Sep 24, 2026J/K move · Enter open · S save

Parameters vs. Context: TRACE Fine-Tuning for Robust Retrieval-Augmented Generation

Zhengchen Huang, Yundong Sun, Minrui Song, Shuanglong Yao+3

School of Information Science & Engineering, LinYi University, China · School of Computer Science and Technology, Harbin Institute of Technology, China

RAG Robustness Retrieval-Augmented Generation Knowledge Conflicts in Language Models

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) mitigates knowledge obsolescence and factual hallucination in large language models by introducing external context. However, when retrieved knowledge conflicts with the model's internal parametric knowledge, the model may either blindly follow misleading context or incorrectly rely on parametric knowledge, leading to unreliable responses. To address this issue, this paper proposes TRACE (Debate-TRace and Answer-Completeness rEgularized fine-tuning), a robust fine-tuning framework for RAG under knowledge conflicts. First, we propose a fine-tuning method that leverages multi-agent debate traces to extract correct candidates, incorrect candidates, and answer-shift patterns, providing fine-grained supervision for reliable knowledge-source selection. In addition, we design an answer completeness regularization mechanism to alleviate empty, overly short, and prematurely terminated responses via answer-tail token reinforcement and premature termination suppression. The fine-tuning objective combines correct-answer supervision, incorrect-candidate suppression, answer-tail token reinforcement, and premature termination suppression, enabling the model to use reliable external context, resist misleading or irrelevant retrieved content, and fall back to parametric knowledge when retrieved evidence is unreliable. Experiments across multiple knowledge-conflict scenarios and datasets show that TRACE improves robustness against misleading retrieved knowledge and reduces incomplete answers. These results demonstrate that multi-agent debate traces and answer completeness regularization jointly enhance knowledge-source selection, conflict robustness, and answer quality in RAG models. Our code is available at https://github.com/PHD-lanyu/TRACE.

Abstract

Retrieval-Augmented Generation

Explore similar work

Parameters vs. Context: TRACE Fine-Tuning for Robust Retrieval-Augmented Generation

KARE-RAG: Knowledge-Aware Refinement and Enhancement for RAG

Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAG