Knowing Before Answering: Decoding Language Models for Reliable RAG

Syed Mahbubul Huq; Christopher Child; Tillman Weyde; Pranava Madhyastha

cs.CLAug 27, 2026

Knowing Before Answering: Decoding Language Models for Reliable RAG

Authors: Syed Mahbubul Huq, Christopher Child, Tillman Weyde, Pranava Madhyastha

Organizations: City St George’s, University of London · The Alan Turing Institute

Abstract

In Retrieval-Augmented Generation (RAG), retrieval may provide insufficient or conflicting information needed to answer a question. The system should not only know when to answer but also be able to identify cases in which the documents provided in RAG are insufficient or contain conflicting information. This can be framed as a three-way classification problem, where we use the model's internal signals to determine whether the provided information in the input can be classified as sufficient, insufficient, or conflicting. We create a controlled benchmark dataset that replicates a RAG setup with fictitious information and labels each instance as answerable, insufficient, or conflicting. We use hidden activations and attention-derived features as inputs to train a lightweight linear model to distinguish among the three classes. Across 16 language models spanning different architectures and a range of model sizes, our feature-based router consistently outperforms prompting-based baselines and the performance of specialised RAG-models. We further conduct analyses into the information dynamics of the models. We show that the most informative signals for the classification are available in the middle layers, with hidden activation states being more effective than attention values or the MLP-feature outputs in most of the tested models. Overall, our results suggest that language models internally encode whether retrieved evidence is sufficient to support answering, and that this signal can be decoded reliably for RAG triage.

Figures & tables

	Prompt-based Techniques						Ours
Model	ATTR [ Zhou et al., 2023 ]	KRE [ Ying et al., 2024 ]	FaithRAG [ Zhang et al., 2025b ]	OPIN_INSTR [ Zhou et al., 2023 ]	Vanilla	NO_CTX	Acc.	Macro-F1	FAR
Falcon-H1-Tiny-90M	0.34	0.34	0.35	0.34	0.33	0.33	0.69	0.68	0.13
Falcon3-7B-1.58bit	0.35	0.35	0.31	0.33	0.22	0.32	0.76	0.73	0.12
OLMo-2-1B	0.31	0.36	0.33	0.33	0.29	0.33	0.75	0.73	0.14
OLMo-2-1B-SFT	0.35	0.32	0.31	0.35	0.27	0.32	0.77	0.76	0.11
OLMo-2-1B-DPO	0.35	0.32	0.31	0.35	0.27	0.32	0.77	0.76	0.11

Table 1: Comparison of accuracy with prompt-based, RAG-specialised, and surface-text baselines. ATTR, KRE, FaithRAG, OPIN_INSTR, Vanilla, and NO_CTX are prompt-based baselines. RA denotes our Router (Activation Features). For FAR, lower values are better.

Model	Overall						Answer		Refuse		Conflict
	Acc.		Macro-F1		FAR
	Base	RA	Base	RA	Base	RA	Base	RA	Base	RA	Base	RA
gpt-oss-20b	0.73	0.82	0.75	0.81	0.19	0.07	0.66	0.69	0.94	0.93	0.58	0.84
Qwen3-4B-Instruct	0.70	0.88	0.70	0.88	0.28	0.08	0.73	0.85	0.66	0.87	0.76	0.94
Olmo-3.1-32B-Instruct	0.61	0.90	0.70	0.90	0.19	0.06	0.41	0.86	0.97	0.93	0.44	0.90

Table 2: Comparison of average accuracy of the best performing baselines (ATTR, KRE, FaithRAG, and OPIN_INSTR) and our router-based model (RA) on the top three highest accuracy language models. The table also compares category-wise accuracy for each language model in correctly classifying Answer, Refuse, and Conflict states. For each metric, the best value is highlighted (Higher value is better for Acc. and Macro-F1. Lower value is better for FAR).

Figure 1: Layer-wise validation accuracy of three routing strategies across sixteen language models. Each subplot shows validation accuracy (y-axis) as a function of transformer layer index (x-axis) for three router types: Hidden State Router (blue), MLP Router (red), and Attention Weight Router (green). Filled circles mark the best-performing layer per router. The heatmap strip below each plot shows the relative best-layer position (best layer / total layers), where darker colours indicate earlier layers and brighter colours indicate later layers. H, M, and A denote Hidden State, MLP, and Attention routers, respectively. The annotation box reports the total layer count and peak validation accuracy with its layer index for each router. An enlarged version of this figure is provided in Figure 7 , Appendix N .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Granite-3.1-8B-Instruct	OLMo-2-1B	Qwen3-4B-Instruct
Previous Val Acc (baseline)	0.90	0.73	0.91
Layer aggregation
Top3-H	0.91	0.80	0.89
Top3-M	0.90	0.81	0.91
All-H	0.90	0.78	0.91
All-M	0.90	0.80	0.90

Appendix

Table 3: Validation accuracy comparison between the baseline setup and ablation variants across three models. H means hidden state features, M means MLP block outputs from transformer. Top N suggests selecting the top-performing layers for training and testing the router. All-H/M suggests combining all the layers for training and testing the router.

Figure 3: Mean accuracy across language models on a subset of 80 examples (40 Refuse, 40 Conflict) under vector steering on hidden state, MLP output, and attention output. The dashed horizontal line indicates the baseline accuracy. Layers are downsampled to 10 layers for ease of comparison.

Model Family	Variant	Parameters	Type	Architecture
Falcon	Falcon-H1-Tiny-90M	90M	Instruct	Hybrid
OLMo	OLMo-2-1B	1B	Base	Dense
OLMo	OLMo-2-1B-SFT	1B	SFT	Dense
OLMo	OLMo-2-1B-DPO	1B	DPO	Dense
OLMo	OLMo-2-1B-Instruct	1B	Instruct	Dense
Qwen	Qwen3.5-2B	2B	Base	Dense

Appendix

Table 4: Model Suite Overview

Figure 5: Patching on different tested language models. Label on y-axis, such as refuse $\rightarrow$ answer means that a hidden state from a refuse example is inserted into an answer example at the patch site. The x-axis shows the patch layer and all subsequent layers, while each cell reports the proportion of examples whose downstream router prediction matches the source class. Higher values after patching indicate that the selected layer acts as a causal control point.

Model	Accuracy	Macro-F1
Falcon3-7B-1.58bit	0.91	0.63
Granite-3.1-8B-Instruct	0.92	0.65
Llama-3.2-3B	0.84	0.65
Qwen3-4B-Instruct	0.91	0.63

Appendix

Table 5: Zero-shot transfer to RAGTruth. The routers are evaluated without any RAGTruth training signal or in-domain adaptation.

Model	Acc.@80%	FAR@80%	Acc.@50%	FAR@50%
Falcon-H1-Tiny-90M	0.754	0.064	0.833	0.021
Falcon3-7B-1.58bit	0.819	0.080	0.901	0.032
Llama-3.2-3B	0.904	0.046	0.957	0.012
Phi-mini-MoE-Instruct	0.904	0.054	0.964	0.017
OLMo-2-1B	0.827	0.058	0.890	0.024
OLMo-2-1B-SFT	0.860	0.055	0.922	0.012

Appendix

Table 6: Accuracy and FAR across confidence thresholds.

Model	Setting	Feature	Acc. $k=5$	Acc. $k=10$	Acc. $k=15$	Acc. $k=20$	FAR $k=20$	Conflict-F1 $k=20$
Qwen3-4B-Instruct	matched- $k$	Last token	0.858	0.842	0.804	0.821	0.113	0.843
Qwen3-4B-Instruct	matched- $k$	Doc-end mean	0.575	0.554	0.454	0.500	0.312	0.639
Qwen3-4B-Instruct	matched- $k$	Doc-token mean	0.637	0.550	0.533	0.454	0.325	0.608
Llama-3.2-3B	matched- $k$	Last token	0.838	0.825	0.746	0.717	0.169	0.724
Llama-3.2-3B	matched- $k$	Doc-end mean	0.588	0.537	0.487	0.517	0.312	0.616
Llama-3.2-3B	matched- $k$	Doc-token mean	0.613	0.537	0.487	0.450	0.350	0.646

Appendix

Table 7: Long-context ablation as the number of retrieved documents increases. Matched-

k

trains and tests at the same context size, while train

k=5

trains only on five-document contexts and evaluates on progressively longer contexts. FAR and Conflict-F1 are reported at

k=20

Model	Real-label Acc.	Random-label Acc.
Granite-3.1-8B-Instruct	0.88	$0.333\pm 0.012$
OLMo-3.1-32B-Instruct	0.90	$0.328\pm 0.012$
gpt-oss-20b	0.82	$0.338\pm 0.014$
Qwen3.5-27B	0.91	$0.330\pm 0.009$
Qwen3.5-2B	0.85	$0.328\pm 0.011$
Qwen3.5-9B	0.91	$0.326\pm 0.019$

Appendix

Table 8: Random-label control. Real-label accuracy is taken from the main results in Table 1 ; random-label accuracy is reported as mean

\pm

standard deviation.

	Three-Way			Binary			Three-Way
Model	Acc.	Macro-F1	FAR	Acc.	Macro-F1	FAR	CONFLICT Acc.
Granite-3.1-8b-Ins	0.879	0.870	0.072	0.877	0.863	0.099	0.900
OLMo-32B	0.904	0.904	0.062	0.900	0.887	0.072	0.919
GPT-OSS	0.823	0.814	0.075	0.874	0.859	0.097	0.915
Qwen-27B	0.912	0.902	0.080	0.896	0.884	0.091	0.919
Qwen-2B	0.846	0.803	0.083	0.833	0.813	0.125	0.852

Appendix

Table 9: Comparison between the three-way and binary router formulations. For FAR, lower values are better. The final column reports CONFLICT accuracy for the three-way router.

Panel of Figure 7 — Figure 7: Zoomed layer-wise validation accuracy

Panel of Figure 8 — Figure 8: Zoomed layer-wise validation accuracy (continued)

RAG Robustness

Retrieval-augmented generation robustness studies how to keep systems reliable when retrieved evidence is noisy, misleading, incomplete, or malicious.

See all 128 papers

AB-RAG: Adaptive Budgeted Retrieval-Augmented Generation for Reliable Question Answering
Jun 27, 2026Ansh KamthanRetrieval-Augmented Generation LLM Answer Verification
Department of Artificial Intelligence and Machine Learning Manipal University Jaipur Jaipur, India
Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAG
Sep 27, 2026Suting Chen, Peichun Hua, Yunming XiaoRetrieval-Augmented Generation Evidence-Grounded QA
Northwestern University · The Chinese University of Hong Kong, Shenzhen
Parameters vs. Context: TRACE Fine-Tuning for Robust Retrieval-Augmented Generation
Sep 24, 2026Zhengchen Huang, Yundong Sun, Minrui Song +4RAG Robustness Retrieval-Augmented Generation
School of Information Science & Engineering, LinYi University, China · School of Computer Science and Technology, Harbin Institute of Technology, China

Jun 27, 2026J/K move · Enter open · S save

AB-RAG: Adaptive Budgeted Retrieval-Augmented Generation for Reliable Question Answering

Ansh Kamthan

Department of Artificial Intelligence and Machine Learning Manipal University Jaipur Jaipur, India

Retrieval-Augmented Generation LLM Answer Verification Confidence Estimation in Language Models Adaptive RAG LLM Reliability LLM Uncertainty Estimation

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) has become the standard way to ground large language models in external knowledge, yet most systems retrieve a fixed number of passages for every question regardless of its difficulty. This wastes computation on easy questions, starves hard ones, and gives no signal for when a generated answer can be trusted. With a growing share of question answering systems built on top of commercial language model APIs, a method that can decide how much to retrieve, and how far to trust its own answers, without retraining the underlying model, is of clear practical value. This paper presents AB-RAG (Adaptive Budgeted Retrieval-Augmented Generation), a training-free and backbone-agnostic framework that generates an answer, estimates its confidence from a combination of three signals, and then decides whether to stop or to retrieve more evidence, subject to a fixed retrieval budget. The estimator combines the model's own certainty, the agreement between the answer and the evidence, and the variance of the retrieval scores. For models that expose token probabilities the certainty signal is read directly; for closed APIs it is approximated by self-consistency, so the method works without access to model internals. Across three backbones and two datasets, the central result is that the confidence estimate reliably separates correct from incorrect answers on every backbone, reaching a clean split of 57.6% against 0% Exact Match between high- and low-confidence answers on a factoid dataset. The adaptive policy improves accuracy on capable backbones, and the study reports its negative and nuanced findings honestly, including a confidence signal that proved unsuitable for short answers and a retrieval signal whose sign was found and corrected through measurement. The entire study was carried out on a single consumer laptop with only a few dollars of API spend.

Abstract

RAG Robustness

Explore similar work

AB-RAG: Adaptive Budgeted Retrieval-Augmented Generation for Reliable Question Answering

Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAG

Parameters vs. Context: TRACE Fine-Tuning for Robust Retrieval-Augmented Generation