How Retrieved Context Shapes Internal Representations in RAG

Samuel Yeh; Sharon Li

cs.CLFeb 23, 2026

How Retrieved Context Shapes Internal Representations in RAG

Authors: Samuel Yeh, Sharon Li

Organizations: Department of Computer Science, University of Wisconsin-Madison

Abstract

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by conditioning generation on retrieved external documents, but the effect of retrieved context is often non-trivial. In realistic retrieval settings, the retrieved document set often contains a mixture of documents that vary in relevance and usefulness. While prior work has largely examined these phenomena through output behavior, little is known about how retrieved context shapes the internal representations that mediate information integration in RAG. In this work, we study RAG through the lens of latent representations. We systematically analyze how different types of retrieved documents affect the hidden states of LLMs, and how these internal representation shifts relate to downstream generation behavior. Across four question-answering datasets and three LLMs, we analyze internal representations under controlled single- and multi-document settings. Our results reveal how context relevancy and layer-wise processing influence internal representations, providing explanations of LLMs' output behaviors and insights for RAG system design.

Figures & tables

Figure 1: The overview of our analysis framework. For each query, we retrieve and rerank a set of documents, and group them as relevant, distracting, and random. We then control the input context of RAG with different type(s) of documents and obtain the hidden representations for comparative analysis.

Figure 2: Representations of prompts paired with semantically similar documents remain close to the no-document baseline, whereas semantically dissimilar documents cause the representations to drift away. We apply PCA on the last prompt token representations across different document types and plot them in 2D.

Figure 3: LLMs are more likely to abstain when context induces large representation shifts. For each context type, we compute the cosine similarity between the representations of with-context prompts and query, and categorize responses as correct, incorrect, or abstain. We show the result of Gemma3-27B. For other models, see Figure 7 .

Figure 4: Base LLMs do not have representation drifts across different context types. We apply PCA on representations of instruction-tuned models and base models. We show the result of Gemma3-27B on Trivia QA. For other models and datasets, see Figure 8 .

Base
	Easy			Hard
Context	Cor	Inc	Abs	Cor	Inc	Abs
Relevant	92.4	4.2	3.4	52.5	35.6	11.9
Distracting	79.5	15.8	4.7	14.8	72.4	12.8
Random	89.4	6.8	3.8	15.6	65.4	19.0
Instruction-tuned

Table 1: Instruction-tuned LLMs tend to abstain when the retrieval document is distracting or random, even if they can answer with the query alone. We report the percentage of correct (Cor), incorrect (Inc), and abstain (Abs) responses for both base and instruction-tuned LLMs. We show the result of Gemma3-27B on Trivia QA. For other models and datasets, see Table 11 .

Figure 5: Representations remain similar when a relevant document is present, regardless of other context. We perform PCA on the last prompt token representations for multiple-document contexts that include one relevant document, and plot them alongside representations with only a relevant document in 2D. We show the result of Gemma3-27B on Trivia QA. For other models and datasets, see Figure 9 .

Gemma3-27B
	Trivia QA		NQ		Pop QA		Strategy QA
Context	Easy	Hard	Easy	Hard	Easy	Hard	Easy	Hard
Relevant	90.4	65.2	79.8	62.1	91.0	70.5	72.7	44.3
+ Distracting	82.6	57.1	73.7	47.8	84.6	58.0	64.0	27.9
+ Random	87.7	60.2	79.7	48.6	84.0	69.8	58.3	30.8
Distracting	8.4	0.7	8.0	0.6	3.6	0.6	22.7	8.6

Table 2: Performance is preserved or even improved if at least one relevant document is presented in the input context. We report the percentage of correct responses for each LLM and dataset across different context types.

Figure 6: Representations with random documents are separated from others in earlier layers, and representations with relevant documents are drawn close to no-document representations in later layers. We perform PCA on the last prompt token representations across different layers and document types and plot them in 2D. We show the result of Llama4-17B on Trivia QA. For other models and datasets, see Figure 10 to 12 .

Gemma3-27B
	Trivia QA		NQ		Pop QA		Strategy QA
Context	Easy	Hard	Easy	Hard	Easy	Hard	Easy	Hard
Relevant Only	90.4	65.2	79.8	62.1	91.0	70.5	72.7	44.3
20 Documents	90.3	61.5	89.2	53.5	85.1	69.1	71.5	24.2
Llama4-17B
Relevant Only	89.8	62.8	85.9	64.0	92.1	79.8	76.3	35.3

Table 3: Performance is preserved when inputting 20 documents without filtering according to relevancy. We report the percentage of correct responses for each LLM and dataset across different context types.

	Trivia QA		NQ		Pop QA		Strategy QA
LLM	Easy	Hard	Easy	Hard	Easy	Hard	Easy	Hard
Gemma3-27B	150	50	117	83	66	134	155	45
Llama4-17B	158	42	118	82	93	107	140	60
Qwen3-80B	159	41	121	79	152	48	140	60

Table 4: Statistics of easy and hard queries for each LLM and dataset.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

LLM	Trivia QA	NQ	Pop QA		Strategy QA
Gemma3-27B	3.117	-1.787	-1.274		14.375
Llama4-17B	5.361	-2.849	2.135	**	8.449
Qwen3-80B	9.525	0.973	-0.663		12.216

Appendix

Table 5: Result of paired-sample one-tailed t-test. We report the t-statistic, where its magnitude indicates how easy we can distinguish the two distributions. *

p<0.05

; **

p<0.01

; ***

p<0.001

	Trivia QA		NQ		Pop QA		Strategy QA
LLM	Acc	AvgDist	Acc	AvgDist	Acc	AvgDist	Acc	AvgDist
Gemma3-27B	1.00 (0.00)	0.78 (0.01)	1.00 (0.00)	0.77 (0.02)	0.99 (0.01)	0.68 (0.01)	1.00 (0.00)	0.73 (0.01)
Llama4-17B	1.00 (0.00)	0.71 (0.01)	1.00 (0.00)	0.70 (0.01)	1.00 (0.00)	0.75 (0.01)	0.99 (0.00)	0.66 (0.01)
Qwen3-80B	0.98 (0.01)	0.54 (0.02)	0.97 (0.00)	0.61 (0.01)	1.00 (0.00)	0.66 (0.02)	0.97 (0.00)	0.44 (0.01)

Appendix

Table 6: In S1, the linear probe achieves both high accuracy and large distance.

	Trivia QA		NQ		Pop QA		Strategy QA
LLM	Acc	AvgDist	Acc	AvgDist	Acc	AvgDist	Acc	AvgDist
Gemma3-27B	0.99 (0.00)	0.31 (0.01)	1.00 (0.00)	0.37 (0.01)	0.98 (0.01)	0.41 (0.01)	1.00 (0.00)	0.40 (0.01)
Llama4-17B	1.00 (0.00)	0.43 (0.01)	0.95 (0.00)	0.40 (0.00)	1.00 (0.00)	0.53 (0.02)	1.00 (0.00)	0.44 (0.01)
Qwen3-80B	0.74 (0.01)	0.25 (0.01)	0.83 (0.01)	0.28 (0.00)	0.89 (0.01)	0.31 (0.00)	0.97 (0.02)	0.24 (0.01)

Appendix

Table 7: In S2, the linear probe achieves a high accuracy but small distance.

	Trivia QA		NQ		Pop QA		Strategy QA
LLM	Acc	AvgDist	Acc	AvgDist	Acc	AvgDist	Acc	AvgDist
Gemma3-27B	0.66 (0.01)	0.22 (0.01)	0.62 (0.01)	0.21 (0.01)	0.65 (0.01)	0.20 (0.01)	0.98 (0.02)	0.26 (0.00)
Llama4-17B	0.70 (0.01)	0.20 (0.00)	0.70 (0.01)	0.24 (0.01)	0.74 (0.03)	0.21 (0.01)	0.93 (0.01)	0.18 (0.00)
Qwen3-80B	0.60 (0.01)	0.26 (0.00)	0.46 (0.02)	0.21 (0.01)	0.67 (0.01)	0.27 (0.02)	0.51 (0.01)	0.21 (0.00)

Appendix

Table 8: In S3, the linear probe achieves both low accuracy and small distance.

	Trivia QA		NQ		Pop QA		Strategy QA
LLM	Acc	AvgDist	Acc	AvgDist	Acc	AvgDist	Acc	AvgDist
Gemma3-27B	0.49 (0.01)	0.18 (0.01)	0.52 (0.01)	0.21 (0.00)	0.63 (0.01)	0.35 (0.01)	0.50 (0.02)	0.25 (0.01)
Llama4-17B	0.62 (0.01)	0.13 (0.00)	0.70 (0.01)	0.14 (0.01)	0.64 (0.01)	0.24 (0.01)	0.68 (0.01)	0.16 (0.00)
Qwen3-80B	0.64 (0.02)	0.14 (0.00)	0.76 (0.02)	0.18 (0.00)	0.77 (0.02)	0.25 (0.00)	0.71 (0.01)	0.21 (0.00)

Appendix

Table 9: In S4, the linear probe achieves both low accuracy and small distance.

	Trivia QA		NQ		Pop QA		Strategy QA
LLM	Acc	AvgDist	Acc	AvgDist	Acc	AvgDist	Acc	AvgDist
Gemma3-27B	1.00 (0.00)	0.54 (0.01)	1.00 (0.00)	0.56 (0.00)	1.00 (0.00)	0.73 (0.00)	1.00 (0.00)	0.57 (0.00)
Llama4-17B	0.99 (0.01)	0.57 (0.01)	1.00 (0.00)	0.46 (0.01)	1.00 (0.00)	0.61 (0.01)	1.00 (0.00)	0.46 (0.01)
Qwen3-80B	1.00 (0.00)	0.44 (0.00)	1.00 (0.00)	0.48 (0.01)	1.00 (0.00)	0.59 (0.02)	1.00 (0.00)	0.50 (0.01)

Appendix

Table 10: In S5, the linear probe achieves both high accuracy and large distance.

Figure 7: The complete result of Figure 3 —relationship between cosine similarity and response type. For each context type, we compute the cosine similarity between the representations of with-context prompts and query, and categorize responses as correct, incorrect, or abstain. The result shows that LLMs are more likely to abstain when context induces large representation shifts.

Figure 8: The complete result of Figure 4 —PCA of base models. We apply PCA on representations of base models. The result shows that base LLMs do not have representation drift across different context types. Note that we only show the result of Gemma3-27B and Llama4-17B as Qwen3-80B did not release its base model.

Gemma3-27B
	Trivia QA						NQ						Pop QA						Strategy QA
	Easy			Hard			Easy			Hard			Easy			Hard			Easy			Hard
Context	Cor	Inc	Abs	Cor	Inc	Abs	Cor	Inc	Abs	Cor	Inc	Abs	Cor	Inc	Abs	Cor	Inc	Abs	Cor	Inc	Abs	Cor	Inc	Abs
Base
Relevant	92.4	4.2	3.4	52.5	35.6	11.9	81.9	10.2	7.9	43.9	39.5	16.6	92.4	2.0	5.6	76.7	20.4	2.9	74.5	16.4	9.1	44.6	42.2	13.2
Distracting	79.5	15.8	4.7	14.8	72.4	12.8	65.7	25.5	8.8	12.4	70.7	16.9	46.2	48.1	5.7	5.2	86.4	8.4	62.0	27.6	10.4	29.8	52.2	18.0

Appendix

Table 11: The complete result of Table 1 We report the percentage of correct (Cor), incorrect (Inc), and abstain (Abs) responses for both base and instruction-tuned LLMs. The result shows that instruction-tuned LLMs tend to abstain when the retrieval document is distracting or random, even if they can answer with the query alone. Note that we only show the result of Gemma3-27B and Llama4-17B as Qwen3-80B did not release its base model.

Figure 9: The complete result of Figure 5 —PCA of multiple-document contexts. We perform PCA on the last prompt token representations for multiple-document contexts that include one relevant document, and plot them alongside representations with only a relevant document in 2D. The result shows that representations remain similar when a relevant document is present, regardless of other context.

Figure 10: The complete result of Figure 6 —evolution of Gemma3-27B representations. We perform PCA on the last prompt token representations of Gemma3-27B across different layers and plot them in 2D.

Figure 11: The complete result of Figure 6 —evolution of Llama4-17B representations. We perform PCA on the last prompt token representations of Llama4-17B across different layers and plot them in 2D.

Figure 12: The complete result of Figure 6 —evolution of Qwen3-80B representations. We perform PCA on the last prompt token representations of Qwen3-80B across different layers and plot them in 2D.

Figure 13: Evolution of Gemma3-27B-base representations. We perform PCA on the last prompt token representations of Gemma3-27B-base across different layers and document types and plot them in 2D.

Figure 14: Evolution of Llama4-17B-base representations. We perform PCA on the last prompt token representations of Llama4-17B-base across different layers and document types and plot them in 2D.

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

On the impact of retrieved content representations in RAG Pipelines
May 29, 2026Jonathan J Ross, Bevan Koopman, Anton van der Vegt +1Retrieval-Augmented Generation RAG Evaluation
The University of Queensland · CSIRO / The University of Queensland
Latent Abstraction for Retrieval-Augmented Generation
Apr 20, 2026Ha Lan N. T, Minh-Anh Nguyen, Dung D. LeRetrieval-Augmented Generation Adaptive RAG
Center for AI Research, VinUniversity, Vietnam
Guaranteeing Knowledge Integration with Joint Decoding for Retrieval-Augmented Generation
Apr 9, 2026Zhengyi Zhao, Shubo Zhang, Zezhong Wang +7Hallucination in RAG Retrieval-Augmented Generation
The Chinese University of Hong Kong · Ministry of Education Key Laboratory of High Confidence Software Technologies, CUHK · University of International Relations +2

May 29, 2026J/K move · Enter open · S save

On the impact of retrieved content representations in RAG Pipelines

Jonathan J Ross, Bevan Koopman, Anton van der Vegt, Guido Zuccon

The University of Queensland · CSIRO / The University of Queensland

Retrieval-Augmented Generation RAG Evaluation

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) supplements a language model's input with retrieved documents, yet most RAG pipelines inherit retrieval components designed for human readers. How retrieved content should be represented when the consumer is a large language model (LLM) rather than a human is less well understood. Recent work has proposed transformations of retrieved content and identified properties that affect generation, but each examines a single transformation or property in isolation, leaving open which features of a document's representation matter most. We address this with a controlled comparison: holding retrieval fixed, we vary only the representation of retrieved documents, comparing an original baseline against thirteen transformations spanning selection, summarisation, and reformulation, in query-dependent and query-independent variants. Across these fourteen representations we measure question-answering accuracy for four generators, and for each representation we also measure answer retention: whether a known answer-bearing document still supports its answer after transformation. We find that answer retention is the primary determinant of generator accuracy; notably, when retention is high, a representation's wording, structure, length, and query-dependence have limited effect. This suggests that accuracy gains attributed to specific mechanisms in prior work may be partly explained by how well those mechanisms preserve answer-bearing content, an attribution that cannot be settled without controlling for retention.

Abstract

Retrieval-Augmented Generation

Explore similar work

On the impact of retrieved content representations in RAG Pipelines

Latent Abstraction for Retrieval-Augmented Generation

Guaranteeing Knowledge Integration with Joint Decoding for Retrieval-Augmented Generation