Contextual Embeddings Beyond the Gold Chunk

Sedigheh Eslami; Markus Krimmel; William Barber; Antoine Chaffin; Maximilian Schall; Jonas Golde; Louis Milliken; Ekaterina Glazkova

cs.LGOct 28, 2026

Contextual Embeddings Beyond the Gold Chunk

Authors: Sedigheh Eslami, Markus Krimmel, William Barber, Antoine Chaffin, Maximilian Schall, Jonas Golde, Louis Milliken, Ekaterina Glazkova,

Organizations: Perplexity AI · turbopuffer

Abstract

Contextual embeddings represent chunks of a document with respect to the document's global context. These models are typically trained and evaluated under the assumption that a relevant document contains only a single relevant chunk. In this work, we argue that in real-world use cases a retriever should not only find answers spread over several chunks but also surface the evidence needed to verify them. To address this, we introduce pplx-embed-v2-context, a family of contextual embedding models trained by distilling graded chunk-level relevance from a query-aware context compression model. This removes the need for gold-chunk labels and makes the training targets independent of the chunking. We utilize a vision-language backbone for training and release pplx-embed-v2-context in 9B and 0.6B sizes with compatible embedding spaces. On the evaluation side, we introduce two benchmarks that label both the chunks that answer a query and the evidence needed to verify the answer. Context-bench is a controlled diagnostic, and Q2D-Chunk is built from real-world search queries. To encourage further research, we also publish Q2D-Chunk as a public leaderboard. Our 9B model sets a new state of the art on both benchmarks and ConTEB. We find that the visual retrieval capabilities of our 9B model transfer from its vision-language base model and propose to extend chunking to image inputs. We demonstrate that our models achieve state-of-the-art performance among dense retrievers on visual ViDoRe-V3 and that performance improves with granular image chunking.

Figures & tables

Panel of Figure 1 — Figure 1: The training approach for the 9B model. The contextual encoder (left) embeds each chunk with respect to its document. Query A, a known match for Document A, is scored against the chunks of all documents, and the chunk scores are max-pooled into document scores (right). Chunk scores are trained toward the context compression model’s predictions within Document A and zero elsewhere, and document scores toward a one-hot target on Document A. Darker blue indicates higher similarity or relevance.

	Context-bench	Q2D-Chunk
Queries (base / with rewordings)	2,099	6,069 / 11,719
paraphrased / indirect / scenario	–	1,876 / 1,888 / 1,886
Documents	38,894	71,100
Chunks	2.46M sentences	9.17M of $\leq$ 128 tokens
Median target document length	$\approx$ 6,100 tokens	$\approx$ 4,700 tokens
Languages / domains	– / 22	10 / 26

Table 1: Statistics of context-bench and Q2D-Chunk.

Panel of Figure 2 — Figure 2: The Q2D-Chunk construction pipeline, from production search logs to the final benchmark.

Figure 3: Where the answer chunks of Q2D-Chunk sit within their pages (128-token chunks, one point per query and page with answers). (a) Relative position on the page of all answer chunks and of the first answer chunk of each page. (b) Answer chunks per page and the number of separate runs of adjacent answer chunks they form. (c) Gap between consecutive answer chunks of a page.

	Answer@ $K$			Evidence recall@ $K$ (in document)				Document@ $K$
Model	@3	@5	@10	@3	@5	@10	All-Evidence@10	@3	@5	@10
pplx-embed-v2-context-9b	23.6	31.1	45.5	23.2	30.8	40.6	31.1	29.7	40.3	61.6
pplx-embed-v2-context-0.6b	9.3	14.0	21.9	14.4	18.8	23.8	18.6	16.4	23.7	39.0
pplx-embed-context-v1-4b	12.1	17.2	26.4	22.0	28.4	36.4	28.6	29.9	40.5	58.1
voyage-context-4	15.2	20.2	31.1	21.6	27.1	35.6	27.6	24.2	33.4	51.6
voyage-4-large †	2.7	4.1	6.4	16.7	19.7	23.4	18.8	17.6	25.4	40.2

Table 2: Results on context-bench: Answer@

K

, evidence recall@

K

within retrieved documents, and Document@

K

. All-Evidence@10 is the fraction of queries with every evidence group recovered. Answer chunks are removed before evidence is ranked. † Non-contextual baseline, which encodes every sentence independently. Scores are in %.

	Answers (Answer@10)				Evidence@10			Both		Pages
									Tokens
Model	base	paraphr.	indirect	scenario	corpus	in document	All-Evidence	$\alpha$ -nDCG@10	to coverage	Document@10
pplx-embed-v2-context-9b	86.4	85.3	82.6	83.9	46.6	72.6	37.5	68.1	1,954	98.4
pplx-embed-v2-context-0.6b	83.1	81.6	77.3	79.9	44.0	69.5	35.2	64.2	2,264	97.9
voyage-context-4	83.9	83.2	80.2	80.9	41.8	68.3	33.1	64.7	2,412	97.6
pplx-embed-context-v1-4b	74.9	73.8	65.4	70.5	41.6	71.5	33.1	53.9	2,826	95.7

Table 3: Answer@10 on the four query configurations of Q2D-Chunk, and evidence, combined and page metrics and median tokens to full coverage on the base queries. † Non-contextual model, which encodes every chunk independently. Scores are in %.

	Health	Law	Finance	Business	Technology	Education	Politics	Science	Consumer goods	Real estate
Model	(1,456)	(1,239)	(719)	(465)	(456)	(413)	(190)	(177)	(145)	(119)
pplx-embed-v2-context-9b	87.7	86.9	85.5	86.1	86.8	86.1	83.9	86.5	82.2	85.9
pplx-embed-v2-context-0.6b	84.9	83.3	83.2	83.3	83.3	82.3	80.7	82.7	77.7	81.9
voyage-context-4	84.1	85.0	84.3	82.0	85.6	83.8	80.9	81.6	80.7	84.3
pplx-embed-context-v1-4b	73.1	76.5	77.5	75.4	76.0	71.4	77.0	72.3	69.0	74.7
pplx-embed-context-v1-0.6b	68.0	67.4	70.0	71.3	69.3	62.9	72.6	66.4	56.8	75.5

Table 4: Answer@10 on Q2D-Chunk per top-level domain, for the ten most frequent labeled domains (base queries; queries per domain in parentheses). Scores are in %. † Non-contextual model, which encodes every chunk independently.

Model	Avg.	Geography	ESG	Insurance	Football	SQuAD	NarrativeQA	MLDR	COVID-QA
pplx-embed-v2-context-9b	83.4	94.3	67.8	99.1	81.6	88.7	84.8	90.1	60.4
pplx-embed-v2-context-0.6b	80.7	94.1	67.3	88.0	79.2	87.3	83.4	89.8	56.1
voyage-context-4	80.4	93.9	57.4	99.7	81.3	84.3	82.5	87.6	56.6
pplx-embed-context-v1-4b	82.1	93.5	63.4	100.0	78.1	83.7	86.9	89.4	62.0
voyage-4-large †	67.0	86.9	61.6	22.6	53.5	78.5	83.5	83.8	65.2
Nemotron-3-Embed-8B †	63.8	80.7	51.6	21.9	39.0	80.5	83.0	81.8	72.0

Table 5: Task-level nDCG@10 on ConTEB in the query-to-chunk setting. The average is the unweighted mean across the eight tasks; best value per column in bold. Scores are in %. † Non-contextual model, which encodes every chunk independently.

$\geq$ 1B parameters or undisclosed
Model	Chunk	Avg.	Fin(En)	Fin(Fr)	Indust.	Pharma	CS	HR	Energy	Physics
pplx-embed-v2-context-9b	multi	61.18	64.19	51.06	49.33	65.42	77.85	62.75	69.58	49.28
pplx-embed-v2-context-9b	single	58.25	59.91	47.05	46.27	65.43	75.10	57.81	65.42	49.02
voyage-context-4	multi	62.76	63.68	50.24	52.64	68.74	80.65	65.83	70.60	49.72
Nemotron-3-Embed-8B †	single	60.63	62.24	47.17	50.88	67.19	77.08	62.21	67.18	51.11
voyage-4-large †	single	59.63	61.14	47.20	48.64	67.67	76.16	61.29	66.39	48.52

Table 6: Text-to-Markdown retrieval on ViDoRe-V3. Scores are nDCG@10 (%), averaged equally over six languages per subtask; Avg. is the mean over eight subtasks.

Figure 4: Retrieval quality versus index size. Left: mean nDCG@10 on ViDoRe-V3 Markdown across eight public tasks versus average index size per document. Our curves use one byte per embedding dimension; baseline points assume FP16 vectors. Nemotron-3-Embed-8B and voyage-4-large use one chunk per document at 4096 and 2048 dimensions, respectively; voyage-context-4 uses 128-token chunks at 2048 dimensions. Right: Q2D-Chunk performance for 1 and 2 KiB INT8 vectors for our 9B model and Voyage, assuming Voyage INT8 matches FP32.

Figure 5: Average nDCG@10 versus index size per page on the eight public ViDoRe-V3 tasks (left) and MIRACL-Vision, averaged across 18 languages (right). Our curves use one byte per embedding dimension and 1, 2, 4, 8, or 16 chunks per page. Dense baselines assume one FP16 vector per page.

Benchmark	Metric	Contrastive only	Distillation only	Both (ours)
Q2D-Chunk	Answer@10	71.1	85.8	86.3
ConTEB	nDCG@10	68.1	82.4	82.7
ViDoRe-V3 Markdown	nDCG@10	58.7	60.3	60.9

Table 7: Training-objective ablation for the 9B model. Scores in %; best per benchmark in bold.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Steps	1600
Warmup / stable / decay steps	300 / 700 / 600
Learning-rate schedule	warmup–stable–decay
Peak learning rate	$1.25\times 10^{-4}$
Optimizer	AdamW, weight decay 0.01
Optimizer batch (pairs)	65,536
In-batch negatives per query	16,384

Appendix

Table 8: Training hyperparameters of the 9B model, trained in a single stage with Matryoshka and int8 quantization-aware training from the first step.

Panel of Figure 6 — Figure 6: Visual document chunking, illustrated with four horizontal bands. The soft visual token sequence is passed through the text tower.

Type	Example query	Answer chunk	Required document context
1. Entity identity	How much did Northlake spend on share repurchases?	The company repurchased $12.8 billion of shares.	The introduction identifies Northlake as “the company.” A similar Southlake report is not relevant.
2. Period or version	Which cartridge fits the M40 revision B?	Install the KC-42 cartridge.	The applicability section says these instructions cover revision B, not revision C.
3. Definitions and aliases	What withdrawal rate did the 15 mg group have?	Zeta had a withdrawal rate of 12.7%.	The report defines Zeta as the 15 mg group. Without that definition, the chunk does not identify the requested group.
4. Table structure	What is the Max model’s battery capacity?	Typical capacity: 4,250 \| 4,700 \| 5,150.	The table header establishes Mini \| Base \| Max, measured in mAh. The answer is therefore 5,150 mAh.
5. Conditions and exceptions	How long is battery coverage for commercial use?	Battery coverage lasts 18 months.	A section introduction restricts this coverage to commercial installations; household installations have a different warranty.
6. References and pronouns	When did Mara become deputy mayor?	She took office in 2021.	The preceding narrative establishes that “she” refers to Mara, not Elin, who is also discussed.

Appendix

Table 9: Twelve types of context dependency tested by context-bench, with illustrative examples.

Panel of Figure 7 — Figure 7: Q2D-Chunk queries in their four wordings. The scenario variant is shown as its post title (italic) followed by its body.

Panel of Figure 8 — Figure 8: An example from Q2D-Chunk: a query, its answer and evidence groups, and an excerpt of one of its pages as 128-token chunks ( answer, evidence, other; text shortened). The answer chunks state the forecasts but not whose forecasts they are or from which edition; only the evidence chunk, the page title, ties them to the ADB’s April 2026 outlook, which matters because the page notes that the forecasts have since been updated. The page gives no figure for Kazakhstan alone, so A1 has no chunk here.

Figure 9: Context-bench query coverage by subject domain and type of context dependency.

Figure 10: Queries per top-level domain in Q2D-Chunk (base queries, 6,069 queries). Highlighted: the ten most frequent domains, reported in Table 4 .

Figure 11: Queries per query language in Q2D-Chunk (base queries, 6,069 queries; log scale).

Model	Avg.	Law	Tech	Finance	Health	Multilingual	Conversation
pplx-embed-v2-context-9b	72.9	68.2	48.0	81.6	83.2	90.4	72.4
voyage-context-4	72.3	65.2	46.5	82.9	82.2	90.9	73.2
pplx-embed-context-v1-4b	64.4	53.9	38.4	74.2	74.3	89.2	70.7
voyage-4-large †	66.5	56.0	37.0	79.1	77.6	88.2	73.1
Nemotron-3-Embed-8B †	66.6	55.2	37.5	77.1	81.8	89.3	74.9

Appendix

Table 10: Retrieval quality across domains on the MTEB-Q2C tasks. Mean nDCG@10 per domain, as unweighted means of the task-level scores in that domain; the average is the unweighted mean across all tasks. Best value per column in bold. Scores are in %. † Non-contextual model, which encodes every chunk independently.

Model	Avg.	Law	Tech	Finance	Health	Multilingual & other	Conversation
pplx-embed-v2-context-9b	80.7	80.6	67.7	80.6	83.0	86.8	71.4
voyage-context-4	82.2	81.8	66.7	82.9	83.7	88.8	72.3
pplx-embed-context-v1-4b	77.0	76.6	61.9	77.7	78.0	82.4	70.5
voyage-4-large †	79.9	79.2	62.8	79.8	81.8	87.3	71.8
Nemotron-3-Embed-8B †	80.0	79.1	64.7	79.7	82.7	86.8	72.7

Appendix

Table 11: Retrieval quality across domains on the MTEB-Q2D tasks. Mean task score per domain, as unweighted means of the task-level main scores in that domain; the multilingual group also contains the remaining tasks, and the average is the unweighted mean across all tasks. Best value per column in bold. Scores are in %. † Non-contextual model, which encodes every chunk independently.

Panel of Figure 12 — Figure 12: (a) Mean nDCG@10 on the MTEB-Q2C tasks of Table 10 against storage per vector, in float32 (solid) and int8 (dashed), with labels giving the dimension. (b) Mean nDCG@10 on the MTEB-Q2D tasks of Table 11 , with documents indexed at the given chunk size.

$\geq$ 1B parameters or undisclosed
Model	Avg.	Fin(En)	Fin(Fr)	Indust.	Pharma	CS	HR	Energy	Physics
pplx-embed-v2-context-9b	58.25	59.91	47.05	46.27	65.43	75.10	57.81	65.42	49.02
Nemotron-3-Embed-8B †	60.63	62.24	47.17	50.88	67.19	77.08	62.21	67.18	51.11
voyage-4-large †	59.63	61.14	47.20	48.64	67.67	76.16	61.29	66.39	48.52
$<$ 1B parameters
pplx-embed-v2-context-0.6b	55.81	59.14	43.51	43.47	63.37	70.69	56.22	63.62	46.47

Appendix

Table 12: Single-chunk text-to-Markdown retrieval on ViDoRe-V3, with one chunk per page. Scores are nDCG@10 in %; – denotes an unavailable result. † Non-contextual model, which encodes every chunk independently.

$\geq$ 1B parameters or undisclosed
Model	Avg.	Fin(En)	Fin(Fr)	Indust.	Pharma	CS	HR	Energy	Physics
pplx-embed-v2-context-9b	61.18	64.19	51.06	49.33	65.42	77.85	62.75	69.58	49.28
voyage-context-4	62.76	63.68	50.24	52.64	68.74	80.65	65.83	70.60	49.72
Nemotron-3-Embed-8B †	58.23	56.25	40.45	49.90	66.74	78.18	59.52	65.29	49.52
voyage-4-large †	57.75	54.40	39.86	48.41	67.06	77.79	59.56	67.04	47.85
$<$ 1B parameters

Appendix

Table 13: Text-to-Markdown retrieval on ViDoRe-V3 with 128-token chunks. Scores are nDCG@10 in %. † Non-contextual model, which encodes every chunk independently.

Figure 13: Chunk localization on ViDoRe-V3 for English queries using the 0.6B and 9B models and eight chunks per page. Bars show the proportion of highlighted regions judged relevant; Average is the unweighted mean across datasets. Error bars show pointwise 95% percentile bootstrap intervals from matched query–page pairs resampled within each dataset. Interval overlap is not a paired significance test.

Figure 14: Ablation of contextual image chunking. We find that the joint (contextual) forward passes over chunks are important. Adding the separation token between chunks improves performance on ViDoRe-V3.

Text Embeddings

Text embeddings map words, sentences, or documents to vectors for measuring meaning, retrieving information, and organizing text.

See all 57 papers

REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling
Aug 30, 2026Devrim Çavuşoğlu, Emre AkbaşLong-Context Retrieval Long-Context Modeling
Middle East Technical University · OBSS AI
Learning to Retrieve via Reinforcement Learning in Embedding Space
Oct 6, 2026Qi Liu, Fengming Liang, Yiqun Chen +2RL for IR Text Embedding Models
Renmin University of China
Rethinking Retrieval in the Long-Context Era: Training-Free Semantic Trees for Adaptive Context Construction
Oct 25, 2026Zhe Xu, Tianhao Liao, Zhenqing Ling +3Long-Context Retrieval Information Retrieval
Sun Yat-sen University · Alibaba Group

Aug 30, 2026J/K move · Enter open · S save

REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling

Devrim Çavuşoğlu, Emre Akbaş

Middle East Technical University · OBSS AI

Long-Context Retrieval Long-Context Modeling Text Embedding Models Dense Retrieval

Open paper AskPDF arXiv

Dense retrieval over long documents is expensive. Token-level encoders scale quadratically in sequence length, and most long-context embedding models reach 32K tokens only through architectural workarounds or by stretching billion-parameter LLMs. We propose REIGN (Refurbished Embeddings with Integrated Guidance Networks), a contrastively trained bi-encoder that operates on sequences of contextualised chunk embeddings from a frozen Guidance Network (GN) rather than on raw tokens. REIGN targets multi-chunk inputs, primarily for document-to-document retrieval; single-chunk inputs stay with the GN. Decoupling token-level processing from document-level reasoning, and caching the GN embeddings to disk, cuts per-document training cost by roughly four orders of magnitude relative to chunked Transformer fine-tuning. We also release a synthetic long-document retrieval benchmark for contrastive training and evaluation at long context lengths. Across an in-distribution Wikipedia benchmark, the LoCo out-of-distribution suite, and a real-world patent retrieval case study, REIGN matches dense long-context retrievers at smaller parameter budgets in each regime. A paired significance test puts it on par with models 1.6-4.3x larger on the patent task, and it stays within 0.65 nDCG@10 of a 20x-larger model on LoCo.

Abstract

Text Embeddings

Explore similar work

REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling

Learning to Retrieve via Reinforcement Learning in Embedding Space

Rethinking Retrieval in the Long-Context Era: Training-Free Semantic Trees for Adaptive Context Construction