When a user supplies a document and wants the answer drawn from it, a language model is expected to follow it even when it contradicts its parametric knowledge. Prior work has focused on making the model rely on context by steering activations, contrasting logits during decoding, or instructing it in plain language. We ask whether the document alone, with no instruction written into it and no access to the model at inference time, is enough. We optimize a document suffix by discrete token search, starting from the answer to a single question and extending the objective to a set of passages, to multi-hop questions that follow from the planted fact, and to later turns of a conversation. On counterfactual passages that contradict parametric knowledge of the model, the suffix flips nearly all items the model resists across three instruction-tuned model families, in most settings outperforms the decoding baseline and the strongest instruction we could write, and matches activation steering. Controls show that the suffix largely acts through the passage rather than injecting the answer, and a single suffix per model transfers to unseen passages. Each extended objective carries the effect to its target, so the objective, not only the document, sets how far the effect reaches.
Figures & tables
Figure 1: An optimized suffix makes the model follow the passage. (a) A 20-token suffix \chipbox [colframe=sfxorange, colback=sfxorange!10, coltext=sfxorange] S is appended to a contextual passage \chipbox [colframe=ctxred, colback=ctxred!8, coltext=ctxred] P that plants a false fact BBC , a hard case for context following since the model knows the true answer. (b) Given only the question \chipbox [colframe=black!35, colback=black!5, coltext=black!60] Q , Llama answers from its parametric knowledge Apple . With \chipbox [colframe=ctxred, colback=ctxred!8, coltext=ctxred] P added, it rejects the contextual knowledge. With \chipbox [colframe=sfxorange, colback=sfxorange!10, coltext=sfxorange] S also appended, it follows the passage and answers BBC .
Figure 2: One item from each dataset. Plum marks the contextual (planted) answer and green the parametric one. (a) BaseFakepedia passages are written to assert the planted answer throughout. (b) CUB NQ passages are real web text with the entity replaced at its first mention, so the rest still describes the true entity: alpha, beta, and gamma efferent neurons are somatic , not autonomic .
Figure 3: A held-out question Q ′ , never in the optimization objective. S is optimized on the base question alone; S ∗ jointly over several multi-hop questions. Only S ∗ carries the planted fact to Q ′ .
Figure 4: Ten phrasings , each in the same encoding position as the suffix, sorted by flip rate on Llama. The rule on each row is the range across the three models. Rankings cross between models: based on the context is fifth for Llama and last for Gemma.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Training loss over the search, on eight training goals per run, log scale. All six runs flatten well before step 500, and the loss is computed on training goals, so checkpoint selection never saw an evaluation item. On Mistral and Gemma the two curves are measured on slightly different goals, so only the shapes of the curves are comparable.
Context-to-LoRA methods encode documents as low-rank parameter updates, allowing language models to answer without retaining the source text in their decoding context. How reliably do these updates preserve facts and support their use under different instructions? Controlled diagnosis of existing context-to-LoRA methods separates fact discrimination, target retention, and response adaptation. Competing sources influence answers, while target-only construction and memory-state optimization improve recovery at fixed update size. Recovering facts also does not ensure that answers follow response instructions. We introduce Cache2LoRA, which uses each query to aggregate evidence from a reusable frozen document cache before generating LoRA updates. Trainable encoding operates only on the query and memory-token suffix, and multi-style supervision teaches both evidence preservation and answer expression. Across 17 tasks, Cache2LoRA improves on existing context-to-LoRA methods and achieves macro F1 of 0.663, approaching full-context in-context learning at 0.671. For 32K documents and 256-token outputs at batch size 16, its decoding throughput is 5.9× that of full-context ICL. At matched memory size, generated LoRA preserves answer accuracy more reliably than the learned-KV variant under long, diverse intervening text and combines more effectively with retained document KV, yielding higher hybrid F1 and EM.
Yi Fang, Gaurav Srivastava, Zhenyu Bi +2
Department of Computer Science, Virginia Tech, Blacksburg, VA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
Large language models often answer from knowledge learned during training, even when a provided passage contradicts that knowledge. To address this problem, contrastive decoding contrasts the predictions under the full and a modified prompt at every token, but it also alters correct answers that the passage supports. We show that checking answers against the passage removes this trade-off. Based on this observation, we propose Accept-or-Repair (AoR), a training-free procedure. AoR keeps a sampled answer if the passage supports it. Otherwise, it tries up to two replacements with contrastive decoding, returning one as soon as it passes this check. It rejects replacements that repeat the answer generated without the passage when the passage favors another answer. Our experiments show that AoR raises the score over sampling by 12.5 points on counterfactual passages and 2.3 on ordinary ones, surpassing eight contrastive and prompting baselines on both, including when each receives the same rejection rule. Replacing only initial answers that the passage does not support improves the average score of every decoder. These results suggest that models can follow passages that contradict their knowledge while preserving accuracy on passages that agree with it. Code will be released at https://github.com/js-lee-AI/AoR.
Jungseob Lee, Seungyoon Lee, Hyeonseok Moon +6
Korea University · Sookmyung Women’s University · Yonsei University Mirae Campus +2
Long-context language models can increasingly process entire documents, making full-context prompting a practical alternative to passage retrieval. However, greater capacity determines how much information a model can receive, but not which information best supports a query. This distinction motivates rethinking retrieval: when the full document already fits, can selecting and organizing its evidence improve answering? The challenge is to retain supporting facts and relations while keeping the context focused on the question. To reconcile these goals, we introduce \ours, a training-free framework that uses semantic abstraction to guide selection while retaining original text for answering. \ours constructs a reusable semantic hierarchy from the full document in a single generation, linking node descriptions to source spans. Given a query, structural--semantic scoring and fine-grained lexical attribution nominate shared source nodes; their union with ancestor context preserves structural relations and complete local text. Across four frontier answer models, \ours improves average QA-F1 by 1.72 points over full-context prompting on three LongBench tasks, surpassing the evaluated baselines on the four-model average while reducing answer-context length by 58.55%. Our code is available at the repository.