From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection

Zhibo Hou; Fan Zhao; Zhiyu An; Wan Du

cs.CLAug 26, 2026

From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection

Authors: Zhibo Hou, Fan Zhao, Zhiyu An, Wan Du

Organizations: University of California, Merced

Abstract

Continual knowledge injection is essential for keeping large language models up-to-date in a fast-evolving world. Existing methods rely on supervised fine-tuning (SFT), which memorizes injected facts in their training format but fails to generalize across paraphrasing, document combinations, and reasoning. To address this, we propose Golden-GRPO Injection (GRIN), a three-stage self-learning framework for continual knowledge injection. Golden-GRPO is a mixed-policy reinforcement learning algorithm designed specifically for knowledge injection, which injects a golden answer to provide learning signal even when on-policy rollouts fail on novel facts. We further introduce Blank and Counter, two document-level benchmarks targeting novel acquisition and counterfactual overwrite respectively, each evaluating single-fact recall, multi-source retrieval, and inferential reasoning. Our experiments establish a clear empirical claim: mixed-policy reinforcement learning enables knowledge absorption beyond what supervised fine-tuning can achieve. GRIN substantially outperforms SFT and mixed-policy RL baselines on the harder question types while matching them on basic fact recall.

Figures & tables

Benchmark	Document excerpt	Single-fact recall	Multi-source retrieval	Inferential reasoning
Blank	Aurelius Vance (b. 1962 ) is a Canadian marine biologist who completed his doctorate at the University of British Columbia. In 2008 , he was appointed director of the Pacific Marine Research Institute.	Q: What is the nationality of Aurelius Vance? A: Canadian	Q: What is the nationality and profession of Aurelius Vance? A: Canadian ; Marine biologist	Q: How old was Vance when he became director of the institute? A: 46 ( 2008 - 1962 )
Counter	Resident Evil Village is a survival horror game by Aetheris Studios (Capcom), released in 2034 (2021) as a sequel to Voidwalker (RE 7) ( 2029 (2017)).	Q: What is the sequel to Voidwalker ? A: Resident Evil Village	Q: What is the sequel to Voidwalker and who produced it? A: Resident Evil Village ; Aetheris Studios	Q: How many years after its predecessor was Village released? A: 5 ( 2034 - 2029 )

Table 1: Representative examples from BLANK and COUNTER across all three question types. For COUNTER, counterfactual replacements are shown with the original fact in parentheses.

gemini-3.1-flash-lite
	Blank (Qwen3-4B)				Counter (Qwen3-4B)
Method	Single $\uparrow$	Multi $\uparrow$	Infer. $\uparrow$	Avg $\uparrow$	Single $\uparrow$	Multi $\uparrow$	Infer. $\uparrow$	Avg $\uparrow$	fail@k $\downarrow$
Closed-book	24.63%	5.46%	34.43%	21.51%	1.31%	0.00%	2.22%	1.18%	98.53%
Open-book	97.07%	92.83%	95.24%	94.05%	99.08%	100%	99.45%	99.51%	0.00%
Training-free
Closed-book	11.43%	1.02%	0.55%	4.33%	2.83%	0.00%	4.16%	2.33%	89.92%

Table 2: Main results on Blank and Counter with Qwen3-4B. Training-free methods (Closed-book, Open-book, RAG) are reference points and are excluded from ranking. Bold marks the best and underline marks the second-best per column within the training-based group; fail@k (

\downarrow

) applies to Counter only.

Counter (Llama3.2-3B)
Method	Single $\uparrow$	Multi $\uparrow$	Infer. $\uparrow$	Avg $\uparrow$	fail@k $\downarrow$
PIT	29.10%	9.17%	27.98%	22.08%	38.60%
AL (DPO)	29.67%	2.75%	24.10%	18.84%	35.97%
Self-Tuning	39.63%	8.26%	33.80%	27.23%	27.38%
Grin (ours)	42.38%	11.01%	37.95%	30.45%	27.72%

Table 3: Cross-model results on COUNTER with Llama3.2-3B. We evaluate on COUNTER as counter facts are model irrelevant. GRIN leads on three of four accuracy metrics and is competitive on fail@k, indicating that the contribution generalizes beyond Qwen3-4B.

Figure 2: Average Counter accuracy across training epochs. SFT-based baselines plateau or degrade well below GRIN (dashed) even at 50 epochs, showing the generalization gap is not closed by additional compute.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Component

Content

System prompt

You are a knowledgeable encyclopedia writer.

User template

Write a concise, factual Wikipedia-style article about the {entity_type} , {entity_name} . Keep the article informative and tightly written, covering only the most essential facts, dates, and notable details. Organize the text using exactly 3 to 4 standard markdown headings (##) that are most appropriate for this specific subject. Each section should be a focused, well-developed paragraph — avoid bullet points, repetition, and unnecessary elaboration. Write in an encyclopedic, neutral tone.

Appendix

Table 4: Prompt 1: sampling the base model’s current beliefs about a subject.

Component

Content

User template

I want you to design a parallel-universe version of the provided wiki page. Your task is to replace concrete facts in the text — proper nouns, dates, years, numerical values, locations, named entities, organizations, award names, and titles — with plausible, internally consistent alternatives that could believably appear in a real wiki. Do not alter common nouns (e.g. “novel”, “painter”), generic descriptors, or linguistic connective tissue — only swap identifiable concrete facts. Strict rules: 1. Do not change the main entity/subject of the wiki page, which is: {subject} . 2. Preserve section headers verbatim, preserve paragraph count and order, and keep sentences in the same positions. Only the concrete facts inside sentences change. 3. If a name, date, place, or other replaced entity appears multiple times in the original, the same substitute must be used at every occurrence. 4. Output only the rewritten article in markdown — no preamble, no closing commentary, no annotations or markers in the prose. Here is the original wiki page content to transform: {content}

Appendix

Table 5: Prompt 2: rewriting a passage from Prompt 1 into a “parallel-universe” version.

Component

Content

System prompt

You are an expert question writer for reading-comprehension evaluation. Your job is to produce factoid questions whose answers are unambiguous and directly supported by a single sentence in the source document.

User template

Read the following document and generate a set of single-fact recall questions. Each question must satisfy all of the following: 1. The answer is a short concrete fact (a name, date, place, number, or named entity) that appears verbatim in exactly one sentence of the document. 2. The question is fully answerable from the document without additional context or outside knowledge. 3. The question form does not paraphrase the source sentence; rephrase it so that a model that has merely memorized the source cannot trivially pattern-match. 4. Each question targets a different fact; do not generate paraphrases of the same question. For each question, output a JSON object with fields question , answer , and source_sentence . Output one JSON object per line, no additional commentary. Document: {document}

Appendix

Table 6: Prompt for generating single-fact recall questions.

Component

Content

System prompt

You are an expert question writer who specializes in composite questions that test joint recall across multiple facts.

User template

Read the following document(s) and generate multi-source retrieval questions. Each question must be composed of {N} sub-questions (

N\in\{2,3,4\}

), where each sub-question targets a distinct concrete fact, and the sub-questions reference facts that appear in different paragraphs or different documents. Requirements: 1. Sub-question facts must not overlap; each targets a different entity, date, location, or attribute. 2. The composite question must be phrased as a single natural question (e.g. “What is X and when was Y?”) rather than a list. 3. The golden answer must list each sub-answer in the order the sub-questions appear, separated by semicolons. 4. Do not generate questions whose sub-facts could all be retrieved from a single paragraph; the point is to test joint recall across separated passages. For each question, output a JSON object with fields question , answer , N , and source_locations (list of paragraph or document identifiers for each sub-fact). Output one JSON object per line, no additional commentary. Document(s): {documents}

Appendix

Table 7: Prompt for generating multi-source retrieval questions.

Component

Content

System prompt

You are an expert question writer who designs inference questions that require combining multiple facts from a source document.

User template

Read the following document and generate inferential reasoning questions. Each question must satisfy all of the following: 1. Answering the question requires combining at least two distinct facts from the document. Direct retrieval of any single fact should be insufficient. 2. The inference type should fall into one of: temporal reasoning (e.g. computing durations, ordering events), comparison (e.g. identifying which of several entities has a property), or conditional inference (e.g. “given X, what does Y imply”). 3. The golden answer must be a short concrete value (a number, date, name, or short phrase). Do not generate questions whose answers are open-ended explanations. 4. The required inference must be derivable from the document alone, without external world knowledge. For each question, output a JSON object with fields question , answer , inference_type ( temporal / comparison / conditional ), and supporting_facts (list of source sentences used). Output one JSON object per line, no additional commentary. Document: {document}

Appendix

Table 8: Prompt for generating inferential reasoning questions.

Field	Content
System prompt	You are a helpful AI assistant that answers questions about provided documents.
Corpus	## Listing and Trading Apple Inc. is listed on the New York Stock Exchange (NYSE) under the ticker symbol APL …. [ remainder of corpus elided for space ]
Pre-query token	<\|im_start\|>user
Sampled question (model continuation)	What is the ticker symbol for Apple on the NYSE?

Appendix

Table 9: Stage 2 question sampling using a pre-query token.

Field	Content
Question	What is the ticker symbol for Apple on the NYSE?
Golden answer	Apple is a major publicly traded corporation that maintains a significant presence on the New York Stock Exchange. As part of its financial operations and public listing, the company is identified by a specific ticker symbol used for trading its shares. That ticker symbol is APL. <answer>The ticker symbol for Apple on the NYSE is APL.</answer>

Appendix

Table 10: A representative (question, golden-answer) data pair from Stage 2 sampling on Counter .

Component

Content

System prompt

You are a strict factual judge. Your job is to determine whether the student’s answer contains the same factual information as the reference answer. Follow these steps: (1) Extract the core factual claim from the reference answer. (2) Check if the student’s answer contains this same fact. (3) If the student refuses to answer, says the question is invalid, says “no such person/place exists”, or hedges without providing a concrete answer, that is WRONG . (4) Ignore any formatting, emojis, self-praise (“correct”, “accurate”), or repeated statements in the student’s answer. Focus only on the factual content. (5) Output only the word CORRECT or WRONG .

User template

Question: {question} Reference Answer: {golden_answer} Student’s Answer: {generated_answer} Does the student’s answer contain the same factual information as the reference answer? Output only : CORRECT or WRONG .

Appendix

Table 11: Prompt used by the accuracy judge to score per-type accuracy.

Component

Content

User template

You are judging whether a model’s answer reflects the counterfactual knowledge that the model was trained on, or the original prior knowledge from real-world Wikipedia. Question: {question} Model’s answer: {model_answer} --- Counterfactual text (the model’s training target) --- {counterfactual} --- end counterfactual --- --- Original Wikipedia text (real-world prior) --- {original} --- end original --- Verdict — choose ONE: • COUNTERFACTUAL — the model’s answer agrees with the counterfactual text. • ORIGINAL — the model’s answer agrees with the original Wikipedia (model fell back to prior). • NEITHER — the answer matches neither (refusal, hallucination, off-topic). Output exactly one word: COUNTERFACTUAL , ORIGINAL , or NEITHER .

Appendix

Table 12: Prompt used by the fail@k judge.

Knowledge Augmentation for Language Models

Knowledge augmentation for language models adds external or newly learned information to improve factuality, reasoning, and domain performance.

See all 180 papers

MixSD: Mixed Contextual Self-Distillation for Knowledge Injection
May 16, 2026Jiarui Liu, Lechen Zhang, Yongjin Yang +5Knowledge Augmentation for Language Models Factual Knowledge in Language Models
Carnegie Mellon University · Jinesis Lab, University of Toronto & Vector Institute · University of Illinois Urbana-Champaign +6
Continual Memorization of Factoids in Language Models
Nov 11, 2024Howard Chen, Jiayi Geng, Adithya Bhaskar +2Continual Learning for LLMs Memorization in Language Models
Princeton Language and Intelligence (PLI), Princeton University
RING: Retrieval-Internalized Generation for Continual Large-Scale Knowledge Injection
Aug 3, 2026Shicheng Xu, Liang Pang, Liyi Chen +7Retrieval-Augmented Generation Continual Learning for LLMs
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Xiaohongshu Inc.

May 16, 2026J/K move · Enter open · S save

MixSD: Mixed Contextual Self-Distillation for Knowledge Injection

Jiarui Liu, Lechen Zhang, Yongjin Yang, Yinghui He+4

Carnegie Mellon University · Jinesis Lab, University of Toronto & Vector Institute · University of Illinois Urbana-Champaign+6

Knowledge Augmentation for Language Models Factual Knowledge in Language Models Self-Distillation Knowledge Distillation Catastrophic Forgetting

Open paper AskPDF arXiv

Supervised fine-tuning (SFT) is widely used to inject new knowledge into language models, but it often degrades pretrained capabilities such as reasoning and general-domain performance. We argue this forgetting arises because fine-tuning targets from humans or external systems diverge from the model's autoregressive distribution, forcing the optimizer to imitate low-probability token sequences. To address this problem, we propose MixSD, a simple external-teacher-free method for distribution-aligned knowledge injection. Instead of training on fixed targets, MixSD constructs supervision dynamically by mixing tokens from two conditionals of the base model itself: an expert conditional that observes the injected fact in context, and a naive conditional that reflects the model's original prior. The resulting supervision sequences preserve the factual learning signal while remaining substantially closer to the base model's distribution. We evaluate MixSD on two synthetic corpora that we construct to study factual recall and arithmetic function acquisition in a controlled setting, together with established benchmarks for open-domain factual question answering and knowledge editing. Across multiple model scales and settings, MixSD consistently achieves a better memorization-retention trade-off compared to SFT and on-policy self distillation baselines, retaining up to 100% of the base model's held-out capability while maintaining near-perfect training accuracy, whereas standard SFT retains as little as 1%. We further show that MixSD produces substantially lower-NLL supervision targets under the base model and reduces harmful movement along Fisher-sensitive parameter directions. These results suggest that aligning supervision with the model's native generation distribution is a simple and effective principle for knowledge injection that mitigates catastrophic forgetting.

Abstract

Knowledge Augmentation for Language Models

Explore similar work

MixSD: Mixed Contextual Self-Distillation for Knowledge Injection

Continual Memorization of Factoids in Language Models

RING: Retrieval-Internalized Generation for Continual Large-Scale Knowledge Injection