SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection

Kexian Tang; Jiani Wang; Shaowen Wang; Kaifeng Lyu

cs.LGMar 23, 2026

SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection

Authors: Kexian Tang, Jiani Wang, Shaowen Wang, Kaifeng Lyu

Organizations: Institute for Interdisciplinary Information Sciences, Tsinghua University

Abstract

While large language models (LLMs) are pretrained on massive amounts of data, their knowledge coverage remains incomplete in specialized, data-scarce domains, motivating extensive efforts to study synthetic data generation for knowledge injection. We propose SPA (Scaling Prompt-engineered Augmentation), a simple but tough-to-beat baseline that uses a small set of carefully designed prompts to generate large-scale synthetic data for knowledge injection. Through systematic comparisons, we find that SPA outperforms several strong baselines. Furthermore, we identify two key limitations of prior approaches: (1) while RL-based methods may improve the token efficiency of LLM-based data augmentation at small scale, they suffer from diversity collapse as data scales, leading to diminishing returns; and (2) while multi-stage prompting may outperform simple augmentation methods, their advantages can disappear after careful prompt tuning. Our results suggest that, for knowledge injection, careful prompt design combined with straightforward large-scale augmentation can be surprisingly effective, and we hope SPA can serve as a strong baseline for future studies in this area. Our code is available at https://github.com/Tangkexian/SPA.

Figures & tables

Figure 1: Overview of Scaling Prompt-engineered Augmentation (SPA). Our baseline method rewrites a small source corpus into a large synthetic corpus by repeatedly prompting a generator with a fixed set of seven human-curated prompt templates, which are designed based on three levels of learning strategies: Concept Learning, Critical Thinking, and Generative Learning.

SQuAD	QA Accuracy(%)
Model: Qwen2.5-7B
Generator: Qwen2.5-7B (self-generated)
Number of Tokens: 120M
$\ast$ Base	31.31
$\star$ Rephrase	86.86
$\star$ QA	89.63

Table 1: Main results at the largest synthetic token scale on each benchmark. QA accuracy (%) is reported across SQuAD, QuALITY, and MultiHop-RAG, with shared settings grouped. For each benchmark, Model refers to the adapted model, Generator refers to the model used for synthetic data generation, and Number of Tokens refers to the total synthetic token budget. Each value corresponds to the final performance at the largest token scale used in that benchmark.

\ast

Base denotes the model without additional training,

\star

denotes simple baselines,

\triangleright

refers to prior high-performing methods, and

\blacktriangleright

denotes our method SPA.

Figure 2: Scaling Curve on SQuAD shows that SPA exhibits strong and consistent scaling behavior. The y-axis represents QA accuracy, and the x-axis represents the synthetic token budget. Note : the PaST data point corresponds to the best performance reported in the original paper.

Method	Generation Model	QA Accuracy(%)
QA	GPT-4-Turbo	52.99
QA	gpt-oss-120b	47.47
SPA	gpt-oss-120b	52.26
QA	GPT-4o	51.31
SPA	GPT-4o	55.49

Table 2: QA accuracy of SPA and QA baseline using different generation models. The training token here is 27M.

Figure 3: Scaling Curve on QuALITY shows that SPA achieves the strongest scaling performance among compared methods as data scales. The y-axis is QA accuracy, and the x-axis is the token budget. Note : For EntiGraph, we use statistics from the original paper, where GPT-4-Turbo is used as the generator, whereas SPA and Active Reading use gpt-oss-120b. The SoG data point corresponds to the best performance reported in the original paper, which uses a stronger base model Llama-3.1-8B-Instruct.

Figure 4: Document-level comparison shows that SPA achieves higher average strategy effectiveness than Active Reading on QuALITY. The table reports the average accuracy (%) of each method across all strategies for each document, showing that SPA consistently attains higher mean accuracy than Active Reading. Bold numbers indicate cases where SPA outperforms Active Reading. The subplots visualize the accuracy of individual strategies for each document, including seven strategies for SPA and a variable number for Active Reading. In each subplot, wide bars denote the average accuracy across all strategies within each method, and the narrow bars denote the accuracy of individual strategies. The gray dashed line denotes the base model’s accuracy.

Method	CR $\downarrow$	Self-Rep. $\downarrow$	Self-BLEU $\downarrow$	CR: POS $\downarrow$
SEAL	19.2468	8.2446	0.0058	21.2072
Rephrase	5.1330	5.6019	0.0014	7.0200
QA	8.9630	6.9882	0.0030	9.7534
Active Reading	4.1788	4.5854	0.0010	5.9388
SPA (ours)	4.3806	4.8207	0.0010	6.0024

Table 3: Diversity evaluation of synthetic data generated by different data augmentation methods on the SQuAD benchmark. Lower values indicate higher diversity. All scores are averaged across five documents sampled with a fixed random seed. Best results are in bold , and second-best are underlined .

Method	QA Acc. (%)	Method	QA Acc. (%)
Key concepts	78.95 (-8.73)	Case studies	83.57 (-4.11)
Mind map	81.93 (-5.75)	Discussions	84.29 (-3.39)
Implications	85.52 (-2.16)	Teacher-style	83.68 (-4.00)
QA-ct	84.70 (-2.98)	Full Set (SPA)	87.68

Table 4: Performance of individual SPA components trained in isolation. All models are trained with approximately 22M tokens on SQuAD. Numbers in parentheses indicate the relative performance drop compared to full SPA.

Prompt Strategies	Acc (SQuAD)
$\mathcal{P}$	87.68%
$\mathcal{P}$ { Key }	86.06%
$\mathcal{P}$ { Key , Mind }	87.27%
$\mathcal{P}$ { Key , Mind , Case }	87.68%
$\mathcal{P}$ { Key , Mind , Case , Teach }	85.01%
$\mathcal{P}$ { Key , Mind , Case , Teach , Disc }	88.19 %

Table 5: Ablation study over different subsets of prompt strategies under a fixed budget of 22M tokens on SQuAD.

\mathcal{P}

denotes the full SPA prompt set, Disc denotes Discussions , Key denotes Key concepts , Mind denotes Mind map , Teach denotes Teacher-style , Case denotes Case Studies , and Imp denotes Implications .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Search Space
Learning Rate	[4e-5, 5e-5 , 6e-5, 7e-5]
Batch Size	[8, 64 ]

Appendix

Table 6: Hyperparameter search space for experiments on SQuAD. Values in bold indicate the hyperparameters used for the largest few experiments within the range of token scales.

Method	Codebase	QA Accuracy (%)
EntiGraph	SPA	55.64
SPA	SPA	57.03
EntiGraph	EntiGraph	54.50 (56.22)
SPA	EntiGraph	56.64

Appendix

Table 7: Training stability comparison on QuALITY across different codebases. Numbers in parentheses indicate results reported in the original EntiGraph paper. All models are trained with 455M synthetic tokens.

Dataset	Method	Avg CR $\downarrow$	Avg Self-Rep. $\downarrow$	Avg Self-BLEU $\downarrow$	Avg CR: POS $\downarrow$
SQuAD	Rephrase	5.1330	5.6019	0.0014	7.0200
	QA	8.9630	6.9882	0.0030	9.7534
	SEAL	19.2468	8.2446	0.0058	21.2072
	Active Reading	4.1788	4.5854	0.0010	5.9388
	SPA	4.3806	4.8207	0.0010	6.0024
QuALITY	QA	6.6744	6.5268	0.0032	9.2556

Appendix

Table 8: Extended diversity evaluation across datasets. Synthetic data are generated under the same settings as the main experiments. Note: For QuALITY, gpt-oss-120b is used for generation, except for EntiGraph, where we evaluate their released dataset generated by GPT-4o-Turbo. Best results are in bold , and second-best are underlined .

Method	MMLU	ARC-C	OpenBookQA	WinoGrande	Avg
Base	71.93	51.37	47.20	73.32	60.96
SEAL	70.26	55.72	45.00	71.74	60.68
Active Reading	70.27	56.31	45.60	72.53	61.18
SPA	70.51	55.12	45.60	73.16	61.10

Appendix

Table 9: Evaluation on general benchmarks after training on SQuAD-based synthetic data. SPA shows slight improvements on some benchmarks while largely retaining the performance of the base model. The best average score is in bold , and the second best is underlined .

Method	MMLU	ARC-C	OpenBookQA	WinoGrande	Avg
Base	62.26	54.18	44.80	73.01	58.56
EntiGraph	55.80	48.89	43.60	68.67	54.24
Active Reading	55.72	50.09	43.00	64.25	53.27
SPA	56.04	48.89	44.20	68.19	54.33

Appendix

Table 10: Evaluation on general benchmarks after training on QuALITY-based synthetic data. SPA exhibits the smallest drop in average performance compared to other baselines. The best average score is in bold , and the second best is underlined .

Knowledge Augmentation for Language Models

Knowledge augmentation for language models adds external or newly learned information to improve factuality, reasoning, and domain performance.

See all 171 papers

Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesis
Jun 22, 2026Songze Li, Yarong Lan, Zhongpu Bo +16Synthetic Data Pretraining Knowledge Augmentation for Language Models
Zhejiang University · ZJU-Ant Group Joint Lab of Knowledge Graph · Ant Group
FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
Jan 29, 2026Ajay Patel, Colin Raffel, Chris Callison-BurchLanguage Model Pretraining Synthetic Data Pretraining
University of Pennsylvania · University of Toronto · Vector Institute +1
Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMs
Aug 31, 2026Jonathan Zheng, Zirui Shao, Alan Ritter +1Synthetic Data Pretraining Knowledge Editing
Georgia Institute of Technology · Zhejiang University

Jun 22, 2026J/K move · Enter open · S save

Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesis

Songze Li, Yarong Lan, Zhongpu Bo, Zhaoyang Wang+15

Zhejiang University · ZJU-Ant Group Joint Lab of Knowledge Graph · Ant Group

Synthetic Data Pretraining Knowledge Augmentation for Language Models Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Knowledge injection via synthetic data is crucial for enhancing Large Language Models (LLMs). However, current synthesis methods simply stop at preset token counts or fixed data ratios, lacking awareness of knowledge distribution. This results in some domains being sparse while others are redundant, limiting LLM knowledge boundaries. We revisit knowledge injection from a distribution perspective and hypothesize that an optimal knowledge distribution exists to maximize knowledge boundary expansion. We propose KDoS (Knowledge Distribution-optimized Synthesis), a framework that introduces knowledge density to drive synthesis through a three-stage feedback mechanism, shifting from blind generation to distribution-optimized synthesis. We construct Wikipedia-based synthetic data with varying knowledge distributions and conduct experiments on models from 0.6B to 16B (Qwen, Ling, LLaMA) and data scales from 1B to 5B tokens. Our key findings are: (1) an optimal knowledge distribution consistently maximizes boundary expansion; (2) this distribution is stable across backbones and scales; (3) KDoS outperforms baselines across six knowledge benchmarks. Our work offers a new perspective and practical framework for synthetic data-driven knowledge injection.

Abstract

Knowledge Augmentation for Language Models

Explore similar work

Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesis

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMs