KoCo: Conditioning Language Model Pre-training on Knowledge Coordinates

Yudong Li; Jiawei Cai; Linlin Shen

cs.CLApr 14, 2026

KoCo: Conditioning Language Model Pre-training on Knowledge Coordinates

Authors: Yudong Li, Jiawei Cai, Linlin Shen

Organizations: School of Computer Science and Software Engineering, Shenzhen University · Department of Electrical Engineering, Tsinghua University · School of Artificial Intelligence, Shenzhen University · Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University

Abstract

Standard Large Language Model (LLM) pre-training typically treats corpora as flattened token sequences, often overlooking the real-world context that humans naturally rely on to contextualize information. To bridge this gap, we introduce Knowledge Coordinate Conditioning (KoCo), a simple method that maps every document into a three-dimensional semantic coordinate. By prepending these coordinates as textual prefixes for pre-training, we aim to equip the model with explicit contextual awareness to learn the documents within the real-world knowledge structure. Experiment results demonstrate that KoCo significantly enhances performance across 10 downstream tasks and accelerates pre-training convergence by approximately 30%. Furthermore, our analysis indicates that explicitly modeling knowledge coordinates helps the model distinguish stable facts from noise, effectively mitigating hallucination in generated outputs.

Figures & tables

Figure 1: KoCo transforms DIKW into an objective representation of the knowledge contained in the corpus.

Figure 2: Overview of Knowledge Coordinate Conditioning (KoCo). Different from standard pre-training (a), KoCo (b) conditions the model on the context tag. By prepending it and applying masked loss, KoCo enables the model to distinguish knowledge properties.

Dimension	Labels and Examples
Source	Academic , Publication (Books/Journals),
	Government , Organization , Industry ,
	Media , Community ,
	Personal (Blogs), Codebase , Others
Content	Instruction , Pedagogical (Textbook),
	Reference , Data (Logs/Tables),

Table 1: The KoCo Taxonomy. This classification scheme categorizes documents based on their objective nature rather than their semantic topic.

Method	COPA	ARC-e	ARC-c	CSQA	IFEval	OBQA	PIQA	SIQA	LogiQA	TruQA	Avg
URL Prefix (MeCo)	82.00	75.40	44.37	63.96	20.02	50.80	73.00	52.90	25.50	36.33	52.42
+ Standard CPT	82.00	74.60	42.83	59.54	22.18	49.60	72.90	52.70	24.88	35.24	51.64
+ Data Selection	82.00	75.00	44.62	63.31	22.42	49.00	74.00	52.60	25.19	35.53	52.36
KoCo	83.00	77.40	44.11	61.83	25.54	51.20	74.80	53.40	26.88	36.61	53.48
	$\uparrow 1.00$	$\uparrow 2.00$	$\downarrow 0.26$	$\downarrow 2.13$	$\uparrow 5.52$	$\uparrow 0.40$	$\uparrow 1.80$	$\uparrow 0.50$	$\uparrow 1.38$	$\uparrow 0.28$	$\uparrow 1.06$

Table 2: Experiment results of pre-training a 1.6B language model on 8B tokens from DCLM. KoCo outperforms existing methods on most tasks and achieves a higher average score.

Figure 3: Pre-training from scratch results comparing KoCo (red) with standard paradigm (blue) on 0.3B and 0.6B models. KoCo consistently outperforms standard training across most tasks and achieves lower training loss, demonstrating accelerated convergence.

Setting	ARC-e	ARC-c	OBQA	PIQA	Avg.
w/o Source	76.20	44.11	50.20	73.70	53.43
w/o Content	76.60	43.60	51.20	74.10	53.46
w/o Stability	76.70	43.09	51.00	73.80	53.32
KoCo (full)	77.40	44.11	51.20	74.80	53.48

Table 3: Ablation study on the impact of each knowledge coordinate dimension.

Model	COPA	IFEval	SIQA	LogiQA	TruQA	Avg.
MeCo	82.00	20.02	52.90	25.50	36.33	52.42
+ cond. prefix	83.00	20.98	53.90	25.65	36.09	52.97
KoCo	83.00	25.54	53.40	26.88	36.61	53.48
+ cond. prefix	84.00	27.50	53.60	28.26	40.39	54.45
	$\uparrow 1.00$	$\uparrow 1.96$	$\uparrow 0.20$	$\uparrow 1.38$	$\uparrow 3.78$	$\uparrow 0.97$

Table 4: Performance comparison with task-specific prefixes.

Source	Content	Stability	TruQA
Personal (x.com)	Opinion	Ephemeral	34.75
Community (4chan.org)	Opinion	Ephemeral	35.07
Community (reddit.com)	Discussion	Ephemeral	35.35
-	-	-	36.61
Academic (Official Docs)	Instructional	Long-term	37.89
Publication (Official FAQ)	Instructional	Long-term	40.39

Table 5: Impact of different knowledge coordinate prefixes on TruthfulQA. Prefixes associated with unreliable or ephemeral sources (top) degrade truthfulness, while reliable sources (bottom) significantly enhance it.

Model Setting	Perplexity
KoCo + random prefix	6.04
KoCo + correct prefix	5.87

Table 6: Perplexity comparison on unseen documents. The correct knowledge coordinate (prefix) significantly reduces perplexity compared to random coordinates.

Figure 4: PCA visualization of the hidden states for factual (blue) vs. opinion-based (red) statements. KoCo learns to separate these concepts into distinct clusters, while the standard model shows substantial overlap.

Figure 5: Distribution of 10,000 documents sampled from the DCLM corpus in the knowledge coordinate.

Figure 6: Comparison of KoCo with different taggers. KoCo with BERT-base achieves comparable performance to KoCo with Qwen3-4B, both outperforming standard pre-training.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameters	Values
Optimizer	AdamW ( $\beta_{1}=0.9$ , $\beta_{2}=0.95$ )
Learning rate	$3e-4$
Weight decay	0.033
Batch size	4M tokens
Warmup	5% linear warmup
Schedule	Cosine decay to 10% of the peak learning rate

Appendix

Table 7: Hyperparameter settings for our experiments.

Param	Layers	Inter	Hidden	Heads
300M	20	3072	768	12
600M	24	4096	1024	16
1.6B	24	5504	2048	16

Appendix

Table 8: Model configurations for our experiments.

Param	300M	600M	1.6B
Tokens	8B	8B	8B
GPU hours	60	96	192

Appendix

Table 9: Resources required to train the models in our experiments (A800 GPU hours).

# Role: Knowledge Taxonomist & Data Curator # Core Task: Your task is to analyze the provided [TEXT] and its source [URL], and generate a single, concise “Knowledge Context Meta-Tag.” This tag serves solely to inform the training model of the “position”, “sourcle”, and “attribute” of the text it will read within the broader human knowledge system. Note that your goal is to classify text, not summarize the content of the text. Do not expand or modify existing information. — # Step Guide: 1. Analysis [URL]: * Check the domain name (e.g. ‘wikipedia.org’, ‘nature.com’, ‘615vgs.com’, ‘nytimes.com’). * Infer Source Type. Classification includes but is not limited to: * ‘Primary Academic’ (such as nature.com) * ‘Secondary Academic’ (such as textbook websites) * .. 2. Analysis [TEXT]: * Determine Topic Domain (e.g., basic physics, aerospace history, finance, entertainment). * Evaluate Context including but not limited to: * ‘Core Principle’ * ‘Specialized Knowledge’ * ‘Factual Report’ * … * Define Temporal Stability ( $F_{S}$ ) to measure the ability of knowledge to resist semantic or structural changes over time. High

F_{S}

indicates that the knowledge has long-term validity, broad consensus, and low temporal fluctuation * Evaluate Temporal Stability (Stability - $F_{S}$ ): * ‘Evergreen’ * ‘Long-term’ * ‘Decaying’ * ‘Ephemeral’ 3. **Generating meta-labels: ** Combine the above analysis into a single descriptive sentence. * Use the following strict format: ‘Source: <Source Type> (<Domain Name>). Content: A paragraph of text describing <Importance> and <Field of Knowledge>. Stability: <Timeliness>’ — # Example: [Example: Newton’s Laws] * [TEXT]: (An excerpt from a textbook describing Newton’s second law F=ma) * [URL]: ‘https://physics.stackexchange.com/questions/123’ * [Output]: ‘Source: Q&A Forum (physics.stackexchange.com). Content: A definition of a Core Principle within the Basic Physics domain. Stability: Evergreen.’ — Please generate meta tags for the corpus and URL you provided. Simply output your results in English [TEXT] {text} [URL] {url}

Appendix

Table 10: The prompt for tagging knowledge coordinate.

Tasks	Source	Content	Stability
COPA	KnowledgeBase (stackenfm.com)	Knowledge	Ephemeral
ARC-Easy	KnowledgeBase(simple.wikipedia.org)	Knowledge	Evergreen
ARC-Challenge	KnowledgeBase(simple.wikipedia.org)	Knowledge	Evergreen
CommonsenseQA	Textbook(textbook.com)	Instructional	Evergreen
IFEval	Official FAQ(faq.com)	Instructional	Long-term
OpenBookQA	KnowledgeBase(simple.wikipedia.org)	Knowledge	Evergreen

Appendix

Table 11: Customized prefixes for conditional inference

Setting	COPA	ARC-e	ARC-c	CSQA	IFEval	OBQA	PIQA	SIQA	LogiQA	TruQA	Avg
w/o Source	84.00	76.20	44.11	62.08	26.38	50.20	73.70	54.00	27.34	36.32	53.43
w/o Content	84.00	76.60	43.60	62.49	26.38	51.20	74.10	53.10	27.04	36.10	53.46
w/o Stability	82.00	76.70	43.09	62.49	25.78	51.00	73.80	53.20	27.34	37.87	53.32
KoCo	83.00	77.40	44.11	61.83	25.54	51.20	74.80	53.40	26.88	36.61	53.48

Appendix

Table 12: Detailed results on ablation study

Tokens	COPA	ARC-e	ARC-c	CSQA	IFEval	OBQA	PIQA	SIQA	LogiQA	TruQA	Avg
Standard
0.8B	58.00	28.90	26.62	20.15	19.90	28.80	52.10	39.40	26.42	50.73	35.10
1.6B	48.00	29.60	23.29	22.19	17.63	27.40	53.30	38.30	24.58	51.38	33.57
2.4B	51.00	31.90	25.00	21.62	18.23	30.60	55.20	38.20	25.96	50.01	34.77
3.2B	52.00	33.10	23.38	21.54	20.38	30.00	57.70	40.30	27.34	50.45	35.62
4.0B	51.00	34.20	24.15	22.36	22.42	27.60	57.20	39.20	24.12	48.89	35.11

Appendix

Table 13: Intermediate checkpoint results for the 0.3B-parameter, 8B-token runs.

Tokens	COPA	ARC-e	ARC-c	CSQA	IFEval	OBQA	PIQA	SIQA	LogiQA	TruQA	Avg
Standard
0.8B	54.00	30.90	24.83	22.36	17.75	32.60	52.10	39.90	25.50	50.59	35.05
1.6B	57.00	30.50	26.37	22.03	13.55	29.40	53.10	39.10	23.35	49.87	34.43
2.4B	61.00	34.10	25.34	24.49	12.11	32.80	55.30	38.50	24.58	50.20	35.84
3.2B	58.00	36.20	25.77	22.52	5.04	31.20	56.50	38.90	25.96	50.92	35.10
4.0B	55.00	36.90	25.77	24.08	8.39	30.80	55.60	39.60	25.81	49.41	35.14

Appendix

Table 14: Intermediate checkpoint results for the 0.6B-parameter, 8B-token runs.

Tokens	COPA	ARC-e	ARC-c	CSQA	IFEval	OBQA	PIQA	SIQA	LogiQA	TruQA	Avg
Standard
6B	48.00	37.60	27.13	25.80	22.54	31.20	56.90	42.00	26.42	48.61	36.62
12B	53.00	42.10	25.85	26.95	25.66	29.40	61.20	40.40	26.73	41.95	37.32
18B	54.00	44.70	25.68	26.54	25.30	32.40	60.90	41.20	26.88	41.08	37.87
24B	57.00	45.20	26.28	27.27	25.06	31.80	61.90	38.90	26.42	41.63	38.15
KoCo

Appendix

Table 15: Intermediate checkpoint results for the 0.3B-parameter extended pre-training runs (up to 24B tokens, 3

\times

more data than the main experiments). KoCo’s advantage over Standard grows from

+0.30

at 6B tokens to

+2.69

at 24B tokens.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 441 papers

KoRe: Compact Knowledge Representations for Large Language Models
May 19, 2026Davide Cavicchini, Fausto Giunchiglia, Jacopo StaianoLLM Grounding Knowledge Augmentation for Language Models
University of Trento, Via Calepina, 14, 38122 Trento TN, Italy
Co-LMLM: Continuous-Query Limited Memory Language Models
Jul 8, 2026Yair Feldman, Linxi Zhao, Nathan Godey +5Language Model Pretraining Memory-Augmented Language Models
Department of Computer Science Cornell University
Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling
Jul 14, 2026Roi Cohen, Yvan Carré, Nick Lechtenbörger +5Language Model Pretraining LLM Grounding
HPI / University of Potsdam · African Institute for Mathematical Sciences · Polytechnique Montréal +2

May 19, 2026J/K move · Enter open · S save

KoRe: Compact Knowledge Representations for Large Language Models

Davide Cavicchini, Fausto Giunchiglia, Jacopo Staiano

University of Trento, Via Calepina, 14, 38122 Trento TN, Italy

LLM Grounding Knowledge Augmentation for Language Models Knowledge Representation Knowledge Graphs

Open paper AskPDF arXiv

Modern Large Language Models (LLMs) have shown impressive performances in user-facing tasks such as question answering, as well as consistent improvements in reasoning capabilities. Still, the way these models encode knowledge seems inherently flawed: by design, LLMs encode world-knowledge within their parameters. This way of representing knowledge is inherently opaque, difficult to debug and update, and prone to hallucinations. On the other hand, Knowledge Graphs can provide human-readable and easily editable world knowledge representations, and their application in knowledge-intensive tasks has consistently proven beneficial to downstream performance. Nonetheless, current integration techniques require extensive retraining or finetuning. To overcome this issue, we introduce KoRe, a methodology to encode 1-hop sub-graphs into compact discrete knowledge tokens and inject them into a LLM backbone. We test the proposed approach on three established benchmarks, and report competitive performances coupled with a significant reduction (up to 10x) in token usage. Our results show that compact discrete KG representations can efficiently and effectively be used to ground modern LLMs.

Abstract

Language Model Pretraining

Explore similar work

KoRe: Compact Knowledge Representations for Large Language Models

Co-LMLM: Continuous-Query Limited Memory Language Models

Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling