Structural Abstraction as an Inductive Bias for Non-Stationary Language Model Training

Elnaz Rahmati; Nona Ghazizadeh; Zhivar Sourati; Nina Rouhani; Morteza Dehghani

cs.LGMar 17, 2026

Structural Abstraction as an Inductive Bias for Non-Stationary Language Model Training

Authors: Elnaz Rahmati, Nona Ghazizadeh, Zhivar Sourati, Nina Rouhani, Morteza Dehghani

Organizations: University of Southern California

Abstract

A foundational principle in cognitive science holds that intelligent agents do not learn by storing experiences as isolated instances, but by forming abstract schemas that capture relational structure shared across situations. Even though this claim is well supported by behavioral and neuroimaging studies, its role as a computational training signal in language models remains underexplored. We target this gap in the setting of non-stationary language model training, asking does biasing learning toward structural abstraction reduce catastrophic interference and improve relational generalization as predicted by human results? To study this question, we introduce Abstraction-Augmented Training (AAT), a lightweight loss-level modification that jointly optimizes over concrete instances and their structural abstractions, and two benchmarks, the Relational Cycle Benchmark (RCB) and the Narrative Abstraction Benchmark (NAB). These resources operationalize core cognitive constructs: entity masking as a computational analog of relational alignment, and proverbs as vehicles for implicit abstract meaning that must be inferred across surface-dissimilar situations. Our empirical results demonstrate that AAT consistently reduces forgetting and improves generalization in a pattern that aligns with cognitive predictions for schema-based learning. Beyond the practical implications for continual learning, these results offer preliminary computational evidence that structural abstraction is a signal for stable learning in non-stationary environments.

Figures & tables

Figure 1: Online accuracy and forgetting over training steps on the RCB.

Method	$\textbf{Acc}_{\textbf{cumul}}^{\textbf{known}}$	$\textbf{Acc}_{\textbf{cumul}}^{\textbf{unknown}}$	$\textbf{F}^{\textbf{known}}$	$\textbf{F}^{\textbf{unknown}}$
Random	0.4756	0.3898	0.1163	0.0845
Category	0.5403	0.4059	0.0713	0.0657
Mask	0.5452	0.4086	0.0679	0.0643

Table 1: Performance across abstraction levels.

Method	$\textbf{Acc}_{\textbf{cumul}}^{\textbf{known}}$	$\textbf{Acc}_{\textbf{cumul}}^{\textbf{unknown}}$	$\textbf{F}^{\textbf{known}}$	$\textbf{F}^{\textbf{unknown}}$
Normal SFT	0.8674	0.6086	0.0565	0.1460
Buffer-100	0.8940	0.5965	0.0631	0.1730
AAT	0.8683	0.6143	0.0557	0.1429

Table 2: Performance comparison on the NAB.

Model	Method	$\textbf{Acc}_{\textbf{online}}^{\textbf{unknown}}$	$\textbf{Acc}_{\textbf{online}}^{\textbf{known}}$	$\textbf{F}^{\textbf{unknown}}$	$\textbf{F}^{\textbf{known}}$	$\textbf{Acc}_{\textbf{cumul}}^{\textbf{unknown}}$	$\textbf{Acc}_{\textbf{cumul}}^{\textbf{known}}$	$\textbf{Acc}_{\textbf{cumul}}^{\textbf{all}}$
	Normal SFT	0.4194	0.5487	0.0707	0.0767	0.4032	0.5330	0.4924
Qwen-1.5B	Buffer-100	0.4023	0.5656	0.0626	0.0635	0.3835	0.5515	0.4990
	AAT	0.4220	0.5568	0.0643	0.0679	0.4086	0.5452	0.5025
	Normal SFT	0.4115	0.5730	0.1166	0.0496	0.3828	0.5552	0.5004
Qwen-3B	Buffer-100	0.4115	0.5818	0.1161	0.0483	0.3802	0.5721	0.5112
	AAT	0.4141	0.5708	0.1035	0.0507	0.3880	0.5552	0.5021

Table 3: Step-averaged online accuracy, forgetting, and cumulative accuracy on the RCB benchmark for Normal SFT, ER with an 100-sample buffer, and AAT across Qwen-1.5B, Qwen-3B, and SmolLM.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Effect of abstraction loss weight $\alpha$ on cumulative accuracy.

Domain	Total	Derivable	% Deriv.	Known Edges	Unknown Edges
Genealogy	500	200	40.0%	700	500
Profession	180	180	100.0%	900	180
Arts	115	50	43.5%	280	115
Science	100	25	25.0%	125	100
History	100	100	100.0%	200	100
Music	100	25	25.0%	125	100

Appendix

Table 4: Dataset statistics for RCB.

Model Size	RCB			NAB
Model Size	Unknown	Known	Total	Unknown	Known	Total
1.5B	0.3469	0.4695	0.4311	0.5420	0.5103	0.5134
3B	0.3020	0.4537	0.4062	0.5222	0.5266	0.5262
7B	0.4357	0.6062	0.5528	0.5157	0.5198	0.5194

Appendix

Table 5: Performance across model sizes on the RCB and the NAB.

Model	Method	$\textbf{Acc}_{\textbf{online}}^{\textbf{unknown}}$	$\textbf{Acc}_{\textbf{online}}^{\textbf{known}}$	$\textbf{F}^{\textbf{unknown}}$	$\textbf{F}^{\textbf{known}}$	$\textbf{Acc}_{\textbf{cumul}}^{\textbf{unknown}}$	$\textbf{Acc}_{\textbf{cumul}}^{\textbf{known}}$	$\textbf{Acc}_{\textbf{cumul}}^{\textbf{all}}$
	Buffer-50-lp1	0.4176	0.5699	0.0792	0.0833	0.3934	0.5275	0.4856
Qwen-1.5B	Buffer-100-lp1	0.3772	0.5283	0.0950	0.0817	0.3450	0.5051	0.4551
	Buffer-50-lp5	0.4014	0.5679	0.0723	0.0689	0.3844	0.5515	0.4993
	Buffer-100-lp5	0.4023	0.5656	0.0626	0.0635	0.3835	0.5515	0.4990

Appendix

Table 6: Step-averaged online accuracy, forgetting, and cumulative accuracy on the RCB benchmark for ER configurations.

Proverb	Narrative Context	Correct Ending	Distractor Ending
Jam tomorrow…	The man believed his friend when he kept borrowing money from him, telling him he would pay him tomorrow. Every day he told him he would pay him soon.	He kept giving him money, thinking of the future, but never saw the loans repaid.	He finally got fed up with the constant requests and decided to save all his money for emergencies.
Keep your powder dry	Margo always knew to start any trip with a spare tire, emergency cash, and a car repair kit. She stayed calm and prepared for any catastrophe.	She was yet to need any of this, but she always figured she’d be best prepared if catastrophe ever struck.	With all the supplies in her car, Margo felt ready for anything and drove off, never expecting to encounter issues.
Laughter is the best medicine	My grandfather was ill and in a somber mood. I read him a funny speech about a plagiarism scandal, and his eyes lit up with fascination.	The somber mood in the room dissipated like dew as we all wondered loudly, and mirthfully, how that was possible.	The somber mood in the room dissipated as my grandfather’s eyes lit up with curiosity about the scandal.
Procrastination is thief of time	My son put off an oil change that would have taken 20 minutes. 6 months later, he was sitting at the side of the road with a seized-up engine.	Ultimately, we figured the delay resulted in over 29 hours of combined waiting, extra travel, and car searching.	Fortunately, my son managed to get the engine fixed quickly by borrowing a friend’s car, saving him from delays.
It takes two to tango	Jeff and Jane were getting into a fight. Jane pointed the finger, saying it was all Jeff’s fault. Their mother knew better though.	She knew that they both were causing trouble and Jeff and Jane were both punished.	She quickly intervened and helped Jeff up off the floor, advising him to apologize to Jane.

Appendix

Table 7: Random samples from the NAB: Proverbs, Narratives and Continuation Task.

Text Context	Known Edges (Sample)	Unknown Edge (Target)
Ljubljana is in Slovenia. Academy of Theatre, Radio, Film and Television is in Ljubljana. Polde Bibič works for University of Ljubljana and was educated at the Academy.	(Polde Bibič, works for, Univ. of Ljubljana), (Polde Bibič, educated at, Acad. of Theatre…), (Ljubljana, country, Slovenia)	(Univ. of Ljubljana, based in country, Slovenia)
Closing time is a dance film and has Mette Ingvartsen as a cast member.	(Closing time, cast member, Mette Ingvartsen), (Closing time, genre, dance)	(Mette Ingvartsen, field of work, dance)

Appendix

Table 8: Random samples from the RCB: Text Context and Knowledge Graph Edges

Panel of Figure 5 — Figure 5: Distractor generation prompt used for NAB.

Panel of Figure 6 — Figure 6: Narrative expansion prompt used for NAB.

LLM Training

Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.

See all 342 papers

Mar 18, 2026

Humans and transformer LMs: Abstraction drives language learning

Categorization is a core component of human linguistic competence. We investigate how a transformer-based language model (LM) learns linguistic categories by comparing its behaviour over the course of training to behaviours which characterize abstract feature-based and concrete exemplar-based accounts of human language acquisition. We investigate how lexical semantic and syntactic categories emerge using novel divergence-based metrics that track learning trajectories using next-token distributions. In experiments with GPT-2 small, we find that (i) when a construction is learned, abstract class-level behaviour is evident at earlier steps than lexical item-specific behaviour, and (ii) that different linguistic behaviours emerge abruptly in sequence at different points in training, revealing that abstraction plays a key role in how LMs learn. This result informs the models of human language acquisition that LMs may serve as an existence proof for.

Jasper Jian, Christopher D. Manning

Stanford University

LLM Training Language Model Pretraining

Ask PDF

Aug 1, 2026

Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction-First Learning

Whether idiosyncratic, item-specific knowledge is learned before abstract class-level generalizations, or vice versa, is a central question in language learning, with exemplar and abstraction-based theories making opposite predictions. Recent methods have claimed to show that, at least for large language models, abstract knowledge is learned first. We show that these methods fall short: pure memorizer models with no abstract representations can appear, by the same criteria, to learn either item-specific or class-level knowledge first, depending on their sensitivity to individual observations, with the transition point governed by the distributional properties of the input. We further argue that the distinction between item-specific and abstract knowledge may be ill-defined for distributed representations, as a word's class-level properties may not be separable from its item-specific properties.

Zachary Nicholas Houghton, Vsevolod Kapatsinski

University of Oregon · Vail Systems, Inc.

Memorization in Language Models Language Modeling

Ask PDF

Sep 17, 2026

Learn Your Own Thoughts: Abstract Token Curriculum

Large Language Models (LLMs) have achieved remarkable reasoning capabilities by utilizing chain-of-thought (CoT) as a scratchpad for intermediate stages of thinking. However, CoT techniques require explicit supervision on thinking tokens, which requires rich, task-specific data. In this work, we propose Abstract Token Curriculum (ATC), a novel curriculum learning framework that elicits effective continuous intermediate representations without direct supervision or manual scratchpad design. ATC gradually increases problem complexity through a sequence of distributions, training the model to develop internal abstract

thoughts'' in the continuous representation space. This paper provides both theoretical and experimental evidence for the benefits of ATC and its advantages over previous methods for training continuous thoughts. Theoretically, we show that for learning parity functions with single-layer softmax attention using ATC, attention naturally focuses on the CoT tokens in the context that provide the

easiest path'' to predicting the next token. Experimentally, we show ATC's effectiveness on graph reachability and arithmetic learning tasks.

Khashayar Gatmiry, Avrajit Ghosh, Parsa Mirtaheri +4

UC San Diego · Google DeepMind and UC Berkeley

CoT Reasoning Curriculum Learning LLM Training Latent Reasoning

Ask PDF

Abstract

LLM Training

Explore similar work

Humans and transformer LMs: Abstraction drives language learning

Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction-First Learning

Learn Your Own Thoughts: Abstract Token Curriculum