Every Part Has Its Role: Part-of-Speech Effects in LLM Post-Training

Laya Iyer; Chen Shani

cs.CLOct 21, 2026

Every Part Has Its Role: Part-of-Speech Effects in LLM Post-Training

Authors: Laya Iyer, Chen Shani

Organizations: Stanford University · Tel Aviv University

Abstract

Large language models are typically trained with next-token prediction (NTP), which rewards predicting specific tokens rather than explicitly accounting for the semantic similarities between different words or expressions. Prior work has addressed this limitation by augmenting training data with minimally varying sentences that replace a target word with conceptually related alternatives, e.g., "The tired coach ran home" $\rightarrow$ {"The tired trainer ran home", "The tired instructor ran home", ...}. Such concept augmentation improves performance, but it remains unclear whether its effectiveness depends on where in the sentence the concept is expressed. We reproduce and extend prior work $\textit{beyond nouns}$ to verbs and adjectives, asking whether the part of speech (POS) carrying the concept signal affects perplexity and downstream performance. We find different patterns across evaluation settings: concept augmentation improves perplexity most consistently on news, independent of the POS used, whereas verbs provide the most reliable downstream gains. Despite these behavioral differences, representational analyses using CKA and linear probing show that the conditions yielding the largest downstream gains are not necessarily those that most alter the model's internal representations. Together, these results suggest that the effectiveness of concept augmentation depends on where and how conceptual information is introduced during training.

Figures & tables

Figure 1: Illustration of POS-specific concept augmentation. For the same input sentence, we generate synonym and hypernym substitutions using either context-aware or dictionary-based concept extraction. Rows show augmentation applied to nouns, verbs, adjectives, or all three POS categories together.

Figure 2: Overview of the concept-augmentation and evaluation pipeline. Datapoints are drawn from arXiv, news, YouTube, or the combined corpus, and target words are selected by part of speech. For each POS condition, training data are either left unchanged (vanilla) or augmented with context-aware or dictionary-based synonym or hypernym substitutions. Each resulting corpus is used to further train Llama-3-8B with the standard next-token prediction objective, producing 96 post-trained models, which are subsequently evaluated through LoRA-based fine-tuning on seven downstream classification tasks.

Figure 3: The effect of concept augmentation on perplexity depends on both post-training corpus and POS. Perplexity improvement of concept-augmented models relative to their matched vanilla baselines, separated by post-training corpus. Positive values indicate lower perplexity after augmentation. Points show context-aware and dictionary-based augmentation for nouns (N), verbs (V), adjectives (Adj), and all three parts of speech (ALL); dotted horizontal lines show the mean improvement for each corpus. Augmentation improves perplexity most consistently when post-trained on news ( $+5.5\%$ ), while it decreases performance on arXiv ( $-8.2\%$ ). The relative effect of each POS also changes substantially across corpora.

Figure 4: Change in language-model quality against change in downstream performance, each measured relative to the model’s own vanilla counterpart. Each point is one post-trained model; the $x$ -axis is the reduction in perplexity averaged over the four evaluation corpora, and the $y$ -axis is the gain in accuracy averaged over the seven downstream tasks. Points in the shaded upper-right quadrant improved on both measures. Marker shape and colour denote the substituted POS; filled markers are hypernym substitution and open markers denote synonym substitution. The correlation is $r=+0.38$ ( $p=0.033$ ) for context augmentation and $r=+0.30$ ( $p=0.096$ ) for dictionary augmentation.

Figure 5: Downstream effects depend on both the augmented POS and evaluation task. (a) Mean accuracy change relative to the matched vanilla baseline for each semantic-relation $\times$ corpus $\times$ POS condition, averaged over the seven downstream tasks and the context-aware and dictionary-based variants. Positive values indicate improvement over vanilla; outlined cells mark the best POS within each row. Verbs achieve the strongest mean performance in six of the eight settings. (b) Range of accuracy changes across augmentation conditions for each downstream task. GLUE, SNLI, and Logical Fallacy are substantially more sensitive to the augmentation choice than Spam, Hate Speech, and Emotion.

Figure 6: POS affects when downstream fine-tuning stops. Mean standardized stopping point across downstream tasks, grouped by the POS augmented during post-training. Adjective-augmented models stop significantly earlier than all-POS models ( $p=0.001$ ), indicating that the breadth of the concept signal affects downstream convergence.

Figure 7: Post-training corpus affects early fine-tuning speed. Standardized accuracy gain between fine-tuning steps 250 and 500, grouped by post-training corpus. Models post-trained on the combined corpus improve significantly faster over the first interval than models post-trained on YouTube ( $p=0.004$ ).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Reproducibility check against the noun-only experiments of Iyer et al. (2026) . Comparison of the original results with our noun-based conditions. Left: perplexity, colored by post-training corpus. Right: downstream accuracy, colored by task. The diagonal indicates identical performance. Our pipeline recovers the same broad trends, with small differences due to data reprocessing and the revised downstream fine-tuning procedure.

Figure 9: Absolute perplexity across post-training and evaluation corpora. Each panel corresponds to one post-training corpus, and the $x$ -axis denotes the evaluation corpus. Co lor denotes the augmented POS. Solid lines show concept-augmented models and dotted lines show their corresponding vanilla conditions. Values are averaged across synonym and hypernym settings.

Relation	Post-training corpus	Nouns	Verbs	Adjectives	All POS
Hypernym	arXiv	$-3.7$	$\mathbf{+4.5}$	$-3.4$	$-3.7$
	News	$-1.4$	$\mathbf{+2.9}$	$-0.4$	$+0.2$
	YouTube	$\mathbf{+2.1}$	$-1.5$	$+1.3$	$+2.0$
	Combined	$-0.1$	$\mathbf{+0.5}$	$-0.7$	$-0.6$
Synonym	arXiv	$-2.7$	$\mathbf{+2.2}$	$-0.3$	$+1.6$
	News	$-1.1$	$-2.4$	$\mathbf{+3.4}$	$+0.6$

Appendix

Table 1: Downstream accuracy change by augmented POS. Mean accuracy change in percentage points relative to matched vanilla models, separated by semantic relation, post-training corpus, and augmented POS. Context-aware and dictionary-based variants are averaged. Bold denotes the strongest POS within each relation-corpus condition. Verb augmentation is strongest in six of the eight conditions.

Relation	Corpus	Context	Dictionary
Hypernym	arXiv	$\mathbf{-1.1}$	$-2.1$
	News	$-0.3$	$\mathbf{+1.0}$
	YouTube	$0.0$	$\mathbf{+1.9}$
	Combined	$-0.7$	$\mathbf{+0.2}$
Synonym	arXiv	$-1.5$	$\mathbf{+1.9}$
	News	$\mathbf{+0.8}$	$-0.5$

Appendix

Table 2: Downstream accuracy change by augmentation source. Mean accuracy change in percentage points relative to matched vanilla models, separated by semantic relation, post-training corpus, and concept source. Results are averaged across POS conditions and downstream tasks. Bold denotes the stronger source within each row.

Post-training corpus	Nouns	Verbs	Adjectives	All POS	Mean
arXiv	0.750	0.788	0.768	0.696	0.751
News	0.811	0.825	0.794	0.770	0.800
YouTube	0.809	0.820	0.791	0.746	0.792
Combined	0.814	0.807	0.807	0.761	0.797
Mean	0.796	0.810	0.790	0.743	0.785

Appendix

Table 3: Final-layer CKA with the base model by POS and post-training corpus. Values are averaged across context-aware, dictionary-based, and matched vanilla hypernym conditions. Lower values indicate greater representational divergence. All-POS models show the greatest divergence, particularly after arXiv post-training, while verb models remain closest to the base model.

	Mean across probed layers		Final probed layer
Target POS	Context	Dictionary	Context	Dictionary
Nouns	$-0.04$	$+0.08$	$-0.12$	$+0.07$
Verbs	$+0.05$	$+0.02$	$+0.06$	$+0.10$
Adjectives	$-0.24$	$-0.05$	$-0.33$	$+0.36$
All POS	$+0.04$	$\mathbf{+0.17}$	$+0.16$	$\mathbf{+0.45}$

Appendix

Table 4: Concept-specific change in POS-probe accuracy relative to matched vanilla models. Values are percentage-point differences between the concept-augmented and matched vanilla probe accuracies. All-POS augmentation shows the clearest positive concept-specific effect, particularly for dictionary-based augmentation. These entries are differences of point estimates; bootstrap confidence intervals in the probe analysis are defined relative to the untouched base model rather than for the concept-minus-vanilla contrast.

Variant	N	V	Adj	All	Mean
Vanilla	0.821	0.805	0.799	0.760	0.796
Context	0.805	0.807	0.789	0.743	0.786
Dict.	0.763	0.817	0.783	0.728	0.773

Appendix

Table 5: Final-layer CKA with the base model by augmentation source. Values are averaged across post-training corpora for the hypernym conditions. Dictionary-based models have lower mean CKA than their matched vanilla models, although the effect is not uniform across POS categories.

Layer	Context	Dictionary	Vanilla
0	$-0.06$	$-0.08$	$-0.08$
4	$-0.10$	$-0.06$	$-0.10$
8	$+0.15$	$+0.27$	$+0.25$
12	$+1.12$	$+1.23$	$+1.23$
16	$+0.31$	$+0.32$	$+0.33$
20	$+0.71$	$+0.76$	$+0.76$

Appendix

Table 6: Mean POS-probe accuracy change relative to the untouched base model. Values are percentage-point changes averaged over POS targets, corpora, and semantic relations. All three variants become more POS-decodable in later layers, including the matched vanilla models.

Language Model Post-Training

Language model post-training adapts pretrained models to follow instructions, reason, use tools, and meet task-specific or human-defined goals.

See all 195 papers

On the (In)effectiveness of AMR Augmentation for Large Language Models
Sep 30, 2026Hoa Quynh Nhung Nguyen, Jacopo Staiano, Michael SullivanLLM Evaluation Knowledge Augmentation for Language Models
Saarland University · University of Trento
Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models
Mar 31, 2026Christine Zhang, Dan Jurafsky, Chen ShaniLanguage Modeling LLM Alignment
Stanford University
Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
Apr 9, 2026Shiwan Zhao, Zhihu Wang, Xuyang Zhao +10LLM Post-Training RL for Language Models
Nankai University, China · Huawei Technologies Ltd., China

Sep 30, 2026J/K move · Enter open · S save

On the (In)effectiveness of AMR Augmentation for Large Language Models

Hoa Quynh Nhung Nguyen, Jacopo Staiano, Michael Sullivan

Saarland University · University of Trento

LLM Evaluation Knowledge Augmentation for Language Models

Open paper AskPDF arXiv

While Abstract Meaning Representation (AMR) has historically improved performance on a range of NLP tasks, the benefit---or lack thereof---of AMR augmentation for modern LLMs is thus far unclear. In this paper, we attempt to reproduce recent work that reported substantial downstream gains from AMR augmentation, finding that these are likely due to specific choices in the experimental settings used: using a consistent and unified protocol for hyperparameter selection, we observe that text-only baselines consistently match or exceed the performance of AMR-augmented models. To investigate this null result, we introduce a perplexity-based probe measuring the degree to which AMR provides an LLM with supplemental relational knowledge not already available to the model. We find that AMR augmentation does not help LLMs improve their understanding of relational content in the sentence, indicating that augmenting these models with AMR offers no clear benefit on downstream tasks.

Abstract

Language Model Post-Training

Explore similar work

On the (In)effectiveness of AMR Augmentation for Large Language Models

Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning