On the (In)effectiveness of AMR Augmentation for Large Language Models

Hoa Quynh Nhung Nguyen; Jacopo Staiano; Michael Sullivan

cs.CLSep 30, 2026

On the (In)effectiveness of AMR Augmentation for Large Language Models

Authors: Hoa Quynh Nhung Nguyen, Jacopo Staiano, Michael Sullivan

Organizations: Saarland University · University of Trento

Abstract

While Abstract Meaning Representation (AMR) has historically improved performance on a range of NLP tasks, the benefit---or lack thereof---of AMR augmentation for modern LLMs is thus far unclear. In this paper, we attempt to reproduce recent work that reported substantial downstream gains from AMR augmentation, finding that these are likely due to specific choices in the experimental settings used: using a consistent and unified protocol for hyperparameter selection, we observe that text-only baselines consistently match or exceed the performance of AMR-augmented models. To investigate this null result, we introduce a perplexity-based probe measuring the degree to which AMR provides an LLM with supplemental relational knowledge not already available to the model. We find that AMR augmentation does not help LLMs improve their understanding of relational content in the sentence, indicating that augmenting these models with AMR offers no clear benefit on downstream tasks.

Figures & tables

Figure 1: Prior work ( Zhang et al. , 2025 ; top) reports substantial improvement over a text-only baseline from AMR-augmentation. We revisit this method under a more robust evaluation (bottom), finding that AMR augmentation yields no improvement.

Figure 2: Abstract Meaning Representation for two different input sentences (a) with the same semantics. (b) shows the AMR graph and (c) represents the corresponding linearization using PENMAN notation. Example adapted from the AMR Guidelines ( https://github.com/amrisi/amr-guidelines ).

Format	Example
Text-only	{ sys } You are an expert in machine translation…
	{ user } Obama receives Netanyahu
	{ asst } Obama empfängt Netanyahu
+AMR	{ sys } You are an expert in machine translation…
	{ user } Sentence: Obama receives Netanyahu
	AMR: (r / receive-01 :ARG0 (p / person :name (n / name :op1 "Obama")) :ARG1 (p2 / person :name (n2 / name :op1 "Netanyahu")))

Table 1: Examples of the text-only (top) and AMR-augmented (bottom) prompt formats for the WMT16 translation task. System messages truncated for illustration; the full prompts are located in Appendix A.2 .

Model	Training	Setup	WiC	SST	SNLI	PM	PAWS	AGN	SPD	CNL	WMT
			(F1)	(F1)	(F1)	(F1)	(F1)	(F1)	(EM)	(F1)	(BLEU)
Qwen3-8B	Joint	Text	$76.8$ $\pm$ 0.3	$\mathbf{96.5}$ $\pm$ 1.2	$92.3$ $\pm$ 0.8	$\mathbf{70.4}$ $\pm$ 2.1	$92.9$ $\pm$ 0.7	$92.3$ $\pm$ 0.5	$59.2$ $\pm$ 1.1	$\mathbf{92.4}$ $\pm$ 0.3	$\mathbf{28.2}$ $\pm$ 0.5
		+AMR	$76.4$ $\pm$ 0.4	$96.1$ $\pm$ 1.9	$92.1$ $\pm$ 1.2	$68.3$ $\pm$ 2.5	$\mathbf{93.0}$ $\pm$ 0.4	$91.2$ $\pm$ 0.3	$\mathbf{61.9}$ $\pm$ 1.2	$92.1$ $\pm$ 0.6	$\mathbf{28.2}$ $\pm$ 0.8
		+AMR +Inter.	$\mathbf{77.5}$ $\pm$ 1.7	$96.2$ $\pm$ 0.3	$\mathbf{92.4}$ $\pm$ 1.3	$67.2$ $\pm$ 4.8	$92.7$ $\pm$ 0.7	$\mathbf{92.5}$ $\pm$ 0.3	$58.8$ $\pm$ 1.5	$92.3$ $\pm$ 1.6	$27.6$ $\pm$ 0.7
	Individual	Text	$\mathbf{75.9}$ $\pm$ 2.1	$96.2$ $\pm$ 0.5	$\mathbf{92.4}$ $\pm$ 1.2	$80.2$ $\pm$ 2.5	$\mathbf{94.3}$ $\pm$ 0.3	$93.2$ $\pm$ 0.8	$\mathbf{57.8}$ $\pm$ 1.2	$93.2$ $\pm$ 0.5	$28.3$ $\pm$ 0.2
		+AMR	$74.6$ $\pm$ 1.7	$\mathbf{96.8}$ $\pm$ 1.4	$91.2$ $\pm$ 1.3	$81.3$ $\pm$ 1.4	$94.1$ $\pm$ 0.4	$\mathbf{93.4}$ $\pm$ 0.5	$57.6$ $\pm$ 1.3	$93.3$ $\pm$ 0.3	$\mathbf{28.5}$ $\pm$ 0.1

Table 2: Joint and individual training results on the nine single-sentence task datasets for Llama-3.1-8B-Instruct and Qwen3-8B, compared to the results reported in Zhang et al. (2025) for Llama-3.1-8B. The +SR scores from Zhang et al. (2025) are averaged over multiple SR frameworks, including AMR (see the discussion in Section 3.2 ). Each cell reports mean

\pm

standard deviation over runs with different random seeds. The best-performing Setup within each Model, Training type, and dataset is indicated in bold . PM=PubMed45, AGN=AGNews, SPD=SPIDER, CNL=CoNLL2003, WMT=WMT16.

Individual training
Dataset	Llama	Qwen	Pooled
WMT	0.370	0.067	0.985
PAWS	0.166	0.273	0.064
WiC	0.954	0.631	0.717
SPIDER	0.790	0.614	0.901
AGNews	0.590	0.297	0.211

Table 3: Results (

p

-values) for two-sided

t

-tests comparing the text-only and AMR-augmented (“+AMR”) scores in Table 2 —significant values (

p<0.05

) are indicated in bold . The “Pooled” column denotes a

t

-test over the combined Llama and Qwen results for each task.

Figure 3: Templates for text-only (left) and AMR-augmented (right) prompts for the multi-sentence tasks.

Dataset	Task	Train	Dev	Test
RAMS	Event Argument Extraction	7000	849	817
CNN/DM	Abstractive Summarization	5000	500	500
ANLI	Natural Language Inference	7000	700	1200
LogiQA	Reading Comprehension	3828	478	480

Table 4: Split sizes for the multi-sentence task datasets.

Model	Training	Setup	RAMS	LogiQA	ANLI	CNN
			(F1)	(Acc.)	(F1)	(ROUGE-L)
Llama-3.1-8B	Joint	Text	50.0 ±0.6	54.9 ±0.5	88.5 ±0.5	31.5 ±0.4
		+AMR	49.1 ±0.5	53.6 ±0.5	88.1 ±0.4	31.2 ±0.4
		+AMR +Inter.	49.5 ±0.7	53.8 ±0.5	86.8 ±0.6	31.4 ±0.4
	Individual	Text	49.4 ±0.7	55.7 ±1.5	87.7 ±0.9	31.0 ±0.7
		+AMR	50.1 ±0.6	57.2 ±0.7	87.4 ±0.5	30.7 ±0.5

Table 5: Joint and individual training results on the four multi-sentence task datasets for Llama-3.1-8B-Instruct and Qwen3-8B. Each cell reports mean

\pm

standard deviation over five runs with different random seeds. The best-performing setup within each model, training type, and dataset is indicated in bold .

Input Type	User Prompt
Text-only	Input Sentence: ‘‘The NBA season of 1975--76 was the 30th season of the National Basketball Association.’’
Text+AMR	Input Sentence: ‘‘The NBA season of 1975--76 was the 30th season of the National Basketball Association.’’ Input AMR: (s / season :ord (o / ordinal-entity :value 30) :poss (l / league :name (n / name :op1 ...)))
Text+AMR-nodes	Input Sentence: ‘‘The NBA season of 1975--76 was the 30th season of the National Basketball Association.’’ Supplement: season, ordinal-entity, 30, league, name, National, Basketball, Association, date-interval, date-entity, 1975, date-entity, 76

Table 6: Input types and their corresponding user prompts. AMR is truncated for presentability. Full example prompts are provided in Appendix E.2 .

Figure 4: Our AMR-NLD generation pipeline, illustrated with an example sentence. The AMR graph for the input sentence is decomposed into core relational branches (e.g. :ord, :poss, :time), each of which is mapped to an individual description in natural language.

Llama
Model	Text	+AMR	+AMR-Nodes
Base	$4.42$	$3.64$	$3.72$
Text-FT	$3.58_{\pm 0.09}$	$3.06_{\pm 0.11}$	$3.16_{\pm 0.13}$
AMR-FT	$3.66_{\pm 0.10}$	$3.24_{\pm 0.17}$	$3.33_{\pm 0.10}$
Inter-FT	$4.66_{\pm 0.15}$	$4.20_{\pm 0.14}$	$4.24_{\pm 0.13}$
Qwen

Table 7: Mean perplexity (

\pm

std) across model configurations and input conditions.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Task	Zhang et al. 2025			Our Experiment
		Train	Val	Test	Train	Val	Test
PAWS	Paraphrase Detection	10,000	N/A	8,000	10,000	1,000	8,000
SNLI	Textual Entailment	10,000	N/A	10,000	10,000	1,000	10,000
WMT16	Translation	10,000	N/A	5,999	10,000	1,000	5,999
CoNLL2003	Named Entity Recog.	10,000	N/A	3,453	10,000	1,000	3,453
LOGIC	Logical Fallacy Det.	10,000	N/A	2,449	N/A	N/A	N/A

Appendix

Table 8: Data statistics for Zhang et al. (2025) ’s fine-tuning experiment and our fine-tuning experiment

Train	Dataset	Setup	LR
Indiv.	PubMed	Text	$5e{-5}$
		AMR	$5e{-5}$
		AMR-i.	$5e{-5}$
	WMT	Text	$5e{-5}$
		AMR	$5e{-5}$
		AMR-i.	$5e{-5}$

Appendix

Table 9: Learning rates for Llama per training configuration. AMR-i. stands for AMR-Intermediate. Dataset names are simplified for illustration purposes.

Train	Dataset	Setup	LR
Indiv.	PubMed	Text	$5e{-5}$
		AMR	$5e{-5}$
		AMR-i.	$5e{-5}$
	WMT	Text	$5e{-5}$
		AMR	$5e{-5}$
		AMR-i.	$5e{-5}$

Appendix

Table 10: Learning rates for Qwen per training configuration. AMR-i. stands for AMR-Intermediate.

Parameter	Value
LLM Related
Attention impl.	flash_attention_2
Dtype	bfloat16
LoRA Configuration
Rank ( $r$ )	64
Alpha	128

Appendix

Table 11: Configuration for single-sentence fine-tuning.

Training	Setup	WiC	SST	SNLI	PM	PAWS	AGN	SPD	CNL	WMT
		(F1)	(F1)	(F1)	(F1)	(F1)	(F1)	(EM)	(F1)	(BLEU)
Joint	Text	76.2	96.1	92.1	67.4	93.9	92.6	59.1	93.0	30.2
Joint	+AMR	78.1	96.7	92.0	73.9	93.5	92.1	61.5	92.4	31.3
Individual	Text	78.0	96.7	92.2	83.9	94.2	93.6	59.4	92.9	31.3
Individual	+AMR	78.5	97.0	92.0	85.2	93.7	93.2	58.8	93.6	28.2

Appendix

Table 12: Joint and individual training results on the nine single-sentence task datasets for Llama-3.1-70B-Instruct. The best-performing setup within each training type and dataset is indicated in bold .

{ System } Generate a sentence for the given Abstract Meaning Representation.

{ User } (r / receive-01 :ARG0 (p / person :name :op1 ‘‘Obama’’) :ARG1 (p2 / person :name ‘‘Netanyahu’’))

{ Assistant } Obama receives Netanyahu

Appendix

Figure 5: Prompt template used for AMR-to-text generation.

LLaMA3.1-8B-Ins
Model Strategy	BLEU Score
Phase 1 checkpoint	46.0
Phase 2 checkpoint	43.9
Qwen3-8B
Phase 1 checkpoint	45.7
Phase 2 checkpoint	43.1

Appendix

Table 13: AMR-to-text generation performance (BLEU) for LLaMA and Qwen models

Train	Model	Dataset	Setup	LR
Indiv.	Llama	RAMS	Text	$5e{-5}$
			AMR	$1e{-5}$
			AMR-i.	$3e{-5}$
		LogiQA	Text	$1e{-5}$
			AMR	$1e{-5}$
			AMR-i.	$3e{-5}$

Appendix

Table 14: Learning rates per training configuration for RAMS, CNN, LogiQA, and ANLI. AMR-i. stands for AMR-Intermediate

Parameter	Value
LLM Related
Attention impl.	flash_attention_2
Dtype	bfloat16
LoRA Configuration
Rank ( $r$ )	64
Alpha	128

Appendix

Table 15: Configuration for multi-sentence fine-tuning.

Strategy	Training	RAMS	LogiQA	ANLI	CNN
		(F1)	(Acc.)	(F1)	(R-L)
GNN	Joint	50.2	54.6	88.8	31.0
GNN	Indiv.	50.7	57.5	88.1	31.2
AMRBART	Joint	49.5	56.9	88.7	32.1
AMRBART	Indiv.	49.4	56.9	87.7	31.2

Appendix

Table 16: Multi-sentence task results for Llama-3.1-8B-Instruct with GNN- and AMRBART-encoded AMR soft prompts.

Strategy	Training	WiC	SST	SNLI	PM	PAWS	AGN	SPD	CNL	WMT
		(F1)	(F1)	(F1)	(F1)	(F1)	(F1)	(EM)	(F1)	(BLEU)
GNN	Joint	74.0	95.8	90.9	71.3	93.2	88.8	53.4	90.9	27.7
GNN	Individual	74.7	96.2	91.0	74.4	93.5	93.0	52.1	92.4	26.6
AMRBART	Joint	72.3	96.0	91.0	81.7	93.7	74.9	52.8	93.0	26.1
AMRBART	Individual	75.0	96.3	91.5	84.4	93.9	91.4	49.8	92.2	28.4

Appendix

Table 17: Single-sentence task results for Llama-3.1-8B-Instruct with GNN- and AMRBART-encoded AMR soft prompts.

Type of Prompt	System Prompt	User Prompt
Text-only	Task: Deconstruct the following sentence into its underlying logical structure. Describe the meaning by identifying core events, the entities involved, and how they relate to one another. Example: Input Sentence: "The NBA season of 1975 -- 76 was the 30th season of the National Basketball Association ." Output: This refers to a season of the National Basketball Association. It was the 30th season of the league. The season took place during the 1975-76 time period.	Sentence: ‘‘Captain’’ was hulked in 1739 , and eventually broken up in 1762 .
Text + AMR	Task: Deconstruct the following sentence into its underlying logical structure. Describe the meaning by identifying core events, the entities involved, and how they relate to one another. You may use the provided AMR (Abstract Meaning Representation) as a supplement to guide you through generation. Example: Input Sentence: "The NBA season of 1975 -- 76 was the 30th season of the National Basketball Association ." Input AMR: (s / season~~2 xxxxx :ord (o / ordinal-entity~~10 xxxxxxxxxx :value 30~~9) xxxxx :poss (l / league~~13 xxxxxxxxxx :name (n / name13 xxxxxxxxxxxxxxx :op1 "National"~~13 xxxxxxxxxxxxxxx :op2 "Basketball"~~14 xxxxxxxxxxxxxxx :op3 "Association"~~15)) xxxxx :time (d3 / date-interval~~4 xxxxxxxxxx :op1 (d / date-entity~~4 xxxxxxxxxxxxxxx :year 1975~~4) xxxxxxxxxx :op2 (d2 / date-entity6 xxxxxxxxxxxxxxx :year 76~6))) Output: This refers to a season of the National Basketball Association. It was the 30th season of the league. The season took place during the 1975-76 time period.	Sentence: ‘‘Captain’’ was hulked in 1739 , and eventually broken up in 1762 . AMR: (a / and xxxxx :op1 (h / hulk-01 xxxxxxxxxx :ARG1 (s / ship xxxxxxxxxxxxxxx :name (n / name xxxxxxxxxxxxxxxxxx :op1 "Captain")) xxxxxxxxxx :time (d / date-entity xxxxxxxxxxxxxxx :year 1739)) xxxxx :op2 (b / break-up-08 xxxxxxxxxx :ARG1 s xxxxxxxxxx :time (e / eventual) xxxxxxxxxx :time (d2 / date-entity xxxxxxxxxxxxxxx :year 1762)))
Text + AMR nodes	Task: Deconstruct the following sentence into its underlying logical structure. Describe the meaning by identifying core events, the entities involved, and how they relate to one another. You may use the supplement provided if you find it helpful. Example: Input Sentence: "The NBA season of 1975 -- 76 was the 30th season of the National Basketball Association ." Supplement: season, ordinal-entity, 30, league, name, "National", "Basketball", "Association", date-interval, date-entity, 1975, date-entity, 76 Output: This refers to a season of the National Basketball Association. It was the 30th season of the league. The season took place during the 1975-76 time period.	Sentence: ‘‘Captain’’ was hulked in 1739 , and eventually broken up in 1762 . Supplement: and, hulk-01, ship, name, "Captain", date-entity, 1739, break-up-08, s, eventual, date-entity, 1762

Appendix

Table 18: Prompt types and their corresponding system prompt

Explore similar work

Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining
Jun 15, 2026Michael K. Chen, Xikun Zhang, Fan Bai +2Large Language Model Pretraining Data Augmentation
1UC San Diego · 2RMIT University · 3Bloomberg AI
MeMo: Memory as a Model
May 14, 2026Ryan Wei Heng Quek, Sanghyuk Lee, Alfred Wei Lun Leong +6Large Language Model Memory
Institute of Data Science, National University of Singapore, Singapore · Integrative Sciences and Engineering Programme, NUSGS, Singapore · Agency for Science, Technology, Research (A*STAR), Singapore +6
When Should LLMs Search? Counterfactual Supervision for Search Routing
Jul 7, 2026Minho KimLarge Language Model Routing
1Sangmyung University · 2DMTLABS.

Jun 15, 2026, cs.LGJ/K move · Enter open · S save

Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining

Michael K. Chen, Xikun Zhang, Fan Bai, Zhengding Hu+1

1UC San Diego · 2RMIT University · 3Bloomberg AI

Large Language Model Pretraining Data Augmentation Augmentation Epoch Overfitting

Open paper AskPDF arXiv

As AI labs approach a data ceiling where compute capacity outpaces the rate of new high-quality text generation, language model pretraining is shifting toward a data-constrained, compute-abundant regime that demands productive multi-epoch training on fixed corpora. Standard autoregressive (AR) pretraining overfits severely in this setting, reaching its optimum early and then continuously deteriorating. We investigate training-time data augmentation as a regularizer to mitigate this overfitting and enable productive training for hundreds of epochs on the same data. We introduce three orthogonal categories of augmentation for AR pretraining: token-level noise (masking, random replacement), sequence permutations (right-to-left prediction, Fill-in-the-Middle), and target offset prediction (

x_{t+i}

for

i > 1

). Through systematic ablations, we find that individual augmentations delay overfitting and lower validation loss relative to the baseline, with random token replacement achieving the best minimum loss among individual methods. Combining augmentation categories further lowers the minimum validation loss. Our experiments demonstrate that data augmentations mitigate AR pretraining's data inefficiency and offer a promising solution to the data-constrained regime~\footnote{All code and data are available at https://github.com/ michaelchen-lab/ data-augmentations-for-pretraining.

Abstract

Explore similar work

Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining

MeMo: Memory as a Model

When Should LLMs Search? Counterfactual Supervision for Search Routing