How Reliable are AMR Parser Outputs? Uncertainty Estimation for AMR Parsing

Anuj Kumar; Shintaro Ozaki; Yamuna Prasad; Katsuhiko Hayashi; Hidetaka Kamigaito; Taro Watanabe

cs.CLOct 9, 2026

How Reliable are AMR Parser Outputs? Uncertainty Estimation for AMR Parsing

Authors: Anuj Kumar, Shintaro Ozaki, Yamuna Prasad, Katsuhiko Hayashi, Hidetaka Kamigaito, Taro Watanabe

Organizations: Indian Institute of Technology Jammu, India · Nara Institute of Science and Technology (NAIST), Japan · The University of Tokyo, Japan

Abstract

Recent advances have significantly improved the performance of Abstract Meaning Representation (AMR) parsers, yet determining when their outputs can be trusted remains largely unexplored. This work presents the first study of uncertainty estimation for AMR parsers in a black-box setting using top-k beam-search candidates. Uncertainty is estimated from pairwise candidate similarities and evaluated using the Prediction Rejection Ratio. This work characterises the relationship between parsing performance and uncertainty, evaluates the reliability of beam candidate distributions, analyses the effect of evaluation metrics on uncertainty assessment, and identifies parsers that provide the strongest balance between performance and reliability. Experimental results show the value of uncertainty estimation as a complementary dimension for evaluating AMR parsers.

Figures & tables

Parser	Smatch	Smatch++
SPRING Bevilacqua et al. (2021)	0.8449	0.8438
MBSE Lee et al. (2022)	0.8374	0.8378
HMG-AMR-BART Sataer et al. (2024)	0.8358	0.8380
IBM-Parser Fernandez Astudillo et al. (2020)	0.8128	0.8181
IBM-Neural Drozdov et al. (2022)	0.8115	0.8174
HMG-AMR-T5 Sataer et al. (2024)	0.7693	0.7797

Table 1: Top-1 AMR parsing performance measured using Smatch and Smatch++.

Figure 1: Mean PRR for each parser and prediction quality metric, averaged across all beam sizes and uncertainty estimators.

Figure 2: Mean PRR averaged across uncertainty estimation methods and prediction quality metrics for beam sizes

k\in\{3,5,8,10\}

Figure 3: Each value shows the mean PRR averaged across uncertainty estimation methods and prediction quality metrics.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

\mathcal{C}_{i}=P(s_{i})=\{g_{i}^{(1)},\ldots,g_{i}^{(k)}\}

Appendix

Algorithm 1 Black-box Uncertainty Estimation for AMR Parsing

Configuration	Runtime
Per parser (all beams)	$\sim$ 89,000–90,000 s ( $\approx$ 25 h)
Per parser per beam	$\sim$ 22,000 s ( $\approx$ 6 h)
Full experiment (6 parsers)	$>$ 150 h wall-clock

Appendix

Table 2: Hardware allocation and observed execution time were measured from job logs.

Parser	Smatch
SPRING Bevilacqua et al. (2021)	0.8449
MBSE Lee et al. (2022)	0.8374
HMG-AMR-BART Sataer et al. (2024)	0.8358
IBM-Parser Fernandez Astudillo et al. (2020)	0.8128
IBM-Neural Drozdov et al. (2022)	0.8115
HMG-AMR-T5 Sataer et al. (2024)	0.7693

Appendix

Table 3: Top-1 AMR parsing performance measured using Smatch.

Figure 4: Relationship between parser accuracy (Smatch) and uncertainty quality (PRR) across beam sizes and evaluation metrics.

Figure 5: Mean PRR of uncertainty estimators across beam sizes aggregated over all parsers. Stable estimators exhibit less variation in performance across beam sizes.

Figure 6: Pairwise Spearman correlation between uncertainty estimators evaluated using Smatch++. Graph-based estimators exhibit strong agreement, while LexSim captures a distinct uncertainty signal.

	SPRING		IBM -Parser		HMG-T5		HMG-BART		MBSE		IBM-Neural
UE Method	S++	SB	S++	SB	S++	SB	S++	SB	S++	SB	S++	SB
LexSim	0.2209	0.1695	0.2660	0.1615	-0.0395	-0.0904	0.0858	0.0395	0.2294	0.1332	0.2659	0.1685
EigV-Smatch	0.1956	0.1421	0.2218	0.1251	-0.0838	-0.1199	0.0756	0.0359	0.1871	0.0802	0.2325	0.1431
EigV-S2match	0.1912	0.1322	0.2319	0.1232	-0.0794	-0.1176	0.0751	0.0330	0.1931	0.0752	0.2448	0.1473
EigV-SEMA	0.2504	0.2254	0.1919	0.1302	0.0334	-0.0060	0.1080	0.0811	0.1666	0.0859	0.1945	0.1328
EigV-WLK	0.2469	0.2231	0.1807	0.1243	0.0348	-0.0002	0.0605	0.0512	0.1555	0.0811	0.0000	0.0000

Appendix

Table 4: PRR at beam size

k=10

for all uncertainty estimation methods across parsers and quality metrics (Smatch++ (S++) and SemBLEU (SB)). Higher PRR indicates better alignment between uncertainty estimates and prediction quality. Green cells denote the best-performing uncertainty estimator within each parser–metric pair. Consistent with the main findings, SPRING achieves the strongest overall reliability, while DegMat-SEMA frequently emerges as the most effective graph-based uncertainty estimator at larger beam sizes.

Figure 7: Parser-level PRR as a function of beam size evaluated using SemBLEU. Each subplot corresponds to one parser and compares uncertainty estimators across beam sizes.

Figure 8: Pairwise Spearman correlation between uncertainty estimators evaluated using S 2 Match.

Figure 9: Parser-level PRR as a function of beam size evaluated using Smatch++. Each subplot corresponds to one parser and compares uncertainty estimators across beam sizes.

Figure 10: Parser-level PRR as a function of beam size evaluated using S 2 Match. Each subplot corresponds to one parser and compares uncertainty estimators across beam sizes.

Figure 11: Pairwise Spearman correlation between uncertainty estimators evaluated using SemBLEU.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 652 papers

On the (In)effectiveness of AMR Augmentation for Large Language Models
Sep 30, 2026Hoa Quynh Nhung Nguyen, Jacopo Staiano, Michael SullivanLLM Evaluation Knowledge Augmentation for Language Models
Saarland University · University of Trento
From Answers to Interpretations: Rethinking Ambiguity-Induced Aleatoric Uncertainty Estimation in LLMs
Sep 3, 2026Omer Nahum, Niv Nayman, Jonathan Fhima +4LLM Uncertainty Estimation Aleatoric Uncertainty
Data and Decision Sciences, Technion, Israel · Amazon Web Services · Computer Vision Group, University of Bern, Switzerland
On the Role of Directionality in Structural Generalization
Jul 2, 2026Zichao WeiSemantic Parsing Compositional Generalization
Saarland University

Sep 30, 2026J/K move · Enter open · S save

On the (In)effectiveness of AMR Augmentation for Large Language Models

Hoa Quynh Nhung Nguyen, Jacopo Staiano, Michael Sullivan

Saarland University · University of Trento

LLM Evaluation Knowledge Augmentation for Language Models

Open paper AskPDF arXiv

While Abstract Meaning Representation (AMR) has historically improved performance on a range of NLP tasks, the benefit---or lack thereof---of AMR augmentation for modern LLMs is thus far unclear. In this paper, we attempt to reproduce recent work that reported substantial downstream gains from AMR augmentation, finding that these are likely due to specific choices in the experimental settings used: using a consistent and unified protocol for hyperparameter selection, we observe that text-only baselines consistently match or exceed the performance of AMR-augmented models. To investigate this null result, we introduce a perplexity-based probe measuring the degree to which AMR provides an LLM with supplemental relational knowledge not already available to the model. We find that AMR augmentation does not help LLMs improve their understanding of relational content in the sentence, indicating that augmenting these models with AMR offers no clear benefit on downstream tasks.

Abstract

Uncertainty Quantification

Explore similar work

On the (In)effectiveness of AMR Augmentation for Large Language Models

From Answers to Interpretations: Rethinking Ambiguity-Induced Aleatoric Uncertainty Estimation in LLMs

On the Role of Directionality in Structural Generalization