Recent advances have significantly improved the performance of Abstract Meaning Representation (AMR) parsers, yet determining when their outputs can be trusted remains largely unexplored. This work presents the first study of uncertainty estimation for AMR parsers in a black-box setting using top-k beam-search candidates. Uncertainty is estimated from pairwise candidate similarities and evaluated using the Prediction Rejection Ratio. This work characterises the relationship between parsing performance and uncertainty, evaluates the reliability of beam candidate distributions, analyses the effect of evaluation metrics on uncertainty assessment, and identifies parsers that provide the strongest balance between performance and reliability. Experimental results show the value of uncertainty estimation as a complementary dimension for evaluating AMR parsers.
Figures & tables
Parser
Smatch
Smatch++
SPRING Bevilacqua et al. (2021)
0.8449
0.8438
MBSE Lee et al. (2022)
0.8374
0.8378
HMG-AMR-BART Sataer et al. (2024)
0.8358
0.8380
IBM-Parser Fernandez Astudillo et al. (2020)
0.8128
0.8181
IBM-Neural Drozdov et al. (2022)
0.8115
0.8174
HMG-AMR-T5 Sataer et al. (2024)
0.7693
0.7797
Table 1: Top-1 AMR parsing performance measured using Smatch and Smatch++.
Figure 1: Mean PRR for each parser and prediction quality metric, averaged across all beam sizes and uncertainty estimators.
Figure 2: Mean PRR averaged across uncertainty estimation methods and prediction quality metrics for beam sizes k∈{3,5,8,10} .
Figure 3: Each value shows the mean PRR averaged across uncertainty estimation methods and prediction quality metrics.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Ci=P(si)={gi(1),…,gi(k)}
Appendix
Algorithm 1 Black-box Uncertainty Estimation for AMR Parsing
Configuration
Runtime
Per parser (all beams)
∼ 89,000–90,000 s ( ≈ 25 h)
Per parser per beam
∼ 22,000 s ( ≈ 6 h)
Full experiment (6 parsers)
> 150 h wall-clock
Appendix
Table 2: Hardware allocation and observed execution time were measured from job logs.
Parser
Smatch
SPRING Bevilacqua et al. (2021)
0.8449
MBSE Lee et al. (2022)
0.8374
HMG-AMR-BART Sataer et al. (2024)
0.8358
IBM-Parser Fernandez Astudillo et al. (2020)
0.8128
IBM-Neural Drozdov et al. (2022)
0.8115
HMG-AMR-T5 Sataer et al. (2024)
0.7693
Appendix
Table 3: Top-1 AMR parsing performance measured using Smatch.
Figure 4: Relationship between parser accuracy (Smatch) and uncertainty quality (PRR) across beam sizes and evaluation metrics.
Figure 5: Mean PRR of uncertainty estimators across beam sizes aggregated over all parsers. Stable estimators exhibit less variation in performance across beam sizes.
Figure 6: Pairwise Spearman correlation between uncertainty estimators evaluated using Smatch++. Graph-based estimators exhibit strong agreement, while LexSim captures a distinct uncertainty signal.
SPRING
IBM -Parser
HMG-T5
HMG-BART
MBSE
IBM-Neural
UE Method
S++
SB
S++
SB
S++
SB
S++
SB
S++
SB
S++
SB
LexSim
0.2209
0.1695
0.2660
0.1615
-0.0395
-0.0904
0.0858
0.0395
0.2294
0.1332
0.2659
0.1685
EigV-Smatch
0.1956
0.1421
0.2218
0.1251
-0.0838
-0.1199
0.0756
0.0359
0.1871
0.0802
0.2325
0.1431
EigV-S2match
0.1912
0.1322
0.2319
0.1232
-0.0794
-0.1176
0.0751
0.0330
0.1931
0.0752
0.2448
0.1473
EigV-SEMA
0.2504
0.2254
0.1919
0.1302
0.0334
-0.0060
0.1080
0.0811
0.1666
0.0859
0.1945
0.1328
EigV-WLK
0.2469
0.2231
0.1807
0.1243
0.0348
-0.0002
0.0605
0.0512
0.1555
0.0811
0.0000
0.0000
Appendix
Table 4: PRR at beam size k=10 for all uncertainty estimation methods across parsers and quality metrics (Smatch++ (S++) and SemBLEU (SB)). Higher PRR indicates better alignment between uncertainty estimates and prediction quality. Green cells denote the best-performing uncertainty estimator within each parser–metric pair. Consistent with the main findings, SPRING achieves the strongest overall reliability, while DegMat-SEMA frequently emerges as the most effective graph-based uncertainty estimator at larger beam sizes.
Figure 7: Parser-level PRR as a function of beam size evaluated using SemBLEU. Each subplot corresponds to one parser and compares uncertainty estimators across beam sizes.
Figure 8: Pairwise Spearman correlation between uncertainty estimators evaluated using S 2 Match.
Figure 9: Parser-level PRR as a function of beam size evaluated using Smatch++. Each subplot corresponds to one parser and compares uncertainty estimators across beam sizes.
Figure 10: Parser-level PRR as a function of beam size evaluated using S 2 Match. Each subplot corresponds to one parser and compares uncertainty estimators across beam sizes.
Figure 11: Pairwise Spearman correlation between uncertainty estimators evaluated using SemBLEU.
While Abstract Meaning Representation (AMR) has historically improved performance on a range of NLP tasks, the benefit---or lack thereof---of AMR augmentation for modern LLMs is thus far unclear. In this paper, we attempt to reproduce recent work that reported substantial downstream gains from AMR augmentation, finding that these are likely due to specific choices in the experimental settings used: using a consistent and unified protocol for hyperparameter selection, we observe that text-only baselines consistently match or exceed the performance of AMR-augmented models. To investigate this null result, we introduce a perplexity-based probe measuring the degree to which AMR provides an LLM with supplemental relational knowledge not already available to the model. We find that AMR augmentation does not help LLMs improve their understanding of relational content in the sentence, indicating that augmenting these models with AMR offers no clear benefit on downstream tasks.
Hoa Quynh Nhung Nguyen, Jacopo Staiano, Michael Sullivan
A key challenge in reliable LLM deployment is recognizing when uncertainty reflects irreducible variability in the task rather than limitations in the model's knowledge. In language tasks, a central source of such aleatoric uncertainty is input ambiguity or underspecification, where multiple interpretations remain plausible. Existing decomposition methods estimate aleatoric uncertainty by generating multiple clarifications of the input, querying the model for an answer under each clarification, and comparing the resulting answers. We argue that answers are not necessary for identifying ambiguity: they are often redundant, add avoidable cost, and can mislead through epistemic leakage. We support this claim theoretically, and propose a clarification-only approach that estimates this ambiguity-induced component directly from the space of plausible interpretations, without answers to the clarified inputs. Using ambiguity detection as an operational evaluation across three benchmarks, this direct approach improves AUROC (63.34 vs. 60.85), reduces computational cost by 4-26x in output tokens and 2.2-3.5x in API calls, and yields estimates with substantially lower correlation with epistemic uncertainty. Overall, our results suggest that ambiguity-induced aleatoric uncertainty is better estimated from the interpretation space than from the response space.
Omer Nahum, Niv Nayman, Jonathan Fhima +4
Data and Decision Sciences, Technion, Israel · Amazon Web Services · Computer Vision Group, University of Bern, Switzerland
Several SLOG test categories explicitly involve directional distinctions (modifier position shifts, argument extraction positions), yet AM-Parser, the previous SOTA, uses an AM algebra whose operations do not encode direction. We redesign the symbolic backend around CCG directed types (deterministic CKY + single linear decoder, 30K learnable parameters). Under the same BERT-base encoder, the system achieves 75.9±6.4% LF exact match, surpassing AM-Parser (70.8±4.3%). Per SLOG's own category groupings, gains are highly directional: the CCG system outperforms AM-Parser on all 5 position-shift categories (+29.9pp), while AM-Parser outperforms on all 6 recursive-depth categories. Replacing the encoder with DeBERTa-v3-large yields 90.7±4.9%, with the largest encoder gains in recursive-depth categories, complementary to directionality's gains. Directional representations shift the bottleneck from the symbolic layer (AM-Parser's 0% category ceiling) to the neural layer, which improves with encoder upgrades.