To fully capture the meaning of a sentence, semantic representations should encode aspect, which describes the internal temporal structure of events. In graph-based meaning representation frameworks such as Uniform Meaning Representations (UMR), aspect lets one know how events unfold over time, including distinctions such as states, activities, and completed events. Despite its importance, aspect remains sparsely annotated across semantic meaning representation frameworks. This has, in turn, hindered not only current manual annotation, but also the development of automatic systems capable of predicting aspectual information. In this paper, we introduce a new dataset of English sentences annotated with UMR aspect labels over Abstract Meaning Representation (AMR) graphs that lack the feature. We describe the annotation scheme and guidelines used to label eventive predicates according to the UMR aspect lattice, as well as the annotation pipeline used to ensure consistency and quality across annotators through a multi-step adjudication process. To demonstrate the utility of our dataset for future automation, we present baseline experiments using three modeling approaches. Our results establish initial benchmarks for automatic UMR aspect prediction and provide a foundation for integrating aspect into semantic meaning representations more broadly.
Figures & tables
Figure 2: UMR aspect lattice with aspectual values utilized in our English annotation highlighted in yellow.
Per-class
Label
Kα∗
Count
None
0.735
514
State
0.706
385
Performance
0.716
360
Process
0.547
100
Habitual
0.704
56
Table 1: Aspect label distribution and inter-annotator agreement metrics. *Per-class Krippendorff’s alpha computed as one-vs-rest. † Fleiss’ κ restricted to the 333 events that received first-pass tie-breaks.
Sentence
Event
Main label
2nd label
(a) <Architect> now above that place red blocks on the grid.
Place
Performance
Process
(b) "Clap your hands, one against the other," the conceited man now directed him.
Clap
Endeavor
Performance
(c) Greed is when you are wealthy and lobby your representatives for special tax breaks because you are over 60 years of age.
Age
State
N/A
Table 2: Selected examples with ambiguous aspect; some annotated with secondary labels.
Type
Model
Acc.
Macro F1
Wtd. F1
Precision
Recall
LLM
LLaMA-3.1-8B-Instruct (zero-shot)
0.31
0.19
0.27
0.29
0.24
LLaMA-3.1-8B-Instruct (3-shot)
0.25
0.16
0.22
0.32
0.21
GPT-5mini (zero-shot)
0.56
0.49
0.56
0.69
0.49
GPT-5mini (3-shot)
0.56
0.46
0.60
0.49
0.46
Neural
Feedforward MLP
0.45
0.27
0.44
0.29
0.32
Symbolic
AutoAspect †
0.39
0.23
0.40
—
—
Table 3: Baseline results on the test split (254 events, 72 sentences). Human performance reflects first-pass annotator accuracy against adjudicated gold labels. Precision and Recall are macro averages across classes. † AutoAspect results are taken from the original publication and are not directly comparable.
Aspect
Little Prince
Minecraft
BOLT DF
WB
Pear Story
Lorelei
UMR 1.0
Total
State
63
20
119
45
121
0
62
430
Habitual
1
0
17
4
28
0
2
52
Process
2
0
5
5
44
1
1
58
Activity
9
0
31
14
57
0
21
132
Performance
35
2
43
18
159
3
57
317
Endeavor
0
0
0
0
2
0
14
16
Table 4: Aspect label distribution from existing UMR data before any additional annotation was done.
Aspect
Little Prince
Minecraft
BOLT DF
WB
Existing Labels
Total
State
172
14
101
14
430
731
Habitual
41
0
4
0
52
97
Process
31
0
38
8
58
135
Activity
15
2
10
3
132
162
Performance
163
43
69
32
317
624
Endeavor
15
0
4
0
16
35
Table 5: Label distribution by corpus and annotated aspect. We report Fleiss’ Kappa between the two initial annotators and do not include disagreements in the reported total.
Category
Metric
Value
Accuracy
State
0.82
Habitual
0.63
Activity
0.52
Performance
0.80
Endeavor
0.11
Overall Accuracy
0.74
Table 6: Practice Annotation Results: Overall Accuracy is the ratio of the total number of correct annotations over the total number of predicates annotated. Perfect Accuracy is the ratio of predicates that were correctly annotated by all annotators. No occurrences of Process aspects were present in the practice set.
Split
Sentences
Events
Train
333
,0 999
Dev
71
,0 220
Test
72
,0 254
Total
476
1,473
Table 7: Stratified 70/15/15 train/dev/test split, divided at the sentence level using dominant aspect label for stratification.
Gold label
Error rate
Error Label
Activity
6.7%
Performance
Endeavor
12.2%
Performance
Habitual
6.8%
Performance
None
6.0%
State
Performance
3.9%
None
Process
9.1%
None
Table 8: Final gold labels and the percent for each that was incorrectly annotated then fixed during adjudication out of the total number of annotations for that label. The error label is the most common incorrect annotation.
Figure 3: Flow diagram illustrating the annotation and adjudication phases, and how many labels were completed at each stage of the process.