The imperfective paradox provides a useful test of compositional semantic analysis. Recent work constructs an NLI benchmark and reports that models frequently infer completed telic events from progressive descriptions, attributing this behavior to a Teleological Bias. It further argues that prompting interventions cause a Calibration Crisis. We reexamine the benchmark and conclusions and show that it is substantially affected by conceptual and evaluation mis-specifications. We identify three conceptual mis-specifications. In particular, Aspectual Reduction affects the benchmark construction, analysis, experiments, and conclusions. Under a strict NLI standard, 76% of Group A instances do not explicitly rule out culmination. In our native-speaker annotation, 38% of Group A examples and 29% of the Group C examples were judged to permit an alternative interpretation. To control these issues and lexical variation, we construct Lexically Matched Minimal Pairs. At the evaluation level, we formulate event-semantic NLI as a Multi-step Reasoning Problem and assess both intermediate semantic decisions and final predictions. Our results show that models often do not affirm culmination but nevertheless accept the corresponding simple-past hypothesis, a pattern we characterize as Sufficiency Bias. We further show that prompting interventions produce a Decision Shift among labels without reliably improving the underlying semantic understanding and reasoning. Intermediate and oracle-guided analyses identify two additional failure modes: errors in compositional aspectual classification and Surface-form Attraction toward surface-associated answers. Our experiments on Qwen-7B with suitable prompts, GPT-5.4, and Qwen-72B provide initial evidence for the context sensitivity of aspectual classification and suggest that these models can achieve performance comparable to that of human annotators.
Figures & tables
Figure 1: Accuracy on Groups A and C across model sizes ( Ma and Miyao, 2026 ) .
Step
Reasoning Component
Description
Example
Step 1
Predicate Identification
Extract the complete event predicate headed by the relevant progressive verb.
Extract build a house , not only build .
Step 2
Aspectual Classification
Determine whether the complete predicate is telic or atelic .
Build a house is telic.
Step 3
Process Verification
Determine whether the premise entails that the event process actually occurred.
Some house-building occurred.
Step 4
Endpoint Identification
For a telic predicate, identify the linguistically encoded culmination or endpoint. For an atelic predicate, indicate that no inherent culmination is introduced by the predicate.
Endpoint: completion of the house.
Step 5
Culmination Verification
Determine whether the premise explicitly entails that the identified endpoint was reached. The existence of an ongoing process alone is insufficient to establish culmination.
Completion is not entailed.
Step 6
Non-Culmination Verification
Determine whether the premise explicitly entails that the endpoint was not reached.
Stopped and never resumed entails non-completion.
Table 1: Multi-step reasoning procedure for event-semantic NLI.
Model
Prompt
Completion
Group-wise Results
A
B
C
D
Acc.
NT
Acc.
NC Rate
Acc.
NC Rate
Acc.
NC Rate
Acc.
NC Rate
Llama-3.1-8B-Instruct
No reasoning
0
100
0
100
0
100
90
90
100
100
Reasoning
85
95
70
100
60
90
90
90
80
80
Qwen2.5-7B-Instruct
No reasoning
45
100
70
90
0
50
75
70
10
15
Reasoning
100
100
90
100
45
100
100
100
100
100
Table 2: Results on the completion and group-wise classification experiments. For the completion experiment, Acc. denotes accuracy and NT denotes the proportion of predictions that are not classified as True . For each group, NC Rate denotes the proportion of predictions not assigned to class C. All values are percentages.
Model
Int.
Occ.
Intr.
NC
Comp.
Llama-3.1-8B
100
0
0
0
100
Qwen2.5-7B
100
20
45
0
100
GLM-4-9B
100
15
0
0
100
DeepSeek-R1-7B
60
10
60
0
100
Table 3: Results on the occurrence experiment. Int., Occ., Intr., NC, and Comp. denote intention, occurrence, interruption, non-completion, and completion, respectively. The reported result is accuracy. The ground-truth label is True only in the Completion set.
Reasoning Step
Zero-shot
DAP
CoT
Counterfactual
Predicate Identification
×
×
×
×
Aspectual Classification
×
×
Unclear
×
Process Verification
×
✓
×
×
Endpoint Identification
×
×
Misleading
×
Culmination Verification
×
✓
✓
×
Non-Culmination Verification
×
×
×
✓
Table 4: Coverage of the reasoning steps under different prompting methods. A checkmark indicates that the step is explicitly elicitepd, while a cross indicates that it is not explicitly modeled.
Model
Method
Original
VSS
Minimal Pairs
A
B
C
D
A-F
A-U
A
B
C
D
A-F
A-U
A
B
C
D
A-F
A-U
Llama-3.1-8B-Instruct
Zero-shot
44
93
7
99
17
27
59
92
7
99
29
31
78
94
0
100
66
12
DAP
54
96
74
82
15
39
76
95
83
85
27
49
74
96
46
50
60
14
CoT
94
34
56
59
6
88
100
37
67
58
10
90
100
4
66
42
36
64
Counterfactual
99
1
95
1
44
55
98
0
95
0
63
35
100
0
90
0
68
32
DAPCoT
78
73
76
66
38
40
86
73
74
72
49
37
88
36
52
46
60
28
Table 5: Performance across the original, VSS, and minimal pairs. A-F and A-U denote the percentages of False and Unknown predictions for Group A, respectively, while A denotes their sum, i.e., the proportion of non- True predictions, computed before rounding. All values are percentages rounded to integers.
Model
Original
VSS
Minimal Pairs
All
Tel.
Proc.
Comp.
Int.
Rel.
Final
Code
All
Tel.
Proc.
Comp.
Int.
Rel.
Final
Code
All
Tel.
Proc.
Comp.
Int.
Rel.
Final
Code
Llama-3.1-8B
50
63
98
86
83
48
48
61
52
64
100
85
82
48
45
57
46
61
96
86
81
54
49
61
GLM-4-9B
48
56
100
99
88
64
64
56
56
72
100
99
84
62
62
68
59
70
100
99
86
66
66
69
Qwen2.5-7B
69
81
100
93
91
55
54
81
69
86
99
99
84
50
50
84
73
87
100
92
91
54
53
88
DeepSeek-R1-7B
43
67
79
97
81
44
44
60
41
61
83
94
79
56
54
54
50
77
82
96
83
52
52
65
Table 6: Accuracy of intermediate semantic decisions and final NLI predictions across the three datasets. All denotes the proportion of outputs with intermediate labels all correct. Tel., Proc., Comp., Int., Rel., Final, and Code denote telicity, process occurrence, completion, interruption, predicted NLI relation, model-final prediction, and code-mapped prediction, respectively. All values are integer percentages.
Model
Method
Original
VSS
Minimal Pairs
A
B
C
D
A
B
C
D
A
B
C
D
Llama-3.1-8B-Instruct
Zero-shot
95
52
96
55
96
46
97
53
96
22
96
18
DAP
100
18
100
16
100
16
100
15
96
4
98
4
CoT
99
28
100
22
100
29
100
29
100
6
100
10
Counterfactual
100
43
99
48
100
41
97
43
100
24
100
26
Qwen2.5-7B-Instruct
Zero-shot
91
76
91
76
96
77
93
77
92
58
92
56
Table 7: Classification accuracy (%) for Groups A–D across three datasets.
Model
Prompt
No Reasoning
Reasoning
A
B
C
D
Avg.
A
B
C
D
Avg.
Llama-3.1-8B-Instruct
S → S
100
30
85
20
59
100
100
90
100
98
O → S
0
15
100
55
43
100
80
100
70
88
S → O
100
0
100
0
50
90
100
100
100
98
O → O
100
50
100
35
71
100
100
100
100
100
Qwen2.5-7B-Instruct
S → S
100
100
90
100
98
100
100
100
100
100
Table 8: Oracle-guided performance under different prompt and reasoning settings. S and O denote semantic and opaque representations, respectively.
Model
Dataset
Setting
A
B
C
D
A-F
A-U
Qwen2.5-72B
Original
No reasoning
91
91
77
99
40
51
Reasoning
82
95
97
78
29
53
One-shot reasoning
86
94
98
90
31
55
VSS
No reasoning
98
92
76
98
59
39
Reasoning
92
94
97
74
51
41
One-shot reasoning
94
95
100
88
47
47
Table 9: Performance of large language models across the original, VSS, and lexically matched minimal-pair datasets. A-F and A-U denote the percentages of False and Unknown predictions for Group A, respectively, while A denotes their sum. All values are percentages rounded to integers.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
A
B
C
D
At least two annotators support an alternative label
38%
2%
29%
1%
Majority vote reproduces original gold
71%
98%
71%
99%
Appendix
Table 10: Human annotation results by benchmark group. The first row reports the proportion of examples for which at least two of three annotators considered a label other than the original gold label acceptable. The second row reports the proportion for which majority voting over annotators’ preferred interpretations reproduces the original gold label.
Prompt-based LLMs are increasingly used for stance detection, but harder examples are not always repaired by clearer instructions, reasoning prompts, retrieval, or debate. We introduce SICI (Stance Inference Complexity Index), a seven-dimensional diagnostic measure of the semantic-pragmatic burden imposed by a target--text pair. Across SemEval-2016 and VAST, SICI predicts LLM accuracy better than surface proxies and shows substantial cross-scorer reliability (α=0.771). More importantly, LLM errors change regime as SICI increases: low-complexity examples invite over-attribution, especially Against predictions; intermediate examples form an unstable boundary; and high-complexity examples rapidly concentrate on None. This phase-transition-like structure persists across GPT-3.5, GPT-4o-mini, DeepSeek-V3, and GPT-4o, although stronger models move the boundaries. A 15-method intervention study further shows that prompting, retrieval, and debate often shift models along the attribution--abstention axis rather than removing the high-complexity bottleneck.
Fuqiang Niu, Bowen Zhang
School of Cyber Science and Technology, University of Science and Technology of China, Hefei, China · School of Artificial Intelligence, Shenzhen Technology University, Shenzhen, China
Do neural models, such as Large Language Models, genuinely acquire compositional abilities for interpretation of natural language? When we talk about semantic interpretation, we can distinguish two complementary aspects: establishing what an expression refers to in the world (which we call the Extensional task) and representing its sense in a structured way (which we call the Intensional task). We evaluate LLMs and humans on both tasks in the setting of the Personal Relation Task (Paperno 2022) in which, given a universe of people and their relationships with each other, one is asked to interpret a noun phrase such as "Amber's parent's friend". Here, for the Intensional task, the answer is the formula "friend(parent(amber))", and for the Extensional task, the person. We find that humans and LLMs show opposite strengths: humans perform better on Extensional than Intensional tasks, and LLMs vice versa. Our methodology brings greater nuance to the understanding of compositional abilities in modern machine learning models. Our results support the notion that the lack of referential grounding in LLM training is a crucial missing component in mimicking human-like language understanding.
Benchmarks are often used as a standard to understand LLM capabilities in different domains. However, aggregate benchmark scores provide limited insight into compositional skill gaps of LLMs and how to improve them. To make these weaknesses visible, we propose Scaffolded Task Design (STaD) framework. STaD generates controlled variations of benchmark tasks based on the concept of scaffolding, which introduces structured, incremental support in a step-by-step manner. Rather than inspecting failures individually, this approach enables systematic and scalable probing of model behavior by identifying the specific reasoning skill compositions they lack. Treating the LLM as a black box, our experiments on six models of varying sizes reveal multiple failure points in three reasoning benchmarks and highlight each model's unique and distinct skill gaps.
Sungeun An, Swanand Ravindra Kadhe, Shailja Thakur +2