Larry Caused the Car to Stop, But the Model Didn't Notice: Transformer Blindness to the M-Heuristic
Authors: Stefania Butnaru, Claudiu Creanga
Organizations: Faculty of Foreign Languages, University of Bucharest, Romania · Faculty of Mathematics and Computer Science, University of Bucharest, Romania · Interdisciplinary School of Doctoral Studies, University of Bucharest, Romania · HLT Research Center, University of Bucharest, Romania
Modern transformer models excel at capturing semantic relationships through sentence embeddings, yet their ability to perform pragmatic reasoning remains understudied. This paper investigates whether encoder-based transformers such as DeBERTa employ the M-Heuristic (the neo-Gricean principle that marked linguistic forms implicate marked meanings). We test this hypothesis by contrasting lexical causatives (e.g., Larry stopped the car'') with periphrastic causatives (e.g., Larry caused the car to stop'') using a Natural Language Inference framework. Our experiments across 188 conditions with 15 ambitransitive verbs reveal that DeBERTa, RoBERTa, and BART show no evidence of capturing the pragmatic distinction between these forms, with DeBERTa predicting ``Neutral'' for 100% of cases. Probing analysis initially suggested a representation-use dissociation, but control experiments reveal the probe was tracking syntactic complexity, not causative pragmatics. Semantic similarity over 30 triplets places periphrastic causatives closer to unmediated manner descriptions in 29/30 cases, opposite to M-Heuristic predictions in the embedding space. Under explicit metalinguistic framing, Gemini Flash-Lite reaches 100% with item-specific traces, so the principle is available under instruction yet unused in default NLI.
Figures & tables
Figure 1: One fixed NLI instance from our tests (M-Heuristic gold: Entailment ). DeBERTa, RoBERTa, BART, and Gemini under direct NLI all predict Neutral on this item. Across all 188 items: chain-of-thought recovers 21.8%; metalinguistic prompting reaches 100%.
Figure 2
Figure 3: Human description preferences from Wolff (2003) . For unmediated events, participants chose lexical causatives ( ∼ 58%) over periphrastic ( ∼ 38%); for mediated events, the pattern reversed sharply, with periphrastic descriptions selected ∼ 85% of the time and lexical only ∼ 5%. This double dissociation is the human baseline our NLI models fail to replicate.
Cond.
Premise Type
Hypothesis Type
Expected Label
1
Lexical causative
Normal interpretation
Entailment
2
Lexical causative
Unusual interpretation
Contradiction
3
Periphrastic causative
Normal interpretation
Contradiction
4
Periphrastic causative
Unusual interpretation
Entailment
Table 1: Experimental Conditions and Expected Human Judgments
Premise Type
Premise
Hypothesis
Expected
Lexical
The chef melted the butter.
The chef did this in a normal, direct way.
Entailment
Periphrastic
The chef caused the butter to melt.
The chef did this in an unusual or indirect way.
Entailment
Table 2: Sample Experimental Items
Model
Correct / Total
Accuracy (%)
DeBERTa-v3-large
0 / 188
0.0
RoBERTa-large
5 / 188
2.7
BART-large
0 / 188
0.0
Table 3: Overall Accuracy on M-Heuristic Inference Task
Expected ↓ / Predicted →
Entailment
Neutral
Contradiction
Entailment
0
94
0
Contradiction
0
94
0
Table 4: Confusion Matrix: DeBERTa-v3-large
Prompt condition
Correct / Total
Accuracy (%)
Direct NLI
0 / 188
0.0
Chain-of-thought
41 / 188
21.8
Metalinguistic (M-Heuristic)
188 / 188
100.0
Table 5: Gemini Flash-Lite accuracy on pragmatic gold labels by prompt condition ( n=188 )
Layer Range
Accuracy
Initial Interpretation
Layer 0 (embedding)
50.0%
Chance level
Layers 1–3
80–99%
Distinction emerges
Layers 4–11
100.0%
Perfect separation
Layers 12–24
94–99%
Slight degradation
Table 6: Linear Probe Accuracy: Lexical vs. Periphrastic Causatives
Layer
Causative Probe
Control Probe
Layer 4
100.0%
87.5%
Layer 8
100.0%
87.5%
Layer 12
97.8%
87.5%
Layer 16
96.7%
97.5%
Layer 20
98.9%
97.5%
Table 7: Probe Accuracy on Syntactic Controls by Layer
Figure 4: Probe accuracy on syntactic controls by layer. Probes achieve 87.5–97.5% accuracy on non-causative periphrastic constructions, matching causative probe accuracy and indicating that the original 100% result tracks syntactic complexity rather than causative pragmatics.
Figure 5: Semantic similarity results. Periphrastic causative anchors cluster with unmediated (direct) descriptions in 29/30 cases, showing that DeBERTa’s representations encode the pragmatically wrong grouping relative to M-Heuristic predictions.
Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.
Discourse relations provide document structure, critical to language understanding and enabling language model performance and ethicality. In this work, we investigate how instruction-tuned Transformer models (LLaMA and Mistral) encode discourse relations in English, with a particular focus on the contrasting relations of causation and antithesis. Framing the task as a next-token prediction task and applying a suite of interpretability techniques to test model internals, our findings show that certain early layers make predictive decisions at mid-sequence tokens, while some mid-level layers finalize their decisions closer to the last token. Most of the remaining layers primarily propagate earlier decisions rather than actively influencing them. Additionally, we observe that some layers exhibit a preference for one answer over alternatives, suggesting asymmetric representation of discourse-based reasoning.\footnote{Our code is available at https://github.com/abhidipbhattacharyya/causation_vs_antithesis}
Abhidip Bhattacharyya, Shira Wein
University of Massachusetts Amherst · University of South Florida
Standard fine-tuning of transformer models on causal reasoning tasks leads to catastrophic model collapse, where models learn trivial solutions such as always predicting "Yes" or "No" regardless of input structure. We demonstrate that fine-tuning Gemma 270M on transitivity and d-separation tasks without semantic loss results in 100% collapse rate, with models achieving misleadingly high accuracy (73.9%) while learning no causal reasoning. We propose a semantic loss function with graph-based logical constraints and dynamic lambda scheduling that prevents this collapse. Our approach achieves 70.4% accuracy on transitivity tasks and 68.6% on d-separation tasks with stable, context-dependent predictions, representing a 42.7% improvement over collapsed baselines. Adversarial evaluation on 1,000 structural reasoning samples shows semantic models achieve 67-70% accuracy while collapsed models fail catastrophically at 43-71%. We validate our findings through comprehensive benchmarking on 200,000+ evaluation samples across five model variants, demonstrating that semantic loss is essential and not optional, for stable causal reasoning in transformers.
Pratik Deshmukh, Atirek Gupta
Technical University of Vienna, Austria · HCLTech, Noida, India
Predicting a label correctly does not necessarily require representing the operation that produces it. Transformer representations are known to carry label-level information, but whether they encode semantic operations producing those labels is unclear. We investigate this in Natural Language Inference using controlled premise-hypothesis pairs that differ by a single semantic transformation. Using layer-wise activations, we estimate operation-level subspaces via SVD and test their causal relevance through activation steering in four open-weight decoder models. Transformation effects are decodable with 84.8-99% accuracy and occupy partially distinct but overlapping subspaces, exceeding random-subspace baselines. Steering experiments show that these directions causally influence predictions, though steerability varies across models; cross-operation steering further reveals structured interference and a dissociation between subspace selectivity and cross-operation independence. These findings indicate that the models encode not only that a hypothesis relates to a premise but also, in part, how it does so, implying that mechanistic analysis and control should operate at the level of semantic operations rather than predicted labels alone.
Nura Aljaafari, Marco Valentino, André Freitas
University of Manchester, United Kingdom · University of Sheffield, United Kingdom · Idiap Research Institute, Switzerland +1