LLMs Struggle with Abstract Meaning Comprehension More Than Expected

Hamoud Alhazmi; Jiachen Jiang

cs.CLApr 13, 2026

LLMs Struggle with Abstract Meaning Comprehension More Than Expected

Authors: Hamoud Alhazmi, Jiachen Jiang

Organizations: Department of Computer Science and Engineering, The Ohio State University, Columbus, Ohio, USA

Abstract

Understanding abstract meanings is crucial for advanced language comprehension. Despite extensive research, abstract words remain challenging due to their non-concrete, high-level semantics. SemEval-2021 Task 4 (ReCAM) evaluates models' ability to interpret abstract concepts by presenting passages with questions and five abstract options in a cloze-style format. Key findings include: (1) Most large language models (LLMs), including GPT-4o, struggle with abstract meaning comprehension under zero-shot, one-shot, and few-shot settings, while fine-tuned models like BERT and RoBERTa perform better. (2) A proposed bidirectional attention classifier, inspired by human cognitive strategies, enhances fine-tuned models by dynamically attending to passages and options. This approach improves accuracy by 4.06 percent on Task 1 and 3.41 percent on Task 2, demonstrating its potential for abstract meaning comprehension.

Figures & tables

Figure 1: The concatenation of passage, question and five options.

Figure 2: An overview of the overall architecture of Bi-Directional Attention

Model	Zero-shot Accuracy
GPT-4o-Mini	65.83%
Gemma-2-9B	64.76%
GPT4-o	64.40%
Qwen2.5-7B	59.02%
Meta-Llama-3.1-8B	52.45%
Vicuna 1.5 7B	24.49%

Table 1: Zero-shot accuracy of different LLMs.

Model	Zero-Shot	One-Shot	Two-Shot
Gemma 2 -9B	64.76%	70.25%	73.60%
GPT-4o-Mini	65.83%	71.45%	72.28%

Table 2: Few-shot learning results for top-performing models.

Pretrained Model	Task1	Task2	Task3
RoBERTa-large	64.47%	70.47%	68.47%
ELECTRA-large	85.89%	88.00%	89.06%

Table 3: Accuracy of fine-tuned BERT-based models on three tasks.

Model	Task1
ELECTRA-large	85.89%
ELECTRA-large + Uni-Attn	87.08% (+1.19%)
ELECTRA-large + Bi-Attn	89.95% (+4.06%)

Table 4: Validation Accuracy For task1 using Uni-Attn and Bi-Attn.

Model	Task2
ELECTRA-large	88.00%
ELECTRA-large + Uni-Attn	89.29% (+1.29%)
ELECTRA-large + Bi-Attn	91.41% (+3.41%)

Table 5: Validation Accuracy For task2 using Uni-Attn and Bi-Attn.

Model	Task3
ELECTRA-large	89.06%
ELECTRA-large + Uni-Attn	89.18% (+0.12%)
ELECTRA-large + Bi-Attn	90.59% (+1.53%)

Table 6: Validation Accuracy For task3 using Uni-Attn and Bi-Attn.

Figure 3: Comparison of LLM-based model v.s. finetuned models on tasks1. The ELELCTRA with Bi-Directional Attention acheieves the best performance over all models.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: An overview of the overall architecture of Uni-Directional Attention

Name	Type	Params	Trainable
Electra	ElectraModel	334 M	No
Classifier	Linear	1.0 K	Yes

Appendix

Table 7: Network layers of Electra and parameter size

Name	Type	Params	Trainable
ELECTRA	ElectraModel	334 M	No
Uni-Attn	MultiheadAtt-	4.2 M	Yes
	entionLayer
Dropouts	ModuleList	0 M	None
Classifier	Linear	1.0 K	Yes

Appendix

Table 8: Network layers of Electra + Uni-Directional Attention and parameter size.

Name	Type	Params	Trainable
ELECTRA	ElectraModel	334 M	No
Bi-Attn	Bi-Directional Attention Layer	8.4 M	Yes
Dropouts	ModuleList	0 M	None
Classifier	Linear	1.0 K	Yes

Appendix

Table 9: Network layers of Electra + Bi-Directional Attention and parameter size.

Layer	Hyperparameter	Value
	token max length	256
Tokenizer	truncation	"only first"
	padding	’max length’
	learning rate	1e-4
	train batch size	2
	eval batch size	2

Appendix

Table 10: Hyperparameter Setting

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Models
Sep 22, 2026David Torres-Moreno, Jorge Hermosillo-ValadezKnowledge Augmentation for Language Models Natural Language Inference
The Grounding Gap: How LLMs Anchor the Meaning of Abstract Concepts Differently from Humans
May 9, 2026Odysseas S. Chlapanis, Orfeas Menis Mastromichalakis, Christos H. PapadimitriouLLM Grounding Cognitive Modeling
Department of Informatics Athens University of Economics and Business Archimedes, Athena Research Center Greece · Instituto de Telecomunicações Portugal · Department of Computer Science Columbia University United States
Measuring Intent Comprehension in LLMs
Jun 19, 2025Nadav Kunievsky, James A. EvansPrompt Sensitivity LLM Evaluation
Knowlesdge Lab, University of Chicago, Chicago, Illinois, USA · Knowledge Lab, Data Science Institute, Department of Sociology, University of Chicago, Chicago, Illinois, USA.

Sep 22, 2026J/K move · Enter open · S save

Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Models

David Torres-Moreno, Jorge Hermosillo-Valadez

Knowledge Augmentation for Language Models Natural Language Inference

Open paper AskPDF arXiv

Despite their outstanding performance on many NLP tasks, LLMs face serious challenges related to semantic abstraction. In this study, we are interested in understanding how LLMs leverage abstract semantic knowledge in natural language inference (NLI), which requires sophisticated linguistic capabilities to interpret implicit meanings, contextual conceptual relationships, and semantic connections between words and phrases. To this end, we propose a methodological framework for constructing new semantic knowledge at a higher level of abstraction, which we define under the notions of semantic compatibility and incompatibility for NLI. In this framework, the meaning of the lexical-semantic relations between the premise and the hypothesis is reconfigured to achieve a more flexible semantic network that induces different reasoning paths in LLMs. These new pathways show a consistent pattern of responses that allows agreement on a single response. The results demonstrate that our proposal allows to discover and compensate for LLMs' semantic knowledge gaps in NLI, achieving significant improvements in accuracy, exceeding 10% for some models, and in particular for the non-entailment class. It is essential to note that LLMs need structured knowledge and not just more data to bridge reasoning gaps. Our hybrid approach directs attention to overlooked word relationships, allowing models to synthesize missing information. We believe that the future lies not in increasing model size, but in creating a semantic scafolding that mimics the flexibility of human thinking. Hopefully, our proposal will enable the development of more robust agents and interpretable reasoning, guiding AI toward reliable language understanding.

Abstract

LLM Evaluation

Explore similar work

Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Models

The Grounding Gap: How LLMs Anchor the Meaning of Abstract Concepts Differently from Humans

Measuring Intent Comprehension in LLMs