Physical commonsense reasoning represents a fundamental capability of human intelligence, enabling individuals to understand their environment, predict future events, and navigate physical spaces. Recent years have witnessed growing interest in reasoning tasks within Natural Language Processing (NLP). However, no prior research has examined the performance of Large Language Models (LLMs) on non-question-answering (non-QA) physical commonsense reasoning tasks in low-resource languages such as Basque. Taking the Italian GITA as a starting point, this paper addresses this gap by presenting BasPhyCo, the first non-QA physical commonsense reasoning dataset for Basque, available in both standard and dialectal variants. We evaluate model performance across three hierarchical levels of commonsense understanding: (1) distinguishing between plausible and implausible narratives (accuracy), (2) identifying the conflicting element that renders a narrative implausible (consistency), and (3) determining the specific physical state that creates the implausibility (verifiability). These tasks were assessed using multiple multilingual LLMs as well as models pretrained specifically for Italian and Basque. Results indicate that, in terms of verifiability, LLMs exhibit limited physical commonsense capabilities in low-resource languages such as Basque, especially when processing dialectal variants.
Figures & tables
Type
Sentence 1
Sentence 2
Sentence 3
Sentence 4
Sentence 5
Plausible
George filled the glass with water.
George put the glass in the microwave.
George turned on the microwave.
The water got hot.
George put a tea bag in the water.
Implausible (order)
George put the glass in the microwave.
George filled the glass with water.
George turned on the microwave.
The water got hot.
George put a tea bag in the water.
Implausible (cloze)
George filled the glass with water.
George put the glass in the microwave.
George turned on the microwave.
The water got cold.
George put a tea bag in the water
Table 1: Example of a story with its plausible and implausible versions.
Standard
Dialectal
Jonek lorategi handi bat dauka. Elur lorategian dago. Jonek lorategiko atea ireki du. Elurrek alde egin du. Lorategia hutsik dago.
Jonek lorategi handi bat dauko . Elur lorategixen dago. Jonek lorategiko atie zabaldu dau . Elurrek ospa egin dau. Lorategixe hutsik dago.
Table 2: Example of a story adapted from Standard Basque to Western Basque.
Figure 1: The number of unique words in BasPhyCo (left) and BasPhyCo west (right), as well as the overlap of both datasets (middle). Additionally, some examples from each dataset.
Test data
Model
Accuracy
Consistency
Verifiability
GITA
Llama-3.1-8B-It
72.47
29.29
14.23
Llama-3.1-70B-It
87.64
65.27
36.40
Gemma-2-9B-It
71.91
35.98
16.74
Gemma-2-27B-It
67.42
30.96
15.06
Latxa-3.1-8B-It
67.42
23.85
13.39
Latxa-3.1-70B-It
85.96
60.25
40.17
Table 3: Overall results for Story Classification, Conflict Detection and Physical State Classification, measured by accuracy, consistency and verifiability, respectively. GITA: original Italian data; BasPhyCo: manually translated Standard Basque data; BasPhyCo west : automatically adapted data into the Western dialect.
Language
Model
Accuracy
Consistency
Verifiability
Order
Cloze
Plausible
Order
Cloze
Order
Cloze
GITA
Llama-3.1-8B-It
64.75
81.20
71.79
14.75
44.44
4.92
23.93
Llama-3.1-70B-It
88.52
95.73
78.63
55.74
74.36
25.41
47.01
Gemma-2-9B-It
49.18
81.20
86.32
15.57
57.26
2.46
31.62
Gemma-2-27B-It
34.43
78.63
90.60
9.02
53.85
4.10
26.50
Latxa-3.1-8B-It
55.74
68.38
78.63
13.11
35.04
6.56
20.51
Table 4: Fine-grained examples for all three metrics. GITA: original Italian data; BasPhyCo: manually translated Standard Basque data; BasPhyCo west : automatically adapted data into the Western dialect.
P. state
total
GITA
BasPhyCo
BasPhyCo west
Avrg
Open
88
20.45
19.32
12.50
17.42
Functional
53
26.41
15.09
16.98
19.49
Exist
47
27.66
23.40
21.28
24.11
Power
36
41.67
36.11
13.63
30.47
In pieces
35
42.86
25.71
31.43
33.33
Location
33
15.15
6.06
6.06
9.09
Table 5: Verifiability results per physical state. These results are for Latxa-3.1-70B-It, the model with the highest verifiability results for Italian, Standard and Western Basque.