Physical commonsense reasoning represents a fundamental capability of human intelligence, enabling individuals to understand their environment, predict future events, and navigate physical spaces. Recent years have witnessed growing interest in reasoning tasks within Natural Language Processing (NLP). However, no prior research has examined the performance of Large Language Models (LLMs) on non-question-answering (non-QA) physical commonsense reasoning tasks in low-resource languages such as Basque. Taking the Italian GITA as a starting point, this paper addresses this gap by presenting BasPhyCo, the first non-QA physical commonsense reasoning dataset for Basque, available in both standard and dialectal variants. We evaluate model performance across three hierarchical levels of commonsense understanding: (1) distinguishing between plausible and implausible narratives (accuracy), (2) identifying the conflicting element that renders a narrative implausible (consistency), and (3) determining the specific physical state that creates the implausibility (verifiability). These tasks were assessed using multiple multilingual LLMs as well as models pretrained specifically for Italian and Basque. Results indicate that, in terms of verifiability, LLMs exhibit limited physical commonsense capabilities in low-resource languages such as Basque, especially when processing dialectal variants.
Figures & tables
Type
Sentence 1
Sentence 2
Sentence 3
Sentence 4
Sentence 5
Plausible
George filled the glass with water.
George put the glass in the microwave.
George turned on the microwave.
The water got hot.
George put a tea bag in the water.
Implausible (order)
George put the glass in the microwave.
George filled the glass with water.
George turned on the microwave.
The water got hot.
George put a tea bag in the water.
Implausible (cloze)
George filled the glass with water.
George put the glass in the microwave.
George turned on the microwave.
The water got cold.
George put a tea bag in the water
Table 1: Example of a story with its plausible and implausible versions.
Standard
Dialectal
Jonek lorategi handi bat dauka. Elur lorategian dago. Jonek lorategiko atea ireki du. Elurrek alde egin du. Lorategia hutsik dago.
Jonek lorategi handi bat dauko . Elur lorategixen dago. Jonek lorategiko atie zabaldu dau . Elurrek ospa egin dau. Lorategixe hutsik dago.
Table 2: Example of a story adapted from Standard Basque to Western Basque.
Figure 1: The number of unique words in BasPhyCo (left) and BasPhyCo west (right), as well as the overlap of both datasets (middle). Additionally, some examples from each dataset.
Test data
Model
Accuracy
Consistency
Verifiability
GITA
Llama-3.1-8B-It
72.47
29.29
14.23
Llama-3.1-70B-It
87.64
65.27
36.40
Gemma-2-9B-It
71.91
35.98
16.74
Gemma-2-27B-It
67.42
30.96
15.06
Latxa-3.1-8B-It
67.42
23.85
13.39
Latxa-3.1-70B-It
85.96
60.25
40.17
Table 3: Overall results for Story Classification, Conflict Detection and Physical State Classification, measured by accuracy, consistency and verifiability, respectively. GITA: original Italian data; BasPhyCo: manually translated Standard Basque data; BasPhyCo west : automatically adapted data into the Western dialect.
Language
Model
Accuracy
Consistency
Verifiability
Order
Cloze
Plausible
Order
Cloze
Order
Cloze
GITA
Llama-3.1-8B-It
64.75
81.20
71.79
14.75
44.44
4.92
23.93
Llama-3.1-70B-It
88.52
95.73
78.63
55.74
74.36
25.41
47.01
Gemma-2-9B-It
49.18
81.20
86.32
15.57
57.26
2.46
31.62
Gemma-2-27B-It
34.43
78.63
90.60
9.02
53.85
4.10
26.50
Latxa-3.1-8B-It
55.74
68.38
78.63
13.11
35.04
6.56
20.51
Table 4: Fine-grained examples for all three metrics. GITA: original Italian data; BasPhyCo: manually translated Standard Basque data; BasPhyCo west : automatically adapted data into the Western dialect.
P. state
total
GITA
BasPhyCo
BasPhyCo west
Avrg
Open
88
20.45
19.32
12.50
17.42
Functional
53
26.41
15.09
16.98
19.49
Exist
47
27.66
23.40
21.28
24.11
Power
36
41.67
36.11
13.63
30.47
In pieces
35
42.86
25.71
31.43
33.33
Location
33
15.15
6.06
6.06
9.09
Table 5: Verifiability results per physical state. These results are for Latxa-3.1-70B-It, the model with the highest verifiability results for Italian, Standard and Western Basque.
We introduce mmPISA-bench, a compact high-quality multilingual reasoning benchmark derived from the OECD Programme for International Student Assessment (PISA). The benchmark consists of 25 multiple-choice questions that require reasoning in order to be answered correctly. Each question is provided in official human translations to 43 languages and complemented with machine-translated counterparts (i.e., 2,150 data points in total). We evaluate two mainstream proprietary LLMs across languages, reasoning effort levels, and translation types in terms of their ability to answer the questions correctly. Our results show that modern LLMs can reason effectively across all evaluated languages, achieve accuracy comparable to human test-takers, with some performance variations across covered languages. We further find that machine-translated questions do not degrade accuracy relative to official human translations which suggests that high-quality machine translation (synthetic data) might often be adequate for large-scale multilingual reasoning evaluations where official translations are not available. Finally, we analyze token usage and related inference cost and find that LLMs usage in some languages is simultaneously more expensive and less accurate.
Yerzhan Sapenov, Jaromir Savelka
School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA
Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.
Although the recent development of LLMs has paved the way for multi-modal interaction, minority languages face difficulties keeping pace. This work presents the first open Speech Language Model for Basque, designed for low-latency, end-to-end spoken interaction. Based on the LLaMA-Omni architecture, the system integrates a Latxa model as its backbone, transferring its Basque capabilities to the spoken modality by fine-tuning several modules. To address data scarcity, the English VoiceAssistant-400K corpus was translated into Basque, and the audio was synthesized using a Basque TTS. Evaluation results confirm the system's viability, achieving 3.86/5 in content quality, a 9.94% WER for spoken intelligibility, and a 3.07 UTMOS for audio quality. This work demonstrates that advanced spoken interaction systems can be developed using modular architectures and synthetic data even in low-resource languages.
Andoni Sudupe, Inma Hernaez, Ander Arriandiaga +3
HiTZ Basque Center for Language Technology - Aholab Signal Processing Laboratory, University of the Basque Country UPV/EHU, Spain