Large language models suffer from content effects in reasoning tasks, particularly in multi-lingual contexts. We introduce a novel method that reduces these biases through explicit structural abstraction that transforms syllogisms into canonical logical representations and applies deterministic parsing to determine validity. Evaluated on the SemEval-2026 Task 11 multilingual benchmark, our approach achieves top-5 rankings across all subtasks while substantially reducing content effects and offering a competitive alternative to complex fine-tuning or activation-level interventions.
Figures & tables
Figure 1: The flowchart illustrates the example step by step the flow of the proposed system.
Normalization
Acc
Bias
Combined
Raw Data
92.10
10.63
26.66
PA Notation
77.48
8.17
24.08
FOL Notation
96.85
3.19
39.80
Syllogism Notation
98.95
2.13
46.23
Table 1: Performance comparison across different normalization strategies on english-only data.
Method
Acc
F1 (Premise)
Bias
Combined
Logical Validity (English)
LLM-only
98.43
-
2.13
45.74
Norm + Parsing
100
-
0.0
100
Relevance Premises (English)
LLM-only
95.78
98.94
5.0
34.87
Norm + Parsing
98.94
95.43
2.0
46.31
Table 2: Comparison of different methods across subtasks. LLM-only denotes direct inference, Norm + Parsing is the deterministic parsing, and EPN refers to English Pivot Normalization.
Figure 2: Content-effect reduction in English-only and Multilingual
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
#
Team Name
Accuracy
Content Effect
Combined
1
rongchuan
100
0.0
100
…
…
…
…
…
9
ITLC
100
0.0
100
…
…
…
…
…
12
ewelinaksiez
97.91
0.02
95.81
13
chisnguyen
99.48
1.06
57.68
Appendix
Table 3: Leaderboard comparison of Subtask 1
#
Team Name
Accuracy
F1
Bias
Combined
1
Habib_TAZ
100
100
0.0
100
2
YNU-NLP
100
100
0.0
100
3
PA
100
98.95
0.0
99.47
4
junhaofu
96.84
94.21
1.13
54.43
5
butasrafael
96.88
95.83
1.17
54.24
6
ITLC
98.94
95.43
2.0
46.31
Appendix
Table 4: Leaderboard comparison of Subtask 2
#
Team Name
Accuracy
Content Effect
Combined
1
Habib_TAZ
100
0.0
100
2
PA
100
0.0
100
3
ITLC
100
0.0
100
4
junhaofu
95.83
0.17
82.59
5
rongchuan
96.35
1.0
56.97
Appendix
Table 5: Leaderboard comparison of Subtask 3
#
Team Name
Accuracy
F1
Bias
Combined
1
PA
100
94.36
0.0
97.18
2
YNU-NLP
90.1
89.58
1.26
49.51
3
sungbin_kai
86.46
86.8
1.2
48.48
4
ufal_cuni
84.9
83.42
1.37
45.2
5
ITLC
91.15
98.31
2.19
43.83
Appendix
Table 6: Leaderboard comparison of Subtask 4
Figure
Valid moods (major–minor–conclusion)
1
AAA, EAE, AII, EIO, AAI, EAO
2
EAE, AEE, EIO, AOO, EAO, AEO
3
AAI, IAI, AII, EAO, OAO, EIO
4
AAI, AEE, IAI, EAO, EIO, AEO
Appendix
Table 7: Lookup table for valid moods by figure
Type
Sentence Pattern
Predicate Logic
Set Theory
Description
A (all)
All S are P
∀x(S(x)→P(x))
S⊆P
Universal Affirmative
E (no)
No S are P
∀x(S(x)→¬P(x))
S∩P=∅
Universal Negative
I (some)
Some S are P
∃x(S(x)∧P(x))
S∩P=∅
Particular Affirmative
O (some-not)
Some S are not P
∃x(S(x)∧¬P(x))
S∖P=∅
Particular Negative
Appendix
Table 8: Four types of categorical sentences and their translations into predicate-logic and set-theoretic notation.
Idx
Content
Interpretation
Role
Inference Issue
0
There are no bikes that can be called cars.
No bikes are cars.
premise
–
1
It is also true that every bike is a type of vehicle.
All bikes are vehicles.
premise
–
2
This has led to the conclusion that a portion of vehicles are bikes.
Some vehicles are bikes.
conclusion
Semantically plausible, but not logically entailed.
GT validity: False LLM-only prediction: True
Appendix
Table 9: LLM-only (Subtask 1) exhibits plausibility bias by accepting a semantically plausible conclusion that is not logically entailed by the premises.
Idx
Content
Post EPN
Interpretation
Role
Error Note
0
No se trata de que cada llave sea un objeto.
It is not the case that every key is an object.
Some keys are not objects. (O)
premise
–
1
Todo lo que sea una llave inglesa es una herramienta.
Everything that is a wrench is a tool.
All wrenches are tools. (A)
premise
–
2
Por tanto, se puede concluir que algunas herramientas no son objetos.
Therefore, some tools are not objects.
Some tools are not objects. (O)
conclusion
LLM fails to convert the negated universal (“no se trata de que cada”) into O-type; treats it as E-type, breaking the inference chain.
Table 10: LLM-only (Subtask 3) fails to resolve a negated universal quantifier in Spanish, misinterpreting “no se trata de que cada” as a universal negative rather than the intended particular negative.
Idx
Original (English)
Content (French)
Interpretation
Role
Error Note
0
It is not true that every dog is a poodle.
Ce n’est pas vrai que tous les chiens sont des caniches.
Some dogs are not poodles. (O)
premise
–
1
Every creature that is a dog is a canine .
Toute créature qui est un chien est un chien .
All A are A. (tautology)
premise
French collapses “canine” into “chien” (dog), destroying the three-term structure.
2
Some canines are not poodles.
Certains chiens ne sont pas des caniches.
Some dogs are not poodles. (O)
conclusion
–
GT validity: True Norm + Parsing prediction: False Language: French
Appendix
Table 11: Norm + Parsing without translation (Subtask 3) fails on a French syllogism where the translation collapses “canine” and “dog” into the same French word “chien,” reducing a valid three-term syllogism to a degenerate two-term structure.
Idx
Premise
Role
Prediction
0
There are no circles that are also three-sided figures.
Relevant
Selected
1
Some isosceles triangles are three-sided figures.
Distractor
Wrongly selected
2
All scalene triangles are three-sided figures.
Distractor
–
3
Every equilateral triangle is a three-sided figure.
Distractor
–
4
All figures that are triangles are three-sided figures.
Relevant
Missed
5
Circles are not triangles.
Distractor
–
Appendix
Table 12: Example of minor representational overlap in Subtask 4. The model selects premise (1) due to shared surface terminology (“three-sided figures”), although premise (4) is required for the correct logical structure.
Idx
Premise
Role
Prediction
0
Some butterflies have colorful wings.
Distractor
–
1
It is true that all beetles have six legs.
Distractor
–
2
There are no webs spun by grasshoppers.
Relevant
Missed
3
Spiders are never ants.
Relevant
Missed
4
Every single bee pollinates flowers.
Distractor
–
5
A few ants are, in fact, insects.
Distractor
–
Appendix
Table 13: Example of LLM-only relevant premises mismatch. Although premises (2) and (3) are required to establish the contradiction structure, the model fails to select any relevant premises and predicts an incorrect validity label.
Idx
Premise
Role
Prediction
0
Some hounds are actually birds.
Distractor
–
1
Any retriever is a type of fish.
Distractor
–
2
Anything that is a poodle is also a canine.
Distractor
Wrongly selected
3
There are no animals that bark.
Distractor
–
4
It is a fact that all collies have scales.
Distractor
–
5
Every single puppy is a kitten.
Distractor
–
Appendix
Table 14: Example of false-positive premise selection. Although no premise is structurally required for the invalid syllogism, the model selects (2) and (7) due to surface-level semantic overlap, leading to an incorrect validity prediction.
Idx
Original (IT)
English
Role
0
Tutto ciò che è un uccello depone le uova
Anything that is a bird lays eggs
GT premise
1
Ogni singola anatra è un uccello
Every single duck is a bird
Distractor
2
Alcuni uccelli sono creature che costruiscono nidi
Some birds are creatures that build nests
Distractor
3
Qualsiasi struzzo è un uccello che depone le uova
Any ostrich is a bird that lays eggs
Distractor
4
Non esistono pinguini che non depongono le uova
There are no penguins that do not lay eggs
Distractor
5
Non esistono polli che non siano uccelli
There are no chickens that are not birds
GT premise
Appendix
Table 15: LLM-only (subtask 4) misses the active premise pair [0,5]; semantically plausible distractors about ostriches and penguins lead the model to predict invalid.
Idx
Original (ES)
English
Role
0
Cada brócoli es una verdura
Every single broccoli is a vegetable
Distractor
1
Es cierto que todas las frutas son comestibles
It is true that all fruits are edible
Distractor
2
Cualquier patata es una verdura
Any potato is a vegetable
Distractor
3
Algunas cosas comestibles son granos
Some edible things are grains
Distractor
4
Hay algunas raíces que son comestibles
There are a few roots that are edible
Distractor
5
Cada vegetal es comestible
Every single vegetable is edible
Wrongly selected as P1
Appendix
Table 16: Norm+Parsing (subtask 4) selects sentences [5,6] forming a plausible carrot → vegetable → edible chain that maps to a valid mood, despite the overall argument being invalid.
Idx
Original (FR)
English
Role
0
Chaque chiot naît en sachant parler français
Every single puppy is born knowing how to speak French
Distractor
1
Aucun chien n’a jamais marché sur quatre pattes
No canine has ever walked on four legs
Distractor
2
Il n’existe pas d’animaux qui soient aussi des chiens
There exist no animals that are also canines
GT premise
3
Il est indéniable que le ciel est vert et l’herbe bleue
The sky is green and the grass is blue
Distractor
4
Tous les chiens sont des robots déguisés
All dogs are actually disguised robots from space
Distractor
5
Certains animaux sont entièrement en verre
Some animals are made entirely of glass
Wrongly selected as P1
Appendix
Table 17: The EPN step (subtask 4) selects sentence [5] (a distractor) as P1 instead of the correct premise [2], producing a structure that the downstream classifier rejects as invalid.
Figure 3: LLM-only prompt for retrieve the validity and relevant premise directly
Figure 4: Norm prompt for normalize sentences into standard categorical form
Figure 5: EPN prompt for Subtask 3 for extract subject term
Figure 6: EPN prompt for Subtask 4 for extract and filter relevant premise and conclusion
Figure 7: EPN prompt with google translated sentence for Subtask 4 for extract and filter relevant premise and conclusion
Method
Acc
F1 (premise)
Bias
Combined
Norm + Parsing
90.63
72.50
7.47
26.01
Norm + Parsing using google translate sentence
88.54
88.39
8.14
27.54
EPN + Norm + Parsing
90.63
90.10
3.00
37.88
+ google translate example
89.58
89.58
4.32
33.53
Appendix
Table 18: Ablation study on the impact of incorporating Google-translated sentences in a multilingual setting
S1
S5
S7
Conclusion
Original (PT)
Alguns mamíferos não são cães
É certo que todo cachorro é um mamífero
Todos os caninos são cães
Todos os caninos são mamíferos
Google Trans.
Some mammals are not dogs
It is true that every dog is a mammal
All canines are dogs
All canines are mammals
Ground Truth
Some mammals are not dogs
Every single dog is a mammal
All canines are dogs
All canines are mammals
Appendix
Table 19: Effect of lexical collapse in the Google-translated reference. Highlighted cells show selected premises per system. Terms are coloured consistently: canines , mammals , dogs .
P1
P2
Conclusion
Original (IT)
Tutto ciò che è una rosa è anche una pianta
Esistono rose che sono fiori
Alcuni fiori sono piante
Google Trans.
Everything that is a rose is also a plant
There are roses that are flowers
Some flowers are plants
Appendix
Table 20: Italian syllogism with Google-translated reference. Terms are coloured consistently: rose , flowers , plants .
While Large Language Models (LLMs) excel in many general NLP tasks, their formal reasoning capabilities are often compromised by content effects, demonstrating a measurable bias towards real-world plausibility. In this paper, we present our system for SemEval-2026 Task 11, which evaluates the ability of models to disentangle formal logic from content across 12 languages with and without distractor premises. We address this challenge using mDeBERTa-v3 networks fine-tuned on a synthetic, rule-based dataset of syllogistic schemes to avoid the semantic noise of LLM-augmented data. To explicitly decouple plausibility from logical structure, our training pipeline employs a multi-objective loss function combining Adaptive Group Distributionally Robust Optimization (DRO), a scheduled differentiable bias penalty, and KL-Divergence consistency regularization. Our system achieved #1 ranks and perfect Ranking Scores (100.0) with 0.00% bias and 100.0% accuracy on Subtask 1 (English), Subtask 2 (Noisy English), and Subtask 3 (Multilingual). On the highly complex Subtask 4 (Noisy Multilingual), the system achieved the 6th rank with 89.06% Accuracy and F1-score, alongside a limited 2.89% Bias and a 37.78 Ranking Score. Our dataset generation engine and codebase are publicly available to facilitate future work on robust logical reasoning.
Abdullah Shaikh, Zain Naqi, Taha Zahid +2
Dhanani School of Science & Engineering Habib University, Pakistan
This paper revisits our pipeline called Syllogistic Evaluation Framework-Common Logic Grammar Construction (SEF-CLGC). We combine formal logical notations with Small Language Models (SLMs) to evaluate reasoning performance on the SemEval-2026 Task 11 Subtask 1: Disentangling Content and Formal Reasoning in Large Language Models. Our experiments show that by relying solely on SLMs, trained on a combination of natural and symbolic languages, our best model achieves a content score of 27.80% on the task while significantly lowering the content bias in reasoning.
Hanna Abi Akl, Fabien Gandon, Catherine Faron +1
Université Côte d’Azur, Inria, CNRS, I3S, Sophia Antipolis, France · Data ScienceTech Institute, Paris, France
This paper describes our system submitted to SemEval-2026 Task 11: Disentangling Content and Formal Reasoning in Large Language Models. We present an efficient modular neuro-symbolic approach, combining a symbolic prover with small reasoning LLMs (4B parameters). The system consists of an LLM-based parser that translates natural language syllogisms to a first-order logic (FOL) representation, an automated theorem prover, and two optional modules: machine translation for multilingual inputs and a symbolic retrieval component for the identification of relevant premises. The system achieves competitive accuracy and relatively low content effect on most subtasks. Our ablations show that this approach outperforms LLM-based zero-shot baselines in this parameter size range, but also reveal limited multilingual capabilities of small LLMs. Finally, we include a discussion of the task's main ranking metric and analyze its limitations.
Ivan Kartáč, Kristýna Onderková, Jan Bronec +3
Institute of Formal and Applied Linguistics, Faculty of Mathematics and Physics, Charles University