ITLC at SemEval-2026 Task 11: Normalization and Deterministic Parsing for Formal Reasoning in LLMs

Wicaksono Leksono Muhamad; Joanito Agili Lopo; Tack Hwa Wong; Muhammad Ravi Shulthan Habibi; Samuel Cahyawijaya

cs.CLMar 3, 2026

ITLC at SemEval-2026 Task 11: Normalization and Deterministic Parsing for Formal Reasoning in LLMs

Authors: Wicaksono Leksono Muhamad, Joanito Agili Lopo, Tack Hwa Wong, Muhammad Ravi Shulthan Habibi, Samuel Cahyawijaya

Organizations: SEACrowd · Mantera Studio · Universiti Teknologi PETRONAS · Universitas Indonesia ·

Abstract

Large language models suffer from content effects in reasoning tasks, particularly in multi-lingual contexts. We introduce a novel method that reduces these biases through explicit structural abstraction that transforms syllogisms into canonical logical representations and applies deterministic parsing to determine validity. Evaluated on the SemEval-2026 Task 11 multilingual benchmark, our approach achieves top-5 rankings across all subtasks while substantially reducing content effects and offering a competitive alternative to complex fine-tuning or activation-level interventions.

Figures & tables

Figure 1: The flowchart illustrates the example step by step the flow of the proposed system.

Normalization	Acc	Bias	Combined
Raw Data	92.10	10.63	26.66
PA Notation	77.48	8.17	24.08
FOL Notation	96.85	3.19	39.80
Syllogism Notation	98.95	2.13	46.23

Table 1: Performance comparison across different normalization strategies on english-only data.

Method	Acc	F1 (Premise)	Bias	Combined
Logical Validity (English)
LLM-only	98.43	-	2.13	45.74
Norm + Parsing	100	-	0.0	100
Relevance Premises (English)
LLM-only	95.78	98.94	5.0	34.87
Norm + Parsing	98.94	95.43	2.0	46.31

Table 2: Comparison of different methods across subtasks. LLM-only denotes direct inference, Norm + Parsing is the deterministic parsing, and EPN refers to English Pivot Normalization.

Figure 2: Content-effect reduction in English-only and Multilingual

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

#	Team Name	Accuracy	Content Effect	Combined
1	rongchuan	100	0.0	100
…	…	…	…	…
9	ITLC	100	0.0	100
…	…	…	…	…
12	ewelinaksiez	97.91	0.02	95.81
13	chisnguyen	99.48	1.06	57.68

Appendix

Table 3: Leaderboard comparison of Subtask 1

#	Team Name	Accuracy	F1	Bias	Combined
1	Habib_TAZ	100	100	0.0	100
2	YNU-NLP	100	100	0.0	100
3	PA	100	98.95	0.0	99.47
4	junhaofu	96.84	94.21	1.13	54.43
5	butasrafael	96.88	95.83	1.17	54.24
6	ITLC	98.94	95.43	2.0	46.31

Appendix

Table 4: Leaderboard comparison of Subtask 2

#	Team Name	Accuracy	Content Effect	Combined
1	Habib_TAZ	100	0.0	100
2	PA	100	0.0	100
3	ITLC	100	0.0	100
4	junhaofu	95.83	0.17	82.59
5	rongchuan	96.35	1.0	56.97

Appendix

Table 5: Leaderboard comparison of Subtask 3

#	Team Name	Accuracy	F1	Bias	Combined
1	PA	100	94.36	0.0	97.18
2	YNU-NLP	90.1	89.58	1.26	49.51
3	sungbin_kai	86.46	86.8	1.2	48.48
4	ufal_cuni	84.9	83.42	1.37	45.2
5	ITLC	91.15	98.31	2.19	43.83

Appendix

Table 6: Leaderboard comparison of Subtask 4

Figure	Valid moods (major–minor–conclusion)
1	AAA, EAE, AII, EIO, AAI, EAO
2	EAE, AEE, EIO, AOO, EAO, AEO
3	AAI, IAI, AII, EAO, OAO, EIO
4	AAI, AEE, IAI, EAO, EIO, AEO

Appendix

Table 7: Lookup table for valid moods by figure

Type	Sentence Pattern	Predicate Logic	Set Theory	Description
$A$ (all)	All $S$ are $P$	$\forall x\,(S(x)\rightarrow P(x))$	$S\subseteq P$	Universal Affirmative
$E$ (no)	No $S$ are $P$	$\forall x\,(S(x)\rightarrow\neg P(x))$	$S\cap P=\emptyset$	Universal Negative
$I$ (some)	Some $S$ are $P$	$\exists x\,(S(x)\wedge P(x))$	$S\cap P\neq\emptyset$	Particular Affirmative
$O$ (some-not)	Some $S$ are not $P$	$\exists x\,(S(x)\wedge\neg P(x))$	$S\setminus P\neq\emptyset$	Particular Negative

Appendix

Table 8: Four types of categorical sentences and their translations into predicate-logic and set-theoretic notation.

GT validity: False LLM-only prediction: True
Idx	Content	Interpretation	Role	Inference Issue
0	There are no bikes that can be called cars.	No bikes are cars.	premise	–
1	It is also true that every bike is a type of vehicle.	All bikes are vehicles.	premise	–
2	This has led to the conclusion that a portion of vehicles are bikes.	Some vehicles are bikes.	conclusion	Semantically plausible, but not logically entailed.

Appendix

Table 9: LLM-only (Subtask 1) exhibits plausibility bias by accepting a semantically plausible conclusion that is not logically entailed by the premises.

Idx	Content	Post EPN	Interpretation	Role	Error Note
0	No se trata de que cada llave sea un objeto.	It is not the case that every key is an object.	Some keys are not objects. (O)	premise	–
1	Todo lo que sea una llave inglesa es una herramienta.	Everything that is a wrench is a tool.	All wrenches are tools. (A)	premise	–
2	Por tanto, se puede concluir que algunas herramientas no son objetos.	Therefore, some tools are not objects.	Some tools are not objects. (O)	conclusion	LLM fails to convert the negated universal (“no se trata de que cada”) into O-type; treats it as E-type, breaking the inference chain.
GT validity: True LLM-only prediction: False Language: Spanish

Appendix

Table 10: LLM-only (Subtask 3) fails to resolve a negated universal quantifier in Spanish, misinterpreting “no se trata de que cada” as a universal negative rather than the intended particular negative.

Idx	Original (English)	Content (French)	Interpretation	Role	Error Note
0	It is not true that every dog is a poodle.	Ce n’est pas vrai que tous les chiens sont des caniches.	Some dogs are not poodles. (O)	premise	–
1	Every creature that is a dog is a canine .	Toute créature qui est un chien est un chien .	All A are A. (tautology)	premise	French collapses “canine” into “chien” (dog), destroying the three-term structure.
2	Some canines are not poodles.	Certains chiens ne sont pas des caniches.	Some dogs are not poodles. (O)	conclusion	–
GT validity: True Norm + Parsing prediction: False Language: French

Appendix

Table 11: Norm + Parsing without translation (Subtask 3) fails on a French syllogism where the translation collapses “canine” and “dog” into the same French word “chien,” reducing a valid three-term syllogism to a degenerate two-term structure.

Idx	Premise	Role	Prediction
0	There are no circles that are also three-sided figures.	Relevant	Selected
1	Some isosceles triangles are three-sided figures.	Distractor	Wrongly selected
2	All scalene triangles are three-sided figures.	Distractor	–
3	Every equilateral triangle is a three-sided figure.	Distractor	–
4	All figures that are triangles are three-sided figures.	Relevant	Missed
5	Circles are not triangles.	Distractor	–

Appendix

Table 12: Example of minor representational overlap in Subtask 4. The model selects premise (1) due to shared surface terminology (“three-sided figures”), although premise (4) is required for the correct logical structure.

Idx	Premise	Role	Prediction
0	Some butterflies have colorful wings.	Distractor	–
1	It is true that all beetles have six legs.	Distractor	–
2	There are no webs spun by grasshoppers.	Relevant	Missed
3	Spiders are never ants.	Relevant	Missed
4	Every single bee pollinates flowers.	Distractor	–
5	A few ants are, in fact, insects.	Distractor	–

Appendix

Table 13: Example of LLM-only relevant premises mismatch. Although premises (2) and (3) are required to establish the contradiction structure, the model fails to select any relevant premises and predicts an incorrect validity label.

Idx	Premise	Role	Prediction
0	Some hounds are actually birds.	Distractor	–
1	Any retriever is a type of fish.	Distractor	–
2	Anything that is a poodle is also a canine.	Distractor	Wrongly selected
3	There are no animals that bark.	Distractor	–
4	It is a fact that all collies have scales.	Distractor	–
5	Every single puppy is a kitten.	Distractor	–

Appendix

Table 14: Example of false-positive premise selection. Although no premise is structurally required for the invalid syllogism, the model selects (2) and (7) due to surface-level semantic overlap, leading to an incorrect validity prediction.

Idx	Original (IT)	English	Role
0	Tutto ciò che è un uccello depone le uova	Anything that is a bird lays eggs	GT premise
1	Ogni singola anatra è un uccello	Every single duck is a bird	Distractor
2	Alcuni uccelli sono creature che costruiscono nidi	Some birds are creatures that build nests	Distractor
3	Qualsiasi struzzo è un uccello che depone le uova	Any ostrich is a bird that lays eggs	Distractor
4	Non esistono pinguini che non depongono le uova	There are no penguins that do not lay eggs	Distractor
5	Non esistono polli che non siano uccelli	There are no chickens that are not birds	GT premise

Appendix

Table 15: LLM-only (subtask 4) misses the active premise pair [0,5]; semantically plausible distractors about ostriches and penguins lead the model to predict invalid.

Idx	Original (ES)	English	Role
0	Cada brócoli es una verdura	Every single broccoli is a vegetable	Distractor
1	Es cierto que todas las frutas son comestibles	It is true that all fruits are edible	Distractor
2	Cualquier patata es una verdura	Any potato is a vegetable	Distractor
3	Algunas cosas comestibles son granos	Some edible things are grains	Distractor
4	Hay algunas raíces que son comestibles	There are a few roots that are edible	Distractor
5	Cada vegetal es comestible	Every single vegetable is edible	Wrongly selected as P1

Appendix

Table 16: Norm+Parsing (subtask 4) selects sentences [5,6] forming a plausible carrot

\to

vegetable

\to

edible chain that maps to a valid mood, despite the overall argument being invalid.

Idx	Original (FR)	English	Role
0	Chaque chiot naît en sachant parler français	Every single puppy is born knowing how to speak French	Distractor
1	Aucun chien n’a jamais marché sur quatre pattes	No canine has ever walked on four legs	Distractor
2	Il n’existe pas d’animaux qui soient aussi des chiens	There exist no animals that are also canines	GT premise
3	Il est indéniable que le ciel est vert et l’herbe bleue	The sky is green and the grass is blue	Distractor
4	Tous les chiens sont des robots déguisés	All dogs are actually disguised robots from space	Distractor
5	Certains animaux sont entièrement en verre	Some animals are made entirely of glass	Wrongly selected as P1

Appendix

Table 17: The EPN step (subtask 4) selects sentence [5] (a distractor) as P1 instead of the correct premise [2], producing a structure that the downstream classifier rejects as invalid.

Panel of Figure 3 — Figure 3: LLM-only prompt for retrieve the validity and relevant premise directly

Panel of Figure 4 — Figure 4: Norm prompt for normalize sentences into standard categorical form

Panel of Figure 5 — Figure 5: EPN prompt for Subtask 3 for extract subject term

Panel of Figure 6 — Figure 6: EPN prompt for Subtask 4 for extract and filter relevant premise and conclusion

Panel of Figure 7 — Figure 7: EPN prompt with google translated sentence for Subtask 4 for extract and filter relevant premise and conclusion

Method	Acc	F1 (premise)	Bias	Combined
Norm + Parsing	90.63	72.50	7.47	26.01
Norm + Parsing using google translate sentence	88.54	88.39	8.14	27.54
EPN + Norm + Parsing	90.63	90.10	3.00	37.88
+ google translate example	89.58	89.58	4.32	33.53

Appendix

Table 18: Ablation study on the impact of incorporating Google-translated sentences in a multilingual setting

	S1	S5	S7	Conclusion
Original (PT)	Alguns mamíferos não são cães	É certo que todo cachorro é um mamífero	Todos os caninos são cães	Todos os caninos são mamíferos
Google Trans.	Some mammals are not dogs	It is true that every dog is a mammal	All canines are dogs	All canines are mammals
Ground Truth	Some mammals are not dogs	Every single dog is a mammal	All canines are dogs	All canines are mammals

Appendix

Table 19: Effect of lexical collapse in the Google-translated reference. Highlighted cells show selected premises per system. Terms are coloured consistently: canines , mammals , dogs .

	P1	P2	Conclusion
Original (IT)	Tutto ciò che è una rosa è anche una pianta	Esistono rose che sono fiori	Alcuni fiori sono piante
Google Trans.	Everything that is a rose is also a plant	There are roses that are flowers	Some flowers are plants

Appendix

Table 20: Italian syllogism with Google-translated reference. Terms are coloured consistently: rose , flowers , plants .

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 926 papers

HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization
Jul 15, 2026Abdullah Shaikh, Zain Naqi, Taha Zahid +2Multilingual Language Model Evaluation Language Model Bias Evaluation
Dhanani School of Science & Engineering Habib University, Pakistan
SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance
Jun 8, 2026Hanna Abi Akl, Fabien Gandon, Catherine Faron +1Logical Reasoning Symbolic Reasoning
Université Côte d’Azur, Inria, CNRS, I3S, Sophia Antipolis, France · Data ScienceTech Institute, Paris, France
UFAL-CUNI at SemEval-2026 Task 11: An Efficient Modular Neuro-symbolic Method for Syllogistic Reasoning
May 6, 2026Ivan Kartáč, Kristýna Onderková, Jan Bronec +3Logical Reasoning Symbolic Reasoning
Institute of Formal and Applied Linguistics, Faculty of Mathematics and Physics, Charles University

Jul 15, 2026J/K move · Enter open · S save

HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization

Abdullah Shaikh, Zain Naqi, Taha Zahid, Sandesh Kumar+1

Dhanani School of Science & Engineering Habib University, Pakistan

Multilingual Language Model Evaluation Language Model Bias Evaluation

Open paper AskPDF arXiv

While Large Language Models (LLMs) excel in many general NLP tasks, their formal reasoning capabilities are often compromised by content effects, demonstrating a measurable bias towards real-world plausibility. In this paper, we present our system for SemEval-2026 Task 11, which evaluates the ability of models to disentangle formal logic from content across 12 languages with and without distractor premises. We address this challenge using mDeBERTa-v3 networks fine-tuned on a synthetic, rule-based dataset of syllogistic schemes to avoid the semantic noise of LLM-augmented data. To explicitly decouple plausibility from logical structure, our training pipeline employs a multi-objective loss function combining Adaptive Group Distributionally Robust Optimization (DRO), a scheduled differentiable bias penalty, and KL-Divergence consistency regularization. Our system achieved #1 ranks and perfect Ranking Scores (100.0) with 0.00% bias and 100.0% accuracy on Subtask 1 (English), Subtask 2 (Noisy English), and Subtask 3 (Multilingual). On the highly complex Subtask 4 (Noisy Multilingual), the system achieved the 6th rank with 89.06% Accuracy and F1-score, alongside a limited 2.89% Bias and a 37.78 Ranking Score. Our dataset generation engine and codebase are publicly available to facilitate future work on robust logical reasoning.

Abstract

LLM Reasoning

Explore similar work

HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization

SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance

UFAL-CUNI at SemEval-2026 Task 11: An Efficient Modular Neuro-symbolic Method for Syllogistic Reasoning