cs.CL · 2606.09157 Copy arXiv ID · Jun 8, 2026 Save SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance Authors: Hanna Abi Akl , Fabien Gandon , Catherine Faron , Pierre Monnin
Organizations: Université Côte d’Azur, Inria, CNRS, I3S, Sophia Antipolis, France · Data ScienceTech Institute, Paris, France
Abstract This paper revisits our pipeline called Syllogistic Evaluation Framework-Common Logic Grammar Construction (SEF-CLGC). We combine formal logical notations with Small Language Models (SLMs) to evaluate reasoning performance on the SemEval-2026 Task 11 Subtask 1: Disentangling Content and Formal Reasoning in Large Language Models. Our experiments show that by relying solely on SLMs, trained on a combination of natural and symbolic languages, our best model achieves a content score of 27.80% on the task while significantly lowering the content bias in reasoning.
Explore similar work Jul 22, 2026 · Hanna Abi Akl, Fabien Gandon, Catherine Faron +1
May 6, 2026 · Ivan Kartáč, Kristýna Onderková, Jan Bronec +3 Neuro-Symbolic Framework Large Language Model Reasoning
Jul 15, 2026 · Abdullah Shaikh, Zain Naqi, Taha Zahid +2 Large Language Model Reasoning Synthetic Training Data
Jul 22, 2026 · cs.CL J/K move · Enter open · S save
Hanna Abi Akl, Fabien Gandon, Catherine Faron, Pierre Monnin
Université Côte d’Azur, Inria, CNRS, I3S, Sophia Antipolis, France · Data ScienceTech Institute, Paris, France
Language models (LMs) struggle with logical tasks like reasoning on syllogisms. It has been shown that Knowledge Representation (KR) plays a crucial role in expressing input information to help models solve tasks. This observation motivates our study of the impact of different formal KR notations on syllogistic reasoning by extending the FOLIO and P-FOLIO datasets. Our experiments on Small Language Models (SLMs) in Supervised Fine-Tuning (SFT) and Zero-Shot (ZS) settings show that the choice of input notation can yield performances competitive with natural language while enabling faster inference. We also propose a syllogistic categorization method (SEF) and use it to enrich ZS prompts with logical definitions, which boost reasoning in small models. We open-source our framework, Common Logic Grammar Construction (CLGC), as the first Python library for automatically generating syllogisms in KR notations and defining their SEF categories.