Argumentation is an important cognitive activity that involves the scrutiny of arguments and counterarguments. When given an argument, it is desirable that its premises entail its claim, and that the premises and claim are consistent, and when given a pair of arguments, it is desirable to check whether one argument supports or attacks another. If the arguments are in natural language, then we could use methods for natural language inference, also known as recognizing textual entailment (RTE). The state-of-the-art RTE methods use machine learning and deep learning, but these are not explainable. An alternative approach which we investigate in this paper is to translate text into an abstract meaning representation (AMR) graph. For this we use a pre-trained AMR parser. We then translate the AMR graph into propositional logic and use a SAT solver for automated reasoning. In text, often common sense suggests that an entailment (or contradiction) relationship holds between a premise and a claim, but because different wordings are used, this is not identified from their logical representations. To address this, we introduce relaxation methods to allow replacement or forgetting of some propositions. Our experimental results show our approach performs well on three RTE datasets. The benefit of our approach is that we have a logical representation of premises and claims, and of the relaxations that are used to identify the entailment and contradiction relations.
Figure 1: AMR graphs for the sentences “The boy wants to go.” (left) and “The boy does not want to go.” (right).
Figure 2: Our sentence-to-label pipeline. The text input is premises and claim, and the output is a label.
Predicted class
ϕ′∧¬ψ′
ϕ∗∧ψ′
Entailment
False
True
Contradiction
True
False
Neutral
True
True
Table 1: Classification of relations. Let ϕ (resp. ψ ) be a premise (resp. claim), let ϕ′ (resp. ψ′ ) be the relaxed premise (resp. relaxed claim), and let ϕ∗ be FgtΓ(ϕ′) where Γ is the set of forgettable atoms of ϕ′ and ψ′ . For columns 2 and 3, PySAT returns true if the formula is consistent, and returns False if it is inconsistent. So for instance the entailment relation holds when ϕ′∧¬ψ′ is False and ϕ∗∧ψ′ is True.
Table 2: Results from the e-SNLI dataset
Figure 3: Results concerning the neuro-matching relaxation threshold. The y-axis represents the scores of precision, recall, F1-score, and accuracy, and the x-axis represents the threshold τ .
Class
Precision
Recall
F1
Support
Entailment
0.659
0.942
0.776
499
Contradiction
0.906
0.810
0.855
500
Neutral
0.826
0.560
0.667
500
Table 3: SICK dataset with forgetting (accuracy 0.771)
Class
Precision
Recall
F1
Support
Entailment
0.610
0.578
0.593
476
Contradiction
0.552
0.374
0.446
494
Neutral
0.459
0.629
0.531
496
Table 4: MultiNLI dataset with forgetting (accuracy 0.527)
The logical reconstruction of argument graphs from natural language text is challenging because of the prevalence of enthymemes (i.e., arguments with implicit premises). There are natural language processing methods for identifying enthymemes in text, and there are symbolic methods based on abduction for identifying missing premises in a logical representation of enthymemes. However, there is a need for methods to generate implicit premises to logically show a known entailment or contradiction relationship between a pair of statements. To address this, we propose a neuro-symbolic pipeline that uses large language models (LLMs) to generate intermediate implicit premises that are translated into logical formulae and used with logical formulae representing explicit premises and explicit claims to show the logical relationships between them (entailment, contradiction, or neutrality). Our approach is evaluated on the Microtext Argumentative Corpus.
Xuyao Feng, Anthony Hunter
Department of Computer Science, University College London, United Kingdom
Real-world arguments in text and dialogues are normally enthymemes (i.e. some of their premises and/or claims are implicit). Natural language processing (NLP) methods for handling enthymemes can potentially identify enthymemes in text but they do not decode their underlying logic, whereas logic-based approaches for handling them assume a knowledgebase with sufficient formulae that can be used to decode them via abduction. There is therefore a lack of a systematic method for translating textual components of an enthymeme into a logical argument and generating the logical formulae required for their decoding, and thereby showing logical entailment. To address this, we propose a pipeline that integrates: (1) a large language model (LLM) to generate intermediate implicit premises based on the explicit premise and claim; (2) another LLM to translate the natural language into logical formulas; and (3) a neuro-symbolic reasoner based on a SAT solver to determine entailment. We evaluate our pipeline on two enthymeme datasets, demonstrating promising performance in selecting the correct implicit premise, as measured by precision, recall, F1-score, and accuracy.
Xuyao Feng, Anthony Hunter
Department of Computer Science, University College London, United Kingdom
Natural language understanding requires interleaving textual and logical reasoning, yet large language models often fail to perform such reasoning reliably. Existing neurosymbolic systems combine LLMs with solvers but remain limited to fully formalizable tasks such as math or program synthesis, leaving natural documents with only partial logical structure unaddressed. We introduce Logitext, a neurosymbolic language that represents documents as natural language text constraints (NLTCs), making partial logical structure explicit. We develop an algorithm that integrates LLM-based constraint evaluation with satisfiability modulo theory (SMT) solving, enabling joint textual-logical reasoning. Experiments on a new content moderation benchmark, together with LegalBench and Super-Natural Instructions, show that Logitext improves both accuracy and coverage. This work is the first that treats LLM-based reasoning as an SMT theory, extending neurosymbolic methods beyond fully formalizable domains.
Hyunseok Oh, Sam Stern, Youngki Lee +1
Seoul National University · U. Mass. Amherst · Microsoft