Real-world arguments in text and dialogues are normally enthymemes (i.e. some of their premises and/or claims are implicit). Natural language processing (NLP) methods for handling enthymemes can potentially identify enthymemes in text but they do not decode their underlying logic, whereas logic-based approaches for handling them assume a knowledgebase with sufficient formulae that can be used to decode them via abduction. There is therefore a lack of a systematic method for translating textual components of an enthymeme into a logical argument and generating the logical formulae required for their decoding, and thereby showing logical entailment. To address this, we propose a pipeline that integrates: (1) a large language model (LLM) to generate intermediate implicit premises based on the explicit premise and claim; (2) another LLM to translate the natural language into logical formulas; and (3) a neuro-symbolic reasoner based on a SAT solver to determine entailment. We evaluate our pipeline on two enthymeme datasets, demonstrating promising performance in selecting the correct implicit premise, as measured by precision, recall, F1-score, and accuracy.
Figure 1: AMR for the sentence “The boy wants to go.” (left) and “The boy does not want to go.” (right).
Figure 2: Our neuro-symbolic pipeline where input is a set of natural language sentences (e.g. premise, implicit premises and claim), and the output is a label.
ARCT
ANLI
Premise
They add a lot to the piece and I look forward to reading comments.
Premise
Jane was a professor teaching piano to students.
Claim
Comment sections have not failed.
Claim
Jane spent the morning sipping coffee and reading a book.
Helpful
Comments sections are a welcome distraction from my work.
Helpful
None of Jane’s students had a lesson that day.
Unhelp.
Comments sections always distract me from my work.
Unhelp.
Two of Jane’s students were early for their lessons.
Table 1: Examples from the ARCT and ANLI datasets
ANLI
ARCT
Class
Step type
F1
τm
τc
Class
Step type
F1
τm
τc
0
original
0.67
0.8
80
0
original
0.67
0.5
90
1-step
0.72
0.6
80
1-step
0.72
0.7
90
2-step
0.74
0.5
80
2-step
0.70
0.75
80
3-step
0.75
0.6
80
3-step
0.72
0.7
90
1
original
0.46
0.55
100
1
original
0.21
0.55
80
Table 2: Best F1-scores and thresholds by dataset, class (0 non-entailment, 1 entailment), and step type.
Dataset
none
original
1-step
2-step
3-step
ANLI
0.496
0.520
0.624
0.660
0.716
ARCT
0.276
0.292
0.480
0.496
0.560
Table 3: Entailment accuracy by dataset and step type. None means no implicit premise; original means the dataset premise; 1-, 2-, and 3-step mean LLM-generated intermediate premises.
Figure 4: Overall accuracy for ANLI and ARCT by step type and thresholds. Original uses dataset premises; 1-, 2-, and 3-step use LLM-generated premises.
Figure 5: A structured argument graph for decoding an enthymeme. The claim that a large insect escaped recently is supported by Implicit Premise A (torn webs result from trapped prey escaping), contradicted by Implicit Premise B (a small insect fled), and neutral with respect to Implicit Premise C (wind tears a spiderweb) and the premise alone. Arc labels show the neuro-matching relations used for support and the neuro-contradict relation used for contradiction. Thus, adding the AMR formulae for Implicit Premise A to those for the premise, together with the neuro-matching relations, entails the claim. Black arrows denote combining the explicit premise with an implicit premise.