Incorporating argumentation frameworks for LLM decision-making: Task-specific design and evaluation
Authors: Bochen Han, Anshu Xiong, Mingyang Ma
Organizations: State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, CHINA · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100190, CHINA
Argumentation frameworks (AFs) provide explicit, formal mechanisms for representing and resolving conflicts and thus have been incorporated into large language model (LLM) decision-making. The benefits vary and depend on how arguments and their relations are constructed, evaluated, and incorporated into the decision process. {We abstract a workflow consisting of four stages based on existing argumentation-augmented methods} and investigate their design choices over tasks of debate-based decision-making and retrieval-based fact verification. Our evaluation with recent LLMs shows that existing methods do not consistently improve performance across the considered tasks. Motivated by this observation, we develop two task-specific methods. The first is called Qnt-Arg, which organizes arguments around the original speakers in debate materials and aggregates model-estimated argument strengths and candidate-answer priors under gradual semantics. The second is Qlt-Arg that uses credibility estimates to establish preferences between conflicting evidence items and applies grounded semantics to select acceptable evidence for subsequent LLM inference. We examine the roles of generative and decision models within the two methods. In the debate setting, Qnt-Arg consistently outperforms other argumentation-augmented methods across matched configurations; Qlt-Arg improves retrieval-based fact-verification performance compared with the baselines. These findings demonstrate the value of combining formal argumentation and LLM while highlighting the importance of aligning with tasks in argumentation construction, model selection, and the role of formal semantic reasoning.
Figures & tables
Figure 1: The four-stage workflow and the design choices of various methods.
Figure 2: The main Qnt-Arg and Qlt-Arg instantiations of the general workflow.
Table 1: Accuracy (%) comparison of all baselines and Qnt-Arg on six debate datasets. Overall accuracy pools all 2,186 binary instances from the six datasets.
Candidate-answer prior
Separate argument and relation assessments
Overall (%)
Δ (pp)
✓
✓
78.41
0.00
—
✓
74.75
−3.66
✓
—
76.72
−1.69
—
—
72.37
−6.04
Table 2: Qnt-Arg ablations on debate tasks at D=2 . Δ is the reported change in percentage points(pp) from the complete configuration.
Figure 3: Accuracy changes from Direct Decision Model (Perplexity Decider) to Qnt-Arg at D=2 , grouped by initial confidence conf=max(β(r),1−β(r)) . Bars show corrections and new errors as percentages of each bin; dots show their net difference.
Method
CONFACT
AVeriTeC
DeepFact-Bench
Top-5
Top-10
Top-5
Top-10
Top-5
Top-10
Direct LLM
78.56
78.56
68.00
68.00
84.00
84.00
Direct Decision Model
65.30
65.30
72.00
72.00
83.00
83.00
RAG LLM
76.92
78.23
76.50
78.00
84.42
83.00
RAG Decision Model
75.94
76.27
79.50
80.00
86.50
86.00
ArgRAG
68.74
68.69
63.50
62.31
81.00
79.50
Table 3: Accuracy (%) on CONFACT, AVeriTeC, and DeepFact-Bench with Top-5 and Top-10 retrieved evidence.
Figure 4: Case studies of argumentation-augmented decision-making. (a) Qnt-Arg corrects an IQ2 outcome prediction using the EW-QBAF. (b) Qlt-Arg supplies credibility scores and grounded acceptance annotations with the same five CONFACT passages.
Credibility scores
Acceptance annotations
Accuracy (%)
Δ (pp)
✓
✓
80.33
0.00
—
✓
76.72
−3.61
✓
—
78.52
−1.80
—
—
78.23
−2.10
Table 4: Qlt-Arg ablations without GE filtering on CONFACT with Top-10 evidence. Δ is the reported change in percentage points from the complete configuration.
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
Large Language Models (LLMs) achieve strong performance in analyzing and generating text, yet they struggle with explicit, transparent, and verifiable reasoning over complex texts such as those containing debates. In particular, they lack structured representations that capture how arguments support or attack each other and how their relative strengths determine overall acceptability. We encompass these limitations by proposing a framework that integrates learning-based argument mining with quantitative reasoning and ontology-based querying. Starting from a raw debate text, the framework extracts a fuzzy argumentative knowledge base, where arguments are explicitly represented as entities, linked by attack and support relations, and annotated with initial fuzzy strengths reflecting plausibility w.r.t. the debate's context. Quantitative argumentation semantics are then applied to compute final argument strengths by propagating the effects of supports and attacks. These results are then embedded into a fuzzy description logic setting, enabling expressive query answering through efficient rewriting techniques. The proposed approach provides a transparent, explainable, and formally grounded method for analyzing debates, overcoming purely statistical LLM-based analyses.
Gianvincenzo Alfano, Sergio Greco, Lucio La Cava +2
Department of Informatics, Modeling, Electronics and System Engineering, University of Calabria, Italy
Argumentative LLMs (ArgLLMs) are an existing approach leveraging Large Language Models (LLMs) and computational argumentation for decision-making, with the aim of making the resulting decisions faithfully explainable to and contestable by humans. Here we propose a web-based system implementing ArgLLM-empowered agents for binary tasks. ArgLLM-App supports visualisation of the produced explanations and interaction with human users, allowing them to identify and contest any mistakes in the system's reasoning. It is highly modular and enables drawing information from trusted external sources. ArgLLM-App is publicly available at https://argllm.app, with a video demonstration at https://youtu.be/vzwlGOr0sPM.
Argumentation skills are an essential toolkit for large language models (LLMs). These skills are crucial in various use cases, including self-reflection, debating collaboratively for diverse answers, and countering hate speech. In this paper, we create the first benchmark for a standardized evaluation of LLM-based approaches to computational argumentation, encompassing 33 datasets from previous work in unified form. Using the benchmark, we evaluate the generalizability of five LLM families across 46 computational argumentation tasks that cover mining arguments, assessing perspectives, assessing argument quality, reasoning about arguments, and generating arguments. On the benchmark, we conduct an extensive systematic analysis of the contribution of few-shot examples, reasoning steps, model size, and training skills to the performance of LLMs on the computational argumentation tasks in the benchmark.
Yamen Ajjour, Carlotta Quensel, Nedim Lipka +1
Leibniz University Hannover · Adobe Research · Leibniz University Hannover, L3S