Incorporating argumentation frameworks for LLM decision-making: Task-specific design and evaluation
Organizations: State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, CHINA · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100190, CHINA
Abstract
Argumentation frameworks (AFs) provide explicit, formal mechanisms for representing and resolving conflicts and thus have been incorporated into large language model (LLM) decision-making. The benefits vary and depend on how arguments and their relations are constructed, evaluated, and incorporated into the decision process. {We abstract a workflow consisting of four stages based on existing argumentation-augmented methods} and investigate their design choices over tasks of debate-based decision-making and retrieval-based fact verification. Our evaluation with recent LLMs shows that existing methods do not consistently improve performance across the considered tasks. Motivated by this observation, we develop two task-specific methods. The first is called Qnt-Arg, which organizes arguments around the original speakers in debate materials and aggregates model-estimated argument strengths and candidate-answer priors under gradual semantics. The second is Qlt-Arg that uses credibility estimates to establish preferences between conflicting evidence items and applies grounded semantics to select acceptable evidence for subsequent LLM inference. We examine the roles of generative and decision models within the two methods. In the debate setting, Qnt-Arg consistently outperforms other argumentation-augmented methods across matched configurations; Qlt-Arg improves retrieval-based fact-verification performance compared with the baselines. These findings demonstrate the value of combining formal argumentation and LLM while highlighting the importance of aligning with tasks in argumentation construction, model selection, and the role of formal semantic reasoning.
Figures & tables
| Candidate-answer prior | Separate argument and relation assessments | Overall (%) | (pp) |
|---|---|---|---|
| 78.41 | |||
| — | 74.75 | ||
| — | 76.72 | ||
| — | — | 72.37 |
| Method | CONFACT | AVeriTeC | DeepFact-Bench | |||
|---|---|---|---|---|---|---|
| Top-5 | Top-10 | Top-5 | Top-10 | Top-5 | Top-10 | |
| Direct LLM | 78.56 | 78.56 | 68.00 | 68.00 | 84.00 | 84.00 |
| Direct Decision Model | 65.30 | 65.30 | 72.00 | 72.00 | 83.00 | 83.00 |
| RAG LLM | 76.92 | 78.23 | 76.50 | 78.00 | 84.42 | 83.00 |
| RAG Decision Model | 75.94 | 76.27 | 79.50 | 80.00 | 86.50 | 86.00 |
| ArgRAG | 68.74 | 68.69 | 63.50 | 62.31 | 81.00 | 79.50 |
| Credibility scores | Acceptance annotations | Accuracy (%) | (pp) |
| 80.33 | |||
| — | 76.72 | ||
| — | 78.52 | ||
| — | — | 78.23 |
LLM Decision-Making
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
See all 279 papers