A comprehensive study of LLM-based argument classification: from Llama through DeepSeek to GPT-5.2

Marcin Pietroń; Filip Gampel; Jakub Gomułka; Andrzej Tomski; Rafał Olszowski

cs.CLFeb 25, 2026

A comprehensive study of LLM-based argument classification: from Llama through DeepSeek to GPT-5.2

Authors: Marcin Pietroń, Filip Gampel, Jakub Gomułka, Andrzej Tomski, Rafał Olszowski

Organizations: Faculty of Computer Science, Electronics, and Telecommunications AGH University of Krakow · Faculty of Humanities AGH University of Krakow · Institute of Mathematics University of Silesia

Abstract

Argument mining (AM) is an interdisciplinary research field focused on the automatic identification and classification of argumentative components, such as claims and premises, and the relationships between them. Recent advances in large language models (LLMs) have significantly improved the performance of argument classification compared to traditional machine learning approaches. This study presents a comprehensive evaluation of several state-of-the-art LLMs, including GPT-5.2, Llama 4, and DeepSeek, on large publicly available argument classification corpora such as Args.me and UKP. The evaluation incorporates advanced prompting strategies, including Chain-of- Thought prompting, prompt rephrasing, voting, and certainty-based classification. Both quantitative performance metrics and qualitative error analysis are conducted to assess model behavior. The best-performing model in the study (GPT-5.2) achieves a classification accuracy of 78.0% (UKP) and 91.9% (Args.me). The use of prompt rephrasing, multi-prompt voting, and certainty estimation further improves classification performance and robustness. These techniques increase the accuracy and F1 metric of the models by typically a few percentage points (from 2% to 8%). However, qualitative analysis reveals systematic failure modes shared across models, including instabilities with respect to prompt formulation, difficulties in detecting implicit criticism, interpreting complex argument structures, and aligning arguments with specific claims. This work contributes the first comprehensive evaluation that combines quantitative benchmarking and qualitative error analysis on multiple argument mining datasets using advanced LLM prompting strategies.

Figures & tables

Dataset	Full For	Full Against	Full NoArg	Full Total	Trim For	Trim Against	Trim NoArg	Trim Total
abortion	680	822	2427	3929	346	418	1236	2000
cloning	706	839	1494	3039	465	552	983	2000
death	457	1111	2083	3651	250	609	1141	2000
gun	787	665	1889	3341	471	398	1131	2000
marijuana	587	626	1262	2475	474	506	1020	2000
nuclear	606	852	2118	3576	339	476	1185	2000

Table 1: UKP class counts for full and trimmed datasets

Dataset	Full For	Full Against	Full Total	Trim For	Trim Against	Trim Total
debatepedia	15787	5406	21193	1490	510	2000
debatewise	8109	5642	13751	1179	821	2000
idebate	6445	6603	13048	988	1012	2000

Table 2: Args.me class counts for full and trimmed datasets

Figure 1: System architecture with voting strategy

Figure 2: Overview of prompt complexity used in the study.

Figure 3: Impact on accuracy due to prompt choice (calculations exclude Llama 3.1 1B)

Figure 4: Standard deviation from model average depending on dataset

Figure 5: Average Fleiss’ kappa by model for UKP and Args.me.

Figure 6: UKP aggregate Accuracy by model

Figure 7: Args.me aggregate Accuracy by model

Figure 8: P1–P4 average vs P5 (CoT) accuracy for UKP and Args.me.

Figure 9: Voting algorithm comparison (simple, tiebreak, weighted) for UKP and Args.me.

	UKP	Args.me
Simple	+4.4%	+2.6%
Tiebreak	+4.5%	+2.8%
Weighted	+2.4%	+2.9%

Table 3: Average improvement in percentage points over baseline accuracy by voting algorithm

Figure 10: Three-prompt vs four-prompt voting by model.

model	P2,3,4	P1,3,4	P1,2,4	P1,2,3	avg	P1,2,3,4
Llama 1b	30.7	47.2	47.6	32.5	39.5	42.2
Llama 3b	59.8	60.9	60.4	62.7	60.9	61.7
Llama 8b	70.3	70.5	68.9	72.9	70.6	71.4
Llama 3.1 70b	74.1	73.8	74.4	75.1	74.3	75.0
Llama 3.3 70b	75.1	74.4	74.7	74.7	74.7	75.6
Llama 4-Scout-70B	78.1	77.1	76.3	78.8	77.6	78.7

Table 4: Ablation study of the impact of presented prompts (UKP tiebreak accuracy).

Figure 11: Heatmap visualizing the performance of the seven top-performing LLM models across debate topics in the UKP and Args.me datasets.

	UKP (mRaR)	UKP (voting)	UKP (CoT)	Args.me (mRaR)	Args.me (voting)	Args.me (CoT)
Llama-3.1-1B	37.3 $\pm$ 0.5	42.2 $\pm$ 0.5	26.6 $\pm$ 0.5	41.6 $\pm$ 4.1	41.5 $\pm$ 4.1	41.0 $\pm$ 4.1
Llama-3.1-3B	53.5 $\pm$ 1.1	61.7 $\pm$ 1.1	55.4 $\pm$ 1.1	58.8 $\pm$ 1.5	61.4 $\pm$ 1.5	55.8 $\pm$ 1.5
Llama-3.1-8B	65.4 $\pm$ 1.4	71.4 $\pm$ 1.4	65.4 $\pm$ 1.4	69.6 $\pm$ 1.9	73.2 $\pm$ 1.9	65.2 $\pm$ 1.9
Llama-3.1-70B	70.9 $\pm$ 1.8	75.0 $\pm$ 1.8	75.3 $\pm$ 1.8	80.0 $\pm$ 2.1	85.7 $\pm$ 2.1	75.8 $\pm$ 2.1
Llama-3.3-70B	72.5 $\pm$ 1.8	75.6 $\pm$ 1.8	76.8 $\pm$ 1.8	80.8 $\pm$ 2.1	86.3 $\pm$ 2.1	76.6 $\pm$ 2.1
Llama-4-Scout-70B	73.1 $\pm$ 1.4	78.7 $\pm$ 1.4	76.9 $\pm$ 1.4	79.6 $\pm$ 1.8	85.2 $\pm$ 1.8	74.2 $\pm$ 1.8

Table 5: Average model Accuracy in argument classification on the UKP and Args.me datasets

	UKP (mRaR)	UKP (voting)	UKP (CoT)	Args.me (mRaR)	Args.me (voting)	Args.me (CoT)
Llama-3.1-1B	25.2 $\pm$ 1.4	35.7 $\pm$ 1.4	22.7 $\pm$ 1.4	27.8 $\pm$ 2.1	35.7 $\pm$ 2.1	30.9 $\pm$ 2.1
Llama-3.1-3B	37.5 $\pm$ 2.0	58.1 $\pm$ 2.0	38.1 $\pm$ 2.0	37.9 $\pm$ 0.5	59.1 $\pm$ 0.5	38.1 $\pm$ 0.5
Llama-3.1-8B	46.2 $\pm$ 1.4	67.2 $\pm$ 1.4	46.6 $\pm$ 1.4	45.4 $\pm$ 1.0	71.7 $\pm$ 1.0	44.0 $\pm$ 1.0
Llama-3.1-70B	52.5 $\pm$ 1.5	74.1 $\pm$ 1.5	74.2 $\pm$ 1.5	52.9 $\pm$ 1.3	84.6 $\pm$ 1.3	50.4 $\pm$ 1.3
Llama-3.3-70B	53.7 $\pm$ 1.4	74.7 $\pm$ 1.4	75.5 $\pm$ 1.4	53.5 $\pm$ 1.3	85.3 $\pm$ 1.3	76.4 $\pm$ 1.3
Llama-4-Scout-70B	52.7 $\pm$ 3.8	76.0 $\pm$ 3.8	73.0 $\pm$ 3.8	52.6 $\pm$ 1.4	83.9 $\pm$ 1.4	49.4 $\pm$ 1.4

Table 6: Average model F1 in argument classification on the UKP and Args.me datasets

	UKP (F1)	Args.me (Acc)
GPT-5.2	77.3	91.9
gpt-oss-120b	74.6	88.5
gpt-oss-20b	73.1	86.2
ds70b	74.6	84.4
Llama 3.3 70b	71.6	80.8
Llama 3.1 8b	61.6	69.6

Table 7: Accuracy comparison across different argument mining studies

Figure 12: Error types in UKP by prompt. Percentages are given with respect to all predicted arguments, including correct ones.

Figure 13: Error types in UKP by dataset. Percentages are given with respect to all predicted arguments, including correct ones.

Figure 14: Error types in UKP by model. Percentages are given with respect to all predicted arguments, including correct ones.

Topic	Dominant error type	Percentage of errors within topic
Death penalty	NA	43.2%
Marijuana legalization	NF	33.0%
Abortion	NA	32.3%
Gun access limitation	NA	32.3%
Cloning	NA	31.2%
Minimum wage	NF	30.3%

Table 8: Dominant error type by topic and its percentage share

Model	Major misclassification categories	Model-specific characteristics
oss120b	• Contrastive discourse structures • Literal interpretation of facts • Complex argumentation • Implicit criticism	More vulnerable to literal interpretation and failure to recognize argumentative intent. This model more frequently misclassifies statements containing numbers, statistics, and research findings as purely neutral or informational content. It performs worse in detecting irony, rhetorical questions, and implicit criticism.
gpt-5.2	• Contrastive discourse structures • Complex argumentation • Misinterpretation of argument structure	High resistance to literal interpretation, but occasional difficulty in processing argument structure. This model performs best in inferring argumentative intent from statistical evidence and rarely misclassifies factual arguments. However, its main limitation involves multi-faceted and contrastive sentences, where it may fail to identify the main conclusion. It may occasionally overinterpret logical structure markers.
ds70b	• Contrastive discourse structures • Literal interpretation of facts • Complex argumentation • Incorporation of external context • Emotional language misinterpreted as argument	More susceptible to emotional and informal language. ds70b demonstrates better recognition of argumentative intent in statistical statements than gpt-oss120b, but performs worse than GPT-5.2. It may misinterpret emotional expressions as indicative of argumentative stance and exhibits difficulties in correctly handling references to external discourse context.

Table 9: Model-specific characteristics in argument misclassification

Figure 15: Gold re-annotation sample breakdown (n=442).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Developer	Release date	Huggingface code	License	Code
Llama 3.2 1B	Meta	Sep 2024	meta-llama/Llama-3.2-1B-Instruct	open	1b
Llama 3.2 3B	Meta	Sep 2024	meta-llama/Llama-3.2-3B-Instruct	open	3b
Llama 3.1 8B	Meta	Jul 2024	meta-llama/Llama-3.1-8B-Instruct	open	8b
Llama 3.1 70B	Meta	Jul 2024	meta-llama/Llama-3.1-70B-Instruct	open	31_70b
Llama 3.3 70B	Meta	Dec 2024	meta-llama/Llama-3.3-70B-Instruct	open	33_70b
Llama 4 Scout	Meta	Apr 2025	meta-llama/Llama-4-Scout-17B-16E-Instruct	open	scout

Appendix

Table 25

No.	Text
1	Is the sentence: " {sentence} " an argument for or against {thesis} , or is it no argument? Return one of the expressions: “For”, “Against” or “No argument”, without any additional commentary.
2	The thesis is: " {thesis} " Indicate if the argument " {sentence} " is for this thesis (F), against this thesis (A), or neutral (N). Please respond with only one letter: F, A, or N, without any additional commentary.
3	In the context of the ongoing public debate, evaluate whether the text " {sentence} " represents an argument supporting or opposing " {thesis} ", or whether it does not qualify as an argument at all. Respond with one of the expressions: “For”, “Against” or “No Argument”.
4	Is the sentence: " {sentence} " an argument for (F) or against (A) {thesis} , or is it no argument (N)? Return a single letter: F, A, or N, without any additional commentary.

Appendix

Table 26

Dataset	Short {thesis}	Long {thesis}
Abortion	abortion	Abortion should be fully accessible.
Cloning	cloning	Cloning should be allowed.
Death penalty	death penalty	The death penalty should be allowed.
Marijuana	legalisation of marijuana	Marijuana should be legal.
Gun laws	stricter gun laws	Gun access should be limited.
Minimum wage	minimum wage	The minimum wage is justified and should be increased.

Appendix

Table 27

No.	Text
1	Is the sentence: " {sentence} " an argument for or against " {thesis} "? Return one of the expressions: "For" or "Against", without any additional commentary.
2	The thesis is: " {thesis} " Indicate if the argument " {sentence} " is for this thesis (F) or against this thesis (A). Please respond with only one letter: F or A, without any additional commentary.
3	In the context of the ongoing public debate, evaluate whether the text " {sentence} " represents an argument supporting or opposing " {thesis} ". Respond with one of the expressions: "For" or "Against".
4	Is the sentence: " {sentence} " an argument for (F) or against (A) {thesis} ? Return a single letter: F or A, without any additional commentary.

Appendix

Table 28

Corpus	CoT Prompt
UKP	Is the sentence: " {sentence} " an argument for or against {thesis} or is it no argument? Solve the argument classification problem. Think through the problem step by step to solve it. Then output one final line exactly in the format: FINAL: <For\|Against\|No Argument> Do not include anything after the FINAL line.
Args.me	Is the sentence: " {sentence} " an argument for or against {thesis} ? Solve the argument classification problem. Think through the problem step by step to solve it. Then output one final line exactly in the format: FINAL: <For\|Against> Do not include anything after the FINAL line.

Appendix

Table 29

Corpus	Prompt No.	Pattern
UKP	1, 3

Appendix

Table 30

Argument Mining

Argument mining identifies claims, premises, and their relations in text to reveal how people construct and contest arguments.

See all 39 papers

Multi-Agent Dialectical Refinement for Enhanced Argument Classification
Mar 29, 2026Jakub Bąba, Jarosław A. ChudziakArgument Mining Computational Argumentation
Faculty of Electronics and Information Technology, Warsaw University of Technology, Poland
An LLM-Based System for Argument Mining
May 13, 2026Paulo Pirozelli, Victor Hugo Nascimento Rocha, Fabio G. Cozman +1Argument Mining Computational Argumentation
Universidade de S˜ao Paulo · Center for Artificial Intelligence (C4AI) · Instituto Mau´a de Tecnologia +1
From Argument Components to Graphs: A Multi-Agent Debate with Confidence Gating for Argument Relations
Jun 14, 2026Jakub Bąba, Jarosław A. ChudziakRelation Extraction Argument Mining
Faculty of Electronics and Information Technology, Warsaw University of Technology, Poland

Mar 29, 2026J/K move · Enter open · S save

Multi-Agent Dialectical Refinement for Enhanced Argument Classification

Jakub Bąba, Jarosław A. Chudziak

Faculty of Electronics and Information Technology, Warsaw University of Technology, Poland

Argument Mining Computational Argumentation Multi-Agent Debate

Open paper AskPDF arXiv

Argument Mining (AM) is a foundational technology for automated writing evaluation, yet traditional supervised approaches rely heavily on expensive, domain-specific fine-tuning. While Large Language Models (LLMs) offer a training-free alternative, they often struggle with structural ambiguity, failing to distinguish between similar components like Claims and Premises. Furthermore, single-agent self-correction mechanisms often suffer from sycophancy, where the model reinforces its own initial errors rather than critically evaluating them. We introduce MAD-ACC (Multi-Agent Debate for Argument Component Classification), a framework that leverages dialectical refinement to resolve classification uncertainty. MAD-ACC utilizes a Proponent-Opponent-Judge model where agents defend conflicting interpretations of ambiguous text, exposing logical nuances that single-agent models miss. Evaluation on the UKP Student Essays corpus demonstrates that MAD-ACC achieves a Macro F1 score of 85.7%, significantly outperforming single-agent reasoning baselines, without requiring domain-specific training. Additionally, unlike "black-box" classifiers, MAD-ACC's dialectical approach offers a transparent and explainable alternative by generating human-readable debate transcripts that explain the reasoning behind decisions.

Abstract

Argument Mining

Explore similar work

Multi-Agent Dialectical Refinement for Enhanced Argument Classification

An LLM-Based System for Argument Mining

From Argument Components to Graphs: A Multi-Agent Debate with Confidence Gating for Argument Relations