LLMs for Qualitative Data Analysis Fail on Security-specificComments in Human Experiments

Maria Camporese; Fabio Massacci; Yuanjun Gong

cs.SEApr 12, 2026

LLMs for Qualitative Data Analysis Fail on Security-specificComments in Human Experiments

Authors: Maria Camporese, Fabio Massacci, Yuanjun Gong

Organizations: University of Trento (Italy) · Vrije Universiteit Amsterdam (The Netherlands)

Abstract

[Background:] Thematic analysis of free-text justifications in human experiments provides significant qualitative insights. Yet, it is costly because reliable annotations require multiple domain experts. Large language models (LLMs) seem ideal candidates to replace human annotators. [Problem:] Coding security-specific aspects (code identifiers mentioned, lines-of-code mentioned, security keywords mentioned) may require deeper contextual understanding than sentiment classification. [Objective:] Explore whether LLMs can act as automated annotators for technical security comments by human subjects. [Method:] We prompt four top-performing LLMs on LiveBench to detect nine security-relevant codes in free-text comments by human subjects analyzing vulnerable code snippets. Outputs are compared to human annotators using Cohen's Kappa (chance-corrected accuracy). We test different prompts mimicking annotation best practices, including emerging codes, detailed codebooks with examples, and conflicting examples. [Negative Results:] We observed marked improvements only when using detailed code descriptions; however, these improvements are not uniform across codes and are insufficient to reliably replace a human annotator. [Limitations:] Additional studies with more LLMs and annotation tasks are needed.

Figures & tables

Figure 2

Figure 3

Figure 4

Figure 5

Figure 1. Annotation by human annotators and LLMs This figure shows the four stages of the human annotation process.

Abbr.	Code Definition	Code applicable Comments	Code inapplicable Comments
Var	Variable/Method identifiers are mentioned	I think it had something to do with inv and debug.	header buffer capacity is not checked.
Lin	Lines number mentioned	In this line, the memory at constant address is copied, this memory will most likely not stay consistent among systems.	The null value may cause the function crash.
Key	Relevant Keywords mentioned	If the file attribute in the code is tampered with for an instance of the BlockDriverState it might happen that the code runs an infinite recursion, essentially stopping the machine.	I don’t see any vulnerabilities.
Vul	Text is related to the specific vulnerability	Might modify environment variable with strtol.	I think that negative checkin triggers the vulnerability for the ML, not sure.
Sec	Text related to security	A check should be added which checks if the buffer is not overflowing.	the datatype of debug wont support strtol.
Exp	Potential exploit is mentioned	The memcpy() function might be injected with a size that is too big for the program to handle, therefore freezing the computer’s resources.	stderr is not created.

Table 1. Example of Codes and Related Comments

Panel of Figure 2 — Figure 2. The pilot interaction where $\texttt{P}_{3pilot}$ is constructed

Panel of Figure 3 — Figure 3. The defined prompt $\texttt{P}_{3+}$ based on $\texttt{P}_{3pilot}$

$\kappa(Acc^{*})$	Replaceability and Rationale
$<0$	Hallucinate (✗) - The LLM has worse performance than random guessing in controlled experiments. It is not recommended to use it.
$[0.0,0.2]$	Unusable (★) - The LLM will rarely generate the expected annotations and will almost always disagree with the human annotator.
$(0.2,0.4]$	Limited (★★), The LLM only partly generates the annotations of a human coder. It might not converge when resolving conflicts with other annotators.
$(0.4,0.6]$	Moderate (★★★), The LLM is not able to generate human annotation for around half the time. It might or might not converge when discussing conflicts.
$(0.6,0.8]$	Substantial (★★★★) - The LLM will generate codes close to a human annotator in practice. Multiple interactions over conflicts are likely to converge.
$(0.8,1.0]$	Almost Perfect (★★★★★) - The LLM would generate almost the same annotations as human coders, or at least will disagree with them only in a few cases.

Table 2. Proposed Thresholds for Human Replaceability

LLM	Precision	Recall	Acc	$\kappa/Acc^{*}$	Replace
GPT-5	0.62	0.47	0.74	0.26	★★
Claude-4	0.57	0.51	0.74	0.30	★★
DeepSeek-V3.2	0.79	0.64	0.83	0.53	★★★
Qwen3-Max	0.74	0.73	0.86	0.61	★★★★

Table 3. Average Results with the Final Prompt

		Human*		GPT-5		Claude-4		DeepSeek-V3.2		Qwen3-Max
	$\hat{\kappa}$	Agreement	$\hat{\kappa}$	Replace	$\hat{\kappa}$	Replace	$\hat{\kappa}$	Replace	$\hat{\kappa}$	Replace
Var	0.98	★★★★★	0.00	★	0.52	★★★	0.79	★★★★	0.79	★★★★
Lin	0.94	★★★★★	0.57	★★★	0.46	★★★	0.31	★★	0.42	★★★
Key	0.98	★★★★★	0.30	★★	0.14	★	0.34	★★	0.56	★★★
Vul	0.98	★★★★★	0.50	★★★	0.44	★★★	0.39	★★	0.50	★★★
Sec	0.98	★★★★★	0.11	★	0.15	★	0.50	★★★	0.66	★★★★

Table 4. The Cohen’s

\kappa

of the human annotator/LLM’s agreement with the reviewers across codes

Prompt	Precision	Recall	Acc	$\kappa(Acc^{*})$	Replaceability
$\texttt{P}_{1}$	0.63	0.51	0.78	0.37	★★
$\texttt{P}_{2}$	0.67	0.56	0.79	0.42	★★★
$\texttt{P}_{3}$	0.59	0.54	0.76	0.35	★★
$\texttt{P}_{3+}$	0.68	0.58	0.79	0.42	★★★

Table 5. Cohen’s

\kappa

(Chance Corrected Accuracy) per Prompt

Contrast		Wilcoxon’s $W$	$p$ -value
$\texttt{P}_{1}$	$\texttt{P}_{2}$	181	0.037
$\texttt{P}_{2}$	$\texttt{P}_{3}$	292	0.890
$\texttt{P}_{3}$	$\texttt{P}_{3+}$	30	0.004

Table 6. Wilcoxon paired tests between increasingly sophisticated prompts

LLM	Contrast		Wilcoxon’s $W$	$p$ -value
GPT-5	$\texttt{P}_{1}$	$\texttt{P}_{2}$	3	0.04
	$\texttt{P}_{2}$	$\texttt{P}_{3}$	45	1.0
	$\texttt{P}_{3}$	$\texttt{P}_{3+}$	0	0.004
Claude-4	$\texttt{P}_{1}$	$\texttt{P}_{2}$	9	0.06
	$\texttt{P}_{2}$	$\texttt{P}_{3}$	31	0.85
	$\texttt{P}_{3}$	$\texttt{P}_{3+}$	16	0.42

Table 7. Wilcoxon paired tests across LLMs and prompts

Panel of Figure 4 — Figure 4. Average Cohen’s $\kappa$ (Chance Corrected Accuracy if human were ground truth) vs Effort (Size of Prompts)

Panel of Figure 5 — Figure 5. Chance corrected accuracy against the required effort in tokens, with each curve corresponding to a different code.

LLM-Assisted Annotation

Large language model-assisted annotation uses AI to label, classify, and structure data, often with human review to improve scale and consistency.

See all 197 papers

Enhancing Reliability in LLM-Based Secure Code Generation
May 22, 2026Mohammed F. Kharma, Mohammad Alkhanafseh, Ahmed Sabbah +1LLM Evaluation CoT Reasoning
the Department of Computer Science, Birzeit University, Ramallah, Palestine · the Department of Computer Science, University of Central Florida, Orlando, FL 32816 USA
Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios
Jul 25, 2026Lixun Ma, Ruolong Ma, Bei Wang +4LLM Safety Benchmarks LLM Security
Zhejiang University, Hangzhou, China · University College London, London, United Kingdom · Artificial Intelligence Institute, CAICT, Beijing, China +1
Observational Equivalence of LLM and Human Annotation
Aug 24, 2026Kentaro Nakamura, Jing Ling Tan, George YeanLLM-Assisted Annotation Annotator Disagreement
Ph.D. student at John F. Kennedy School of Government, Harvard University · Ph.D. student at Department of Government, Harvard University

May 22, 2026J/K move · Enter open · S save

Enhancing Reliability in LLM-Based Secure Code Generation

Mohammed F. Kharma, Mohammad Alkhanafseh, Ahmed Sabbah, David Mohaisen

the Department of Computer Science, Birzeit University, Ramallah, Palestine · the Department of Computer Science, University of Central Florida, Orlando, FL 32816 USA

LLM Evaluation CoT Reasoning LLM Security Secure Code Generation Prompt Engineering

Open paper AskPDF arXiv

Large language models (LLMs) are widely used for code generation, but their security reliability remains inconsistent across languages and prompting strategies. Existing prompt engineering improves functional correctness but rarely ensures consistent security outcomes. We introduce the Mitigation-Aware Chain-of-Thought (MA-CoT) framework, which embeds task-specific CWE mitigation guidance and language-aware safeguards to reduce recurring vulnerabilities in generated code. We evaluate MA-CoT across three LLMs (gpt-5, claude-4.5, gemini-2.5), three programming languages (C, Java, Python), and four prompting strategies (Vanilla, Zero-shot, CoT, MA-CoT) on a 200-task primary dataset, with external validation on LLMSecEval. Using static analysis with expert validation, MA-CoT reduces total security findings from 92 to 39 (57.6%) on the primary dataset and from 73 to 4 (94.5%) on LLMSecEval. High-severity findings (Blocker + Critical) drop from 90 to 39 (56.7%) and from 45 to 2 (95.6%), respectively. Across both datasets, MA-CoT is the only strategy that consistently improves security reliability; Zero-shot and CoT are less reliable and may increase vulnerability, especially in C. We further introduce a strict layered attribution of vulnerability drivers (language-core vs. stack layers) and show that residual risk concentrates in hardening-oriented patterns (e.g., OS- and toolchain-dependent), motivating secure-by-construction primitives alongside prompting.

Abstract

LLM-Assisted Annotation

Explore similar work

Enhancing Reliability in LLM-Based Secure Code Generation

Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

Observational Equivalence of LLM and Human Annotation