LLMs for Survey Text Analysis - A Performance Comparison Between Humans and GPT-5 on Inductive Content Analysis

Leonardo Bergmann; Renata Gheorghiu; Ana Gvritishvili; Alex Mican; Chris Stewart; Topias Tolonen-Weckström

cs.AIAug 23, 2026

LLMs for Survey Text Analysis - A Performance Comparison Between Humans and GPT-5 on Inductive Content Analysis

Authors: Leonardo Bergmann, Renata Gheorghiu, Ana Gvritishvili, Alex Mican, Chris Stewart, Topias Tolonen-Weckström

Organizations: Vienna Doctoral School in Cognition, Behavior and Neuroscience (VDS CoBeNe), University of Vienna, Austria · Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Austria · International Institute for Advanced Studies in Psychotherapy and Applied Mental Health, Babeș-Bolyai University, Cluj-Napoca, Romania · School of Education, Humanities and Social Sciences, International Black Sea University, Georgia ·

Abstract

Large language models (LLMs) are increasingly used to support text analysis in qualitative research, yet evidence on their performance in inductive content analysis remains limited. This study compares human and LLM-based inductive coding of open-ended survey responses from 903 answers across six variables from a European PhD student survey. Five human coders performed inductive content analysis following a standardized coding scheme, while an LLM (GPT-5.4) conducted the same task using an established prompting procedure. Agreement between human and LLM outputs was assessed using the Adjusted Rand Index (ARI). Results showed an alignment between humans and the LLM, with ARI values of 0.61 for coding and 0.54 for theme generation. These values were close to the internal consistency of coding and theme results within humans (ARI = 0.68) and the LLM (ARI = 0.76). Agreement varied widely across variables, with low within-entity consistency consistently linked to low between-entity agreement, underscoring the role of data characteristics and individual performance in reliability. Overall, the findings suggest that LLMs can approximate human coding in this case-specific setting, particularly at the coding level, and may serve as a scalable support tool for inductive qualitative analysis.

Human-AI Agreement

Human-AI agreement measures when AI judgments, decisions, and collaborative actions match human assessments and where they diverge.

See all 15 papers

So Many Opinions, So Many LLMs: Comparing Large Language Models to Traditional Machine Learning for Open- Ended Survey Analysis
May 4, 2026Abdullah Akinde, Mariam Akinde, Rasheedat Emiola +1LLM Evaluation LLM Interpretability
Austin Peay State University, United States
Human-LLM Collaborative Inductive Coding for Conceptualizing K-12 Educator AI Use
Jul 30, 2026Alex Liu, Min Sun, Lief Esbenshade +4LLM-Assisted Annotation Human-in-the-Loop Annotation
College of Education, University of Washington, Seattle, WA, United States · Colleague AI
Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth
Jul 30, 2026Alex Liu, Lief Esbenshade, Michael Xiao +4LLM-Assisted Annotation Human-in-the-Loop Evaluation
University of Washington · Colleague AI/ · Colleague AI

May 4, 2026J/K move · Enter open · S save

So Many Opinions, So Many LLMs: Comparing Large Language Models to Traditional Machine Learning for Open- Ended Survey Analysis

Abdullah Akinde, Mariam Akinde, Rasheedat Emiola, Ahmed Akinsola

Austin Peay State University, United States

LLM Evaluation LLM Interpretability Sentiment Analysis Text Classification

Open paper AskPDF arXiv

Open-ended surveys offer valuable insights, but they are notoriously difficult to analyze at scale. Building on previous work that employed traditional machine learning to classify text ("So Many Responses, So Little Time: A Machine-Learning Approach to Analyzing Open-Ended Survey Data") [1], this study investigates how different large language models (LLMs) understand and analyze NSSE open-ended survey responses. We focus on several cutting-edge LLMSs-OpenAI's GPT series, Twitter-roBERTa-base model, and Meta's LLaMA-and compare their performance to the previous machine learning models in tasks like sentiment analysis and thematic classification. Our research analysis assesses model agreement, classification accuracy, and interpretability of reasoning. The findings reveal that current LLMs routinely beat classic machine learning models in classification accuracy, particularly in understanding complex mood and theme patterns in student replies. While LLMs have superior accuracy, they differ greatly in how explicitly and consistently they justify their predictions and apply category boundaries. These distinctions highlight crucial trade-offs when using LLMs for qualitative analysis: increased predictive strength comes with issues in consistency and explainability. Our findings illustrate the benefits and drawbacks of utilizing various LLMs for large-scale qualitative research, and we provide practical advice for researchers looking to balance automation and interpretive rigor.

Abstract

Human-AI Agreement

Explore similar work

So Many Opinions, So Many LLMs: Comparing Large Language Models to Traditional Machine Learning for Open- Ended Survey Analysis

Human-LLM Collaborative Inductive Coding for Conceptualizing K-12 Educator AI Use

Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth