Can AI interview? Comparing AI-led to Human-led Chat Interviews
Authors: Semra Guven, Tobias Gårdhus, Andreas Bjerre-Nielsen, Anna Rogers, Hjalmar Bang Carlsen
Organizations: University of Copenhagen, Denmark · Center for Social Data Science, University of Copenhagen, Denmark · Data Science, IT University of Copenhagen, Denmark
Large language model chatbots have been proposed as AI-interviewers for collecting qualitative interview data at scale, potentially bridging social science's longstanding divide between the breadth of surveys and the depth of qualitative interviews. This paper provides the first direct empirical comparison of data quality in AI-led and human-led interviews. In a randomized experiment (N = 41), university students were interviewed via synchronous text chat by either trained non-expert interviewers or a locally hosted AI system; we evaluate exposure, specificity, and relevance. Median interview duration was similar across conditions, but AI-interviewers asked more than twice as many questions per interview and elicited roughly 65 percent more respondent text per interview. Per-question responses were longer with human interviewers, though not after multiple-testing correction; only AI-condition participants skipped questions. Responses were almost uniformly on-topic in both conditions; for specificity we detect no difference, but the design cannot exclude one of up to three quarters of a point. These results were obtained with a small, locally hosted open-weight model (7B parameters), which addresses concerns about data security, GDPR compliance, and scientific reproducibility. Our findings support the viability of AI-interviewing for large-N qualitative data collection.
Table 1: Overview of AI-Interviewer System Implementations
Figure 1: Proposed implementation of the AI-interviewer. The system is provided with the interview guide and a set of predefined main questions and suggested probes. Three agents — classification, reformulation, and probing — coordinate to generate contextually responsive follow-up questions while avoiding duplication.
Figure 2: Screenshot of the chat interface used for both human-led and AI-led interviews. The interface displayed conversation history in the upper portion of the screen; a resizable text box at the bottom was used by interviewees to type their responses. The send button was disabled until the next question appeared, prompting participants to submit their response in a single message.
Measure
Exact
Within 1
Krippendorff α
Cohen κw
Specificity
0.527
0.937
0.648
0.628
Relevance
0.934
0.980
0.365
0.411
Table 2: Inter-rater Reliability Between the Two Coders
Figure 3: Human- and AI-led interviews compared. Exposure: (A) mean characters per answer, (B) total characters per interview, (C) questions answered, (D) questions answered per minute, (E) characters per minute, (F) mean question formulation time. Data quality: (G) mean specificity and (H) mean relevance per interview (scale 1–5). Violin plots (A–E) show the kernel density with the median (white dot), interquartile range (thick bar) and 1.5 IQR whiskers (thin line); histograms (F–H) mark each group’s median with a dashed line. Labels give Holm-corrected Mann-Whitney U p -values; ns = not significant.