Can AI interview? Comparing AI-led to Human-led Chat Interviews
Authors: Semra Guven, Tobias Gårdhus, Andreas Bjerre-Nielsen, Anna Rogers, Hjalmar Bang Carlsen
Organizations: University of Copenhagen, Denmark · Center for Social Data Science, University of Copenhagen, Denmark · Data Science, IT University of Copenhagen, Denmark
Large language model chatbots have been proposed as AI-interviewers for collecting qualitative interview data at scale, potentially bridging social science's longstanding divide between the breadth of surveys and the depth of qualitative interviews. This paper provides the first direct empirical comparison of data quality in AI-led and human-led interviews. In a randomized experiment (N = 41), university students were interviewed via synchronous text chat by either trained non-expert interviewers or a locally hosted AI system; we evaluate exposure, specificity, and relevance. Median interview duration was similar across conditions, but AI-interviewers asked more than twice as many questions per interview and elicited roughly 65 percent more respondent text per interview. Per-question responses were longer with human interviewers, though not after multiple-testing correction; only AI-condition participants skipped questions. Responses were almost uniformly on-topic in both conditions; for specificity we detect no difference, but the design cannot exclude one of up to three quarters of a point. These results were obtained with a small, locally hosted open-weight model (7B parameters), which addresses concerns about data security, GDPR compliance, and scientific reproducibility. Our findings support the viability of AI-interviewing for large-N qualitative data collection.
Table 1: Overview of AI-Interviewer System Implementations
Figure 1: Proposed implementation of the AI-interviewer. The system is provided with the interview guide and a set of predefined main questions and suggested probes. Three agents — classification, reformulation, and probing — coordinate to generate contextually responsive follow-up questions while avoiding duplication.
Figure 2: Screenshot of the chat interface used for both human-led and AI-led interviews. The interface displayed conversation history in the upper portion of the screen; a resizable text box at the bottom was used by interviewees to type their responses. The send button was disabled until the next question appeared, prompting participants to submit their response in a single message.
Measure
Exact
Within 1
Krippendorff α
Cohen κw
Specificity
0.527
0.937
0.648
0.628
Relevance
0.934
0.980
0.365
0.411
Table 2: Inter-rater Reliability Between the Two Coders
Figure 3: Human- and AI-led interviews compared. Exposure: (A) mean characters per answer, (B) total characters per interview, (C) questions answered, (D) questions answered per minute, (E) characters per minute, (F) mean question formulation time. Data quality: (G) mean specificity and (H) mean relevance per interview (scale 1–5). Violin plots (A–E) show the kernel density with the median (white dot), interquartile range (thick bar) and 1.5 IQR whiskers (thin line); histograms (F–H) mark each group’s median with a dashed line. Labels give Holm-corrected Mann-Whitney U p -values; ns = not significant.
There are now multiple proposals for systems based on Large Language Models (LLMs) to conduct automated qualitative interviews, but most of the current solutions rely on proprietary LLMs, which compromises reproducibility and data security. They also rely on LLMs for all interview tasks, which limits standardisation of question wording as well as control over question order. To address these issues, we introduce the AInterviewer platform, an opensource solution based on a multi-agent pipeline that combines controlled question administration of survey software with the flexibility of LLMs. AInterviewer is an interdisciplinary effort designed to implement best practices of qualitative interviewing in social science, and it can run with locally hosted models to ensure security, transparency, and reproducibility. Our platform provides a web-based GUI supporting each phase of data collection: from interview guide design and pilot testing to interview distribution and data collection monitoring.
Tobias Priesholm Gardhus, Nikolas Vitsakis, Fie Lejre Frederiksen +2
University of Copenhagen · IT University of Copenhagen
Increasingly, AI interviewers are being developed to elicit open-ended responses in applications like market research, public polling, preference elicitation, and social science research. However, evaluating AI interviewers is challenging because they function in extended, multi-turn interactions where they must adapt to participant behaviors. To address this need, we develop InterviewPlayground, a simulation environment for evaluating AI interviewers using simulated study participants whose behaviors are grounded in social theory. Simulated studies in InterviewPlayground produce an InterviewReportCard, which assesses the performance of AI interviewers using a suite of validated measures. To test whether our simulation-based evaluations predict performance with human participants, we conduct 15 real qualitative studies with five AI interviewers, three interview topics, and 450 human participants and compare them to simulated studies in InterviewPlayground. We find that AI interviewer performance in InterviewPlayground predicts performance in human studies with an average Pearson correlation of 0.86 across 12 measures, and the simulated interactions from InterviewPlayground reproduce key findings from behavioral analysis of AI interviewers in the human studies. Together, these findings support the validity of InterviewPlayground in assessing AI interviewer performance and examining potential failure modes. Our work contributes a simulation environment for AI interviewers supported with empirical validation, and more broadly, a roadmap for future work to develop validated, simulation-based evaluations of conversational AI systems.
AI-moderated interviews are emerging as a scalable market-research method for generating consumer insights and building consumer "digital twins." Yet it remains unclear whether they match human-moderated interviews or improve on simpler, static data collection methods. In a pre-registered, between-subjects study (N = 317) with three industry partners, we compare AI-moderated (N = 139), human-moderated (N = 24), and static interviews (N = 154). AI moderation matches human moderation in depth, covers more themes, and, holding budget constant, recovers significantly more customer needs than human moderation or static interviews. However, participants sound more emotionally engaged when speaking to a live human. We then create digital twins using interview data and evaluate each twin against the participant's own held-out responses to six real-world marketing stimuli. We find that digital twins created from AI-moderated interviews predict consumer responses better than demographics-only personas. However, the additional richness from AI moderation does not translate into better quantitative predictions compared to static interviews. By analyzing open-ended thoughts generated from humans versus their twins, we find that prediction errors are connected both to differences in (self-reported) thinking styles between twins and humans, and to gaps between training and validation data (i.e., asking questions that are too far out of distribution).
Yuting Deng, Jingxuan Liu, Olivier Toubia +1
Columbia Business School, Columbia University · The Wharton School, University of Pennsylvania