cs.HCFeb 24, 2026

SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery

Authors: David Anugraha, Vishakh Padmakumar, Diyi Yang

Organizations: Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Qualitative insights from user experiences are critical for informing product and policy decisions, but collecting such data at scale is constrained by the time and availability of experts to conduct semi-structured interviews. Recent work has explored using large language models (LLMs) to automate interviewing, yet existing systems lack a principled mechanism for balancing systematic coverage of predefined topics with adaptive exploration, or the ability to pursue follow-ups, deep dives, and emergent themes that arise organically during conversation. In this work, we formulate adaptive semi-structured interviewing as an optimization problem over the interviewer's behavior. We define interview utility as a trade-off between coverage of a predefined interview topic guide, discovery of relevant emergent themes, and interview cost measured by length. Based on this formulation, we introduce SparkMe, a multi-agent LLM interviewer that performs deliberative planning via simulated conversation rollouts to select questions with high expected utility. We evaluate SparkMe through controlled experiments with LLM-based interviewees, showing that it achieves higher interview utility, improving topic guide coverage (+4.7% over the best baseline) and eliciting richer emergent insights while using fewer conversational turns than prior LLM interviewing approaches. We further validate SparkMe in a user study with 70 participants across 7 professions on the impact of AI on their workflows. Domain experts rate SparkMe as producing high-quality adaptive interviews that surface helpful profession-specific insights not captured by prior approaches. The code, datasets, and evaluation protocols for SparkMe are available as open-source at https://github.com/SALT-NLP/SparkMe.

Figures & tables

Figure 1: Collecting qualitative data at scale with LLM-based systems requires balancing coverage of a predefined topic guide with exploration of emergent conversational themes without unnecessarily burdening the interviewee. We formalize this trade-off as a tractable utility function that guides the design of interviewer agents ( Section 2 ). Motivated by the lack of explicit mechanisms for emergence in prior systems, we design SparkMe to prioritize both coverage and emergence ( Section 3 ) by periodically simulating conversation rollouts and selecting directions with high expected utility gain.
System NameInterview StructureStopping ConditionCoverage- awareDeliberative ExplorationOpen Source
AI Recruiter ( Pathak and Pandey, 2025 )Semi-structuredUser-driven✓--
Persona ChatBot ( Wei et al., 2024 )UnstructuredUser-driven---
VirtualInterviewer ( Gomez et al., 2025 )Semi-structuredScript-based✓--
Anthropic Interviewer ( Handa et al., 2025 )Semi-structuredScript-based✓--
Interview-GPT ( Wuttke et al., 2025 )UnstructuredUser-driven--✓
MimiTalk ( Liu and Yu, 2025 )Semi-structuredUser-driven✓-✓
Table 1: Comparison of LLM-based interview systems. Systems vary in interview structure, stopping condition (user-, script-, or utility-driven), coverage awareness, and design mechanisms to support deliberative exploration. Unlike prior approaches, SparkMe jointly balances coverage of the topic guide with exploration beyond it while minimizing interview cost.
Figure 2: ExplorationPlanner (EP) runs asynchronously every kk turns, simulating multiple conversation rollouts and scoring them by expected utility gain to propose conversation directions for prioritization.
Figure 3: Predefined subtopic coverage and utility of subtopics ( yy -axis) as a function of the number of interview turns ( xx -axis) for different systems ( Section 4.1.3 ), using Qwen3-30B-A3B-Instruct-2507 as the backbone for the interviewer LLM and user agent. SparkMe (Ours) efficiently converges to a higher coverage and utility value, consistently outperforming other baselines ( Section 4.1.5 ).
Figure 4: The first two panels show the Flesch-Kincaid Grade Level and Flesch Reading Ease of the questions in the automated evaluation. The last three panels show local coherence, transition quality, and follow-up contingency of the interview questions, rated on a 1–-5 Likert scale, to evaluate the overall quality and flow of the interviews.
Figure 5: Number of emergent subtopics identified and covered between with and without EP. The number of covered emergent subtopics for other baselines is not shown since they have 0 covered emergent subtopics according to our evaluation setup.
Figure 6: The left conversation illustrates a high-utility simulated rollout generated during planning to prioritize emergence, while the right conversation shows the resulting real interaction. The EP selects the next question to explore stake-based triage of AI use , which is identified as a high-potential emergent subtopic based on partial coverage and estimated emergence value.
Figure 7: Overall human evaluation of SparkMe and MimiTalk across three dimensions (5-point Likert scale). Content quality (blue) is evaluated based on coverage, depth, and correctness of information. Interaction quality (green) is evaluated based on clarity, adaptiveness, and conversational comfort. Emergent insights (red) is evaluated based on topic relevance, topic-level emergence, surprise, and analytical value of newly surfaced ideas.
SubtopicSparkMeMimiTalk
 Participant 1Participant 2Participant 3Participant 4
Predefined subtopics
General outlook on AI’s broader societal and industry impact✓✓-✓
Personal beliefs about the ethics and risks of AI in the workplace✓✓✓-
Missing AI tools or features that would be most beneficial in the future✓✓✓✓
Predicted evolution of their job in the next 5–10 years with AI integration✓✓✓✓
Table 2: An illustrative coverage comparison across 4 participants interviewed with SparkMe and MimiTalk on the topic "AI Attitudes and Future Outlook". While both systems and participants follow the same interview topic guide, SparkMe consistently covers predefined subtopics while eliciting emergent themes that differ across interviews. In contrast, MimiTalk covers a subset of predefined topics and does not surface additional themes.
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Predefined subtopic coverage and utility of subtopics ( yy -axis) as a function of the number of interview turns ( xx -axis) by ablating different features of SparkMe , using Qwen3-30B-A3B-Instruct-2507 as the backbone for the interviewer LLM and user agent.
Figure 9: (a) Predefined subtopic coverage and (b) utility of subtopics ( yy -axis) as a function of the number of interview turns ( xx -axis) for different systems ( Section 4.1.3 ), using GPT-4.1-mini as the backbone for the interviewer LLM and user agent. SparkMe (Ours) efficiently converges to a higher coverage and utility value, consistently outperforming other baselines
Figure 10: (a) Predefined subtopic coverage and (b) utility of subtopics ( yy -axis) as a function of the number of interview turns ( xx -axis) for SparkMe , by varying the horizon length, using Qwen3-30B-A3B-Instruct-2507 as the backbone for the interviewer LLM and user agent.
Figure 11: (a) Predefined subtopic coverage and (b) utility of subtopics ( yy -axis) as a function of the number of interview turns ( xx -axis) for SparkMe using Qwen3-30B-A3B-Instruct-2507 as the backbone for the interviewer LLM and various LLMs for the user agent.
Figure 12: Statistics per turn between SparkMe , LLM-Roleplay , StorySage , MimiTalk , and Interview-GPT .
Figure 13: Utility score for different γ\gamma with α=148\alpha=\frac{1}{48} and β\beta of 0.
TopicSubtopics
Introduction & Background• Educational background or training • Specific job title and role description • Current industry or sector (e.g., tech, finance, manufacturing) • Company size and environment • Type of business or market segment • Duration/years of experience in current role • Professional seniority or career level
Core Responsibilities and Decision-Making• Primary job responsibilities and regular daily tasks • Approximate proportion of time spent on core activities • Level of autonomy and scope of decision-making in the role
Task Proficiency, Challenge, and Engagement• Tasks that feel easiest or most natural to perform • Tasks perceived as most challenging or complex • Tasks that are repetitive, data-heavy, or suitable for automation • Tasks that are most enjoyable or engaging versus those that feel boring or tedious • Common pain points or inefficiencies in completing tasks • How enjoyment, skill level, and productivity relate to one another
Tech Learning Comfort• Attitude towards learning new technologies and tools • Perceived adaptability to new software/methods • Willingness to invest time in tech training • Motivations or barriers to learning new tech (e.g., workload, relevance) • Influence of peers or management on willingness to adopt new tools
Primary Tools and Technologies Used in Work• Specific software, platforms, or systems used daily • Essential non-AI tools for workflow • Familiarity with industry-standard technologies • Interoperability or integration issues between tools
Appendix
Table 3: Interview Topics and Subtopics (Part 1 of 2)
TopicSubtopics
AI Experience and Tool Adoption• Familiarity with fundamental AI/ML concepts and terminology • Names of specific AI software/platforms currently used in work • Frequency and purpose of AI tool application (specific use cases) • Specific examples of AI success and failure experiences (lessons learned) • Availability of organizational training or peer resources for AI use
AI Interaction Style and Workflow Change• Preferred mode of interaction: independent versus step-by-step collaboration • Style of human-AI teaming (e.g., advisor, assistant, co-worker) • Willingness and openness to adopting new AI-driven workflows • Preference for conversational vs. command-based interfaces (communication dynamics)
Trust and Control Over AI• Extent to which tasks rely on specialized, tacit domain knowledge • Level of trust in AI outputs for work tasks and critical decisions • Ideal balance of human effort and AI automation for specific tasks • Conditions under which high automation is acceptable or threatening
AI Impact on Skills and Job Security• Perceived impact of AI on the importance of existing skills (enhanced vs. reduced) • Emerging skills or new areas of responsibility created by AI • Level of concern about AI replacing specific tasks or the overall role • Availability of override mechanisms or manual checks for AI-driven processes • Perceived change in team or company policies regarding AI adoption
AI Attitudes and Future Outlook• General outlook on AI’s broader societal and industry impact • Personal beliefs about the ethics and risks of AI in the workplace • Missing AI tools or features that would be most beneficial in the future • Predicted evolution of their job in the next 5-10 years with AI integration • Concrete steps they would want their organization to take regarding AI strategy
Appendix
Table 4: Interview Topics and Subtopics (Part 2 of 2)
ModelKendall’s τ\tauExact AgreementWithin 1 pointWithin 2 points
mR3-Qwen3-14B0.54945.2%88.2%98.9%
Qwen3-30B-A3B-Instruct0.52441.9%74.2%88.2%
GPT-4.10.43931.2%75.3%91.4%
GPT-4.1-mini0.40523.7%62.4%78.5%
Appendix
Table 5: Kendall’s τ\tau correlation between LLM-as-a-judge and human rating annotations for coverage assessment on 100100 samples total, with 2020 instances per rating from 1 to 5. Each sample is annotated by 33 humans, where the human inter-annotator agreement is Krippendorff’s α=0.65\alpha=0.65 .
Figure 14: The front page of our interviewer platform, along with the instructions and PII notice.
Figure 15: Our web-interviewer platform, where SparkMe will display their response along with their voice using text-to-speech (TTS). The interviewee may respond in either text (through the chat box) or audio, depending on their convenience.
Figure 16: A sample of our annotation platform for the post-interview questionnaire. The annotators are given the instruction rubrics based on Table 7 , Table 8 , and Table 9 , along with the subtopic and summary or notes collected during the interview.
Figure 17: The instruction given for human annotations to verify our LLM-as-a-judge to evaluate coverage against ground truth.
Figure 18: A sample of information presented for human annotations to verify our LLM-as-a-judge to evaluate coverage against ground truth.
SystemLeading Questions
SparkMeQ1. It’s interesting that prompt writing has become an important skill for you. Could you tell me more about how learning to write effective prompts has changed the way you work with AI? Are there particular challenges or benefits you’ve noticed as you’ve developed this skill?
 Q2. It’s great to hear you’ve had positive experiences with several AI platforms. Shifting gears a bit, I’m curious about how you feel AI has impacted the importance of your existing skills as a data scientist. Do you find that AI has enhanced the value of your current expertise, or has it reduced the need for some skills? How has that influenced the way you approach your work?
MimiTalkQ1. What motivates you to dedicate 3 to 4 hours daily to learning new technologies, and are there any challenges you face in maintaining this learning habit?
 Q2. Can you tell me about a recent time when an AI chat tool helped you solve a tricky coding or IT issue? What was the problem and how did the AI assist you?
 Q3. Why do you feel that none of your tasks are suitable for automation? Are there specific reasons or concerns that make automation less appropriate?
Appendix
Table 6: All detected leading questions posed during the user study interview by the SparkMe and MimiTalk
DimensionDescriptionScale (1–5)
CoverageHow comprehensively did the interview cover the intended topics?1 : Missed most key areas 2 : Covered some areas but missed significant parts 3 : Covered main points but lacked detail 4 : Covered most areas well 5 : Comprehensive coverage of all intended topics
DepthDid the interviewer probe deeply enough to get meaningful answers?1 : Very superficial; surface-level only 2 : Mostly superficial with rare follow-up 3 : Adequate depth but missed opportunities 4 : Good depth; probed well 5 : Exceptional depth; uncovered profound insights
CorrectnessHow accurate were the interviewer’s summaries and interpretations?1 : Frequent major inaccuracies 2 : Several errors or misunderstandings 3 : Mostly accurate with some minor errors 4 : Accurate with rare minor slips 5 : Flawless interpretation and summary
Appendix
Table 7: Evaluation rubrics given to human annotators to evaluate the interview’s content quality.
DimensionDescriptionScale (1–5)
Topical RelevanceHow much did this emergent subtopic add new dimensions to the planned topic being discussed?1 : Irrelevant or distracting 2 : Weakly relevant 3 : Somewhat relevant 4 : Mostly relevant 5 : Highly relevant and aligned
Topic-Level EmergenceDid this subtopic reveal novel perspectives within the planned sub-topics being discussed? Did it go beyond obvious aspects to uncover new angles?1 : Adds nothing new 2 : Minor/redundant connections 3 : Moderately emergent 4 : Clearly emergent/meaningful 5 : Highly emergent; substantial novel directions
SurpriseHow unexpected were the emergent ideas relative to what had already been discussed?1 : Entirely expected/routine 2 : Slightly unexpected 3 : Moderately unexpected 4 : Highly unexpected 5 : Strongly surprising while remaining relevant
Analytical ValueHow much does this contribute to understanding AI workforce issues across the industry?1 : Distracting/uninformative 2 : Little analytical insight 3 : Some insight, limited utility 4 : Meaningful analytical insight 5 : Substantially deepens understanding
Appendix
Table 8: Evaluation rubrics given to human annotators to cross-evaluate emergent insights in different interviews.
DimensionDescriptionScale (1–5)
ClarityHow clear were the questions and the flow of the conversation?1 : Confusing and disjointed 2 : Often unclear 3 : Generally clear but sometimes vague 4 : Very clear and easy to follow 5 : Crystal clear and perfectly structured
AdaptivenessHow well did the interviewer adapt to the interviewee’s responses?1 : Rigid; ignored responses 2 : Struggled to adapt 3 : Adapted adequately 4 : Adapted well to flow 5 : Seamlessly adapted to every nuance
Comfort & FairnessDid the interviewee feel comfortable and treated fairly?1 : Hostile or unfair environment 2 : Uncomfortable or biased 3 : Neutral/Adequate 4 : Comfortable and fair 5 : Extremely supportive and unbiased
Overall ExperienceTaking everything into account, how would the interviewee rate the session?1 : Poor 2 : Below Average 3 : Average 4 : Good 5 : Excellent
Appendix
Table 9: Evaluation rubrics given to human annotators to evaluate the interview overall interaction quality.
More in this topic

LLM Agent Planning

Large language model (LLM) agent planning studies how agents decompose goals, choose actions and tools, and adapt plans from feedback.

See all 225 papers

Explore similar work

CardsList
May 15, 2026

AInterviewer: A Platform for Designing and Conducting AI-led Qualitative Interviews

There are now multiple proposals for systems based on Large Language Models (LLMs) to conduct automated qualitative interviews, but most of the current solutions rely on proprietary LLMs, which compromises reproducibility and data security. They also rely on LLMs for all interview tasks, which limits standardisation of question wording as well as control over question order. To address these issues, we introduce the AInterviewer platform, an opensource solution based on a multi-agent pipeline that combines controlled question administration of survey software with the flexibility of LLMs. AInterviewer is an interdisciplinary effort designed to implement best practices of qualitative interviewing in social science, and it can run with locally hosted models to ensure security, transparency, and reproducibility. Our platform provides a web-based GUI supporting each phase of data collection: from interview guide design and pilot testing to interview distribution and data collection monitoring.

Tobias Priesholm Gardhus, Nikolas Vitsakis, Fie Lejre Frederiksen +2

University of Copenhagen · IT University of Copenhagen

LLM Agents
AskPDF
Oct 8, 2026

Evidence-Traceable Dynamic Interviewer Architecture for Expertise-Adaptive Qualitative Interviews Using Local LLMs

Automated interviewers and conversational agents are increasingly used in research, recruitment, customer service, and education. However, many existing systems rely on fixed question sequences and provide limited context-based personalization without considering participants' knowledge, which can lead to repetitive or irrelevant follow-up questions. Therefore, there is a need for an adaptive interviewing system that can adjust question depth while maintaining conversational continuity and semantic progression. To address this, an Evidence-Traceable Dynamic Interviewer Architecture is presented using a locally hosted Large Language Model (LLM), with the interview continuously adapted throughout the entire conversation based on the participant's responses and evolving context. The interviewer profiles participants' expertise in real time to generate knowledge-appropriate questions, well-articulated responses, and smooth transition messages that support conversational continuity. A five-module prompt-driven architecture and persistent interview-state record support these functions. The interviewer was evaluated with 246 participants. Expertise Profiling module (M3) showed 78.9% exact agreement with independently reported participant expertise, with a weighted Cohen's K of 0.80. Generate Iterative Questions module (M4) showed a strong expertise-complexity association (p=.79, p<.001), and participants reported high relevance (mean 4.41), engagement (mean 4.32), and satisfaction (mean 4.38), providing evidence that the architecture's adaptive components operated consistently with their intended functions while participants reported a positive interview experience.

Aisvarya Adeseye, Jouni Isoaho, Adeyemi Adeseye +2

Department of Computing, University of Turku, Turku, Finland

Conversational AgentsLLM Agent Evaluation
AskPDF
Jul 11, 2026

PolyInterview: An LLM-based Platform for Immersive Mock Interview Practice with Comprehensive Multimodal Assessment

Preparing for job interviews is important for securing desired positions, yet realistic practice remains difficult to access: real interviews are infrequent, expert mock coaching is costly, and self-practice offers neither adaptive dialogue nor structured assessment. Existing systems typically address only parts of this need through fixed question sequences, limited communication channels, or feedback with little supporting evidence. We present PolyInterview, an LLM-based platform for immersive mock interview practice with comprehensive multimodal assessment. PolyInterview uses the target job description and CV to generate questions tailored to the role and candidate, conducts multi-turn spoken interviews with a lip-synced digital human interviewer that asks answer-aware follow-up questions, and evaluates response content, vocal delivery, and non-verbal behavior. Four parallel evaluators produce 13 behavior-level features that are aggregated into 10 assessment aspects and two competency tracks. Guided by the KSA and STAR frameworks, the report links each score to behavioral evidence and actionable recommendations. PolyInterview is publicly accessible. Its current all-account snapshot contains 101 accounts, 1,564 interview sessions, 7,665 generated questions, and 1,422 five-stage question sets. Generated questions are more closely aligned with their matched job description than with cross-role job descriptions in 93.7% of sessions. An evaluation by ten experts found strong question plans and actionable feedback.

Zhiyuan Wen, Jiannong Cao, Zijian Wang +4

Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China · School of Artificial Intelligence, Chongqing University of Posts and Telecommunications, China

Topic pending
AskPDF