Evidence-Traceable Dynamic Interviewer Architecture for Expertise-Adaptive Qualitative Interviews Using Local LLMs

Aisvarya Adeseye; Jouni Isoaho; Adeyemi Adeseye; Seppo Virtanen; Mohammad Tahir

cs.AIOct 8, 2026

Evidence-Traceable Dynamic Interviewer Architecture for Expertise-Adaptive Qualitative Interviews Using Local LLMs

Authors: Aisvarya Adeseye, Jouni Isoaho, Adeyemi Adeseye, Seppo Virtanen, Mohammad Tahir

Organizations: Department of Computing, University of Turku, Turku, Finland

Abstract

Automated interviewers and conversational agents are increasingly used in research, recruitment, customer service, and education. However, many existing systems rely on fixed question sequences and provide limited context-based personalization without considering participants' knowledge, which can lead to repetitive or irrelevant follow-up questions. Therefore, there is a need for an adaptive interviewing system that can adjust question depth while maintaining conversational continuity and semantic progression. To address this, an Evidence-Traceable Dynamic Interviewer Architecture is presented using a locally hosted Large Language Model (LLM), with the interview continuously adapted throughout the entire conversation based on the participant's responses and evolving context. The interviewer profiles participants' expertise in real time to generate knowledge-appropriate questions, well-articulated responses, and smooth transition messages that support conversational continuity. A five-module prompt-driven architecture and persistent interview-state record support these functions. The interviewer was evaluated with 246 participants. Expertise Profiling module (M3) showed 78.9% exact agreement with independently reported participant expertise, with a weighted Cohen's K of 0.80. Generate Iterative Questions module (M4) showed a strong expertise-complexity association (p=.79, p<.001), and participants reported high relevance (mean 4.41), engagement (mean 4.32), and satisfaction (mean 4.38), providing evidence that the architecture's adaptive components operated consistently with their intended functions while participants reported a positive interview experience.

Figures & tables

Figure 1: Overview of the Evidence-Traceable Dynamic Interviewer Architecture

System-Prompt Component	Main Function
Role and Interview Identity	Defines the AI interviewer role and responsibilities
General Information	Specifies objectives, research areas, priorities, and question allocation
Purpose and Context	Defines language, question type, style, tone, and conversational flow
Ethical Guidelines	Controls privacy, sensitive questions, neutrality, and inclusive interaction
Knowledge Handling Rules	Controls reliable information use, uncertainty, and hallucination avoidance
Behavioral Consistency	Maintains the defined rules across interview turns

Table 1: System-Prompt Components in M1

M1 User-Prompt Component	Main Function
LLM Role Specification	Defines the model as a system-prompt generator
Task Scope	Limits generation to the supplied interview configuration
Clarity of Instruction	Specifies which configuration elements must be included
Language Emphasis	Maintains the selected interview language and terminology
Output Format Enforcement	Requires a structured, machine-readable output
Action Limitation	Prevents M1 from conducting the interview or generating questions

Table 2: M1 User-Prompt Components

Figure 2: System prompt component structure in M1. The figure presents the persistent research, behavioral, ethical, knowledge-handling, and output constraints governing the interview.

Figure 3: M1 user prompt component structure. The figure presents the instruction structure used to transform the design-time interview configuration into the reusable system prompt.

M2 Component	Function
Research Area and Priority	Selects an appropriate high-priority area for beginning the interview
Question Complexity	Starts with a low-complexity, broadly understandable question
Contextual Framing	Aligns the question with the research objectives and interview context
Question Requirements	Produces one clear, open-ended, non-leading question
Behavioral Constraints	Prevents unnecessary explanation, repetition, or multiple questions
Output Structure	Returns the initial question in the required structured format

Table 3: M2 User-Prompt Components for Initial Question Generation

Figure 4: M2 user prompt component structure for initial question generation. The prompt defines the research-area selection, initial complexity, contextual framing, question requirements, behavioral constraints, and output structure used to generate the first interview question.

Expertise Factor	What M3 Examines
F1. Domain Terminology and Specificity	Appropriate use of domain terminology, concepts, technologies, processes, or specific examples
F2. Conceptual Correctness and Relevance	Whether objectively verifiable domain statements are coherent, relevant, and technically appropriate
F3. Reasoning Depth	Ability to explain relationships, causes, consequences, dependencies, or trade-offs
F4. Application and Evaluation	Ability to apply knowledge to situations, compare alternatives, justify decisions, or propose solutions

Table 4: Expertise Factors Evaluated by M3

Figure 5: M3 user prompt component structure for dynamic expertise profiling. The prompt defines the participant-response input, expertise factors, classification levels, evidence requirements, behavioral constraints, and structured output used to generate the dynamic expertise profile.

M4 Output	Function
Response Message	Briefly acknowledges or reflects the participant’s previous response
Transition Message	Creates a natural conversational connection between the previous response and the next question
Next Question	Generates a context-aware, expertise-aligned, open-ended follow-up question

Table 5: Participant-Facing Outputs Generated by M4

Figure 6: M4 user prompt component structure for adaptive iterative question generation. The prompt combines participant response, expertise profile, interview history, research priorities, and conversational constraints to generate the response message, transition message, and next question.

M5 Component	Function
Candidate Question Input	Receives the proposed next question generated by M4
Question-History Comparison	Compares the candidate question with previously asked questions
Semantic Overlap Assessment	Identifies conceptual similarity beyond exact wording
Research Progression Check	Determines whether the question contributes new information or advances the discussion
Validation Decision	Produces an Accept or Regenerate decision
Structured Output	Returns the validation result and internal justification

Table 6: M5 Components for Semantic Question Validation

Figure 7: M5 user prompt component structure for semantic question uniqueness validation. The prompt defines the candidate-question input, question-history comparison, semantic-overlap assessment, research-progression criteria, behavioral constraints, validation decision, and structured output.

Research Area	Priority	Total Question Allocation
Awareness and knowledge of LLMs among employees	High	4
Application of LLMs in the organization	Medium	3
Skill levels and training in using LLMs	High	3
Data privacy and security in LLM use	Medium	4
Organizational guidelines for LLM use and adoption	Low	2

Table 7: Research Areas and Interview Configuration

Evaluation Dimension	Purpose
Question Relevance and Coherence	Evaluates whether questions were appropriate, logically connected, and relevant to the ongoing discussion
Cognitive and Emotional Engagement	Evaluates whether the interaction maintained participants’ attention and involvement
Overall User Satisfaction	Evaluates participants’ overall perception of the AI-powered interview experience

Table 8: Participant-Based Evaluation Dimensions

Verification Component	Comparison / Evidence	Main Purpose
Expertise Profiling	M3 expertise vs. participant-reported expertise	Assess agreement in expertise classification
Question Adaptation	M3 expertise vs. subsequent M4 question complexity	Assess expertise-driven adaptation
Question Uniqueness	M5 decisions and question-history comparisons	Examine semantic-overlap filtering and regeneration
Question Relevance and Coherence	Participant post-interview rating	Assess conversational quality
Engagement	Participant post-interview rating	Assess participant involvement
Overall Satisfaction	Participant post-interview rating	Assess overall interview experience

Table 9: Summary of Experimental Verification Components

Characteristic	$n$ (%)	Characteristic	$n$ (%)
Gender		Age
Male	130 (52.8)	18–24 years	31 (12.6)
Female	111 (45.1)	25–34 years	78 (31.7)
Prefer not to say	5 (2.0)	35–44 years	70 (28.5)
		45–54 years	43 (17.5)
		55 years or above	24 (9.8)

Table 10: Participant Demographic and Background Characteristics (

N=246

)

Self-Reported Expertise	M3 Novice	M3 Basic	M3 Advanced	M3 Expert
Novice	44	7	1	0
Basic Knowledge	8	54	10	1
Advanced Knowledge	1	9	58	8
Expert	0	1	6	38

Table 11: M3 Expertise Classification Compared with Participant Self-Assessment

M3 Expertise Level	Mean Complexity	SD
Novice	1.42	0.55
Basic Knowledge	2.08	0.60
Advanced Knowledge	2.91	0.63
Expert	3.56	0.51

Table 12: Question Complexity by M3 Expertise Level

Expertise Level	Representative Question
Novice	“Have you ever thought about privacy when using an LLM at work? What kinds of information would you avoid sharing with it?”
Basic Knowledge	“What privacy or security problems do you think could arise when employees enter work-related information into publicly available LLMs?”
Advanced Knowledge	“How would you assess the risk of sensitive organizational information being exposed through employees’ use of public LLM services?”
Expert	“Suppose an organization wants to permit employees to use public LLMs while protecting confidential information. What technical and governance controls would you recommend, and how would you evaluate whether those controls are effective?”

Table 13: Representative Questions Across Expertise Levels

Evaluation Dimension	Mean	SD
Question Relevance and Coherence	4.41	0.58
Cognitive and Emotional Engagement	4.32	0.64
Overall User Satisfaction	4.38	0.61

Table 14: Participant Evaluation of the AI-Powered Interview

Explore similar work

May 15, 2026cs.HC

AInterviewer: A Platform for Designing and Conducting AI-led Qualitative Interviews

There are now multiple proposals for systems based on Large Language Models (LLMs) to conduct automated qualitative interviews, but most of the current solutions rely on proprietary LLMs, which compromises reproducibility and data security. They also rely on LLMs for all interview tasks, which limits standardisation of question wording as well as control over question order. To address these issues, we introduce the AInterviewer platform, an opensource solution based on a multi-agent pipeline that combines controlled question administration of survey software with the flexibility of LLMs. AInterviewer is an interdisciplinary effort designed to implement best practices of qualitative interviewing in social science, and it can run with locally hosted models to ensure security, transparency, and reproducibility. Our platform provides a web-based GUI supporting each phase of data collection: from interview guide design and pilot testing to interview distribution and data collection monitoring.

Tobias Priesholm Gardhus, Nikolas Vitsakis, Fie Lejre Frederiksen +2

University of Copenhagen · IT University of Copenhagen

LLM Agents

Ask PDF

May 28, 2026cs.CL

Adaptive Interviewing for Persona Simulation in LLMs: Evidence-Grounded Reasoning Improves Decision Alignment

Accurately simulating the decisions of a specific individual remains challenging for large language models (LLMs), partly because persona information is often provided as static descriptions that miss the values, experiences, and contextual cues needed for individual-level decision simulation. We propose an adaptive interview framework that gathers persona-relevant information through a structured three-stage dialogue: core questions, dynamic follow-ups, and a synthesized personality summary. Using the resulting interview transcripts, we evaluate whether LLMs can simulate participants' decisions in moral dilemma scenarios. We compare three conversational contexts -- Core-10 responses, the full interview dialogue, and a summarized persona representation. We find that adaptive interviewing functions less as a uniform accuracy booster and more as a selective grounding mechanism: follow-up-derived evidence is incorporated in around 40% of full-interview traces, and these follow-up-grounded predictions are more accurate than core-only grounded ones (45.5% vs. 39.3%). These findings highlight that richer persona context alone is insufficient: improvements arise only when models actually ground their decisions in user-specific evidence.

Ruoxi Su, Yuhan Liu, Jingyu Hu

Independent Researcher · University of Cambridge

LLM Grounding Human Behavior Simulation LLM Decision-Making

Ask PDF

Jul 11, 2026cs.CL

PolyInterview: An LLM-based Platform for Immersive Mock Interview Practice with Comprehensive Multimodal Assessment

Preparing for job interviews is important for securing desired positions, yet realistic practice remains difficult to access: real interviews are infrequent, expert mock coaching is costly, and self-practice offers neither adaptive dialogue nor structured assessment. Existing systems typically address only parts of this need through fixed question sequences, limited communication channels, or feedback with little supporting evidence. We present PolyInterview, an LLM-based platform for immersive mock interview practice with comprehensive multimodal assessment. PolyInterview uses the target job description and CV to generate questions tailored to the role and candidate, conducts multi-turn spoken interviews with a lip-synced digital human interviewer that asks answer-aware follow-up questions, and evaluates response content, vocal delivery, and non-verbal behavior. Four parallel evaluators produce 13 behavior-level features that are aggregated into 10 assessment aspects and two competency tracks. Guided by the KSA and STAR frameworks, the report links each score to behavioral evidence and actionable recommendations. PolyInterview is publicly accessible. Its current all-account snapshot contains 101 accounts, 1,564 interview sessions, 7,665 generated questions, and 1,422 five-stage question sets. Generated questions are more closely aligned with their matched job description than with cross-role job descriptions in 93.7% of sessions. An evaluation by ten experts found strong question plans and actionable feedback.

Zhiyuan Wen, Jiannong Cao, Zijian Wang +4

Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China · School of Artificial Intelligence, Chongqing University of Posts and Telecommunications, China

Topic pending

Ask PDF