cs.CLApr 6, 2026
SaveWhat Makes a Good Response? An Empirical Analysis of Quality in Qualitative Interviews
Organizations: Johns Hopkins University
Abstract
Qualitative interviews provide essential insights into human experiences when they elicit high-quality responses. While qualitative and NLP researchers have proposed various measures of interview quality, these measures lack validation that high-scoring responses actually contribute to the study's goals. In this work, we identify, implement, and evaluate 10 proposed measures of interview response quality to determine which are actually predictive of a response's contribution to the study findings. To conduct our analysis, we introduce the Qualitative Interview Corpus, a newly constructed dataset of 343 interview transcripts with 16,940 participant responses from 14 real research projects. We find that direct relevance to a key research question is the strongest predictor of response quality. We additionally find that two measures commonly used to evaluate NLP interview systems, clarity and surprisal-based informativeness, are not predictive of response quality. Our work provides analytic insights and grounded, scalable metrics to inform the design of qualitative studies and the evaluation of automated interview systems.
Figures & tables
| Characteristic | Definition | Source |
| Specificity (Palpability) | The extent to which a response provides detailed examples rather than abstract generalizations. | Kvale and Brinkmann (2009) ; Charmaz (2014) ; Xiao et al. (2020b) ; Small and Calarco (2022) |
| Clarity | How clear and understandable a response is. | Kvale and Brinkmann (2009) ; Xiao et al. (2020b) |
| Immediate Relevance | How relevant the response is to the specific question asked by the interviewer. | Patton (2015) ; Xiao et al. (2020b) |
| Research Question Relevance | How relevant the response is to the overall research question. | Kvale and Brinkmann (2009) ; Charmaz (2014) ; Patton (2015) |
| Spontaneity | The extent to which the response provides information beyond what is provided in the question. | Kvale and Brinkmann (2009) |
| Self-reportedness | How understandable a response is if taken out of context. | Kvale and Brinkmann (2009) |
Table 1: We identify 10 key characteristics of interview responses from qualitative literature and NLP interview systems, shown here with their definitions and where they were proposed.
| Research Project | # Interviews |
| Mindfulness for Firefighters and EMS Workers ( Steinberg et al., 2024 ) | 11 |
| Drug Shortage Management ( Shuman, 2021 ) | 16 |
| Ghanaian Healthcare Workers During COVID-19 ( Alvarez, 2025 ) | 20 |
| Socializing Policy Feedback ( Micatka, 2025 ) | 30 |
| Perspectives on Political Representation ( Ruedin and Murahwa, 2025 ) | 23 |
| Nutrition Interventions in Rural Ethiopia ( Mersha, 2025 ) | 21 |
Table 2: The Qualitative Interview Corpus is built from 14 research projects across a diverse set of domains. This table lists each project and the number of interviews that it contributed to the corpus.
| Conceptual Measure | Human Agreement | Human-LLM Agreement |
| Attributed Meaning | 0.750 | 0.868 |
| Spontaneity | 0.716 | 0.797 |
| Specificity | 0.732 | 0.789 |
| Immediate Relevance | 0.602 | 0.764 |
| Response Quality | 0.754 | 0.757 |
| Research Question Relevance | 0.714 | 0.690 |
Table 3: From 2,400 human judgments of our conceptual measures, we find strong agreement between human ratings (Human) and between the median human ratings and LLM judge ratings (Human-LLM) as measured with Krippendorff’s alpha.
| Characteristic | Std. Coef. | P-Value |
| Research Question Relevance | 0.536 | <0.001 |
| Attributed Meaning | 0.137 | <0.001 |
| Specificity | 0.056 | <0.001 |
| Response Length | 0.048 | <0.001 |
| Immediate Relevance | 0.039 | <0.001 |
| Spontaneity | 0.037 | <0.001 |
Table 4: Using a linear mixed-effect model, we identify which characteristics of participant responses are most predictive of response quality. The model has marginal and conditional .
Figure 1: We compare the distributions of quality in responses elicited using different interview techniques. We group techniques into five distinct groups (G1–G5) that correspond to the theoretical function of the techniques in the group. Each member of a group has a statistically significant difference in median with the members of all other groups.
Figure 2: We compare the distribution of quality responses over the course of the interviews. Our quality criterion reveals a temporal trend where participants tend to provide the lowest quality responses during the beginnings and ends of interviews.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: LLM Judge prompt used to evaluate the attributed meaning of participant statements.
Figure 4: LLM Judge prompt used to evaluate the clarity of participant statements.
Figure 5: LLM Judge prompt used to evaluate the relevance of participant statements to the interviewer’s question.
Figure 6: LLM Judge prompt used to evaluate the self-reportedness of participant statements.
Figure 7: LLM Judge prompt used to evaluate the specificity of participant statements.
Figure 8: LLM Judge prompt used to evaluate the spontaneity of participant statements.
Figure 9: LLM Judge prompt used to evaluate the relevance of participant statements to a key research question.
Figure 10: LLM Judge prompt used to evaluate the likelihood that participant statements contributed to the results section.
Figure 11: LLM Judge prompt used to identify the techniques used in interviewer statements.
Figure 12: An example of the interface used to collect human annotations. The participant response is redacted to comply with the Qualitative Data Repository’s terms of use.
| Research Project | Subjects | Keywords | # Interviews | Avg. Word Count |
| Mindfulness for Firefighters and EMS Workers ( Steinberg et al., 2024 ) | Medicine, Health and Life Sciences | firefighters, mindfulness, emergency medical service (EMS) providers, barriers, facilitators, implementation | 11 | 5,520 |
| Drug Shortage Management ( Shuman, 2021 ) | Medicine, Health and Life Sciences | pharmacy, inventory control, inventory shortages, cooperation, drug shortages | 16 | 5,462 |
| Ghanaian Healthcare Workers During COVID-19 ( Alvarez, 2025 ) | Medicine, Health and Life Sciences | COVID-19, healthcare worker | 20 | 3,891 |
| Socializing Policy Feedback ( Micatka, 2025 ) | Social Sciences | adolescence, welfare, politics, attitude, civic, government, youth, policy | 30 | 6,039 |
| Perspectives on Political Representation ( Ruedin and Murahwa, 2025 ) | Social Sciences | political representation, politics, voting | 23 | 2,588 |
| Nutrition Interventions in Rural Ethiopia ( Mersha, 2025 ) | Medicine, Health and Life Sciences | nutrition, nutrition-sensitive, nutrition-specific, community health, agriculture, multisectoral | 21 | 1,535 |
Table 5: A detailed view of the research projects used in the Qualitative Interview Corpus including their self-identified subjects and keywords from the Qualitative Data Repository, the total number of interviews that they contributed to the corpus, and the average length of their interviews measured in number of words.
| Metric | Word Count | Interviewer Utterances | Participant Utterances | Excerpts |
| Total | 2,157,939 | 27,254 | 31,434 | 16,940 |
| Average Per Project | 154,138.50 | 1,946.71 | 2,245.29 | 1,210 |
| Average Per Interview | 6,147.97 | 79.46 | 91.64 | 49.39 |
| Average Per Excerpt | 127.39 | 1.61 | 1.86 | — |
Table 6: The word count, number of interviewer utterances, number participant utterances, and number of excerpts in the Qualitative Interview Corpus.
| Predictor Variable | VIF |
| Response Length | 2.25 |
| Specificity | 2.21 |
| Spontaneity | 1.86 |
| Attributed Meaning | 1.75 |
| Self-reportedness | 1.69 |
| Response Length Ratio | 1.64 |
Table 7: Variance Inflation Factors (VIF) for our mixed-effects model.
| Technique | Description |
| Introduction & Contextualization | Open-ended questions designed to understand the participant or context, often unrelated to core research questions. |
| Support & Rapport Building | Statements designed to build a connection, provide support, or validate the participant’s contribution. |
| Follow-up | Brief interjections (e.g., "uh-huh") or direct calls to encourage the participant to continue talking. |
| Specifying | Follow-up questions (who, what, where, when, how) to obtain a detailed picture of an experience. |
| Direction Questioning | Questions that directly introduce specific topics or dimensions to the respondent. |
| Indirect Questioning | Questions about others’ attitudes to indirectly surface the participant’s own motivations or emotions. |
Table 8: Taxonomy of interview techniques from Kvale and Brinkmann (2009)
Figure 13: P-values from Dunn’s post-hoc test for difference in median response quality between pairs of interview techniques.
Figure 14: Correlations observed in the Qualitative Interview Corpus between each pair of characteristics in our framework.
More in this topic
Automated Evaluation
Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.
See all 396 papers