cs.HCMar 10, 2026
SaveInterview-Informed Generative Agents for Product Discovery: A Validation Study
Organizations: Adobe Research San Jose, CA, USA
Abstract
Large language models (LLMs) have shown strong performance on standardized social science instruments, but their value for product discovery remains unclear. We investigate whether interview-informed generative agents can simulate user responses in concept testing scenarios. Using in-depth workflow interviews with knowledge workers, we created personalized agents and compared their evaluations of novel AI concepts against the same participants' responses. Our results show that agents are distribution-calibrated but identity-imprecise: they fail to replicate the specific individual they are grounded in, yet approximate population-level response distributions. These findings highlight both the potential and the limits of LLM simulation in design research. While unsuitable as a substitute for individual-level insights, simulation may provide value for early-stage concept screening and iteration, where distributional accuracy suffices. We discuss implications for integrating simulation responsibly into product development workflows.
Figures & tables
Figure 1. Study design and simulation setup. We recruit knowledge workers to conduct in-depth interviews and surveys. This data is used to ground our generative simulation agents. To evaluate the agents’ performance, we compare their responses to those of human participants in a concept testing session. This allows us to explore the application of these agents in product discovery contexts. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.
Figure 2. Prototype concepts for AI document workflow assistance. The four designs span different levels of intervention and integration: (a) Multi-document Q&A Assistant, (b) Smart Highlights Assistant, (c) Audio Assistant, and (d) Workflow Actions Assistant. These concepts represent a spectrum from passive support to active automation, enabling evaluation across varied user needs. Higher resolution figures are available in the Supplementary Material D. description
Figure 3. Simulation architecture for interview-informed generative agents. The agent creation module builds memory and scratchpad representations from interview transcripts, while the agent interaction module retrieves relevant turns, reflects, and generates answers to concept test questions. Together, these components define how each agent simulates a target participant.
Figure 4. Individual-level accuracy of simulated responses to categorical questions. Compared to human participants, all agent variants show higher error (MAE) and lower agreement/correlation, with no significant differences across designs. Error bars indicate 95% bootstrap confidence intervals over participants (more details in Section 5.2.1 ).
Figure 5. Overall population-level alignment of simulated responses. The two bar charts of categorical responses, Likert-scale and NPS-scale respectively, show interview-informed agents capture more realistic variability compared to scratchpad-only and no-information baselines. The bottom Summary statistics table using Wasserstein distance confirm that interview-informed agents achieve the best distributional similarity to both human response attempts compared to alternative agent designs.
Figure 6. Evaluation of open-ended response quality. Across sentiment, explanation, topic coverage, and tone, interview-informed agents outperform other agent variants but remain consistently below human participants. The largest gap arises in tone, reflecting agents’ inability to capture conversational style from transcripts.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7. MAE and correlation results per question construct.
| Ablations | MAE | Corr |
|---|---|---|
| human | 0.91 (0.77, 1.08) | 0.84 (0.79, 0.87) |
| base | 1.28 (1.07, 1.49) | 0.73 (0.66, 0.80) |
| full_scratch | 1.47 (1.26, 1.69) | 0.71 (0.65, 0.77) |
| basic_scratch | 1.53 (1.30, 1.75) | 0.69 (0.62, 0.76) |
| nothing | 1.37 (1.10, 1.63) | 0.76 (0.68, 0.83) |
| image_ablation | 1.26 (1.06, 1.47) | 0.75 (0.68, 0.81) |
Table 1. Ablation study results showing mean absolute error (MAE) and correlation (Corr). We show the mean metric value as well as the confidence interval.
| Human participant’s response | Agent’s response |
| Uh, for me personally, it’s just really a matter of scope because my work doesn’t tend to Uh, involve much of our mobile applications, and we use those much more for uh external customer facing things. So that’s, it’s really not a knock against the product, it’s just that it doesn’t apply to the day to day work that I do. Although, like I said, I do think that there could be some crossover with a lot of the training materials and some of the internal communication documents. So, it’s not to say that there wouldn’t be any use, but it’s definitely more aimed at um other people in my organization versus what I do on the day to day. | One major concern would be the accuracy of the AI tool in transforming content into audio experiences. I would worry about potential mistakes in the narration or misinterpretation of the content, which could lead to misinformation. This is quite significant since accuracy is paramount in my industry. Additionally, data privacy and security would be a big concern. Uploading sensitive documents to an AI tool could pose risks, and I would need assurances and robust security measures in place to feel comfortable. Another barrier is the lack of organizational support and established policies. Without clear guidelines and training from my organization, it would be difficult to adopt this tool. To address these concerns, there would need to be strong guarantees for accuracy, comprehensive security protocols, and organizational buy-in with proper training and policies. |
Table 2. Illustrative example comparing agent’s generated response versus its corresponding human participant’s response.
| Comparison | Statistic | p-value | Lower CI | Upper CI |
|---|---|---|---|---|
| (human – interview-based) | -0.397 | 0.040 | -0.781 | -0.013 |
| (human – scratchpad only) | -0.540 | 0.002 | -0.924 | -0.156 |
| (human – no info) | -0.469 | 0.010 | -0.853 | -0.085 |
| (interview-based – scratchpad only) | -0.143 | 0.770 | -0.527 | 0.241 |
| (interview-based – no info) | -0.072 | 0.962 | -0.456 | 0.312 |
| (scratchpad only – no info) | 0.071 | 0.964 | -0.313 | 0.455 |
Table 3. Tukey’s HSD Pairwise Group Comparisons (95.0% Confidence Interval) for the MAE results in Figure 4 .
Figure 8. Concept A: Q&A Assistant.
Figure 9. Concept B: Smart Highlights Assistant.
Figure 10. Concept C: Audio Assistant.
Figure 11. Concept D: Workflow Actions Assistant.
| Comparison | Statistic | p-value | Lower CI | Upper CI |
|---|---|---|---|---|
| (human – interview-based) | 0.099 | 0.093 | -0.011 | 0.210 |
| (human – scratchpad only) | 0.115 | 0.037 | 0.005 | 0.225 |
| (human – no info) | 0.077 | 0.275 | -0.034 | 0.187 |
| (interview-based – scratchpad only) | 0.015 | 0.983 | -0.095 | 0.126 |
| (interview-based – no info) | -0.023 | 0.951 | -0.133 | 0.087 |
| (scratchpad only – no info) | -0.038 | 0.806 | -0.148 | 0.072 |
Table 4. Tukey’s HSD Pairwise Group Comparisons (95.0% Confidence Interval) for the Gwet’s AC2 results in Figure 4 .
Figure 12. Distribution of the four constructs for the “Audio Assistant” concept as well as the Wasserstein distance comparing annotators themselves and the three different agent designs.
Figure 13. Distribution of the four constructs for the “Highlight Assistant” concept as well as the Wasserstein distance comparing annotators themselves and the three different agent designs.
Figure 14. Distribution of the four constructs for the “Q&A Assistant” concept as well as the Wasserstein distance comparing annotators themselves and the three different agent designs.
Figure 15. Distribution of the four constructs for the “Workflow Assistant” concept as well as the Wasserstein distance comparing annotators themselves and the three different agent designs.
More in this topic
User Simulation
User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.
See all 127 papers