cs.HCSep 14, 2026

When the AI Leaves the Tailorshop: Measuring What an LLM Advisor Leaves Behind in Complex Problem Solving

Authors: Robin Welsch

Organizations: Aalto University, Espoo, Finland

Abstract

Complex problem solving depends on acting effectively and understanding how a system works. AI advice may support these outcomes unequally. Two preregistered experiments compared participants managing a simulated clothing factory with and without an LLM advisor. Across studies, AI-supported participants reported greater confidence and understanding with less effort. In the first study (N=200), assistance increased company value but produced no detectable prediction-accuracy difference. After withdrawal, previously supported participants outperformed controls when decisions were scored against repeating previous choices, but not default settings. Within the AI-supported group, more frequent recommendation alterations predicted better unaided performance. In the second study (N=198), AI-supported participants went bankrupt less often and showed a small knowledge advantage in the registered analysis, largely associated with remaining solvent. More frequent recommendation alterations predicted higher knowledge within the AI-supported group. Applied HAI evaluation should assess users' understanding and independent capability alongside the performance achieved with AI support.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Available but Unclaimed: An Empirical Study of Human-AI Synergy

    Sep 15, 2026Robin Welsch, Michelle Rausch, Pascal Knierim +4Human-AI ComplementarityAI-Assisted Decision Making

  2. AI advice suppresses people's willingness to say "I don't know", even when the advice is wrong and accuracy is incentivized

    Jul 15, 2026Chiara Marcoccia, Walter Quattrociocchi, Valerio CapraroAI-Assisted Decision MakingHuman-AI Decision Making

  3. How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles

    Aug 24, 2026Shang Wu, Catarina G Belem, Shuyuan Fu +2Human-AI Collaboration