As generative artificial intelligence (AI) becomes more capable and widely used, understanding when and how humans should work with these systems is becoming increasingly important. However, the empirical evidence in this space offers conflicting recommendations, which leaves individuals and organizations without clear guidance on when to combine human and AI capabilities. Existing meta-analyses can help reconcile these findings, but they require substantial manual effort and cannot readily incorporate new evidence as it emerges. To address this challenge, we introduce AutoAnalysis: an automated, self-updating meta-analysis of when humans and AI work better together, and when they work better alone. AutoAnalysis' most recent run synthesizes 1,955 effect sizes from 420 papers. Across this body of work, we found that, on average, human-AI combinations performed significantly better than humans alone, but they performed significantly worse than the better of humans or AI alone. These findings highlight the important distinction between improving human performance and achieving human-AI synergy and underscore the need to evaluate collaboration against both human and AI performance alone. As new studies emerge, AutoAnalysis will integrate their findings into the meta-analysis, providing cumulative evidence about when humans and AI perform better together and how human-AI performance varies across AI agents, tasks, and processes.
Figures & tables
Figure 1: Published papers with “human-AI” in the title, 2000–2026.
Figure 2: The AutoAnalysis pipeline.
Figure 3: Forest plots of human augmentation and human–AI synergy.
Figure 4: Moderator analyses of human augmentation and human–AI synergy.
Stage
Task
Model
Cleaning
Check if a paper is peer-reviewed, and verify the title, authors, year, venue, DOI and abstract
gpt-6.1-sol (high)
Screening
Decide eligibility based on title and abstract
gpt-6.1-sol (high)
Reviewing and extracting
Locate a copy of the paper and, if found, make inclusion decision; if applicable, extract effect-sizes in the paper
gpt-6-astra (xhigh)
Auditing
Re-screen, re-review, or re-check a random sample of decisions
Claude Fable 5.1 (high)
Table S1: Stages of the AutoAnalysis pipeline.
Database
Search string
ACM Digital Library
[Abstract: human] OR [Abstract: humans] OR [Abstract: participant] OR [Abstract: participants] AND [Abstract: ”generative AI”] OR [Abstract: genAI] OR [Abstract: ”large language model”] OR [Abstract: GPT] OR [Abstract: Claude] OR [Abstract: LLM] OR [Abstract: ”diffusion model”] OR [Abstract: DALL-E] OR [Abstract: DALLE] OR [Abstract: Midjourney] OR [Abstract: ”Stable Diffusion”] AND [Abstract: collaborate] OR [Abstract: assist] OR [Abstract: aid] OR [Abstract: interact] OR [Abstract: help] AND [Abstract: experiment] OR [Abstract: experiments] OR [Abstract: ”user study”] OR [Abstract: ”user studies”]
Web of Science Core Collection
AB=(human OR humans OR participant OR participants) AND AB=(”generative AI” OR genAI OR ”large language model” OR GPT OR Claude OR LLM OR ”diffusion model” OR DALL-E OR DALLE OR Midjourney OR ”Stable Diffusion”) AND AB=(collaborate OR assist OR aid OR interact OR help) AND AB=(experiment OR experiments OR ”user study” OR ”user studies”)
Table S2: Search strings for the database component of the seed dataset.
Review
Scope
Al Naqbi et al. (2024)
Generative AI and work productivity across professional sectors
Belkina et al. (2025)
Case studies implementing generative AI in higher education
Deng et al. (2025)
Experimental studies of ChatGPT in education
Doo and Park (2026)
Experimental studies of ChatGPT and learning outcomes
Holzner et al. (2025)
Meta-analysis of generative AI and creativity
del Rio-Chanona et al. (2025)
Generative AI, labor markets and worker productivity
Table S3: Systematic reviews and meta-analyses included in our initial search.
AI agents are reshaping the workspace, leading to drastic change of how humans work. Despite the considerable potential of human-agent collaboration both in preserving human agency and generating economic value, this paradigm remains largely absent from occupational task evaluation, hindered by the difficulty of gathering real human data and accounting for inter-human variability. We introduce CollabSkill, a framework for evaluating human-agent collaboration on real-world occupational tasks. CollabSkill pairs real human workers with AI agents on tasks matched to their occupational background, collecting data that capture the complexity of economically valuable tasks and the usage patterns of real workers. To account for inter-human variability, CollabSkill employs a Bayesian skill rating system to disentangle and quantify the skill contributions of both humans and AI agents. Drawing on over 1,500 prompts from 386 working sessions contributed by 93 human workers, our analysis yields insights on two fronts: on the agent side, rankings on CollabSkill diverge meaningfully from those of existing fully autonomous benchmarks where Codex leads, with Claude Code ranking first; on the human side, CollabSkill reveals that practical experience emerges as the primary driver of collaboration skill, with hands-on collaboration meaningfully shifting workers' AI literacy. Together, we hope CollabSkill enables the community to invest in systematic evaluation of human-agent collaboration and spurs development efforts aimed at building AI agents that genuinely augment human workers.
Yijia Shao, Zora Zhiruo Wang, Neel Ahuja +3
Stanford University · Carnegie Mellon University · University of Rochester +1
Evidence shows that humans and AI systems perform better together, by collaborating, than alone. This paper examines two key design dimensions of human-AI collaboration (autonomy and initiative) and explores the collaboration patterns that they generate. Documenting these patterns starts with identifying the underlying problems and solutions, followed by examining the internal tensions within the problems. The paper uses a paradox perspective to analyze those tensions. It describes a process for surfacing the tensions and mapping the underlying paradoxes. It also illustrates how the pattern descriptions can be derived from mapping these paradoxes. Finally, the paper documents four human-AI collaboration patterns: Instruction, Delegation, Assistance, and Co-creation.
Automated AI agents are increasingly capable, yet many scientific and professional tasks require human judgment and contextual expertise. We use simulated shared-workspace human-AI teams as a controlled testbed for studying how collaboration structure shapes team behavior. Using the Collaborative Gym environment with tasks from DiscoveryBench, we vary team compositions and collaboration structures across 1,482 sessions. We find that adding additional collaborators can lower performance when coordination structure is absent. We then evaluate collaboration scaffolding that combines shared group memory with simulated human-in-the-loop (HITL) gates, where selected actions require approval from a designated simulated participant. This scaffolding improves performance, most clearly in three-person teams, with clearer responsibility signals and stronger routing of expertise to team actions. Overall, our results suggest that coordination structure is central to whether available capability improves team outcomes.