When humans collaborate with AI agents on long-horizon tasks, what happens to the agents' contributions? For solving long-horizon tasks, humans place greater importance on ongoing interaction with AI agents (e.g., revising plans) than on an agent's ability to complete the task in a single attempt. However, most agentic benchmarks have focused more on automation. Strong performance on such benchmarks does not imply effective real-world collaboration. Professional workflows offer a useful setting, as they often involve interdependent steps. To address this gap, we introduce Kolab, a new dataset of 115+ hours of screen recordings of 50 professionals from 40 occupations collaborating with agents on realistic, long-horizon problems. Kolab also contains participants' collaboration and output quality judgments. Our analysis reveals diverse collaboration patterns, highlighting the importance of when and how professionals coordinate with agents. A collaborative agent must do two things that automation benchmarks never test: anticipate whether experts will adopt, modify, or reject its contribution once it leaves the chat window to guide its next steps, and assess collaboration quality from the unfolding workflow without constant expert feedback. Our benchmark operationalizes both: (1) forecasting how professionals use agent contributions at diverse workflows, and (2) verifying their judgments of collaboration satisfaction and output quality. Findings show that frontier models, both agentic and non-agentic, overestimate collaboration with experts and struggle to distinguish between high- and low-quality collaboration. Longer workflow context does not consistently improve these predictions. As frontier agents cannot yet understand long-horizon collaboration with experts, we position Kolab as a valuable dataset for studying human-agent collaboration grounded in real professional workflows.
Figures & tables
Figure 1: An example human-agent collaborative workflow for an architect. Each step is a subgraph of human and agent actions; annotations link a source node, the target AI output it triggers, and the human response to it (here, Modify and Adopt ).
Dataset
Human Workflow
H-AI Collab.
Software Diversity
#Domains
Modalities
Agents’ Last Exam ( Sun et al., 2026b )
✗
✗
✓
55
Image, text, files
Workflow-GYM ( Zhu et al., 2026 )
✓
✗
✓
6
Image
WeaveBench ( Li et al., 2026 )
✗
✗
✓
8
Image, text
TheAgentCompany ( Xu et al., 2026 )
✗
✗
✗
6
Text
GDPVal ( Patwardhan et al., 2026 )
✗
✗
✓
44
Text, files
DeskCraft ( Wang et al., 2026 )
✗
✓
✓
5
Image
Table 1: Comparison of Kolab with long-horizon, multi-domain workflow benchmarks. “H-AI Collab.” indicates the presence of human-agent collaborative workflows; “Software Diversity” refers to tasks spanning multiple software applications; “Modalities” lists types of publicly released data.
Figure 2: Kolab domain distributions.
Response
Source Node Action
Target AI Node Action
Current Node Action
Software
Adopt
Request Afrobeats/techno background music in ElevenLabs (67:33–68:30)
Capture the ElevenLabs track and import it into the DaVinci Resolve project (72:05–74:36)
ElevenLabs, DaVinci Resolve
Modify
Upload the existing shell plan and request a full room layout (24:10–25:28)
Claude delivers a dimensioned floor-plan layout for the lobby/storage circulation (51:21–56:20)
Rebuild walls/partitions in Revit, keeping the liked zoning but fixing lobby/storage circulation (58:40–72:59)
Revit
Reject
Ask Kling AI to generate a Zanzibar schoolyard video clip (36:30–39:04)
Kling AI generates the requested video clip (39:45–41:17)
Judge the clip’s audio unusable and abandon it, out of credits to iterate further (41:17–42:06)
Kling AI
Table 2: Real examples of Adopt , Modify , and Reject responses from Kolab , drawn from a music producer workflow, architecture workflow, and a video-production workflow respectively.
Video
Graph
Annotation (692 labels)
Total duration
115 hrs
Workflow steps
508
Adopt
65.9% (456)
Avg. duration/session
69 mins
Avg. steps/session
5.24
Modify
19.1% (132)
Avg. nodes/step
8.68
Reject
15.0% (104)
Table 3: Summary statistics of Kolab for videos, workflow graphs, and human action annotations.
Figure 3: Collaboration dynamics in Kolab . (a) Burstiness vs. solo lead-in; (b) workflow branching vs. collaboration rating; (c) timing of Adopt , Modify , and Reject responses across sectors. *** refers to p<0.001 and * is p<0.05 .
Experiment Task
Observed Workflow
Response Visible?
Prediction
RQ1: AI adoption
1a: Recognition
Tsourcestart→Tadoptionend
✓
Adopt Modify Reject
1b: Forecasting
Tsourcestart→TAIend
✗
1c: Step- N forecasting
Tstepstart→TAIend
✗
1d: Workflow-level forecasting
Tworkflowstart→TAIend
✗
RQ2: Human rating
Table 4: Benchmark tasks. Txstart and Txend denote the start and end timestamps of event or segment x , respectively; x can refer to the source node, target AI node, human adoption node, or the full workflow .
1a: Recognition
1b: Forecasting
1b: Per-label F1
Model
F1
Prec.
Rec.
F1
Adopt
Modify
Reject
Majority-class
0.265
0.220
0.333
0.265
0.794
0.000
0.000
∙ Gemini 3.1 Pro
0.617 ± 0.007
0.461 ± 0.026
0.402 ± 0.011
0.391 ± 0.013
0.710 ± 0.012
0.333 ± 0.018
0.129 ± 0.031
∙ Gemini 3.8 Flash
0.587 ± 0.010
0.437 ± 0.016
0.383 ± 0.008
0.383 ± 0.009
0.758 ± 0.009
0.218 ± 0.027
0.172 ± 0.013
∙ GPT-6 Astra +
0.713
0.539 ± 0.003
0.524 ± 0.013
0.513 ± 0.003
0.721 ± 0.023
0.582 ± 0.098
0.235 ± 0.070
∙ Claude Opus 5.5
0.676
0.610
0.483
0.492
0.790
0.434
0.250
Table 5: Benchmarking performance for AI adoption recognition (Experiment 1a) and for forecasting (Experiment 1b), for video only, mean ± std over 3 runs per model. The best result in each column is shaded and the second best result is bolded .
Figure 4: Change in macro-F1 as forecasting context grows. X-axis shows how many N previous steps are included as a context. Full refers to the start of the whole workflow.
Exp. 2a: Collaboration
Exp. 2b: Output Quality
Model
MAE ↓
Within-1 ↑
Spearman ρ↑
MAE ↓
Within-1 ↑
Spearman ρ↑
Mode baseline
0.682
0.854
–
0.735
0.831
–
∙ Gemini 3.1 Pro
0.685 ± 0.006
0.850 ± 0.003
0.078 ± 0.057
0.764 ± 0.016
0.821 ± 0.006
0.046 ± 0.039
∙ Gemini 3.8 Flash
0.800 ± 0.029
0.819 ± 0.014
0.011 ± 0.003
0.792 ± 0.006
0.811 ± 0.002
0.042 ± 0.012
∙ GPT-6 Astra +
0.945 ± 0.070
0.841 ± 0.032
0.379 ± 0.130
0.912 ± 0.084
0.805 ± 0.030
0.352 ± 0.101
∙ Claude Opus 5.5 †
0.858
0.865
0.136
0.817
0.849
0.208
Table 6: Rating prediction for Experiments 2a and 2b, mean ± std over 3 runs. The best result in each column is shaded ; the second best result is bolded .
Figure 5: Full results for seven input-modality combinations across the four benchmark tasks. V stands for video, G: graph, and C: chat log; “+” is a combination of modalities, “All” refers to using video, graph, and chat as the input. The best score in each row is in bold.
Figure 6: How each modality helps.
Figure 7: Distributions of model-generated labels for Modify and Reject cases in AI adoption forecasting.
Figure 8: Mean predicted rating vs. human rating for collaboration (Exp 2a) and output quality (Exp 2b).
Appendix figures & tables42 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Item
Count
Coverage
Total participants
50
O*NET sectors
10
Occupations
40
Education
PhD
10
Master’s
20
Bachelor’s
15
Appendix
Table 7: Summary statistics of the 50 professionals.
Figure 9: Software used by participants in their day-to-day work, grouped by category.
Figure 10: Occupation distributions of Kolab . “Other” refers to three O*NET sectors: Life & Social Science, Sales & Related, and Healthcare.
Figure 11: Time spent in each recording session working with agentic AI versus the participant’s estimated time to solve the same problem without agentic AI.
Figure 12: Example workflow graph for one step of a participant’s session.
Rating
Session 1
Session 2
Total
Collaboration Satisfaction
5
117 (53.7%)
122 (56.2%)
239 (54.9%)
4
61 (28.0%)
68 (31.3%)
129 (29.7%)
3
28 (12.8%)
16 (7.4%)
44 (10.1%)
2
8 (3.7%)
7 (3.2%)
15 (3.4%)
1
4 (1.8%)
4 (1.8%)
8 (1.8%)
Appendix
Table 8: Distribution of collaboration satisfaction and output quality ratings, by session.
Figure 13: How three professionals (musician, marketing specialist, and attorney) adopt, modify, and reject agent outputs.This is based on the real data point in Kolab .
Response
Source Node Action
Target AI Node Action
Current Node Action
Software
Adopt
Request Afrobeats/techno background music in ElevenLabs (67:33–68:30)
Capture the ElevenLabs track and import it into the DaVinci Resolve project (72:05–74:36)
ElevenLabs, DaVinci Resolve
Ask Claude to add a faith-based closing line to the voiceover script (13:07–13:41)
Claude adds the requested faith-based reference to the script (34:17–34:22)
Generate the final voiceover in ElevenLabs and import it into DaVinci Resolve (34:22–36:30)
Claude, ElevenLabs, DaVinci Resolve
Modify
Upload the existing shell plan and request a full room layout (24:10–25:28)
Claude delivers a dimensioned floor-plan layout for the lobby/storage circulation (51:21–56:20)
Rebuild walls/partitions in Revit, keeping the liked zoning but fixing lobby/storage circulation (58:40–72:59)
Revit
Ask whether CapCut’s AI Relight tool requires the desktop app (63:49–63:55)
Claude explains AI Relight also works in the web version (64:01–64:04)
Retry AI Relight in the browser; still blocked by the desktop-app requirement (64:04–65:17)
CapCut
Reject
Ask Kling AI to generate a Zanzibar schoolyard video clip (36:30–39:04)
Kling AI generates the requested video clip (39:45–41:17)
Judge the clip’s audio unusable and abandon it, out of credits to iterate further (41:17–42:06)
Kling AI
Ask ChatGPT to generate the same video clip (40:09–40:28)
ChatGPT explains it cannot generate video and offers a still image instead (40:28–41:03)
Review ChatGPT’s limitation and move to another tool (41:03–41:17)
ChatGPT
Appendix
Table 9: Real examples of Adopt , Modify , and Reject responses from Kolab , drawn from a music producer workflow, architecture workflow, and a video-production workflow respectively.
Figure 14: A webpage for professionals to align timestamps with steps and verify the generated graphs. Professionals first verify workflow steps and their corresponding video timestamps (top), and subsequently inspect the workflow graph for each step (bottom).
Experiment
# Clips
Mean ± std
Min
Max
Experiment 1a
692
14.0 ± 13.7
0.8
73.0
Experiment 1b
691
8.8 ± 11.7
0.2
72.0
Experiment 1c
638
37.9 ± 22.7
1.5
117.1
Experiment 1d: last1
114
10.1 ± 9.6
0.4
53.6
Experiment 1d: last2
114
20.7 ± 11.7
3.6
72.7
Experiment 1d: last3
114
32.3 ± 13.8
7.0
87.8
Appendix
Table 10: Length (in minutes) of the video clips given to the models in each experiment.
Macro
Per-class F1
Model
Setting
P
R
F1
Adopt
Modify
Reject
GPT-6-Astra
video-only
0.742
0.725
0.713
0.804
0.602
0.732
chat-only
0.648
0.673
0.658
0.777
0.644
0.552
video-graph-chat
0.699
0.742
0.673
0.812
0.480
0.727
Claude
video-only
0.736
0.648
0.676
0.843
0.556
0.630
chat-only
0.685
0.584
0.618
0.831
0.449
0.574
Appendix
Table 11: Result for Experiment 1a. Mean ± std over runs where a model has several; bold marks the best score within each model.
Macro
Per-class F1
Model
Setting
P
R
F1
Adopt
Modify
Reject
GPT-6-Astra
video-only
0.539 ± 0.003
0.524 ± 0.013
0.513 ± 0.003
0.721 ± 0.023
0.582 ± 0.098
0.235 ± 0.070
chat-only
0.365
0.371
0.368
0.695
0.408
0.000
video-graph-chat
0.546
0.512
0.512
0.696
0.592
0.250
Claude
video-only
0.610
0.483
0.492
0.790
0.434
0.250
chat-only
0.428
0.414
0.408
0.770
0.378
0.076
Appendix
Table 12: Result for Experiment 1b. Mean ± std over runs where a model has several; bold marks the best score within each model.
Overall
MAE by true rating
Model
Setting
MAE ↓
Exact acc.
Within-1
Spearman ρ
Low 1–2
Moderate 3
Mod-high 4
High 5
GPT-6-Astra
video-only
0.945 ± 0.070
0.244 ± 0.031
0.841 ± 0.032
0.379 ± 0.130
1.775 ± 0.391
0.684 ± 0.031
0.577 ± 0.150
0.988 ± 0.062
chat-only
1.246
0.183
0.611
0.107
1.714
0.636
0.759
1.480
video-graph-chat
0.740
0.365
0.887
0.263
1.429
0.917
0.373
0.802
Claude
video-only
0.858
0.266
0.865
0.136
1.675
0.742
0.275
1.131
chat-only
0.857
0.276
0.867
0.244
1.778
0.633
0.346
1.048
Appendix
Table 13: Result for Experiment 2a. Mean ± std over runs where a model has several; bold marks the best score within each model. Band columns give the MAE within each true-rating band.
Overall
MAE by true rating
Model
Setting
MAE ↓
Exact acc.
Within-1
Spearman ρ
Low 1–2
Moderate 3
Mod-high 4
High 5
GPT-6-Astra
video-only
0.912 ± 0.084
0.332 ± 0.067
0.805 ± 0.030
0.352 ± 0.101
1.853 ± 0.255
0.711 ± 0.077
0.710 ± 0.157
0.855 ± 0.105
chat-only
1.134
0.221
0.713
0.163
1.500
0.727
0.788
1.320
video-graph-chat
0.765
0.438
0.854
0.311
1.500
0.750
0.500
0.741
Claude
video-only
0.817
0.344
0.849
0.208
1.568
0.625
0.405
1.018
chat-only
0.973
0.267
0.809
0.137
1.789
1.059
0.610
1.052
Appendix
Table 14: Result for Experiment 2b. Mean ± std over runs where a model has several; bold marks the best score within each model. Band columns give the MAE within each true-rating band.
Figure 15: Professionals’ ratings of collaboration satisfaction and output quality across all steps.
Model
F1
Runs
GPT-6-Astra
0.713
1
Claude
0.699
1
Gemini-3.1-Pro-Preview
0.667
3
Gemini-3.8-Flash
0.620
3
Qwen3-VL-30B-A3B-Instruct
0.560
3
Qwen3-VL-32B-Instruct
0.374
3
Appendix
Table 15: F1 for Experiment 1a on the common set of 279 instances from 32 participants. The common set is limited by GPT-6-Astra, which has 279 valid outputs.
Model
F1
Runs
Claude
0.533
1
GPT-6-Astra
0.511
1
Gemini-3.1-Pro-Preview
0.399
3
Gemini-3.8-Flash
0.392
3
Qwen3-VL-30B-A3B-Instruct
0.375
3
Qwen3-VL-32B-Instruct
0.221
3
Appendix
Table 16: F1 for Experiment 1b on the common set of 349 instances from 37 participants. The common set is limited by GPT-6-Astra, which has 349 valid outputs. GPT-6-Astra’s score is based on its single run covering all 349 instances.
Figure 16: Mean rating (1–10) for each of the 26 AI-collaboration behavior items, sorted low to high (error bars: 1 SD). The dashed line marks the scale midpoint.
(a) Q1. After watching the session 1 recording again and looking at the extracted workflow, how natural do you think your workflow there compared to your general workflow?
Figure 34Figure 35Figure 36
(a) Q10. How many hours on average did it take or do you think it takes for you to finish the problem for session 2 without agentic AI’s help?
AI agents are reshaping the workspace, leading to drastic change of how humans work. Despite the considerable potential of human-agent collaboration both in preserving human agency and generating economic value, this paradigm remains largely absent from occupational task evaluation, hindered by the difficulty of gathering real human data and accounting for inter-human variability. We introduce CollabSkill, a framework for evaluating human-agent collaboration on real-world occupational tasks. CollabSkill pairs real human workers with AI agents on tasks matched to their occupational background, collecting data that capture the complexity of economically valuable tasks and the usage patterns of real workers. To account for inter-human variability, CollabSkill employs a Bayesian skill rating system to disentangle and quantify the skill contributions of both humans and AI agents. Drawing on over 1,500 prompts from 386 working sessions contributed by 93 human workers, our analysis yields insights on two fronts: on the agent side, rankings on CollabSkill diverge meaningfully from those of existing fully autonomous benchmarks where Codex leads, with Claude Code ranking first; on the human side, CollabSkill reveals that practical experience emerges as the primary driver of collaboration skill, with hands-on collaboration meaningfully shifting workers' AI literacy. Together, we hope CollabSkill enables the community to invest in systematic evaluation of human-agent collaboration and spurs development efforts aimed at building AI agents that genuinely augment human workers.
Yijia Shao, Zora Zhiruo Wang, Neel Ahuja +3
Stanford University · Carnegie Mellon University · University of Rochester +1
Existing multi-agent benchmarks primarily test in competitive settings, short-horizon interactions under 20 steps, or simply aggregate individual performance, failing to isolate and highlight genuine collaboration capabilities of LLM-based agents. We introduce AgentWorld, a benchmark of 100 human-annotated tasks (with 100 augmented variants) for evaluating long-horizon, multi-agent collaboration. Tasks span 50+ interaction rounds across a rich MMORPG sandbox and require 3-20 agents with asymmetric roles and abilities to coordinate through communication, joint planning, and resource sharing under a blackbox setting where each agent acts independently without access to others' internal states. To quantify collaboration effectiveness in addition to conventional binary task success, we propose Causal Collaboration Effectiveness (CCE), a graph-based metric that traces causal dependencies between agent actions and measures what fraction of a team's effort actually contributed to the outcome. Experiments with Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, and DeepSeek R1-70B show that even the best model achieves only 52.0% task success, with systematic failure modes including communication breakdowns, role confusion, and inability to maintain shared plans across rounds. AgentWorld is fully open-source.
Raphael Shu, Yusen Zhang, Young Min Cho +7
OpenAgents · Columbia University · University of Pennsylvania +2
Real-world professional desktop workflows in specialized creative and engineering software unfold over long horizons and often require human-in-the-loop coordination, where agents proactively seek necessary information and users provide additional instructions, clarifications, feedback, or corrections as the task progresses. Yet existing desktop GUI benchmarks mostly reduce this setting to short, simplified tasks with all user instructions provided upfront. To address this issue, we introduce DeskCraft, a desktop GUI benchmark targeting long horizon creative and engineering workflows and proactive human-agent collaboration. DeskCraft organizes tasks into a multilevel difficulty taxonomy, with long horizon tasks requiring over 50 execution steps, and covers professional creative software across design, video, audio, and 3D creation. Furthermore, DeskCraft formalizes human-agent collaboration into an interaction protocol covering mid-turn and post-turn exchanges. Mid-turn interaction captures both agent-initiated clarification under uncertainty and user-initiated interruption during execution, while post-turn interaction accommodates user-driven feedback after the agent signals completion, together spanning the full space of realistic collaboration patterns. We evaluate 18 proprietary and open source agents on 538 tasks and find that GPT-5.4 reaches 31.6% on standard tasks and 27.6% on interactive tasks. Further analyses reveal persistent failures in long horizon workflow delivery and proactive clarification. We will open-source all evaluation codes, tasks, and data at https://github.com/mrwwk/DeskCraft.
Wenkai Wang, Tao Xiong, Jingchen Ni +6
1Zhejiang University · 2Tsinghua University · 3Tencent +1