Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows

Shirley Anugrah Hayati; Patricia Anugrah Setiani; Ruizi Wang; Zhouyi Li; Khanh Chi Le; Young-Jun Lee; Dongyeop Kang

cs.AIOct 15, 2026

Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows

Authors: Shirley Anugrah Hayati, Patricia Anugrah Setiani, Ruizi Wang, Zhouyi Li, Khanh Chi Le, Young-Jun Lee, Dongyeop Kang

Organizations: Department of Computer Science, University of Minnesota · Department of Economics, University of Minnesota

Abstract

When humans collaborate with AI agents on long-horizon tasks, what happens to the agents' contributions? For solving long-horizon tasks, humans place greater importance on ongoing interaction with AI agents (e.g., revising plans) than on an agent's ability to complete the task in a single attempt. However, most agentic benchmarks have focused more on automation. Strong performance on such benchmarks does not imply effective real-world collaboration. Professional workflows offer a useful setting, as they often involve interdependent steps. To address this gap, we introduce Kolab, a new dataset of 115+ hours of screen recordings of 50 professionals from 40 occupations collaborating with agents on realistic, long-horizon problems. Kolab also contains participants' collaboration and output quality judgments. Our analysis reveals diverse collaboration patterns, highlighting the importance of when and how professionals coordinate with agents. A collaborative agent must do two things that automation benchmarks never test: anticipate whether experts will adopt, modify, or reject its contribution once it leaves the chat window to guide its next steps, and assess collaboration quality from the unfolding workflow without constant expert feedback. Our benchmark operationalizes both: (1) forecasting how professionals use agent contributions at diverse workflows, and (2) verifying their judgments of collaboration satisfaction and output quality. Findings show that frontier models, both agentic and non-agentic, overestimate collaboration with experts and struggle to distinguish between high- and low-quality collaboration. Longer workflow context does not consistently improve these predictions. As frontier agents cannot yet understand long-horizon collaboration with experts, we position Kolab as a valuable dataset for studying human-agent collaboration grounded in real professional workflows.

Figures & tables

Figure 1: An example human-agent collaborative workflow for an architect. Each step is a subgraph of human and agent actions; annotations link a source node, the target AI output it triggers, and the human response to it (here, Modify and Adopt ).

Dataset	Human Workflow	H-AI Collab.	Software Diversity	#Domains	Modalities
Agents’ Last Exam ( Sun et al., 2026b )	✗	✗	✓	55	Image, text, files
Workflow-GYM ( Zhu et al., 2026 )	✓	✗	✓	6	Image
WeaveBench ( Li et al., 2026 )	✗	✗	✓	8	Image, text
TheAgentCompany ( Xu et al., 2026 )	✗	✗	✗	6	Text
GDPVal ( Patwardhan et al., 2026 )	✗	✗	✓	44	Text, files
DeskCraft ( Wang et al., 2026 )	✗	✓	✓	5	Image

Table 1: Comparison of Kolab with long-horizon, multi-domain workflow benchmarks. “H-AI Collab.” indicates the presence of human-agent collaborative workflows; “Software Diversity” refers to tasks spanning multiple software applications; “Modalities” lists types of publicly released data.

Response	Source Node Action	Target AI Node Action	Current Node Action	Software
Adopt	Request Afrobeats/techno background music in ElevenLabs (67:33–68:30)	ElevenLabs generates Afrobeats instrumental candidates (69:50–70:04)	Capture the ElevenLabs track and import it into the DaVinci Resolve project (72:05–74:36)	ElevenLabs, DaVinci Resolve
Modify	Upload the existing shell plan and request a full room layout (24:10–25:28)	Claude delivers a dimensioned floor-plan layout for the lobby/storage circulation (51:21–56:20)	Rebuild walls/partitions in Revit, keeping the liked zoning but fixing lobby/storage circulation (58:40–72:59)	Revit
Reject	Ask Kling AI to generate a Zanzibar schoolyard video clip (36:30–39:04)	Kling AI generates the requested video clip (39:45–41:17)	Judge the clip’s audio unusable and abandon it, out of credits to iterate further (41:17–42:06)	Kling AI

Table 2: Real examples of Adopt , Modify , and Reject responses from Kolab , drawn from a music producer workflow, architecture workflow, and a video-production workflow respectively.

Video		Graph		Annotation (692 labels)
Total duration	115 hrs	Workflow steps	508	Adopt	65.9% (456)
Avg. duration/session	69 mins	Avg. steps/session	5.24	Modify	19.1% (132)
		Avg. nodes/step	8.68	Reject	15.0% (104)

Table 3: Summary statistics of Kolab for videos, workflow graphs, and human action annotations.

Panel of Figure 3 — Figure 3: Collaboration dynamics in Kolab . (a) Burstiness vs. solo lead-in; (b) workflow branching vs. collaboration rating; (c) timing of Adopt , Modify , and Reject responses across sectors. *** refers to $p<0.001$ and * is $p<0.05$ .

RQ1: AI adoption
Experiment Task	Observed Workflow	Response Visible?	Prediction
1a: Recognition	$T_{\mathrm{source}}^{\mathrm{start}}\rightarrow T_{\mathrm{adoption}}^{\mathrm{end}}$	✓	Adopt Modify Reject
1b: Forecasting	$T_{\mathrm{source}}^{\mathrm{start}}\rightarrow T_{\mathrm{AI}}^{\mathrm{end}}$	✗
1c: Step- $N$ forecasting	$T_{\mathrm{step}}^{\mathrm{start}}\rightarrow T_{\mathrm{AI}}^{\mathrm{end}}$	✗
1d: Workflow-level forecasting	$T_{\mathrm{workflow}}^{\mathrm{start}}\rightarrow T_{\mathrm{AI}}^{\mathrm{end}}$	✗
RQ2: Human rating

Table 4: Benchmark tasks.

T_{x}^{\mathrm{start}}

and

T_{x}^{\mathrm{end}}

denote the start and end timestamps of event or segment

x

, respectively;

x

can refer to the

\mathrm{source}

node, target

\mathrm{AI}

node, human

\mathrm{adoption}

node, or the full

\mathrm{workflow}

	1a: Recognition	1b: Forecasting			1b: Per-label F1
Model	F1	Prec.	Rec.	F1	Adopt	Modify	Reject
Majority-class	0.265	0.220	0.333	0.265	0.794	0.000	0.000
$\bullet$ Gemini 3.1 Pro	0.617 $\pm$ 0.007	0.461 $\pm$ 0.026	0.402 $\pm$ 0.011	0.391 $\pm$ 0.013	0.710 $\pm$ 0.012	0.333 $\pm$ 0.018	0.129 $\pm$ 0.031
$\bullet$ Gemini 3.8 Flash	0.587 $\pm$ 0.010	0.437 $\pm$ 0.016	0.383 $\pm$ 0.008	0.383 $\pm$ 0.009	0.758 $\pm$ 0.009	0.218 $\pm$ 0.027	0.172 $\pm$ 0.013
$\bullet$ GPT-6 Astra +	0.713	0.539 $\pm$ 0.003	0.524 $\pm$ 0.013	0.513 $\pm$ 0.003	0.721 $\pm$ 0.023	0.582 $\pm$ 0.098	0.235 $\pm$ 0.070
$\bullet$ Claude Opus 5.5	0.676	0.610	0.483	0.492	0.790	0.434	0.250

Table 5: Benchmarking performance for AI adoption recognition (Experiment 1a) and for forecasting (Experiment 1b), for video only, mean

\pm

std over 3 runs per model. The best result in each column is shaded and the second best result is bolded .

Figure 4: Change in macro-F1 as forecasting context grows. X-axis shows how many N previous steps are included as a context. Full refers to the start of the whole workflow.

	Exp. 2a: Collaboration			Exp. 2b: Output Quality
Model	MAE $\downarrow$	Within-1 $\uparrow$	Spearman $\rho\uparrow$	MAE $\downarrow$	Within-1 $\uparrow$	Spearman $\rho\uparrow$
Mode baseline	0.682	0.854	–	0.735	0.831	–
$\bullet$ Gemini 3.1 Pro	0.685 $\pm$ 0.006	0.850 $\pm$ 0.003	0.078 $\pm$ 0.057	0.764 $\pm$ 0.016	0.821 $\pm$ 0.006	0.046 $\pm$ 0.039
$\bullet$ Gemini 3.8 Flash	0.800 $\pm$ 0.029	0.819 $\pm$ 0.014	0.011 $\pm$ 0.003	0.792 $\pm$ 0.006	0.811 $\pm$ 0.002	0.042 $\pm$ 0.012
$\bullet$ GPT-6 Astra +	0.945 $\pm$ 0.070	0.841 $\pm$ 0.032	0.379 $\pm$ 0.130	0.912 $\pm$ 0.084	0.805 $\pm$ 0.030	0.352 $\pm$ 0.101
$\bullet$ Claude Opus 5.5 †	0.858	0.865	0.136	0.817	0.849	0.208

Table 6: Rating prediction for Experiments 2a and 2b, mean

\pm

std over 3 runs. The best result in each column is shaded ; the second best result is bolded .

Figure 5: Full results for seven input-modality combinations across the four benchmark tasks. V stands for video, G: graph, and C: chat log; “+” is a combination of modalities, “All” refers to using video, graph, and chat as the input. The best score in each row is in bold.

Figure 6: How each modality helps.

Figure 7: Distributions of model-generated labels for Modify and Reject cases in AI adoption forecasting.

Figure 8: Mean predicted rating vs. human rating for collaboration (Exp 2a) and output quality (Exp 2b).

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Item	Count
Coverage	Total participants	50
	O*NET sectors	10
	Occupations	40
Education	PhD	10
	Master’s	20
	Bachelor’s	15

Appendix

Table 7: Summary statistics of the 50 professionals.

Figure 9: Software used by participants in their day-to-day work, grouped by category.

Figure 10: Occupation distributions of Kolab . “Other” refers to three O*NET sectors: Life & Social Science, Sales & Related, and Healthcare.

Figure 11: Time spent in each recording session working with agentic AI versus the participant’s estimated time to solve the same problem without agentic AI.

Figure 12: Example workflow graph for one step of a participant’s session.

Collaboration Satisfaction
Rating	Session 1	Session 2	Total
5	117 (53.7%)	122 (56.2%)	239 (54.9%)
4	61 (28.0%)	68 (31.3%)	129 (29.7%)
3	28 (12.8%)	16 (7.4%)	44 (10.1%)
2	8 (3.7%)	7 (3.2%)	15 (3.4%)
1	4 (1.8%)	4 (1.8%)	8 (1.8%)

Appendix

Table 8: Distribution of collaboration satisfaction and output quality ratings, by session.

Figure 13: How three professionals (musician, marketing specialist, and attorney) adopt, modify, and reject agent outputs.This is based on the real data point in Kolab .

Response	Source Node Action	Target AI Node Action	Current Node Action	Software
Adopt	Request Afrobeats/techno background music in ElevenLabs (67:33–68:30)	ElevenLabs generates Afrobeats instrumental candidates (69:50–70:04)	Capture the ElevenLabs track and import it into the DaVinci Resolve project (72:05–74:36)	ElevenLabs, DaVinci Resolve
	Ask Claude to add a faith-based closing line to the voiceover script (13:07–13:41)	Claude adds the requested faith-based reference to the script (34:17–34:22)	Generate the final voiceover in ElevenLabs and import it into DaVinci Resolve (34:22–36:30)	Claude, ElevenLabs, DaVinci Resolve
Modify	Upload the existing shell plan and request a full room layout (24:10–25:28)	Claude delivers a dimensioned floor-plan layout for the lobby/storage circulation (51:21–56:20)	Rebuild walls/partitions in Revit, keeping the liked zoning but fixing lobby/storage circulation (58:40–72:59)	Revit
	Ask whether CapCut’s AI Relight tool requires the desktop app (63:49–63:55)	Claude explains AI Relight also works in the web version (64:01–64:04)	Retry AI Relight in the browser; still blocked by the desktop-app requirement (64:04–65:17)	CapCut
Reject	Ask Kling AI to generate a Zanzibar schoolyard video clip (36:30–39:04)	Kling AI generates the requested video clip (39:45–41:17)	Judge the clip’s audio unusable and abandon it, out of credits to iterate further (41:17–42:06)	Kling AI
	Ask ChatGPT to generate the same video clip (40:09–40:28)	ChatGPT explains it cannot generate video and offers a still image instead (40:28–41:03)	Review ChatGPT’s limitation and move to another tool (41:03–41:17)	ChatGPT

Appendix

Table 9: Real examples of Adopt , Modify , and Reject responses from Kolab , drawn from a music producer workflow, architecture workflow, and a video-production workflow respectively.

Panel of Figure 14 — Figure 14: A webpage for professionals to align timestamps with steps and verify the generated graphs. Professionals first verify workflow steps and their corresponding video timestamps (top), and subsequently inspect the workflow graph for each step (bottom).

Experiment	# Clips	Mean $\pm$ std	Min	Max
Experiment 1a	692	14.0 $\pm$ 13.7	0.8	73.0
Experiment 1b	691	8.8 $\pm$ 11.7	0.2	72.0
Experiment 1c	638	37.9 $\pm$ 22.7	1.5	117.1
Experiment 1d: last1	114	10.1 $\pm$ 9.6	0.4	53.6
Experiment 1d: last2	114	20.7 $\pm$ 11.7	3.6	72.7
Experiment 1d: last3	114	32.3 $\pm$ 13.8	7.0	87.8

Appendix

Table 10: Length (in minutes) of the video clips given to the models in each experiment.

		Macro			Per-class F1
Model	Setting	P	R	F1	Adopt	Modify	Reject
GPT-6-Astra	video-only	0.742	0.725	0.713	0.804	0.602	0.732
	chat-only	0.648	0.673	0.658	0.777	0.644	0.552
	video-graph-chat	0.699	0.742	0.673	0.812	0.480	0.727
Claude	video-only	0.736	0.648	0.676	0.843	0.556	0.630
	chat-only	0.685	0.584	0.618	0.831	0.449	0.574

Appendix

Table 11: Result for Experiment 1a. Mean

\pm

std over runs where a model has several; bold marks the best score within each model.

		Macro			Per-class F1
Model	Setting	P	R	F1	Adopt	Modify	Reject
GPT-6-Astra	video-only	0.539 $\pm$ 0.003	0.524 $\pm$ 0.013	0.513 $\pm$ 0.003	0.721 $\pm$ 0.023	0.582 $\pm$ 0.098	0.235 $\pm$ 0.070
	chat-only	0.365	0.371	0.368	0.695	0.408	0.000
	video-graph-chat	0.546	0.512	0.512	0.696	0.592	0.250
Claude	video-only	0.610	0.483	0.492	0.790	0.434	0.250
	chat-only	0.428	0.414	0.408	0.770	0.378	0.076

Appendix

Table 12: Result for Experiment 1b. Mean

\pm

std over runs where a model has several; bold marks the best score within each model.

		Overall				MAE by true rating
Model	Setting	MAE $\downarrow$	Exact acc.	Within-1	Spearman $\rho$	Low 1–2	Moderate 3	Mod-high 4	High 5
GPT-6-Astra	video-only	0.945 $\pm$ 0.070	0.244 $\pm$ 0.031	0.841 $\pm$ 0.032	0.379 $\pm$ 0.130	1.775 $\pm$ 0.391	0.684 $\pm$ 0.031	0.577 $\pm$ 0.150	0.988 $\pm$ 0.062
	chat-only	1.246	0.183	0.611	0.107	1.714	0.636	0.759	1.480
	video-graph-chat	0.740	0.365	0.887	0.263	1.429	0.917	0.373	0.802
Claude	video-only	0.858	0.266	0.865	0.136	1.675	0.742	0.275	1.131
	chat-only	0.857	0.276	0.867	0.244	1.778	0.633	0.346	1.048

Appendix

Table 13: Result for Experiment 2a. Mean

\pm

std over runs where a model has several; bold marks the best score within each model. Band columns give the MAE within each true-rating band.

		Overall				MAE by true rating
Model	Setting	MAE $\downarrow$	Exact acc.	Within-1	Spearman $\rho$	Low 1–2	Moderate 3	Mod-high 4	High 5
GPT-6-Astra	video-only	0.912 $\pm$ 0.084	0.332 $\pm$ 0.067	0.805 $\pm$ 0.030	0.352 $\pm$ 0.101	1.853 $\pm$ 0.255	0.711 $\pm$ 0.077	0.710 $\pm$ 0.157	0.855 $\pm$ 0.105
	chat-only	1.134	0.221	0.713	0.163	1.500	0.727	0.788	1.320
	video-graph-chat	0.765	0.438	0.854	0.311	1.500	0.750	0.500	0.741
Claude	video-only	0.817	0.344	0.849	0.208	1.568	0.625	0.405	1.018
	chat-only	0.973	0.267	0.809	0.137	1.789	1.059	0.610	1.052

Appendix

Table 14: Result for Experiment 2b. Mean

\pm

std over runs where a model has several; bold marks the best score within each model. Band columns give the MAE within each true-rating band.

Figure 15: Professionals’ ratings of collaboration satisfaction and output quality across all steps.

Model	F1	Runs
GPT-6-Astra	0.713	1
Claude	0.699	1
Gemini-3.1-Pro-Preview	0.667	3
Gemini-3.8-Flash	0.620	3
Qwen3-VL-30B-A3B-Instruct	0.560	3
Qwen3-VL-32B-Instruct	0.374	3

Appendix

Table 15: F1 for Experiment 1a on the common set of 279 instances from 32 participants. The common set is limited by GPT-6-Astra, which has 279 valid outputs.

Model	F1	Runs
Claude	0.533	1
GPT-6-Astra	0.511	1
Gemini-3.1-Pro-Preview	0.399	3
Gemini-3.8-Flash	0.392	3
Qwen3-VL-30B-A3B-Instruct	0.375	3
Qwen3-VL-32B-Instruct	0.221	3

Appendix

Table 16: F1 for Experiment 1b on the common set of 349 instances from 37 participants. The common set is limited by GPT-6-Astra, which has 349 valid outputs. GPT-6-Astra’s score is based on its single run covering all 349 instances.

Figure 16: Mean rating (1–10) for each of the 26 AI-collaboration behavior items, sorted low to high (error bars: 1 SD). The dashed line marks the scale midpoint.

(a) Q1. After watching the session 1 recording again and looking at the extracted workflow, how natural do you think your workflow there compared to your general workflow?

(a) Q10. How many hours on average did it take or do you think it takes for you to finish the problem for session 2 without agentic AI’s help?

(a) Q35. I use AI to execute a plan rather than determine one.

(a) Q38. AI helps me discover assumptions I hadn’t considered.

(a) Q39. What is the average hourly wage (in US Dollar) for a job similar to yours?

Figure 37: Q40. How serious would an undetected material error be in the tasks you did for this study?

Figure 38: Q41. Estimated financial/resource cost of an undetected error:

Figure 39: Q42. Did you notice any issues in the AI content during the tasks you did for this study?

(a) Q43. Estimated chance your final output still contains a material error:

Human-AI Collaboration

Human-AI collaboration studies how people and AI systems divide, coordinate, and adapt work to support decisions, creativity, and learning.

See all 440 papers

CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks
Apr 20, 2026Yijia Shao, Zora Zhiruo Wang, Neel Ahuja +3AI Agent Evaluation Human-in-the-Loop Evaluation
Stanford University · Carnegie Mellon University · University of Rochester +1
AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs
Sep 25, 2026Raphael Shu, Yusen Zhang, Young Min Cho +7Multi-Agent Collaboration AI Agent Benchmarks
OpenAgents · Columbia University · University of Pennsylvania +2
DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration
Jun 2, 2026Wenkai Wang, Tao Xiong, Jingchen Ni +6Computer-Use Agent Benchmarks GUI Agents
1Zhejiang University · 2Tsinghua University · 3Tencent +1

Apr 20, 2026J/K move · Enter open · S save

CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks

Yijia Shao, Zora Zhiruo Wang, Neel Ahuja, Yicheng Wang+2

Stanford University · Carnegie Mellon University · University of Rochester+1

AI Agent Evaluation Human-in-the-Loop Evaluation Human-AI Collaboration AI and Labor

Open paper AskPDF arXiv

AI agents are reshaping the workspace, leading to drastic change of how humans work. Despite the considerable potential of human-agent collaboration both in preserving human agency and generating economic value, this paradigm remains largely absent from occupational task evaluation, hindered by the difficulty of gathering real human data and accounting for inter-human variability. We introduce CollabSkill, a framework for evaluating human-agent collaboration on real-world occupational tasks. CollabSkill pairs real human workers with AI agents on tasks matched to their occupational background, collecting data that capture the complexity of economically valuable tasks and the usage patterns of real workers. To account for inter-human variability, CollabSkill employs a Bayesian skill rating system to disentangle and quantify the skill contributions of both humans and AI agents. Drawing on over 1,500 prompts from 386 working sessions contributed by 93 human workers, our analysis yields insights on two fronts: on the agent side, rankings on CollabSkill diverge meaningfully from those of existing fully autonomous benchmarks where Codex leads, with Claude Code ranking first; on the human side, CollabSkill reveals that practical experience emerges as the primary driver of collaboration skill, with hands-on collaboration meaningfully shifting workers' AI literacy. Together, we hope CollabSkill enables the community to invest in systematic evaluation of human-agent collaboration and spurs development efforts aimed at building AI agents that genuinely augment human workers.

Abstract

Human-AI Collaboration

Explore similar work

CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks

AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs

DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration