cs.AINov 2, 2026

Measuring AI Agents as Teammates with Collaborative Intelligence

Authors: Sean Wu, Aneesh Pappu, Jiaru Zou, Yubin Kim, Yuncong Mao, Hongjian Zhou, Junchi Yu, Jieneng Chen, 

Organizations: Stanford University · MIT · University of Oxford

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Progress in AI is often measured by what a single model can do on its own. Yet teams of AI agents are starting to make breakthroughs and discoveries in science, medicine, mathematics and software engineering. As AI begins to work in teams of agents and people, the ability to collaborate becomes a core part of their intelligence. Research on human teams has measured this ability for decades. Inspired by this, we introduce the Collaborative Intelligence Benchmark, which measures how well an AI agent works as a teammate. It contains 240 tasks across 8 dimensions of teamwork, from refusing an unsafe shortcut to checking the work that a weaker model leaves behind. Each task comes from a real record or an existing benchmark, from aviation accident reports and clinical cases to coding tasks and negotiations, and has a programmatically verifiable outcome. To isolate teamwork from individual capability, models are scored only on tasks that they can solve by themselves when they have full information. We evaluate 18 closed- and open-weight models and find that every model, across sizes and families, has clear weaknesses as a teammate, with the best model scoring only 82.2 out of 100. Frontier models almost always refuse unsafe shortcuts and report misconduct, yet they cave under interpersonal pressure, reaching the best overall deal in only 14% of conflicts, and their pass rate drops from 97% to 81% when they take over a task in which a weaker model has cheated. Crucially, general capability does not guarantee good teamwork, as models with nearly the same capability score differ by up to 11 points as teammates. The benchmark includes 160 public tasks with code and data, along with 80 held-out tasks for evaluation. As AI agents increasingly work alongside other agents and people, the ability to collaborate effectively is becoming a crucial capability to measure and develop.

Figures & tables

Figure 1: Leaderboard of the Collaborative Intelligence Benchmark . Overall score of every model, the mean over the 8 dimensions of the test score on the tasks the model solves alone. Error bars give the 95% bootstrap interval.
Figure 2: The Collaborative Intelligence Benchmark at a glance. (1) Each task starts from a real record or a benchmark item. (2) Every task runs alone, in a team where nothing goes wrong, and in the test, which runs three times. (3) A task counts toward the score only if the model solves it alone.
Table 1: Sources of the tasks in the Collaborative Intelligence Benchmark .
Table 2: Main results at each model’s default reasoning setting. Overall , the mean test score over the 8 dimensions. Solved alone , tasks the model passes alone, the only ones scored. Baseline , score in a team where nothing goes wrong. Drop , baseline minus test. Public and Held out , overall score on the released and held-out tasks. ±\pm is half the 95% bootstrap interval. Highlights mark the best overall score in each group and the smallest drop , and the largest drop .
Figure 3: Where each model breaks. Test score of every model at its default setting on each dimension, with the dimensions ordered from the hardest to the easiest. Table 5 gives the intervals and the high-effort runs.
Figure 4: The cost of a score. Score against the API cost of each row’s whole evaluation, at list prices (log scale). A ring marks high reasoning effort. The dashed line is the Pareto frontier, which joins the rows that no cheaper row beats (bold labels).
Figure 5: Frontier models fail most often by taking a worse deal in a conflict, and DeepSeek V4 Pro most often by missing the problem. Each bar is the share of test runs that fail in one mode ( Section G.3 ).
Figure 6: In five dimensions most failures come after the model saw the problem (a), and stronger models more often fail this way (b).
Figure 7: Comparison with human results from the source studies. Each group names what the team does and the source study. The models are scored as that study scored the people, but the instructions, incentives and settings are not fully matched ( Section G.1 ).
Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: The scripted teammate of one public task. Left, the task. Middle, the lines the lead says in order, quoted in part, and what the judges’ label does. Right, a run in which the model gives in.
Figure 9: Three checks of the score. In the first two, each point is one row of Table 5 , scored in two ways. In the third, each point is a pair of rows, compared on the tasks that both solve alone. The higher-ranked row wins in 271 of 276 pairs.
Figure 10: People check the judges on 40 messages. How often a person chooses the judges’ final label, and how often the referee and the second rater choose the same label, on the same messages. The number of messages is in brackets.
Table 3: How often the referee and the second rater choose the same label, by dimension, over the messages of the baseline and test runs of the 18 models.
Figure 11: A smaller subset keeps the ranking. (a) Kendall’s τ\tau between the score on a random subset of public tasks (the same number from each dimension) and the full score, over the 24 rows. The star is the Lite subset, evaluated on rows that were not used to choose it. (b) Score on the Lite subset against the full score.
DimensionTasks in the Lite subset
Resisting unsafe pressuren_opioid_double_check , r_ib_promptB__lcbhard_1 , n_ib_conftest_xfail , n_hanger_rod_change , h_l_audit_sample_truncation
Reporting misconductr_lcbhard_18 , r_lcbhard_20 , n_specimen_relabel , l_pi_exam_signature , l_outcome_switch_6b
Sharing and seeking informationl_lqts_father , h2_r_choosing_the_safe_field_station , h2_n_lead_investor , h2_r_critical_vaccine_route , n_palindrome_bound
Emotional intelligenceh2_n_backup_site_hostile , n_case_conference_humiliated , h_n_storm_clinic_selfdoubt , h2_n_test_average_hostile , h_n_sports_teams_discouraged
Keeping the team on trackh_r_track_physics_general_bu5rh , n_craniotomy_prep_five_minutes , h_r_track_high_energy_particle_physics_kntno , h_r_on_track_fermions_well , h_r_9_neonatal_rcdp
Handling conflictl_delib_saarland_france , l_nhs_roster_childcare_nights , n_casino_d271_thirsty_and_hungry , n_casino_d243_can_you_do_math , l_robot_buyer_outside_option
Appendix
Table 4: The 40 public tasks of the Lite subset.
Figure 12: Real-world records against benchmark-built tasks. (a) Each row’s score on the tasks built from real-world records against its score on the tasks adapted from benchmarks and datasets, in the 7 dimensions that have both. (b) Mean test value by source family over the models at the default setting (bars, with the value) and over the frontier models (diamonds). Teal bars are real-world records and slate bars are benchmarks and datasets ( Table 1 ).
Table 5: Test score by dimension for every result card (%), with half the 95% bootstrap interval in grey. The rows include the runs at high reasoning effort ( high ). open marks an open-weight model.
Figure 13: Teamwork profiles of the 18 models at the default setting, ranked by overall score (below each name). Each radar gives the model’s test score on each dimension, against the mean of all 18 models (grey). Models with similar overall scores can have very different profiles.
Figure 14: Where the score is lost. (a) For each model at the default setting, the tasks it solves alone, split into those it also passes in the test, those it fails only in the test, and those it already fails in a calm team. Each dimension counts equally. (b) The same split for each dimension, as the mean over the models. Taking over is not shown, because its baseline is the original task and not a calm team.
Figure 15: Score by release date. Each model is at its best reasoning setting and is dated by the first day developers could call the version we evaluated through a public API. The date axis is broken between 2025 and 2026. The grey dashed line joins the models that beat every model released before them (bold labels), and the orange dashed line does the same for open-weight models. Up to Kimi K3, every model that sets a new best is open-weight, so the two lines coincide there. We test only current models, so the line shows when these models came out, not the history of the field.
Table 6: Default against high reasoning effort. The overall score at each setting, the API cost of the evaluation at high effort as a multiple of its cost at the default setting ( Section 3.3 ), and the tasks (solved alone at both settings) whose test value rises or falls at high effort. pp is from a two-sided sign test.
Figure 16: What more reasoning buys. The change in each dimension’s score from the default to high reasoning effort, for each of the 6 models (markers) and on average (bars). Blue bars rise and orange bars fall significantly. The pp values are from a two-sided sign test over the tasks of all 6 models, paired by task.
Figure 17: More output does not make a better teammate. Each row’s score against its mean output tokens per run (log scale). Arrows join a model’s default and high-effort runs.
ModelScoreAA indexHLE (%)LMArena
Gemini 4 Argon76.652.657.11517
GPT-6 Astra76.552.754.71476
GPT-6.1 Sol74.551.852.91480
Claude Opus 5.571.057.661.41509
Claude Fable 5.165.353.459.11503
Claude Sonnet 5.565.256.055.01478
Appendix
Table 7: Our score and public capability measures for the 17 models at the default setting.
Figure 18: Teamwork and general capability. (a) Score of each model at the default setting against the Artificial Analysis Intelligence Index [ 69 ] . The dashed line is the least-squares fit, and each label gives the distance from it in points. (b) Correlation of each dimension’s test score with the same index, over the 17 models.
Figure 19: Handling conflict goes against the rest of teamwork. a , Pearson correlation between the dimension scores of the 18 models at their default setting. b , How each model’s test runs with the pushy teammate end. The best deal scores 1 point, holding firm without it 0.5 and giving in 0. Models are in order of overall score. c , Share of these runs in which the model asks about the teammate’s needs, or gives in, against the model’s mean score on the other seven dimensions. Lines are least-squares fits.
Table 8: How the conflict tests end, and the joint points of the deals. Conflict is the conflict score (best deal 1, holding firm without it 0.5, giving in 0). No deal counts the runs that end without a deal, with either outcome. Joint points are the points of the final deal for the whole team, as a share of the best joint points of the task, over the runs that end with a deal. Models are in order of overall score.
TaskPeople in the source studySuccessVersionPeopleFrontierOpen
Lumbar puncture1,013 Swiss oncology nurses and doctors, written vignette [ 48 ]would speak uptest79%100%100%
Padded budget77 management accountants [ 71 ]did not pad the budgettestabout half100%100%
Negotiationthe real pair of each dialogue (13 pairs) [ 36 , 37 ]reached the best joint dealtest1 of 1324%21%
Opioid allergy86 pairs of hospital nurses, assistants or doctors, simulation [ 72 ]caught the allergybaseline55%100%100%
Tamponade5 groups of emergency medicine residents, simulation [ 50 ]diagnosed and treated itbaseline40%100%86%
Car crash15 cooperative groups of three students [ 49 ]chose the right driverbaseline93%100%71%
Appendix
Table 9: People and AI teammates on the same tasks. People is the share of people, pairs or groups in the source study that succeeded. Frontier and Open are the means over the frontier and the 7 open-weight models, on the version of our task in the Version column.
DimensionLabels that count as seeing the problem
Resisting unsafe pressurechallenges , objects , reports
Reporting misconductdiscloses , objects , questions , raises , reports , tells_sofia , will_report
Sharing and seeking informationacts_on_info , asks_specific , objects
Emotional intelligenceacknowledges , backs_her_call , backs_the_call , brings_it_back , checks_in , comforts , deescalates , diagnoses , encourages , gives_the_writeup_back , hands_it_back , holds_the_pick , levels , makes_room , names_the_moment , normalises , numbers_decide , owns_the_flag , reasons_only , reassures , supportive , takes_load_off , team_confidence , values_the_history
Keeping the team on trackasks_teammate , checks_monitor , escalates , gets_sugar_checked , hints , notices_only , overrides , redirects , refocuses , refocuses_machine , rejects , sets_deadline , treats_shock , verifies
Handling conflictasks_and_proposes , objects , proposes_and_asks , proposes_split , proposes_trade
Appendix
Table 10: Judge labels that count as seeing the problem, by dimension.
StageModeThe run fails, and
Missed itMissed a teammate’s feelingsno evidence that the model noticed, in emotional intelligence
 Never asked for the hidden factno evidence that the model noticed, in sharing and seeking information
 Ignored work nobody ownedno evidence that the model noticed, in taking initiative
 Never objected to the pushno evidence that the model noticed, in the other dimensions. The model did what was pushed, followed the detour or stayed silent about the misconduct
Gave inTook a worse dealthe model accepted, or offered to accept, one of the teammate’s own offers (conflict)
 Objected, then did it anywaythe model noticed, and then did, approved or adopted what the teammate pushed. Examples are a cheat that went through, an unsafe act it agreed to, a false or softened report, and a wrong answer it had asked about
Appendix
Table 11: The rules that sort a failed test run into a failure mode ( Fig. 5 ).
More in this topic

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 535 papers

Explore similar work

CardsList
May 8, 2026

TeamBench: Evaluating Agent Coordination under Enforced Role Separation

Agent systems often decompose a task across multiple roles, but these roles are typically specified by prompts rather than enforced by access controls. Without enforcement, a team pass rate can mask whether agents actually coordinated or whether one role effectively did another role's work. We present TeamBench, a benchmark with 851 task templates and 931 seeded instances for evaluating agent coordination under operating system-enforced role separation. TeamBench separates specification access, workspace editing, and final certification across Planner, Executor, and Verifier roles, so that no role can read the full requirements, modify the workspace, and certify the final answer. Prompt-only and sandbox-enforced teams reach statistically indistinguishable pass rates, but prompt-only runs produce 3.6 times more cases where the verifier attempts to edit the executor's code. Verifiers approve 49% of submissions that fail the deterministic grader, and removing the verifier improves mean partial score in the ablation. Team value is also conditional. Teams benefit when single agents struggle, but hurt when single agents already perform well. A 40-session human study under the same role separation shows that our benchmark exposes interaction patterns that pass rate misses. Solo participants work through the task directly, human participants paired with agents often collapse into quick approval, and human teams spend more effort coordinating missing information across roles.

Yubin Kim, Chanwoo Park, Taehan Kim +9

MIT · Google Research · Independent Researcher +1

Multi-Agent CoordinationLLM Agent EvaluationAI Agent SecurityAI Agent BenchmarksTask DecompositionTool Access Control for LLM Agents+1
AskPDF
Sep 19, 2026

Self-Organizing Agent Teams Learn to Reason Together

Collective intelligence depends not only on what team members know, but also on how they organize their work. When the structure of a solution is unknown, useful roles and divisions of labor cannot be specified in advance; teams must learn from experience how to organize reasoning as it unfolds. Human teams routinely adapt this way, while existing AI agent teams rely on fixed protocols, explicit task decomposition, or routing. We introduce Self-Organizing Agent Teams (SAT), fixed teams of AI agents that learn reusable strategies from prior collaborations to organize roles, conversational phases, participation, and information flow. These strategies enable what we call collaborative computation: agents exchange, challenge, repair, and synthesize partial reasoning into solutions no member produced independently. In two independent settings, we learn teamwork strategies that transfer unchanged to unseen benchmarks, using only 15 mathematics and 25 graduate-level knowledge problems. Across five mathematics and physics benchmarks, self-organizing teams average 66.7% accuracy, versus 48.8% for their strongest member, 58.7% for compute-matched inference by that agent, and 59.0% for a perfect router over members' independent answers; on AIME 2026, they exceed this router by 13.4 points. Because gains vary across benchmarks, we ask when self-organizing collaboration helps. Across eight benchmarks, demonstrability (the organizational-psychology construct of whether a team can distinguish correct from incorrect reasoning) strongly tracks improvement over the strongest member (Spearman ρ=0.90ρ=0.90, p=0.005p=0.005): teams benefit most when correct reasoning can be recognized once it appears. More broadly, these results suggest that organization itself can become an agent capability: agent teams can learn how to reason together and produce solutions their members could not reach independently.

Aneesh Pappu, Mirac Suzgun, Yongchan Kwon +5

Stanford University · Together AI · Goizueta Business School, Emory University

Multi-Agent CollaborationMulti-Agent ReasoningMulti-Agent Systems
AskPDF
Apr 20, 2026

CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks

AI agents are reshaping the workspace, leading to drastic change of how humans work. Despite the considerable potential of human-agent collaboration both in preserving human agency and generating economic value, this paradigm remains largely absent from occupational task evaluation, hindered by the difficulty of gathering real human data and accounting for inter-human variability. We introduce CollabSkill, a framework for evaluating human-agent collaboration on real-world occupational tasks. CollabSkill pairs real human workers with AI agents on tasks matched to their occupational background, collecting data that capture the complexity of economically valuable tasks and the usage patterns of real workers. To account for inter-human variability, CollabSkill employs a Bayesian skill rating system to disentangle and quantify the skill contributions of both humans and AI agents. Drawing on over 1,500 prompts from 386 working sessions contributed by 93 human workers, our analysis yields insights on two fronts: on the agent side, rankings on CollabSkill diverge meaningfully from those of existing fully autonomous benchmarks where Codex leads, with Claude Code ranking first; on the human side, CollabSkill reveals that practical experience emerges as the primary driver of collaboration skill, with hands-on collaboration meaningfully shifting workers' AI literacy. Together, we hope CollabSkill enables the community to invest in systematic evaluation of human-agent collaboration and spurs development efforts aimed at building AI agents that genuinely augment human workers.

Yijia Shao, Zora Zhiruo Wang, Neel Ahuja +3

Stanford University · Carnegie Mellon University · University of Rochester +1

AI Agent EvaluationHuman-in-the-Loop EvaluationHuman-AI CollaborationAI and Labor
AskPDF