cs.AINov 2, 2026
SaveMeasuring AI Agents as Teammates with Collaborative Intelligence
Organizations: Stanford University · MIT · University of Oxford
Abstract
Progress in AI is often measured by what a single model can do on its own. Yet teams of AI agents are starting to make breakthroughs and discoveries in science, medicine, mathematics and software engineering. As AI begins to work in teams of agents and people, the ability to collaborate becomes a core part of their intelligence. Research on human teams has measured this ability for decades. Inspired by this, we introduce the Collaborative Intelligence Benchmark, which measures how well an AI agent works as a teammate. It contains 240 tasks across 8 dimensions of teamwork, from refusing an unsafe shortcut to checking the work that a weaker model leaves behind. Each task comes from a real record or an existing benchmark, from aviation accident reports and clinical cases to coding tasks and negotiations, and has a programmatically verifiable outcome. To isolate teamwork from individual capability, models are scored only on tasks that they can solve by themselves when they have full information. We evaluate 18 closed- and open-weight models and find that every model, across sizes and families, has clear weaknesses as a teammate, with the best model scoring only 82.2 out of 100. Frontier models almost always refuse unsafe shortcuts and report misconduct, yet they cave under interpersonal pressure, reaching the best overall deal in only 14% of conflicts, and their pass rate drops from 97% to 81% when they take over a task in which a weaker model has cheated. Crucially, general capability does not guarantee good teamwork, as models with nearly the same capability score differ by up to 11 points as teammates. The benchmark includes 160 public tasks with code and data, along with 80 held-out tasks for evaluation. As AI agents increasingly work alongside other agents and people, the ability to collaborate effectively is becoming a crucial capability to measure and develop.
Figures & tables
Figure 1: Leaderboard of the Collaborative Intelligence Benchmark . Overall score of every model, the mean over the 8 dimensions of the test score on the tasks the model solves alone. Error bars give the 95% bootstrap interval.
Figure 2: The Collaborative Intelligence Benchmark at a glance. (1) Each task starts from a real record or a benchmark item. (2) Every task runs alone, in a team where nothing goes wrong, and in the test, which runs three times. (3) A task counts toward the score only if the model solves it alone.
Table 1: Sources of the tasks in the Collaborative Intelligence Benchmark .
Table 2: Main results at each model’s default reasoning setting. Overall , the mean test score over the 8 dimensions. Solved alone , tasks the model passes alone, the only ones scored. Baseline , score in a team where nothing goes wrong. Drop , baseline minus test. Public and Held out , overall score on the released and held-out tasks. is half the 95% bootstrap interval. Highlights mark the best overall score in each group and the smallest drop , and the largest drop .
Figure 3: Where each model breaks. Test score of every model at its default setting on each dimension, with the dimensions ordered from the hardest to the easiest. Table 5 gives the intervals and the high-effort runs.
Figure 4: The cost of a score. Score against the API cost of each row’s whole evaluation, at list prices (log scale). A ring marks high reasoning effort. The dashed line is the Pareto frontier, which joins the rows that no cheaper row beats (bold labels).
Figure 5: Frontier models fail most often by taking a worse deal in a conflict, and DeepSeek V4 Pro most often by missing the problem. Each bar is the share of test runs that fail in one mode ( Section G.3 ).
Figure 6: In five dimensions most failures come after the model saw the problem (a), and stronger models more often fail this way (b).
Figure 7: Comparison with human results from the source studies. Each group names what the team does and the source study. The models are scored as that study scored the people, but the instructions, incentives and settings are not fully matched ( Section G.1 ).
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: The scripted teammate of one public task. Left, the task. Middle, the lines the lead says in order, quoted in part, and what the judges’ label does. Right, a run in which the model gives in.
Figure 9: Three checks of the score. In the first two, each point is one row of Table 5 , scored in two ways. In the third, each point is a pair of rows, compared on the tasks that both solve alone. The higher-ranked row wins in 271 of 276 pairs.
Figure 10: People check the judges on 40 messages. How often a person chooses the judges’ final label, and how often the referee and the second rater choose the same label, on the same messages. The number of messages is in brackets.
Table 3: How often the referee and the second rater choose the same label, by dimension, over the messages of the baseline and test runs of the 18 models.
Figure 11: A smaller subset keeps the ranking. (a) Kendall’s between the score on a random subset of public tasks (the same number from each dimension) and the full score, over the 24 rows. The star is the Lite subset, evaluated on rows that were not used to choose it. (b) Score on the Lite subset against the full score.
| Dimension | Tasks in the Lite subset |
|---|---|
| Resisting unsafe pressure | n_opioid_double_check , r_ib_promptB__lcbhard_1 , n_ib_conftest_xfail , n_hanger_rod_change , h_l_audit_sample_truncation |
| Reporting misconduct | r_lcbhard_18 , r_lcbhard_20 , n_specimen_relabel , l_pi_exam_signature , l_outcome_switch_6b |
| Sharing and seeking information | l_lqts_father , h2_r_choosing_the_safe_field_station , h2_n_lead_investor , h2_r_critical_vaccine_route , n_palindrome_bound |
| Emotional intelligence | h2_n_backup_site_hostile , n_case_conference_humiliated , h_n_storm_clinic_selfdoubt , h2_n_test_average_hostile , h_n_sports_teams_discouraged |
| Keeping the team on track | h_r_track_physics_general_bu5rh , n_craniotomy_prep_five_minutes , h_r_track_high_energy_particle_physics_kntno , h_r_on_track_fermions_well , h_r_9_neonatal_rcdp |
| Handling conflict | l_delib_saarland_france , l_nhs_roster_childcare_nights , n_casino_d271_thirsty_and_hungry , n_casino_d243_can_you_do_math , l_robot_buyer_outside_option |
Table 4: The 40 public tasks of the Lite subset.
Figure 12: Real-world records against benchmark-built tasks. (a) Each row’s score on the tasks built from real-world records against its score on the tasks adapted from benchmarks and datasets, in the 7 dimensions that have both. (b) Mean test value by source family over the models at the default setting (bars, with the value) and over the frontier models (diamonds). Teal bars are real-world records and slate bars are benchmarks and datasets ( Table 1 ).
Table 5: Test score by dimension for every result card (%), with half the 95% bootstrap interval in grey. The rows include the runs at high reasoning effort ( high ). open marks an open-weight model.
Figure 13: Teamwork profiles of the 18 models at the default setting, ranked by overall score (below each name). Each radar gives the model’s test score on each dimension, against the mean of all 18 models (grey). Models with similar overall scores can have very different profiles.
Figure 14: Where the score is lost. (a) For each model at the default setting, the tasks it solves alone, split into those it also passes in the test, those it fails only in the test, and those it already fails in a calm team. Each dimension counts equally. (b) The same split for each dimension, as the mean over the models. Taking over is not shown, because its baseline is the original task and not a calm team.
Figure 15: Score by release date. Each model is at its best reasoning setting and is dated by the first day developers could call the version we evaluated through a public API. The date axis is broken between 2025 and 2026. The grey dashed line joins the models that beat every model released before them (bold labels), and the orange dashed line does the same for open-weight models. Up to Kimi K3, every model that sets a new best is open-weight, so the two lines coincide there. We test only current models, so the line shows when these models came out, not the history of the field.
Table 6: Default against high reasoning effort. The overall score at each setting, the API cost of the evaluation at high effort as a multiple of its cost at the default setting ( Section 3.3 ), and the tasks (solved alone at both settings) whose test value rises or falls at high effort. is from a two-sided sign test.
Figure 16: What more reasoning buys. The change in each dimension’s score from the default to high reasoning effort, for each of the 6 models (markers) and on average (bars). Blue bars rise and orange bars fall significantly. The values are from a two-sided sign test over the tasks of all 6 models, paired by task.
Figure 17: More output does not make a better teammate. Each row’s score against its mean output tokens per run (log scale). Arrows join a model’s default and high-effort runs.
| Model | Score | AA index | HLE (%) | LMArena |
|---|---|---|---|---|
| Gemini 4 Argon | 76.6 | 52.6 | 57.1 | 1517 |
| GPT-6 Astra | 76.5 | 52.7 | 54.7 | 1476 |
| GPT-6.1 Sol | 74.5 | 51.8 | 52.9 | 1480 |
| Claude Opus 5.5 | 71.0 | 57.6 | 61.4 | 1509 |
| Claude Fable 5.1 | 65.3 | 53.4 | 59.1 | 1503 |
| Claude Sonnet 5.5 | 65.2 | 56.0 | 55.0 | 1478 |
Table 7: Our score and public capability measures for the 17 models at the default setting.
Figure 18: Teamwork and general capability. (a) Score of each model at the default setting against the Artificial Analysis Intelligence Index [ 69 ] . The dashed line is the least-squares fit, and each label gives the distance from it in points. (b) Correlation of each dimension’s test score with the same index, over the 17 models.
Figure 19: Handling conflict goes against the rest of teamwork. a , Pearson correlation between the dimension scores of the 18 models at their default setting. b , How each model’s test runs with the pushy teammate end. The best deal scores 1 point, holding firm without it 0.5 and giving in 0. Models are in order of overall score. c , Share of these runs in which the model asks about the teammate’s needs, or gives in, against the model’s mean score on the other seven dimensions. Lines are least-squares fits.
Table 8: How the conflict tests end, and the joint points of the deals. Conflict is the conflict score (best deal 1, holding firm without it 0.5, giving in 0). No deal counts the runs that end without a deal, with either outcome. Joint points are the points of the final deal for the whole team, as a share of the best joint points of the task, over the runs that end with a deal. Models are in order of overall score.
| Task | People in the source study | Success | Version | People | Frontier | Open |
|---|---|---|---|---|---|---|
| Lumbar puncture | 1,013 Swiss oncology nurses and doctors, written vignette [ 48 ] | would speak up | test | 79% | 100% | 100% |
| Padded budget | 77 management accountants [ 71 ] | did not pad the budget | test | about half | 100% | 100% |
| Negotiation | the real pair of each dialogue (13 pairs) [ 36 , 37 ] | reached the best joint deal | test | 1 of 13 | 24% | 21% |
| Opioid allergy | 86 pairs of hospital nurses, assistants or doctors, simulation [ 72 ] | caught the allergy | baseline | 55% | 100% | 100% |
| Tamponade | 5 groups of emergency medicine residents, simulation [ 50 ] | diagnosed and treated it | baseline | 40% | 100% | 86% |
| Car crash | 15 cooperative groups of three students [ 49 ] | chose the right driver | baseline | 93% | 100% | 71% |
Table 9: People and AI teammates on the same tasks. People is the share of people, pairs or groups in the source study that succeeded. Frontier and Open are the means over the frontier and the 7 open-weight models, on the version of our task in the Version column.
| Dimension | Labels that count as seeing the problem |
|---|---|
| Resisting unsafe pressure | challenges , objects , reports |
| Reporting misconduct | discloses , objects , questions , raises , reports , tells_sofia , will_report |
| Sharing and seeking information | acts_on_info , asks_specific , objects |
| Emotional intelligence | acknowledges , backs_her_call , backs_the_call , brings_it_back , checks_in , comforts , deescalates , diagnoses , encourages , gives_the_writeup_back , hands_it_back , holds_the_pick , levels , makes_room , names_the_moment , normalises , numbers_decide , owns_the_flag , reasons_only , reassures , supportive , takes_load_off , team_confidence , values_the_history |
| Keeping the team on track | asks_teammate , checks_monitor , escalates , gets_sugar_checked , hints , notices_only , overrides , redirects , refocuses , refocuses_machine , rejects , sets_deadline , treats_shock , verifies |
| Handling conflict | asks_and_proposes , objects , proposes_and_asks , proposes_split , proposes_trade |
Table 10: Judge labels that count as seeing the problem, by dimension.
| Stage | Mode | The run fails, and |
| Missed it | Missed a teammate’s feelings | no evidence that the model noticed, in emotional intelligence |
| Never asked for the hidden fact | no evidence that the model noticed, in sharing and seeking information | |
| Ignored work nobody owned | no evidence that the model noticed, in taking initiative | |
| Never objected to the push | no evidence that the model noticed, in the other dimensions. The model did what was pushed, followed the detour or stayed silent about the misconduct | |
| Gave in | Took a worse deal | the model accepted, or offered to accept, one of the teammate’s own offers (conflict) |
| Objected, then did it anyway | the model noticed, and then did, approved or adopted what the teammate pushed. Examples are a cheat that went through, an unsafe act it agreed to, a false or softened report, and a wrong answer it had asked about |
Table 11: The rules that sort a failed test run into a failure mode ( Fig. 5 ).
More in this topic
AI Agent Evaluation
AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.
See all 539 papers