cs.HCSep 15, 2026
SaveAvailable but Unclaimed: An Empirical Study of Human-AI Synergy
Organizations: Aalto University, Espoo, Finland · University of Innsbruck, Innsbruck, Austria · Humboldt-Universität zu Berlin, Berlin, Germany · Ludwig-Maximilians-Universität München, Munich, Germany
Abstract
People increasingly reason with large language models (LLMs), yet complementary capabilities do not guarantee outperforming both components. In a between-subjects study, participants (N=535) solved a 40-item battery of matrix reasoning, mental rotation, syllogisms, and letter-string analogies, unaided or with GPT-5.6-Luna, Claude Opus 4.8, Gemini 3.6 Flash, or Kimi K3. Each assisted trial required consultation with the model. Each model answered every item alone 100 times under matched elicitation. The assisted-unaided accuracy difference increased with item-level LLM competence. Deference varied across tasks and increased with competence within tasks. Post-advice confidence distinguished correct from incorrect answers less strongly than unaided confidence. In a reference comparison, about half the increase in LLM accuracy carried through to assisted accuracy. How much of that accuracy gain reached participants differed across the models. These findings motivate evaluating LLMs in interaction with humans and designing support for selective deference that preserves independent reasoning.
Figures & tables
Figure 1. Assisted and assistant-alone accuracy differ across tasks. Each assistant’s own accuracy (left, 100-run chat-parity benchmark) and its teams’ accuracy (right), overall and per task. The dotted rule marks unaided accuracy and the dashed rule chance (undefined for letter strings). Small points show per-item competences. Bars show 95% Jeffreys intervals on pooled runs and trials, treating observations as independent. Five panels, overall and four tasks, with a small schematic of the task at the top of each and the task name at its foot. In each panel four lines run from an assistant-alone point on the left to a team point on the right, each marked with its provider's logo, and a cloud of small points behind the left column shows that assistant's per-item competence spread. A dotted line labelled unaided marks the unaided group. A dashed line labelled chance marks each task's chance level. The lines are not parallel. The assistants that gain most between the two columns are not the ones highest on the left. Thin vertical bars behind each mark show 95% intervals. They are narrow throughout and mostly hidden by the marks.
| Task type | Difficulty mix | Response format | |
|---|---|---|---|
| Matrix reasoning | 10 | 2 medium, 4 hard, 4 expert | Multiple choice, 8 options |
| Mental rotation | 10 | 2 medium, 4 hard, 4 expert | Multiple choice, 2 options |
| Syllogistic reasoning | 10 | 1 easy, 4 medium, 5 hard a | Multiple choice, 3 options |
| Letter-string analogies | 10 | 3 hard, 4 expert, 3 combination | Free text entry |
Table 1. Composition of the administered 40-item battery. Difficulty labels are the generator’s designed levels, not empirical difficulties.
Figure 2. One example item per task type; (d) is excerpted for legibility. (a) matrix item mat_025 (correct answer option 1), (b) mental-rotation item rot_014 ( depth rotation, correct answer same object ), (c) letter-string item ls_014 (correct answer > + ! $ & ), and (d) syllogism item syl_013 (correct answer cannot be determined ). The premises in (d) are embedded in record-keeping filler so the quantifiers must be inferred from prose. Four example items, one per task type. (a) A matrix-reasoning item shows a three-by-three grid of cells whose triangles vary in number, size, and orientation, with the final cell replaced by a question mark. Eight answer options appear below in two rows, numbered 1–4 and 5–8. (b) A mental-rotation item shows two three-dimensional shapes made of cubes side by side, labelled A and B, with B rotated in depth. The response options are same object or mirrored object. (c) A letter-string item shows two example transformations of short symbol strings followed by a new string whose transformed form must be typed as free text. (d) A syllogism item shows a short workplace narrative containing quantified statements, followed by a conclusion to judge as follows, does not follow, or cannot be determined.
Figure 3. The study application used a split interface, with the reasoning task and the response and confidence controls on the left and the assistant on the right. In no-AI the same pages were shown without the chat panel. Screenshot of the study web application in the assisted condition. The left side shows the current reasoning item with its answer controls and a progress indicator. The right side shows a chat panel with the AI assistant, a message history, and a text box for sending messages. A header carries the task name and a timer.
Figure 4. Every assisted group outperforms the unaided one, and none exceeds its own assistant by much. Distribution of individual accuracy over the 40 items, one row per group. Note. Each row is a kernel density over participants above jittered points, one per participant. Group sizes are given beside each row. The dashed rule and diamond in each assisted row mark that assistant’s own accuracy on the same 40 items without a person in the loop ( Section 4.4 ). Five stacked rows of individual accuracy, each a density curve above a strip of jittered points, one point per participant. The unaided curve is broad and centered near .53. The four assisted curves sit further right and are narrower, centered near .69, .71, .73 and .79, and their point strips are visibly sparser, the unaided group and gpt-5.6-luna rows carrying three times as many participants as the others. A dashed vertical line in each assisted row marks that assistant's own accuracy, falling inside the bulk of its own team's distribution in every case except kimi-k3, where it sits above it.
Figure 5. Teams track the assistant where it is strong and the unaided participant where the assistant is weak. Per-item accuracy for the unaided group, gpt-5.6-luna alone, and the group working with it, with the 40 items ordered by LLM competence (the proportion of one hundred benchmark runs answered correctly). Vertical lines are 95% credible intervals on the item mean. Letters classify each item by point estimates, not resolved differences, with S where it exceeded both of its parts (synergy, 14 items), A where it exceeded one (augmentation, 24), and I where it exceeded neither (interference, 2). The strip at the top left gives the three shares. Point-and-interval plot of three accuracy series across 40 items ordered by LLM competence. The LLM series rises from zero to one across the ordering, the unaided group series is roughly flat, and the assisted series follows the unaided group series on the left of the plot and the assistant series on the right. A letter above each item classifies the team on that item, and a small stacked strip at the top left summarizes the shares, with synergy on 35 percent of items, augmentation on 60 percent, and interference on 5 percent.
| Realized synergy | Synergy capture | Excess co-failure | ||||
|---|---|---|---|---|---|---|
| Task | Est. | 95% HDI | Est. | 95% HDI | Est. | 95% HDI |
| Mental rotation | [ , .027] | .071 | [ , .547] | .062 | [.035, .090] | |
| Matrices | .015 | [ , .078] | .643 | [.383, .888] | .074 | [.046, .101] |
| Letter strings | [ , ] | .484 | [.293, .689] | .013 | [ , .041] | |
| Syllogisms | [ , ] | .820 | [.641, .997] | [ , .023] | ||
Table 2. Realized synergy never resolves positive in any task, capture is uneven across them, and error overlap resolves only on the image-delivered tasks. Per-task estimates for the three item-level quantities, with tasks ordered by LLM competence. Note. compares assisted accuracy with the better component on each item. Capture is relative to independent-error headroom above unaided accuracy. It is modelled on an unbounded scale, so an interval may extend above 1, and mental rotation leaves almost no headroom, so its ratio is unidentified. Excess co-failure is the amount by which the pair failed together above the product of its two failure rates. Bracketed values are 95% credible intervals. An interval excluding zero is what we call resolved, and resolved estimates are set in bold throughout the paper’s tables. Matrices and mental rotation were delivered as images, letter strings and syllogisms as text. Every task contributes 10 items.
| Quantity | Estimate [95% HDI] |
|---|---|
| The effect function | |
| Effect at mean competence | .200 [.112, .277] |
| Slope on competence | .401 [.193, .612] |
| Effect where the assistant always fails | [ , ] |
| Zero crossing | .238 [ , .527] |
| Effect within competence band | |
Table 3. Benefits rise with competence, with uncertain effects in the lower competence bands. The effect function over the 160 item-by-assistant cells, the effect within each competence band, and the slope for each assistant separately. Note. The assisted–unaided difference is per-item team accuracy minus unaided accuracy on the same item. Cell-level quantities are on the accuracy scale. The trial-level slope is in log-odds. Resolved estimates are in bold . The zero crossing and the two lowest competence bands are unresolved, so these estimates do not identify a sharp decision threshold.
Figure 6. Assisted–unaided differences increase with LLM competence. The 160 item–assistant cells, coloured by assistant, with a pooled fit and 95% interval. Negative values indicate lower assisted accuracy. The pooled line summarizes the trend; assistant-specific slopes are reported separately. A scatter plot of the per-item assisted–unaided difference against the assistant's competence on that item, with one upward fitted line and its credible band. Points from all four assistants are interleaved rather than separated, with no assistant occupying a distinct region.
Figure 7. Assisted–unaided differences vary across observed score strata and assistant reliability. Assisted minus unaided accuracy by within-group score tertile, shown separately for items where the assistant is reliable ( ) and unreliable ( ). Note. Tertiles use observed scores within each group, not pretreatment ability. Contrasts come from separate trial-level models for reliable and unreliable items. Bars are 95% credible intervals, and solid fill marks a contrast whose interval excludes zero. The interval spans benefit and harm only in the lowest score tertile on unreliable items, marked by a hollow bar. Grouped bar chart of assisted minus unaided accuracy across three score tertiles. Reliable-item bars are positive throughout and shrink from low to high scores, from about .45 to .16. Unreliable-item bars are near zero and unresolved in the lowest stratum, clearly negative in the middle stratum at about , and slightly positive in the highest stratum at about .07.
Figure 8. Correct-to-incorrect switching is more frequent on low-competence items. Probability of switching away from a correct answer, and of switching to a correct answer, after the assistant replied, by LLM competence on the item. Note. Error bars are 95% credible intervals from models with crossed random effects for participant and item. Probabilities condition on initial correctness, not on disagreement with advice. The analysis covers the 35% of assisted trials with an answer recorded before the assistant’s first reply. The test each series carries is its change across the competence range, [ , ] for abandoning and .425 [.296, .550] for correcting. The two series do not cross at any level of competence. Two lines across four competence bands. The probability of giving up a correct answer falls from about .20 to about .02, while the probability of correcting a wrong answer rises from about .24 to about .68.
| Task | Trials | Advice correct | Deference | Overreliance | Discrim. |
|---|---|---|---|---|---|
| Syllogisms | 3,402 | .969 | .947 | .851 | .100 |
| Letter strings | 3,384 | .846 | .751 | .482 | .319 |
| Mental rotation | 3,335 | .652 | .902 | .806 | .154 |
| Matrices | 3,425 | .577 | .867 | .775 | .161 |
| All tasks | 13,546 | .751 | .859 | .743 | .165 |
| Widest intervals are on advice accuracy in matrices and mental rotation ( ), | |||||
Table 4. Extracted deference rates differ across tasks. Advice accuracy, deference and overreliance by task, on the 13,546 classified trials that carried committed advice, pooled over all four assistants and ordered by advice accuracy. Note. Advice correct is the share of trials on which the assistant’s extracted answer was right, and deference the share on which the participant submitted it. Overreliance is deference where the advice was wrong, and discrimination is deference to right minus deference to wrong advice. Task rows are model-based estimates. The pooled row gives observed trial shares. Interval widths are summarized below.
Figure 9. Stated confidence barely moves while accuracy swings by a third. Mean stated confidence and realized accuracy by LLM competence on the item, on trials following the intended sequence. Note. Post-consultation certainty ratings are divided by 100. Their gaps from accuracy are descriptive scale comparisons, not elicited probability calibration. The figures along the foot give the confidence–accuracy gap in each band, posterior median and 95% interval. Capped bars on the markers are 95% credible intervals. Two lines across four competence bands. Realized accuracy rises steeply from about .49 to about .86 while stated confidence stays almost flat near .75, so the vertical gap between them closes and then reverses.
Figure 10. Answer agreement and reference-policy gains describe different aspects of the interaction. (A) Submitted-answer agreement against the assistant’s item-level advantage, with task means. (B) Three reference policies, each as a gain over observed performance. Note. Task-level selection gains [ , ], item-level selection [ , ], and an independent-error oracle [ , ]. Two panels. The left panel plots submitted-answer agreement against the assistant's advantage across 160 item–assistant cells, with larger points marking task means. The right panel shows three reference rules as gains over the observed teams, each with an interval above zero.
| Assistant | LLM alone | Team | Realized synergy | Pass-through | |
|---|---|---|---|---|---|
| gemini-3.6-flash | 52 | .761 [.674, .845] | .781 [.720, .841] | [ , ] | 2.81 [2.37, 3.25] |
| claude-opus-4-8 | 61 | .735 [.653, .816] | .733 [.673, .791] | [ , ] | 2.21 [1.80, 2.61] |
| kimi-k3 | 56 | .730 [.648, .812] | .691 [.631, .750] | [ , ] | 1.39 [.89, 1.88] |
| gpt-5.6-luna (low) | 179 | .713 [.627, .796] | .711 [.651, .770] | [ , ] | 2.76 [2.39, 3.14] |
| unaided | 187 | — | .528 [.474, .583] | — | — |
| pooled | 348 | .735 [.633, .834] | .728 [.622, .828] | [ , ] | 2.46 [2.18, 2.75] |
Table 5. Solo and assisted point-estimate rankings differ. Pass-through also varies by group. Per-assistant solo accuracy, team accuracy, realized synergy, and pass-through, ordered by solo accuracy. Note. All values are posterior medians with 95% credible intervals. Resolved estimates are in bold . Pass-through is the slope of team correctness on the assistant’s item competence, in log-odds, partially pooled across assistants. Slopes are group-specific. Realized synergy compares the team with the better component separately on each item. The pooled row is over the 348 assisted participants. The unaided row is shown for reference only. The lower section gives the six pairwise differences in pass-through.
| Group | Confidence gap | Drop vs. unaided |
|---|---|---|
| unaided | [.092, .112] | — |
| gpt-5.6-luna (low) | [.051, .073] | [ , ] |
| claude-opus-4-8 | [.037, .075] | [ , ] |
| gemini-3.6-flash | [.037, .080] | [ , ] |
| kimi-k3 | [.035, .072] | [ , ] |
Table 6. Post-advice confidence separates right from wrong answers less in every assisted group than unaided, and the drop does not scale with what the assistant knows. The confidence gap between correct and incorrect answers, per group, and each assisted group’s drop against unaided. Note. The gap is the posterior difference in mean stated confidence (rescaled to – ) between correct and incorrect answers, from one model over every rated trial. Drops are differences from the unaided gap on the same scale. Bracketed values are 95% credible intervals. Every interval excludes zero, so all estimates are set in bold .
| Panel A. Who declines wrong advice (log-odds per ) | ||
|---|---|---|
| Predictor | Overreliance | Discrimination |
| Metacognitive discrimination ( ), | [ , ] | [ , .117] |
| Own-answer accuracy, | [ , ] | [1.559, 1.889] |
| Own-answer accuracy, disjoint trials, | [ , ] | — |
Table 7. Selected own-answer accuracy and metacognitive AUC are associated with resistance to wrong advice. Posterior effect of each candidate predictor, per standard deviation of that predictor. Note. Overreliance and discrimination are as defined in Table 4 . Both are trial-level outcomes, so Panel A is in log-odds. The two s give participants in the overreliance and discrimination fits, respectively. AUC coefficients adjust for shrunken own-work ability. Own-answer coefficients are unadjusted. The disjoint estimate uses declined correct-advice trials (median three per person) to predict separate wrong-advice trials. Capture is the realised share of headroom, a participant-level quantity, so Panel B is on the capture scale itself. Bracketed values are 95% credible intervals, is the posterior probability that the effect carries the sign of its median, and resolved estimates are in bold .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Level | Attributes varying | Relations drawn from | Additional constraint |
|---|---|---|---|
| easy | 2 | classic only | none |
| medium | 3 | classic only | none |
| hard | 4 | all four | at least one distribute-three |
| expert | 4 a | all four | at least two distribute-three |
Table 8. Difficulty levels for matrix items.
| Level | Angular disparity | Rotation axis (depth condition) |
|---|---|---|
| easy | world | |
| medium | world | |
| hard | world | |
| expert | world |
Table 9. Difficulty levels for mental rotation. The rotation axis applies to the depth condition only, and it switches with the angle, so no axis effect is separable from an effect of angular disparity. With the camera azimuth fixed at the switch is not itself a difficulty manipulation.
| Tier | Mental models | Mean estimated human accuracy |
|---|---|---|
| easy | 1 | |
| medium | 2 | |
| hard | 3 |
Table 10. Difficulty levels for syllogism items, and the mean of the hand-typed per-form human-accuracy estimates the computed tiers replaced (shown for validation only).
| Level | Alphabet | Step | Base length a | Grouping | Axes active |
|---|---|---|---|---|---|
| medium (baseline) | -letter | 1 | 5 | no | 0 |
| hard | -symbol | 2 | 5 | no | 1 |
| expert | -letter b | 2 | 9 | yes | 3 |
| combination | mixed c | 2 | 5 | yes | 3 c |
Table 11. Parameters of the letter-string levels. The medium level is shown as the baseline against which generalization axes are counted, but was not administered.
| Group | Age | Men | Women | Other | Bachelor+ | Weekly+ | Daily | OpenAI pref. | |
|---|---|---|---|---|---|---|---|---|---|
| unaided | 187 | 36.8 (10.9) | 53 | 47 | 1 | 73 | 77 | 37 | 55 |
| gpt-5.6-luna | 179 | 34.3 (10.8) | 54 | 45 | 1 | 68 | 84 | 47 | 56 |
| claude-opus-4-8 | 61 | 37.7 (10.8) | 52 | 48 | 0 | 77 | 85 | 54 | 56 |
| gemini-3.6-flash | 52 | 33.0 (10.0) | 33 | 67 | 0 | 71 | 79 | 42 | 67 |
| kimi-k3 | 56 | 32.3 (10.4) | 45 | 52 | 4 | 71 | 79 | 43 | 50 |
Table 12. Composition of the analyzed sample by group. Age is ( ). Remaining columns are percentages. Other combines non-binary, multiple categories, and undisclosed gender. AI use is the share reporting at least weekly, and daily, use. Provider is the share naming OpenAI as preferred provider.