Better together? A performance theory of human-machine skill policies under generalization difficulty

Riccardo Zanardelli

cs.HCOct 12, 2026

Better together? A performance theory of human-machine skill policies under generalization difficulty

Authors: Riccardo Zanardelli

Abstract

Organizations pairing a human and a machine on a task decide, often by default, which executes each instance and what happens when both do. We account for what this attains under the squared rule, with performance distributed over instances to capture generalization difficulty. A gross stake lies between the skills' mean and best per instance. It splits at the best single skill into a systematic part, known from separate assessments and secured by commitment, and a residual, the expected value of perfect information on which skill each instance favors, known only from paired observation and split further by classes. Skills set the stake; workflows decide the take, via two channels. Routing directs work, weighting or selecting responses; with binary truth it is capped by the best skill per instance. Reading responses as joint evidence is not routing; what they jointly reveal caps it. Synergy lets skills interact before responding; bounded by no skill either way, its lift over independent responses must clear a break-even and can fall below one. As algorithms, arrangements form a ladder stating what each reads, holds and records, and a procedure whose checks keep earlier gains when class means are known and lose little on finite records given a margin. The theory extends to many skills, each worth what it adds where it beats all others held, and to competing skills, where at parity a single failure's allowed cost decides. In a public chest-radiograph study with 227 radiologists, they are near parity with the model and the stake is mostly residual; assisted reading, no better on average originally, returned less than averaging both responses; recalibrating the model alone beats any deployable arrangement of raw responses. Organizations can learn from their records what is at stake, which arrangement collects it and what evidence each step needs; pricing costs and errors is future work.

Figures & tables

Panel of Figure 1 — Figure 1: A skill policy $\pi$ , which an organization runs as a workflow. The policy holds the available set $\mathcal{S}=\{i,j,\dots\}$ ; on each instance the engagement criterion decides which of its skills to engage, that is, to execute, so a skill can be available and not engaged on the instance drawn. Without a protocol, the engaged skills run separately and their responses meet only in the aggregation, which weights them or selects one (solid connectors); the policy routes where the criterion or the aggregation directs work between the skills, by class or by instance, through assignment, ensembling or selection. With a protocol $\rho$ (dashed), the protocol surrounds the engaged skills, lets what one of them produces enter another’s execution, fixes who is shown what and in what order, and itself returns the delivered response: the policy operates synergy, and routes as well where the engaged skills or their weights are set per instance. The scoring rule of the setting compares the delivered response $\hat{r}_{\pi}$ with the correct response $r$ to yield the performance $\theta_{\pi}$ .

Panel of Figure 2 — Figure 2: Routing and synergy as flows. Above: a policy that engages one skill on every instance, by ignorance or by an ex ante commitment, has nothing to augment, and its criterion makes no choice among skills at instance level. Below: augmentation requires at least two skills available and the use of at least one of them based on a criterion that runs at instance level. A policy routes when information about the class or the instance, read or inferred from signals, directs which skills are engaged or how their contributions are weighted; it operates synergy when a protocol $\rho$ (frame) surrounds the engaged skills and returns the joint response after the exchanges among them. Where no synergy, the criterion may engage one of the skills or several (and/or), so that the criterion can implement assignment, ensembling and selection. Without routing the criterion is constant; without synergy the skills run separately and an aggregation forms the response. The quadrant of neither holds the fixed use of several skills, pooling with fixed weights, which buys insurance against a wrong commitment rather than augmentation. Routing without a protocol is bounded by the best response on every instance where the truth is an endpoint; a protocol is bounded by none of the skills, and is measured by its lift.

Panel of Figure 3 — Figure 3: The anchors and the decomposition of the stake, as defined in Equation 2 , for an available set of any size, drawn with two skills. (a) On one instance: $m$ is the mean of the skills’ performances, $\mathcal{M}$ the best of them, and the gap $\Delta=\mathcal{M}-m$ . (b) Over the instance space: the distributions of skills $i$ (dotted) and $j$ (dash-dotted), of the mean anchor $m$ (dashed) and of the routed anchor $\mathcal{M}$ (solid, hatched), drawn for two independent Beta-distributed skills, independence being what lets the distributions of $m$ and $\mathcal{M}$ follow from the skills’ own (Appendix A ); below, their expectations magnified, with $S$ between $\mathbb{E}[m]$ and $b$ and $R$ between $b$ and $\mathbb{E}[\mathcal{M}]$ .

Panel of Figure 4 — Figure 4: The full decomposition of the stake, along expected performance. Between the mean anchor $\mathbb{E}[m]$ and the routed anchor $\mathbb{E}[\mathcal{M}]$ , the gross stake $G$ splits at the informed baseline $b$ into the systematic component $S$ and the residual $R$ (Equation 2 ); where instances fall into classes, the residual splits again, at $\sum_{k}p_{k}b_{k}$ , into the part $R_{\mathrm{class}}$ that sending each class to its incumbent reaches and the part $R_{\mathrm{inst}}$ that only a policy reading the instance itself can (Proposition 7 ). Below the axis, what attains each anchor and what evidence identifies each component. Positions are schematic.

						Score
$r$	Flags	Share	Responses	Pooled	Combined	Each skill	Pooling	$\mathcal{M}$	Combination
1	2	0.081	0.333, 0.333	0.333	0.692	0.556	0.556	0.556	0.905
1	1	0.018	0.333, 0.014	0.174	0.059	0.291	0.317	0.556	0.114
1	0	0.001	0.014, 0.014	0.014	0.002	0.027	0.027	0.027	0.003
0	2	0.036	0.333, 0.333	0.333	0.692	0.889	0.889	0.889	0.521
0	1	0.288	0.333, 0.014	0.174	0.059	0.944	0.970	1.000	0.997

Table 1: A combination outside the hull. Two calibrated skills, independent given the truth, on instances one in ten of which is positive; each flags a positive instance with probability

0.9

and a negative one with probability

0.2

. For each truth and number of flags: the share of instances, the two responses, the pooled and the combined response, and the scores under the squared rule, the first of them the mean of the two skills’ scores. The last row is the expectation

Figure 5: What each arrangement of the worked example takes, component by component (Corollary 3 ): two skills, two classes and four instances, scored under the squared rule (Table 2 ). At the top, the stake and its three components. Each row is a policy. In the left panel the lanes are the components, each filled from its start with the part the policy takes, and a cross-hatched bar left of

\mathbb{E}[m]

marks a share below zero; the triple gives the shares. The right panel shows what the policy returns that is not stake, the dividend of combining responses (hatched) and the lift of the consensus protocol (dotted), and the take,

\mathbb{E}[\theta_{\pi}]-\mathbb{E}[m]

. Assignment on a class predicted correctly with probability

a=3/4

resolves neither class and is shown under both crediting rules (Figure 18 ). The class-1 router resolves class 1 but not class 2: it takes the whole of

R_{\mathrm{inst}}

, falls

S

below the mean anchor at the level of the class, and ends at

b

			Responses				Performances
Instance	Class $k$	$r$	$\hat{r}_{i}$	$\hat{r}_{j}$	$\hat{r}_{\bar{w}}$	$\hat{r}_{\rho}$	$\theta_{i}$	$\theta_{j}$	$m$	$\mathcal{M}$	$\theta_{\bar{w}}$	$\theta(\hat{r}_{\rho},r)$
1	1	1	0.4	1.0	0.7	0.9	0.64	1.00	0.82	1.00	0.91	0.99
2	1	0	0.3	0.5	0.4	0.2	0.91	0.75	0.83	0.91	0.84	0.96
Mean over class 1							0.775	0.875	0.825	0.955	0.875	0.975
3	2	1	1.0	0.4	0.7	0.9	1.00	0.64	0.82	1.00	0.91	0.99
4	2	0	0.3	0.3	0.3	0.5	0.91	0.91	0.91	0.91	0.91	0.75

Table 2: A worked example: four instances, two classes, two skills and one protocol. Responses are probabilities that the instance is positive; performances are their scores under the squared rule,

\theta=1-(\hat{r}-r)^{2}

Figure 6: The decomposition of Equation ( 2 ) on a grid of synthetic skill pairs, both skills Beta-distributed and independent on each instance, parameterized by their mean and a concentration

\nu

, with

\alpha=\mathbb{E}[\theta]\,\nu

and

\beta=(1-\mathbb{E}[\theta])\,\nu

;

\mathbb{E}[\theta_{H}]=0.40

in every panel, so the informed baseline is

b=0.40

throughout. Rows set the gap

\mathbb{E}[\theta_{H}]-\mathbb{E}[\theta_{M}]

and therefore

S=\mathrm{gap}/2

(Proposition 1 ); columns set the concentration, larger

\nu

giving tighter distributions. gray curves are the exact densities of the two skills (

\theta_{H}

solid,

\theta_{M}

dotted), and the shaded densities those of the anchors,

m

with a dashed outline and

\mathcal{M}

with a solid one; vertical lines mark

\mathbb{E}[m]

(dotted),

b

(solid) and

\mathbb{E}[\mathcal{M}]

(dashed), and the arrows and the column on the right give

G

S

and

R

with the routable share

R/G

Figure 7: Equal means, three shapes. Both skills carry

\mathbb{E}[\theta]=0.40

, so

\mathbb{E}[m]

and

b

coincide and

S=0

; only the machine’s density changes, from identical to the human’s, through dispersed, to U-shaped. Reshaping moves

\mathbb{E}[\mathcal{M}]

and raises

G

and

R

together, from

0.061

0.154

, leaving the routable share at

100\%

. The invariance is not a property of the Beta family: with equal means the shapes cancel in the ratio whatever they are. Conventions as in Figure 6 ; the machine’s density in the third panel is unbounded at both ends

Profile A , $t=0$
	$d$	$\mathbb{E}[\theta_{H}]$	$\mathbb{E}[\theta_{M}]$	$b$	$G$	$S$	$R$	$R_{\mathrm{class}}$	$R/G$	$\mathbb{E}[\theta_{\bar{w}}]$	$\gamma^{*}$	$R_{\mathrm{cop}}$
	Low	0.600	0.833	0.833 (M)	0.132	0.117	0.015	—	12%	0.740	1.126	0.005
	Med	0.500	0.667	0.667 (M)	0.118	0.083	0.035	—	30%	0.598	1.114	0.015
	High	0.400	0.333	0.400 (H)	0.111	0.033	0.078	—	70%	0.375	1.065	0.047
	Task	0.500	0.611	0.611 (M)	0.121	0.056	0.065	0.022	54%	0.571	1.070	0.044
Profile A , $t=T$

Table 3: The stake profile of the illustrative skill pairs of Appendix G , per difficulty level and for the task as a whole, taking the three levels as equiprobable classes. At the level of the task, the residual splits into the part a class-level rule reaches,

R_{\mathrm{class}}

, and the rest,

R_{\mathrm{inst}}=R-R_{\mathrm{class}}

(Proposition 7 ); within a level the whole residual is instance-level.

\mathbb{E}[\theta_{\bar{w}}]

is what pooling the two responses attains under the squared rule, taking the profiles’ distributions as performances under that rule, with the two responses on the same side of the truth, and

\gamma^{*}=b/\mathbb{E}[\theta_{\bar{w}}]

the break-even lift of Proposition 15 . The last column relaxes an assumption of the profiles:

R_{\mathrm{cop}}

is the residual with the two performances coupled by a Gaussian copula of parameter

0.5

instead of independent (Proposition 17 ). Expectations of the maximum are computed by numerical integration, performances being independent on each instance by construction of the profiles

Level of evidence	Record	Layer	What it identifies
Separate assessments	$\mathcal{R}_{s}$ for every $s$	1: the means, and the marginal distributions	$b$ , $S$ and the commitment $i^{*}$ ; bounds on $R$ (Proposition 18 )
Separate assessments by class	$\mathcal{R}_{s,k}$ for every $s$ and $k$	1, by class	the incumbents $\bar{s}(k)$ and $R_{\mathrm{class}}$
Paired assessment	$\mathcal{R}_{\mathcal{S}}$ , with the responses retained	2: pairwise moments; 3: the joint law	$\Lambda$ and $\gamma^{*}$ ; $\mathbb{E}[\mathcal{M}]$ , and with it $G$ , $R$ and $R_{\mathrm{inst}}$
A third arm	the workflow’s output on the instances of $\mathcal{R}_{\mathcal{S}}$ , with the independent responses retained	none: the take is a property of the workflow	the take of a routing arrangement; the lift $\gamma_{\pi}$ of a protocol
A finite record	any of the above	4: finite records	each of the above with error; $\hat{b}$ biased upward, $\hat{R}$ downward

Table 4: The levels of evidence and the layers of knowledge. Each level of evidence is a kind of record (Appendix F ); each layer is what is known of the skills (Appendix A.1 ). The two do not coincide: paired observation reaches two layers at once, the third arm reaches none, since what it measures belongs to the workflow, and a record of finite size is not a further level but the error with which every level is read

Setup

Execution

Adjudication

Figure 8: What exists when. A policy reads what the instance carries and what the records have already yielded, and returns a response; everything on the right arrives later and on some instances only. The records a policy leaves behind are not the records it needs:

\mathcal{R}_{\mathcal{S}}

grows only where every skill executed, so an arrangement that commissions one consumes that record and replenishes none of it

Rung	Skill policy	Attains	Rout.	Syn.	Arch.	Depl.
Ceilings on what each routing family can attain (Section 5.6 )
B3	$ij(w\mid\mathrm{inst})$	$\mathbb{E}[\mathcal{M}]$	Y	N	$HM{+}{+}_{\gamma=1}$	N
B2	$i,j\mid\mathrm{inst}$	$\mathbb{E}[\mathcal{M}]$	Y	N	—	N
B1	$i,j\mid\mathrm{class}$	$\sum_{k}p_{k}\,\mu_{\bar{s}(k),k}=b+R_{\mathrm{class}}$	Y †	N	—	N
The ladder
5	$ij(w\mid Z_{q},W_{q}){+}$	$\gamma_{\pi}\,\mathbb{E}[\theta(\textstyle\sum_{s}\hat{w}_{s}\hat{r}_{s},r)]$	Y	Y	—	Y

Table 5: The spectrum of skill policies: the ceilings of each routing family above, the ladder below, its rungs read from the bottom up. Sums over

s

run over the available set; attainment is an expectation over the instance space, per instance

Rung	Configuration	Pays and buys	Anchors
5	Combining information, not just responses : routing, then an exchange in which each skill can revise its response in light of the other’s	Synergy on top of rung 4’s routing; the lift is measured, is bounded by none of the skills, and can fall below one	Second opinions between skills; centaur mode ( Randazzo et al., 2025 ; Dell’Acqua et al., 2026 )
4	Directing work given signals : a policy conditioned on the class or on the instance assigns, weights or selects	Collects $R$ in proportion to the value of the signal, up to $R_{\mathrm{class}}$ on a class signal and up to all of it at perfect routing; requires paired evidence, which assignment stops producing	Function allocation per instance ( Parasuraman et al., 2000 ) ; learning to defer ( Madras et al., 2018 ; Mozannar and Sontag, 2020 ) ; dynamic selection ( Cruz et al., 2018 )
3	Together against uncertainty : independent responses, then an exchange before the joint response is fixed	Inherits rung 2’s insurance at cost $S-\Lambda$ ; recovers it only past $\gamma^{*}$ ; lift measured against the pre-exchange mean response	Consensus double reading ( Johnston and Brown, 1999 ) ; the organizational loop ( Herrmann and Pfeiffer, 2023 )
2	Putting both skills at work : independent responses averaged, no routing	Pays $S-\Lambda$ ; caps the shortfall at $S-\Lambda$ against $2S$ for a wrong commitment (Proposition 10 ); narrows dispersion on both tails; the default landing place of oversight by mandate	Supervisory-control failure ( Parasuraman and Manzey, 2010 ; Green, 2022 ) ; the human in the average ( Crootof et al., 2023 )
1	One decision takes all : commit ex ante from separate assessments	Secures $S$ for the assessment alone; a wrong commitment forfeits $2S$	Comparative advantage ( Autor et al., 2003 ; Acemoglu, 2025 )
0	Nothing to decide	No assessment of the skill in use, and no alternative considered	The status quo

Table 6: What each rung pays and buys in performance, with its anchors in the literature

Ceilings (Section 5.6 )
Rung	Skill policy	Standing cost $\kappa_{\pi}$	Expected execution cost $\mathbb{E}[c_{\pi}]$
B3	$ij(w\mid\mathrm{inst})$	$\sum_{s}\kappa_{s}$	$\sum_{s}c_{s}$
B2	$i,j\mid\mathrm{inst}$	$\sum_{s}\kappa_{s}$	$\mathbb{E}[c_{\dot{s}}]$
B1	$i,j\mid\mathrm{class}$	$\sum_{s\in\bar{s}(\mathcal{K})}\kappa_{s}$	$\sum_{k}p_{k}\,c_{\bar{s}(k)}$
The ladder
5	$ij(w\mid Z_{q},W_{q}){+}$	$\sum_{s}\kappa_{s}$	$\sum_{s}c_{s}+c_{\rho}$

Table 7: The cost profile of each arrangement: the standing cost of what it holds available, incurred whatever the volume, and the expected execution cost of what it engages on an instance. Sums over

s

run over the available set;

\bar{s}(\mathcal{K})

is the set of skills that lead at least one class

Panel of Figure 9 — Figure 9: The lower rungs as skill policies (Section 3.5 ). Rungs 0 to 2 operate no channel: the criterion engages the same skills on every instance and the aggregation treats every instance alike. Rung 3 adds a protocol $\rho$ (dashed), which surrounds the engaged skills and returns the delivered response. Dotted boxes are skills available and not engaged. Random assignment (Algorithm 2r ) is not drawn: its criterion engages one skill per instance, as at rung 1, but draws it at random instead of committing to $i^{*}$ .

Panel of Figure 10 — Figure 10: Rungs 4 and 5 as skill policies. Under assignment (4a to 4c) the criterion engages one skill per instance from what it reads, $i$ on the instance drawn and $j$ on others (dotted); under ensembling (4d) and selection (4e) it engages both, and the aggregation weights their responses or selects one. Rung 5 runs a protocol $\rho$ (dashed) whose weights are set per instance, and so routes and operates synergy at once. Assignment with audit (4c+a, Algorithm 4c+a ) is not drawn: it is 4c with the other skill also engaged on a random fraction of instances, its response retained for the record and not returned.

Panel of Figure 11 — Figure 11: The ceilings of the routing families as skill policies (Section 5.6 ). Each receives from an oracle what no workflow can have: the true class (B1), the winner of the instance (B2), or the weighting that would have been best (B3), which on an endpoint truth places all weight on the winner. B1 and B2 close assignment, B3 ensembling.

Figure 12: The ladder against the decomposition, illustrative Profile A at the initial date, one column per level of difficulty. Above, the exact densities of the two skills in gray (

\theta_{H}

solid,

\theta_{M}

dotted) and of the within-instance anchors,

m

with a dashed outline and

\mathcal{M}

with a solid one, with dotted lines at

\mathbb{E}[m]

b

and

\mathbb{E}[\mathcal{M}]

; below, the performance each rung of Table 6 can attain, rung 0 showing both skills and rung 2 the pooled response scored under the squared rule, above

\mathbb{E}[m]

by the disagreement

\Lambda

; random assignment (Algorithm 2r ) is not drawn, since it sits on the dotted line at

\mathbb{E}[m]

. Hollow marks carry no lift; solid marks apply a uniform lift

\gamma=1.15

in the reduced form of Proposition 16 , computed by clipping each instance and averaging afterwards, with an asterisk where the ceiling clips more than

1\%

of instances (Proposition 26 ). Rung 4 is drawn solid from

b

to perfect routing and dashed down to

\mathbb{E}[m]-G

, where routing the wrong way on every instance would leave it (Proposition 13 ). The ladder orders arrangements by how much of the two channels they operate, not by what they attain.

\gamma^{*}

is reported in each column footer

Figure 13: The ladder against the decomposition at the final date, illustrative Profile A; conventions as in Figure 12 . Compared with the initial date, the informed baseline rises only at high difficulty, where the improving human skill overtakes the machine; at medium difficulty the machine remains the leader (

0.667

against

0.625

), and the share of the stake carried by the residual inverts across the three levels.

Step	Arrangement	Record it builds	What it identifies	Take without error
0	rung 0, its executions scored (Alg. 0 )	$\mathcal{R}_{s}$ of the skill in use	$\mathbb{E}[\theta_{s}]$ of that skill	$\mathbb{E}[\theta_{s}]-\mathbb{E}[m]$
1	rung 1, after separate assessments (Alg. 1 )	$\mathcal{R}_{s}$ for every $s$	whether the leader beats a reporter that reads nothing; $b$ and $S$ ; an upper bound on $R$	$S$
2	assignment on the class: B1 where the class is observed at intake, 4a where it is predicted (Algs. B1 , 4a )	$\mathcal{R}_{s,k}$ and $\mathcal{R}^{Z}$	the incumbents and $R_{\mathrm{class}}$	$S+R_{\mathrm{class}}$
3	the same, with audit at a high rate (Alg. 4c+a , on the class assignment)	$\mathcal{R}^{Z,W}_{\mathcal{S}}$	$\mathbb{E}[\mathcal{M}]$ , $R_{\mathrm{inst}}$ , $\Lambda$ and $\gamma^{*}$	$S+R_{\mathrm{class}}$
4	assignment on the instance, with audit (Algs. 4b , 4c , 4c+a )	$\mathcal{R}^{Z,W}_{\mathcal{S}}$	the share of $R_{\mathrm{inst}}$ the criterion reaches	$S+R_{\mathrm{class}}+T_{\mathrm{inst}}$
5	optionally, pooling, ensembling, selection or a protocol (Algs. 2 , 4d , 4e , 3 , 5 )	$\mathcal{R}^{Z,W,\hat{r}}_{\mathcal{S}}$ and the independent responses	the dividend; the lift $\gamma_{\pi}$	adds the dividend or the lift

Table 8: A procedure for climbing the ladder. Each step deploys an arrangement, builds the record the next step requires, and identifies what the next step would pursue. The last column is the take when the step is carried out without error; errors in reading the class or the instance make it smaller, and split it between the components only by convention (Section 4.5 )

Panel of Figure 14 — Figure 14: The procedure as a flow. Each step deploys an arrangement and builds the record the next step needs; each check either lets the climb continue, stops it where the next component is not worth what pursuing it costs, or returns the classes that fall back to what the previous step assigned them while the criterion is refitted. The checks after steps 2 and 4 are made on each class as the policy reads it, observed at intake or predicted, against what the previous step attains on the same instances, by a margin of $z$ standard errors where the attainments are estimated (Proposition 32 ); how the take splits between the components is read afterwards, on the true classes (Section 4.5 ). From step 3 the audit keeps the paired record growing; step 5 grows it without one. The stated takes are those of a step carried out without error

Unit	What it is	In the file	How many	In the theory
Film	one patient’s chest radiograph, which comes with a short clinical history	patient_id	324	the unit on which folds and resamples are drawn
Finding	a condition every film is asked about, such as pneumothorax, whether or not it turns out to be present: the diagnostic question	pathology	15 of 104	the class $k$ , named at intake, so that the signal $Z$ gives it exactly
Film-finding	one finding on one film: whether pneumothorax is present on film 154	patient_id and pathology	4,860	carries the truth $r$ and the model’s response $\hat{r}_{M}$ , which is also the signal $W$ ; each weighs the same
Reading	one radiologist reading one film in one condition, answering all its findings at once	experiment_id , patient_id and treatment	5,240, about 16 per film	the human skill executing the 15 instances of a film
Read	one radiologist’s probability for one finding on one film, what a reading returns on one film-finding	a row, probability	78,600, 7 to 30 per film-finding	an instance: $\hat{r}_{H}$ , paired with $\hat{r}_{M}$ and $r$ ; each weighs $1/n$ of its film-finding

Table 9: The units of the Collab-CXR data, and their place in the theory. Counts are for one arm; the columns of the file are those of Table 18

Figure 15: The CXR experiment, arm with clinical history, squared rule. (a) What each arrangement attains, with 95% intervals over resamples of films and the value in the column on the right; the vertical lines are the mean anchor, the informed baseline (the model’s), the class ceiling B1 and the routed anchor, and the gray bands the two reporters that read nothing, each finding’s prevalence, within

0.0001

of B1, and always zero. The intervals are wide because the films differ in difficulty for every arrangement alike; differences between arrangements are estimated far more tightly (text). The open marker is the arm with the model’s prediction in view; rung 4a is the one built in Appendix H.6 . (b) Each of the 227 radiologists, on the films he or she read: the model’s lead over the reader, and the residual of the pair; the dashed line is the systematic component

S_{r}

, half the absolute lead.

		Squared rule		Linear rule
		X-ray only	With history	X-ray only	With history
Skills and anchors
	$\mathbb{E}[\theta_{H}]$ , radiologist	0.9312	0.9330	0.8908	0.8927
	$\mathbb{E}[\theta_{M}]$ , model	0.9366	0.9366	0.8490	0.8490
	$\mathbb{E}[m]$ , mean anchor	0.9339	0.9348	0.8699	0.8709
	$b$ , informed baseline	0.9366 ( $M$ )	0.9366 ( $M$ )	0.8908 ( $H$ )	0.8927 ( $H$ )

Table 10: The stake and the take in the CXR experiment, by arm and scoring rule. Instances are a film and a finding read by one radiologist, over the 15 findings the model reports; each of the 4,860 film-findings weighs the same.

H

is a radiologist reading unaided, without or with clinical history,

M

the model. Brackets: 95% intervals over 1,000 resamples of films; the interval of

S

is that of half the leader’s margin,

(\mathbb{E}[\theta_{L}]-\mathbb{E}[\theta_{U}])/2

with the leader held at its estimate, which can fall below zero. B1 and rungs 4b to 4e are cross-fitted over five folds of films. The two rows below the routed anchor are its analogues when a whole reading, one radiologist’s film, or a whole film must go to one skill: the mean, over readings or over films, of the better of the two skills’ mean performances on its film-findings

Film 277, a typical film
		Alone		Rung 3	Rung 2	B1	4c, checked	4d	4e
Finding	$r$	$H$	$M$	$M$ in view	pooling	by finding	+ probability	ensembling	selection
Any abnormality	0	0.46	0.46	0.72	0.46	0.46 H	0.46 H	0.46	0.46 H
Airspace opacity	1	0.17	0.19	0.19	0.18	0.19 M	0.19 M	0.18	0.17 H
Atelectasis	1	0.02	0.07	0.02	0.05	0.07 M	0.07 M	0.06	0.02 H
Consolidation	0	0.51	0.02	0.02	0.26	0.02 M	0.02 M	0.04	0.02 M

Table 11: Two films through the arrangements, each read by one radiologist of the third design, unaided (

H

) and later with the model’s prediction in view;

M

is the model and

r

the standard. Each column gives the probability the arrangement reports for the finding, with the skill whose response it returns where it returns one; the arrangements are fitted on the responses as reported, on the folds of films that exclude the film. B1 assigns each finding to its incumbent, and 4c, checked, also reads the model’s probability where the check keeps it. The film rows give the mean performance over the 15 findings, squared rule, and the findings given to the radiologist; under the hierarchy she answers only those shown to her. The model runs on the film under every arrangement but

H

alone. Table 24 gives every finding, and Tables 25 and 26 every read of four of the film-findings

Figure 16: The procedure of Section 6 walked on the Collab-CXR data, arm with clinical history, squared rule. (a) What each step attains, fitted on four fifths of the films and scored on the fifth, with its value in the column attains , and what its check finds, the steps read from the bottom up. The two lowest rows score the radiologists as they report; every row above them uses the two skills recalibrated within each finding, as the check of step 1 requires. Where a row shows both skills, the circle is the radiologist (

H

) and the square the model (

M

); open markers are not adopted. The gray bands are the reporters that read nothing, the black line routing omniscience between the recalibrated skills. Step 5 includes pooling, which is ensembling with equal weights, and the arm the study ran, which is an exchange; above the rule, for comparison, the logistic combination, which is not a rung. (b) The same results as a path from current practice: the radiologists recalibrated, the recalibrated model alone, and ensembling the two; under each bar, the share of films a radiologist reads

Line of work	What it holds	What this paper adds
Earlier paper ( Zanardelli, 2025 )	$G=S+R$ for a pair, with bounds; the mean, collaborative and superpower archetypes; a ladder whose upper rungs are bounded but not run; Monte Carlo analysis of utility	Any number of skills; $R$ split by class; the squared rule and the bridge between responses and performances; the two channels; every rung as an algorithm; the procedure; a measurement on field data
Value of information ( Howard, 1966 ; Raiffa and Schlaifer, 1961 )	$R$ as the expected value of perfect information	The lower anchor and $S$ ; the split of $R$ by class; which record identifies each part
Algorithm selection and model routing ( Rice, 1976 ; Kerschke et al., 2019 ; Hu et al., 2024 ; Ong et al., 2025 )	The gap between the virtual and the single best solver, which is $R$ , and the share of it a selector closes	What lies below the single best solver, $S$ ; the class and instance parts of $R$ ; combining responses and protocols, which per-instance selection does not represent; bounds on $R$ from separate assessments
Learning to defer, triage, dynamic selection ( Madras et al., 2018 ; Mozannar and Sontag, 2020 ; Raghu et al., 2019 ; Okati et al., 2021 ; Ko et al., 2008 )	Criteria learned per instance, those of rungs 4b and 4c, with the oracle as their ceiling	The ceiling identified before any criterion is learned; the evidence each criterion consumes, and an audit that replenishes it; comparison with pooling and with protocols
Ensembles and forecast combination ( Krogh and Vedelsby, 1994 ; Bates and Granger, 1969 ; Soll and Larrick, 2009 ; Mannes et al., 2014 )	The ambiguity decomposition; when combining beats choosing	With an endpoint truth, combination capped at the best skill on every instance (Proposition 8 ); pooling as insurance with a stated premium (Proposition 10 )
Human–AI complementarity ( Bansal et al., 2021 ; Donahue et al., 2022 ; Hemmer et al., 2025 ; Peng et al., 2025 )	Complementary team performance; complementarity potentials anchored at perfect execution; existence and worst-case results	Anchors at the mean and at the best skill, and so $S$ and $G$ ; magnitudes for a given pair rather than worst cases; synergy measured by a lift

Table 12: What adjacent lines of work hold, and what the accounting adds

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 17: What each layer adds, and what becomes known. The figure is drawn for illustrative Beta skills, but what each layer determines does not depend on them. Middle: what the layer adds to what is known of the skills in each class. Ticks mark the class means

\mu_{s,k}

, those of skill

i

above the line and those of skill

j

below it; the bars in the second row are their spreads; the dotted and dash-dotted curves are the distributions of skills

i

and

j

within each class (the coupling, which the law also fixes, is not drawn); and the filled dots and empty squares in the last row are observed performances of skills

i

and

j

, a sample of pairs that in practice takes the place of the law, drawn faintly behind it because it is not known. Third column: what is then known of the anchors and of the stake. The means fix

\mathbb{E}[m]

b

and

\sum_{k}p_{k}b_{k}

, and with them

S

and

R_{\mathrm{class}}

, but leave

\mathbb{E}[\mathcal{M}]

and

R_{\mathrm{inst}}

unknown; pairwise moments bound them from above (hatched, Proposition 20 ) and, taken on the distances

\sqrt{1-\theta_{s}}

, give the disagreement

\Lambda

that pooling earns, and with it rung 2 and

\gamma^{*}

; the joint law fixes them; and a record of 200 paired instances yields estimates of

\mathbb{E}[\mathcal{M}]

scattered around its value. Right: what can be drawn of the anchors themselves. The means give

\mathbb{E}[m]

alone, and pairwise moments add its spread and an upper bound on

\mathbb{E}[\mathcal{M}]

, but neither determines the distributions of

m

and

\mathcal{M}

; the joint law gives them exactly (

m

dashed and shaded,

\mathcal{M}

solid and hatched); and a record of 200 paired instances gives histograms that estimate them, the exact curves drawn faintly behind.

Figure 18: Where the split of a take is a convention. Assignment on a predicted class, with two classes whose incumbents differ and illustrative values

S=0.020

and

R_{\mathrm{class}}=0.050

, as the accuracy

a

of the class prediction runs from chance to one. The black line is its take at the level of the class,

(2a-1)(S+R_{\mathrm{class}})

, fixed by Proposition 23 under any crediting rule; the patterns split it into

T_{S}

(hatched) and

T_{\mathrm{class}}

(dotted), (a) in order and (b) in proportion. The two agree at

a=1

, where the policy resolves both classes and takes

(S,R_{\mathrm{class}})

, and at

a=\frac{1}{2}

, where it takes nothing; between them every point is a blend of right and wrong assignments, and the split is a convention. The dotted lines mark the commitment, which takes

S

, and B1, which takes

S+R_{\mathrm{class}}

;

a^{*}

is the accuracy at which the policy reaches

b

Result	Squared rule	Linear rule
Pooling, rung 2	$\mathbb{E}[m]+\Lambda=b-S+\Lambda$	$\mathbb{E}[m]=b-S$
Fixed combinations (Corollary 1 )	can exceed $b$ ; never above $\mathbb{E}[\mathcal{M}]$	never above $b$
Pooling against randomizing (Proposition 12 )	pooling higher by $\Lambda$	equal in expectation
The insurance contract (Proposition 10 )	premium $S-\Lambda-D$ ; pooling preferable iff $D>S-\Lambda$ ; for a pair iff $p_{a}<\frac{1}{2}+\frac{\Lambda}{2S}$	premium $S-D$ ; iff $D>S$ ; for a pair iff $p_{a}<\frac{1}{2}$
Break-even of synergy (Proposition 15 )	$\gamma^{}=\frac{b}{b-S+\Lambda}$ , $\delta^{}=S-\Lambda$	$\gamma^{}=\frac{b}{b-S}$ , $\delta^{}=S$
Archetype $HM$ (Proposition 16 )	$\theta_{HM}=\theta_{\bar{w}}$ ; attains $b-S+\Lambda$	$\theta_{HM}=m$ ; attains $b-S$

Appendix

Table 13: Results that depend on the scoring rule. The squared entries are those stated in the text; the linear entries set

\Lambda=0

Setting	Why	Rule
Hard responses: classes, yes or no, decisions acted on as they stand	the two rules coincide on every instance	either; the linear reads more simply
Graded responses that are actions, with errors costing in proportion to their size	the linear rule is the economic valuation	linear
Skills assessed only separately	pooling, the insurance premium and the break-even of synergy follow from the marginals	linear, for pricing before paired data exist
Graded responses combined with others, mechanically or through a protocol	a mechanical combination is scored above the mean of its parts by their disagreement; a protocol is measured against that combination, and can use only the evidence the responses still carry	squared
Graded responses read by a criterion, or compared with a threshold other than one half	the decision depends on the probability, not on the class	squared
Comparing, selecting or training probabilistic skills	the linear rule can favor responses that have discarded the strength of their evidence	squared

Appendix

Table 14: Where each scoring rule fits

Record	Instances it holds	$r$	$k$	$Z_{q}$	$W_{q}$	$\{\hat{r}_{s}\}$	What it supports
$\mathcal{R}_{s}$	worked by skill $s$	✓	—	—	—	—	$\hat{\mathbb{E}}[\theta_{s}]$ , hence $i^{*}$
$\mathcal{R}_{s,k}$	the part of $\mathcal{R}_{s}$ of class $k$	✓	✓	—	—	—	$\hat{\mu}_{s,k}$ , hence $\bar{s}(k)$
$\mathcal{R}^{Z}$	any, whether or not a skill executed them	—	✓	✓	—	—	$\widehat{\Pr}(k\mid Z_{q})$
$\mathcal{R}_{\mathcal{S}}$	worked by every skill in $\mathcal{S}$	✓	—	—	—	—	the winner $\dot{s}$ of each instance; with the responses retained, $\hat{\Lambda}$
$\mathcal{R}_{\mathcal{S}}^{W}$	the same, carrying $W_{q}$	✓	—	—	✓	—	$\hat{\mathbb{E}}[\theta_{s}\mid W_{q}]$
$\mathcal{R}_{\mathcal{S}}^{Z,W}$	the same, carrying both	✓	—	✓	✓	—	$\hat{\mathbb{E}}[\theta_{s}\mid Z_{q},W_{q}]$

Appendix

Table 15: The records. Each is a set of past instances, distinguished by which skills worked them and what was retained about them. A tick marks what an instance in that record carries; a dash marks what it need not

		Profile A				Profile B
		$t=0$		$t=T$		$t=0$		$t=T$
Skill	$d$	$\alpha$	$\beta$	$\alpha$	$\beta$	$\alpha$	$\beta$	$\alpha$	$\beta$
$H$	Low	3	2	6	2	3	2	6	2
	Med	5	5	5	3	5	5	5	3
	High	2	3	3	2.5	2	3	3	2.5
$M$	Low	10	2	10	2	10	2	10	2

Appendix

Table 16: Parameters

(\alpha,\beta)

of the illustrative performance distributions, by skill, difficulty level and date

Panel of Figure 19 — Figure 19: The illustrative performance distributions of Table 16 , for both skills and all difficulty levels, at the initial date $t=0$ and the final date $t=T$ , the human skill improving between the two and the machine skill static; densities are exact, and the ticks on the axis mark each skill’s mean

The data
Rows of the file	2,717,728; 314,400 used ( $11.6\%$ ): the four arms and the 15 findings, without practice or panel reads
Films	324, of which 122 are positive on at least one of the 15 findings
Findings, the classes	15 of the 104, those the model reports: 4 at the top of the hierarchy, 11 below
Film-findings	4,860, of which 302 positive ( $6.2\%$ )
Radiologists	227: 159, 33 and 35 in the three designs; the panel’s reads excluded
Readings	5,240 in each of the four arms: one radiologist reading one film in one condition, about 16 per film

Appendix

Table 17: The Collab-CXR data as used here, and the study in the terms of the theory. Counts are for the arm with clinical history unless stated; the other arms have the same structure

Figure 20: The Collab-CXR data, arm with clinical history. (a) The 15 findings, by the share of the 324 films on which the standard is positive, with the number of positive films; the four top-level findings in gray. (b) The number of radiologists reading each film-finding. (c) and (d) The probabilities the two skills report where the standard is negative and where it is positive, the radiologists’ reads weighted so that each film-finding counts once.

Figure 21: Performance in the Collab-CXR data, arm with clinical history, squared rule, against two reporters that read nothing: one that gives every film the prevalence of each finding, and one that always reports zero. (a) The mean performance of each skill and each reporter, finding by finding, with the two skills’ means beside the panel, the higher of the two, the finding’s incumbent, in bold. (b) Among the responses in each band of reported probability, the share positive on the standard, marker area growing with the number of responses; the dotted line is perfect calibration. (c) The share of film-findings in each band of performance, where the finding is absent and where it is present, with the means.

Column	What it records	Use here
Readers, films and findings
experiment_id	the radiologist: the 227 participants and the 10 readers of the two diagnostic panels	the reader; within-reader pairs
uid_clean	the radiologist in one session; differs from experiment_id only in design 2, whose readers have four sessions	not used
experiment_session	the session, 0 to 3; sessions 1 to 3 exist only in design 2	not used
design	the design, 1, 2 or 3; missing for the panel readers	the within-reader designs, 2 and 3
patient_id	the film, 0 to 323, which links to the image and clinical history of the Stanford release	the film; folds and resamples

Appendix

Table 18: The columns of the primary file of the Collab-CXR release ( Moehring et al., 2025 ) , version of 16 September 2026, and their use here. Each row of the file is a read, one radiologist’s response for one finding on one film in one session, 2,717,728 rows in all (Table 9 ). Descriptions follow the release’s variable list (its Table 1), checked against the file; counts are from the file; where the list does not document a column, or documents it differently, the description says what the file shows

Finding	Prevalence	$\mu_{H,k}$	$\mu_{M,k}$	$\bar{s}(k)$	$R_{k}$	Bounds on $R_{k}$	Corr.	$\bar{s}(k)$ , linear
Any abnormality	0.194	0.7293	0.6066	$H$	0.0692	[0.0086, 0.1959]	0.44	$H$
Airspace opacity	0.160	0.8760	0.8798	$M$	0.0645	[0.0336, 0.1123]	0.45	$H$
Consolidation	0.034	0.9575	0.9735	$M$	0.0129	[0.0030, 0.0263]	0.50	$H$
Bacterial or lobar pneumonia	0.012	0.9762	0.9856	$M$	0.0097	[0.0041, 0.0144]	0.43	$H$
Edema	0.019	0.9468	0.9217	$H$	0.0290	[0.0069, 0.0527]	0.32	$H$
Atelectasis	0.111	0.9040	0.9286	$M$	0.0237	[0.0079, 0.0709]	0.76	$H$

Appendix

Table 19: The CXR experiment by finding, in the arm with clinical history under the squared rule. Prevalence is the share of films positive on the diagnostic standard;

\mu_{H,k}

and

\mu_{M,k}

are the class means,

\bar{s}(k)

the incumbent, under the squared rule and, in the last column, under the linear rule;

R_{k}

is the residual within the finding, with the bounds that the two class marginals alone establish (Proposition 33 ); Corr. is the correlation of the two performances across reads

Variant	$\mathbb{E}[\theta_{H}]$	$\mathbb{E}[\theta_{M}]$	$S$	$R$	$R_{\mathrm{class}}$	$\Lambda$	Pool $-\,b$	Arm $-\,b$	$\gamma$
All 15 findings	0.9330	0.9366	0.0018	0.0334	0.0100	0.0101	0.0083	$-$ 0.0001	0.991
Top level (4)	0.8479	0.8328	0.0075	0.0716	0.0156	0.0209	0.0133	0.0040	0.989
Below top level (11)	0.9640	0.9743	0.0051	0.0140	0.0025	0.0062	0.0010	$-$ 0.0071	0.992
Two or more positives (11)	0.9107	0.9150	0.0021	0.0442	0.0134	0.0131	0.0110	$-$ 0.0002	0.988
Isotonic recalibration	0.9574	0.9644	0.0035	0.0137	0.0001	0.0049	0.0014	$-$ 0.0037	0.995
Logistic recalibration	0.9563	0.9655	0.0046	0.0120	0.0000	0.0045	$-$ 0.0001	$-$ 0.0063	0.994

Appendix

Table 20: The CXR experiment under variations, arm with clinical history, squared rule. The first row is the measurement of Table 10 . Recalibration maps each skill’s probability within each finding, fitted on four fifths of the films and applied to the fifth; the reads with the model’s prediction in view are recalibrated on their own. The row Standard, panellists shown uses a standard built from the panellists who were shown each finding; the last two keep only the reads of findings the radiologist was shown, with the model’s paired reads, on which the arm with the prediction in view is not compared, since the findings shown differ between the arms (text). The last three columns are pooling and the arm with the prediction in view against the informed baseline, and the lift of the arm against pooling

As reported
	Estimate	Films	Radiologists	Both
$S$	0.0018	[ $-$ 0.0001, 0.0038]	[0.0005, 0.0030]	[ $-$ 0.0004, 0.0042]
$R$	0.0334	[0.0312, 0.0356]	[0.0323, 0.0344]	[0.0309, 0.0357]
Pool $-\,b$	0.0083	[0.0067, 0.0098]	[0.0073, 0.0094]	[0.0063, 0.0100]
$\gamma$	0.991	[0.989, 0.993]	[0.989, 0.993]	[0.988, 0.994]
Recalibrated

Appendix

Table 21: Three ways of resampling, arm with clinical history, squared rule: 95% intervals over 1,000 resamples of films, as in the text, of radiologists, and of both, drawn independently, each read weighing the product of its film’s and its radiologist’s multiplicities.

S

is half the leader’s margin, which can fall below zero; Pool

-\,b

is pooling against the commitment, and

\gamma

the lift of the arm with the prediction in view against pooling. Recalibrated: as in Table 20

			Radiologist’s share
Step	Policy	Attains	findings given	films read
	routing omniscience, $\mathbb{E}[\mathcal{M}]$	0.9781	—	—
	logistic combination of the two, outside the ladder	0.9674	100%	100%
5	exchange: the model in view, rung 3	0.9607	100%	100%
5	selection	0.9643	100%	100%
5	ensembling	0.9660	100%	100%

Appendix

Table 22: The procedure of Section 6 run on the Collab-CXR data, arm with clinical history, squared rule: what each step attains, and how much of a radiologist’s work it needs. As the check of step 1 requires, both skills are recalibrated within each finding on the training films; every step is fitted on four fifths of the films and evaluated on the fifth, and the check of step 4 is made finding by finding on inner folds of the training films. The steps read from the bottom up; above them, what no step reaches, and below them, the two reporters that read nothing

Rung	On an arriving film	films read	findings given	Attains
		Radiologist’s share
B3	both report every finding, and the oracle returns the better probability	100%	100%	0.9699
B2	an oracle names, for each finding, the skill that will score better, and that skill alone reports it	100%	81%	0.9699
5	assembled from the study’s arms, not run: a radiologist reads it with the prediction in view, and her read is averaged with the model’s under the weights of rung 4d	100%	100%	0.9475
4e	both report every finding, and one of the two probabilities is returned, chosen on the finding and the bands of both	100%	100%	0.9528
4d	both report every finding, and the report is a weighted average, the weight set by the finding and the band of the model’s probability	100%	100%	0.9497

Appendix

Table 23: The arrangements of Section 5 in the CXR setting: how each would run on an arriving film, the share of a radiologist’s films she reads and of the film-findings given to her, and what it attains in the arm with clinical history under the squared rule, on the responses as reported, rung 0 at the foot. Fitted policies are cross-fitted over five folds of films

Film 277, a typical film
		Alone		Rung 3	Rung 2	B1	4c, checked	4d	4e
Finding	$r$	$H$	$M$	$M$ in view	pooling	by finding	+ probability	ensembling	selection
Any abnormality	0	0.46	0.46	0.72	0.46	0.46 H	0.46 H	0.46	0.46 H
Airspace opacity	1	0.17	0.19	0.19	0.18	0.19 M	0.19 M	0.18	0.17 H
Consolidation	0	0.51	0.02	0.02	0.26	0.02 M	0.02 M	0.04	0.02 M
Bacterial or lobar pneumonia	0	0.01	0.02	0.00	0.02	0.02 M	0.02 M	0.02	0.02 M

Appendix

Table 24: The two films of Table 11 , every finding. Columns as there

		Airspace opacity		Support device or hardware
experiment_id	design	unaided	in view	unaided	in view
advtele_22d1lv15	1	—	0.07	—	1.00
advtele_3pmlgwb2	1	—	0.04	—	0.96
advtele_8v77oqem	1	—	0.04	—	0.46
advtele_aq9gyefs	1	—	0.03	—	0.79
advtele_av5u00hh	1	—	0.15	—	0.81

Appendix

Table 25: Every read of two film-findings of film 277, in the arm with clinical history, as recorded in the primary file (Table 18 ): one row per radiologist who read the film in that arm, unaided, with the prediction in view, or both, the reader of Table 11 in bold; a dagger marks a finding not shown to the reader, participant or panel, and recorded as zero. The measurement uses every unaided read, each weighing equally within the film-finding, so that the mean of the reads is what a radiologist drawn at random returns on it. Below, the model’s response and the five probabilities of the panel that set the standard. The film was also read by 29 radiologists without history and 30 without history with the prediction in view

		Cardiomegaly		Any abnormality
experiment_id	design	unaided	in view	unaided	in view
advtele_2aoiqxz7	1	—	0.96	—	1.00
advtele_413wwwzp	1	0.77	—	0.74	—
advtele_59dyemkk	1	0.70	—	0.60	—
advtele_6gqvd9r0	1	—	0.57	—	0.73
advtele_9mkc2cw0	1	—	0.66	—	0.84

Appendix

Table 26: Every read of two film-findings of film 154, in the arm with clinical history, as recorded in the primary file (Table 18 ): one row per radiologist who read the film in that arm, unaided, with the prediction in view, or both, the reader of Table 11 in bold; a dagger marks a finding not shown to the reader, participant or panel, and recorded as zero. The measurement uses every unaided read, each weighing equally within the film-finding, so that the mean of the reads is what a radiologist drawn at random returns on it. Below, the model’s response and the five probabilities of the panel that set the standard. The film was also read by 13 radiologists without history and 11 without history with the prediction in view

		Alone		Rung 3	Rung 2	B1	4c, checked	4d	4e
Finding	$r$	$H$	$M$	$M$ in view	pooling	by finding	+ probability	ensembling	selection
Any abnormality	1	0.08	0.72	0.73	0.40	0.08 H	0.08 H	0.08	0.08 H
Airspace opacity	1	0.02	0.27	0.36	0.14	0.27 M	0.27 M	0.12	0.02 H
Consolidation	1	0.00	0.13	0.01	0.07	0.13 M	0.13 M	0.08	0.00 H
Bacterial or lobar pneumonia	1	0.00	0.13	0.00	0.07	0.13 M	0.13 M	0.08	0.13 M
Edema	0	0.00	0.15	0.35	0.07	0.00 H	0.15 M	0.10	0.00 H

Appendix

Table 27: Film 77, on which four findings are present; the radiologist put any abnormality at

0.08

and the airspace opacity at

0.02

, so that the findings beneath the airspace opacity, two of the four among them, were not shown to her and are recorded as zero. Another radiologist of the third design; columns as in Table 11

Explore similar work

Apr 20, 2026

CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks

AI agents are reshaping the workspace, leading to drastic change of how humans work. Despite the considerable potential of human-agent collaboration both in preserving human agency and generating economic value, this paradigm remains largely absent from occupational task evaluation, hindered by the difficulty of gathering real human data and accounting for inter-human variability. We introduce CollabSkill, a framework for evaluating human-agent collaboration on real-world occupational tasks. CollabSkill pairs real human workers with AI agents on tasks matched to their occupational background, collecting data that capture the complexity of economically valuable tasks and the usage patterns of real workers. To account for inter-human variability, CollabSkill employs a Bayesian skill rating system to disentangle and quantify the skill contributions of both humans and AI agents. Drawing on over 1,500 prompts from 386 working sessions contributed by 93 human workers, our analysis yields insights on two fronts: on the agent side, rankings on CollabSkill diverge meaningfully from those of existing fully autonomous benchmarks where Codex leads, with Claude Code ranking first; on the human side, CollabSkill reveals that practical experience emerges as the primary driver of collaboration skill, with hands-on collaboration meaningfully shifting workers' AI literacy. Together, we hope CollabSkill enables the community to invest in systematic evaluation of human-agent collaboration and spurs development efforts aimed at building AI agents that genuinely augment human workers.

Yijia Shao, Zora Zhiruo Wang, Neel Ahuja +3

Stanford University · Carnegie Mellon University · University of Rochester +1

AI Agent Evaluation Human-in-the-Loop Evaluation Human-AI Collaboration AI and Labor

Ask PDF

Sep 29, 2026

Recursive Organization Improvement: A Modeling Specification for Human--Agent Organizations

Stronger AI agents do not automatically produce better organizations: teams must also learn which work arrangements to retain and when to reconsider them. We propose a modeling specification for recursive organization improvement and evaluate it through an executable checker, a public-record mapping, and controlled simulation. The specification connects actor-visible histories, organizational memory, decision rights, and evidence-carrying change contracts. The mechanism study crosses six decision rules, three memory conditions, and three task environments under fixed resource ceilings. In a stationary environment, cumulative evidence raises balanced evaluation's normalized net value per task from 0.45224 to 0.48007. Repeated reassessment's disadvantage relative to this comparator falls from 0.01702 with reset evidence to 0.00007 with cumulative evidence. A reversal of the best workflow reveals the opposite cost: indefinite retention delays adaptation, while a finite window restores eventual performance at a transition cost. In exploratory controls, matching trial acquisition and label reuse reduces the apparent reassessment gain from 0.00607 to 0.00191. Program replacement adds no stable benefit across the tested reversal times. The study identifies evidence acquisition, reuse, and timely updating as mechanisms that must be separated from evaluator replacement when assessing organizational improvement.

Zilong Wang

CataX AI

Multi-Agent Collaboration Human-AI Collaboration

Ask PDF

Mar 3, 2026

Path Dependence under Adaptive AI Delegation

Repeated AI assistance can improve immediate task performance while reducing the skill available for future independent work. We develop a mathematical framework for this long-run tradeoff. The model tracks two state variables: a latent human skill level governing expected independent performance, and a delegation level representing the learner's evolving tendency to rely on AI. Skill changes through error-driven learning under practice and decay under delegation; delegation responds to observed performance, increasing when AI-assisted work appears to outperform independent work. We analyze the resulting dynamics and contrast them with fixed delegation. With fixed delegation, skill follows a one-dimensional learning-decay process with a single stable equilibrium. With adaptive delegation, the coupled system has two attracting equilibria separated by the stable manifold of an interior saddle. The existence and geometry of this separatrix require a global phase-plane analysis of the coupled dynamics. The system is path-dependent: small differences in initial skill or reliance can lead to different long-run outcomes. We use this characterization to show that AI assistance can improve short-run performance while producing worse long-run performance than a no-AI baseline. Increasing AI capability can enlarge the basin of attraction of the low-skill equilibrium, making delegation appear beneficial for longer while increasing the risk of eventual skill loss. The qualitative picture is observed to persist across alternative specifications. Together, these results show that the risk is not AI assistance itself, but the coupling between performance-driven reliance and use-dependent skill change.

Lingxiao Huang, Nisheeth K. Vishnoi

Nanjing University · Yale University

Human-AI Collaboration

Ask PDF