When a multi-agent system changes its team, it also changes the messages and final answer it produces, so an end-to-end accuracy gap does not by itself identify a routing effect. We introduce method, an evaluation contract that fixes a public information boundary, downstream stack G, and finite legal team family before outcomes are generated. Complete coverage identifies exact finite-benchmark oracle regret conditional on that stack. For any finite collection of frozen policies, executing the union of their distinct selected teams is the minimal assumption-free support for every pairwise policy contrast, though not for absolute oracle regret. Two controlled tables with source-ID-disjoint splits test the instrument. On MuSiQue-12, a pre-specified privileged positive control improves regret from 0.532 to 0.402; a later public-interface control reaches 0.424 versus 0.554 but is retrospective. On HotpotQA-4, a pre-specified public direct scorer improves regret from 0.313 to 0.110. In fixed-stack Llama execution, verified route regret improves by 0.190, while the raw-answer gain is 0.010 with an interval crossing zero. A five-family ToolSandbox variant-shift validation exhaustively evaluates 16 declared teams on 14 untouched task variants (224/224 valid rows): the declared-family oracle reaches 0.768 safe-evidence completion, while the prospectively frozen router gets 0.637 (regret 0.131), failing the predeclared 0.10 criterion. A later retrospective comparator reaches 0.655, matching all-workers with 4.57 versus 5.00 workers on average. Thus COVER exposes selection headroom without manufacturing a routing win. A crossed-stack diagnostic shows absolute scores depend on G but finds no detectable router-by-finalizer interaction. COVER is an auditable measurement methodology, not a claim of stack-invariant or universal agent-routing superiority.
Figures & tables
Figure 1: COVER changes only the selected coalition. Orange information is unavailable when the route is chosen.
Stage
Available
Deliberately unavailable
Route selection
task text, public cards, declared budget
private evidence, answers, outcome table, required team
Worker execution
selected worker’s private evidence
other workers’ private evidence
Finalization
canonical selected messages
router identity and unselected messages
Offline scoring
frozen table values and route files
any held-out training outcome
Table 1: Information visible at each point in a COVER evaluation.
Coverage regime
Interventions/task
Identified quantity
Complete finite family
all legal teams
exact finite-benchmark oracle regret
Precommitted sampled family
declared sampled teams
sampled-action regret or bounds, not an oracle
Frozen route union
distinct selected teams
all pairwise policy contrasts; inclusion-minimal
Table 2: COVER’s compute/identification regimes. Costs are interventions per task; only the first regime identifies absolute finite-benchmark oracle regret.
Tests direct team scoring without a partition decoder
Table 3: The two intervention-complete controlled instances. A task, rather than a card pair or intervention, is the unit of inference.
Study
Tasks
Result
Interpretation
MuSiQue-12 public
500
0.424 vs 0.554
Deployment-relevant public interface; retrospective control
MuSiQue-12 privileged
500
0.402 vs 0.532
Pre-specified positive control of instrument sensitivity
HotpotQA-4
300
0.110 vs 0.313
Direct set scoring is a useful bias in a four-way decision
Table 4: Controlled findings and their evidential status. Regret is exact over each declared offline family conditional on its frozen stack; exact scoring does not make a privileged policy deployable or a retrospective policy confirmatory.
Figure 2: Controlled public-interface comparisons. The MuSiQue pair is retrospective; the HotpotQA pair is pre-specified. Regret is exact within each declared action space; scales are not compared across instances.
Family
Router
Verified evidence complete
Raw answer exact
Raw route regret
Llama
Unified-COVER
81.67%
36.33%
0.050
Llama
leave-one-out
62.67%
35.33%
0.060
Table 5: Frozen genuine execution. Executed-route regret is calculated only on the union of the frozen selected routes, not over all possible teams.
Policy or reference
Completion
Regret
Mean workers
Per-task declared-family oracle
.768
—
—
All workers
.655
.113
5.00
Development-best fixed team
.655
.113
5.00
Learned value router
.655
.113
4.57
ToolBench-style card retriever
.655
.113
5.00
Frozen public capability router
.637
.131
3.00
Table 6: Five-family ToolSandbox variant-shift validation (14 tasks, 16 declared coalitions/task). Completion is the official safe-evidence endpoint; lower regret is better. “Style” rows are interface-level adaptations, not exact published-system reproductions.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Field
Router at selection time
Worker/finalizer after selection
Question and public capability cards
yes
yes
Private passages and evidence units
no
selected worker only
Gold answer, required team, component ID
no
no
Held-out intervention value
no
evaluation only
Finalizer identity and canonical message order
fixed, not routed
fixed
Appendix
Table 7: Public/private boundary used in both controlled instances.
Study
Comparator
COVER
Improvement
Status
MuSiQue-12 binary
calibrated graph 0.5320
0.4020
0.1300
positive control
HotpotQA-4 binary
leave-one-out 0.3133
0.1100
0.2033
primary
Llama verified route regret
leave-one-out 0.2900
0.1000
0.1900
primary
Appendix
Table 8: Originally specified offline comparisons. Lower regret is better. Holm correction applies to these three tests; the MuSiQue row is a privileged positive control, not deployment evidence.
Check
Result
Interpretation
Same local score, no partition
0.4660 vs 0.4020; gap 0.0640, CI [0.0400,0.0880]
global completion contributes beyond fixed local scores
Graded evidence coverage
both offline advantages remain positive
conclusion is not only a proof-gate effect
Outcome-supervised public interface
partition 0.4240 vs graph 0.5540; no-partition 0.4540
train outcomes and public text suffice; retrospective, no latent membership labels
Zero-shot Llama public router
MuSiQue regret 0.4860 (500 tasks)
external practitioner comparator; between public partition and public graph
GPT-OSS strict 300-task rerun
raw gain 0.0433, CI [0.0133,0.0733]
secondary matched family support
GPT-OSS source-disjoint 1k
raw gain 0.0870, CI [0.0640,0.1110]
late prospective provider-pipeline confirmation
Appendix
Table 9: Attribution and robustness checks. All rows are secondary and unadjusted.
Worker messages
Finalizer
Leave-one-out
Unified-COVER
Gain [95% CI]
Llama
Llama
.4833
.5000
.0167 [ −.0067 ,.0400]
Llama
GPT-OSS
.5767
.6000
.0233 [ −.0067 ,.0533]
GPT-OSS
Llama
.4833
.5267
.0433 [.0133,.0733]
GPT-OSS
GPT-OSS
.5533
.5900
.0367 [.0033,.0700]
Appendix
Table 10: Completed 300-task 2×2 worker-message/finalizer diagnostic. Values are raw exact accuracy; gain is Unified-COVER minus leave-one-out with a task-bootstrap 95% interval.
Department of Electrical and Computer Engineering, University of Miami, Coral Gables, FL 33146 USA · Department of Mathematics, University of Miami, Coral Gables, FL 33146 USA