Credit Where Credit Is Due: Contrastive Attribution Topology Search for Multi-Agent Systems

Fabian Tritthart; Tom Zehle

cs.CLOct 21, 2026

Credit Where Credit Is Due: Contrastive Attribution Topology Search for Multi-Agent Systems

Authors: Fabian Tritthart, Tom Zehle

Organizations: University of Freiburg

Abstract

An LLM multi-agent system (MAS) is only as good as the agents it chooses to trust: its accuracy and cost hinge on its topology, meaning which agents are included and how they communicate. A single task score, however, does not reveal which agents help or hinder performance. We propose Contrastive Attribution Topology Search (CATS), which searches for compact teams within a fixed supergraph of specialized agents. An LLM attributer compares execution traces across candidate topologies and assigns credit to individual agents. CATS uses these credits to update supergraph edges while progressively pruning weak edges and agents, yielding a small team for deployment. With small open-weight models and matched candidate evaluations, CATS improves mean accuracy on BigCodeBench by 5.3 percentage points over score-only heuristic search while using 54 $\%$ fewer deployment tokens. On MBPP, CATS matches the strongest baselines with fewer tokens than the other searches. In a HotpotQA prompt-poisoning stress test, CATS excludes all adversarially prompted agents in all three runs, whereas random search retains some at comparable accuracy. Per-agent credit thus supports both team optimization and the pruning of potentially harmful agents.

Figures & tables

Figure 1: CATS pipeline. At each of

T

steps,

K

subgraphs are sampled from the supergraph and evaluated on

N

tasks. The depicted supergraph is a simplified example.

Figure 2

Figure 3: Selected topologies on HotpotQA (seed 13). Poisoned agents are in orange.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Initial supergraphs and fixed baseline. Supergraph edges run forward between layers, and start edges select entry agents. Orange nodes in (c) are adversarially prompted. These labels are hidden from the search and attributer. Agent codes are listed in Table 3 .

Benchmark	Seed	Agents	Edges
BigCodeBench	13	DC, IM, TD	start $\to$ DC, DC $\to$ IM, IM $\to$ TD
	14	DC, VE	start $\to$ DC, start $\to$ VE
	15	DB, DC	start $\to$ DB, start $\to$ DC
MBPP	13	DC, IM, TD	start $\to$ DC, start $\to$ IM, start $\to$ TD
	14	DC, IM, TL	start $\to$ DC, start $\to$ IM, start $\to$ TL, TL $\to$ IM
	15	DC, IM	start $\to$ DC, start $\to$ IM, DC $\to$ IM

Appendix

Table 4: Selected deployment topologies

G^{\star}

. These are the highest-scoring step-4 candidates on the search set and are the systems reported as “CATS” in the main tables. Agent codes are resolved in Table 3 . Each listed edge is directed. start denotes an entry edge.

Hyperparameter	Value
Search tasks	$100$
Seeds (search and evaluation)	$13$ , $14$ , $15$
Search steps, $T$	$4$
Candidates per step, $K$	$4$
Learning rate, $\eta$	$6$
Unfired-edge weight, $\rho$	$0.25$

Appendix

Table 5: Run settings. Left: CATS hyperparameters, shared across benchmarks. Right: per-benchmark inference settings and mean CATS cost. Attribution accounted for 9–11% of coding-search tokens.

Method/read-out	Definition
Random search	Draw 16 candidates at fixed $\alpha=0.15$ . Return the best search-set score, breaking ties by fewer agents and then random.
Score-only search	Sample $K=4$ candidates at each of $T=4$ steps. For candidate $k$ , compute the score-only advantage $A_{k}=\bar{s}_{k}-(K-1)^{-1}\sum_{j\neq k}\bar{s}_{j}$ , where $\bar{s}_{k}$ is its mean search-set score. Assign $A_{k}$ to every active agent and update edge logits using the same learning rate, unfired-edge rule, and pruning schedule as our method, but without LLM attribution. This heuristic omits the Bernoulli score-function factor $I_{e}-\alpha_{e}$ required by REINFORCE [ 11 ] . Return the highest-scoring step- $T$ candidate, breaking ties by fewer agents and then the earlier candidate.
Direct LLM (MBPP)	One greedy zero-shot call, without tools, execution feedback, or retry. The prompt follows the task template of Austin et al. [1] .
Direct LLM (BigCodeBench)	One greedy zero-shot call, without tools, execution feedback, or retry. The prompt uses the instruction_prefix and response_prefix defaults from the official BigCodeBench generation harness .
ReAct single agent	One native-tool loop with the same prompt body on both benchmarks, and a tool-call budget of five. ReAct is the method of Yao et al. [13] . The exact string is from LangChain Hub entry hwchase17/react . {tools} and {tool_names} are filled with the environment toolset and the runtime’s native tool-use protocol block.
Hand-crafted MAS	A fixed planner–executor–verifier topology (Figure 4(d) ).

Appendix

Table 6: Operational definitions of the baselines. All baselines use the same task model, grader, decoding limits, and seed-specific held-out tasks.

Benchmark	Snapshot and search split	Grading
BigCodeBench	v0.1.4 Instruct, 100 disjoint tasks	Last stored program against hidden tests
MBPP	Revision 4bb6404f , 100 disjoint tasks	Last stored program against hidden tests
HotpotQA	distractor , 100 disjoint tasks	Exact match on last submit_answer phrase

Appendix

Table 7: Dataset splits and grading. Each held-out set contains 100 tasks.

	Seed 13				Seed 14				Seed 15
Method	Score	Tokens	$V/E$	$P$	Score	Tokens	$V/E$	$P$	Score	Tokens	$V/E$	$P$
BigCodeBench
CATS	53	7.16	3/3	—	43	4.70	2/2	—	48	8.28	2/2	—
Random	35	9.06	5/6	—	28	11.08	2/2	—	42	5.93	1/1	—
Score-only search	45	6.62	3/3	—	37	23.71	4/5	—	46	13.75	5/8	—
MBPP

Appendix

Table 8: Per-seed evaluations supporting the headline comparisons. Each seed groups results for the selected topologies on its 100-task held-out set. Scores are percentages. Tokens are thousands of input plus output tokens per task.

V/E

counts selected agents/edges, including start edges.

P

counts poisoned agents (HotpotQA only).

Panel of Figure 5 — Figure 5: CATS dynamics on MBPP. The corresponding BigCodeBench panels appear in the main paper. Lines show means over three seeds. Bands show sample standard deviations.

		Pooled		Within-task
Seed	$n$ pairs	$r$	$\rho$	$r$	$\rho$
13	1600	0.876	0.854	0.785	0.742
14	1600	0.879	0.834	0.841	0.802
15	1596	0.834	0.846	0.746	0.687
Mean $\pm$ SD over seeds	—	0.863 $\pm$ 0.025	0.845 $\pm$ 0.010	0.791 $\pm$ 0.048	0.744 $\pm$ 0.058

Appendix

Table 9: Credit–success consistency check. Correlation between the mean credit over participating agents,

\bar{c}_{k,q}

, and the binary task score

s_{k,q}

, over all (candidate, task) pairs. Within-task columns correlate within each query. Scores are binary, so Pearson

r

is the point-biserial coefficient. Four pairs are dropped from seed 15 due to one unparsed transcript. The attributer sees system_score , so this is not independent validation of credit quality.

Benchmark	Method	Task model	Attributer	Total
BigCodeBench	CATS	14.88	1.54	$16.42\pm 5.35$
	Random search	16.49	0	$16.49\pm 5.66$
	Score-only search	22.89	0	$22.89\pm 10.89$
MBPP	CATS	12.00	1.53	$13.53\pm 1.51$
	Random search	11.45	0	$11.45\pm 2.71$
	Score-only search	20.89	0	$20.89\pm 3.47$

Appendix

Table 10: Offline search tokens, separate from deployment. Millions of input plus output tokens for 16 candidates on 100 search tasks, averaged over seeds 13–15; totals include sample SD.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,039 papers

Agents that Matter: Optimizing Multi-Agent LLMs via Removal-Based Attribution
May 26, 2026Mingyu Lu, Yushan Huang, Chris Lin +1Multi-Agent System Optimization Cooperative Game Theory
Paul G. Allen School of Computer Science & Engineering, University of Washington
CANTANTE: Optimizing Agentic Systems via Contrastive Credit Attribution
May 13, 2026Tom ZehleMulti-Agent LLM Systems Multi-Agent System Optimization
University of Freiburg · ELLIS Institute Tübingen
The Trace Is the State: Exact Credit Assignment for LLM Agent Teams
Mar 6, 2026Yanjun Chen, Yirong Sun, Hanlin Wang +5Credit Assignment Counterfactual Credit Assignment
Eastern Institute of Technology · The Hong Kong Polytechnic University · Harbin Institute of Technology

May 26, 2026J/K move · Enter open · S save

Agents that Matter: Optimizing Multi-Agent LLMs via Removal-Based Attribution

Mingyu Lu, Yushan Huang, Chris Lin, Su-In Lee

Paul G. Allen School of Computer Science & Engineering, University of Washington

Multi-Agent System Optimization Cooperative Game Theory Feature Attribution Multi-Agent LLMs

Open paper AskPDF arXiv

As multi-agent systems (MAS) become increasingly complex, identifying the contributions of individual agents is critical for system optimization. However, existing approaches lack a rigorous, unified framework for credit assignment. In this work, we formalize agent attribution as a cooperative game, parameterized by the coalition distribution, removal protocol, and target metric. Using this framework, we show that Leave-One-Out (LOO) identifies bottleneck agents as effectively as combinatorial methods, but at a fraction of the computational cost. We also demonstrate that removal protocols induce distinct games: Agent ablation isolates structural bottlenecks, whereas introspective LLM judges fail to faithfully approximate this behavior. Furthermore, to evaluate the utility of specific agent backbones, we introduce attribution via model replacement. By substituting underlying models of low-contribution agents, we improve task performance by up to 17% while reducing cost by up to 35% across three benchmarks. Finally, we apply our framework to audit a medical MAS, revealing that agent contributions to diagnostic accuracy and ethical behavior are often decoupled. By intervening on counterproductive roles, we observe an increase in ethics alignment while maintaining diagnostic accuracy. Overall, this work provides a principled approach for cost-effective MAS attribution and intervention.

Abstract

Multi-Agent LLM Systems

Explore similar work

Agents that Matter: Optimizing Multi-Agent LLMs via Removal-Based Attribution

CANTANTE: Optimizing Agentic Systems via Contrastive Credit Attribution

The Trace Is the State: Exact Credit Assignment for LLM Agent Teams