Compositional RL: Evaluating and Training LLMs for Collaborative Reasoning

Shangbin Feng; Yulia Tsvetkov; Jan Kautz; David Acuna

cs.CLOct 12, 2026

Compositional RL: Evaluating and Training LLMs for Collaborative Reasoning

Authors: Shangbin Feng, Yulia Tsvetkov, Jan Kautz, David Acuna

Organizations: University of Washington · NVIDIA

Abstract

Language models are often evaluated alone but increasingly deployed together, routed between, made to debate, ensembled, or collaboratively trained. For a model inside a system, what matters most is not how well it performs on its own, but how much the system gains from its presence. We call this a model's compositional strength and formalize it using the Shapley value of a cooperative game over coalitions of a model pool, and use a model's leave-one-out contribution to the full coalition as a tractable proxy, requiring n+1 rather than 2^n system evaluations. Evaluating individual vs. compositional strengths across five model pools, twelve collaboration methods, and twelve datasets, we find them largely uncorrelated: in 12 of 15 (pool, method-family) settings, individual strength does not translate into compositional advantages, and the coupling is weakest among general-purpose models that practitioners are most likely to deploy. Standalone benchmarks are therefore often poor proxies for models' value in collaborative systems. We then propose compositional reinforcement learning (C-RL), which trains a policy inside a live multi-model system and rewards the outcome produced by the system as a whole, rather than the policy's correctness in isolation. The leave-one-out baseline in Shapley values cancels under a group policy gradient, so this needs no additional rollout cost beyond RL on the collaborative system's outcome and never requires running a coalition without the policy. Across three collaboration protocols and eight tasks spanning reasoning, coding, and scientific discovery, C-RL produces the strongest collaborator in 21 of 24 settings, lifting collaborative performance by +4.0 to +5.4 pts on average (~3x the gain of baselines), up to +13.1 pts.

Figures & tables

Figure 1: Individual strength (solo score) vs. compositional strength (leave-one-out Shapley value in a collaborative system, § 2.1 ), broken down by model pool (rows) and collaboration-method family (columns). Each point is one model evaluated on one dataset under one collaboration method; color denotes the dataset’s task category.

\rho

is the Spearman correlation with a permutation-test

p

-value (

10{,}000

permutations within model). Individual and compositional strength are largely uncorrelated : in 12 of 15 (pool, method) panels the two show no significant relationship, visible from each panel’s near-flat fitted line and its printed

\rho

(

|\rho|\leq 0.29

throughout) and

p

-value (marked n.s. when

p>0.05

); the decoupling is sharpest among pools of general-purpose models, namely

>

100B Open Models and Proprietary Models (rows 3-4), where all six panels are n.s.

Figure 2: Frequency when each of the 12 collaboration methods achieves the best (#1), second-best (#2), or third-best (#3) full-coalition performance

P_{m,t}(\mathcal{N})

, out of all (pool, dataset) pairs in which it was evaluated (

n

= number of such pairs). Ties share a rank.

Figure 3: Best compositional strength achieved by collaboration methods, for each (model pool, dataset category). Each cell reports $\max_{m,t\in\text{category}}\bar{\phi}(m,t)$ , where $\bar{\phi}(m,t)$ is the mean compositional strength across models for $m$ , $t$ .

Collaboration protocol: LLM-Blender
Method	GPQA-dia	Theorem	KernelB	SMDD	AssayB	ChemB	Culture-H	MMLU-PX	Avg.
Base	.818	.613	.144	.635	.035	.720	.806	.804	.572
GRPO-indiv	.849	.647	.157	.635	.033	.732	.825	.822	.587
DAPO-indiv	.859	.657	.129	.635	.041	.750	.819	.812	.588
prompt-collab	.838	.630	.159	.654	.036	.742	.822	.827	.588
RAFT-collab	.808	.645	.147	.652	.031	.708	.827	.796	.577

Table 1: Collaborative performance : each policy is evaluated inside the full collaborative system. Best per column within each collaboration protocol in bold , second-best underlined ; tied values share the same mark. Avg. is the mean over all eight datasets. Confidence intervals are reported in Table 4 ; individual-mode scores in Appendix A.1 .

Figure 4: Collaboration performance on two held-out datasets, APEX-Shortlist and BixBench. Prompt-collab is not applicable because it performs no training and produces no checkpoint, so it has nothing to transfer.

Figure 5: Change in collaborative performance relative to Base under two protocols never seen during training: Mixture-of-Agents (MoA) and Heterogeneous Swarms (H-Swarm). Bars are GRPO-indiv, DAPO-indiv, RAFT-collab and C-RL, using the KernelBench and AssayBench checkpoints trained under LLM-Blender. Orange marks a method that performs worse than the untrained policy. Prompt-collab is omitted because it trains nothing and has no checkpoint to transfer.

Method	KernelBench		AssayBench
Method	score	$\Delta$	score	$\Delta$
Base	.174	–	.044	–
GRPO-indiv	.185	$+1.1$	.033	$-1.1$
DAPO-indiv	.223	$+4.9$	.026	$-1.9$
prompt-collab	.171	$-0.3$	.043	$-0.2$
RAFT-collab	.215	$+4.1$	.032	$-1.2$

Table 2: Frontier collaborators. Collaborative performance with GPT-5.5, Gemini 3.5 Flash and Claude Opus 4.8 as the pool, under LLM-Blender.

\Delta

is the change in percentage points relative to the untrained Base policy in the same pool. Best per dataset in bold .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	GPQA-dia	Theorem	KernelB	SMDD	AssayB	ChemB	Culture-H	MMLU-PX	Avg.
Base	.687	.547	.103	.279	.023	.694	.783	.758	.484
GRPO-indiv	.808	.608	.111	.322	.026	.714	.832	.784	.526
DAPO-indiv	.758	.640	.114	.261	.024	.724	.819	.820	.520
prompt-collab	.697	.580	.110	.260	.026	.700	.818	.796	.498
RAFT-collab	.768	.637	.108	.410	.017	.684	.806	.764	.524
C-RL , LLM-Blender	.889	.690	.097	.297	.024	.710	.842	.826	.547

Appendix

Table 3: Individual performance : the trained policy answers alone, outside any collaborative system. The five baselines do not depend on the collaboration protocol and are reported once; C-RL is reported per training protocol. Best per column in bold . Avg. is the mean over all eight datasets.

Collaboration protocol: LLM-Blender
Method	GPQA-dia	Theorem	KernelB	SMDD	AssayB	ChemB	Culture-H	MMLU-PX	Avg.
Base	.818	.613	.144	.635	.035	.720	.806	.804	.572
C-RL (ours)	.879	.708 ∗	.183	.686	.046	.776 ∗	.843 ∗	.840 ∗	.620 ∗
C-RL (ours)	[.808,.939]	[.662,.752]	[.120,.256]	[.606,.759]	[.024,.069]	[.738,.812]	[.814,.873]	[.808,.872]	[.602,.638]
Collaboration protocol: Multiagent Debate
Base	.768	.623	.155	.655	.036	.720	.817	.818	.574

Appendix

Table 4: Confidence intervals for C-RL. For each collaboration protocol we report the untrained Base policy alongside C-RL , with 95% confidence intervals for C-RL in brackets. ∗ marks cells where the lower bound of the C-RL interval still exceeds the Base score.

	LLM-Blender		Multiagent Debate		SLM-MUX
Task	rescue	drag	rescue	drag	rescue	drag
GPQA-dia	.162 $\to$ .040	.030 $\to$ .051	.152 $\to$ .111	.071 $\to$ .030	.172 $\to$ .172	.020 $\to$ .040
Theorem	.098 $\to$ .048	.033 $\to$ .030	.100 $\to$ .052	.025 $\to$ .037	.085 $\to$ .152	.065 $\to$ .033
KernelB	.136 $\to$ .168	.064 $\to$ .032	.144 $\to$ .160	.056 $\to$ .024	.140 $\to$ .220	.120 $\to$ .020
SMDD	.438 $\to$ .438	.080 $\to$ .088	.474 $\to$ .423	.095 $\to$ .095	.504 $\to$ .526	.051 $\to$ .073
AssayB	.006 $\to$ .009	.000 $\to$ .000	.000 $\to$ .000	.000 $\to$ .003	–	–

Appendix

Table 5: Where the policy and the system disagree. Each cell reads base

\to

C-RL . Rescue is the fraction of test items the policy answers incorrectly on its own but the system gets right; drag is the fraction the policy answers correctly on its own but the system gets wrong. Cells marked -- lack a usable log. Drag falls under all three protocols.

What happens if aromatic nitro compounds are mixed with sodium hydroxide? (A) Nothing happens (B) They may explode (C) They decompose (D) They form a salt Correct: B
	option proposed by each model				system	policy len.
	Policy	gpt-oss	kimi-k3	nemotron	system	policy len.
Base	A	A	B	A	A	34,350
C-RL	B	A	B	A	B	4,059
Which of the following statements is true about halogenated organic compounds? (A) They are incompatible with strong oxidizing and reducing agents. (B) They react slowly with metals. (C) The more halogen atoms, the more reactive they become. (D) They are highly reactive. Correct: A
Base	D	A	A	A	D	7,292

Appendix

Table 6: Two ChemBench items where C-RL changes the system’s answer. Each model proposes a candidate and a fuser combines them; the letter shown is the option each candidate ends on, with the correct option in bold . In the first item the untrained policy proposes the wrong option and the system follows it; after C-RL the policy proposes the correct option and the system does too. In the second the untrained policy dissents from three collaborators that are all correct and the system adopts the policy’s error. The last column gives the length of the policy’s own candidate.

Q. How many ways are there to arrange the letters in the word BANANA up to the symmetries of the word?
Base	The phrase ”up to the symmetries” implies treating the arrangement as a bracelet, where two arrangements are considered identical if one can be rotated or flipped to match the other (the dihedral group $D_{6}$ ). Applying Burnside’s Lemma, the identity symmetry fixes all 60 distinct linear permutations, while the five non-trivial rotations fix none because the letter counts (3 A’s, 2 N’s, 1 B) cannot form the required periodic patterns. Among the six reflections, the three axes passing through opposite letter positions each fix 4 arrangements (by assigning the AA and NN pairs to the two swapped slots and the remaining A and B to the fixed slots), while the three axes passing between slots fix none. Summing the fixed counts gives $60+3\times 4=72$ , and dividing by the 12 total symmetries yields 6 distinct arrangements. Final Answer: 6
C-RL	First, calculate the number of distinct linear permutations of the letters in BANANA, which is $6!/(3!\,2!)=60$ . The phrase ”up to symmetries” for a linear word means we consider an arrangement equivalent to its reverse. Since the letter counts for ’A’ (3) and ’B’ (1) are odd, no arrangement can be a palindrome, meaning every permutation is distinct from its reverse and forms a unique pair. Dividing the total linear permutations by 2 to account for this reversal symmetry gives 30 distinct arrangements. Final Answer: 30
Q. Determine the product obtained when 3,4-dimethylhexanedial is reacted with the following set of reagents: 1. KOH, H2O, THF, Heat 2. CH3CH2MgBr, H3O+ 3. PCC, CH2Cl2 4. O3, H2O A. 3,4-dimethyl-5,6-dioxooctanoic acid $\ldots$ D. 3,4-dimethyl-5,6-dioxooctanal
Base	Base-catalyzed aldol condensation of 3,4-dimethylhexanedial forms a cyclohexene derivative, which subsequently undergoes Grignard addition and PCC oxidation to introduce an ethyl ketone group. Ozonolysis cleaves the ring double bond to yield an acyclic chain containing aldehyde and ketone functionalities. The resulting 8-carbon skeleton and carbonyl positions precisely match 3,4-dimethyl-5,6-dioxooctanal, while ozonolysis of an internal alkene does not produce carboxylic acids, eliminating option A. Answer: D
C-RL	3,4-Dimethylhexanedial undergoes an intramolecular aldol condensation under basic conditions to form a five-membered cyclic $\alpha$ , $\beta$ -unsaturated aldehyde. Subsequent Grignard addition and PCC oxidation convert the side-chain aldehyde into an ethyl ketone. Finally, oxidative ozonolysis cleaves the ring double bond, opening the ring to yield a linear dicarbonyl carboxylic acid chain. Counting the carbons and functional groups confirms the product is 3,4-dimethyl-5,6-dioxooctanoic acid. Answer: A

Appendix

Table 7: Collaborative responses the system produces, before and after C-RL. Two items the untrained system answers incorrectly and the C-RL system answers correctly, quoted in full from the logs (the fuser’s internal deliberation is omitted). Both are under LLM-Blender, on TheoremQA and GPQA-Diamond respectively; in both the disagreement is interpretive rather than arithmetic. The second item’s two remaining distractors are elided: they are identical to one another in the source dataset and neither system proposes either of them.

Model pool	QA	Reasoning	Coding	Pluralism & Safety	Science
Specialized Expert	65.8 $\to$ 72.4 (+6.6)	32.3 $\to$ 45.5 (+13.1)	44.1 $\to$ 65.5 (+21.4)	38.7 $\to$ 54.0 (+15.3)	41.2 $\to$ 55.4 (+14.2)
8-30B Open	82.9 $\to$ 94.9 (+12.0)	24.2 $\to$ 35.6 (+11.4)	70.7 $\to$ 82.5 (+11.8)	78.2 $\to$ 81.2 (+3.1)	57.7 $\to$ 60.8 (+3.1)
$>$ 100B Open	80.0 $\to$ 83.7 (+3.7)	48.1 $\to$ 60.0 (+11.9)	48.8 $\to$ 71.2 (+22.4)	77.0 $\to$ 82.0 (+4.9)	53.7 $\to$ 68.3 (+14.6)
Proprietary API	86.8 $\to$ 90.0 (+3.2)	58.5 $\to$ 69.4 (+10.9)	66.7 $\to$ 94.1 (+27.5)	32.1 $\to$ 37.7 (+5.6)	73.1 $\to$ 80.6 (+7.5)
Diverse Academic	66.1 $\to$ 70.0 (+3.9)	32.4 $\to$ 65.3 (+32.9)	37.4 $\to$ 52.7 (+15.4)	64.6 $\to$ 71.6 (+7.0)	50.4 $\to$ 55.1 (+4.7)

Appendix

Table 8: The best-performing collaboration method across 25 (model pool, task category) values underlying the collaboration-effectiveness claim in § 2.3 . The mean of the 25 gains is

+11.5

with a

95\%

CI of

[+8.7,+14.7]

Figure 6: Maximum compositional strength achieved by any collaboration method, for each (model pool, dataset) pair: the per-dataset counterpart to the domain-level aggregation in Figure 3 . Datasets are grouped by category (column separators and colored labels); each cell reports $\max_{m}\bar{\phi}(m,t)$ for dataset $t$ .

Figure 7: Leave-one-out strength against the Banzhaf value (left) and the exact Shapley value (right) on the Specialized Expert pool. Each point is one (model, dataset, method) cell, colored by collaboration method; the dotted line is

y=x

and the dashed line is the least-squares fit.

Pool	$n$	Members
Specialized Expert	4	Qwen2.5-7B-Instruct, Qwen2.5-Math-7B, Qwen2.5-Coder-7B-Instruct, Qwen2.5-7B-Instruct-1M
8–30B Open	7	Qwen3-14B, Nemotron-3-Nano-30B-A3B, gpt-oss-20b, Llama-3.1-8B-Instruct, gemma-3-12b-it, phi-4, DeepSeek-R1-Distill-Qwen-14B
$>$ 100B Open	4	gpt-oss-120b, deepseek-v4-pro, kimi-k2.6, nemotron-3-ultra
Proprietary API	3	gpt-5.5, gemini-3.5-flash, bedrock-claude-opus-4-8
Diverse Academic ( Feng et al., 2026e )	8	parti_0_full, parti_1_full, parti_2_full, parti_3_full, parti_4_full, parti_5_full, parti_6_full, parti_7_full

Appendix

Table 9: The five model pools. Each pool defines a separate cooperative game; compositional strengths are computed within a pool and never compared across pools. Diverse Academic is a set of independently post-trained research models, each specialised for a different task (narrative claim verification, tool use, planner optimisation, architecture ablation, instruction following, instruction induction, conditional-probability estimation, and multi-document reasoning respectively).

Dataset	Category	Test items
ARC-Challenge ( Clark et al., 2018 )	QA	1172
MMLU-Redux ( Gema et al., 2025 )	QA	1000
GPQA-Diamond ( Rein et al., 2024 )	Reasoning	99
BBH ( Suzgun et al., 2023 )	Reasoning	1000
TheoremQA ( Chen et al., 2023 )	Reasoning	400
HumanEval ( Chen et al., 2021 )	Coding	114

Appendix

Table 10: The twelve evaluation datasets. Coalition values are computed on the full test split. All scoring is rule-based or execution-based; no model acts as a judge.

Dataset	Category	Development	Test
GPQA-Diamond ( Rein et al., 2024 )	Answer-matching	99	99
TheoremQA ( Chen et al., 2023 )	Answer-matching	400	400
MMLU-ProX ( Xuan et al., 2025 )	Answer-matching	500	500
CulturalBench-Hard ( Chiu et al., 2024 )	Answer-matching	613	613
ChemBench ( Mirza et al., 2025 )	Answer-matching	498	500
KernelBench ( Ouyang et al., 2025 )	Execution-verified	125	125

Appendix

Table 11: The ten datasets used in § 3 . The eight training datasets (top) list both their MoCo development split, which we further divide

8{:}2

into training and validation for C-RL, and their test split, reported for reference. APEX-Shortlist and BixBench (bottom) are used only for the held-out generalization evaluation of § 3.4 , so only their test-set size is listed. Answer-matching , execution-verified , and reference-ranking follow the scoring split of § 3.2 .

Multi-LLM Collaboration

Multi-LLM collaboration studies how large language models share work, exchange information, and combine outputs to improve capability, reliability, or cost.

See all 85 papers

Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
Aug 18, 2026Yunhao Yang, Yuexin Bian, Yunjie Tian +6RL for Language Model Reasoning Reinforcement Learning
Johns Hopkins University · UC San Diego · Independent Researcher +1
COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference
Sep 22, 2026Norah Alballa, Wenxuan Zhang, Salma Kharrat +4LLM Inference Efficiency LLM Routing
KAUST · LUMS
TRACER: Turn-level Regret Matching with Inner Reinforcement Credit for Cooperative Multi-LLM Reasoning
May 27, 2026Chusen Li, Zhou Liu, Shuigeng Zhou +1Regret Minimization Multi-Agent Reasoning
1Fudan University · 2Zhongguancun Academy · 3Academy for Advanced Interdisciplinary Studies, Peking University

Aug 18, 2026J/K move · Enter open · S save

Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

Yunhao Yang, Yuexin Bian, Yunjie Tian, Di Fu+5

Johns Hopkins University · UC San Diego · Independent Researcher+1

RL for Language Model Reasoning Reinforcement Learning Cooperative MARL Heterogeneous MARL RL for VLMs

Open paper AskPDF arXiv

Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs. Code is available at https://github.com/DrStranded/Co-RL.

Abstract

Multi-LLM Collaboration

Explore similar work

Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference

TRACER: Turn-level Regret Matching with Inner Reinforcement Credit for Cooperative Multi-LLM Reasoning