Collaborative multi-agent large language models (LLMs) can solve complex reasoning tasks by decomposing roles, but reinforcement learning for such systems is limited by credit assignment: shared terminal rewards obscure individual contributions and can encourage free-riding. We introduce two optimizer-agnostic credit assignment methods for converting joint outcomes into agent-specific learning signals. Counterfactual Credit for Policy Optimization (CCPO) estimates an agent's marginal contribution by comparing the realized joint outcome with a counterfactual outcome where that agent is removed. Self-Evaluated Credit for Policy Optimization (SEPO) uses constrained self- and peer-evaluations as a verifier-anchored credit signal while keeping the external task outcome dominant. Both operate at the reward-construction layer rather than as policy optimizers, producing role-specific rewards or advantages for GRPO, GSPO, or REINFORCE++. We instantiate these credit signals in a sequential Think--Solve setting and evaluate them on mathematical reasoning benchmarks. Results show that explicit credit assignment often improves dual-agent reasoning, especially on MATH500 and several out-of-distribution settings, while gains vary across models and datasets. Our code is available at: https://github.com/bhai114/ccpo.
Figures & tables
Figure 1: Overview of the reward-construction layer. Shared rewards, CCPO, and SEPO define credit signals that are passed to a separate policy optimizer.
Model
Dataset
Untrained
Shared
CCPO
qwen2.5 1.5b instruct
MATH500
54.00
60.00
61.00
AIME25
-
-
-
AMC23
32.50
20.00
35.00
Gaokao2023
41.04
43.90
41.60
MinervaMath
12.50
13.60
14.70
llama3.1 8b instruct
MATH500
46.20
51.80
53.40
Table 1: Dual-agent reasoning performance under the GRPO optimizer with different credit signals on mathematical benchmarks (Accuracy %).
Model
Dataset
Un- trained
Shared
CCPO
SEPO
qwen2.5 1.5b instruct
MATH500
54.00
57.80
59.20
59.20
AIME25
-
6.670
6.670
6.670
AMC23
32.50
42.50
42.50
35.00
Gaokao2023
41.04
46.23
47.01
46.49
MinervaMath
12.50
16.18
15.44
16.91
olmo3 7b instruct
MATH500
82.60
87.00
84.40
87.00
Table 2: Reward-construction comparison under the fixed GSPO optimizer.
Model
Dataset
Untrained
Shared
CCPO
SEPO
GroupA
MATH500
73.60
87.20
87.00
87.40
AIME25
20.00
33.33
36.67
33.33
AMC23
52.50
72.50
80.00
82.50
Gaokao2023en
55.32
69.09
68.31
69.09
MinervaMath
20.22
27.94
30.15
25.37
Table 3: Performance (%) of heterogeneous LLM collaboration under GSPO with different credit signals. GroupA denotes olmo3-7b-instruct&qwen2.5-1.5b-instruct.
Model
Full collaboration
Agent 1 removed
qwen2.5-1.5b-instruct
61.00
56.40
llama3.1-8b-instruct
53.40
52.00
qwen2.5-7b-instruct
77.60
76.00
Table 4: Ablation of the Think–Solve handoff.
Figure 2: Reward distributions under shared rewards and CCPO credit assignment. The first row shows the Thinker and the second row shows the Solver; CCPO rewards separate contribution patterns more clearly than shared rewards (J: joint-answer score; S: Solver-only score).
Figure 3: Training efficiency and validation accuracy of GRPO with CCPO rewards versus GRPO with shared rewards.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Category
Hyperparameter
Value
Policy Optimization
Learning rate
1×10−6
Batch size
64
Samples per prompt ( n )
4
Clip ratio ( ϵ )
0.2
Gradient clip
1.0
Reward Shaping
Contribution sensitivity ( α )
1.0
Appendix
Table 5: Training and reward-shaping hyperparameters.