General-sum games are more challenging than fully cooperative or adversarial settings because agents may deceive or exploit one another. Achieving high social welfare requires sustained cooperation, while a single defection can drive agents toward a welfare suboptimal Nash equilibrium. Moreover, assuming access to other agents' rewards or internal states is often unrealistic and implicitly introduces additional cooperation into the setting. To address this issue, we introduce Reciprocal Q-Augmentation (RQA), a method that uses a latent-variable opponent model to augment the Q-function with estimates of how one's actions affects other agents. RQA allows agents to learn both cooperative and defective policies using only their own Q-function. We show that, in social dilemmas, agents that all follow the cooperative policy of RQA learn to cooperate faster and reach higher levels of cooperation than fully cooperative algorithms. Moreover, a single RQA agent that switches appropriately between its cooperative and defecting policies can induce cooperation in agents trained with standard RL algorithms, which would otherwise end up defecting. We evaluate our approach on the Iterated Prisoner's Dilemma, Iterated Traveler's Dilemma, Iterated Stag Hunt, and CleanUp-Lite. Our experiments show that the algorithms implementing RQA outperforms fully cooperative methods with shared rewards and enables cooperation with algorithms that otherwise fail to reach high social welfare.
Figures & tables
Figure 1: Schematic of the different modules of the RQA method.
Figure 2: Social welfare for the matrix games, averaged average over 5 different seeds and smoothed over 1000 training steps; the shaded area represents the 95% confidence interval. (a-b-c): the three symmetric matrix games. In all three, RQA-MASAC converges to the optimal solution and maintains it, while the other algorithms struggle. (d-e-f): same games with asymmetric payoffs, where RQA-MASAC again converges consistently to the optimum, beating the other algorithms.
Table 1: Payoff tables for the three matrix games.
Figure 3: (a-b): Plot of the social welfare and equality, averaged across 5 seeds, smoothed over 200 episodes of 64 steps each, and where the shaded area represents the 95% confidence interval. (c): efficiency of each algorithm during evaluation, the error bar represents the standard deviation.
Game
coop-RQA
PPO
IPD
0.7±0.8
3.7±0.3
ISH
2.9±1.7
3.8±0.3
ITD
4.9±0.3
8.5±0.2
Table 2: Rewards for fixed cooperative MASAC-RQA (coop-RQA) and PPO, averaged over the last 5 episodes of training.
Figure 4: Rewards for the fixed cooperative RQA agent against PPO in the three matrix games. Each line is the average over 10 runs, with the shaded area representing the 95% confidence interval.
Game
Social welfare
self-play
frozen
learning
IPD
6
2±0
5.99±0.02
5.99±0.02
ISH
8
4±0
8.00±0.01
7.98±0.04
ITD
20
14±4
19.3±1.7
16.8±2.9
Table 3: Social welfare for PPO under different settings, averaged over the last 5 episodes of training.
Figure 5: (a-c): Plots of the cooperation rate (time passed in cooperative mode by the RQA agent), and the cumulative return for the test with a frozen RQA agent implementing the switching logic against a PPO agent. The dashed lines represent the theoretical maximum welfare. (b-d): Same plots but for a learning RQA agent, implementing the switching logic while training its policies. In both cases and for all matrix games we see that the cooperation rate is very close to 1 by the end of training, and the cumulative reward converges toward the optimal solution.
self-play
co-RQA vs PPO
sw-RQA vs PPO
Attempted
11±1.7
31±18
60±29
Successful
3.3±0.6
6±5
12±5
Table 4: Total number of zaps during training for RQA-MASAC in different settings, averaged over 5 runs. The values are ×103 .
PPO vs PPO
co-RQA vs PPO
sw-RQA vs PPO
Apples
40±40
100±60
80±29
Dirt
160±40
160±90
189±22
SW
14±5
18±5
16±2
Table 5: Statistic for the different test with PPO. Values obtained averaging over 5 runs and considering the last 10 episodes of training. Apples and dirt refer to the PPO agent.