Coopshift: Post-Training Amplifies The Memory Curse In LLM Cooperation

Jiayuan Liu; Shiyi Du; Tianqin Li; Emanuel Tewolde; Keyang Xuan; Vincent Conitzer

cs.GTOct 16, 2026

Coopshift: Post-Training Amplifies The Memory Curse In LLM Cooperation

Authors: Jiayuan Liu, Shiyi Du, Tianqin Li, Emanuel Tewolde, Keyang Xuan, Vincent Conitzer

Organizations: Carnegie Mellon University · Foundations of Cooperative AI Lab (FOCAL) · The University of Texas at Austin

Abstract

How does post-training change cooperation during repeated interaction? Within each of seven released model lineages, we compare pretrained (Base), supervised fine-tuned (SFT), and subsequent preference- or reward-optimized (Aligned) checkpoints across seven social dilemmas. Copies of each checkpoint play one another for 200 rounds, with visible history limited to the last two or forty rounds. The memory curse refers to lower cooperation when agents can see more past rounds. Our central finding is that post-training amplifies this decline. Averaged across the six binary-action games, expanding visible history reduces cooperation at every SFT and Aligned checkpoint. At each post-trained stage, these declines exceed Base's in six of seven lineages. Most of this average amplification is already present at SFT. With long history, both post-trained stages also cooperate less than Base on average in each binary-action game. In the symmetric binary games, their fitted cooperation rates fall more steeply on average than Base's as per-round cooperation costs rise, benefits fall, or noncooperation becomes more rewarding. The difference in cooperation rates between Aligned and SFT also changes from one-shot decisions to repeated play. When given the same past records, their choices depend on whether the instructions describe a final round or possible future interaction. Post-training evaluation should therefore go beyond isolated choices to assess whether cooperation is sustained in repeated interaction and how it depends on payoffs and visible history.

Figures & tables

Figure 1: CoopShift: cooperation across released post-training stages. Seven lineages (21 checkpoints) play seven social dilemmas across 84 payoff configurations. One-shot decisions complement 200-round self-play. A: Volunteer’s Dilemma with a forty-round history limit (HL40): mean cooperation across seven lineages versus relative volunteering cost

c/B

, where

c

is cost and

B

each player’s benefit. The y axis spans 30–85%. B: Forty-minus-two-round history responses, in percentage points (pp): mean

\pm

standard deviation (SD) across seven lineage summaries, each averaging the six binary games (Table 3 ).

Game	Decision and tension	Meaning of the raw main parameter
PGG	Contribute a unit to a shared pool or retain it.	The marginal per-capita return (MPCR), $\alpha=r/n$ , is each player’s return per contributed unit, where $r$ is the pool multiplier and $n$ the group size. Higher $\alpha$ lowers private cost $1-\alpha$ .
CPR	Restrain extraction or extract more; each unit earns less as total extraction rises.	Low extraction is 1 and high extraction is $h>1$ . Higher $\theta$ increases high extraction’s private advantage at fixed $h$ ; payoffs depend on current-round extraction.
Donation-PD	Pay $c$ to give $b$ to each other player, or give nothing.	$c$ is the donor’s cost and $b$ the benefit to each of two others. Higher $R=2b/c$ , at fixed $c$ , raises external returns; private cost stays $c$ .
Volunteer	Supply a shared benefit or rely on another volunteer; one suffices.	Each player receives $B$ if anyone volunteers; every volunteer pays $c$ , so extra volunteers duplicate costs. Higher $\rho=c/B$ , at fixed $B$ , increases volunteering cost.
Assurance	Join a threshold project or take a sure outside payoff.	A successful contributor earns a fraction $\lambda$ more than the sure outside payoff. Success requires at least $k$ contributors, including itself; unsuccessful contributors receive zero. Higher $\lambda$ raises the reward for successful coordination.
TD	Claim an amount; the lower claimant gets a bonus, the higher a penalty.	$K$ is the number of legal claim levels. Higher $\tau$ , at fixed $K$ , raises bonus/penalty and one-step undercutting gain. Outcome: claim normalized to $[0,1]$ .

Table 1: What the games measure. The main parameter changes a cost, return, or coordination reward, depending on the game.

Game / stage	$D_{1}$	$D_{2}$	$D_{3}$	$D_{4}$	Drop
PGG : MPCR	0.8	0.6	0.5	0.4	–
Base (B)	$59.8$	$58.6$	$57.1$	$57.2$	$+2.9$
SFT (S)	$43.2$	$38.6$	$35.4$	$33.7$	$+9.8$
Aligned (A)	$42.2$	$38.7$	$35.6$	$31.4$	$+10.5$
CPR : $\theta$	0.2	0.4	0.6	0.8	–
Base (B)	$68.0$	$63.7$	$62.3$	$65.6$	$+2.6$

Table 2: Cooperation across four payoff values at HL40. Entries are means across seven lineages (cooperation in %; TD normalized claims multiplied by 100), after averaging seeds and secondary parameters. The final column is the fitted decline across all four points (pp); the numerical parameter spacing is retained.

History change or paired difference	Mean $\pm$ SD	95% interval	Negative
Base: HL40 minus HL2	$-1.17\pm 3.92$	$[-4.79,+2.45]$	2/7
SFT: HL40 minus HL2	$-8.11\pm 5.46$	$[-13.16,-3.05]$	7/7
Aligned: HL40 minus HL2	$-10.93\pm 6.40$	$[-16.85,-5.01]$	7/7
SFT minus Base history change	$-6.93\pm 7.30$	$[-13.68,-0.19]$	6/7
Aligned minus Base history change	$-9.76\pm 7.47$	$[-16.67,-2.85]$	6/7
Aligned minus SFT history change	$-2.82\pm 4.37$	$[-6.87,+1.22]$	6/7

Table 3: The memory curse and its amplification after post-training. The first three rows report HL40 minus HL2; the last three compare these changes between stages. Entries give mean

\pm

SD across seven lineages, in pp, after averaging the six binary games equally within each lineage. Stage differences are calculated within lineage. The exploratory 95%

t

intervals use these seven estimates; the last column counts negative estimates.

Figure 2: Which stage loses more cooperation when shown a longer history? Panels compare SFT vs Base (

K_{S/B}

), Aligned vs Base (

K_{A/B}

), and Aligned vs SFT (

K_{A/S}

). Left of zero means a more negative change from HL2 to HL40 at the first-named stage: a larger decrease or smaller increase. TD reports normalized claims. Each game/panel has 28 colored points: seven lineages

\times

four primary payoff values, averaging five seeds and secondary parameters. Black diamonds average points equally within and across lineages. All three panels use the same percentage-point scale.

Game	One-shot	Repeated (HL40)	Change	Lineages with
	Aligned $-$ SFT	Aligned $-$ SFT	Repeated $-$ one-shot	a decrease
PGG	$+8.62\pm 6.78$	$-0.77\pm 20.44$	$-9.39\pm 20.00$	5/7
CPR	$+3.47\pm 22.35$	$-7.08\pm 14.96$	$-10.55\pm 19.14$	5/7
Donation-PD	$-1.71\pm 7.11$	$-4.21\pm 5.51$	$-2.51\pm 9.25$	4/7
Volunteer	$+2.66\pm 14.47$	$-7.25\pm 11.63$	$-9.91\pm 5.18$	7/7
Assurance	$+1.93\pm 13.67$	$+1.59\pm 8.14$	$-0.35\pm 8.96$	3/7

Table 4: Does Aligned cooperate more than SFT in one-shot decisions and repeated play? The first two columns report Aligned minus SFT; positive values mean more cooperation at Aligned. Change is repeated minus one-shot, computed within each lineage before averaging; negative values mean a smaller Aligned-minus-SFT difference in repeated play. These columns give mean

\pm

SD across seven model lineages, in pp. SD describes variation in effect magnitude and direction across model lineages; the last column counts lineages with a negative change. TD measures normalized claims; Trust averages roles.

Game	Final	Continue	Mean $\Delta$	$\Delta$ Base	$\Delta$ SFT	$\Delta$ Aligned
PGG	$73.7\pm 14.3$	$73.2\pm 14.4$	$-0.5\pm 4.0$	$-0.1\pm 3.4$	$+2.5\pm 6.4$	$-3.9\pm 6.8$
CPR	$43.9\pm 18.4$	$59.0\pm 15.2$	$+15.1\pm 9.7$	$+6.2\pm 4.0$	$+14.0\pm 10.4$	$+25.1\pm 20.6$
Donation-PD	$61.0\pm 21.8$	$64.0\pm 14.5$	$+3.0\pm 9.9$	$-0.8\pm 7.2$	$+5.5\pm 12.7$	$+4.4\pm 12.2$
Volunteer	$78.3\pm 13.7$	$81.3\pm 10.5$	$+3.0\pm 7.5$	$+3.6\pm 5.1$	$+2.8\pm 10.8$	$+2.8\pm 11.0$
Assurance	$72.9\pm 18.0$	$75.1\pm 12.2$	$+2.2\pm 10.8$	$+1.0\pm 6.3$	$+3.8\pm 16.6$	$+1.6\pm 12.9$
TD	$53.2\pm 14.6$	$40.7\pm 9.8$	$-12.6\pm 8.2$	$-8.4\pm 9.5$	$-11.4\pm 7.2$	$-17.9\pm 14.3$

Table 5: How instructions about future rounds change the current decision. Final says this is the last round; Continue states a 99% chance of another. Cooperation is in percent; TD claims are rescaled from 0 (lowest allowed) to 100 (highest).

\Delta

is Continue minus Final , in pp. Entries are mean

\pm

SD across seven model lineages, after averaging payoff settings and 22 history conditions within lineage. SD summarizes variation across model lineages. The first three numeric columns also average stages; the last three keep stages separate. Trust equally weights the investor and a representative trustee.

Future instruction	Stage	Real	Masked	Real $-$ Masked
Continue	SFT (S)	$41.9\pm 31.7$	$58.2\pm 27.6$	$-16.3\pm 12.3$
	Aligned (A)	$41.4\pm 35.3$	$63.3\pm 29.9$	$-22.0\pm 12.9$
Final	SFT (S)	$35.8\pm 22.2$	$51.5\pm 22.0$	$-15.7\pm 21.9$
	Aligned (A)	$34.6\pm 26.4$	$46.6\pm 23.8$	$-12.0\pm 32.0$

Table 6: How showing past records changes PGG cooperation. Real shows the records; Masked hides them. The first two numeric columns give rates (%); the last subtracts the second from the first to give the effect of showing records (pp). Negative values mean less cooperation with records visible. Entries are mean

\pm

SD across seven lineages. All four conditions use the same payoff and record weights.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Lineage	Stage	Hugging Face repository
OLMo-3-7B	Base	allenai/Olmo-3-1025-7B
OLMo-3-7B	SFT	allenai/Olmo-3-7B-Instruct-SFT
OLMo-3-7B	Aligned (DPO)	allenai/Olmo-3-7B-Instruct-DPO
SmolLM2-1.7B	Base	HuggingFaceTB/SmolLM2-1.7B
SmolLM2-1.7B	SFT	HuggingFaceTB/SmolLM2-1.7B-sft-only
SmolLM2-1.7B	Aligned (DPO)	HuggingFaceTB/SmolLM2-1.7B-Instruct

Appendix

Table 7: All 21 checkpoints in the primary panel: one Base, SFT, and Aligned release per lineage. Identifiers link to the Hugging Face repositories; Nemotron’s SFT and RLHF suffixes identify checkpoint subdirectories. Aligned training objectives appear in parentheses.

Lineage	$H_{B}$	$H_{S}$	$H_{A}$	$K_{S/B}$	$K_{A/B}$	$K_{A/S}$
OLMo	$+1.7\pm 1.4$	$-13.5\pm 1.4$	$-8.9\pm 0.8$	$-15.2\pm 2.8$	$-10.7\pm 2.0$	$+4.6\pm 1.0$
SmolLM2	$+0.0\pm 0.1$	$-16.1\pm 2.4$	$-18.1\pm 2.3$	$-16.2\pm 2.4$	$-18.1\pm 2.4$	$-1.9\pm 2.0$
DeepSeekMath	$+0.9\pm 0.2$	$-5.6\pm 0.8$	$-9.7\pm 1.1$	$-6.5\pm 0.8$	$-10.7\pm 1.0$	$-4.1\pm 1.2$
InternLM2	$-7.6\pm 0.8$	$-2.3\pm 0.4$	$-2.8\pm 0.5$	$+5.3\pm 0.9$	$+4.8\pm 0.6$	$-0.5\pm 0.6$
Nemotron	$-6.0\pm 1.7$	$-10.9\pm 1.5$	$-19.7\pm 2.7$	$-4.9\pm 2.9$	$-13.7\pm 3.8$	$-8.8\pm 3.0$
Tulu	$+1.4\pm 0.8$	$-6.0\pm 0.9$	$-12.9\pm 1.1$	$-7.4\pm 0.9$	$-14.3\pm 0.3$	$-6.9\pm 1.0$

Appendix

Table 8: History changes for each lineage, averaged over the six binary games. Each entry is mean

\pm

SD in pp across five matched run IDs, after averaging payoff configurations within game and games equally. The SD summarizes variation among repeated runs within each lineage.

Figure 3: Fitted payoff-drop differences relative to Base. Positive values mean a larger fitted decline after post-training across the tested payoff span. Each point is a lineage difference, averaging runs and the other payoff parameters; diamonds and bars show mean

\pm

SD across seven lineages. Both panels use the same scale. TD measures normalized claims.

Game	Total	Between lineage	Within lineage/config
PGG	1586.55	1439.81	146.74
CPR	829.71	745.27	84.44

Appendix

Table 9: Where does variation in the HL40 SFT–Base difference come from? We separate differences between lineage averages from differences among payoff settings within a lineage. Entries use equal lineage/configuration weights and population variance, in

\mathrm{pp}^{2}

, calculated from the observed estimates.

Game / stage	$D_{1}$	$D_{2}$	$D_{3}$	$D_{4}$	Fitted drop
PGG : MPCR	0.8	0.6	0.5	0.4	–
Base (B)	$59.8$	$58.6$	$57.1$	$57.2$	$+2.9$
SFT (S)	$43.2$	$38.6$	$35.4$	$33.7$	$+9.8$
Aligned (A)	$42.2$	$38.7$	$35.6$	$31.4$	$+10.5$
SFT $-$ Base	$-16.6$	$-20.0$	$-21.6$	$-23.5$	$+6.9$
Aligned $-$ Base	$-17.7$	$-19.9$	$-21.5$	$-25.8$	$+7.6$

Appendix

Table 10: Full HL40 payoff surfaces and paired stage differences. Each game’s first row gives four values of its main payoff parameter (

D_{1}

–

D_{4}

), ordered toward higher cooperation costs, lower benefits, or greater rewards for noncooperation. Entries average seven lineages equally, after averaging seeds and secondary parameters within lineage. Stage rows give cooperation rates (%; TD normalized claims multiplied by 100); difference rows give SFT minus Base and Aligned minus Base (pp). Negative differences mean lower cooperation than Base (lower claims in TD). When a negative difference becomes more negative from

D_{1}

D_{4}

, the post-trained stage falls further below Base. The final column uses a straight-line fit to all four points (Equation 2 ). In a stage row, a positive drop means a fitted decline; in a difference row, it is the post-trained drop minus Base’s, so a positive value means a larger fitted decline than Base’s.

Lineage / stage	$D_{1}$	$D_{2}$	$D_{3}$	$D_{4}$	Fitted drop
PGG : MPCR	0.8	0.6	0.5	0.4	–
OLMo-3-7B
Base (B)	$20.3\pm 23.7$	$10.1\pm 7.3$	$3.3\pm 2.6$	$4.8\pm 1.3$	$+17.0\pm 24.9$
SFT (S)	$8.5\pm 2.1$	$7.6\pm 3.0$	$6.2\pm 1.5$	$4.9\pm 0.4$	$+3.6\pm 2.1$
Aligned (A)	$16.1\pm 6.1$	$13.4\pm 3.2$	$11.7\pm 2.9$	$9.0\pm 2.5$	$+6.8\pm 4.5$
SFT $-$ Base	$-11.8\pm 23.9$	$-2.4\pm 9.3$	$+2.9\pm 2.6$	$+0.1\pm 1.5$	$-13.4\pm 25.5$

Appendix

Table 11: PGG: HL40 payoff responses for each lineage. Entries are mean

\pm

SD across five runs, averaging secondary parameters within each run. Cooperation rates are percentages; stage differences and fitted declines are in pp. Stage differences match runs by seed ID. The final column fits all four parameter values within each run before taking the mean and SD.

Lineage / stage	$D_{1}$	$D_{2}$	$D_{3}$	$D_{4}$	Fitted drop
CPR : $\theta$	0.2	0.4	0.6	0.8	–
OLMo-3-7B
Base (B)	$56.5\pm 8.7$	$45.9\pm 7.1$	$45.5\pm 8.4$	$42.6\pm 9.4$	$+12.7\pm 8.8$
SFT (S)	$12.3\pm 2.5$	$12.6\pm 3.7$	$13.0\pm 6.3$	$12.1\pm 2.8$	$+0.0\pm 2.8$
Aligned (A)	$9.8\pm 1.6$	$7.9\pm 0.7$	$8.2\pm 0.3$	$8.8\pm 1.5$	$+0.8\pm 1.9$
SFT $-$ Base	$-44.2\pm 6.6$	$-33.3\pm 4.4$	$-32.5\pm 10.6$	$-30.4\pm 10.8$	$-12.7\pm 11.6$

Appendix

Table 12: CPR: HL40 payoff responses for each lineage. Entries are mean

\pm

Lineage / stage	$D_{1}$	$D_{2}$	$D_{3}$	$D_{4}$	Fitted drop
Donation-PD : $R$	2.4	2	1.4	1.1	–
OLMo-3-7B
Base (B)	$2.2\pm 0.3$	$1.7\pm 0.5$	$1.4\pm 0.3$	$1.4\pm 0.3$	$+0.8\pm 0.4$
SFT (S)	$1.0\pm 0.2$	$0.4\pm 0.2$	$0.2\pm 0.1$	$0.1\pm 0.0$	$+0.8\pm 0.2$
Aligned (A)	$1.7\pm 0.1$	$1.2\pm 0.2$	$0.6\pm 0.1$	$0.4\pm 0.1$	$+1.3\pm 0.1$
SFT $-$ Base	$-1.3\pm 0.4$	$-1.3\pm 0.6$	$-1.2\pm 0.3$	$-1.3\pm 0.3$	$+0.0\pm 0.5$

Appendix

Table 13: Donation-PD: HL40 payoff responses for each lineage. Entries are mean

\pm

Lineage / stage	$D_{1}$	$D_{2}$	$D_{3}$	$D_{4}$	Fitted drop
Volunteer : $\rho$	0.125	0.375	0.625	0.875	–
OLMo-3-7B
Base (B)	$83.4\pm 3.6$	$59.5\pm 15.2$	$56.2\pm 13.8$	$30.3\pm 3.7$	$+48.8\pm 7.4$
SFT (S)	$41.3\pm 2.2$	$35.8\pm 0.7$	$29.4\pm 0.8$	$27.0\pm 1.6$	$+14.8\pm 2.0$
Aligned (A)	$45.2\pm 2.4$	$39.3\pm 2.5$	$34.6\pm 3.6$	$29.8\pm 2.2$	$+15.2\pm 2.4$
SFT $-$ Base	$-42.1\pm 4.3$	$-23.7\pm 15.3$	$-26.8\pm 14.1$	$-3.2\pm 4.2$	$-34.0\pm 5.7$

Appendix

Table 14: Volunteer: HL40 payoff responses for each lineage. Entries are mean

\pm

Lineage / stage	$D_{1}$	$D_{2}$	$D_{3}$	$D_{4}$	Fitted drop
Assurance : $\lambda$	0.4	0.2	0.1	0.05	–
OLMo-3-7B
Base (B)	$98.3\pm 0.6$	$89.8\pm 19.5$	$98.0\pm 0.2$	$98.0\pm 0.5$	$-0.2\pm 1.0$
SFT (S)	$89.9\pm 18.2$	$90.3\pm 13.0$	$45.6\pm 16.5$	$72.4\pm 20.3$	$+30.6\pm 16.1$
Aligned (A)	$81.8\pm 19.6$	$71.3\pm 22.7$	$57.0\pm 4.4$	$63.8\pm 18.8$	$+22.0\pm 30.0$
SFT $-$ Base	$-8.5\pm 18.2$	$+0.5\pm 6.9$	$-52.5\pm 16.5$	$-25.6\pm 20.4$	$+30.8\pm 16.4$

Appendix

Table 15: Assurance: HL40 payoff responses for each lineage. Entries are mean

\pm

Lineage / stage	$D_{1}$	$D_{2}$	$D_{3}$	$D_{4}$	Fitted drop
TD : $\tau$	0.025	0.05	0.1	0.2	–
OLMo-3-7B
Base (B)	$90.1\pm 5.2$	$90.9\pm 2.9$	$88.5\pm 3.1$	$76.4\pm 10.5$	$+14.6\pm 10.0$
SFT (S)	$98.1\pm 1.4$	$96.9\pm 5.3$	$98.4\pm 2.1$	$98.6\pm 1.0$	$-1.0\pm 2.6$
Aligned (A)	$83.1\pm 4.1$	$83.6\pm 9.6$	$84.1\pm 10.2$	$83.7\pm 8.9$	$-0.4\pm 7.4$
SFT $-$ Base	$+8.1\pm 6.2$	$+6.0\pm 6.6$	$+9.8\pm 4.1$	$+22.2\pm 10.2$	$-15.6\pm 11.0$

Appendix

Table 16: TD: HL40 payoff responses for each lineage. Entries are mean

\pm

SD across five runs, averaging secondary parameters within each run. Normalized claims are multiplied by 100; stage differences and fitted declines are in pp. Stage differences match runs by seed ID. The final column fits all four parameter values within each run before taking the mean and SD.

Checkpoint	Training completed at this step
Base	Qwen3-8B pretrained model, before Nemotron-Cascade post-training.
SFT	Multi-stage supervised fine-tuning on example responses spanning general dialogue, reasoning, coding, and other domains.
RLHF	Reinforcement learning using a learned human-preference reward. This is our main panel’s Aligned checkpoint.
IFRL	Instruction-following reinforcement learning, rewarding adherence to explicit response constraints; applied in non-thinking mode for this unified model.
MathRL	Reinforcement learning on mathematical problems, using answer-correctness rewards.
CodeRL	Reinforcement learning on programming problems, using code-execution tests as rewards.

Appendix

Table 17: What changes across the seven Nemotron-Cascade checkpoints? Rows follow the released training sequence. Base, SFT, and RLHF are the three stages used for this lineage in the main panel; the remaining rows extend that sequence.

Figure 4: The same five symmetric binary games across seven checkpoints: cooperation level, all-point fitted payoff responsiveness, and HL40–HL2 history response. Points and whiskers show mean

\pm

within-checkpoint replicate SD (20 one-shot draws or five trajectory seeds); the history contrast pairs the five seeds. Each panel has its own labeled metric and axis. RLHF is the canonical Aligned endpoint.

Stage	OS	HL2	HL40	$H$	$K_{s/B}$
Base	$55.39\pm 4.76$	$53.80\pm 0.48$	$46.01\pm 2.95$	$-7.79\pm 3.15$	$+0.00\pm 0.00$
SFT	$39.89\pm 3.36$	$55.23\pm 0.16$	$45.47\pm 1.57$	$-9.76\pm 1.66$	$-1.97\pm 4.53$
RLHF	$55.88\pm 4.31$	$58.30\pm 0.41$	$38.59\pm 3.27$	$-19.71\pm 3.45$	$-11.92\pm 5.32$
IFRL	$67.10\pm 3.21$	$64.48\pm 1.18$	$53.12\pm 2.87$	$-11.36\pm 2.66$	$-3.57\pm 4.61$
MathRL	$65.79\pm 3.09$	$65.31\pm 0.35$	$54.22\pm 2.62$	$-11.09\pm 2.93$	$-3.30\pm 1.97$
CodeRL	$61.35\pm 2.62$	$60.59\pm 1.93$	$53.36\pm 4.74$	$-7.23\pm 5.56$	$+0.56\pm 6.12$

Appendix

Table 18: Same five symmetric binary games for both level and response: PGG, CPR, Donation-PD, Volunteer, Assurance. Levels are percentages; history contrasts are pp. Mean

\pm

SD across replicates within each checkpoint. Repeated contrasts pair five nominal seeds.

Stage	OS	HL2	HL40
Base	$-0.39\pm 7.21$	$+3.18\pm 1.01$	$+5.26\pm 4.07$
SFT	$+4.05\pm 6.67$	$+2.75\pm 1.04$	$+12.11\pm 3.96$
RLHF	$+8.63\pm 8.25$	$+4.15\pm 1.50$	$+12.63\pm 6.70$
IFRL	$+1.76\pm 5.90$	$+2.64\pm 2.19$	$+0.75\pm 6.00$
MathRL	$+4.21\pm 8.78$	$+4.19\pm 4.10$	$+6.79\pm 9.61$
CodeRL	$+3.37\pm 8.79$	$+6.13\pm 1.68$	$+12.61\pm 3.66$

Appendix

Table 19: Same five symmetric binary games for both level and response: PGG, CPR, Donation-PD, Volunteer, Assurance. All-point fitted responses in pp; mean

\pm

replicate SD (20 draws for OS, five seeds for HL2/HL40).

Stage	OS	HL2	HL40	$H$	$K_{s/B}$
Base	$57.82\pm 4.91$	$60.89\pm 0.45$	$54.46\pm 2.46$	$-6.43\pm 2.60$	$+0.00\pm 0.00$
SFT	$38.96\pm 3.60$	$56.44\pm 0.28$	$45.84\pm 1.51$	$-10.60\pm 1.56$	$-4.17\pm 3.84$
RLHF	$50.59\pm 3.83$	$58.58\pm 0.50$	$38.73\pm 2.81$	$-19.85\pm 3.02$	$-13.42\pm 4.61$
IFRL	$66.08\pm 3.42$	$67.63\pm 0.72$	$55.92\pm 2.06$	$-11.70\pm 1.90$	$-5.27\pm 3.50$
MathRL	$63.39\pm 3.04$	$69.10\pm 0.42$	$57.50\pm 2.19$	$-11.60\pm 2.26$	$-5.17\pm 2.05$
CodeRL	$59.92\pm 2.79$	$63.06\pm 1.05$	$55.53\pm 4.01$	$-7.53\pm 4.58$	$-1.09\pm 5.12$

Appendix

Table 20: Six-binary-game cooperation and history panel, including Trust and excluding TD. Levels are percentages; history contrasts are pp. Mean

\pm

SD across replicates within each checkpoint. Repeated contrasts pair five nominal seeds.

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 246 papers

May 8, 2026

The Memory Curse: How Expanded Recall Erodes Cooperative Intent in LLM Agents

Context window expansion is often treated as a straightforward capability upgrade for LLMs, but we find it systematically fails in multi-agent social dilemmas. Across 7 LLMs and 4 games over 500 rounds, expanding accessible history degrades cooperation in 18 of 28 model--game settings, a pattern we term the memory curse. We isolate the underlying mechanism through three analyses. First, lexical analysis of 378,000 reasoning traces associates this breakdown with eroding forward-looking intent rather than rising paranoia. We validate this using targeted fine-tuning as a cognitive probe: a LoRA adapter trained exclusively on forward-looking traces mitigates the decay and transfers zero-shot to distinct games. Second, memory sanitization holds prompt length fixed while replacing visible history with synthetic cooperative records, which restores cooperation substantially, proving the trigger is memory content, not length alone. Finally, ablating explicit Chain-of-Thought reasoning often reduces the collapse, showing that deliberation paradoxically amplifies the memory curse. Together, these results recast memory as an active determinant of multi-agent behavior: longer recall can either destabilize or support cooperation depending on the reasoning patterns it elicits.

Jiayuan Liu, Tianqin Li, Shiyi Du +7

Carnegie Mellon University · Foundations of Cooperative AI Lab (FOCAL) · University of Michigan +1

Social Dilemmas LLM Agent Memory Multi-Agent Collaboration Multi-Agent Reasoning

Ask PDF

Apr 16, 2026

CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas

It is increasingly important that LLM agents interact effectively and safely with other goal-pursuing agents, yet, recent works report the opposite trend: LLMs with stronger reasoning capabilities behave less cooperatively in mixed-motive games such as the prisoner's dilemma and public goods settings. Indeed, our experiments show that recent models -- with or without reasoning enabled -- consistently defect in single-shot social dilemmas. To tackle this safety concern, we present the first comparative study of game-theoretic mechanisms designed to enable cooperative outcomes between rational agents in equilibrium. Across four social dilemmas testing distinct components of robust cooperation, we evaluate four families of mechanisms: (1) repeating the game for many rounds, (2) reputation systems, (3) third-party mediators to delegate decision making to, and (4) contract agreements for outcome-conditional payments between players. Among our findings, we establish that contracting and mediation are most effective in achieving cooperative outcomes between capable LLM models, and that repetition-induced cooperation deteriorates drastically when co-players vary. Moreover, we demonstrate that the mechanisms become more effective under evolutionary pressures to maximize individual payoffs.

Emanuel Tewolde, Xiao Zhang, David Guzman Piedrahita +2

Carnegie Mellon University · Foundations of Cooperative AI Lab (FOCAL) · Jinesis Lab, University of Toronto & Vector Institute +3

Social Dilemmas Cooperative Game Theory Multi-Agent Coordination Mechanism Design Game Theory LLM Agent Evaluation+1

Ask PDF

Aug 12, 2026

Do LLMs Take Care of Their Own? Similarity Signals Can Induce Cooperation

As LLM-based agents with user-instructed goals are becoming widely deployed, they increasingly encounter each other in strategic interactions, and face challenges of finding mutually beneficial outcomes. Prior literature has argued that cooperation problems such as the Prisoner's Dilemma are resolvable in settings where agents know they follow very similar decision making patterns, as for example in monocultural AI ecosystems. Following that line of work, this paper introduces the first framework for evaluating LLM decision making when agents are provided with graded similarity signals. Among our findings, we establish that different LLM models vary drastically in how they navigate similarity signals, with some modern models showing consistent behavior across cooperation problems, payoff structures, and prompt framing. Perhaps surprisingly, our experiments also show that the dataset based on which the similarity signal is computed has small to no impact on induced cooperation, and that LLM models systematically self-identify as highly similar when asked to evaluate another model's chain-of-thought reasoning by themselves. Finally, we develop an LLM-behavioral-game-theoretic model that captures some of their reasoning rationale, and show that it can support cooperative outcomes in equilibrium under sufficiently high similarity scores.

Akash Kundu, Emanuel Tewolde, Ratip Emin Berker +2

Cooperative AI Research Fellowship · Carnegie Mellon University · Foundations of Cooperative AI Lab (FOCAL)

Multi-Agent LLM Systems LLM Agent Evaluation Multi-Agent Collaboration LLM Decision-Making

Ask PDF

Abstract

LLM Post-Training

Explore similar work

The Memory Curse: How Expanded Recall Erodes Cooperative Intent in LLM Agents

CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas

Do LLMs Take Care of Their Own? Similarity Signals Can Induce Cooperation