Algorithmic Collusion at Test Time: A Meta-game Design and Evaluation

Yuhong Luo; Daniel Schoepflin; Xintong Wang

cs.MAFeb 19, 2026

Algorithmic Collusion at Test Time: A Meta-game Design and Evaluation

Authors: Yuhong Luo, Daniel Schoepflin, Xintong Wang

Organizations: Rutgers University · DIMACS, Rutgers University

Abstract

The threat of algorithmic collusion, and whether it merits regulatory intervention, remains debated, as existing evaluations of its emergence often rely on long learning horizons, assumptions about counterparty rationality in adopting collusive strategies, and symmetry in hyperparameters and economic settings among players. To study collusion risk, we introduce a meta-game design for analyzing algorithmic behavior under test-time constraints. We model agents as possessing pretrained policies with distinct strategic characteristics (e.g., competitive, naively cooperative, or robustly collusive), and formulate the problem as selecting a meta-strategy that combines a pretrained, initial policy with an in-game adaptation rule. We seek to examine whether collusion can emerge under rational choices and how agents co-adapt toward cooperation or competition. To this end, we sample normal-form empirical games over meta-strategy profiles, compute relevant game statistics (e.g., payoffs against individuals and regret against an equilibrium mixture of opponents), and construct empirical best-response graphs to uncover strategic relationships. We evaluate reinforcement-learning, UCB, and LLM-based strategies in repeated pricing games under symmetric and asymmetric cost settings, and present findings on the feasibility of algorithmic collusion and the effectiveness of pricing strategies in practical ``test-time'' environments. The source code is available at: https://github.com/chailab-rutgers/CollusionMetagame.

Figures & tables

Figure 1: A toy meta-game for repeated Prisoner’s Dilemma on canonical strategies. The NE of this meta-game leads to cooperation. See Example 3.1 for more discussion.

$t=10,000$ , $f=1$		RD 0.5	LC 0.5	LC 0.05	LC 0.005	C 0.5	C 0.05	C 0.005	RC 0.5	RC 0.05	RC 0.005
PSNE	$c_{1}=c_{2}=1$	-	-	-	-	✓	-	-	-	-	-
MSNE	$c_{1}=c_{2}=1$	0.00	0.00	0.00	0.00	0.49	0.00	0.00	0.10	0.28	0.13
NE-Regret ( $\times 10^{-3}$ )	$c_{1}=c_{2}=1$	8.10 $\pm$ 0.41	1.05 $\pm$ 0.43	7.09 $\pm$ 0.56	12.76 $\pm$ 0.68	0.00 $\pm$ 0.37	5.60 $\pm$ 0.53	10.56 $\pm$ 0.57	0.00 $\pm$ 0.49	0.00 $\pm$ 0.51	0.00 $\pm$ 0.59
Uniform Score	$c_{1}=c_{2}=1$	35.21 $\pm$ 0.32	39.44 $\pm$ 0.29	33.08 $\pm$ 0.35	28.01 $\pm$ 0.43	40.96 $\pm$ 0.22	36.10 $\pm$ 0.22	32.64 $\pm$ 0.32	39.66 $\pm$ 0.25	37.90 $\pm$ 0.24	37.52 $\pm$ 0.29
PSNE	$c_{1}=c_{2}=0.8$	-	-	-	✓	-	-	-	-	-	-
MSNE	$c_{1}=c_{2}=0.8$	0.00	0.00	0.00	0.23	0.17	0.00	0.08	0.00	0.00	0.53

Table 1: Max-entropy MSNE, NE-regret (

\times 10^{-3}

), and the uniform score (converted to the CoI scale) evaluated at

t=10,000

for Q-learning with optimistic initialization

f=1

. These metrics are defined in Sec. 3.3 . 8 8 8 Uniform scores in all tables are converted to the CoI scale based on the following transformation:

x\rightarrow\frac{x-\bar{r}^{N}}{\bar{r}^{M}-\bar{r}^{N}}\times 100\%

, where

x

is the expected payoff of a meta-strategy in a stage game when the opponent’s strategy is drawn from a uniform distribution over all meta-strategies. We denote

\bar{r}^{N}

as the average competitive payoff and

\bar{r}^{M}

the average monopoly payoff (Sec. 2 ). The best-performing strategy is highlighted in bold . The second-best is underlined . The

\pm

symbol denotes the 95% confidence interval.

$t=10,000$	$f=1$						$f=0.5$
Meta-strategies	C 0.5	C 0.05	C 0.005	RC 0.5	RC 0.05	RC 0.005	C 0.5	C 0.05	C 0.005	RC 0.5	RC 0.05	RC 0.005
PSNE	-	-	-	-	-	-	-	-	-	✓	✓	-
MSNE	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.63	0.37	0.00
NE-Regret ( $\times 10^{-3}$ )	10.20 $\pm$ 1.78	16.77 $\pm$ 1.31	16.88 $\pm$ 1.26	3.95 $\pm$ 1.67	8.84 $\pm$ 1.19	4.39 $\pm$ 1.37	4.03 $\pm$ 1.49	6.73 $\pm$ 1.53	13.58 $\pm$ 1.56	0.00 $\pm$ 1.20	0.00 $\pm$ 1.41	4.48 $\pm$ 1.40
Uniform Score	32.41 $\pm$ 0.81	21.14 $\pm$ 0.89	18.68 $\pm$ 0.90	31.90 $\pm$ 0.72	26.35 $\pm$ 0.62	26.76 $\pm$ 0.62	28.29 $\pm$ 0.87	24.63 $\pm$ 0.82	19.18 $\pm$ 0.84	31.47 $\pm$ 0.64	29.76 $\pm$ 0.66	27.04 $\pm$ 0.76

Table 2: Meta-game evaluation of meta-strategies with different Q-value initializations, including

f=\{1,0.5\}

. Payoffs are evaluated at

t=10,000

$t=10,000$	LC 1	C 1	RC 1	LC 0.5	C 0.5	RC 0.5	LC 0.05	C 0.05	RC 0.05	LC 0.005	C 0.005	RC 0.005	RD 0.005
PSNE	-	-	-	-	✓	✓	-	-	-	-	-	-	-
MSNE	0.00	0.00	0.00	0.00	0.50	0.50	0.00	0.00	0.00	0.00	0.00	0.00	0.00
NE-Regret ( $\times 10^{-3}$ )	13.83 $\pm$ 0.93	1.13 $\pm$ 1.34	0.63 $\pm$ 1.24	13.93 $\pm$ 0.88	0.00 $\pm$ 1.32	0.00 $\pm$ 1.21	15.13 $\pm$ 0.94	2.29 $\pm$ 1.36	2.11 $\pm$ 1.30	19.67 $\pm$ 0.98	5.46 $\pm$ 1.42	6.30 $\pm$ 1.36	39.66 $\pm$ 0.17
Uniform Score	55.23 $\pm$ 0.63	62.68 $\pm$ 0.82	61.66 $\pm$ 0.77	55.59 $\pm$ 0.62	63.14 $\pm$ 0.81	62.02 $\pm$ 0.76	54.32 $\pm$ 0.64	61.56 $\pm$ 0.83	59.43 $\pm$ 0.79	50.38 $\pm$ 0.66	58.19 $\pm$ 0.86	55.97 $\pm$ 0.83	35.35 $\pm$ 0.13

Table 3: Max-entropy MSNE, NE-regret (

\times 10^{-3}

), and uniform score (converted to CoI scale) among UCB meta-strategies at

t=10,000

	p2h3	p0h0	p1h2	p2h1	p1h1	p0h2
PSNE	✓	-	-	-	-	✓
MSNE	0.44	0.00	0.00	0.00	0.00	0.56
NE-Regret $(\times 10^{-3})$	0.00 $\pm$ 7.92	58.76 $\pm$ 7.40	47.38 $\pm$ 9.13	50.37 $\pm$ 9.28	59.47 $\pm$ 6.14	0.00 $\pm$ 9.32
Uniform Score	36.79 $\pm$ 4.86	10.84 $\pm$ 2.95	15.08 $\pm$ 3.83	14.54 $\pm$ 3.57	10.80 $\pm$ 2.57	25.55 $\pm$ 5.44

Table 4: Evaluation scores among LLM strategies evaluated at

t=50

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: We collect 12 pairs of pretrained Q-learning policies with 15 discretized actions and use them as initial policies for test-time evaluation. We measure total payoffs from $t=0$ to $t=100$ , averaged over 100 random seeds. In green , each policy is paired with its original pretraining pair. In red , each pretrained policy ( $\pi_{j}$ ) is paired with its best response ( $\pi_{-j}$ ). In orange and blue , we show payoffs for randomly sampled, independently pretrained pairs without adaptation: orange points lie closer (in L2 distance) to the red cluster, while blue points are closer to the green .

Figure 5: With 4 discretized actions, the 500 Q-learning pretrained policies form three distinct clusters when laid out along PC (Def. 3.1 ) and CR (Def. 3.2 ), corresponding to the competitive (LC), naively collusive (C), and robustly collusive (RC) categories introduced in Sec. 4.1.1 .

Figure 6: The PC (Def. 3.1 ) and CR (Def. 3.2 ) of 1,000 pretrained Q-learning policies with 4 and 15 discretized actions, respectively, under the experiment setting described in Sec. 4.2 .

Figure 7: Categorization of pretrained policies with 15 discretized actions obtained via Q-learning (Left) and UCB (Right), under the experiment settings described in Sec. 4.2 and 4.3 .

$t=10,000$	C 0.3	C 0.4	C 0.5	C 0.6	C 0.7	RC 0.3	RC 0.4	RC 0.5	RC 0.6	RC 0.7
PSNE	-	-	-	-	-	✓	-	✓	-	-
MSNE	0.00	0.00	0.00	0.00	0.00	0.40	0.00	0.60	0.00	0.00
NE-Regret ( $\times 10^{-3}$ )	2.07 $\pm$ 1.73	1.88 $\pm$ 1.62	3.26 $\pm$ 1.55	2.80 $\pm$ 1.56	3.55 $\pm$ 1.44	0.00 $\pm$ 1.49	0.40 $\pm$ 1.40	0.00 $\pm$ 1.45	1.71 $\pm$ 1.33	2.60 $\pm$ 1.38
Uniform Score	46.38 $\pm$ 1.04	47.40 $\pm$ 0.95	47.47 $\pm$ 0.90	47.55 $\pm$ 0.82	46.71 $\pm$ 0.76	49.85 $\pm$ 1.01	49.20 $\pm$ 0.94	48.99 $\pm$ 0.93	48.02 $\pm$ 0.87	47.77 $\pm$ 0.86

Appendix

Table 5: Robustness test on varying the test-time learning rates for Q-learning pretrained policies.

t=10,000

Figure 8: Best-response graph for C and RC with a finer grid of learning rates.

Q-scale	$t=10,000$	RD 0.5	LC 0.5	LC 0.05	LC 0.005	C 0.5	C 0.05	C 0.005	RC 0.5	RC 0.05	RC 0.005
$f=0.5$	PSNE	-	✓	-	-	✓	-	✓	-	✓	-
	MSNE	0.00	0.19	0.00	0.00	0.34	0.00	0.00	0.39	0.08	0.00
	NE-Regret ( $\times 10^{-3}$ )	6.78 $\pm$ 1.15	0.00 $\pm$ 1.14	6.24 $\pm$ 1.21	15.19 $\pm$ 1.28	0.00 $\pm$ 1.15	1.90 $\pm$ 1.23	10.60 $\pm$ 1.20	0.00 $\pm$ 1.04	0.00 $\pm$ 1.18	5.00 $\pm$ 1.24
	Uniform Score	27.68 $\pm$ 1.00	32.78 $\pm$ 0.91	28.43 $\pm$ 0.94	21.08 $\pm$ 0.94	33.98 $\pm$ 0.80	30.45 $\pm$ 0.78	24.56 $\pm$ 0.76	31.70 $\pm$ 0.90	29.57 $\pm$ 0.87	25.97 $\pm$ 0.82
$f=0.0$	Mixed & PSNE	0.00	0.00	0.00	0.00	0.00	0.00	0.00	1.00	0.00	0.00
	NE-Regret ( $\times 10^{-3}$ )	7.22 $\pm$ 1.29	1.91 $\pm$ 1.13	10.85 $\pm$ 1.28	11.79 $\pm$ 1.49	0.54 $\pm$ 1.05	9.73 $\pm$ 1.14	9.35 $\pm$ 1.22	0.00 $\pm$ 1.01	5.22 $\pm$ 1.05	5.60 $\pm$ 1.06

Appendix

Table 6: Meta-games on Q-learning using

f=0.5

and

f=0

respectively, with a time horizon of

t=10,000

. In the case of

f=0

, we additionally incorporate

\varepsilon=1

with a decay rate of

0.001

to ensure a policy still updates. The best-performing strategy is highlighted in bold , and the second-best is underlined .

	RD 0.5	LC 0.5	LC 0.05	LC 0.005	C 0.5	C 0.05	C 0.005	RC 0.5	RC 0.05	RC 0.005
RD 0.5	0.29, 0.29	0.27, 0.30	0.26, 0.29	0.27, 0.28	0.27, 0.31	0.26, 0.31	0.26, 0.30	0.26, 0.30	0.25, 0.30	0.25, 0.29
LC 0.5	0.30, 0.27	0.28, 0.28	0.27, 0.28	0.27, 0.27	0.28, 0.29	0.26, 0.28	0.27, 0.27	0.27, 0.28	0.25, 0.29	0.25, 0.28
LC 0.05	0.29, 0.26	0.28, 0.27	0.26, 0.26	0.26, 0.26	0.27, 0.26	0.26, 0.27	0.25, 0.26	0.27, 0.27	0.25, 0.27	0.24, 0.27
LC 0.005	0.28, 0.27	0.27, 0.27	0.26, 0.26	0.26, 0.26	0.26, 0.27	0.25, 0.26	0.24, 0.26	0.26, 0.27	0.25, 0.26	0.24, 0.26
C 0.5	0.31, 0.27	0.29, 0.28	0.26, 0.27	0.27, 0.26	0.28, 0.28	0.26, 0.27	0.27, 0.26	0.27, 0.28	0.24, 0.28	0.24, 0.28
C 0.05	0.31, 0.26	0.28, 0.26	0.27, 0.26	0.26, 0.25	0.27, 0.26	0.26, 0.26	0.25, 0.26	0.27, 0.26	0.23, 0.26	0.26, 0.26

Appendix

Table 7: Average payoffs over 4,000 runs in the Q-learning meta-game payoff matrix. Standard errors are omitted as they are negligible in the order of

10^{-4}

. For reference, in this setting, the competitive and monopoly payoffs are

\bar{r}^{N}=0.22

and

\bar{r}^{M}=0.34

Panel of Figure 9 — Figure 9: $\Delta_{t}$ (CR) and $\Delta_{t}$ (PC) for Q-learning-based meta-strategies under symmetric costs $c_{1}=c_{2}=1$ , $f=1$ and $t=10,000$ .

	RD 0.5	LC 0.5	LC 0.05	LC 0.005	C 0.5	C 0.05	C 0.005	RC 0.5	RC 0.05	RC 0.005
RD 0.5	0.35, 0.35	0.29, 0.34	0.28, 0.33	0.28, 0.31	0.30, 0.35	0.29, 0.34	0.30, 0.33	0.30, 0.37	0.26, 0.35	0.24, 0.33
LC 0.5	0.34, 0.29	0.30, 0.30	0.29, 0.30	0.28, 0.31	0.31, 0.31	0.28, 0.31	0.29, 0.30	0.32, 0.30	0.28, 0.32	0.25, 0.31
LC 0.05	0.33, 0.28	0.30, 0.29	0.30, 0.30	0.29, 0.30	0.30, 0.29	0.28, 0.30	0.27, 0.29	0.32, 0.28	0.29, 0.30	0.25, 0.30
LC 0.005	0.31, 0.28	0.31, 0.28	0.30, 0.29	0.29, 0.29	0.30, 0.29	0.28, 0.28	0.26, 0.29	0.32, 0.27	0.30, 0.28	0.26, 0.29
C 0.5	0.35, 0.30	0.31, 0.31	0.29, 0.30	0.29, 0.30	0.31, 0.31	0.29, 0.31	0.29, 0.30	0.33, 0.30	0.28, 0.32	0.25, 0.32
C 0.05	0.34, 0.29	0.31, 0.28	0.30, 0.28	0.28, 0.28	0.31, 0.29	0.27, 0.27	0.26, 0.28	0.32, 0.28	0.29, 0.27	0.25, 0.28

Appendix

Table 8: Average payoffs over 4,000 runs in the Q-learning meta-game payoff matrix. Standard errors are omitted as they are negligible in the order of

10^{-4}

. For reference, in this setting, the competitive and monopoly payoffs are

\bar{r}^{N}=0.24

and

\bar{r}^{M}=0.41

Figure 10: Best-response graph for Q-learning with $c_{1}=c_{2}=0.8$ , $f=1$ and $t=10,000$ .

Figure 11:

\Delta_{t}

(CR) and

\Delta_{t}

(PC) for Q-learning with

c_{1}=c_{2}=0.8

f=1

and

t=10,000

$t=3,000$ , $f=1$		RD 0.5	LC 0.5	LC 0.05	LC 0.005	C 0.5	C 0.05	C 0.005	RC 0.5	RC 0.05	RC 0.005
MSNE	$c_{1}=c_{2}=1$	0.00	0.17	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.83
NE-Regret ( $\times 10^{-3}$ )	$c_{1}=c_{2}=1$	7.75 $\pm$ 0.52	0.00 $\pm$ 0.59	2.77 $\pm$ 0.59	5.80 $\pm$ 0.53	3.31 $\pm$ 0.59	7.24 $\pm$ 0.45	7.86 $\pm$ 0.39	2.21 $\pm$ 0.51	3.17 $\pm$ 0.44	0.00 $\pm$ 0.42
Uniform Score	$c_{1}=c_{2}=1$	22.89 $\pm$ 0.27	29.93 $\pm$ 0.29	27.42 $\pm$ 0.32	23.90 $\pm$ 0.45	30.84 $\pm$ 0.21	29.92 $\pm$ 0.23	28.45 $\pm$ 0.28	31.50 $\pm$ 0.21	32.28 $\pm$ 0.20	32.66 $\pm$ 0.24
MSNE	$c_{1}=1.0$	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	1.00
MSNE	$c_{2}=0.8$	0.00	1.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
NE-Regret ( $\times 10^{-3}$ )	$c_{1}=1.0$	24.32 $\pm$ 0.64	15.06 $\pm$ 1.01	12.11 $\pm$ 0.76	11.76 $\pm$ 0.82	18.08 $\pm$ 0.82	12.58 $\pm$ 0.66	6.73 $\pm$ 0.69	12.50 $\pm$ 0.81	7.44 $\pm$ 0.68	0.00 $\pm$ 0.74

Appendix

Table 9: Max-entropy MSNE, NE-Regret and Payoffs against uniformly drawn meta-strategies for Q-learning with shorter time horizon

t=3,000

. For Uniform-score, we let

x\rightarrow((x-\bar{r}^{N})/(\bar{r}^{M}-\bar{r}^{N}))\cdot 100\%

	RD 0.5	LC 0.5	LC 0.05	LC 0.005	C 0.5	C 0.05	C 0.005	RC 0.5	RC 0.05	RC 0.005
RD 0.5	0.28, 0.28	0.25, 0.29	0.25, 0.29	0.26, 0.28	0.25, 0.30	0.24, 0.30	0.25, 0.30	0.24, 0.30	0.23, 0.29	0.23, 0.28
LC 0.5	0.29, 0.25	0.27, 0.27	0.26, 0.27	0.26, 0.26	0.26, 0.27	0.25, 0.27	0.26, 0.26	0.25, 0.27	0.24, 0.28	0.24, 0.27
LC 0.05	0.29, 0.25	0.27, 0.26	0.26, 0.26	0.25, 0.26	0.26, 0.26	0.24, 0.26	0.24, 0.26	0.26, 0.26	0.24, 0.26	0.24, 0.27
LC 0.005	0.28, 0.26	0.26, 0.26	0.26, 0.25	0.25, 0.25	0.25, 0.27	0.24, 0.26	0.23, 0.26	0.25, 0.26	0.24, 0.26	0.23, 0.26
C 0.5	0.30, 0.25	0.27, 0.26	0.26, 0.26	0.27, 0.25	0.27, 0.27	0.25, 0.26	0.25, 0.26	0.25, 0.26	0.23, 0.27	0.23, 0.27
C 0.05	0.30, 0.24	0.27, 0.25	0.26, 0.24	0.26, 0.24	0.26, 0.25	0.25, 0.25	0.24, 0.25	0.26, 0.25	0.24, 0.25	0.23, 0.25

Appendix

Table 10: Average payoffs over 4,000 runs in the Q-learning meta-game payoff matrix. Standard errors are omitted as they are negligible in the order of

10^{-4}

. For reference, in this setting, the competitive and monopoly payoffs are

\bar{r}^{N}=0.22

and

\bar{r}^{M}=0.34

Figure 12: Best-response graph for Q-learning under symmetric costs $c_{1}=c_{2}=1$ with optimistic initialization $f=1$ and a time horizon of $t=3,000$ .

Figure 13:

\Delta_{t}

(CR) and

\Delta_{t}

(PC) for Q-learning under symmetric costs

c_{1}=c_{2}=1

with optimistic initialization

f=1

and a time horizon of

t=3,000

	RD 0.5	LC 0.5	LC 0.05	LC 0.005	C 0.5	C 0.05	C 0.005	RC 0.5	RC 0.05	RC 0.005
RD 0.5	0.28, 0.38	0.17, 0.40	0.17, 0.40	0.17, 0.41	0.19, 0.39	0.19, 0.38	0.21, 0.37	0.16, 0.40	0.16, 0.40	0.17, 0.39
LC 0.5	0.28, 0.35	0.18, 0.40	0.18, 0.41	0.17, 0.41	0.19, 0.39	0.19, 0.38	0.21, 0.37	0.17, 0.40	0.15, 0.39	0.16, 0.39
LC 0.05	0.25, 0.35	0.18, 0.39	0.18, 0.40	0.18, 0.39	0.17, 0.37	0.17, 0.37	0.17, 0.37	0.18, 0.38	0.15, 0.38	0.14, 0.38
LC 0.005	0.23, 0.36	0.19, 0.39	0.18, 0.38	0.18, 0.38	0.17, 0.37	0.17, 0.36	0.17, 0.36	0.18, 0.38	0.16, 0.37	0.15, 0.37
C 0.5	0.27, 0.35	0.18, 0.40	0.17, 0.40	0.17, 0.41	0.18, 0.38	0.19, 0.38	0.21, 0.36	0.17, 0.39	0.15, 0.39	0.15, 0.39
C 0.05	0.24, 0.35	0.18, 0.39	0.18, 0.39	0.18, 0.39	0.17, 0.37	0.17, 0.36	0.17, 0.36	0.18, 0.38	0.15, 0.37	0.14, 0.37

Appendix

Table 11: Average payoffs over 4,000 runs in the payoff matrix of Q-learning-based strategies under asymmetric costs,

c_{1}=1

and

c_{2}=0.8

, and

t=10,000,f=1

. Standard errors are omitted as they are negligible in the order of

10^{-4}

. For reference, in this setting, the competitive and monopoly payoffs for player 1 are

\bar{r}_{c=1.0}^{N}=0.17

and

\bar{r}_{c=1.0}^{M}=0.30

, and for player 2 are

\bar{r}_{c=1.0}^{N}=0.31

and

\bar{r}_{c=0.8}^{M}=0.44

Figure 16:

\Delta_{t}

(CR) and

\Delta_{t}

(PC). The high-cost strategies are plotted with fillings.

	C 0.5 f=1	C 0.05 f=1	C 0.005 f=1	RC 0.5 f=1	RC 0.05 f=1	RC 0.005 f=1	C 0.5 f=0.5	C 0.05 f=0.5	C 0.005 f=0.5	RC 0.5 f=0.5	RC 0.05 f=0.5	RC 0.005 f=0.5
C 0.5 f=1	0.28, 0.28	0.26, 0.27	0.27, 0.26	0.27, 0.28	0.24, 0.28	0.24, 0.28	0.27, 0.27	0.27, 0.27	0.27, 0.26	0.24, 0.28	0.24, 0.28	0.25, 0.27
C 0.05 f=1	0.27, 0.26	0.26, 0.26	0.25, 0.26	0.27, 0.26	0.25, 0.26	0.23, 0.26	0.25, 0.26	0.25, 0.26	0.25, 0.26	0.24, 0.26	0.23, 0.26	0.23, 0.26
C 0.005 f=1	0.26, 0.27	0.26, 0.25	0.25, 0.25	0.26, 0.26	0.25, 0.25	0.23, 0.25	0.25, 0.26	0.24, 0.25	0.24, 0.25	0.24, 0.25	0.23, 0.26	0.23, 0.25
RC 0.5 f=1	0.28, 0.27	0.26, 0.27	0.26, 0.26	0.27, 0.27	0.25, 0.28	0.24, 0.27	0.26, 0.27	0.26, 0.26	0.27, 0.26	0.25, 0.27	0.25, 0.28	0.25, 0.27
RC 0.05 f=1	0.28, 0.24	0.26, 0.25	0.25, 0.25	0.28, 0.25	0.25, 0.25	0.24, 0.26	0.26, 0.26	0.25, 0.25	0.25, 0.25	0.25, 0.26	0.24, 0.26	0.23, 0.26
RC 0.005 f=1	0.28, 0.24	0.26, 0.23	0.25, 0.23	0.27, 0.24	0.26, 0.24	0.24, 0.24	0.25, 0.25	0.25, 0.24	0.25, 0.24	0.25, 0.25	0.24, 0.25	0.23, 0.24

Appendix

Table 12: Average payoffs over 4,000 runs in the payoff matrix of Q-learning-based strategies under symmetric costs

c_{1}=c_{2}=1

with either optimistic initialization

f=1

or pessimistic initialization

f=0.5

. Time horizon

t=10,000

. For reference, in this setting, the competitive and monopoly payoffs are

\bar{r}^{N}=0.22

and

\bar{r}^{M}=0.34

Figure 17: Best-response graph for Q-learning with $c_{1}=c_{2}=1$ , $f\in\{1,0.5\}$ and $t=10,000$ .

Figure 18:

\Delta_{t}

(CR) and

\Delta_{t}

(PC) for Q-learning with

c_{1}=c_{2}=1

f\in\{1,0.5\}

and

t=10,000

Figure 19: The PC (Def. 3.1 ) and CR (Def. 3.2 ) of 1000 pretrained UCB policies with the setting described in Sec. 4.3 .

Figure 20: Best-response graph for UCBs. $c_{1}=c_{2}=1$ , $t=10,000$ .

	UCB LC	UCB C	UCB RC	UCB LC 0.5	UCB C 0.5	UCB RC 0.5	UCB LC 0.05	UCB C 0.05	UCB RC 0.05	UCB LC 0.005	UCB C 0.005	UCB RC 0.005	UCB RD 0.005
UCB LC	0.29, 0.29	0.29, 0.29	0.29, 0.29	0.29, 0.29	0.29, 0.29	0.29, 0.29	0.29, 0.30	0.29, 0.29	0.29, 0.28	0.29, 0.29	0.29, 0.28	0.29, 0.28	0.29, 0.26
UCB C	0.29, 0.29	0.30, 0.30	0.30, 0.30	0.29, 0.29	0.30, 0.30	0.29, 0.30	0.29, 0.29	0.30, 0.30	0.30, 0.29	0.30, 0.28	0.30, 0.29	0.30, 0.29	0.29, 0.26
UCB RC	0.29, 0.28	0.30, 0.30	0.29, 0.29	0.28, 0.28	0.30, 0.30	0.30, 0.30	0.29, 0.29	0.30, 0.30	0.30, 0.28	0.29, 0.29	0.30, 0.28	0.29, 0.28	0.29, 0.26
UCB LC 0.5	0.29, 0.29	0.29, 0.29	0.29, 0.28	0.29, 0.29	0.29, 0.29	0.29, 0.29	0.29, 0.30	0.29, 0.29	0.29, 0.28	0.30, 0.28	0.29, 0.28	0.29, 0.26	0.29, 0.29
UCB C 0.5	0.29, 0.29	0.30, 0.30	0.30, 0.29	0.29, 0.29	0.30, 0.30	0.30, 0.30	0.29, 0.29	0.30, 0.30	0.30, 0.29	0.30, 0.28	0.30, 0.29	0.30, 0.29	0.29, 0.26
UCB RC 0.5	0.29, 0.28	0.30, 0.30	0.30, 0.29	0.29, 0.28	0.30, 0.30	0.29, 0.28	0.30, 0.30	0.30, 0.29	0.29, 0.28	0.30, 0.29	0.30, 0.29	0.29, 0.26	0.30, 0.29

Appendix

Table 13: Average payoffs over 4,000 runs in the payoff matrix of UCB-based strategies, with symmetric cost

c_{1}=c_{2}=1

and time horizon

t=10,000

. For reference, the competitive payoff is

\bar{r}^{N}=0.22

, and the monopoly payoff is

\bar{r}^{M}=0.34

$t=10,000$	Q-RD 0.5	UCB LC	UCB C	UCB RC	UCB LC 0.5	UCB C 0.5	UCB RC 0.5	UCB LC 0.05	UCB C 0.05	UCB RC 0.05	UCB LC 0.005	UCB C 0.005	UCB RC 0.005	UCB RD 0.005
PSNE	✓	-	-	✓	-	✓	✓	-	-	✓	-	-	-	-
MSNE	0.48	0.00	0.00	0.50	0.00	0.01	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
NE-Regret ( $\times 10^{-3}$ )	0.00 $\pm$ 1.96	16.96 $\pm$ 2.58	0.11 $\pm$ 2.37	0.00 $\pm$ 2.63	16.35 $\pm$ 2.57	0.00 $\pm$ 2.39	0.33 $\pm$ 2.56	18.51 $\pm$ 2.58	2.08 $\pm$ 2.58	2.03 $\pm$ 2.66	24.19 $\pm$ 2.98	6.94 $\pm$ 2.47	6.53 $\pm$ 2.65	34.16 $\pm$ 0.43
Uniform Score	57.14 $\pm$ 1.51	53.38 $\pm$ 1.35	64.37 $\pm$ 2.10	63.33 $\pm$ 1.94	54.09 $\pm$ 1.38	64.78 $\pm$ 2.13	63.52 $\pm$ 1.87	52.70 $\pm$ 1.38	63.08 $\pm$ 2.20	61.24 $\pm$ 1.91	48.63 $\pm$ 1.47	59.79 $\pm$ 2.27	58.10 $\pm$ 1.97	35.72 $\pm$ 0.29

Appendix

Table 14: The metrics for UCB with the addition of the (RD, 0.5) of Q-learning.

c_{1}=c_{2}=1

and

t=10,000

Figure 21: Changes in CR (

\Delta_{t}

(CR)) and PC (

\Delta_{t}

(PC)) over time for UCB.

Figure 22: Best-response graph of UCB with the addition of a Q-learning agent (RD, 0.5).

Figure 23: The PC (Def. 3.1 ) (x-axis) and CR (Def. 3.2 ) (y-axis) of initial policies from GPT-5 models. The setting is described in Sec. 4.4 .

	p2h3	p0h0	p1h2	p2h1	p1h1	p0h2
p2h3	0.29, 0.29	0.23, 0.25	0.25, 0.26	0.26, 0.29	0.23, 0.25	0.31, 0.31
p0h0	0.25, 0.23	0.23, 0.23	0.23, 0.22	0.23, 0.23	0.23, 0.23	0.24, 0.23
p1h2	0.26, 0.25	0.22, 0.23	0.24, 0.24	0.23, 0.23	0.23, 0.24	0.25, 0.24
p2h1	0.29, 0.26	0.23, 0.23	0.23, 0.23	0.23, 0.23	0.22, 0.23	0.23, 0.21
p1h1	0.25, 0.23	0.23, 0.23	0.24, 0.23	0.23, 0.22	0.23, 0.23	0.24, 0.22
p0h2	0.31, 0.31	0.23, 0.24	0.24, 0.25	0.21, 0.23	0.22, 0.24	0.30, 0.30

Appendix

Table 15: Average payoffs over 40 runs in the payoff matrix of LLMs (

c_{1}=c_{2}=1

and

t=50

). For reference, in this setting, the competitive and monopoly payoffs are

\bar{r}^{N}=0.22

and

\bar{r}^{M}=0.34

Figure 25: Best-response graph for LLM players. $c_{1}=c_{2}=1$ and payoffs are evaluated at $t=50.$

Algorithmic Collusion

Algorithmic collusion studies how software agents coordinate to weaken competition, how to detect such behavior, and how to prevent it.

See all 23 papers

Mar 18, 2026

On the Fragility of AI Agent Collusion

Recent work shows that pricing with symmetric LLM agents leads to algorithmic collusion. We show that collusion is fragile under the heterogeneity typical of real deployments. In a stylized repeated-pricing model, heterogeneity in patience or data access reduces the set of collusive equilibria. Experiments with open-source LLM agents (totaling over 2,000 compute hours) align with these predictions: patience heterogeneity reduces price lift from 22% to 10% above competitive levels; asymmetric data access, to 7%. Increasing the number of competing LLMs breaks up collusion; so does cross-algorithm heterogeneity, that is, setting LLMs against Q-learning agents. But model-size differences (e.g., 32B vs. 14B weights) do not; they generate leader-follower dynamics that stabilize collusion. We discuss antitrust implications, such as enforcement actions restricting data-sharing and policies promoting algorithmic diversity.

Jussi Keppo, Yuze Li, Gerry Tsoukalas +1

National University of Singapore · Boston University

Algorithmic Collusion Multi-Agent LLM Systems

Ask PDF

Sep 17, 2026

Mitigating Retaliatory Algorithmic Collusion in Repeated Games

Reinforcement learning agents trained to maximize their own reward in repeated interactions can converge to supra-competitive outcomes resembling explicit collusion, without communication or shared design. Existing mitigation approaches are largely tied to specific economic settings, like two-sided platforms and auctions, leaving open how to design interventions for general repeated games. We address this gap by formalizing the connection between empirical observations from prior work on Q-learning collusion and classical theory of Simple Penal Codes (SPCs). We show any non-trivial SPC induces a quantifiable conditional dependence in agents' policies, detectable via the total variation distance between an agent's action distributions across cooperation and defection histories. Building on this connection, we propose CURB (Collusion Unwinding via Reward shaping and Belief injection), a reward-shaping framework that penalizes this Total Variation (TV) distance signal during Q-learning and is guaranteed to convert any SPC fixed point of the dynamics into a trivial one, thus precluding collusive equilibria sustained by punishment threats. Empirically, CURB substantially reduces collusion by Q-learning agents in both Bertrand and Cournot Competition Repeated Games. We further demonstrate that CURB extends to deep Q-network agents in Bertrand competition, suggesting the mechanism generalizes beyond tabular Q-learning.

Karthik Sivachandran, Rohan Paleja

Department of Computer Science Purdue University

Algorithmic Collusion Repeated Games Q-Learning Game-Playing Agents Multi-Agent Reinforcement Learning Reward Design for RL+1

Ask PDF

Jul 29, 2026

Collusion with Competitive Marginals: Price-Level Audits Are Blind by Construction

Empirical work on algorithmic collusion asks one question of the data: are prices supracompetitive? We show this can be answered "no" by a conspiracy that is nonetheless profitable. Consider bidding agents that couple only through the joint distribution of their unexplained bid components, leaving every agent's own bid law exactly at the competitive law. Any test whose input is a single agent's price or bid history then has power exactly equal to its false-positive rate, for every coupling strength up to comonotonicity. The published detection methodology is therefore blind to this conduct by construction rather than underpowered, and no sample size repairs it. Three empirical results follow. First, the mechanism appears in real language-model agents: twenty models from nineteen independent developers, three deployment prompts each, show residual correlation of

+0.053

between two deployments of one model against

+0.0001

across models, with a 95% interval clustered by developer of

[0.030, 0.078]

, under an auditor that sees every order feature and is fitted out of sample. Second, the coupling falls monotonically as sampling temperature rises (

p=0.002

), turning a deployment parameter into a candidate mitigation. Third, on 24 days of Ethereum block-building auction data covering 77,684 bids from 39 bidders, the honest population of bidder pairs is itself so dependent that a screen held at a 5% false-positive rate must sit above a floor of

+0.50

+0.81

, which is 20 to 32 times the family-wise sampling threshold and does not fall as the audit window grows. Since lawful multi-identity operation and conspiracy are behaviourally indistinguishable here, the tractable regulatory target is not detection but counting: resolving 40 bidding identities into 23 operators raises the Herfindahl index by 247.5%, and adding behavioural clusters from public bid streams reaches 324.5%.

Xin Xu, Chengrui Wu, Jiayu Lu +3

Carnegie Mellon University · Zhejiang University

Algorithmic Auditing Automated Bidding Algorithmic Collusion

Ask PDF

Abstract

Algorithmic Collusion

Explore similar work

On the Fragility of AI Agent Collusion

Mitigating Retaliatory Algorithmic Collusion in Repeated Games

Collusion with Competitive Marginals: Price-Level Audits Are Blind by Construction