Beyond Supra-Competitive Outcomes: Collusive Behaviour in Deep Reinforcement Learning for Optimal Execution Games

Christos Spyridon Koulouris; Carlo Campajola

q-fin.TRSep 30, 2026

Beyond Supra-Competitive Outcomes: Collusive Behaviour in Deep Reinforcement Learning for Optimal Execution Games

Authors: Christos Spyridon Koulouris, Carlo Campajola

Organizations: Institute of Finance and Technology University College London Gower Street, London WC1E 6BT, United Kingdom · UZH Blockchain Center Andreasstrasse 15, 8050 Zürich, Switzerland

Abstract

In this paper, we extend earlier findings of supra-competitive outcomes in optimal-execution games by identifying a learned punitive mechanism that deters deviations and provides behavioural evidence of collusion. We investigate this mechanism in a two-player, finite-horizon Almgren-Chriss liquidation game. Independent proximal policy optimisation agents with access to within-episode price and action histories achieve costs below the Nash benchmark. We identify a profitable deviation by training against the mean learned liquidation schedule, then impose its first trade on one of the original agents. The opponent responds by accelerating liquidation. This response more than offsets the deviator's gain in every run and both player roles, while leaving the punisher's average payoff materially unchanged relative to not punishing under the same deviation. The punisher imposes greater losses on the deviator while preserving its own average payoff, despite the availability of more profitable, less punitive liquidation plans. Matching deviations and subsequent additional selling rise and later decline during training, while final policies retain an effective punitive response. We formalise two checks: whether punishment outweighs the gain from deviating, and whether the change in trading behaviour is large enough to account for the loss imposed. Both checks hold for the tested deviation. Together, these findings provide behavioural and economic evidence supporting a collusive interpretation of the learned supra-competitive outcomes.

Figures & tables

Figure 1. Testing implementation shortfall for the two players. Each circle averages 500 test episodes from one trained pair; stars mark the closed-loop Nash and joint TWAP benchmarks. Lower values on both axes indicate a joint improvement in execution costs. Ten run centroids lie below the Nash benchmark in both players' implementation shortfall, but above the TWAP benchmark.

Figure 2. Mean testing inventory paths (left) and trade quantities (right), compared with closed-loop Nash. The shaded region is the full pointwise range across both players, ten trained runs and 500 test episodes per run. It measures realised trajectory dispersion. Mean inventory and trading paths with a minimum-to-maximum envelope covering every deterministic testing trajectory, compared with the closed-loop Nash schedule. Learned liquidation is slower initially, and the envelope is narrow.

Figure 3. Punisher’s trade under the learned response to the deviation minus its trade along the learned supra-competitive path without the deviation. The line averages these differences over both role assignments, 500 paired test episodes and ten trained runs; shading spans the full range of run means. Grey and hatched regions mark the deviation and compulsory final liquidation. The punisher sells more in timesteps two to five than in the unforced episode, then sells less at later timesteps. Its first trade is unchanged.

Figure 4. Execution-cost differences given the same forced deviation: learned punishment minus no punishment (orange), and sampled alternative minus no punishment (green).Positive values indicate a loss; negative values indicate a gain. The learned response has a near-zero cost effect on the punisher and a positive cost effect on the deviator. The alternative schedules lower both players' costs relative to learned punishment, while the deviator's cost remains above the replay control.

Figure 5. Occurrence of the tested first-step deviation during training. Orange counts matching episodes; blue counts the subset followed by additional selling from the other player. Counts pool a 200-episode trailing window from each of ten runs. Faint curves show rolling counts; bold curves are centred averages over 2,000 successive window endpoints. Matching deviations and the subset followed by additional selling rise during training, peak near episode eighteen thousand, and then decline.

Figure 6. Deviator gain without punishment, $G$ (blue), and loss from punishment, $H$ (orange), in $\mathrm{IS}/N$ . Connected points pair values from the same trained run, averaging both player identities and all test episodes. Boxes summarise the ten run means. The punishment loss exceeds the gain in every run. For every trained run, the orange loss point lies to the right of its connected blue gain point.

Figure 7. Measured exposure-weighted TV and its required boundary from ( 18 ). Connected points compare the two quantities within each trained run; boxes summarise their distributions and black diamonds show pooled values. The measured change exceeds the boundary in every run. The measured timing change lies above its required boundary in all ten trained runs and in the pooled comparison.

Explore similar work

May 19, 2026q-fin.CP

Memory-Induced Supra-Competitive Outcomes Between Deep Reinforcement Learning Agents in Optimal Trade Execution

In this paper, we investigate whether deep reinforcement-learning agents interacting in a shared optimal-execution environment can sustain supra-competitive outcomes, in the sense of achieving lower implementation shortfalls than the relevant game-theoretical competitive benchmark. We study a two-agent Almgren-Chriss liquidation game and examine how learned behavior depends on intra-episode environment feedback, the ability to interpret the mid-price and the agent's knoledge of the past. We first use ex-ante schedule-learning agents to remove intra-episode feedback and isolate what can arise when agents commit to complete liquidation trajectories before execution begins. We then allow agents to condition on the evolving state using a variety of DDQN architectures. We find that, when agents are given access to intra-episode history, especially recent prices and own past actions, supra-competitive outcomes become substantially more frequent and more persistent. These findings indicate that supra-competitive behavior in this execution game is driven not by multi-agent learning or by current price observation alone, but by feedback, memory, and state-contingent interaction along the realized execution path.

Christos Spyridon Koulouris, Carlo Campajola

Institute of Finance and Technology, University College London, Gower Street WC1E 6BT London, United Kingdom · UZH Blockchain Center, Andreasstrasse 15, 8050 Zürich, Switzerland

Multi-Agent Reinforcement Learning Single Trajectory

Ask PDF

Sep 17, 2026cs.LG

Mitigating Retaliatory Algorithmic Collusion in Repeated Games

Reinforcement learning agents trained to maximize their own reward in repeated interactions can converge to supra-competitive outcomes resembling explicit collusion, without communication or shared design. Existing mitigation approaches are largely tied to specific economic settings, like two-sided platforms and auctions, leaving open how to design interventions for general repeated games. We address this gap by formalizing the connection between empirical observations from prior work on Q-learning collusion and classical theory of Simple Penal Codes (SPCs). We show any non-trivial SPC induces a quantifiable conditional dependence in agents' policies, detectable via the total variation distance between an agent's action distributions across cooperation and defection histories. Building on this connection, we propose CURB (Collusion Unwinding via Reward shaping and Belief injection), a reward-shaping framework that penalizes this Total Variation (TV) distance signal during Q-learning and is guaranteed to convert any SPC fixed point of the dynamics into a trivial one, thus precluding collusive equilibria sustained by punishment threats. Empirically, CURB substantially reduces collusion by Q-learning agents in both Bertrand and Cournot Competition Repeated Games. We further demonstrate that CURB extends to deep Q-network agents in Bertrand competition, suggesting the mechanism generalizes beyond tabular Q-learning.

Karthik Sivachandran, Rohan Paleja

Department of Computer Science Purdue University

Collusion Q-Learning Equilibrium

Ask PDF

Jun 3, 2026cs.MA

Failure Modes of Deep Multi-Agent RL in Asynchronous Pricing: Reproducible Triggers, Trace Diagnostics, and a Partial Fix

We study two reproducible failure modes of deep multi-agent reinforcement learning in continuous-time pricing markets: (i) tacit cartel formation between competing DDPG agents, and (ii) actor--critic instability at high event rates. We instantiate both inside a single CT-MARL benchmark (Poisson-clocked price updates, observation latency

δ

, interior-optimum logit demand), show that synchronous DDPG agents reliably trigger Failure Mode 1 with collusion index

Δ= 0.69 \pm 0.11

, and quantify a partial microstructure fix: asynchrony alone cuts collusion by 48% and adding latency drives it to a minimum of

Δ= 0.28

. The fix has clearly documented costs: it is partial (

Δ

remains supra-Bertrand), it is non-monotone in

δ

, and it does not survive Failure Mode 2, which emerges as DDPG critic divergence at

λ= 5

and corrupts the phase-diagram cell at

(λ{=}5, δ{=}1)

. We accompany the scalar collusion index with trajectory-level trace diagnostics that expose the within-episode signalling collapse and the post-shock non-recovery.

Shree Murthy, Rohan Pandey

DigitalOcean, USA

Multi-Agent Reinforcement Learning Dynamic Pricing Critic Learning Coalition Formation Mathematical Finance Reproducibility

Ask PDF