econ.GNMay 28, 2025

Learning to Charge More: How Q-Learning Firms Lock In Supracompetitive Prices

Authors: Cristian Chica, Yinglong Guo, Gilad Lerman

Organizations: School of Mathematics, University of Minnesota.

Abstract

There is growing experimental evidence that QQ-learning agents can learn to charge supracompetitive prices. We ask when such behavior persists, whether it emerges, and which strategies support it. In a repeated pricing game in which firms condition on the previous period's price profile, we give conditions on the learned action values under which supracompetitive prices become an absorbing state once experimentation ends. We then show that under softmax exploration these conditions are reached with positive and explicitly bounded probability, both from a random and from an all-zero initial QQ-table. In the latter case firms lock in supracompetitive prices because they never learn that undercutting pays; the same mechanism can lock in the competitive price, so the long-run outcome is selected by the exploration path. Finally, we give sufficient conditions on the learned table under which the induced strategy is naive collusion, a grim trigger strategy or an increasing-price strategy; grim trigger strategies are subgame perfect under a standard discounting condition, whereas naive collusion is generally not an equilibrium.

Explore similar work

Jul 15, 2026

Equilibrium stability as a driver of cooperation among Q-learners

Algorithmic collusion among pricing algorithms has raised concerns about sustained supra-competitive prices and their implications for social welfare. Existing work has largely focused on the probability that reinforcement-learning algorithms converge to cooperative strategies, typically under the assumption that exploration vanishes over time. Motivated by the observation that algorithms deployed in practice are likely to continue exploring in order to remain adaptive to changing environments, we study learning dynamics under constant exploration. In this setting, the relevant question is no longer whether an algorithm converges to a particular strategy profile, but rather what fraction of time the algorithms spend playing cooperative strategies. Even in the benchmark case of the repeated Prisoner's Dilemma with one-period memory, this yields high-dimensional stochastic learning dynamics, for which a complete analytic treatment is intractable. We show that cooperative strategies can be dominant in this time-averaged sense and derive a boundary predicting when such dominance arises, based on the expected dynamics of the Q-learning process. Extensive simulations show that this boundary is a strong predictor for non-defection-dominated behaviour under epsilon-greedy Q-learning.
Sep 28, 2026

Persistent Partners Raise Prices Among Learning Agents

When pricing agents meet repeatedly on a platform, the platform decides who faces whom. We ask whether that choice moves the prices the agents learn, and whether a rise comes with learned punishment. In a pre-registered randomised experiment in the Bertrand duopoly of Calvano et al., each agent's price is set by a tabular Q-learning module, not by the small language model attached to it, and we randomise whether each agent keeps its partner, sees its rival's prices and can send messages. Keeping the same partner raises the level of profits, averaged over training, by 0.27 of the gap between competitive and monopoly profit (95% CI 0.20 to 0.35, all twenty paired runs positive), our registered primary result, and the resting price by 0.17 of the Nash-to-monopoly range (post hoc). A plain tabular learner reproduces the effect in all 25 further blocks, and there one permanent partner raises the level more than about three do (+0.23 against +0.05, exploratory). Where rival prices are hidden, the price-setting module cannot see a cut, so cannot punish it, yet the resting price rises as much and the rise lasts to the end of training, while with visible rivals it shrinks with longer training (post hoc). Where the rival is visible, a static best responder accounts for a third to a half of what a forced-deviation probe reads as punishment, on the starts where the rival can see the cut, and net of it the registered test of learned punishment is inconclusive. A test that looks only for punishment would thus miss the rise where the rival is hidden, while a check for profitable deviations flags most of those prices (post hoc). In an exploratory extension, untrained Qwen2.5 7B and 14B models under one prompt show the effect when the rival's price is left out of the prompt and inconsistently when it is shown, the 7B result replicating on fresh blocks, while two other model families show none.
May 15, 2026

Misspecified Estimate-then-Optimize Leads to Supra-Competitive Prices

We study whether simple algorithmic pricing systems can systematically produce collusive-like prices in multi-firm markets. We consider firms that price using a myopic estimate-then-optimize rule: each repeatedly fits a demand model to its own price and sales history and sets the price that maximizes estimated profit. This demand model is misspecified, omitting competitors' prices. We analyze the dynamics of this rule when it is initialized by an exploration phase of independent random prices. We characterize when this pipeline converges to supra-competitive prices above the Nash equilibrium, via a fluid-limit ordinary differential equation analysis. We show that supra-competitive prices arise when firms initially explore within similar price ranges on the same side of the Nash price. Moreover, prices can be substantially above the Nash price; we show that prices can reach monopoly levels under symmetric exploration. Simulations calibrated to a real multifamily rental market confirm that supra-competitive outcomes arise robustly beyond our theoretical assumptions, including under finite horizons, heterogeneous products, and nonlinear logit demand.