Applying Reinforcement Learning (RL) to economic modeling reveals a fundamental tension: standard RL agents do not behave as competitive price-takers. While macroeconomic models typically assume agents take aggregate conditions as given, a naive single-agent RL algorithm learns to internalize its impact on the environment, effectively exerting market power. We illustrate this divergence in a search-and-matching model with concave production, showing that a standard RL agent converges to a monopsonistic allocation rather than the competitive equilibrium. Furthermore, we show that RL's approach to time discounting distorts standard intertemporal economic trade-offs. To resolve these issues, we propose a Calibrated Mean-Field Reinforcement Learning framework that restricts the agent to optimize against an exogenous aggregate state to restore price-taking behavior, and recalibrates the RL reward structure to reflect true economic opportunity costs. By iteratively updating the aggregate state, our algorithm converges to a stationary fixed point that recovers the competitive equilibrium. This approach provides a method for embedding learning algorithms into dynamic general equilibrium models.
Figures & tables
Symbol
Description
Value
A
Productivity
1
a
Matching efficiency
0.471
α
Parameter in production function
0.667
λ
Separation rate
0.0144
η
Worker’s bargaining power
0.6
c
Vacancy cost
0.268
Table 1: Default experimental parameters.
Figure 1: Market tightness θ : theoretical benchmark vs. RL outcome. The left panel illustrates the variation of reward during the training process, where the solid line represents the reward for the final step, the dotted line for the accumulated reward, and a dashed horizontal line for the theoretical reward value 0.168 of the steady-state derived from economic models. The stabilization of reward indicates that the agent has nearly converged to the optimal policy. The right panel depicts the changes in θ over the course of training; once the agent’s strategy has converged, the market tightness fluctuates around 0.1 , which is significantly lower than the theoretical value of 0.78 derived from economic models. The shaded regions indicate the standard deviations from five independent runs.
Figure 2: Comparison between theoretical equilibrium and fully corrected RL simulation. The figure depicts the changes in θ over iterations; once it has converged, the market tightness fluctuates around the theoretical value of 0.78 derived from economic models. The shaded regions indicate the standard deviations from five independent runs.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
l
u
q
w
v
θ
0.967
0.033
0.547
0.831
0.025
0.78
Appendix
Table 2: Theoretical steady state.
Figure 3: Market tightness θ across different simulation settings. The left panel shows the situation where there is only structural correction, while the right panel shows the situation where there is only parametric correction. The shaded regions indicate the standard deviations from five independent runs.
Figure 4: The learned policy function: Market Tightness θ vs. Unemployment Rate ut . The ⋆ marker indicates the theoretical steady-state equilibrium (0.033,0.78) . The shaded regions indicate the standard deviations from five independent runs.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
This survey (re)introduces reinforcement learning methods to economists. The curse of dimensionality limits how far exact dynamic programming can be effectively applied, forcing us to rely on suitably "small" problems or our ability to convert "big" problems into smaller ones. While this reduction has been sufficient for many classical applications, a growing class of economic models resists such reduction. Reinforcement learning algorithms offer a natural, sample-based extension of dynamic programming, extending tractability to problems with high-dimensional states, continuous actions, and strategic interactions. I review the theory connecting classical planning to modern learning algorithms and demonstrate their mechanics through simulated examples in pricing, inventory control, strategic games, and preference elicitation. I also examine the practical vulnerabilities of these algorithms, noting their brittleness, sample inefficiency, sensitivity to hyperparameters, and the absence of global convergence guarantees outside of tabular settings. The successes of reinforcement learning remain strictly bounded by these constraints, as well as a reliance on accurate simulators. When guided by economic structure, reinforcement learning provides a remarkably flexible framework. It stands as an imperfect, but promising, addition to the computational economist's toolkit. A companion survey (Rust and Rawat, 2026b) covers the inverse problem of inferring preferences from observed behavior. All simulation code is publicly available.
Multi-agent reinforcement learning (MARL) offers a principled way of populating simulated economies with many heterogeneous decision makers. Its adoption, pioneered in finance, has nonetheless been slow in macroeconomics because of two main limitations: the lack of a shared frame of reference connecting MARL systems to dynamic stochastic general equilibrium (DSGE) models the profession already trusts, and the very long training times. The first limitation was recently addressed by the Multi-Agent Reinforcement Learning Business Cycle (MARL-BC), which places deep RL households in a real business cycle economy and reduces to classic DSGE models in specific limits. Here we address the second limitation. We present JaxMARL-BC, a JAX-accelerated simulator that trains MARL-BC economies in seconds (between 75x and 724x faster than the reference implementation), and scales to 200,000 independent learning households. We verify its correctness against the known DSGE limits, characterise its scaling through a simple cost law, and demonstrate its value by a proof-of-concept calibration of an extended model with heterogeneous returns on capital, in a two-hour experiment that would previously have taken over a month. We release JaxMARL-BC as open source.
Daniele Montagnani, Andrea Coletta, Antonietta Mira +1
In this paper, we investigate whether a model-free RL agent can identify and exploit price manipulation opportunities more effectively than a traditional model-based approach that assumes correct specification of the data-generating process but relies on noisy parameter estimates. We consider a single-asset market in which prices evolve according to an Almgren-Chriss framework with non-linear permanent impact and linear temporary impact. We first establish the existence of price-manipulative strategies in discrete time and compute the optimal benchmark strategy using Sequential Least Squares Quadratic Programming under full information. We then compare two finite-sample learning approaches: a model-based procedure that estimates impact parameters from simulated execution data and an agnostic RL approach based on Deep Deterministic Policy Gradient, trained directly on the same amount of data. For intermediate volatility, the RL agent successfully discovers profitable manipulative strategies without explicit knowledge of the underlying model, even when training data are quite limited. More importantly, RL consistently outperforms the model-based approach when parameter estimates are affected by sampling error, despite the latter benefiting from the correct model specification. For large volatility, all methods are unable to identify manipulation opportunities, while for small volatility, the model based approach outperforms RL. These findings highlight both the effectiveness of RL in complex control problems and the risks associated with deploying learning algorithms in financial markets without appropriate safeguards.
Ioanna-Yvonni Tsaknaki, Andrea Macrì, Fabrizio Lillo
Scuola Normale Superiore, Piazza dei Cavalieri 7, Pisa 56126, Italy.