Automated market makers (AMMs) are a cornerstone of decentralised finance (DeFi). Constant product markets with concentrated liquidity, such as UniswapV3, are now a well-established design. In these markets, liquidity providers (LPs) face a sequential decision problem: they must decide when to rebalance their positions and which price ranges to allocate capital to as market conditions evolve. We formulate dynamic liquidity provision as a stochastic impulse control problem and use reinforcement learning (RL) to solve it, focusing on providing interpretable solutions. We show that learned policies exhibit rich state-dependent behaviour, allocating liquidity according to mispricing, rebalancing costs, uncertainty, inventory exposure, and heterogeneous risk preferences. These behaviours help compress the left tail of the Profit and Loss (PnL) distribution and avoid catastrophic outcomes under high uncertainty. Finally, we benchmark the RL agents against baseline and sophisticated agents from the AMM microstructure literature and analyse their performance.
Figures & tables
σ=0.01,g=2
σ=0.02,g=4
Agent
PnL
CVaR
PnL
CVaR
DeployNarrow
15.85
-14.52
6.48
-35.37
DeployWide
12.78
-15.11
4.09
-36.48
ArrivalRebalance
49.72
6.86
5.69
-36.23
CDM
35.18
11.55
15.01
-26.23
Risk neutral
Table 1. Mean PnL and 5% CVaR in USDC over 1000 common evaluation trajectories. Values following ± denote the standard deviation across 10 independently trained policies. Bold marks the best RL and benchmark method. p tests their difference, treating training seeds as replication units and additionally propagating trajectory sampling error.
σ=0.03,g=6
Agent
PnL
CVaR
DeployNarrow
0.10
-56.1 7
DeployWide
-2.12
-57.33
ArrivalRebalance
-5.38
-62.64
CDM
-3.10
-56.93
Risk neutral
Risk averse ( ϕ=50 )
Table 2. Under adversarial conditions ( σ=0.03 , g=6 ), the risk-averse RL agents learn mainly not to deploy.
Figure 1. Differences in the mean decomposed PnL components between the RL agents across volatility, by risk profile (rows) and gas cost (columns). ΔNetPnL=ΔFees+(−ΔIL)+(−ΔGas)=PPO_narrow(PnL)−PPO(PnL) denotes the difference in mean PnL and is shown by the diamond marker. A positive ΔFees (green), negative −ΔIL (purple), and negative −ΔGas (orange) indicate that PPO_narrow earns more fees, incurs greater IL, and pays more gas than PPO, respectively. Each quantity is averaged over 1,000 trajectories. Missing bars indicate that one or both agents did not deploy.
Figure 2. Position width and rebalancing behaviour of PPO and PPO_narrow. Panel ( 2(a) ) shows mean quoted width, while Panel ( 2(b) ) shows per-decision rebalance rates. Labels and Legends as in Figure 1 .
Figure 3. Liquidity provision behaviour of all strategies for one random simulation from the evaluation trajectories (scenario: σ=0.01,g=2, risk neutral agents). The red thin vertical lines show when agents rebalance.
Figure 4. Illustration of the empirical CL shapes around pool prices (left) in our model-based environment (right). Left figure shows the normalised liquidity distribution around the current tick for Optimism (red), Arbitrum (black) and Ethereum (purple) as observed from real on-chain data calculated in ( Adams, 2024 ) . Right figure shows a superposition of all four PPO risk-profile agents, PPO_narrow, and a wide background liquidity. Each agent’s position is her reported average width for σ=0.02 and g=2 with indicative relative wealths.
Figure 5. The full action response that a risk-averse PPO agent has for fixed wealth, with ϕ=50 for gas_cost=4 and σ=0.01 at t=0.4 , across two important dimensions: the amount of the risky asset she holds (e.g. ETH), and the adversarial signal (mispricing). It is assumed that the current action of the agent is a symmetrical narrow position (-1,1) around the current tick. The shaded areas in the first two plots are where the agent holds her existing position and the chosen actions can be ignored.
Figure 6. Action responses of the risk-neutral PPO_narrow agent for σ=0.01 with gas cost g=2 (left panel) and g=6 (right panel) at time t=0.4 . The x-axis shows the mispricing (USDC), while the y-axis shows the policy output, i.e. the chosen range centre. The colour bar indicates the current centre (ticks). Both the chosen action and current state are expressed as centre offsets. Dark purple denotes a position well below the current tick, bright yellow well above it, and teal positions partially in range. Solid line segments indicate rebalancing; dashed segments indicate that the agent ignores its centre output and holds its position.