Automated market makers (AMMs) are a cornerstone of decentralised finance (DeFi). Constant product markets with concentrated liquidity, such as UniswapV3, are now a well-established design. In these markets, liquidity providers (LPs) face a sequential decision problem: they must decide when to rebalance their positions and which price ranges to allocate capital to as market conditions evolve. We formulate dynamic liquidity provision as a stochastic impulse control problem and use reinforcement learning (RL) to solve it, focusing on providing interpretable solutions. We show that learned policies exhibit rich state-dependent behaviour, allocating liquidity according to mispricing, rebalancing costs, uncertainty, inventory exposure, and heterogeneous risk preferences. These behaviours help compress the left tail of the Profit and Loss (PnL) distribution and avoid catastrophic outcomes under high uncertainty. Finally, we benchmark the RL agents against baseline and sophisticated agents from the AMM microstructure literature and analyse their performance.
Figures & tables
σ=0.01,g=2
σ=0.02,g=4
Agent
PnL
CVaR
PnL
CVaR
DeployNarrow
15.85
-14.52
6.48
-35.37
DeployWide
12.78
-15.11
4.09
-36.48
ArrivalRebalance
49.72
6.86
5.69
-36.23
CDM
35.18
11.55
15.01
-26.23
Risk neutral
Table 1. Mean PnL and 5% CVaR in USDC over 1000 common evaluation trajectories. Values following ± denote the standard deviation across 10 independently trained policies. Bold marks the best RL and benchmark method. p tests their difference, treating training seeds as replication units and additionally propagating trajectory sampling error.
σ=0.03,g=6
Agent
PnL
CVaR
DeployNarrow
0.10
-56.1 7
DeployWide
-2.12
-57.33
ArrivalRebalance
-5.38
-62.64
CDM
-3.10
-56.93
Risk neutral
Risk averse ( ϕ=50 )
Table 2. Under adversarial conditions ( σ=0.03 , g=6 ), the risk-averse RL agents learn mainly not to deploy.
Figure 1. Differences in the mean decomposed PnL components between the RL agents across volatility, by risk profile (rows) and gas cost (columns). ΔNetPnL=ΔFees+(−ΔIL)+(−ΔGas)=PPO_narrow(PnL)−PPO(PnL) denotes the difference in mean PnL and is shown by the diamond marker. A positive ΔFees (green), negative −ΔIL (purple), and negative −ΔGas (orange) indicate that PPO_narrow earns more fees, incurs greater IL, and pays more gas than PPO, respectively. Each quantity is averaged over 1,000 trajectories. Missing bars indicate that one or both agents did not deploy.
Figure 2. Position width and rebalancing behaviour of PPO and PPO_narrow. Panel ( 2(a) ) shows mean quoted width, while Panel ( 2(b) ) shows per-decision rebalance rates. Labels and Legends as in Figure 1 .
Figure 3. Liquidity provision behaviour of all strategies for one random simulation from the evaluation trajectories (scenario: σ=0.01,g=2, risk neutral agents). The red thin vertical lines show when agents rebalance.
Figure 4. Illustration of the empirical CL shapes around pool prices (left) in our model-based environment (right). Left figure shows the normalised liquidity distribution around the current tick for Optimism (red), Arbitrum (black) and Ethereum (purple) as observed from real on-chain data calculated in ( Adams, 2024 ) . Right figure shows a superposition of all four PPO risk-profile agents, PPO_narrow, and a wide background liquidity. Each agent’s position is her reported average width for σ=0.02 and g=2 with indicative relative wealths.
Figure 5. The full action response that a risk-averse PPO agent has for fixed wealth, with ϕ=50 for gas_cost=4 and σ=0.01 at t=0.4 , across two important dimensions: the amount of the risky asset she holds (e.g. ETH), and the adversarial signal (mispricing). It is assumed that the current action of the agent is a symmetrical narrow position (-1,1) around the current tick. The shaded areas in the first two plots are where the agent holds her existing position and the chosen actions can be ignored.
Figure 6. Action responses of the risk-neutral PPO_narrow agent for σ=0.01 with gas cost g=2 (left panel) and g=6 (right panel) at time t=0.4 . The x-axis shows the mispricing (USDC), while the y-axis shows the policy output, i.e. the chosen range centre. The colour bar indicates the current centre (ticks). Both the chosen action and current state are expressed as centre offsets. Dark purple denotes a position well below the current tick, bright yellow well above it, and teal positions partially in range. Solid line segments indicate rebalancing; dashed segments indicate that the agent ignores its centre output and holds its position.
Concentrated liquidity provision in decentralized exchanges presents a fundamental Impulse Control problem. Liquidity Providers (LPs) face a non-trivial trade-off between maximizing fee accrual through tight price-range concentration and minimizing the friction costs of rebalancing, including gas fees and swap slippage. Existing methods typically employ heuristic or threshold strategies that fail to account for market dynamics. This paper formulates liquidity management as an optimal control problem and derives the corresponding Hamilton-Jacobi-Bellman quasi-variational inequality (HJB-QVI). We present an approximate solution RAmmStein, a Deep Reinforcement Learning method that incorporates the mean-reversion speed (theta) of an Ornstein-Uhlenbeck process among other features as input to the model. We demonstrate that the agent learns to separate the state space into regions of action and inaction. We further extend the framework with RAmmStein-Width, which jointly optimizes rebalancing timing and position width via a 6-action DDQN. We evaluate the framework using high-frequency 1Hz Coinbase trade data comprising over 6.8M trades on a realistic environment (10M TVL, 1% default width). Experimental results show that RAmmStein achieves a net ROI of 1.60%, the highest among all realistic (non-omniscient) strategies, while greedy strategies lose up to -8.4% to gas costs. Notably, the agent reduces rebalancing frequency by 85% compared to greedy rebalancing. RAmmStein-Width discovers extreme parsimony on its own, executing only 9 rebalances and $40 in gas, and degrades more slowly than all active strategies at elevated gas costs. Our results demonstrate that regime-aware laziness can significantly improve capital efficiency by preserving the returns that would otherwise be eroded by the operational costs.
We present SAiFE_gym, a Python module that provides a collection of simulation environments for studying trading problems in Constant Product Markets (CPMs) with Concentrated Liquidity (CL). These markets give Liquidity Providers (LPs) granular control over how their capital is allocated and enable them to adjust their range of liquidity provision dynamically based on market conditions, which in turn, dictates how they earn fees. We decompose the microstructure of CPMs with CL in interactive components that allow researchers and practitioners to capture various economic settings. We employ a vectorized approach to optimize our environments, making them scalable for high dimensional Reinforcement Learning (RL) workflows that best describe sequential decision problems. We demonstrate the benefits of our environments by evaluating the performance of RL agents in CPMs with CL under uncertainty in market parameters.
Automated Market Makers (AMMs), as a core infrastructure of decentralized finance (DeFi), uniquely drive on-chain asset pricing through a deterministic reserve ratio mechanism. Unlike traditional markets, AMM price dynamics is triggered largely by on-chain events (e.g., swap) that change the reserve ratio, rather than by continuous responses to off-chain information. This makes event-level analysis crucial for understanding price formation mechanisms in AMMs. However, existing research generally neglects the micro-structural dynamics at the AMMs level, lacking both a comprehensive dataset covering multiple protocols with fine-grained event classification and an effective framework for event-aware modeling. To fill this gap, we construct a dataset containing 8.9 million on-chain event records from four representative AMMs protocols: Pendle, Uniswap v3, Aave and Morpho, with precise annotations of transaction type and block height timestamps. Furthermore, we propose an Uncertainty Weighted Mean Squared Error (UWM) loss function, which incorporates the block interval regression term into the traditional Time-Point Process (TPP) objective function by weighting the uncertainty with homoscedasticity. Extensive experiments on eight advanced TPP architectures demonstrate that this loss function reduces the time prediction error by an average of 56.41% while maintaining the accuracy of event type prediction, establishing a robust benchmark for event-aware prediction in the AMMs ecosystem. This work provides the necessary data foundation and methodological framework for modeling the discreteness and event-driven characteristics of on-chain price discovery. All datasets and source code are publicly available. https://github.com/yosen-king/Deep-AMM-Events
Huaiyu Jia, Jiehshun You, Yizhi Luo +2
HKUST(GZ) · GUANGZHOU, China · Hong Kong Polytechnic University +1