RAmmStein: Regime Adaptation in Mean-reverting Markets with Stein Thresholds -- Optimal Impulse Control in Concentrated AMMs

Pranay Anchuri

cs.LGFeb 23, 2026

RAmmStein: Regime Adaptation in Mean-reverting Markets with Stein Thresholds -- Optimal Impulse Control in Concentrated AMMs

Authors: Pranay Anchuri

Organizations: Offchain Labs

Abstract

Concentrated liquidity provision in decentralized exchanges presents a fundamental Impulse Control problem. Liquidity Providers (LPs) face a non-trivial trade-off between maximizing fee accrual through tight price-range concentration and minimizing the friction costs of rebalancing, including gas fees and swap slippage. Existing methods typically employ heuristic or threshold strategies that fail to account for market dynamics. This paper formulates liquidity management as an optimal control problem and derives the corresponding Hamilton-Jacobi-Bellman quasi-variational inequality (HJB-QVI). We present an approximate solution RAmmStein, a Deep Reinforcement Learning method that incorporates the mean-reversion speed (theta) of an Ornstein-Uhlenbeck process among other features as input to the model. We demonstrate that the agent learns to separate the state space into regions of action and inaction. We further extend the framework with RAmmStein-Width, which jointly optimizes rebalancing timing and position width via a 6-action DDQN. We evaluate the framework using high-frequency 1Hz Coinbase trade data comprising over 6.8M trades on a realistic environment (10M TVL, 1% default width). Experimental results show that RAmmStein achieves a net ROI of 1.60%, the highest among all realistic (non-omniscient) strategies, while greedy strategies lose up to -8.4% to gas costs. Notably, the agent reduces rebalancing frequency by 85% compared to greedy rebalancing. RAmmStein-Width discovers extreme parsimony on its own, executing only 9 rebalances and $40 in gas, and degrades more slowly than all active strategies at elevated gas costs. Our results demonstrate that regime-aware laziness can significantly improve capital efficiency by preserving the returns that would otherwise be eroded by the operational costs.

Figures & tables

Parameter	Value
Learning rate	$10^{-4}$
Discount factor $\gamma$	0.99
Replay buffer size	100,000
Batch size	128
Target update frequency	100 steps
$\epsilon$ start / end / decay	1.0 / 0.05 / 0.9998

TABLE I: Training Hyperparameters

Parameter	Value
Range width	1% (100 bps)
Pool fee tier ( $\phi$ )	0.05%
Gas cost ( $G$ )	$2.00 USD
Initial capital ( $K$ )	$10,000 USD
Pool TVL ( $L_{pool}$ )	$10,000,000 USD
DEX/CEX volume ratio ( $\alpha$ )	10%

TABLE II: Environment Parameters

Strategy	Active%	Rebal.	Fees ($)	Gas ($)	Net ROI
Set & Forget (Oracle)	100.0%	1	198	4.5	1.94%
Fixed Passive	19.7%	1	117	4.5	1.13%
Fixed Active (Greedy)	100.0%	344	644	1,483	$-$ 8.40%
LSTM Predictor	95.1%	237	580	1,036	$-$ 4.55%
RAmmStein	57.7%	51	389	228	1.60%
RAmmStein-Width	9.3%	9	75	40	0.35%

TABLE III: Strategy Comparison on Test Data (

10M TVL, 1% width,

2 gas)

Fig. 1: Learned decision boundary with actual rebalance events (blue dots, $n=50$ ). The heatmap shows $Q(\text{rebalance})-Q(\text{hold})$ ; green = hold preferred, red = rebalance preferred. White dashed lines mark the range edges ( $d_{\text{edge}}=\pm 1$ ). All rebalances occur outside the range at low $\theta$ .

Gas (USD)	Fixed Active	RAmmStein	RAmmStein-Width
$1.00	$-$ 4.96%	1.86%	0.44%
$2.00	$-$ 8.40%	1.60%	0.35%
$5.00	$-$ 18.72%	0.83%	0.08%
$10.00	$-$ 35.94%	$-$ 0.50%	$-$ 0.36%
$20.00	$-$ 70.38%	$-$ 3.16%	$-$ 1.24%

TABLE IV: Net ROI vs. Gas Cost

Fig. 2: RAmmStein-Width: $Q(\text{best rebalance})-Q(\text{hold})$ landscape with actual rebalance events (left), and per-event $d_{\text{edge}}$ with chosen width annotation (right). All 8 rebalances are downside exits at $\theta\approx 0$ ; 7 of 8 select 1% width.

Strategy	Fees ($)	Gas ($)	Net PnL ($)	Fee/Gas
Set & Forget (Oracle)	198	4.5	194	44.1
Fixed Passive	117	4.5	113	26.1
Fixed Active	644	1,483	$-$ 840	0.4
LSTM Predictor	580	1,036	$-$ 455	0.6
RAmmStein	389	228	160	1.7
RAmmStein-Width	75	40	35	1.9

TABLE V: Fee and Gas Decomposition

Explore similar work

Aug 19, 2026

Concentrated Liquidity Provision: a Reinforcement Learning Perspective

Automated market makers (AMMs) are a cornerstone of decentralised finance (DeFi). Constant product markets with concentrated liquidity, such as UniswapV3, are now a well-established design. In these markets, liquidity providers (LPs) face a sequential decision problem: they must decide when to rebalance their positions and which price ranges to allocate capital to as market conditions evolve. We formulate dynamic liquidity provision as a stochastic impulse control problem and use reinforcement learning (RL) to solve it, focusing on providing interpretable solutions. We show that learned policies exhibit rich state-dependent behaviour, allocating liquidity according to mispricing, rebalancing costs, uncertainty, inventory exposure, and heterogeneous risk preferences. These behaviours help compress the left tail of the Profit and Loss (PnL) distribution and avoid catastrophic outcomes under high uncertainty. Finally, we benchmark the RL agents against baseline and sophisticated agents from the AMM microstructure literature and analyse their performance.

Georgios Chionas, Charalampos Kleitsikas, Stefanos Leonardos +2

University of Oxford Oxford, UK

Decentralized Finance Algorithmic Trading Reinforcement Learning Stochastic Optimal Control

Ask PDF

Jul 12, 2026

Reinforcement Learning for Execution under Dynamic Fees in a Closed-Loop DEX Simulator

Trader-facing dynamic fees are increasingly proposed for automated market makers (AMMs), but historical data do not identify how order flow would respond: trader-facing fees do not vary, trader types are latent, and a replayed tape is not a sequential decision environment. We therefore construct a minimal closed-loop simulator in which the missing signal exists by construction: two constant-product pools repriced by an equilibrium-inspired dynamic-fee rule, fee-sensitive noise flow, and closed-form CEX--AMM arbitrage. Equilibrium is used as a closure principle, not as an object the trader learns. Against a tuned benchmark ladder of schedule, planning, lookahead, and tabular policies, a small DQN is the only evaluated valid policy whose paired improvement over tuned one-step routing excludes zero. On a reserved final block of 1{,}000 seeds with completion forced to 1.0 for every policy, it reduces implementation shortfall under every tested intra-step ordering, by

13.3\bps

of order notional under the pre-specified agent-last ordering, and the edge is concentrated in, and learned from, dynamic-fee environments: under constant fees the paired difference is indistinguishable from zero. The result is model-conditioned counterfactual evidence about execution control in AMMs, not evidence about historical traders, equilibrium play, or deployable profit.

Wen-Ting Wang

Decentralized Finance Finance Algorithmic Trading

Ask PDF

Sep 15, 2026

SAiFE-gym: Model-based Environments for Automated Market Making with Concentrated Liquidity

We present SAiFE_gym, a Python module that provides a collection of simulation environments for studying trading problems in Constant Product Markets (CPMs) with Concentrated Liquidity (CL). These markets give Liquidity Providers (LPs) granular control over how their capital is allocated and enable them to adjust their range of liquidity provision dynamically based on market conditions, which in turn, dictates how they earn fees. We decompose the microstructure of CPMs with CL in interactive components that allow researchers and practitioners to capture various economic settings. We employ a vectorized approach to optimize our environments, making them scalable for high dimensional Reinforcement Learning (RL) workflows that best describe sequential decision problems. We demonstrate the benefits of our environments by evaluating the performance of RL agents in CPMs with CL under uncertainty in market parameters.

Georgios Chionas, Charalampos Kleitsikas, Stefanos Leonardos +2

University of Oxford Oxford, UK

Reinforcement Learning Decentralized Finance Algorithmic Trading

Ask PDF