Counteractive RL: Rethinking Core Principles for Efficient and Scalable Deep Reinforcement Learning

Ezgi Korkmaz

cs.LGMar 16, 2026

Counteractive RL: Rethinking Core Principles for Efficient and Scalable Deep Reinforcement Learning

Authors: Ezgi Korkmaz

Abstract

Following the pivotal success of learning strategies to win at tasks, solely by interacting with an environment without any supervision, agents have gained the ability to make sequential decisions in complex MDPs. Yet, reinforcement learning policies face exponentially growing state spaces in high dimensional MDPs resulting in a dichotomy between computational complexity and policy success. In our paper we focus on the agent's interaction with the environment in a high-dimensional MDP during the learning phase and we introduce a theoretically-founded novel paradigm based on experiences obtained through counteractive actions. Our analysis and method provide a theoretical basis for efficient, effective, scalable and accelerated learning, and further comes with zero additional computational complexity while leading to significant acceleration in training. We conduct extensive experiments in the Arcade Learning Environment with high-dimensional state representation MDPs. The experimental results further verify our theoretical analysis, and our method achieves significant performance increase with substantial sample-efficiency in high-dimensional environments.

Figures & tables

Panel of Figure 1 — Figure 1: Human normalized scores median and 80 ${}^{\textrm{th}}$ percentile over all games in the Arcade Learning Environment (ALE) 100K benchmark for CoAct TD Learning and the canonical temporal difference learning with $\epsilon$ -greedy for QRDQN. Left: Median. Right: 80 ${}^{\textrm{th}}$ Percentile.

Panel of Figure 2 — Figure 2: Learning curves in the chain MDP with our proposed algorithm CoAct TD Learning, the canonical algorithm $\epsilon$ -greedy and the UCB algorithm with variations in $\epsilon$ .

Panel of Figure 3 — Figure 3: Temporal difference for our proposed algorithm CoAct TD Learning and the canonical $\epsilon$ -greedy algorithm in the Arcade Learning Environment 100K benchmark. Dashed lines report the temporal difference for the $\epsilon$ -greedy algorithm and solid lines report the temporal difference for the CoAct TD Learning algorithm. Colors indicate games.

Panel of Figure 4 — Figure 4: The learning curves for our proposed method CoAct TD Learning algorithm and canonical temporal difference learning in the Arcade Learning Environment with 200 million frame training. Left: JamesBond. MiddleLeft: Gravitar. MiddleRight: Surround. Right: Tennis.

Method	CoAct TD	$\epsilon$ -greedy	NoisyNetworks
Median	0.0927 $\pm$ 0.0050	0.0377 $\pm$ 0.0031	0.0457 $\pm$ 0.0035
20 ${}^{\textrm{th}}$ Percentile	0.0145 $\pm$ 0.0003	0.0056 $\pm$ 0.0017	0.0102 $\pm$ 0.0018
80 ${}^{\textrm{th}}$ Percentile	0.3762 $\pm$ 0.0137	0.2942 $\pm$ 0.0233	0.1913 $\pm$ 0.0144

Table 1: Human normalized scores median, 20

{}^{\textrm{th}}

and 80

{}^{\textrm{th}}

percentile across all of the games in the Arcade Learning Environment 100K benchmark for CoAct TD Learning,

\epsilon

-greedy and NoisyNetworks with DDQN.

Panel of Figure 5 — Figure 5: Human normalized scores median and 80 ${}^{\textrm{th}}$ percentile over all games in the Arcade Learning Environment (ALE) 100K benchmark in DDQN for CoAct TD Learning algorithm and the canonical temporal difference learning with $\epsilon$ -greedy. Left:Median. Right: 80 ${}^{\textrm{th}}$ Percentile.

Panel of Figure 6 — Figure 6: Left and Middle: Normalized temporal difference $\mathcal{TD}$ gain median across all games in the Arcade Learning Environment 100K benchmark for CoAct TD Learning and NoisyNetworks. Right: Temporal difference $\mathcal{TD}$ when exploring chain MDP with Upper Confidence Bound (UCB) method, $\epsilon$ -greedy and our proposed algorithm CoAct TD Learning.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Breaking the Computational Barrier: Provably Efficient Actor-Critic for Low-Rank MDPs
May 2, 2026Ruiquan Huang, Donghao Li, Yingbin Liang +1Markov Decision Processes Reinforcement Learning
University of Kentucky. · University of Virginia. · Ohio State University..
Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
Jun 4, 2026Johan Obando-Ceron, Lu Li, Scott Fujimoto +3Reinforcement Learning Representation Learning for RL
Mila – Québec AI Institute · Université de Montréal · McGill University +2
Simple Actors and Deep Critics for Scalable Reinforcement Learning
Aug 27, 2026Guhyeon Kang, Jaehwi Lee, Minhae KwonOffline RL Actor-Critic Methods
Sungkyunkwan University Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea · Soongsil University Republic of Korea

May 2, 2026J/K move · Enter open · S save

Breaking the Computational Barrier: Provably Efficient Actor-Critic for Low-Rank MDPs

Ruiquan Huang, Donghao Li, Yingbin Liang, Jing Yang

University of Kentucky. · University of Virginia. · Ohio State University..

Markov Decision Processes Reinforcement Learning Linear Markov Decision Processes Actor-Critic Methods Policy Learning Sample-Efficient RL

Open paper AskPDF arXiv

Reinforcement learning (RL) is a fundamental framework for sequential decision-making, in which an agent learns an optimal policy through interactions with an unknown environment. In settings with function approximation, many existing RL algorithms achieve favorable sample complexity, but often rely on computationally intractable oracles. In this paper, we use supervised learning as a computational proxy to establish a clear hierarchy of commonly adopted RL oracles under low-rank Markov Decision Processes (MDPs). This hierarchy shows that policy evaluation is the most computationally efficient oracle, provided that supervised learning can be efficiently solved. Motivated by this observation, we propose a novel optimistic actor-critic algorithm that relies solely on the policy evaluation oracle. We prove that our algorithm outperforms the existing sample complexity guarantees for low-rank MDPs while avoiding computationally expensive planning or optimization oracles commonly assumed in prior works. We further extend our theoretical results to approximately low-rank MDPs and demonstrate that this setting captures a broad class of real-world environments. Finally, we validate our theoretical results with experiments on several standard Gym environments.

Abstract

Reinforcement Learning

Explore similar work

Breaking the Computational Barrier: Provably Efficient Actor-Critic for Low-Rank MDPs

Representation Learning Enables Scalable Multitask Deep Reinforcement Learning

Simple Actors and Deep Critics for Scalable Reinforcement Learning