cs.ROOct 27, 2026

Echo: Scaling Visual Reinforcement Learning with Off-policy Experience

Authors: Gagan Khandate

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Scaling visual reinforcement learning (RL) with GPU-accelerated simulation is computationally expensive due to rendering and visual-policy inference costs. Consequently, two-stage teacher--student distillation is commonly used to train visual policies, but introduces an imitation gap. In this preliminary study, we introduce \echomethod{}, a single-stage off-policy RL framework that scales experience collection without proportionally increasing visual computation. Through inverse distillation, a state-based auxiliary policy imitates the evolving visual policy and collects inexpensive, unrendered experience. Echo leverages this experience for off-policy value learning, directly improving the visual policy through a shared privileged critic. We instantiate \echomethod{} with FastTD3 and evaluate it on three visual control tasks and two dexterous manipulation tasks in Genesis. Relative to \sdpgmethod{} under the evaluated configurations, Echo reduces time to reference returns by an average of 60% across the three locomotion tasks and 56% across both hand tasks, while improving peak mean returns by 62% and 12%, respectively. Both hand tasks also achieve higher mean returns at matched simulator-transition budgets. These initial results suggest that scaling off-policy experience is a promising approach to efficient direct visual RL.

Figures & tables

Figure 1: Scaling visual RL with off-policy auxiliary experience. A state-based imitator collects unrendered transitions, while a small rendered subset supplies visual-policy experience. Both streams train a privileged critic that directly improves the visual policy.
Figure 2: Two key ideas in Echo . (1) Imitate the visual policy: a state-based imitator matches noise-free visual actions with stopped gradients. (2) Update the visual policy with the shared critic: both experience streams train the critic, whose action gradient improves the visual policy while critic parameters remain fixed. The imitator’s EMA copy supplies TD continuation actions without successor images (dashed purple). Gray arrows denote experience flow.
Figure 3: Visual locomotion benchmarks in Genesis. Episode returns as a function of wall-clock training time (top, excluding evaluation) and total simulator transitions (bottom, including auxiliary interactions). Solid lines and shaded bands indicate mean and sample standard deviation across three seeds, respectively; faint lines show individual runs. Reference hyperparameters are in Appendix C ; the Ant baseline qualification is in Appendix D .
Figure 4: Visual dexterous manipulation in Genesis. Episode returns on Allegro Hand (top) and Shadow Hand (bottom) as a function of wall-clock training time (left, excluding evaluation) and total simulator transitions (right, including auxiliary interactions). All runs use a three-hour training budget. Solid lines and shaded bands indicate mean and sample standard deviation across three seeds, respectively; faint lines show individual runs. Curves are linearly interpolated over the common support of all six runs. Differences in initial returns arise from shaped rewards and policy initialization.
Figure 5: Echo dexterous manipulation rollouts. Selected in-hand object reorientation episodes on Allegro Hand and Shadow Hand, showing target orientations, timestamps, and measured orientation errors. Both policies maintain object grasp throughout the episodes. Rollouts are generated using earlier checkpoints than those evaluated in Figure 4 and are illustrative rather than representative of aggregate success rates.
Figure 6: Inverse distillation and critic-gradient diagnostics. (Top) Online imitation loss (teal, solid) and post-update EMA action error (purple, dashed) on fresh visual-policy states, averaged over 60-second intervals. (Middle and bottom) Critic action-gradient dot product and cosine similarity between visual-policy and imitator actions on paired rendered replay states, measured in separate diagnostic runs. Lines and shaded bands show mean and sample standard deviation across three seeds; faint lines show individual runs.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

HyperparameterValue
Simulation and observations
Total parallel environments4096
Rendered environments64
Auxiliary environments4032
Image resolution84×8484\times 84 RGB
Replay history per environment512 transitions
Appendix
Table 1: Echo training hyperparameters.
More in this topic

Visual RL

Visual reinforcement learning trains agents to perceive images and learn actions through interaction, enabling control when states are not directly observed.

See all 60 papers

Explore similar work

CardsList

  1. Efficient On-policy Visual-RL via Stochastic Decoupled Policy Gradient

    May 26, 2026Haoxiang You, Yilang Liu, Davis Zong +5Policy OptimizationVisuomotor Policy Learning

    Yale University, Department of Mechanical Engineering. · Yale University, Department of Computer Science. · Shanghai Jiao Tong University, School of Computer Science. +1

  2. V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control

    Aug 8, 2026Donghu Kim, Youngdo Lee, Hojoon Lee +6Reinforcement LearningVisual RL

    KAIST · Mila – Québec AI Institute · Université de Montréal +2

  3. On-Policy Visual Evidence Distillation

    Sep 29, 2026Shaohang Wei, Feifan Song, Guangyue Peng +9VLM DistillationVisually Grounded Reasoning

    Peking University · CUHK · Nanjing University +1