Reinforcement-learning policies for robot locomotion can fail at deployment when operating conditions shift (e.g., rough or slippery terrain, payload changes, sensing degradation), and retraining or manual recalibration is often infeasible under on-device latency and memory constraints. We study test-time adaptation (TTA) for robotic control: adapting a pretrained policy online and in a self-supervised manner during execution, without access to rewards. Our starting point is a lightweight baseline that updates only the actor using a frozen pretrained critic as a deployment-time performance surrogate. Building on this, we introduce TEMPO, a test-time entropy-regularized, memory-guided policy optimization method that stabilizes critic-driven updates under severe shift. TEMPO combines (i) an entropy-promoting regularizer on the critic's predictive distribution to mitigate overconfident extrapolation, and (ii) a score-based memory, inspired by prioritized replay, that prioritizes low-value ("hard") observations and applies importance sampling to focus limited computation on the most informative deployment conditions. Across Go1, Go2, and G1 in MuJoCo Playground under four representative shifts and multiple architectures, TEMPO yields consistent gains, reaching up to 80% improvement in a single-device setting. We further validate on a physical Unitree Go2 with an added calf payload, where fully on-device adaptation improves performance over the non-adapted baseline across real-world episodes.
Figures & tables
Figure 1: Motivation and pipeline. Left: Pretrained locomotion policies degrade as shift severity increases. Middle: Off-the-shelf TTA and prior RL adaptation baselines fail under a rocky-terrain shift. Right: TEMPO loop: execute πθt , update memory B , and periodically adapt via Eq. ( 2 ).
Figure 2: Main results on Go2 under deployment-time distribution shift. Top: episodic return. Bottom: average forward linear velocity. Shifts (left → right): 1− friction, calf mass, rough terrain, torso mass (severity increases along x-axis). As the severity of the domain shift increases, performance degrades more rapidly. Naive adaptation via QMAX recovers part of the performance drop. Our proposed TEMPO consistently provides the largest performance gain, especially under severe shifts.
Figure 3
Mass on Calf = 2 kg
Mass on Torso = 5 kg
Floor friction
Rocky terrain
Floor friction
Rocky terrain
Method
0.90
0.85
0.03
0.06
0.90
0.85
0.03
0.06
No Adapt
9.2±0.1
12.9 ±0.2
19.9±0.1
14.5±0.2
4.2 ±0.2
6.2 ±0.1
14.7 ±0.3
7.8 ±0.2
QMAX
9.7±0.2
13.4 ±0.4
20.7±0.1
15.9±0.2
4.8 ±0.1
7.0 ±0.3
17.3 ±0.5
9.9 ±0.2
TEMPO
10.4 ±0.2
14.4 ±0.1
21.2 ±0.3
17.5 ±0.2
5.6 ±0.2
9.0 ±0.4
20.1 ±0.2
14.6 ±0.2
Table 2: Performance Comparison Under Mixed Corruptions. We report the environment reward (six seeds) for the non-adapted policy, QMAX, and our proposed TEMPO when the policy is subjected to a composition of corruptions (e.g. mass on torso and rocky terrain simultaneously). Our proposed TEMPO consistently provides the best performance by a significant margin.
Figure 4: Continual TTA under progressively increasing shift severity. We report environment rewards versus adaptation steps as the deployment shift intensifies over time. TEMPO recovers faster and converges to a higher performance plateau than QMAX across all corruptions.
Table 3: Ablating TEMPO . Average episodic return under increasing shift severity for three corruption families. Adding entropy regularization and score-based memory each improves over QMAX , and combining both ( TEMPO ) yields the best robustness across all severities.
Figure 5: RMA ablation. Frozen RMA vs native RMA adaptation vs RMA+TEMPO. TEMPO provides consistent and significant additional gains.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8
Figure 6: Generalization across architectures and embodiments. Rocky-terrain results (left → right): Fast-TD3 trained with domain randomization, SimBA-V2, and the G1 humanoid. TEMPO transfers without architecture-specific modifications and remains effective across settings.
Go1
G1
Method
MC=4
MC=5
MT=7.5
MT=10
MT=2
MT=3
MT=5
No Adapt
18.9
16.3
13.6
6.8
21.3
17.5
10.0
QMAX
19.6
17.6
13.6
6.9
24.1
22.1
17.4
TEMPO
20.5
18.6
16.9
11.0
24.5
23.4
18.5
Appendix
Table 4: Additional results on Go1 and G1 under payload perturbations. MC denotes mass on calf and MT denotes mass on torso.
Figure 7: Robustness to on-device adaptation hyperparameters. TEMPO is robust and performs best across different memory sizes (left), batch sizes (middle), and adaptation intervals (right). All experiments are run with Fast-TD3 on Go2 with Mass on Torso Corruption with severity of 7.5 kg.
Figure 8: Ablating λ .
Figure 9: Continual TTA followed by return to the source domain. Performance under a non-stationary deployment stream in which the severity of the domain shift increases over time and is then reset to nominal conditions. Left: mass-on-calf shift. Right: mass-on-torso shift. Architecture: Fast - TD3. Both QMAX and TEMPO recover source-domain performance immediately after the shift is removed, indicating no catastrophic forgetting despite prolonged adaptation under shift.
Figure 10: Angular velocity tracking on Go2 under deployment-time distribution shift. We report angular velocity tracking error as a function of shift severity for four corruption families. Results are averaged over five seeds. Critic-guided test-time adaptation improves tracking accuracy relative to no adaptation, with TEMPO achieving the most stable and accurate angular velocity tracking, particularly under severe distribution shifts.
Visuomotor policies trained via imitation learning often inherit the unnecessarily slow timing of teleoperated demonstrations. Yet uniform speedup is unreliable because different phases of a manipulation task tolerate acceleration differently. In this work, we introduce AdaTempo, a self-supervised method that accelerates visuomotor policies by exploiting shared relative-tempo structure in demonstrations. AdaTempo establishes phase correspondence, aggregates the aligned relative tempo into a consensus, and maps it to a continuous speedup profile used to resample demonstrations into accelerated training trajectories. Training standard policies such as ACT or Diffusion Policy on these resampled trajectories directly embeds the desired tempo in the learned behavior, without runtime tempo selection or online retiming. Extensive evaluations show that AdaTempo achieves up to a 3.57× speedup and yields a stronger success--speed trade-off than the original policies and representative acceleration baselines.
While pretrained robotic policies exhibit impressive capabilities in controlled environments, unobserved physical properties and dynamics require these policies to rapidly adapt during deployment. Existing test-time adaptation methods typically rely on sparse scalar rewards, failing to exploit the rich geometric and dynamic feedback from the environment during physical interaction. To address this challenge, we propose SCOUT, a dynamics-aware meta-learning framework that enables manipulation policies to rapidly adapt by continuously revising their internal beliefs about environment dynamics. Our approach couples an action-prediction policy with a forward dynamics model via a shared belief latent space. During meta-training, an inner loop updates this shared belief latent by minimizing the dynamics prediction error against the observed action outcome, while the outer loop optimizes the network for action selection. At deployment, this structure allows the agent to infer and adapt to unknown physical dynamics on the fly. By updating its latent belief based on action-outcome mismatches, the policy automatically adapts without risking catastrophic forgetting. We demonstrate that SCOUT significantly accelerates online adaptation across simulated manipulation benchmarks and achieves robust sim-to-real transfer in the real world. Project webiste can be found here: https://liy1shu.github.io/SCOUT/
Yishu Li, Liyuan Geng, Xinyi Mao +2
Robotics Institute, Carnegie Mellon University · Computer Science and Technology, Tsinghua University
Generalist robot policies carry broad manipulation priors from large-scale data, but specializing them to a new task remains the deployment bottleneck. This requires eliciting task-specific behavior from limited demonstrations without degrading their broad capabilities. We introduce Proxy Policy Steering (PPS), an inference-time adaptation method that resolves this challenge by training two lightweight proxy policies whose calibrated velocity-space difference steers the frozen base sampler. A reference proxy models the frozen base's behavior on target-task observations, and a task proxy, initialized from the reference, captures how this behavior changes under task supervision. Their difference forms a calibrated velocity-space residual that steers the frozen base sampler at every denoising step. We identify the conditions under which this residual isolates the change induced by task supervision, and validate them empirically. Because the base is never directly modified, its broad capabilities remain available at inference, including behaviors such as recovery from failure that the demonstrations themselves do not exercise. Adaptation requires only forward velocity predictions from the base, making PPS lightweight to train and applicable even without access to the base's parameters. On 8 real-world and 4 simulation manipulation tasks, PPS lifts the state-of-the-art pi 0.5 base policy by 53% absolute success rate on average, with zero-to-one gains on tasks the base never solves, while preserving the base's broad capabilities. PPS outperforms LoRA fine-tuning, from-scratch specialists, residual policies, and prior inference-time steering methods.