Register-Routed Delayed Fusion: Rewiring Shortcut-Prone Observation Fusion in Visuomotor Imitation

Jieting Long; Weidong Cai; Weiming Zhi

cs.ROOct 2, 2026

Register-Routed Delayed Fusion: Rewiring Shortcut-Prone Observation Fusion in Visuomotor Imitation

Authors: Jieting Long, Weidong Cai, Weiming Zhi

Organizations: School of Computer Science, The University of Sydney, Australia.

Abstract

Visuomotor imitation policies combine high-dimensional visual observations with compact signals such as proprioception, and their fusion topology determines when and through which tokens these streams interact. In dense token fusion, visual tokens may attend directly to compact tokens from the first encoder layer, allowing action-predictive compact cues to influence spatial visual representations early in their formation. We ask whether controlling this route improves visual responsiveness and policy behavior. We introduce Register-Routed Delayed Fusion (RRDF), which masks direct compact-visual attention and stages cross-modal interaction through a learned register workspace. Its isolate-collect-route schedule protects an early stream-separated prefix and later permits only register-mediated exchange, while compact conditioning remains available to the native action generator. Across five simulation tasks and three real-robot tasks, RRDF matches or improves dense ACT under nominal conditions. Appearance-shift evaluations on four simulation tasks and held-out-position evaluations on three real-robot tasks also favor RRDF. Phase-matched input probes show lower measured state-to-image sensitivity, while ablations indicate that adding registers alone does not reproduce the full performance gain. These results support controlling cross-modal propagation while retaining compact action conditioning.

Figures & tables

Fig. 1 : Representative behavioral difference under a held-out object-position shift. The dense-fusion policy produces similar trajectories for seen and shifted object configurations, failing to accommodate the shifted scene. RRDF instead adapts its trajectory and completes the task, illustrating improved scene responsiveness under the proposed delayed-fusion topology.

Fig. 2 : Overview of RRDF . Dense fusion permits direct compact–visual attention from the first encoder layer. RRDF instead uses source-specific register accumulation, joint collection without primary-stream readback, and later register-mediated feedback. The outer panels show representative dense-fusion failure and RRDF success under a held-out-position shift.

Fig. 3 : Depth-dependent directed topology of RRDF . Left to right: dense fusion, source-specific register collection during Isolation, joint workspace formation during Collection, and register-mediated feedback during Routing. Direct

\mathcal{C}\leftrightarrow\mathcal{V}

attention remains masked throughout; arrows denote permitted source-to-destination information flow.

Panel of Figure 4 — Fig. 4 : Controlled appearance-shift stress tests in simulation. (a) Nominal, bright-floor, and recolored observations for each task (left to right); geometry and dynamics are unchanged. (b) Closed-loop success under each shift. Solid bars show dense-fusion success, hatched extensions the RRDF gain, and total height the RRDF result.

Fig. 5 : Qualitative real-robot behaviours. Matched dense-fusion and RRDF rollouts illustrate Spatial Adaptation on Soap Pick-up (top), Precision Alignment on Plate Stacking (bottom-left), and Precision Placement on Toolbox (bottom-right), under nominal and held-out-position conditions. Red/green boxes mark representative dense-fusion failures and RRDF successes.

ACT Baseline
Task	Action	Perf. (%) $\uparrow$	SD.	Time (s) $\downarrow$	SD.
Can	joint	99.33 /96.67	1.15/1.44	7.08/7.08	0.08/0.02
Square	joint	75.33/73.53	3.06/2.78	10.56/10.49	0.68/0.15
Stack-D1	EEF	78.67/66.33	2.31/4.92	6.11/6.17	0.43/0.27
Coffee-D2	EEF	51.33/37.07	9.45/5.98	11.93/11.50	0.22/0.26
Push-T	2D	77.61/72.76	34.28/13.73	22.72/23.06	7.79/7.37

TABLE I : Quantitative results in simulation. Performance denotes success rate or Push-T score. Entries show best-of-final-ten / final-ten summaries; statistical definitions follow Section V-A .

ACT
	Nominal (ID)		Held-out Position (OOD)
Task	SR (%) $\uparrow$	Time (s) $\downarrow$	SR (%) $\uparrow$	Time (s) $\downarrow$
Soap Pick-up	$55.0\pm 11.1$	$12.6\pm 2.9$	$30.0\pm 10.2$	$11.0\pm 3.0$
Plate Stacking	$65.0\pm 10.7$	$28.5\pm 3.5$	$30.0\pm 10.2$	$29.5\pm 4.0$
Toolbox	$50.0\pm 11.2$	$35.5\pm 3.5$	${0.0\pm 0.0}^{\dagger}$	–
ACT- RRDF

TABLE II : Quantitative results on real robots. Success rate (SR, %): observed success rate

\pm

estimated binomial standard error. Time: mean

\pm

sample SD across successful rollouts; “–” denotes no success.

Fig. 6 : Illustrative closed-loop recovery. Red/green arrows show the baseline/ RRDF rollout progression. The red box marks an intermediate placement failure; green boxes highlight RRDF ’s subsequent corrective actions.

Fig. 7 : Phase-matched state-to-image and action sensitivity analysis. (a) State-induced visual change (circles) and state-to-image ratio $\Gamma^{\ell}$ (triangles); background bands mark RRDF’s stages. (b) Action changes $D_{s}$ and $D_{I}$ (bars) and ratio $\Lambda_{A}$ (diamonds). Shading and error bars denote SD across episode means.

Variant	Reg.	Prefix	Topology	Nom. $\uparrow$	$\Gamma^{L}$
(a) Global topology controls
Dense			Dense	$75.3{\pm}3.1$	$1.39{\pm}0.20$
Dense + R	$\checkmark$		Dense	$77.3{\pm}3.1$	$0.96{\pm}0.29$
Delayed Dense		Sep.	Dense	$87.3{\pm}2.3$	$0.81{\pm}0.16$
R-Route	$\checkmark$		Routed	$90.7{\pm}2.3$	$0.39{\pm}0.01$
(b) Pre-routing prefix decomposition

TABLE III : Fusion-topology ablations on Square. Nominal success rate (Nom.): the best-of-final-ten success, mean

\pm

SD. Sensitivity statistics for

\Gamma^{L}

follow Section V-C .

Fig. 8 : Sensitivity to workspace width and routing onset on Square. R $i$ D $j$ denotes $R=i$ registers and $L_{\rm delay}=j$ , with $L=4$ ; D2/D3/D4 use $(L_{\rm iso},L_{\rm col},L_{\rm route})=(1,1,2)/(1,2,1)/(1,3,0)$ . Left: register-count sweep with $L_{\rm delay}=2$ ; right: routing-delay sweep with $R=2$ . D4 is the no-encoder-readback boundary case. Bars show success and gains over dense fusion; lines and shading show mean completion time $\pm$ SD.

	Dense ACT		RRDF
Visual backbone	SR (%) $\uparrow$	Time (s) $\downarrow$	SR (%) $\uparrow$	Time (s) $\downarrow$
ResNet-18	$75.3\pm 3.1$	$10.56\pm 0.68$	$\mathbf{93.3\pm 3.1}$	$9.74\pm 0.25$
DINOv2-Base	$76.7\pm 9.0$	$9.89\pm 2.25$	$\mathbf{88.0\pm 4.0}$	$10.04\pm 2.25$

TABLE IV : Visual-backbone ablation on Square. Best-of-final-ten results, mean

\pm

SD.

Fusion	SR (%) $\uparrow$	Time (s) $\downarrow$
Feature concat	$81.1\pm 3.8$	$7.50\pm 0.27$
Dense fusion	$80.8\pm 3.9$	$7.19\pm 0.16$
RRDF	$\mathbf{84.3\pm 3.3}$	$7.26\pm 0.08$

TABLE V : Diffusion Policy transfer on Square. Success and successful-rollout time: mean ± sample SD across the final five checkpoint means.

Explore similar work

Jun 12, 2026

TRACE: Trajectory-Routed Causal Memory for Delayed-Evidence Visuomotor Imitation

Robots under autonomous operation may require decisions based on evidence that is no longer visible. We study delayed-evidence tasks, where an early cue disappears before a later decision point, so visually similar observations can require different actions. In these settings, the current observation is not a sufficient state for control. We introduce TRAjectory-routed Causal Evidence (TRACE), a memory framework for visuomotor imitation policies. TRACE stores task-relevant visual and robot-state evidence, such as object identity, target choice, or route-dependent state, in a fixed-size latent memory that remains bounded over long episodes. Instead of indexing memory by raw time or manually provided task labels, TRACE uses path signatures: compact, order-sensitive features of the executed robot-state trajectory. These signatures do not store the visual cue itself; rather, they provide trajectory-conditioned keys for writing and retrieving the evidence stored when the cue was visible. When the robot later reaches an ambiguous observation, the policy conditions on TRACE memory to recover the missing context and choose the correct branch. TRACE attaches through lightweight adapters to policies, without changing the policy backbone, action head, or imitation objective. Across real-world long-horizon manipulation tasks with visually ambiguous branch points, TRACE improves branch selection and task success over alternative baselines, including short-history and recurrent memory. Project page: https://jeong-zju.github.io/trace

Zihao Li, Ranpeng Qiu, Yincong Chen +2

School of Computer Science and the Australian Centre for Robotics at the University of Sydney.

Visuomotor Policy Learning Robotic Manipulation Robot Imitation Learning Memory-Augmented Agents

Ask PDF

Oct 9, 2026

Dual-Stream Memory for Vision-Language-Action Policies

Memory-dependent manipulation requires a policy to track execution state and recover visual evidence that is no longer observable. We propose dual-stream memory for vision-language-action policies, assigning these requirements to complementary representations. A task-agnostic Gated DeltaNet compresses history into one temporal token, supervised by progress prediction and action-conditioned feature reconstruction. A task-conditioned retriever retains frames above a fixed relevance threshold; linear multi-scale fusion encodes each frame into sixteen spatial tokens. We then freeze the temporal compressor and jointly optimize retrieval and policy learning. Across sixteen RoboMME tasks, the complete model achieves 49.22% mean success versus 44.51% for the published FrameSamp+Modul reference, improving fourteen tasks and all four suite means. On four real-robot tasks, mean success reaches 62.5% versus 37.5%, with the largest gains on repeated actions requiring execution-stage discrimination. These results support combining execution context with selective visual access, while failures under cup swaps expose persistent identity tracking as an unresolved requirement.

Yang Wang, Ping Liu, Ruiyu Wang +3

Magic-Lab Team, Magiclab Robotics Inc. · University of California, Los Angeles, USA

Memory-Augmented VLMs Visuomotor Policy Learning Vision-Language-Action Models Robotic Manipulation

Ask PDF

Sep 14, 2026

StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation

Recent advances in robot imitation learning have produced visuomotor policies that predict actions directly from visual observations. Yet visually similar scenes can require different actions as target position, object height, or contact geometry changes. Pretrained RGB features may map these geometrically distinct states to similar policy inputs, while simply adding depth requires the policy to learn RGB-depth correspondence from the same limited demonstrations used to learn control. We introduce StereoPatch, a patch-aligned RGB-depth representation that binds registered metric geometry directly to the RGB patches used for action prediction. On a shared 2-D patch grid, asymmetric cross-attention incorporates depth information into the corresponding RGB features before action decoding. The resulting StereoPatch Tokens provide a geometry-aware visual representation that can condition general visuomotor policies without changing their underlying learning objectives. Across six real-robot tasks, StereoPatch achieves higher closed-loop success than appearance-only, geometry-only, raw RGB-D, and late-fusion baselines. Additional experiments across three simulation suites evaluate compatibility across visuomotor policy architectures, spatial generalization, and operating limits. Results suggest that resolving control-relevant geometric ambiguity benefits from aligning depth directly with the visual features used for action prediction, rather than supplying it as an independent modality. Project page: https://aus.bot/research/stereopatch/.

Yanan Zhou, Zhaoyan Qian, James Zhao +1

School of Computer Science, The University of Sydney, Australia. · Australian Centre for Robotics, The University of Sydney, Australia.

Generalization in Robotic Manipulation Visuomotor Policy Learning Robotic Manipulation Multimodal Fusion Robotic Perception

Ask PDF