Register-Routed Delayed Fusion: Rewiring Shortcut-Prone Observation Fusion in Visuomotor Imitation

Jieting Long; Weidong Cai; Weiming Zhi

cs.ROOct 2, 2026

Register-Routed Delayed Fusion: Rewiring Shortcut-Prone Observation Fusion in Visuomotor Imitation

Authors: Jieting Long, Weidong Cai, Weiming Zhi

Organizations: School of Computer Science, The University of Sydney, Australia.

Abstract

Visuomotor imitation policies combine high-dimensional visual observations with compact signals such as proprioception, and their fusion topology determines when and through which tokens these streams interact. In dense token fusion, visual tokens may attend directly to compact tokens from the first encoder layer, allowing action-predictive compact cues to influence spatial visual representations early in their formation. We ask whether controlling this route improves visual responsiveness and policy behavior. We introduce Register-Routed Delayed Fusion (RRDF), which masks direct compact-visual attention and stages cross-modal interaction through a learned register workspace. Its isolate-collect-route schedule protects an early stream-separated prefix and later permits only register-mediated exchange, while compact conditioning remains available to the native action generator. Across five simulation tasks and three real-robot tasks, RRDF matches or improves dense ACT under nominal conditions. Appearance-shift evaluations on four simulation tasks and held-out-position evaluations on three real-robot tasks also favor RRDF. Phase-matched input probes show lower measured state-to-image sensitivity, while ablations indicate that adding registers alone does not reproduce the full performance gain. These results support controlling cross-modal propagation while retaining compact action conditioning.

Figures & tables

Fig. 1 : Representative behavioral difference under a held-out object-position shift. The dense-fusion policy produces similar trajectories for seen and shifted object configurations, failing to accommodate the shifted scene. RRDF instead adapts its trajectory and completes the task, illustrating improved scene responsiveness under the proposed delayed-fusion topology.

Fig. 2 : Overview of RRDF . Dense fusion permits direct compact–visual attention from the first encoder layer. RRDF instead uses source-specific register accumulation, joint collection without primary-stream readback, and later register-mediated feedback. The outer panels show representative dense-fusion failure and RRDF success under a held-out-position shift.

Fig. 3 : Depth-dependent directed topology of RRDF . Left to right: dense fusion, source-specific register collection during Isolation, joint workspace formation during Collection, and register-mediated feedback during Routing. Direct

\mathcal{C}\leftrightarrow\mathcal{V}

attention remains masked throughout; arrows denote permitted source-to-destination information flow.

Panel of Figure 4 — Fig. 4 : Controlled appearance-shift stress tests in simulation. (a) Nominal, bright-floor, and recolored observations for each task (left to right); geometry and dynamics are unchanged. (b) Closed-loop success under each shift. Solid bars show dense-fusion success, hatched extensions the RRDF gain, and total height the RRDF result.

Fig. 5 : Qualitative real-robot behaviours. Matched dense-fusion and RRDF rollouts illustrate Spatial Adaptation on Soap Pick-up (top), Precision Alignment on Plate Stacking (bottom-left), and Precision Placement on Toolbox (bottom-right), under nominal and held-out-position conditions. Red/green boxes mark representative dense-fusion failures and RRDF successes.

ACT Baseline
Task	Action	Perf. (%) $\uparrow$	SD.	Time (s) $\downarrow$	SD.
Can	joint	99.33 /96.67	1.15/1.44	7.08/7.08	0.08/0.02
Square	joint	75.33/73.53	3.06/2.78	10.56/10.49	0.68/0.15
Stack-D1	EEF	78.67/66.33	2.31/4.92	6.11/6.17	0.43/0.27
Coffee-D2	EEF	51.33/37.07	9.45/5.98	11.93/11.50	0.22/0.26
Push-T	2D	77.61/72.76	34.28/13.73	22.72/23.06	7.79/7.37

TABLE I : Quantitative results in simulation. Performance denotes success rate or Push-T score. Entries show best-of-final-ten / final-ten summaries; statistical definitions follow Section V-A .

ACT
	Nominal (ID)		Held-out Position (OOD)
Task	SR (%) $\uparrow$	Time (s) $\downarrow$	SR (%) $\uparrow$	Time (s) $\downarrow$
Soap Pick-up	$55.0\pm 11.1$	$12.6\pm 2.9$	$30.0\pm 10.2$	$11.0\pm 3.0$
Plate Stacking	$65.0\pm 10.7$	$28.5\pm 3.5$	$30.0\pm 10.2$	$29.5\pm 4.0$
Toolbox	$50.0\pm 11.2$	$35.5\pm 3.5$	${0.0\pm 0.0}^{\dagger}$	–
ACT- RRDF

TABLE II : Quantitative results on real robots. Success rate (SR, %): observed success rate

\pm

estimated binomial standard error. Time: mean

\pm

sample SD across successful rollouts; “–” denotes no success.

Fig. 6 : Illustrative closed-loop recovery. Red/green arrows show the baseline/ RRDF rollout progression. The red box marks an intermediate placement failure; green boxes highlight RRDF ’s subsequent corrective actions.

Fig. 7 : Phase-matched state-to-image and action sensitivity analysis. (a) State-induced visual change (circles) and state-to-image ratio $\Gamma^{\ell}$ (triangles); background bands mark RRDF’s stages. (b) Action changes $D_{s}$ and $D_{I}$ (bars) and ratio $\Lambda_{A}$ (diamonds). Shading and error bars denote SD across episode means.

Variant	Reg.	Prefix	Topology	Nom. $\uparrow$	$\Gamma^{L}$
(a) Global topology controls
Dense			Dense	$75.3{\pm}3.1$	$1.39{\pm}0.20$
Dense + R	$\checkmark$		Dense	$77.3{\pm}3.1$	$0.96{\pm}0.29$
Delayed Dense		Sep.	Dense	$87.3{\pm}2.3$	$0.81{\pm}0.16$
R-Route	$\checkmark$		Routed	$90.7{\pm}2.3$	$0.39{\pm}0.01$
(b) Pre-routing prefix decomposition

TABLE III : Fusion-topology ablations on Square. Nominal success rate (Nom.): the best-of-final-ten success, mean

\pm

SD. Sensitivity statistics for

\Gamma^{L}

follow Section V-C .

Fig. 8 : Sensitivity to workspace width and routing onset on Square. R $i$ D $j$ denotes $R=i$ registers and $L_{\rm delay}=j$ , with $L=4$ ; D2/D3/D4 use $(L_{\rm iso},L_{\rm col},L_{\rm route})=(1,1,2)/(1,2,1)/(1,3,0)$ . Left: register-count sweep with $L_{\rm delay}=2$ ; right: routing-delay sweep with $R=2$ . D4 is the no-encoder-readback boundary case. Bars show success and gains over dense fusion; lines and shading show mean completion time $\pm$ SD.

	Dense ACT		RRDF
Visual backbone	SR (%) $\uparrow$	Time (s) $\downarrow$	SR (%) $\uparrow$	Time (s) $\downarrow$
ResNet-18	$75.3\pm 3.1$	$10.56\pm 0.68$	$\mathbf{93.3\pm 3.1}$	$9.74\pm 0.25$
DINOv2-Base	$76.7\pm 9.0$	$9.89\pm 2.25$	$\mathbf{88.0\pm 4.0}$	$10.04\pm 2.25$

TABLE IV : Visual-backbone ablation on Square. Best-of-final-ten results, mean

\pm

SD.

Fusion	SR (%) $\uparrow$	Time (s) $\downarrow$
Feature concat	$81.1\pm 3.8$	$7.50\pm 0.27$
Dense fusion	$80.8\pm 3.9$	$7.19\pm 0.16$
RRDF	$\mathbf{84.3\pm 3.3}$	$7.26\pm 0.08$

TABLE V : Diffusion Policy transfer on Square. Success and successful-rollout time: mean ± sample SD across the final five checkpoint means.

Visuomotor Policy Learning

Visuomotor policy learning trains robots to turn camera observations, often with language and proprioception, into actions for manipulation and navigation.

See all 275 papers

TRACE: Trajectory-Routed Causal Memory for Delayed-Evidence Visuomotor Imitation
Jun 12, 2026Zihao Li, Ranpeng Qiu, Yincong Chen +2Visuomotor Policy Learning Robotic Manipulation
School of Computer Science and the Australian Centre for Robotics at the University of Sydney.
Dual-Stream Memory for Vision-Language-Action Policies
Oct 9, 2026Yang Wang, Ping Liu, Ruiyu Wang +3Memory-Augmented VLMs Visuomotor Policy Learning
Magic-Lab Team, Magiclab Robotics Inc. · University of California, Los Angeles, USA
StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation
Sep 14, 2026Yanan Zhou, Zhaoyan Qian, James Zhao +1Generalization in Robotic Manipulation Visuomotor Policy Learning
School of Computer Science, The University of Sydney, Australia. · Australian Centre for Robotics, The University of Sydney, Australia.

Jun 12, 2026J/K move · Enter open · S save

TRACE: Trajectory-Routed Causal Memory for Delayed-Evidence Visuomotor Imitation

Zihao Li, Ranpeng Qiu, Yincong Chen, Guoqiang Ren+1

School of Computer Science and the Australian Centre for Robotics at the University of Sydney.

Visuomotor Policy Learning Robotic Manipulation Robot Imitation Learning Memory-Augmented Agents

Open paper AskPDF arXiv

Robots under autonomous operation may require decisions based on evidence that is no longer visible. We study delayed-evidence tasks, where an early cue disappears before a later decision point, so visually similar observations can require different actions. In these settings, the current observation is not a sufficient state for control. We introduce TRAjectory-routed Causal Evidence (TRACE), a memory framework for visuomotor imitation policies. TRACE stores task-relevant visual and robot-state evidence, such as object identity, target choice, or route-dependent state, in a fixed-size latent memory that remains bounded over long episodes. Instead of indexing memory by raw time or manually provided task labels, TRACE uses path signatures: compact, order-sensitive features of the executed robot-state trajectory. These signatures do not store the visual cue itself; rather, they provide trajectory-conditioned keys for writing and retrieving the evidence stored when the cue was visible. When the robot later reaches an ambiguous observation, the policy conditions on TRACE memory to recover the missing context and choose the correct branch. TRACE attaches through lightweight adapters to policies, without changing the policy backbone, action head, or imitation objective. Across real-world long-horizon manipulation tasks with visually ambiguous branch points, TRACE improves branch selection and task success over alternative baselines, including short-history and recurrent memory. Project page: https://jeong-zju.github.io/trace

Abstract

Visuomotor Policy Learning

Explore similar work

TRACE: Trajectory-Routed Causal Memory for Delayed-Evidence Visuomotor Imitation

Dual-Stream Memory for Vision-Language-Action Policies

StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation