Learn the Task, Not the Schedule: Reducing Spurious Cross-Arm Coupling in Bimanual Imitation Learning

Shuyuan Hu; Deng-Ke Wu; Yifan Jin; Qianhang Luo; Mingchen Ju; Kai Ye; Bo Zhao

cs.ROOct 19, 2026

Learn the Task, Not the Schedule: Reducing Spurious Cross-Arm Coupling in Bimanual Imitation Learning

Authors: Shuyuan Hu, Deng-Ke Wu, Yifan Jin, Qianhang Luo, Mingchen Ju, Kai Ye, Bo Zhao

Organizations: School of AI, Shanghai Jiao Tong University · Central South University · South China Normal University · University of Sydney

Abstract

A bimanual demonstration specifies both what the robot should do and one particular schedule for when each arm should act. Standard imitation learning does not distinguish task-required cross-arm constraints from incidental scheduling choices, and may therefore exploit opposite-arm progress as a shortcut cue, creating spurious cross-arm coupling even where relative timing is irrelevant for the current task phase. We introduce Constraint-preserving Temporal Recombination (CTR), a training-time data intervention that preserves recorded arm-local sensorimotor content while retiming the corresponding segments to realize admissible schedules that preserve task-required cross-arm constraints. We use CTR to train Diffusion Policy, Action Chunking with Transformers, and $π_{0.5}$ , and evaluate them on four simulated and four real-robot tasks. Across six real-robot bimanual progress-state initializations, CTR raises worst-state success from 56% for the strongest non-CTR schedule mixture to 76%. With $π_{0.5}$ on an Independent task, a 4 s pause transfers 2.56 s of delay to the opposite arm under the Concurrent schedule distribution, but only 0.25 s with CTR. On real robots, CTR violates task constraints in only 0--4% of trials, versus 36--98% for narrow-schedule baselines. Code will be released at https://github.com/MINT-SJTU/CTR.

Figures & tables

Figure 1: CTR data-processing pipeline. Foreground segmentation supports observation composition, and temporal recombination realizes alternative admissible schedules while preserving arm-local sensorimotor content and task-required cross-arm constraints. The displayed masks and sampled schedules are schematic.

Figure 2: The eight evaluation tasks span Independent, Shared-Workspace Exclusion, Coordination, and Prerequisite relations, testing both invariance where relative timing is irrelevant and responsiveness where cross-arm constraints matter. Columns group tasks by relation type; the top and bottom rows show simulated and real-robot tasks, respectively, with two representative frames per task.

Figure 3: The five training distributions vary relative arm timing: the baselines concentrate on one or more canonical schedules, whereas CTR covers a broader range of admissible timings. Histograms show onset-difference densities from Sort Blocks simulation trajectories, with

\Delta_{\mathrm{start}}=t_{R}^{\rm start}-t_{L}^{\rm start}>0

indicating that the left arm starts first; vertical density scales differ across panels. Curves show smoothed density estimates except for Concurrent; smoothing-induced tails should not be interpreted as additional observed support. Appendix 0 gives construction and measurement details.

Figure 4: Phase-conditioned action sensitivity on Deliver Water for ACT, DP, and $\pi_{0.5}$ . We measure changes in one arm’s predicted actions while keeping its state and observations fixed and varying the opposite arm’s state and observations. Curves show the mean action difference for samples within each time interval. CTR reduces sensitivity during the independent phases while retaining responsiveness during Serve Water . Appendix 0 shows the original scatter plots for all three models.

Task	Data	C00	Cmid	L-lead	R-lead	CL	CR	Worst SR
Put Object Cabinet (sim)	$L\!\rightarrow\!R$	40 / 16.4	56 / 3.8	74 / 6.4	0 / –	60 / 9.1	2 / 8.1	0
	Serial	36 / 16.9	90 / 3.6	76 / 6.4	30 / 8.8	52 / 9.3	44 / 10.6	30
	Concurrent	22 / 12.1	98 / 2.9	0 / –	0 / –	0 / –	0 / –	0
	Mixed	54 / 10.1	100 / 3.0	64 / 7.0	50 / 7.4	48 / 10.1	80 / 7.8	48
	CTR (ours)	66 / 11.0	100 / 3.2	72 / 6.1	62 / 11.3	56 / 8.7	96 / 6.6	56
Sort Letters (real)	$L\!\rightarrow\!R$	90 / 32.1	82 / 29.1	36 / 48.7	72 / 39.3	92 / 22.2	98 / 18.0	36

Table 1: Bimanual progress-state robustness for

\pi_{0.5}

on the Put Object Cabinet task from the RoboTwin 2.0 benchmark ( Chen et al., 2026 ) and on Sort Letters on the real robot. During phases in which the two arms’ subtasks are independent, we expect policy performance to remain unchanged across initializations with different relative progress between the arms. Each measured-state cell (

n=50

) reports SR (%) /

T_{\rm Pen}

(s). When SR=0,

T_{\rm Pen}

is undefined and shown as “–” because its failure penalty depends on the cell’s mean successful completion time. Cell tint is determined solely by SR, with darker blue indicating higher success rates. Worst SR is the minimum over the six states. Appendix 0 specifies the penalty.

Table 6

Task	Baseline	Violation (%)		Release latency (s)
Task	Baseline	Baseline	CTR	Baseline	CTR
Put Object Cabinet (real)	$L\!\rightarrow\!R$	98	0	0.01	0.27
Organize Hardware	Concurrent	36	4	0.09	0.15

Table 3: Task-constraint interventions with

\pi_{0.5}

on two real-robot tasks (

n=50

). Put Object Cabinet compares

L\!\rightarrow\!R

with CTR; Organize Hardware compares Concurrent with CTR. Lower is better for both violation rate and release latency; CTR lowers violations but has longer release latencies.

Task	DP		ACT		$\pi_{0.5}$
Task	Serial	CTR	Serial	CTR	Serial	CTR
Sort Blocks	70	72	90	92	94	94
Place Dual Shoes	0	0	26	20	38	38
Scan Object	0	2	4	10	62	70
Put Object Cabinet (sim)	2	4	8	24	36	66
Sort Letters	0	2	12	18	92	94

Table 4: Default-condition SRs (%) for DP, ACT, and

\pi_{0.5}

trained on Serial and CTR data. The first four rows are simulated tasks and the last four are real-robot tasks. Each entry is task-level SR (

n=50

per task, model, and training condition).

Figure 5: Opposite-arm pause on Independent Sort Letters with

\pi_{0.5}

. Delay transfer is the unpaused arm’s extra completion time relative to a matched no-pause control. Numbers show means (s); faint bands show pointwise 95% confidence intervals (50 trials per condition and duration). Straight segments join only the four tested durations.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	Released	Reschedulable	Diversity-covered	Locked
Dataset	Domain	tasks	tasks	tasks	(%)
RoboTwin 2.0	Simulation	50	12	0	100.0
RoboCOIN	Real	639	231	0	100.0
DexMimicGen	Simulation	9	8	6	25.0
BiCoord	Simulation	18	15	3	80.0
Pooled		716	266	9	96.6

Appendix

Table 6: Dataset size and schedule coverage for the surveyed bimanual datasets. Locked is the percentage of reschedulable tasks demonstrated under a single observed schedule.

ID	Domain	Task	Class
S1	Sim.	Sort Blocks	Independent
S2	Sim.	Place Dual Shoes	Shared-Workspace Exclusion
S3	Sim.	Scan Object	Coordination
S4	Sim.	Put Object Cabinet	Prerequisite
R1	Real	Sort Letters	Independent
R2	Real	Organize Hardware	Shared-Workspace Exclusion

Appendix

Table 7: Task inventory used in the experiments.

Figure 6: Initial-scene setups for the six progress-state evaluations. The first two rows show Put Object Cabinet in simulation; the last two rows show Sort Letters on the real robot. Each panel is labeled by its setup name and normalized bimanual progress state.

Panel of Figure 7 — Figure 7: Idealized schedule geometry. Top: idealized bimanual progress-state support with left-arm progress on the horizontal axis and right-arm progress on the vertical axis. Arrows indicate increasing progress. Axis padding separates the trajectories from the axes without changing their shapes; in the top-row CTR panel, the two interior paths nearest the serial envelope are omitted for clarity. Bottom: the corresponding schedule families as arm progress over independently normalized time; solid teal and dashed orange denote the left and right arms, respectively. Both rows are schematic.

Source	$L\!\rightarrow\!R$	Concurrent	$R\!\rightarrow\!L$
Demos average	17.10	0.00	$-17.22$
DP	14.74	0.05	$-15.15$
ACT	20.31	$-0.04$	$-19.78$
$\pi_{0.5}$	14.52	0.02	$-14.43$

Appendix

Table 8: Schedule inheritance on Sort Letters. Policy entries are mean

\Delta_{\rm start}=t_{R}^{\rm start}-t_{L}^{\rm start}

in seconds. The Demos average row reports dataset-average demonstration references:

+17.10

s for

L\!\rightarrow\!R

(mean left-arm task duration),

0.00

s for Concurrent, and

-17.22

s for

R\!\rightarrow\!L

(negative mean right-arm task duration).

Model	Training data	Full chunk	First step	First 8 steps	Phase-contained full chunk
ACT	CTR	$+2.26$	$+3.32$	$+3.16$	$+2.22$
ACT	Concurrent	$-5.71$	$-5.12$	$-5.62$	$-6.35$
DP	CTR	$+1.60$	$+1.14$	$+1.60$	$+1.60$
DP	Concurrent	$-3.22$	$-3.06$	$-3.22$	$-3.22$
$\pi_{0.5}$ 20k	CTR	$+1.11$	$+0.33$	$+0.61$	$+1.05$
$\pi_{0.5}$ 20k	Concurrent	$-1.22$	$+0.11$	$0.00$	$-1.63$

Appendix

Table 9: Normalized phase-selectivity gap

G_{\rm sel}

in degree-equivalent units. Gripper differences are scaled by

0.7609

degree-equivalent/mm using median

q_{90}-q_{10}

spans of

42.9902^{\circ}

for joints and

56.5000

mm for grippers. A positive value means sensitivity is higher during Serve Water than the mean of Place Bottles and Return to Standby . The phase-contained column excludes chunks that extend beyond the sampled phase boundary; the first-step and first-eight-step columns include all sampled groups.

Figure 8: Phase-conditioned cross-arm sensitivity for ACT checkpoints trained on Concurrent data (left) and CTR data (right). Yellow circles measure right-state-to-left-Action sensitivity; blue triangles measure left-state-to-right-Action sensitivity. The two outer regions are labeled Independent and the middle region is Synchronized.

Figure 9: Phase-conditioned cross-arm sensitivity for DP checkpoints trained on Concurrent data (left) and CTR data (right). Yellow circles measure right-state-to-left-Action sensitivity; blue triangles measure left-state-to-right-Action sensitivity. The two outer regions are labeled Independent and the middle region is Synchronized.

Figure 10: Phase-conditioned cross-arm sensitivity for

\pi_{0.5}

20k checkpoints trained on Concurrent data (left) and CTR data (right). Yellow circles measure right-state-to-left-Action sensitivity; blue triangles measure left-state-to-right-Action sensitivity. The two outer regions are labeled Independent and the middle region is Synchronized.

Robot Imitation Learning

Robot imitation learning trains robots to perform tasks by reproducing and adapting behaviors shown in demonstrations.

See all 680 papers

RoTri-Diff: A Spatial Robot-Object Triadic Interaction-Guided Diffusion Model for Bimanual Manipulation
Mar 7, 2026Zixuan Chen, Nga Teng Chan, Yiwen Hou +8Bimanual Robotic Manipulation Robotic Manipulation
Zixuan Chen and Jing Huo are with the School of Computer Science, Nanjing University, China. · Yiwen Hou, Chenrui Tie, Zixuan Liu, Haonan Chen, Junting Chen and Lin Shao are with the School of Computing, National University of Singapore, Singapore. · Nga Teng Chan is with the Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, China. +1
Tri-Manual Visuomotor Imitation Learning of Robot Policies
Jul 28, 2026James Zhao, Mingyuan Ba, Weiming ZhiRobot Teleoperation Visuomotor Policy Learning
School of Computer Science and 2Australian Center For Robotics, The University of Sydney, Australia · College of Connected Computing, Vanderbilt University, TN, USA
RESTA: Time-Scaling Augmentation for Data-Efficient Robot Imitation Learning
Oct 13, 2026Yunpeng Jiang, Jianshu Hu, Han Fang +3Data-Efficient Robot Learning Robot Imitation Learning
Global College, Shanghai Jiao Tong University, Shanghai, China · Shanghai Institute of Physical Intelligence and Robotics (PAIR), Shanghai, China · Duke Kunshan University, Jiangsu, China

Mar 7, 2026J/K move · Enter open · S save

RoTri-Diff: A Spatial Robot-Object Triadic Interaction-Guided Diffusion Model for Bimanual Manipulation

Zixuan Chen, Nga Teng Chan, Yiwen Hou, Chenrui Tie+7

Zixuan Chen and Jing Huo are with the School of Computer Science, Nanjing University, China. · Yiwen Hou, Chenrui Tie, Zixuan Liu, Haonan Chen, Junting Chen and Lin Shao are with the School of Computing, National University of Singapore, Singapore. · Nga Teng Chan is with the Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, China.+1

Bimanual Robotic Manipulation Robotic Manipulation Robot Imitation Learning Motion Representation Learning

Open paper AskPDF arXiv

Bimanual manipulation is a fundamental robotic skill that requires continuous and precise coordination between two arms. While imitation learning (IL) is the dominant paradigm for acquiring this capability, existing approaches, whether robot-centric or object-centric, often overlook the dynamic geometric relationship among the two arms and the manipulated object. This limitation frequently leads to inter-arm collisions, unstable grasps, and degraded performance in complex tasks. To address this, in this paper we explicitly models the Robot-Object Triadic Interaction (RoTri) representation in bimanual systems, by encoding the relative 6D poses between the two arms and the object to capture their spatial triadic relationship and establish continuous triangular geometric constraints. Building on this, we further introduce RoTri-Diff, a diffusion-based imitation learning framework that combines RoTri constraints with robot keyposes and object motion in a hierarchical diffusion process. This enables the generation of stable, coordinated trajectories and robust execution across different modes of bimanual manipulation. Extensive experiments show that our approach outperforms state-of-the-art baselines by 10.2% on 11 representative RLBench2 tasks and achieves stable performance on 4 challenging real-world bimanual tasks. Project website: https://rotri-diff.github.io/.

Abstract

Robot Imitation Learning

Explore similar work

RoTri-Diff: A Spatial Robot-Object Triadic Interaction-Guided Diffusion Model for Bimanual Manipulation

Tri-Manual Visuomotor Imitation Learning of Robot Policies

RESTA: Time-Scaling Augmentation for Data-Efficient Robot Imitation Learning