Organizations: Graduate School of Information Science and Technology, The University of Osaka · D3 Center, The University of Osaka · Graduate School of Maritime Sciences, Kobe University
Bilateral control-based imitation learning captures both position and force information, making it well suited to contact-rich manipulation. However, existing approaches provide limited means for an operator to specify how a learned task should be executed at inference time, such as slowly or quickly, gently or firmly. We propose Bi-MoDe, a modifier-conditioned decoding framework that injects a constrained latent into every layer of the Transformer action decoder via adaLN-Zero, allowing behavioral directives to directly influence action-chunk generation. We evaluate the method on a real-world whiteboard wiping task with combinations of temporal and physical modifiers. Bi-MoDe improves physical directive following over the action-chunking baseline while maintaining comparable temporal control. An ablation further shows that decoder conditioning and latent-space composition interact, and that their combination is important for accurate physical directive following. Additional material is available at the https://mertcookimg.github.io/bi-mode/
Figures & tables
Fig. 1: Concept of Bi-MoDe
Fig. 2: Overview of Bi-MoDe. The constrained latent zc is appended as a token to the Transformer encoder input and, through adaLN-Zero, modulates every layer of the action decoder.
Fig. 3: One encoder layer and one decoder layer. A zero-initialized MLP regresses the scale, shift and gate of all three decoder sub-layers from zc .
Fig. 4: The follower robot, with Joint2 and Joint3 highlighted.
Fig. 5: Effect of the temporal modifier in the demonstrations. Rows are fast, moderate and slow from top to bottom, with frames taken at equal elapsed time: over six seconds the three conditions complete three, one and a half, and one stroke.
Fig. 6: Effect of the physical modifier in the demonstrations. Rows are weak, moderate and strong, with columns showing the start, forward and backward phases of one stroke. The arm follows the same path at every level, while the Joint2 and Joint3 torques below separate by commanded force.
Parameter
Value
Encoder layers
4
Decoder layers
4
Hidden dimension
512
Feedforward dimension
2048
Attention heads
8
Latent dimension ( zc / zu )
2 / 1
TABLE I: Training configuration, identical across the four configurations except for the two ablated factors.
Fig. 7: Cycle duration versus commanded speed (top) and contact torque versus commanded force (bottom). Columns are the four configurations of Table II from left to right, each with the reference line fitted to the demonstrations.
Fig. 8: Joint2 and Joint3 profiles, with the demonstrations on the left and Bi-MoDe on the right. Angles are shown for the three commanded speed levels and torques for the three commanded force levels.
Configuration
Design
TSR
Physical
Temporal
Decoder cond.
zu
MDE
slope
MDE
slope
Baseline (ACT-based) [ 10 ]
–
✓
45/45
0.123
0.896
0.161
1.104
No zu
–
–
45/45
0.300
0.754
0.093
1.069
Decoder conditioned
✓
✓
45/45
0.139
0.854
0.039
0.959
Bi-MoDe (Ours)
✓
–
45/45
0.036
0.968
0.116
1.096
TABLE II: Task success rate and directive-following performance. MDE ↓ ; slope ratio →1.0 .
CMDIR extends Manifold-Decomposed Impedance Retargeting (MDIR) to transform fixed-impedance demonstrations into continuous variable-impedance controllers, which can also serve as structured supervision for imitation learning. Continuous Task-Manifold Impedance Representation (TMIR) pairs an evolving task frame with controller instructions. Demo-relative Compromise dynamics retain moving-basis transport and control/physical metric mismatch, yielding displacement, reaction-impulse, and perturbation-sensitivity criteria. Quality-to-Fast automatically compiles a solver structure from development paths within a predefined finite space, re-instantiates that structure for each demonstration, and certifies the resulting candidate by multi-resolution evaluation. Across 225 retargeted-controller trials in three real contact tasks, full CMDIR improves mean task-proxy retention and reduces mean pose deviation, force fluctuation, and peak force relative to discrete MDIR. FastMPO achieves a 5.8--9.4× speedup over C-MPO with comparable closed-loop outcomes. Downstream experiments demonstrate learnability of the complete TMIR supervision interface; lower force fluctuation and peak force are observed among successful executions, while completion reliability remains uneven across tasks and environments.
Jiahao Liu, Kento Kawaharazuka, Tasuku Makabe +1
Department of Mechano-Informatics Graduate School of Information Science and Technology, The University of Tokyo Tokyo, Japan
To operate effectively across diverse contexts, robots must not only perform manipulation tasks accurately but also adapt how their actions unfold to the task, object, and interaction setting. We ask whether this execution-level variation can be learned as a reusable behavioral factor shared across tasks. We present MoMo, a two-stage imitation-learning framework consisting of a spatiotemporal action tokenizer and a behavior-cloning transformer that takes task and a continuous motion-mode condition as inputs. Across six real-robot manipulation tasks, varying this condition produces steady, dynamic, and intermediate behaviors that human raters can distinguish and that differ in joint speed, acceleration, and end-effector approach pitch. On tasks demonstrated in only one mode, MoMo transfers the unseen requested mode while largely preserving task success. Together, these results provide evidence of compositional generalization to unseen task--mode combinations and show that motion mode can be reused across tasks to control how a manipulation skill is performed.
Long-horizon contact-rich robotic manipulation remains challenging due to partial observability and unstable subtask transitions under contact uncertainty. While hierarchical architectures improve temporal reasoning and bilateral imitation learning enables force-aware control, existing approaches often rely on flat policies that struggle with long-horizon coordination. We propose Bi-HIL, a bilateral control-based multimodal hierarchical imitation learning framework for long-horizon manipulation. Bi-HIL stabilizes hierarchical coordination by integrating keyframe memory with subtask-level progress rate that models phase progression within the active subtask and conditions both high- and low-level policies. We evaluate Bi-HIL on unimanual and bimanual real-robot tasks, demonstrating consistent improvements over flat and ablated variants. The results highlight the importance of explicitly modeling subtask progression together with force-aware control for robust long-horizon manipulation. For additional material, please check: https://mertcookimg.github.io/bi-hil