Adapt as You Say: Online Interactive Bimanual Skill Adaptation via Human Language Feedback

Zhuo Li; Dianxi Li; Tao Teng; Quentin Rouxel; Zhipeng Dong; Dennis Hong; Darwin Caldwell; Fei Chen

cs.ROMar 27, 2026

Adapt as You Say: Online Interactive Bimanual Skill Adaptation via Human Language Feedback

Authors: Zhuo Li, Dianxi Li, Tao Teng, Quentin Rouxel, Zhipeng Dong, Dennis Hong, Darwin Caldwell, Fei Chen

Organizations: Collaborative and Versatile Robots (CLOVER) Laboratory, T-Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong · Department of Mechanical and Aerospace Engineering, University of California, Los Angeles, USA · Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy

Abstract

Developing general-purpose robots capable of autonomously operating in human living environments requires the ability to adapt to continuously evolving task conditions. However, adapting high-dimensional coordinated bimanual skills to novel task variations at deployment remains a fundamental challenge. In this work, we present BiSAIL (Bimanual Skill Adaptation via Interactive Language), a novel framework that enables zero-shot online adaptation of offline-learned bimanual skills through interactive language feedback. The key idea of BiSAIL is to adopt a hierarchical reason-then-modulate paradigm, which first infers generalized adaptation objectives from multimodal task variations, and then adapts bimanual motions via diffusion modulation to achieve the inferred objectives. Extensive real-robot experiments across six bimanual tasks and two dual-arm platforms demonstrate that BiSAIL significantly outperforms existing methods in human-in-the-loop adaptability, task generalization and cross-embodiment scalability. This work enables the development of adaptive bimanual assistants that can be flexibly customized by non-expert users via intuitive verbal corrections. Experimental videos and code are available at https://rip4kobe.github.io/BiSAIL/.

Figures & tables

Fig. 1: Illustration of Online Interactive Bimanual Skill Adaptation. (a) Offline-learned bimanual skills often encounter diverse task variations when deployed in human-centric environments. (b) BiSAIL enables online interactive adaptation of learned bimanual skills through human language feedback, facilitating zero-shot generalization to unseen task variations.

Fig. 2: An Overview of BiSAIL. (1) High-level Adaptation Objective Reasoning: ESA-CoT infers a structured bimanual adaptation objective from multimodal task variations; (2) Mid-level Bimanual Motion Modulation: Initial motion proposals sampled from BMP are first iteratively optimized toward the adaptation objective, and then modulated through compositional sampling to enforce dual-arm coordination and task compatibility; (3) Low-level Skill Adaptation Refinement: A closed-loop reflection mechanism evaluates post-adaptation outcomes to refine both the adaptation objective and the resulting bimanual motion.

Fig. 3: Problem Formulation. We formulate the online interactive bimanual skill adaptation as a constrained probabilistic optimization problem.

Fig. 4: Bimanual Skill Adaptation Objective Reasoning. ESA-CoT enables the objective reasoning through a structured chain-of-thought, augmented with robot embodied information and bimanual domain knowledge.

Fig. 5: Bimanual Motion Prior Learning. BMP is implemented as an unconditional diffusion model with Transformer encoder-only architecture.

Fig. 6: BMP Dataset Generation. Self-supervised bimanual motion data generation pipeline for training BMP.

q(\tau)=\frac{\exp\left(\zeta R(\tau;l,o)\right)}{\sum_{i=1}^{M}\exp\left(\zeta R(\tau^{i};l,o)\right)}

Algorithm 1 Online Bimanual Motion Modulation

Fig. 7: Bimanual Motion Modulation via Compositional Sampling. We employ MCMC sampling to optimize the joint motion distribution over adaptation objective, task-specific policy and coordination constraint.

Fig. 8: Experimental Setup. (a) Overview of the dual-arm robot platform. (b) Evaluation tasks comprising six bimanual scenarios.

Fig. 9: Qualitative Results of Adaptation Objective Reasoning. ESA-CoT infers feasible rewards and coordination across diverse task variations.

Method	RA (%) $\uparrow$	ER (%) $\uparrow$	ROUGE-L $\uparrow$
Few-shot Prompting	25.3 $\pm$ 1.2	17.6 $\pm$ 1.5	N/A
Vanilla CoT	44.5 $\pm$ 0.6	38.7 $\pm$ 0.4	0.55 $\pm$ 0.08
ESA-COT (w/o UIU)	85.7 $\pm$ 0.6	79.4 $\pm$ 0.7	0.88 $\pm$ 0.05
ESA-COT (w/o TCA)	80.1 $\pm$ 0.7	75.7 $\pm$ 0.2	0.77 $\pm$ 0.03
ESA-COT (w/o EA)	58.7 $\pm$ 0.5	49.6 $\pm$ 1.1	0.68 $\pm$ 0.06
ESA-COT (w/o AOR)	75.4 $\pm$ 0.7	69.7 $\pm$ 0.9	0.67 $\pm$ 0.07

TABLE I: Quantitative results of adaptation objective reasoning

Method	FID $\downarrow$	Diversity $\uparrow$	Smoothness $\downarrow$	Time (s)
U-Net	20.61 $\pm$ 0.07	0.58	0.04 $\pm$ 0.01	0.10
Transformer (Dec-only)	13.54 $\pm$ 0.13	0.73	0.12 $\pm$ 0.03	0.27
Transformer (Enc-Dec)	3.21 $\pm$ 0.05	0.94	0.03 $\pm$ 0.04	0.51
BMP (w/o auxiliary)	3.73 $\pm$ 0.02	0.91	0.09 $\pm$ 0.03	0.16
BMP (Ours)	2.48 $\pm$ 0.02	0.93	0.02 $\pm$ 0.06	0.16

TABLE II: Quantitative results of BMP evaluation

Fig. 10: Qualitative Results of Online Bimanual Motion Modulation. Left: Task variation and original bimanual motion distribution. Right: Comparison of modulated motions from various baselines. Our method (h) achieves the best balance of intent alignment, task compatibility, and dual-arm coordination.

Fig. 11: Qualitative Results of BMP Evaluation. (a)–(c): Motions sampled from task-specific DP under different settings. (d)–(e): Motions sampled from BMP architectural variants. (f): Our BMP generates motion proposals that balance spatial diversity, temporal smoothness, and sampling efficiency.

Fig. 12: Comparison of MCMC and CG sampling. Bimanual motions are projected to 2D using PCA and visualized via KDE [ 36 ] . (a) Original component distributions. (b) CG sampling produces OOD motions. (c) Our MCMC sampling yields motions near the intersection of all components, avoiding distribution shift.

Fig. 13: Real Robot Experiment Results. BiSAIL demonstrates flexible and generalizable online interactive bimanual skill adaptation across diverse task variations, manipulation scenarios and coordination categories.

Fig. 14: Comparison with End-to-End Adaptation Baseline. (a) In ID settings, BiSAIL and YAY show comparable IA error. (b) In OOD settings, YAY shows a marked increase in IA error, while BiSAIL maintains low error.

Method	IA (%) $\uparrow$	TA (%) $\uparrow$	CS (%) $\uparrow$
Post-hoc Ranking	53.2 $\pm$ 1.1	40.5 $\pm$ 1.4	38.9 $\pm$ 2.2
BMM (w/ MPPI)	65.4 $\pm$ 1.7	68.2 $\pm$ 1.5	63.6 $\pm$ 2.8
BMM (w/ CEM)	71.6 $\pm$ 1.3	72.4 $\pm$ 2.1	70.2 $\pm$ 1.2
BMM (w/o MCMC)	76.5 $\pm$ 2.4	55.7 $\pm$ 2.6	58.3 $\pm$ 2.2
BMM (w/ CG, g=300)	78.9 $\pm$ 1.5	75.8 $\pm$ 1.3	72.1 $\pm$ 1.4
BMM (w/ CG, g=500)	64.7 $\pm$ 1.6	62.3 $\pm$ 2.7	64.5 $\pm$ 1.2

TABLE III: Quantitative results for online bimanual motion modulation

Object Variations
Method \ Tasks	PlateWiping	VegetableCutting	StirFry	SteakFlip	PotPlace	BowlShelve	Avg.TSR
DP [ 21 ]	0.33 $\pm$ 0.11	0.13 $\pm$ 0.09	0.27 $\pm$ 0.08	0.20 $\pm$ 0.09	0.33 $\pm$ 0.12	0.20 $\pm$ 0.11	0.24 $\pm$ 0.08
ACT [ 41 ]	0.20 $\pm$ 0.07	0.20 $\pm$ 0.06	0.27 $\pm$ 0.09	0.13 $\pm$ 0.10	0.27 $\pm$ 0.10	0.33 $\pm$ 0.12	0.23 $\pm$ 0.07
IDMP [ 6 ]	0.40 $\pm$ 0.11	0.33 $\pm$ 0.08	0.27 $\pm$ 0.07	0.40 $\pm$ 0.10	0.73 $\pm$ 0.11	0.67 $\pm$ 0.12	0.47 $\pm$ 0.18
DSRL [ 11 ]	0.73 $\pm$ 0.04	0.67 $\pm$ 0.12	0.80 $\pm$ 0.09	0.67 $\pm$ 0.08	0.87 $\pm$ 0.09	0.80 $\pm$ 0.12	0.76 $\pm$ 0.08
YAY [ 13 ]	0.60 $\pm$ 0.13	0.47 $\pm$ 0.11	0.60 $\pm$ 0.13	0.53 $\pm$ 0.11	0.73 $\pm$ 0.13	0.73 $\pm$ 0.10	0.61 $\pm$ 0.05

TABLE IV: Real-robot experiment results of online interactive bimanual skill adaptation

Task	IA (%) $\uparrow$	TA (%) $\uparrow$	CS (%) $\uparrow$
SteakFlip	78.3 $\pm$ 1.8	76.5 $\pm$ 2.2	75.2 $\pm$ 2.5
VegetableCutting	71.5 $\pm$ 1.4	73.3 $\pm$ 1.9	67.8 $\pm$ 1.1
PlateWiping	82.7 $\pm$ 2.2	84.2 $\pm$ 1.9	83.6 $\pm$ 2.3
StirFry	80.4 $\pm$ 2.6	78.9 $\pm$ 1.0	79.3 $\pm$ 1.4

TABLE V: Quantitative results of cross-embodiment evaluation

Fig. 15: Cross-Embodiment Experiment Results. BiSAIL consistently adapts to four bimanual tasks on a dual-arm Franka platform.

Robot Skill Learning

Robot skill learning studies how robots acquire, adapt, and combine reusable behaviors from demonstrations, interaction, and other data.

See all 320 papers

VLBiMan++: Expanding the Generalization Boundary of Vision-Language Anchored One-Shot Bimanual Manipulation
Sep 13, 2026Huayi Zhou, Wei Gao, Yiyang Han +2Bimanual Robotic Manipulation Robotic Manipulation
Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, College of Computer Science and Software Engineering, Shenzhen University
Scaling Latent Motor Adaptation to Coordinated Bimanual Manipulation
Oct 10, 2026Seung-Bin Joo, Alessandro Albini, Luca Wiehe +3Generalization in Robotic Manipulation Adaptive Control
Visual Geometry Group, University of Oxford, Oxford, United Kingdom · Soft Robotics Lab, Oxford Robotics Institute, University of Oxford, Oxford, United Kingdom · Technical University of Munich, Munich, Germany +1
Bimanual Robot Manipulation via Multi-Agent In-Context Learning
Apr 22, 2026Alessio Palma, Indro Spinelli, Vignesh Prasad +4ICL for Robotics Language Model-Based Control
Sapienza University of Rome, Italy · TU Darmstadt, Germany · Hessian.AI, Germany

Sep 13, 2026J/K move · Enter open · S save

VLBiMan++: Expanding the Generalization Boundary of Vision-Language Anchored One-Shot Bimanual Manipulation

Huayi Zhou, Wei Gao, Yiyang Han, Kui Jia+1

Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, College of Computer Science and Software Engineering, Shenzhen University

Bimanual Robotic Manipulation Robotic Manipulation Generalization in Robotic Manipulation Cross-Embodiment Robot Learning Long-Horizon Robotic Manipulation VLMs for Robotics

Open paper AskPDF arXiv

Generalizable bimanual robotic manipulation requires a reusable task prior that can persist across increasingly diverse tasks, objects, scenes, embodiments, and execution conditions, thus avoiding the prohibitive cost of large-scale teleoperated demonstrations and policy retraining. In this work, we present VLBiMan++, an extended framework that expands the generalization boundary of vision-language anchored one-shot bimanual manipulation. Starting from a single human demonstration, VLBiMan++ performs task-aware decomposition to identify reusable and adaptable skill components, and employs vision-language grounded geometric adaptation to transfer these skills to novel configurations without retraining. Building on this foundation, we systematically extend generalization along five dimensions: task generalization through diverse and long-horizon skill compositions; object generalization across unseen categories, varying geometries, and more complex articulated or deformable objects; scene generalization under clutter, occlusion, and dynamic interference; embodiment generalization across heterogeneous dual-arm robotic platforms; and deployment generalization through prolonged closed-loop execution under repeated external perturbations. To support this broader scope, we further introduce object-state-aware adaptation and lightweight trajectory optimization mechanisms that accommodate changes beyond simple rigid 6-DoF pose variations while preserving reliable bimanual coordination. Extensive real-world experiments demonstrate that VLBiMan++ maintains strong task success and adaptation capability across these increasingly challenging settings. Overall, VLBiMan++ advances one-shot bimanual manipulation from demonstrating isolated transferability toward a more systematic and scalable framework for generalization across tasks, objects, scenes, embodiments, and long-term deployment conditions.

Abstract

Robot Skill Learning

Explore similar work

VLBiMan++: Expanding the Generalization Boundary of Vision-Language Anchored One-Shot Bimanual Manipulation

Scaling Latent Motor Adaptation to Coordinated Bimanual Manipulation

Bimanual Robot Manipulation via Multi-Agent In-Context Learning