Body-Grounded Replanning for Physically Adaptive Manipulation

Namiko Saito; Hiroshi Kera

cs.ROSep 24, 2026

Body-Grounded Replanning for Physically Adaptive Manipulation

Authors: Namiko Saito, Hiroshi Kera

Organizations: Microsoft Research Asia - Tokyo · Waseda University · Chiba University · National Institute of Informatics

Abstract

Manipulation requires not only reasoning about the external environment, but also about the robot's physical condition. A strategy may remain geometrically feasible while becoming physically unsuitable due to increased joint load or limited mobility, yet internal physical state is typically used only for low-level control. We propose body-grounded high-level replanning, which uses internal physical state to adapt manipulation strategies during execution. Body-state events trigger strategy replanning, and an LLM interprets the underlying joint-level state, recent execution statistics, and execution history to select a context-dependent alternative, while leaving the task objective and low-level controller unchanged. We evaluate the framework on a reaching task under controlled load and asymmetric mobility constraints in simulation and on a real robot. Our experiments show that body-grounded replanning maintains high task success while reducing physical effort and enabling more efficient strategy adaptation. Additional contact-rich manipulation experiments demonstrate the applicability of the same replanning interface beyond reaching. These results show that internal physical state can inform not only low-level control, but also high-level decisions about how a manipulation task should be performed.

Figures & tables

Figure 1 : Environment-centered and body-grounded strategies reach the same task-space goal, but the body-grounded strategy reduces load on the affected joint (red, Joint 4) by selecting an alternative approach.

Figure 2 : Body-grounded high-level replanning: internal physical state triggers strategy reconsideration, and an LLM selects a context-dependent alternative.

Load ( $+500$ g)
	Success (%) $\uparrow$				Mean joint-torque norm $\downarrow$				Restricted time [s] $\downarrow$
Condition	Fixed	Env.	Rule	Ours	Fixed	Env.	Rule	Ours	Fixed	Env.	Rule	Ours
Wrist	99	100	91	98	$52.8{\scriptstyle\,\pm 5.7}$	$52.6{\scriptstyle\,\pm 5.8}$	$52.6{\scriptstyle\,\pm 9.2}$	$\mathbf{49.5{\scriptstyle\,\pm 7.4}}$	$\mathbf{3.21{\scriptstyle\,\pm 1.39}}$	$3.35{\scriptstyle\,\pm 1.48}$	$3.47{\scriptstyle\,\pm 3.09}$	$3.38{\scriptstyle\,\pm 2.27}$
Forearm	99	100	91	99	$51.4{\scriptstyle\,\pm 5.7}$	$51.4{\scriptstyle\,\pm 5.6}$	$51.2{\scriptstyle\,\pm 9.3}$	$\mathbf{48.4{\scriptstyle\,\pm 7.7}}$	$3.24{\scriptstyle\,\pm 1.49}$	$3.33{\scriptstyle\,\pm 1.46}$	$3.44{\scriptstyle\,\pm 3.09}$	$\mathbf{3.13{\scriptstyle\,\pm 2.00}}$
Upper arm	99	100	92	97	$49.7{\scriptstyle\,\pm 5.6}$	$49.9{\scriptstyle\,\pm 5.4}$	$49.8{\scriptstyle\,\pm 9.3}$	$\mathbf{46.9{\scriptstyle\,\pm 7.3}}$	$\mathbf{3.24{\scriptstyle\,\pm 1.48}}$	$3.28{\scriptstyle\,\pm 1.42}$	$3.32{\scriptstyle\,\pm 2.96}$	$3.45{\scriptstyle\,\pm 2.58}$
Mean	99.0	100.0	91.3	98.0	$51.3$	$51.3$	$51.2$	$\mathbf{48.3}$	$\mathbf{3.23}$	$3.32$	$3.41$	$3.32$

Table 1 : Simulation results over

100

paired episodes per condition. Torque is mean

\pm

std over episodes; restricted completion time assigns

12.45

s to failures and excludes LLM-inference pauses. Mean rows average the corresponding condition-wise means. Bold indicates the best value per row and metric.

Load ( $+250$ g)
	Success (%) $\uparrow$			Mean actuator-effort norm $\downarrow$			Restricted time [s] $\downarrow$
Condition	Env.	Rule	Ours	Env.	Rule	Ours	Env.	Rule	Ours
Wrist	100	75	100	$2.53{\scriptstyle\,\pm 0.15}$	$2.33{\scriptstyle\,\pm 0.16}$	$\mathbf{2.26{\scriptstyle\,\pm 0.26}}$	$32.74{\scriptstyle\,\pm 5.56}$	$33.03{\scriptstyle\,\pm 10.99}$	$\mathbf{28.79{\scriptstyle\,\pm 4.80}}$
Forearm	75	75	100	$2.43{\scriptstyle\,\pm 0.10}$	$\mathbf{2.02{\scriptstyle\,\pm 0.37}}$	$2.03{\scriptstyle\,\pm 0.33}$	$34.94{\scriptstyle\,\pm 8.79}$	$28.49{\scriptstyle\,\pm 12.62}$	$\mathbf{27.07{\scriptstyle\,\pm 8.26}}$
Upper arm	87.5	62.5	87.5	$2.38{\scriptstyle\,\pm 0.19}$	$1.97{\scriptstyle\,\pm 0.11}$	$\mathbf{1.86{\scriptstyle\,\pm 0.35}}$	$31.35{\scriptstyle\,\pm 7.70}$	$32.84{\scriptstyle\,\pm 13.64}$	$\mathbf{28.51{\scriptstyle\,\pm 10.18}}$
Mean	87.5	70.8	95.8	$2.45$	$2.11$	$\mathbf{2.05}$	$33.01$	$31.45$	$\mathbf{28.12}$

Table 2 : Real-robot results over

8

trials per method and condition. Actuator effort is mean

\pm

std over trials; restricted completion time assigns

45

s to failures. Mean rows average the corresponding condition-wise means. Bold indicates the best value per row and metric.

Figure 4 : Additional real-robot demonstrations: obstacle-avoiding button pressing, box-lid opening, and lint-roller extraction. The robot changes its strategy in response to the encountered physical difficulty and completes the task.

Figure 5 : Body-state signals and replanning decisions during three additional real-robot manipulation tasks. Numbered markers indicate replanning decisions within each task. Replanning changes the strategy in response to physical difficulty, which can reduce joint effort or restore task progress.

Explore similar work

CoRAL: Contact-Rich Adaptive LLM-based Control for Robotic Manipulation
May 4, 2026Berk Çiçek, Mert K. Er, Ozgur S. OguzRobotic Manipulation Contact-Rich Manipulation
LiRA Lab, Department of Computer Engineering, Bilkent University, Ankara, Türkiye
Video Replanning via Latent Embedding Refinement and Rejection-Based Sampling
Oct 20, 2025Po-Chen Ko, Yu Chan, Yu-Hsiang Fu +7
National Taiwan University · MIT CSAIL · Cornell University +1
Learning Holistic Whole-Body Loco-Manipulation with a Bipedal Mobile Manipulator
Sep 16, 2026Zhongyu Chen, Yuxuan Nai, Qian Chen +8Whole-Body Control Humanoid Loco-Manipulation
ZJU-UIUC Institute. · LimX Dynamics.

May 4, 2026, cs.ROJ/K move · Enter open · S save

CoRAL: Contact-Rich Adaptive LLM-based Control for Robotic Manipulation

Berk Çiçek, Mert K. Er, Ozgur S. Oguz

LiRA Lab, Department of Computer Engineering, Bilkent University, Ankara, Türkiye

Robotic Manipulation Contact-Rich Manipulation Efficient Latent Reasoning

Open paper AskPDF arXiv

While Large Language Models (LLMs) and Vision-Language Models (VLMs) demonstrate remarkable capabilities in high-level reasoning and semantic understanding, applying them directly to contact-rich manipulation remains a challenge due to their lack of explicit physical grounding and inability to perform adaptive control. To bridge this gap, we propose CoRAL (Contact-Rich Adaptive LLM-based control), a modular framework that enables zero-shot planning by decoupling high-level reasoning from low-level control. Unlike black-box policies, CoRAL uses LLMs not as direct controllers, but as cost designers that synthesize context-aware objective functions for a sampling-based motion planner (MPPI). To address the ambiguity of physical parameters in visual data, we introduce a neuro-symbolic adaptation loop: a VLM provides semantic priors for environmental dynamics, such as mass and friction estimates, which are then explicitly refined in real time via online system identification, while the LLM iteratively modulates the cost-function structure to correct strategic errors based on interaction feedback. Furthermore, a retrieval-based memory unit allows the system to reuse successful strategies across recurrent tasks. This hierarchical architecture ensures real-time control stability by decoupling high-level semantic reasoning from reactive execution, effectively bridging the gap between slow LLM inference and dynamic contact requirements. We validate CoRAL on both simulation and real-world hardware across challenging and novel tasks, such as flipping objects against walls by leveraging extrinsic contacts. Experiments demonstrate that CoRAL outperforms state-of-the-art VLA and foundation-model-based planner baselines by boosting success rates over 50% on average in unseen contact-rich scenarios, effectively handling sim-to-real gaps through its adaptive physical understanding.

Abstract

Explore similar work

CoRAL: Contact-Rich Adaptive LLM-based Control for Robotic Manipulation

Video Replanning via Latent Embedding Refinement and Rejection-Based Sampling

Learning Holistic Whole-Body Loco-Manipulation with a Bipedal Mobile Manipulator