Embodied Learning of Reward for Musculoskeletal Control with Vision Language Models

Saraswati Soedarmadji; Yunyue Wei; Chen Zhang; Yisong Yue; Yanan Sui

cs.RODec 28, 2025

Embodied Learning of Reward for Musculoskeletal Control with Vision Language Models

Authors: Saraswati Soedarmadji, Yunyue Wei, Chen Zhang, Yisong Yue, Yanan Sui

Organizations: Tsinghua University · California Institute of Technology

Abstract

Discovering effective reward functions remains a fundamental challenge in motor control of high-dimensional musculoskeletal systems. While humans can describe movement goals explicitly such as "walking forward with an upright posture," the underlying control strategies that realize these goals are largely implicit, making it difficult to directly design rewards from high-level goals and natural language descriptions. We introduce Motion from Vision-Language Representation (MoVLR), a framework that leverages vision-language models (VLMs) to bridge the gap between goal specification and movement control. Rather than relying on handcrafted rewards, MoVLR iteratively explores the reward space through iterative interaction between control optimization and VLM feedback, aligning control policies with physically coordinated behaviors. Our approach transforms language and vision-based assessments into structured guidance for embodied learning, enabling the discovery and refinement of reward functions for high-dimensional musculoskeletal locomotion and manipulation. These results suggest that VLMs can effectively ground abstract motion descriptions in the implicit principles governing physiological motor control.

Figures & tables

Figure 1: Workflow of MoVLR . Policy optimization is performed to provide high-dimensional musculoskeletal dynamics of the reward candidate. A VLM evaluates the corresponding movement video $\boldsymbol{\zeta}^{(i)}$ to update the current best reward design $r^{*}$ and suggest biomechanical improvements $\mathcal{F}$ for a LLM to refine reward generation of $r^{(i+1)}$ .

Figure 2: Overview of the six evaluated tasks. The top row illustrates the environment setup for each task, and the bottom row visualizes the relative weighting of learned reward terms.

Panel of Figure 3 — Figure 3: Performance comparison of MoVLR against baselines across (a) locomotion and (b) manipulation tasks. Locomotion is measured by total distance walked in 10s (higher is better), while manipulation is evaluated by object distance and orientation errors (lower is better).

Figure 4: Progressive improvement of the musculoskeletal model’s gait across training stages based on movement video.

Panel of Figure 5 — Figure 5: Weighted reward terms per stage for (a) locomotion task, (b) manipulation task

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Example inputs and outputs of the (a) VLM, and (b) LLM. The VLM analyzes a video motion sequence based on the given motion description and provides diagnostic feedback. The LLM uses this feedback to explore corresponding code modifications to the reward function.

Figure 8: Comparison of reward terms designed by LLM only (blue) and by human experts (green), with shared terms shown in orange, across three musculoskeletal systems.

Multimodal Reward Modeling

Multimodal reward modeling builds evaluators that score images, video, audio, and agent behavior against human preferences or task goals.

See all 93 papers

ROOT: Discovering Rewards for User-Specified Embodied Behaviors
Oct 3, 2026Eren Sadikoglu, Aditya Taparia, Xinyuan Liu +1Reward Design for RL Robotic RL
Robotics and Autonomous Systems, Arizona State University, Tempe, Arizona, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA
VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning
Jul 1, 2026Kuan-Chen Chen, Winston Chen, Wei-Fang Sun +1Multimodal Reward Modeling Reward Design for RL
Department of Computer Science, National Tsing Hua University · NVIDIA AI Technology Center (NVAITC)
Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models
Mar 17, 2026Yanru Wu, Weiduo Yuan, Esteban Martinez Licon +4Long-Horizon Robotic Manipulation Robot Skill Learning
USC Physical Superintelligence Lab

Oct 3, 2026J/K move · Enter open · S save

ROOT: Discovering Rewards for User-Specified Embodied Behaviors

Eren Sadikoglu, Aditya Taparia, Xinyuan Liu, Ransalu Senanayake

Robotics and Autonomous Systems, Arizona State University, Tempe, Arizona, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA

Reward Design for RL Robotic RL LLM-Guided RL RL for Legged Locomotion

Open paper AskPDF arXiv

Reinforcement learning for embodied control remains constrained by the difficulty of reward specification. Although recent large language model (LLM)-based methods can synthesize reward functions from natural-language descriptions, they often fail to capture subtle behavioral properties that humans care about, such as natural gait, posture, and movement style. This limitation arises because many desired behaviors are easier to recognize visually than to encode in a reward function. We introduce Reward Optimization via Observable Trees (ROOT), a framework for discovering reward functions that align learned policies with user-specified embodied behaviors. Rather than relying solely on scalar training statistics, ROOT casts reward design as an observation-guided search over a persistent experiment tree that stores reward programs, trained policies, and rollout observations, together with behavioral insights distilled by a video-language model, to diagnose behavioral failures and guide subsequent reward refinements. We evaluate ROOT on seven tasks across four embodiments: simulated Hopper, HalfCheetah, Ant, Unitree Go2, and as well as the real-world Unitree Go2. ROOT produces behaviors that better align with user intent than those generated by existing LLM-based reward-generation methods, achieving up to 86.8% locomotion-completeness accuracy and improving Vid-LLM behavioral alignment from 3.56/5 to 4.14/5, a 16.5% improvement over baselines. Human evaluations further support these results, with ROOT preferred in 51-63% of pairwise comparisons.

Abstract

Multimodal Reward Modeling

Explore similar work

ROOT: Discovering Rewards for User-Specified Embodied Behaviors

VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning

Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models