cs.RONov 13, 2026

Objectives as Policies: Vision-Language Model Cost Programs for Contact-Rich Manipulation

Authors: Zihao Lu, Trong Gia Hung Nguyen, Fuquan Wang, Yifan Zhu

Organizations: University of Illinois Chicago, Chicago, IL, USA

Abstract

Vision-language models (VLMs) have shown strong semantic and spatial reasoning capabilities in 3D environments. Recently, connecting VLMs with model-predictive controllers (MPC) has emerged as a paradigm for combining strong semantic reasoning with physics-based planning. However, connecting these two for contact-rich robotic manipulation requires a task representation that bridges the two and captures how physical interactions can be reasoned by VLMs. In this work, we introduce Objectives as Policies (OaP), a framework that connects a pretrained VLM to a shared sampling-based MPC through an editable objective program. Each program specifies a sequence of stages for completing a task, combining running and terminal costs from a cost library with stage completion conditions. Given a natural-language task instruction and scene observation, we build a simulation-based digital twin through real-to-sim, and the VLM generates an objective program that is executed by the MPC in the simulation. The program is iteratively updated by the VLM based on MPC rollouts, before being deployed in the real world. We evaluate our framework in both simulation and the real world on four tabletop manipulation tasks, and show that it outperforms strong baselines. Additional videos and examples are available on the project website: https://zihaolu001.github.io/oap-project/

Figures & tables

Explore similar work

CardsList
  1. DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models

    Mar 17, 2026Emily Yue-Ting Jia, Weiduo Yuan, Tianheng Shi +3VLMs for RoboticsRL for VLMs

  2. PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models

    Aug 27, 2026Davood Soleymanzadeh, Kaidi Zhang, Zhiyuan Zhang +4Vision-Language-Action ModelsReasoning for VLA Models

  3. AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding

    Jun 4, 2026Qize Yu, Jiadi You, Yuran Wang +12Language-Conditioned Robot ManipulationRobotic Manipulation