Know Your Body: A Harness for Direct and Self-Improving Robot Control with VLMs

Zeyu Lou; Yanhong Zeng; Yong Wang; Chenyang Si

cs.ROSep 22, 2026

Know Your Body: A Harness for Direct and Self-Improving Robot Control with VLMs

Authors: Zeyu Lou, Yanhong Zeng, Yong Wang, Chenyang Si

Organizations: Nanjing University · Ant Group · Zhejiang University

Abstract

A general-purpose vision-language model can understand a task goal without knowing how a particular robot's motion and functional parts produce the intended effect. We introduce KnowBody, a harness that makes these action-relevant body relations explicit, queryable, and revisable while keeping the model weights frozen. Initialized from one off-task trajectory, a partial body model guides action selection and the interpretation of past interactions. New evidence refines the model, and knowledge dependent on revised body estimates is rechecked before reuse. Across 32 fixed-budget trials on four real-robot tasks, initialized KnowBody achieves 75% completion versus 25% for the native harness and requires fewer planner rounds on successful trials in tasks completed by both. With persistent updates enabled, planner rounds decrease by 29-53% from the first to the fifth recorded success.

Figures & tables

Figure 1: Knowing the task is not knowing the body. A VLM is able to recognize what to write, but the robot receives an end-effector pose while writing acts at the held tip. KnowBody makes this command–effect relation explicit for action decision and outcomes interpretation.

Figure 2: KnowBody separates online state feedback from validated cross-episode revision. At each decision, the VLM receives the task, current observations, current state $M$ , eligible rules $K$ , and selected original evidence $E$ . It queries the partial body model, issues waypoints and gripper commands, and receives execution feedback through Observe $\rightarrow$ Record $\rightarrow$ Check $\rightarrow$ Update $M$ ; only $M$ returns directly to the next decision. Qualified evidence enters the revision lane, where candidate $B/K$ changes are independently validated and rules dependent on a changed $B$ are re-evaluated. Dashed purple arrows publish validated $B_{k+1}$ to the body model and eligible $K_{k+1}$ to the decision context for later episodes. One off-task trajectory initializes supported relations in $B_{0}$ .

Figure 3: Action-relevant relations exposed by the body model. (a) Achieved robot motion is paired with the displacement of stationary landmarks in the wrist image, supporting visual-direction queries within the observed domain. (b) Permanent end-effector geometry is composed with the current attachment to map the commanded frame to the acting tool tip. The blue glyph schematically denotes the declared panda_link8 frame; the green point marks the visible writing tip. (c) For a distant target, a valid endpoint solution does not certify the intervening motion. The translucent cutout anchors the current physical embodiment; the overlaid paths and body poses schematically contrast an endpoint-only transition with a discontinuous intermediate IK branch (red) against sequentially seeded waypoints that preserve a continuous full-arm path (green) to the same target. The query evaluates path feasibility using the kinematic relations in $B$ and the current joint and attachment state in $M$ before execution.

Figure 4: Real-robot evaluation setup. A Franka Research 3 with a Robotiq 2F-85 gripper operates over the worktable. A ZED Mini provides the wrist view, while two ZED 2i cameras provide third-person views. The same robot and camera arrangement is used in both evaluation protocols.

Figure 5: Task examples and two evaluation protocols. Left: illustrative development sequences for duck placement, apple pushing, writing, and pouring. Upper right: reasoning rounds for five successive successful completions per task under the continual-interaction protocol. Knowledge is retained across attempts and tasks; intervening failures contribute updates but are not plotted. The horizontal axis indexes successful trials, and vertical ranges differ across tasks. Lower right: all 32 fixed-budget trials, with four trials per method and task. Native Codex denotes the native Codex harness using the same frozen VLM as KnowBody .Numbers denote reasoning rounds for successful trials; F denotes failure. KnowBody starts each trial from the same initialization with cross-episode updates disabled. Objects are randomly repositioned before each attempt in both protocols.

Method	Duck	Apple	Writing	Pouring
KnowBody	4/4	3/4	3/4	2/4
$\pi_{0.5}$ -DROID	19/20	2/20	0/20	0/20
$\pi_{0}$ -DROID	18/20	0/20	0/20	0/20

Table 1: Task success on the DROID setup. Entries report successful trials over total attempts.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Clock	Advances when
Call $i$	A model role is invoked. Planner, verifier, and optional evidence-assistant calls are attributed separately.
Interaction $t$	A new observation is captured or a requested action is dispatched and its achieved prefix is observed. Read-only queries do not advance $t$ .
Episode $e$	A new physical evidence group begins. A process restart or pause does not by itself create an independent group.
Task $\tau$	A new task contract fixes the goal, success criteria, diagnostic scope, and resource accounting shared by its episodes.

Appendix

Table A1: The four clocks used by the harness. A read-only query can occur within a model call without advancing the physical-interaction clock.

Component	Reads	Produces	Does not authorize
Context compiler	Task contract, current observation, $M_{t}$ , pinned $B_{e}/K_{e}$ , indexed $\mathcal{E}$	One bounded, source-linked model input	Actions, body revisions, or rule publication
Frozen VLM planner	Compiled text, attached images, read-only tool results	One structured choice and proposed state updates	Hardware execution or writes to active $B/K$
Body-query layer	Frozen observation, body pin, measured state, attachment revision	Supported, unknown, or invalid numerical relation with provenance	Safety or execution
Executor and fixed gates $S$	Requested motion and fresh telemetry	Planned and achieved motion, rejection/interruption status	Task success or reusable knowledge
Evidence verifier	Fixed criteria, observations, actual receipts, unresolved counterevidence	Scoped Pass , Needs_Work , or Blocked review	Motion, changes to criteria, or publication of $B/K$
Revision pipeline	Qualified original evidence, frozen candidates and validation plans	Published body bundle and eligible rules for a later episode	Changes to the current episode’s pin

Appendix

Table A2: Authority boundaries in KnowBody. Model assessments can propose state changes or candidates, whereas execution and publication pass through separate contracts.

	Authoritative content	Writer and lifetime	Model view	Invalidation or withholding
$\mathcal{E}$	Immutable observations, requested/planned/achieved motion, receipts, outcomes, reviews, and source hashes	Runtime and tools append events; records persist across episodes	Current images and short causal facts; older records retrieved by stable evidence ID	Never rewritten. A later interpretation appends a new record and cites the original.
$B$	Versioned configuration $C$ , permanent geometry $G$ , local response $\mathsf{R}$ , support domain $D$ , and provenance/validation $V$	Estimators propose; validation and publication activate a bundle for later episodes	Pin and component versions remain resident; numerical relations enter through supported query results	Configuration mismatch, unsupported domain, contradicted component, or unpublished candidate
$M$	Current scene relations, attachment revision, task progress, hypotheses, counterevidence, and pending tests	Sensor-grounded updates and source-checked model proposals; rebuilt for each episode	Current compact state plus explicit changes and unresolved risks	Entity, camera, gripper, body, or attachment changes stale only dependent claims
$K$	Conditional action–effect rules, scope, outcome, counterexamples, sources, and body dependencies	Learning pipeline publishes after independent validation; persists across compatible episodes	Conditions are checked first; only eligible rules are ranked and shown	False/unknown condition, changed dependency, incompatible configuration, or counterevidence

Appendix

Table B1: Representation contract. “Model view” is the selected projection used for a decision, not necessarily the complete stored object.

Choice	Required fields
motion	rationale, motion plan, nullable evidence request, predicted observation, re-observation condition
observe	rationale and a non-null evidence request for a temporal observation
done	rationale; completion is subsequently checked by the verifier
give_up	concrete blocker and why admissible observation or recovery cannot resolve it

Appendix

Table B2: Choice-specific planner output.

Order	Block headings	Lifetime	Source and role
1–2	STABLE CONTROL CONTRACT ; WRIST DIRECTION MEMORY	Episode pin; updated when qualified local response changes	Interface reminder and compact supported/withheld visual-direction relations
3–8	RIG FACTS ; CONTROLLER FACTS ; SCALE FACTS ; RUNTIME LIMITS ; COVERAGE ; GRASP ATTEMPT MILESTONES	Resident configuration plus measured episode history	Embodiment, controller, units, fixed limits, response coverage, and source-linked grasp attempts
9–10	TASK ; CURRENT SCENE	Task resident; scene refreshed	Verbatim task contract reference and current observation/view inventory
11	CURRENT BODY STATE	Refreshed	Measured end-effector pose, gripper state, diagnostics, and joint-limit headroom
12–14	RETRIEVED EMBODIMENT EVIDENCE ; RECENT CAUSAL MEMORY ; CONTRADICTIONS	Selected accumulated evidence	Eligible records, requested/planned/achieved transitions, $M$ , and unresolved conflicts
15–17	EFFICIENCY TARGET ; MODE ; DECISION REQUIREMENTS	Per call	Resource guidance, nominal/recovery mode, and concise output requirements

Appendix

Table C1: Text blocks for one planner call, in serialization order. The final method-context block carries the durable task contract, remaining resources, unresolved reviews, and continuation provenance.

Role	Text	Images/evidence access
Planner	12k target; 24k hard maximum	At most 10 attached images; indexed archive queries
Reviewer	48k hard maximum	At most 10 images; at most 6 archive reads per call
Assistant	12k hard maximum	Up to 6 registered text records; no images

Appendix

Table C2: Bounded model-facing inputs. These limits bound context, not the number of physical trials.

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 627 papers

RoboHarness: A Simple Harness Outperforms VLA and World Action Models
Sep 29, 2026Bince Qu, Wei Chen, Bo ZhangAgentic AI for Robotics LLM Agent Harnesses
Zhejiang University
Show-Harness: Just a VLM Agent Can Play Robots
Sep 9, 2026Yanzhe Chen, Zechen Bai, Zhijun Cao +7Cross-Embodiment Robot Learning Language-Conditioned Robot Control
Show Lab, National University of Singapore
REKI: Registering Knowledge in VLAs
Oct 15, 2026Juliette Marrie, Paul Pacaud, Adrien Ramanana Rahary +2Vision-Language-Action Models Efficient VLA Models
Kyutai · Inria · École nationale des ponts et chaussées

Sep 29, 2026J/K move · Enter open · S save

RoboHarness: A Simple Harness Outperforms VLA and World Action Models

Bince Qu, Wei Chen, Bo Zhang

Zhejiang University

Agentic AI for Robotics LLM Agent Harnesses Robotic Control Tool-Using Vision-Language Agents

Open paper AskPDF arXiv

The rise of large language models (LLMs) has opened a path toward generalist embodied AI, raising the question of how LLMs can act in the physical world. One straightforward approach is training the model to generate actions: collect robot-action data and video demonstration, refine LLM architecture with an action head, and train a vision-language-action (VLA) model at scale. However, we argue that coding agents have the potential to realize generalist embodied manipulation. All they need is a simple yet effective robot harness. We present RoboHarness: a robotic harness that gives LLM agents a visual-geometric control panel so that they can directly understand and invoke embodied tasks. With RoboHarness, Qwen3.8-Flash-Next outperforms state-of-the-art VLA models on BEHAVIOR Challenge 2025 tasks. The implementation is open source.

Abstract

VLMs for Robotics

Explore similar work

RoboHarness: A Simple Harness Outperforms VLA and World Action Models

Show-Harness: Just a VLM Agent Can Play Robots

REKI: Registering Knowledge in VLAs