Robo-Harness K1: Harnessing Robot-Use Agents via Perception Augmentation

Zexi Li; Yehang Zhang; Wenqian Li; Haojian Huang; Chenxu Wang; Shiyuan Deng; Yangkai Wei; Tianyi Zhang

cs.ROSep 24, 2026

Robo-Harness K1: Harnessing Robot-Use Agents via Perception Augmentation

Authors: Zexi Li, Yehang Zhang, Wenqian Li, Haojian Huang, Chenxu Wang, Shiyuan Deng, Yangkai Wei, Tianyi Zhang, +7 more

Organizations: The Chinese University of Hong Kong · Knowin AI · The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Foundation vision-language models (VLMs) understand objects, instructions, and spatial relations, yet translating this capability into robotic manipulation remains difficult. Vision-language-action (VLA) models require extensive demonstrations and may compromise pretrained understanding, while direct RGB-only VLM control is costly and strongly dependent on model capability. We introduce Robo-Harness K1, a robot-use agent (RUA) framework that exposes perception as tools. The agent queries calibrated depth, persistent visual anchors, spatial measurements, and grasp hypotheses, then selects generic motions from the returned evidence. This interface makes 3D geometry accessible without changing the VLM architecture or training a depth encoder. On matched LIBERO-PRO tasks, Gemini 3.7 Flash with K1 reaches 77.8% accuracy, surpassing GPT-6 Astra's 61.1% with an RGB-only harness; K1 further improves Astra to 88.9%. Without target fine-tuning, Gemini with K1 transfers to three RoboSuite arms and dual-arm RoboTwin tasks. On RoboTwin, it achieves 32.0% on Easy and 28.0% on Hard, showing resilience to visual and environmental perturbations. K1 also produces tool-call traces aligned with next-token training. A Qwen3.5-9B student trained on only 107 teacher episodes reaches 44.2% accuracy on new initial states versus 30.2% for OpenVLA, and 13.9% on held-out task conditions versus 0.0% for OpenVLA. These results suggest that perception-augmented RUAs offer a promising route to sample-efficient, generalizable robotic policies that leverage VLM capabilities through an accessible tool interface.

Figures & tables

Figure 1 : Robo-Harness K1 with Gemini 3.7 Flash surpasses GPT-6 Astra on LIBERO-PRO tasks. Astra with Robo-Harness K1 obtains 27.8% performance gains. Published baselines are from Table 1 ( Zhang et al., 2026a ; Sun et al., 2026 ) . Evaluation subsets, trial counts, and comparison scope are detailed in Appendix C.3 .

Figure 2 : Robo-Harness K1: perception-augmented Robot-Use Agents. (A) Agent harness loop. The VLM receives RGB views, the instruction, and robot state. It queries region grounding, calibrated geometry, anchor tracking, and grasp hypotheses as needed, while depth and calibration stay behind the tool interface. (B) Training. Qwen3.5-9B learns from tool-call traces via SFT, and generalizes well. Bars show epoch-5 accuracy on new-state and held-out-task evaluation. Qwen K1 RUA uses our harness, Qwen RUA is the RGB-only direct-action baseline. All methods share source episodes but differ in inputs and tools. (C) Transfer. Gemini with K1 transfers from LIBERO to RoboSuite (Panda, UR5e, IIWA) without target fine-tuning, using explicit embodiment adapters and the same Panda gripper.

Figure 3 : Four perceptual tool families through the VLM’s existing RGB interface. Recorded image crops show (a) labeled regions, (b) measured surface geometry, (c) a persistent anchor after camera motion, and (d) a projected grasp hypothesis. The complete tool schemas, overlay conventions, and source calls appear in Appendix A.5 .

	Object		Goal		Spatial		Average
Method	Position	Task	Position	Task	Position	Task
VLA Methods
OpenVLA	0.0	0.0	0.0	0.0	0.0	0.0	0.0
$\pi_{0}$	0.0	0.0	0.0	0.0	0.0	0.0	0.0
$\pi_{0.5}$	17.0	1.0	38.0	0.0	20.0	1.0	12.8
MolmoAct	6.0	0.0	0.0	0.0	0.0	0.0	1.0

Table 1: LIBERO-PRO task accuracy (%). Position denotes the position-swap suite. MolmoAct, NORA, X-VLA, and AtomVLA are from Sun et al. (2026) , Table IV; other VLA and CaP rows are from RATs ( Zhang et al., 2026a ) . Averages use the six displayed conditions.

Figure 4 : Task accuracy across training epochs. A: in-domain tests with exact training configurations. B: OOD new initial states (development set). C: OOD held-out task conditions. Qwen K1 RUA uses our harness. Qwen RUA is the RGB-only text-action baseline. Qwen VLA is the modified VLA with the same Qwen3.5-9B backbone. All methods share the same 107 teacher episodes. Training details are in Appendix B .

	References ( Zhang et al., 2026a )		K1+ Gemini (ours)
Task/Methods or Arms	CaP-Agent0	+ RATs skills	Panda	UR5e	IIWA
Cube lifting	68.0	84.0	100.0	100.0	100.0
Cube restacking	34.0	46.0	100.0	100.0	100.0
Cube stacking	46.0	60.0	100.0	100.0	100.0
Nut assembly	0.0	0.0	60.0	55.0	45.0
Spill wiping	100.0	100.0	65.0	–	–

Table 2: RoboSuite zero-shot transfer accuracy (%). K1 uses Gemini without target fine-tuning. CaP-Agent0 and RATs are published results ( Zhang et al., 2026a ) ; – denotes untested tasks.

Figure 5 : RoboTwin zero-shot transfer. Task-macro accuracy under Easy and Hard conditions. Gray: trained-policy references from RoboTwin ( Chen et al., 2025b ) . Blue: K1 + Gemini without target-task fine-tuning. Per-task scores are in Appendix D.2 .

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Contract and boundary
find_regions	Text-prompted SAM 3 masks from a current original RGB image, with RGB-D summaries and stable S identifiers. Multiple candidates require model-side disambiguation.
inspect_region	Box-prompted segmentation or explicit refresh of a region. The supplied label does not guide semantic segmentation. Returns visible geometry, not a full-object pose.
measure_depth	Back-projects a selected valid visible pixel to world coordinates, with TCP-relative displacement. Creates a trackable point reference, not a grasp target.
fit_geometry	Fits local visible geometry in an image region. A fitted plane or unsigned principal direction is not an articulation model.
region_relation	Measures region-to-robot or region-to-region displacement and available plane relations.
check_region_view	Projects an existing measured region into the other available camera, with a depth consistency check. This is not a newly rendered viewpoint.

Appendix

Table 3: Tool families in the evaluated harness. Measurements and tool execution do not certify task success.

Figure 6 : Perception as tools, illustrated by recorded model-visible feedback. Each row corresponds to one tool family. (a–c) Text grounding, box refinement, and depth-consistent projection into the other existing view. (d–f) Metric points and visible-surface geometry, with selected receipt values below the images. (g–i) The same wrist-camera anchor changes pixel location but has only 0.89 mm estimated world displacement, before becoming invalid. These are automatic updates, not additional tracking calls. (j–l) Three grasp hypotheses from one tool response, shown as enlarged crops rather than executed hand poses. Green contours denote regions; orange lines, principal axes; purple lines, TCP–surface references. No views or scene geometry are synthesized.

Arm	Trainable components	Learning rate	Examples/epoch
Qwen K1 RUA	Language LoRA $r=16$ , $\alpha=32$ , dropout .05	$10^{-4}$	2,482
Qwen RUA	Same language LoRA, native text output head	$10^{-4}$	1,670
Qwen VLA (RGB-only / RGB-D)	Same LoRA plus numeric-state/action head, optional depth encoder	$10^{-4}$	10,241
$\pi_{0.5}$	Language LoRA $r=16$ , $\alpha=32$ , plus full action expert/projections	$2.5\times 10^{-5}$	10,241
OpenVLA	All-linear LoRA $r=32$ , $\alpha=16$ , dropout .05	$5\times 10^{-4}$	40,808

Appendix

Table 4: Adaptation recipe. All arms use seed 17, batch size 1 with accumulation 8, and save checkpoints at epochs 1/3/5.

Property	Original RGB interface	K1
Scene information	Two RGB views	Same two views plus depth-backed measurements
Text history	Original complete history	Last eight exchanges plus retrieval
Image history	Current and previous decision	Current and previous decision, optional retrieved images
Motion	Original interpolated pose tool	Local corrections, persistent poses, full-target transit
Perception	Model interprets RGB	Explicit region, depth, tracking, grasp tools
Progress memory	Dialogue	Dialogue plus model-maintained milestones

Appendix

Table 5: Interface differences in the same-GPT comparison.

Suite	Trained condition indices (A/B)	Held-out (C)
goal_swap	0, 1, 4, 7, 8, 9	2, 6
goal_task	3, 4, 5, 7, 8, 9	2, 6
object_swap	2, 3, 4, 5, 6, 7, 8, 9	0, 1
object_task	2, 3, 4, 5, 6, 7, 8, 9	0, 1
spatial_swap	1, 2, 5, 6, 7, 8, 9	0, 3
spatial_task	1, 2, 3, 4, 6, 7, 8, 9	0, 5

Appendix

Table 6: Task-index inventory. A uses one actually trained initial state for each listed condition. B uses the same condition at state 3. C uses all three states 40–42.

System	Spatial	Object	Goal	Average
RGB + GPT-6 Astra	33.3	83.3	66.7	61.1
K1 + Gemini 3.7 Flash	66.7	100.0	66.7	77.8
K1 + GPT-6 Astra	83.3	100.0	83.3	88.9

Appendix

Table 7: Paired system comparison: task accuracy (%). Tasks and initial states are matched. Each system is evaluated on six cases per category, eighteen in total. The GPT comparison changes perception, memory, control interface, and completion feedback together.

Model	Epoch	A	B	C
Qwen K1 RUA	1	7/43 (16.3%)	10/43 (23.3%)	3/36 (8.3%)
Qwen K1 RUA	3	14/43 (32.6%)	20/43 (46.5%)	5/36 (13.9%)
Qwen K1 RUA	5	22/43 (51.2%)	19/43 (44.2%)	5/36 (13.9%)
Qwen RUA	1	0/43 (0.0%)	0/43 (0.0%)	0/36 (0.0%)
Qwen RUA	3	0/43 (0.0%)	0/43 (0.0%)	0/36 (0.0%)
Qwen RUA	5	3/43 (7.0%)	2/43 (4.7%)	0/36 (0.0%)

Appendix

Table 8: Full checkpoint ledger. Incomplete groups are marked ∗ , and unrun groups are –.

Model	Group	Spatial	Object	Goal
Qwen K1 RUA	A	6/15 (40.0%)	11/16 (68.8%)	5/12 (41.7%)
Qwen K1 RUA	B	7/15 (46.7%)	7/16 (43.8%)	5/12 (41.7%)
Qwen K1 RUA	C	3/12 (25.0%)	1/12 (8.3%)	1/12 (8.3%)
Qwen RUA	A	0/15 (0.0%)	1/16 (6.2%)	2/12 (16.7%)
Qwen RUA	B	0/15 (0.0%)	1/16 (6.2%)	1/12 (8.3%)
Qwen RUA	C	0/12 (0.0%)	0/12 (0.0%)	0/12 (0.0%)

Appendix

Table 9: Epoch-5 success by task category. Denominators: 15/16/12 for A and B; 12 per category for C.

Method	Robot	Task	Successes / trials	Accuracy (%)
K1 + Gemini	Panda	Cube lifting	20/20	100.0
K1 + Gemini	Panda	Cube restacking	20/20	100.0
K1 + Gemini	Panda	Cube stacking	20/20	100.0
K1 + Gemini	Panda	Nut assembly	12/20	60.0
K1 + Gemini	Panda	Spill wiping	13/20	65.0
K1 + Gemini	Panda	Two-arm handover	1/20	5.0

Appendix

Table 10: Complete transfer ledger. Each tested task has five reset configurations and four rollouts per configuration. Official-base VLA rows are not the LIBERO-fine-tuned models in Figure 4 .

Task	Easy	Hard	Excluded	Action limit
Grab Roller	8/10 (80.0%)	9/10 (90.0%)	0/0	400
Handover Mic	2/10 (20.0%)	0/10 (0.0%)	0/0	600
Lift Pot	2/10 (20.0%)	2/10 (20.0%)	0/0	400
Move Can Pot	4/8 (50.0%)	5/8 (62.5%)	2/2	400
Place Dual Shoes	0/8 (0.0%)	0/6 (0.0%)	2/4	600
Place Phone Stand	2/10 (20.0%)	0/8 (0.0%)	0/2	400

Appendix

Table 11: K1 + Gemini on RoboTwin. Scores are successes/retained episodes followed by accuracy. Excluded counts are initialization failures in Easy/Hard order.

Task	ACT	DP	DP3	RDT	$\pi_{0}$
Grab Roller	66/6	98 /1	77/1	74/43	96/ 80
Handover Mic	9/0	53/0	93/7	98/ 41	100 /13
Lift Pot	7/2	37/0	85 /0	82/19	84/ 36
Move Can Pot	0/0	42/0	28/0	47/23	74 / 32
Place Dual Shoes	0/0	7/0	1/0	4/ 4	15 /0
Place Phone Stand	0/0	17/0	8/1	15/6	35 / 7

Appendix

Table 12: Published RoboTwin task accuracies (%). Each cell is Easy/Hard. Averages are recomputed over the ten displayed tasks from RoboTwin ( Chen et al., 2025b ) , Table 6.

Figure 7 : How Gemini allocates its model decisions. Shares and counts over 6,282 responses in 180 LIBERO-PRO trajectories, including unsuccessful episodes. Each response occupies exactly one category. Output limit denotes responses without a tool call.

Category	Tool	Requests
Motion	move_toward	1,835
	move_relative	654
	move_to_pose	547
	rotate_toward	271
	align_axis	48
	set_gripper	883

Appendix

Table 13: Tool-request counts for Figure 7 , including requests that return errors.

Figure 8 : Longer prompt history does not consistently improve accuracy. Gemini 3.7 Flash with the frozen K1 harness, evaluated on the same 18 LIBERO-PRO cases per setting. Left: image history

K

at fixed

N=8

. Right: text history

N

at fixed

K=1

, with a base-2 logarithmic horizontal axis. Point labels show observed accuracy, and lines are visual guides. The protocol and interpretation boundaries are detailed below.

$K$	$N$	Successes	Accuracy (%)	Mean calls
0	8	15/18	83.3	28.3
1	8	14/18	77.8	33.0
2	8	14/18	77.8	41.2
3	8	11/18	61.1	47.4
4	8	12/18	66.7	45.9
1	1	13/18	72.2	70.0

Appendix

Table 14: Memory-context ablation outcomes. Each row uses the same 18 cases. Mean calls include successes and failures. Both evaluations share

K=1,N=8

, counted once.

Figure 9 : Recorded LIBERO-PRO trajectory. GPT-6 Astra with K1 on Spatial swap task 1/state 0. Panels show original external/wrist RGB at four decision frames, without candidate or region overlays. The final panel precedes native termination. Table 15 summarizes the tool evidence.

Call	Operation	Observable role
0	find_regions	Search for black bowls. S2 is selected from the existing external view based on the ramekin relation.
2	grasp_candidates	Request nominal grasp hypotheses for S2. No motion is executed by this call.
3–9	Transit and rotation	Approach with open jaws, rotate in bounded steps, and begin the persistent pose approach.
10, 17	inspect_region	Refresh S2 using a wrist-image bounding box while approaching the rim.
20	set_gripper	Close at frame 159 and allow native settling.
21–23	Inspect, lift, update	Reinspect S2, request a 25 mm lift, and update the carry milestone from the resulting observation.

Appendix

Table 15 : Selected calls from the LIBERO-PRO trace. Skipped calls adjust poses or retrieve history; no resets are omitted.

Figure 10 : RoboSuite nut assembly with regrasping. Recorded external and wrist RGB at four decision frames. The nut remains on the table at frame 326; after the third closure, it is visible near the gripper at frame 428 and above the peg at frame 504. The last panel precedes the final descent and native success at frame 534.

Call	Operation	Observable role
0–3	Ground and inspect	Find the nut, refine its handle as S2, measure depth, and request grasp hypotheses.
4–9	Approach and close	Record progress, approach using wrist depth, close at frame 79, and request a 3 cm lift.
10–14	Measure and revisit	Measure the scene and move toward the peg. A new search at frame 171 finds the nut still near its original position.
15–23	Second attempt	Return with open jaws, remeasure, close at frame 235, and repeat lift and transport.
24–29	Reground and regrasp	Find the nut again at frame 326, use wrist depth to refine approach, and close at frame 384.
30–31	Lift and inspect	Lift with closed jaws. The frame-428 wrist image shows the nut near the gripper; a depth query follows.

Appendix

Table 16 : Selected calls from the 35-call RoboSuite episode. Omitted calls perform measurements or local motion within the same episode.

Figure 11 : Dual-arm grasping in a randomized RoboTwin scene. Recorded head, left-wrist, and right-wrist RGB. Command indices count native action requests, not physics steps or video seconds. The final panel is the recorded terminal observation after both arms lift; it is not an additional model call.

Call	Operation	Observable role
0–2	Ground and propose	Find roller region S1 in the head view, then request grasp hypotheses separately for the left and right arms.
3–5	move_effectors	Approach, lower, and close both grippers using paired arm targets.
6–7	Adjust right grasp	Reopen and reposition the right gripper, then close it while retaining the left grasp command.
8–9	Correct a failed target	A negative left-arm height produces planner failures. The next call restores a positive height and refines the right grasp.
10	done	Native success is false at command 30. The completion claim is rejected and the episode remains active.
11	move_effectors	Lift both closed grippers by approximately 6 cm in achieved motion. Native success follows at command 31.

Appendix

Table 17 : Calls from the 12-call RoboTwin episode, including the failed target and rejected completion check.

Explore similar work

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents
Jul 9, 2026Yixian Zhang, Huanming Zhang, Feng Gao +14Goal-Conditioned Dynamic Manipulation Visuomotor Control
Tsinghua University · Striding AI · Purdue University +4
Show-Harness: Just a VLM Agent Can Play Robots
Sep 9, 2026Yanzhe Chen, Zechen Bai, Zhijun Cao +7Robot Systems Embodiment
Show Lab, National University of Singapore
World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal
Sep 24, 2026Yehang Zhang, Haojian Huang, Yifan Chang +13Robotic Manipulation Efficient World-Action Model
HKUST(GZ) · Knowin AI · CUHK

Jul 9, 2026, cs.ROJ/K move · Enter open · S save

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

Yixian Zhang, Huanming Zhang, Feng Gao, Xiao Li+13

Tsinghua University · Striding AI · Purdue University+4

Goal-Conditioned Dynamic Manipulation Visuomotor Control Frozen Vision-Language Models

Open paper AskPDF arXiv

Language-conditioned manipulation requires both precise contact-rich control and robust reasoning over language, scenes, and long horizons. End-to-end Vision-Language-Action (VLA) models provide strong local visuomotor skills, but they are trained on in-distribution task trajectories and often fail under deployment perturbations such as semantic retargeting, goal re-binding, spatial-layout shifts, and unstable local contacts. LLM coding agents provide complementary semantic and compositional reasoning, but purely analytic primitives struggle with irregular grasping, constrained placement, and articulated-object interaction. We present Harness VLA, a memory-augmented agentic framework that exposes a frozen VLA as a retryable contact-rich primitive and composes it with a small fixed library of analytic primitives for grounding, staging, transport, navigation, and release. Rather than expanding the skill library, the harness learns the operating range of these fixed primitives from task-specific execution traces, global success rules, and failure models. By lifting semantic re-grounding, non-contact execution, and VLA re-staging to the planner while reserving the frozen VLA for local contact-rich phases, Harness VLA extends pretrained VLAs beyond their original trajectory distribution without finetuning. Across perturbed tabletop, household kitchen, and clean-to-randomized bimanual manipulation, Harness VLA improves over the strongest relevant baselines by 38.6 and 25.4 percentage points on LIBERO-Pro and RoboCasa365, respectively, and reaches 58.4% on RoboTwin C2R. Real-world demonstrations on dual-Franka robots further show target redirection, grasp recovery, and new task compositions with the same frozen VLA. Code is available at https://github.com/RLinf/RPent.

Abstract

Explore similar work

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

Show-Harness: Just a VLM Agent Can Play Robots

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal