cs.RONov 1, 2026
SaveDemocratizing Robot-Use Agents: A Unified Harness with Open Models
Organizations: Tsinghua University · UC, Berkeley
Abstract
Robot-use Agents (RUAs) have shown promising zero-shot manipulation capabilities, but reliance on frontier, proprietary models limits research into further training. To democratize RUA research, we introduce uni-robot-agent, a unified harness for competitive open-weight robot use that supports further training. Without privileged inputs, an out-of-the-box Qwen3.8-Flash-Next achieves a RoboDojo Score of 31.39, surpassing the published GPT-6 Astra-RoboProbe result of 28.97 and many VLA/WAM baselines, and performs competitively on 19 real-world tasks. Our analysis across model generations, families, and sizes links effective robot use to four general capabilities, suggesting that future RUA training should develop them jointly.
Figures & tables
Figure 1: uni-robot-agent: results and system overview. Top: shared tools, execution, and feedback connect models to environment adapters, supporting generalizable agentic training across environments. Bottom: Simulation and real-world scores.
Figure 2: Five logical layers of uni-robot-agent. Requests flow from agent runtimes through orchestration and shared tools to domain services and native backends. Context policies and concrete backends are replaceable components; arrows summarize logical interactions rather than individual network connections.
Figure 3: Tool reuse across environments. Shared tools avoid implementations. : tool; : adapter; : environment.
Figure 4: Serial interaction and pipelining. Reasoning ( ) overlaps sequential tools ( ); queues while runs. Solid/dashed arrows show submissions/results; shading marks overlap. Timings are schematic.
Figure 5: Key design choices. Each column pairs two designs for physical consistency, action precision, or context management.
| Model | Harness | Gen. | Prec. | Long-Hor. | Memory | Open | Avg. |
|---|---|---|---|---|---|---|---|
| VLA & WAM | |||||||
| – | 8.17/13.38 | 5.50/12.40 | 14.67/23.54 | 4.67/5.89 | 1.67/1.98 | 6.93/11.44 | |
| Hy-Embodied-0.5-VLA | – | 8.39/11.78 | 8.00/13.81 | 14.92/25.74 | 12.11/13.37 | 0.58/0.65 | 8.80/13.07 |
| OpenWAM- | – | 14.83/20.71 | 9.25/18.45 | 25.33/34.93 | 9.11/10.41 | 1.08/1.41 | 11.92/17.18 |
| Xiaomi-Robotics-1 | – | 17.00/23.54 | 18.83/26.69 | 23.67/38.39 | 6.56/7.81 | 3.58/3.94 | 13.93/20.07 |
| Agent | |||||||
Table 1: Results on RoboDojo. Each entry reports SR/Score, where success rate (SR) is a percentage and Score is on a 0–100 scale. Our uni-robot-agent configurations exclude task-verifier feedback. Depth-enabled results are shown in gray for reference; our primary results do not use depth. Avg. is the unweighted mean of each metric over the five capability dimensions. † GAP (Hybrid) incorporates .
Figure 6: Representative real-world tasks across five dimensions. Scene photographs and close-ups show task details, with example instructions and illustrative tool calls.
| VLA | WAM | uni-robot-agent | ||
|---|---|---|---|---|
| Capability | Dream-1 | GPT-6 Astra | Qwen3.8-Flash-Next | |
| Spatial understanding | 11.25 | 6.50 | 17.00 | 9.00 |
| Object generalization | 12.50 | 13.50 | 17.50 | 10.00 |
| Instruction following | 14.00 | 9.50 | 18.00 | 13.50 |
| Compositional generalization | 6.00 | 10.00 | 20.00 | 13.00 |
| Unseen tasks | 4.00 | 4.00 | 8.00 | 8.00 |
Table 2: Results on real-world evaluation. Each capability is scored out of 20; Total is out of 100. Per-task results and scoring details are provided in Appendix B .
Figure 7: Robot-use performance across model generations, families, and sizes. Success rate (top) and Score (bottom) on RoboDojo with uni-robot-agent. Robot use is neither exclusive to frontier models nor unique to the latest generation: GPT-5.6 Sol already achieves nontrivial performance, while Qwen3.8-27B succeeds where larger, earlier Qwen models fail. Shading indicates closed-source models.
Figure 8: Four capabilities associated with effective robot use. Correlation between each model’s RoboDojo Score and its errors in (a) general visual understanding, (b) spatial grounding, (c) coding in geometry, and (d) embodied reasoning. Lower error rates across these dimensions are associated with higher task performance, suggesting that effective robot use draws on a combination of these capabilities.
Figure 9: Robot-use performance and general model capabilities. Each point denotes a model, with benchmark performance on the horizontal axis and RoboDojo Score on the vertical axis. Model colors and symbols follow Figure 8 . Dashed lines show linear fits; Pearson correlations are computed across the six models.
Figure 10: Robot use with less geometric information. Recorded progress through 771 tool calls and final score across 42 tasks, with skills enabled.
Figure 12: Model-level resource–performance trade-offs. Astra, Qwen, and DeepSeek each use 252 episodes. Bars show score and mean resource usage per episode.
Figure 14: Model–action overlap in pipelined execution. Blue and green bars show model requests and physical actions; shading marks overlap. Percentages indicate the fraction of action time overlapping model requests. Each panel starts at the first action and uses its own time range.
Figure 15: Episode duration on five selected tasks. Pipelined and serial execution are compared using one episode per configuration and task, excluding simulator initialization.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| ID | Task | Task variation and scoring criterion | Max |
|---|---|---|---|
| Spatial Understanding | 20 | ||
| 1 | Pick and place (fruits) | Move cherries, bananas, apples, oranges, and melons from bowls or plates into a basket. One point for each fruit type placed successfully in every trial. | 5 |
| 2 | Fold rag | Fold large, small, and obliquely placed rags. One point for each condition folded successfully. | 3 |
| 3 | Bimanual plate pull-out | Lift the bottle with one arm and remove and stack the plate with the other. Each arm’s required action receives 0.5 points per trial. | 4 |
| 4 | Insert test tubes into rack | Transfer test tubes from one rack and insert them into another. Each pickup receives 0.5 points and each insertion 0.5 points. | 4 |
Table 3: Complete Real Eval test catalogue. Each dimension has a 20-point budget, and score caps follow the source specification. IDs match Table 4 .
| VLA | WAM | uni-robot-agent | ||||
| ID | Task | Max | Dream-1 | GPT-6 Astra | Qwen3.8-Flash-Next | |
| Spatial Understanding | ||||||
| 1 | Pick and place (fruits) | 5 | 2 | 3 | 5 | 3 |
| 2 | Fold rag | 3 | 1.75 | 1.5 | 3 | 1 |
| 3 | Bimanual plate pull-out | 4 | 3 | 0 | 2 | 1 |
| 4 | Insert test tubes into rack | 4 | 2 | 1 | 3 | 2 |
Table 4: Per-task results on Real Eval. Each capability subtotal is out of 20, and Total is out of 100. Task IDs correspond to Table 3 ; scoring details are given in Appendix B.5 .
| Model / pricing condition | Currency | Input | Cached input | Output |
|---|---|---|---|---|
| GPT-6 Astra ( OpenAI, 2026b ) | USD | 10.00 | 1.00 | 50.00 |
| Qwen3.8-Flash ( Alibaba Cloud, 2026 ) | CNY | 0.80 | 0.10 | 2.70 |
| DeepSeek-V4.1-Flash (peak) ( DeepSeek, 2026 ) | CNY | 2.00 | 0.04 | 8.00 |
| DeepSeek-V4.1-Flash (off-peak) | CNY | 1.00 | 0.02 | 4.00 |
Table 5: Reference API rates for the cost analysis. Rates are per million tokens in the listed currency.
Figure 16: Agent-written triangulation in insert_tubes . Matching pixels marked A define viewing rays, illustrated schematically above the executed code excerpt. Recorded coordinates and reprojection errors connect the calculation to its output; the final frame follows subsequent grasping and insertion.
Figure 17: Geometric hypothesis testing in cover_blocks . The agent varies a candidate table height and computes cup dimensions from selected pixels. The schematic illustrates the ray–plane intersections computed by hit() in the code excerpt; the right-hand frame shows a later placement over the red block.
Figure 18: Grasp adjustment and supported sliding in align_blocks . The upper row summarizes four stages, each illustrated by a representative observation; a stage may contain multiple tool calls. Green and blue bars show physical tools and model requests; shading marks overlap. The marked interval is enlarged below, where the model prepares a clearance check during release. Image timestamps and the main axis are relative to the first displayed observation.
Figure 19: Orientation preparation during digit transport. The upper row summarizes four stages rather than individual tool calls; the lanes below show recorded execution. The enlarged interval shows a model request preparing rotation candidates while transport continues. The later divergence is retained to show how execution feedback affects the prepared plan. Colors and time conventions match Figure 18 .
Figure 20: Score variability with fewer rollouts. Shading shows the central 95% range across 10,000 subsamples of the depth-enabled Qwen evaluation. The dashed line marks the 50-rollout reference; the marker and error bar show the mean and range for six rollouts per task variant.
More in this topic
Robotic Tool Use
Robotic tool use studies how robots select, grasp, and operate tools to perform tasks beyond their native capabilities.
See all 48 papers