cs.ROOct 20, 2026
SaveJARVIS: From Visual Manuals to Autonomous Robotic Assembly
Organizations: A*STAR Institute of Advanced Intelligence and Computing · Monash University · National University of Singapore · University of Utah
Abstract
Visual manuals communicate procedural knowledge through stepwise diagrams, but translating these instructions into reliable long-horizon robotic execution remains challenging. We introduce JARVIS, a method-agnostic benchmark for autonomous visual-manual-guided execution without additional language instructions. JARVIS comprises 236,679 LEGO assembly tasks across 8,987 scenes, with a carefully curated JARVIS-v1 suite of 50 tasks for interactive robotic evaluation. Tasks span varying initial structural complexities and assembly horizons, requiring agents to interpret manuals, ground actions in an evolving workspace, and execute the intended procedure. A structure-aware evaluation protocol captures geometric accuracy, structural relations, construction order, partial progress, and completion. We further introduce JARVIS-agentic, a tool-grounded embodied agent that composes robot primitives into executable programs and uses execution feedback for verification and recovery, without task-specific action trajectories or model parameter updates. Across nine multimodal and reasoning models, task-oriented primitives increase the average assembly score. Performance nevertheless declines sharply with assembly horizon, and execution errors remain difficult to correct. Prior-task experience improves initial performance, while repeated reflection provides inconsistent benefits, highlighting persistent limitations in connecting visual procedural reasoning with reliable physical execution.
Figures & tables
Figure 1: Overview of the visual planning, execution, and reflection workflow in JARVIS.
| Split | #Tasks | Assembly Horizon | Initial Complexity | Total Bricks | |||
| avg. | range | avg. | range | avg. | range | ||
| Easy | 20 | 2.45 | 1-4 | 33.05 | 2-105 | 35.50 | 4-107 |
| Medium | 15 | 4.13 | 2-7 | 78.80 | 2-213 | 82.93 | 7-215 |
| Hard | 15 | 9.87 | 3-29 | 127.00 | 57-236 | 136.87 | 75-239 |
Table 1: Statistics of the JARVIS-v1 benchmark.
Figure 2: Overview of JARVIS-V1. The framework integrates multimodal task context, agentic code generation, simulation execution, evaluation, and self-reflection in a closed-loop assembly process.
| Model | Tool library | Soft Score | Valid (%) | Failed actions per task | |||
| Easy | Medium | Hard | Overall | ||||
| Gemini-3-Pro [ 8 ] | Generic | 13.27 | 6.61 | 4.68 | 8.19 | 98.00 | 6.04 |
| Task-oriented | 22.16 | 19.30 | 9.94 | 17.13 | 100.00 | 4.96 | |
| Claude-Opus-4.5 [ 3 ] | Generic | 15.84 | 16.13 | 9.51 | 13.83 | 92.00 | 7.36 |
| Task-oriented | 22.20 | 18.58 | 10.35 | 17.04 | 100.00 | 5.31 | |
| Claude-Haiku-4.5 [ 2 ] | Generic | 11.69 | 3.22 | 2.88 | 5.93 | 90.00 | 6.70 |
Table 2: Model and tool-library performance across task difficulty levels. Underline and blue denote the best generic and task-oriented results; bold marks the best Overall score within each library.
| Assembly horizon | ||||
| Model | Initial complexity | 1–2 | 3–5 | 6+ |
| Gemini-3-Pro [ 8 ] | 0–29 | 43.42 (5) | 15.40 (10) | – |
| 30–78 | 28.03 (4) | 17.99 (8) | 10.40 (6) | |
| 79+ | 16.40 (4) | 12.96 (7) | 6.01 (6) | |
| Claude-Opus-4.5 [ 3 ] | 0–29 | 36.46 (5) | 18.38 (10) | – |
| 30–78 | 30.87 (4) | 14.47 (8) | 11.88 (6) | |
Table 3: Joint effect of initial scene complexity and assembly horizon. Each cell reports the mean soft score, with the number of evaluated tasks in parentheses. Results are computed from the latest no-reflection execution round for each model.
| Model | Input | Position Error (mm) | Orientation Error ( ∘ ) | Matched Steps | Missing Steps | Coverage (%) |
| Gemini-3-Pro [ 8 ] | RGB-D observation | 78.08 | 16.87 | 40 | 219 | 15.44 |
| RGB-D + 3D descriptors | 16.76 | 12.00 | 34 | 225 | 13.13 | |
| Claude-Opus-4.5 [ 3 ] | RGB-D observation | 49.49 | 18.00 | 65 | 194 | 25.10 |
| RGB-D + 3D descriptors | 29.09 | 23.57 | 63 | 196 | 24.32 | |
| Claude-Haiku-4.5 [ 2 ] | RGB-D observation | 184.12 | 20.96 | 50 | 209 | 19.31 |
| RGB-D + 3D descriptors | 30.34 | 16.67 | 29 | 230 | 11.20 |
Table 4: Manual planning and spatial grounding before execution. Matched and missing steps count correctly identified and omitted assembly steps; coverage is the fraction of required steps matched. Position and orientation errors measure placement accuracy over matched steps. Position and orientation errors are computed on matched steps. 3D denotes structured world-frame descriptors added to RGB-D observations. More details about 3D descriptors are in supplementary.
| w/ Time Penalty | w/o Time Penalty | |||||||||||
| Model | Reflection Framework | Difficulty | R1 | R2 | R3 | R1 R2 | R2 R3 | R1 | R2 | R3 | R1 R2 | R2 R3 |
| (%) | (%) | (%) | (%) | |||||||||
| Easy | 13.926 | 9.922 | 9.361 | 14.311 | 13.079 | 11.220 | ||||||
| Medium | 12.617 | 11.352 | 10.241 | 13.651 | 12.320 | 11.760 | ||||||
| Hard | 11.770 | 11.891 | 9.218 | 12.452 | 13.475 | 10.703 | ||||||
| Standard Reflection | Overall | 12.771 | 11.055 | 9.607 | 13.471 | 12.958 | 11.228 | |||||
Table 5: Performance across reflection rounds with the task-oriented tool library. R1-R3 denote scores after one to three reflection rounds, respectively. Prior-task summaries are distilled from completed assembly tasks. Percentage changes are relative to the preceding round, and Overall is the macro-average across Low, Medium, and High. Blue highlights the largest improvement in each percentage column.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Overview of the benchmark construction and evaluation pipeline. Visual LEGO instructions are first aligned with their corresponding 3D structural models. Long construction procedures are then decomposed into consecutive, self-contained episodes, each defined by an initial state, a set of required bricks, and a target state. These episodes are instantiated as physically interactive robotic assembly environments in NVIDIA Isaac Sim. During evaluation, a method observes the current scene, produces robot actions, and receives updated observations through closed-loop interaction. Performance is measured from the resulting construction state using the proposed Soft Score.
Figure 4: Distribution of the LOGO training set. (a) Distribution of assembly horizons measured by the number of action bricks required in each task. (b) Distribution of initial-scene complexity measured by the number of bricks present before execution. (c) Joint distribution of assembly horizon and initial-scene complexity, where each cell reports the horizon composition within the corresponding scene-complexity interval.
Figure 5: Distributions of the LOGO validation set and the selected JARVIS Benchmark. (a) Assembly-horizon distributions measured by the number of action bricks required in each task. (b) Initial-scene complexity distributions measured by the number of bricks present before execution. (c) Benchmark coverage over the joint distribution of assembly horizon and initial-scene complexity, where each cell reports the number of selected Benchmark tasks relative to the number of eligible validation tasks in the corresponding interval, together with the selection rate.
Figure 6: Distribution of the 1,067 eligible VAL tasks and the selected Benchmark-50 tasks across normalized scene complexity and action horizon. For each occupied grid cell, color intensity represents the mean normalized difficulty of the tasks within that cell, with lighter and darker colors indicating lower and higher difficulty, respectively. Empty cells contain no eligible tasks. The overlaid markers denote the selected Easy, Medium, and Hard tasks, while the dashed boundaries indicate the 40th and 70th percentiles used to define the three difficulty strata.
Figure 7: A representative Hard assembly task from Benchmark-50. The initial structure contains 74 pre-assembled bricks, and the agent must place 29 additional bricks to obtain the 103-brick target construction. The task has and , yielding and a normalized difficulty of among the 1,067 eligible VAL tasks. Its position in the candidate-task distribution is marked in red.
Figure 8: Example of grounded program execution in the interactive assembly environment. The left panel shows four stages from an actual Isaac Sim execution: approaching the selected brick, grasping and lifting it, transferring it above the target, and vertically placing and releasing it. The right panel shows the corresponding high-level program generated for the same operation. The program specifies the selected brick and target pose, while the shared execution primitives handle pose validation, grasp verification, collision-aware motion, and failure recovery.
Figure 9: No-reflection execution comparison on Task 000003. The four frames in each row show the initial scene, approach and grasp, transfer, and final placement generated by the corresponding model. Gemini-3-Pro, Claude Opus-4.5, and DeepSeek-V4.1 obtain soft scores of , , and , respectively. All three models complete the manipulation sequence, with the remaining performance differences primarily reflecting final placement accuracy.
Figure 10: No-reflection execution comparison on Task 000024. Each row presents four representative stages of execution for one backbone model. Gemini-3-Pro, Claude Opus-4.5, and DeepSeek-V4.1 achieve soft scores of , , and , respectively. Their closely grouped scores indicate similar execution quality, with only minor differences in the final placement pose.
Figure 11: No-reflection execution comparison on Task 000048. Gemini-3-Pro, Claude Opus-4.5, and DeepSeek-V4.1 obtain soft scores of , , and , respectively. All three models generate executable manipulation sequences, but their final placement accuracy differs substantially. DeepSeek-V4.1 produces the highest-quality final state, whereas the larger residual pose error of Gemini-3-Pro results in a lower soft score.
Figure 12: Comparison of Gemini-3-Pro execution with and without structured reflection on the same assembly task. Without reflection, an unsuccessful or misaligned grasp remains undetected and propagates to subsequent actions, producing a soft score of . With structured reflection, the model uses updated visual observations to reassess the workspace state and generate corrective actions, improving the soft score to . The numbered frames show representative stages from the initial scene to the final assembly state.
Figure 13: Comparison of non-reflective execution by Claude Opus-4.5 and GPT-5.1 on the same low-scoring task. The numbered frames show representative stages from the initial scene through grasping and transfer to the final state. Although both models generate executable manipulation sequences, unintended collisions with surrounding LEGO bricks disturb the existing structure and cause bricks to shift or scatter. Without visual feedback and corrective replanning, the resulting scene deviation persists until termination, yielding soft scores of for Claude Opus-4.5 and for GPT-5.1.
Figure 14: Comparison between Standard Reflection and Reflection + Prior-Task Experience Summary on the same Gemini-3-Pro assembly task. The numbered frames show representative stages of the two executions. Incorporating prior-task experience produces a more accurate placement, particularly in orientation, increasing the soft score from to . Although the pose error is reduced, neither setting satisfies the strict completion criterion.
| Model | Effective actions | Average score |
| Gemini 3 Pro | 150 | 24.25 |
| Claude Opus 4.5 | 138 | 19.68 |
| Claude Haiku 4.5 | 112 | 17.66 |
| DeepSeek-V4.1 | 142 | 22.66 |
| Qwen3-VL-235B | 127 | 19.23 |
| Kimi-K2 | 112 | 17.12 |
Table 6: Effective-action performance under ground-truth-guided assembly. Scores are averaged over effective actions against brick-specific ground truth, without execution-time penalties.
More in this topic
Robotic Assembly
Robotic assembly studies how robots perceive, plan, and join parts reliably, from precision insertion to collaborative construction.
See all 88 papers