Visual manuals communicate procedural knowledge through stepwise diagrams, but translating these instructions into reliable long-horizon robotic execution remains challenging. We introduce JARVIS, a method-agnostic benchmark for autonomous visual-manual-guided execution without additional language instructions. JARVIS comprises 236,679 LEGO assembly tasks across 8,987 scenes, with a carefully curated JARVIS-v1 suite of 50 tasks for interactive robotic evaluation. Tasks span varying initial structural complexities and assembly horizons, requiring agents to interpret manuals, ground actions in an evolving workspace, and execute the intended procedure. A structure-aware evaluation protocol captures geometric accuracy, structural relations, construction order, partial progress, and completion. We further introduce JARVIS-agentic, a tool-grounded embodied agent that composes robot primitives into executable programs and uses execution feedback for verification and recovery, without task-specific action trajectories or model parameter updates. Across nine multimodal and reasoning models, task-oriented primitives increase the average assembly score. Performance nevertheless declines sharply with assembly horizon, and execution errors remain difficult to correct. Prior-task experience improves initial performance, while repeated reflection provides inconsistent benefits, highlighting persistent limitations in connecting visual procedural reasoning with reliable physical execution.
Figures & tables
Figure 1: Overview of the visual planning, execution, and reflection workflow in JARVIS.
Split
#Tasks
Assembly Horizon
Initial Complexity Ninit
Total Bricks
avg.
range
avg.
range
avg.
range
Easy
20
2.45
1-4
33.05
2-105
35.50
4-107
Medium
15
4.13
2-7
78.80
2-213
82.93
7-215
Hard
15
9.87
3-29
127.00
57-236
136.87
75-239
Table 1: Statistics of the JARVIS-v1 benchmark.
Figure 2: Overview of JARVIS-V1. The framework integrates multimodal task context, agentic code generation, simulation execution, evaluation, and self-reflection in a closed-loop assembly process.
Model
Tool library
Soft Score ↑
Valid (%) ↑
Failed actions per task ↓
Easy
Medium
Hard
Overall
Gemini-3-Pro [ 8 ]
Generic
13.27
6.61
4.68
8.19
98.00
6.04
Task-oriented
22.16
19.30
9.94
17.13
100.00
4.96
Claude-Opus-4.5 [ 3 ]
Generic
15.84
16.13
9.51
13.83
92.00
7.36
Task-oriented
22.20
18.58
10.35
17.04
100.00
5.31
Claude-Haiku-4.5 [ 2 ]
Generic
11.69
3.22
2.88
5.93
90.00
6.70
Table 2: Model and tool-library performance across task difficulty levels. Underline and blue denote the best generic and task-oriented results; bold marks the best Overall score within each library.
Assembly horizon Nact
Model
Initial complexity Ninit
1–2
3–5
6+
Gemini-3-Pro [ 8 ]
0–29
43.42 (5)
15.40 (10)
–
30–78
28.03 (4)
17.99 (8)
10.40 (6)
79+
16.40 (4)
12.96 (7)
6.01 (6)
Claude-Opus-4.5 [ 3 ]
0–29
36.46 (5)
18.38 (10)
–
30–78
30.87 (4)
14.47 (8)
11.88 (6)
Table 3: Joint effect of initial scene complexity and assembly horizon. Each cell reports the mean soft score, with the number of evaluated tasks in parentheses. Results are computed from the latest no-reflection execution round for each model.
Model
Input
Position Error (mm) ↓
Orientation Error ( ∘ ) ↓
Matched Steps ↑
Missing Steps ↓
Coverage (%) ↑
Gemini-3-Pro [ 8 ]
RGB-D observation
78.08
16.87
40
219
15.44
RGB-D + 3D descriptors
16.76
12.00
34
225
13.13
Claude-Opus-4.5 [ 3 ]
RGB-D observation
49.49
18.00
65
194
25.10
RGB-D + 3D descriptors
29.09
23.57
63
196
24.32
Claude-Haiku-4.5 [ 2 ]
RGB-D observation
184.12
20.96
50
209
19.31
RGB-D + 3D descriptors
30.34
16.67
29
230
11.20
Table 4: Manual planning and spatial grounding before execution. Matched and missing steps count correctly identified and omitted assembly steps; coverage is the fraction of required steps matched. Position and orientation errors measure placement accuracy over matched steps. Position and orientation errors are computed on matched steps. 3D denotes structured world-frame descriptors added to RGB-D observations. More details about 3D descriptors are in supplementary.
w/ Time Penalty
w/o Time Penalty
Model
Reflection Framework
Difficulty
R1
R2
R3
R1 → R2
R2 → R3
R1
R2
R3
R1 → R2
R2 → R3
(%)
(%)
(%)
(%)
Easy
13.926
9.922
9.361
−28.75
−5.65
14.311
13.079
11.220
−8.61
−14.21
Medium
12.617
11.352
10.241
−10.03
−9.79
13.651
12.320
11.760
−9.75
−4.55
Hard
11.770
11.891
9.218
+1.03
−22.48
12.452
13.475
10.703
+8.22
−20.57
Standard Reflection
Overall
12.771
11.055
9.607
−13.44
−13.10
13.471
12.958
11.228
−3.81
−13.35
Table 5: Performance across reflection rounds with the task-oriented tool library. R1-R3 denote scores after one to three reflection rounds, respectively. Prior-task summaries are distilled from completed assembly tasks. Percentage changes are relative to the preceding round, and Overall is the macro-average across Low, Medium, and High. Blue highlights the largest improvement in each percentage column.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Overview of the benchmark construction and evaluation pipeline. Visual LEGO instructions are first aligned with their corresponding 3D structural models. Long construction procedures are then decomposed into consecutive, self-contained episodes, each defined by an initial state, a set of required bricks, and a target state. These episodes are instantiated as physically interactive robotic assembly environments in NVIDIA Isaac Sim. During evaluation, a method observes the current scene, produces robot actions, and receives updated observations through closed-loop interaction. Performance is measured from the resulting construction state using the proposed Soft Score.
Figure 4: Distribution of the LOGO training set. (a) Distribution of assembly horizons measured by the number of action bricks required in each task. (b) Distribution of initial-scene complexity measured by the number of bricks present before execution. (c) Joint distribution of assembly horizon and initial-scene complexity, where each cell reports the horizon composition within the corresponding scene-complexity interval.
Figure 5: Distributions of the LOGO validation set and the selected JARVIS Benchmark. (a) Assembly-horizon distributions measured by the number of action bricks required in each task. (b) Initial-scene complexity distributions measured by the number of bricks present before execution. (c) Benchmark coverage over the joint distribution of assembly horizon and initial-scene complexity, where each cell reports the number of selected Benchmark tasks relative to the number of eligible validation tasks in the corresponding interval, together with the selection rate.
Figure 6: Distribution of the 1,067 eligible VAL tasks and the selected Benchmark-50 tasks across normalized scene complexity and action horizon. For each occupied grid cell, color intensity represents the mean normalized difficulty of the tasks within that cell, with lighter and darker colors indicating lower and higher difficulty, respectively. Empty cells contain no eligible tasks. The overlaid markers denote the selected Easy, Medium, and Hard tasks, while the dashed boundaries indicate the 40th and 70th percentiles used to define the three difficulty strata.
Figure 7: A representative Hard assembly task from Benchmark-50. The initial structure contains 74 pre-assembled bricks, and the agent must place 29 additional bricks to obtain the 103-brick target construction. The task has Sscene=0.244 and Sact=1.000 , yielding D=0.3Sscene+0.7Sact=0.773 and a normalized difficulty of 1.00 among the 1,067 eligible VAL tasks. Its position in the candidate-task distribution is marked in red.
Figure 8: Example of grounded program execution in the interactive assembly environment. The left panel shows four stages from an actual Isaac Sim execution: approaching the selected brick, grasping and lifting it, transferring it above the target, and vertically placing and releasing it. The right panel shows the corresponding high-level program generated for the same operation. The program specifies the selected brick and target pose, while the shared execution primitives handle pose validation, grasp verification, collision-aware motion, and failure recovery.
Figure 9: No-reflection execution comparison on Task 000003. The four frames in each row show the initial scene, approach and grasp, transfer, and final placement generated by the corresponding model. Gemini-3-Pro, Claude Opus-4.5, and DeepSeek-V4.1 obtain soft scores of 66.67 , 63.94 , and 67.39 , respectively. All three models complete the manipulation sequence, with the remaining performance differences primarily reflecting final placement accuracy.
Figure 10: No-reflection execution comparison on Task 000024. Each row presents four representative stages of execution for one backbone model. Gemini-3-Pro, Claude Opus-4.5, and DeepSeek-V4.1 achieve soft scores of 59.26 , 58.57 , and 59.64 , respectively. Their closely grouped scores indicate similar execution quality, with only minor differences in the final placement pose.
Figure 11: No-reflection execution comparison on Task 000048. Gemini-3-Pro, Claude Opus-4.5, and DeepSeek-V4.1 obtain soft scores of 42.68 , 55.06 , and 57.78 , respectively. All three models generate executable manipulation sequences, but their final placement accuracy differs substantially. DeepSeek-V4.1 produces the highest-quality final state, whereas the larger residual pose error of Gemini-3-Pro results in a lower soft score.
Figure 12: Comparison of Gemini-3-Pro execution with and without structured reflection on the same assembly task. Without reflection, an unsuccessful or misaligned grasp remains undetected and propagates to subsequent actions, producing a soft score of 0.00 . With structured reflection, the model uses updated visual observations to reassess the workspace state and generate corrective actions, improving the soft score to 17.05 . The numbered frames show representative stages from the initial scene to the final assembly state.
Figure 13: Comparison of non-reflective execution by Claude Opus-4.5 and GPT-5.1 on the same low-scoring task. The numbered frames show representative stages from the initial scene through grasping and transfer to the final state. Although both models generate executable manipulation sequences, unintended collisions with surrounding LEGO bricks disturb the existing structure and cause bricks to shift or scatter. Without visual feedback and corrective replanning, the resulting scene deviation persists until termination, yielding soft scores of 0.43 for Claude Opus-4.5 and 0.00 for GPT-5.1.
Figure 14: Comparison between Standard Reflection and Reflection + Prior-Task Experience Summary on the same Gemini-3-Pro assembly task. The numbered frames show representative stages of the two executions. Incorporating prior-task experience produces a more accurate placement, particularly in orientation, increasing the soft score from 45.12 to 56.32 . Although the pose error is reduced, neither setting satisfies the strict completion criterion.
Model
Effective actions
Average score
Gemini 3 Pro
150
24.25
Claude Opus 4.5
138
19.68
Claude Haiku 4.5
112
17.66
DeepSeek-V4.1
142
22.66
Qwen3-VL-235B
127
19.23
Kimi-K2
112
17.12
Appendix
Table 6: Effective-action performance under ground-truth-guided assembly. Scores are averaged over effective actions against brick-specific ground truth, without execution-time penalties.
We introduceWorkBenchMark, a LEGO Duplo-based robotic assembly benchmark motivated by the RoboCup Smart Manufacturing League. Robotic assembly couples low-level manipulation with task-level symbolic reasoning under physical constraints, a combination that current end-to-end learning methods do not yet solve reliably. The benchmark provides 400 tasks across four complexity tiers. We provide an open-vocabulary perception, Assembly-by-Disassembly baseline solution. Our planning-based pipeline outperforms a modern vision-language-action approach across all tiers. The benchmark, simulation environment, and baseline implementation will be released openly to support the broader robotic assembly community.
Wenbo Ma, Daniel Swoboda, Matteo Tschesche +1
Chair of Machine Learning and Reasoning (i6), RWTH Aachen University, Aachen, Germany · MASCOR Institute, FH Aachen University of Applied Sciences, Aachen, Germany
Autonomous robotic assembly of interlocking bricks demands seamless integration of long-horizon task reasoning, spatial grounding, and fine-grained manipulation. This paper presents BrickCraft, a compositional framework designed for long-horizon and generalizable interlocking brick assembly. BrickCraft models the assembly process using a relative formulation, where each step is anchored to a reference brick within the partial structure, thereby decomposing complex tasks into a finite set of reusable primitive skills. BrickCraft bridges the gap between high-level assembly plans and physical execution through situated manuals, which provide explicit spatial guidance for learned visuomotor skills by projecting the assembly intent onto real-time robot observations. Finally, BrickCraft employs a modular execution pipeline that chains these spatially grounded skills to accomplish long-horizon assembly tasks. Extensive experimental validations demonstrate that BrickCraft acquires proficient assembly skills from a limited set of demonstrations and effectively composes them for diverse long-horizon assembly tasks. The project website is available at https://intelligent-control-lab.github.io/BrickCraft.
Multi-pair robotic assembly in unstructured environments faces spatial interference and contact uncertainties. Existing paradigms fail to bridge cognitive decision-making and physical execution, as they either encounter state-space explosion and knowledge bottlenecks or suffer from logical hallucinations and topological conflicts. We propose an end-to-end neuro-symbolic framework that solves the challenge hierarchically: generating optimal subgraphs for each pair, decoupling generality from edge cases, and then resolving cross-pair interferences. Given an eye-on-hand RGB-D assembly scene, the framework extracts semantic instance identity and state while quantifying the scene for divergence calculation. For each pair, optimal subgraph is generated via LLM using barely basic actions to mitigate hallucinations. Supportive actions for edge cases are reasoned and inserted with a lightweight discriminator. Driven by the divergence between the quantified baseline and current scene, it is easily extensible at low cost. Augmented subgraphs are topologically coordinated into global sequences while preserving internal behavioral coherence. Dynamic behavior trees embedding atomic skills close the force-aware execution loop. Offline evaluation on 100 real-world scenes achieves 97.00% global executability, outperforming classical and state-of-the-art planners. Real-robot deployment on a UR3 arm attains 90% success rate with 0.5 mm tolerance under strong interference, demonstrating a unified and verifiable solution for complex autonomous assembly.
Xinyi Li, Aiguo Song, Linhu Wei +1
School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China.