Organizations: University of Chinese Academy of Sciences, Beijing, China · Qilu University of Technology (Shandong Academy of Sciences), Shandong Computer Science Center, Jinan, China · China University of Petroleum (East China), China · College of Computer Science, Chongqing University, Chongqing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China
A GUI command can be correct at the interface level yet still produce an incorrect assembly. An agent must identify the relevant components, respect their structural relationships, and determine which parts should move during recovery. We introduce MAGAR-Bench, a benchmark of 4,000 multimodal samples from 26 assembly families that separates Assembly Decision, Atomic Action, and Minimal Rework. For rework, the objective counts distinct moved parts within the validated recovery candidate set for each state, rather than the number of actions. We also introduce MAGAR, a selective correction framework built on a frozen policy obtained by supervised fine-tuning. Its key design matches correction granularity to output dependencies. Structured candidates revise assembly fields and rework scope, while complete action candidates keep native GUI primitives coupled with their parameters. A learned selector retains the original prediction unless a candidate passes a task-specific intervention threshold. On recorded states from held-out assembly families, MAGAR improves Overall, the task-balanced aggregate, from 37.50 to 41.39, with gains on all three tasks. Across 209 interventions, 70 correct an error and 22 introduce one. These results support task-specific correction while highlighting the remaining tradeoff between fixing errors and preserving reliable predictions.
Figures & tables
Figure 1: Motivation of MAGAR. Locally executable GUI actions may still violate the underlying assembly structure.
Figure 2: Overview of MAGAR. A frozen SFT backbone provides task-specific predictions, while Module A performs structured assembly and rework intervention and Module B performs complete-action intervention.
Figure 3: Construction pipeline of MAGAR-Bench.
Model
Assembly Decision
Atomic Action
Minimal Rework
Overall ↑
LLM-as-judge
OPG ↑
OBA ↑
SGF ↑
IBC ↑
ATA ↑
PCA ↑
WCS ↑
MPA ↑
RCA ↑
Score ↑
Open-source multimodal models
Qwen3-VL-8B-Instruct ( Bai et al., 2025 )
28.97
25.31
7.56
3.78
28.40
16.81
16.65
6.00
4.25
14.05
1.66
DeepSeek-VL2-Small ( Wu et al., 2024 )
0.47
3.72
3.78
—
5.89
—
1.58
3.00
7.00
3.16
0.41
InternVL2.5-8B ( Chen et al., 2024 )
8.88
17.57
10.50
3.36
29.91
9.37
13.41
5.00
2.50
10.46
1.10
Kimi-VL-A3B-Instruct ( Team et al., 2025 )
0.93
5.85
3.78
—
29.91
18.84
17.96
7.00
8.00
10.79
0.88
Table 1: Main results on recorded task instances. Models are grouped into open-source multimodal models, proprietary multimodal models, and GUI agent baselines. The best results are highlighted in bold, and the second-best results are underlined.
Model
Assembly Decision
Atomic Action
Minimal Rework
Overall ↑
LLM-as-judge
OPG ↑
OBA ↑
SGF ↑
IBC ↑
ATA ↑
PCA ↑
WCS ↑
MPA ↑
RCA ↑
Score ↑
Vanilla-SFT
36.45
32.76
42.86
33.61
43.96
29.37
50.64
24.00
45.50
37.50
2.66
SFT + Module A
39.25
33.62
44.54
35.29
43.96
29.37
50.64
30.00
48.75
39.63
2.67
SFT + Module B
36.45
32.76
42.86
33.61
48.19
37.39
54.24
24.00
45.50
39.26
2.79
MAGAR(Ours)
39.25
33.62
44.54
35.29
48.19
37.39
54.24
30.00
48.75
41.39
2.82
Table 2: Ablation of the two intervention modules. The best results are highlighted in bold.
Figure 5: Representative MAGAR corrections across three task families.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The curves report Overall, Assembly Decision, Atomic Action, and the task-balanced Minimal Rework score, computed as the mean of MPA and RCA.
Benchmark / Work
GUI
Config. Target
3D Assembly
Error Repair
Min. Moved Parts
VisualWebArena ( Koh et al., 2024 )
✓
X
X
X
X
OSWorld ( Xie et al., 2024 )
✓
X
X
X
X
AndroidWorld ( Rawles et al., 2025 )
✓
X
X
X
X
PAGE Bench / PAGER ( Wei et al., 2026 )
✓
✓
X
X
X
LEGO-ECA / SCANet ( Wan et al., 2024 )
X
✓
✓
✓
X
FurnitureBench ( Heo et al., 2025 )
X
✓
✓
X
X
Appendix
Table 3: Capability coverage of related benchmarks. A check mark denotes explicit coverage, and X denotes that the capability is not a dedicated evaluation target.
Setting
Value
Backbone
Qwen3-VL-8B-Instruct
Adaptation
LoRA
LoRA rank / scaling factor
8 / 32
LoRA dropout
0.05
Frozen components
Visual encoder and multimodal aligner
Optimizer
AdamW
Appendix
Table 4: Backbone adaptation and decoding configuration.
Figure 8: Fine-grained performance across task subcategories. Rows denote subcategories of Assembly Decision, Atomic Action, and Minimal Rework, and columns denote evaluated models.
Task family
Test samples
Interventions
Rate (%)
Assembly Decision
238
14
5.88
Atomic Action
662
184
27.79
Minimal Rework
100
11
11.00
All tasks
1,000
209
20.90
Appendix
Table 5: Intervention frequency by task family. Rates use the corresponding Test-set sizes.
The task of 3D assembly requires translating an understanding of parts and their relationships into precise spatial arrangements. Can pretrained general-purpose agents assemble objects through visual interaction without additional assembly-specific fine-tuning? To investigate this question, we introduce AssemblyWorld, an interactive 3D environment in which agents inspect rendered views and manipulate supplied rigid parts, guided by images or assembly manuals when available. Agents perceive part geometry through 2D views rather than direct access to mesh vertices or faces, while their resulting assemblies are evaluated geometrically. Building on this environment, we construct AssemblyWorldBench, comprising 100 assembly tasks across 80 objects spanning furniture, industrial assembly, and fracture reassembly. Evaluating eight agent systems reveals substantial differences in their capabilities. The strongest system achieves 80.9% part accuracy but 59.4% complete-assembly success. The evaluated open-source systems lag substantially behind their stronger closed-source peers in both execution reliability and assembly accuracy. Analyses of visual references, interaction trajectories, and failures show how agents revise assemblies while leaving residual positioning errors. AssemblyWorld provides a common setting for both assessing the capabilities of interactive assembly agents and characterizing the gap between approximate structure recovery and precise reconstruction.
Jiahao Zhang, Yeying Fan, Moitreya Chatterjee +5
The Australian National University · Tsinghua University · Mitsubishi Electric Research Laboratories (MERL) +1
Large language models have demonstrated strong capabilities in structured reasoning and code generation, yet identifying an assembly fault does not guarantee the ability to generate executable rework programs. In interlocking assemblies, a locally plausible repair program may violate blocking dependencies, omit required recovery operations, or terminate in an incorrect configuration. We identify this challenge as fault-conditioned executable rework generation, where models must jointly reason about structural failures and generate programs that restore constrained assembly states. To benchmark this capability, we introduce FORGE-Bench, containing 4,000 tasks across 26 interlocking assemblies with five assemblies held out for evaluating generalization to unseen structures. We further propose FORGE (Fault-Oriented Rework Code Generation for Executable Interlocking Assembly), a structured state-to-program framework that decomposes rework generation into Rework Structuring and Action Grounding. Rework Structuring captures fault-induced recovery dependencies, while Action Grounding maps these dependencies into state-conditioned executable actions. Experiments show that locally reasonable repair code is insufficient for reliable assembly recovery, and that connecting structural recovery reasoning with executable action realization is essential for verified target restoration.
Fuqiang Wang, Jiahao Fu, Ren Hefeng +5
Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan 250014, China · Shandong Provincial Key Laboratory of Industrial Network and Information System Security, Shandong Fundamental Research Center for Computer Science, Jinan 250014, China · Quan Cheng Laboratory, Jinan, China +4
Robot learning policies fail in characteristic ways: they stall in uncertain states, drift during contact-rich alignment, and miss targets by millimetres in precision tasks. Yet training datasets consist largely of successful demonstrations, while real-world benchmarks often reduce performance to binary success. This limits both supervision for recovery and analysis of where failures occur. We introduce REBOOT (Recovery Episode Benchmark for Off-nominal Trajectories), the first robot manipulation benchmark designed around failure as a first-class signal. REBOOT contains 2,160 demonstrations across 18 precision assembly tasks, each decomposed into five shared phases: Align(pick), Engage(pick), Transport, Align(place), and Engage(place), enabling phase-level evaluation beyond terminal success. Failures are introduced across phases and paired with expert recovery trajectories that return the system to a valid continuation state. Tasks are annotated with rotational symmetry, engagement-clearance precision tier, and assembly direction through matched install-remove pairs. Failure episodes are labeled by phase and categorical failure mode, enabling attribution to kinematic stage and tolerance violation. Data includes synchronized RGB-D observations from four viewpoints and grounded natural-language descriptions of phase-level success and failure conditions. Half the dataset contains expert demonstrations; the other half contains recovery demonstrations sampled to reflect failures observed in imitation-learned policy rollouts. We benchmark action-chunked transformer, diffusion, and π0-FAST policies using phase-level completion rates, revealing model-specific failure points hidden by binary evaluation. Dataset and code: https://nanayawoa.github.io/REBOOT
Nana Yaw Owusu Ofori-Ampofo, Samira Ebrahimi Kahou, Joseph Thekinen
University of Calgary · University of Calgary, Mila