MAGAR: Multimodal Agents for GUI-Based 3D Mechanical Assembly and Minimal Rework

Jingxuan Wei; Jiahao Fu; Fuqiang Wang; Xiaoming Wu; Yuwen Liu; Kaiwen Wei; Ming Yang; Junnan Zhu

cs.AINov 2, 2026

MAGAR: Multimodal Agents for GUI-Based 3D Mechanical Assembly and Minimal Rework

Authors: Jingxuan Wei, Jiahao Fu, Fuqiang Wang, Xiaoming Wu, Yuwen Liu, Kaiwen Wei, Ming Yang, Junnan Zhu

Organizations: University of Chinese Academy of Sciences, Beijing, China · Qilu University of Technology (Shandong Academy of Sciences), Shandong Computer Science Center, Jinan, China · China University of Petroleum (East China), China · College of Computer Science, Chongqing University, Chongqing, China ·

Abstract

A GUI command can be correct at the interface level yet still produce an incorrect assembly. An agent must identify the relevant components, respect their structural relationships, and determine which parts should move during recovery. We introduce MAGAR-Bench, a benchmark of 4,000 multimodal samples from 26 assembly families that separates Assembly Decision, Atomic Action, and Minimal Rework. For rework, the objective counts distinct moved parts within the validated recovery candidate set for each state, rather than the number of actions. We also introduce MAGAR, a selective correction framework built on a frozen policy obtained by supervised fine-tuning. Its key design matches correction granularity to output dependencies. Structured candidates revise assembly fields and rework scope, while complete action candidates keep native GUI primitives coupled with their parameters. A learned selector retains the original prediction unless a candidate passes a task-specific intervention threshold. On recorded states from held-out assembly families, MAGAR improves Overall, the task-balanced aggregate, from 37.50 to 41.39, with gains on all three tasks. Across 209 interventions, 70 correct an error and 22 introduce one. These results support task-specific correction while highlighting the remaining tradeoff between fixing errors and preserving reliable predictions.

Figures & tables

Figure 1: Motivation of MAGAR. Locally executable GUI actions may still violate the underlying assembly structure.

Figure 2: Overview of MAGAR. A frozen SFT backbone provides task-specific predictions, while Module A performs structured assembly and rework intervention and Module B performs complete-action intervention.

Figure 3: Construction pipeline of MAGAR-Bench.

Model	Assembly Decision				Atomic Action			Minimal Rework		Overall $\uparrow$	LLM-as-judge
Model	OPG $\uparrow$	OBA $\uparrow$	SGF $\uparrow$	IBC $\uparrow$	ATA $\uparrow$	PCA $\uparrow$	WCS $\uparrow$	MPA $\uparrow$	RCA $\uparrow$	Overall $\uparrow$	Score $\uparrow$
Open-source multimodal models
Qwen3-VL-8B-Instruct ( Bai et al., 2025 )	28.97	25.31	7.56	3.78	28.40	16.81	16.65	6.00	4.25	14.05	1.66
DeepSeek-VL2-Small ( Wu et al., 2024 )	0.47	3.72	3.78	—	5.89	—	1.58	3.00	7.00	3.16	0.41
InternVL2.5-8B ( Chen et al., 2024 )	8.88	17.57	10.50	3.36	29.91	9.37	13.41	5.00	2.50	10.46	1.10
Kimi-VL-A3B-Instruct ( Team et al., 2025 )	0.93	5.85	3.78	—	29.91	18.84	17.96	7.00	8.00	10.79	0.88

Table 1: Main results on recorded task instances. Models are grouped into open-source multimodal models, proprietary multimodal models, and GUI agent baselines. The best results are highlighted in bold, and the second-best results are underlined.

Model	Assembly Decision				Atomic Action			Minimal Rework		Overall $\uparrow$	LLM-as-judge
Model	OPG $\uparrow$	OBA $\uparrow$	SGF $\uparrow$	IBC $\uparrow$	ATA $\uparrow$	PCA $\uparrow$	WCS $\uparrow$	MPA $\uparrow$	RCA $\uparrow$	Overall $\uparrow$	Score $\uparrow$
Vanilla-SFT	36.45	32.76	42.86	33.61	43.96	29.37	50.64	24.00	45.50	37.50	2.66
SFT + Module A	39.25	33.62	44.54	35.29	43.96	29.37	50.64	30.00	48.75	39.63	2.67
SFT + Module B	36.45	32.76	42.86	33.61	48.19	37.39	54.24	24.00	45.50	39.26	2.79
MAGAR(Ours)	39.25	33.62	44.54	35.29	48.19	37.39	54.24	30.00	48.75	41.39	2.82

Table 2: Ablation of the two intervention modules. The best results are highlighted in bold.

Figure 5: Representative MAGAR corrections across three task families.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: The curves report Overall, Assembly Decision, Atomic Action, and the task-balanced Minimal Rework score, computed as the mean of MPA and RCA.

Benchmark / Work	GUI	Config. Target	3D Assembly	Error Repair	Min. Moved Parts
VisualWebArena ( Koh et al., 2024 )	$\checkmark$	X	X	X	X
OSWorld ( Xie et al., 2024 )	$\checkmark$	X	X	X	X
AndroidWorld ( Rawles et al., 2025 )	$\checkmark$	X	X	X	X
PAGE Bench / PAGER ( Wei et al., 2026 )	$\checkmark$	$\checkmark$	X	X	X
LEGO-ECA / SCANet ( Wan et al., 2024 )	X	$\checkmark$	$\checkmark$	$\checkmark$	X
FurnitureBench ( Heo et al., 2025 )	X	$\checkmark$	$\checkmark$	X	X

Appendix

Table 3: Capability coverage of related benchmarks. A check mark denotes explicit coverage, and X denotes that the capability is not a dedicated evaluation target.

Setting	Value
Backbone	Qwen3-VL-8B-Instruct
Adaptation	LoRA
LoRA rank / scaling factor	8 / 32
LoRA dropout	0.05
Frozen components	Visual encoder and multimodal aligner
Optimizer	AdamW

Appendix

Table 4: Backbone adaptation and decoding configuration.

Figure 8: Fine-grained performance across task subcategories. Rows denote subcategories of Assembly Decision, Atomic Action, and Minimal Rework, and columns denote evaluated models.

Task family	Test samples	Interventions	Rate (%)
Assembly Decision	238	14	5.88
Atomic Action	662	184	27.79
Minimal Rework	100	11	11.00
All tasks	1,000	209	20.90

Appendix

Table 5: Intervention frequency by task family. Rates use the corresponding Test-set sizes.

Multimodal Agents

Multimodal agents perceive and reason across text, images, audio, and video to use tools, interact with interfaces, and act in the world.

See all 396 papers

AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents
Sep 30, 2026Jiahao Zhang, Yeying Fan, Moitreya Chatterjee +5Robot Manipulation Benchmarks Robotic Assembly
The Australian National University · Tsinghua University · Mitsubishi Electric Research Laboratories (MERL) +1
FORGE: Fault-Oriented Rework Code Generation for Executable Interlocking Assembly
Nov 2, 2026Fuqiang Wang, Jiahao Fu, Ren Hefeng +5Robotic Program Synthesis Robot Failure Recovery
Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan 250014, China · Shandong Provincial Key Laboratory of Industrial Network and Information System Security, Shandong Fundamental Research Center for Computer Science, Jinan 250014, China · Quan Cheng Laboratory, Jinan, China +4
REBOOT: From Failure to Recovery - A Dataset and Benchmark for Precision Assembly
Sep 18, 2026Nana Yaw Owusu Ofori-Ampofo, Samira Ebrahimi Kahou, Joseph ThekinenRobot Manipulation Benchmarks Robot Manipulation Datasets
University of Calgary · University of Calgary, Mila

Sep 30, 2026J/K move · Enter open · S save

AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents

Jiahao Zhang, Yeying Fan, Moitreya Chatterjee, Suhas Lohit+4

The Australian National University · Tsinghua University · Mitsubishi Electric Research Laboratories (MERL)+1

Robot Manipulation Benchmarks Robotic Assembly

Open paper AskPDF arXiv

The task of 3D assembly requires translating an understanding of parts and their relationships into precise spatial arrangements. Can pretrained general-purpose agents assemble objects through visual interaction without additional assembly-specific fine-tuning? To investigate this question, we introduce AssemblyWorld, an interactive 3D environment in which agents inspect rendered views and manipulate supplied rigid parts, guided by images or assembly manuals when available. Agents perceive part geometry through 2D views rather than direct access to mesh vertices or faces, while their resulting assemblies are evaluated geometrically. Building on this environment, we construct AssemblyWorldBench, comprising 100 assembly tasks across 80 objects spanning furniture, industrial assembly, and fracture reassembly. Evaluating eight agent systems reveals substantial differences in their capabilities. The strongest system achieves 80.9% part accuracy but 59.4% complete-assembly success. The evaluated open-source systems lag substantially behind their stronger closed-source peers in both execution reliability and assembly accuracy. Analyses of visual references, interaction trajectories, and failures show how agents revise assemblies while leaving residual positioning errors. AssemblyWorld provides a common setting for both assessing the capabilities of interactive assembly agents and characterizing the gap between approximate structure recovery and precise reconstruction.

Abstract

Multimodal Agents

Explore similar work

AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents

FORGE: Fault-Oriented Rework Code Generation for Executable Interlocking Assembly

REBOOT: From Failure to Recovery - A Dataset and Benchmark for Precision Assembly