On-Demand Robotic Assembly via Differentiable Geometric Part Repair

Millicent Schlafly; Fabio Schaub; Diogo Costa Pais; Luca Lelli; Janne Dvorak; Claire Colmont; Sven Marti; Mark D. Fuge

cs.ROOct 7, 2026

On-Demand Robotic Assembly via Differentiable Geometric Part Repair

Authors: Millicent Schlafly, Fabio Schaub, Diogo Costa Pais, Luca Lelli, Janne Dvorak, Claire Colmont, Sven Marti, Mark D. Fuge

Organizations: ETH Zurich, Zurich, Switzerland

Abstract

Transitioning from a digital design to a robotic assembly process currently requires months of expert manual tuning to reconcile part geometries with robotic constraints. This paper presents an end-to-end, autonomous pipeline for the design and physical construction of bespoke wooden assemblies. A generative AI agent translates user prompts into initial 3D geometries, balancing the visual fidelity of the design with select physical constraints. The assemblability of the design is further improved by a gradient-based repair stage that backpropagates through a graph attention network surrogate to adjust component geometries. In addition to correcting for disjointed and overlapping components, we demonstrate hardware-specific corrections, differentiably optimizing the geometry of components to enable robot screwdriving for 86.7% of 60 novel natural language inputs, significantly outperforming prior work by a factor of ten. For ten of the structures, we physically demonstrate assemblability with two UR5e robots. This work marks a meaningful step toward on-demand robotic manufacturing, enabling the rapid production of customized, low-volume goods.

Figures & tables

Fig. 1: Woodworking On-Demand Pipeline. A. A generative AI agent interprets a natural language input to produce $8$ initial geometries, of which many fail due to physics and motion planning checks. B. A differentiable graph attention network, trained from $174,000$ block configurations labeled in NVIDIA Isaac Sim, performs gradient-based optimization to satisfy hardware-specific constraints, such as screwdriver clearance. C. Two UR5e robots physically assemble the optimized design to validate the framework’s feasibility for bespoke parts.

Panel of Figure 2 — Fig. 2: Late-fork GAT architecture: Two shared GATv2Conv layers process the assembly graph, then split into a regression branch predicting standardized overlap and thickness ( $\hat{\boldsymbol{y}}_{\text{reg}}$ ) and a classification branch driving a binary feasibility head ( $\hat{\ell}_{\text{feas}}$ ) and an auxiliary failure-mode head ( $\hat{\boldsymbol{\ell}}_{\text{mode}}$ ). Edge features are reused as attention inputs at every GAT layer with hidden width 128, 4 attention heads and dropout 0.1.

Fig. 3: Final Assemblies for Ten Natural Language Inputs . For each prompt, we show the best design candidate produced by the generative AI agent (left) followed by the modified design (right). We physically construct each object to validate assemblability.

	Gap/ Concurrence	Structural Tipping	Sufficient Overlap	Thickness	Overall Screwable	End-to-End (Full System)
Ours	$93.3\%\;(56)$	$96.7\%\;(58)$	$\mathbf{98.3\%}\;(59)$	$\mathbf{86.7\%}\;(52)$	$\mathbf{86.7\%}\;(52)$	$\mathbf{36.7\%}\;(22)$
Blox-Net	$\mathbf{100\%}\;(60)$	$\mathbf{100\%}\;(60)$	$85.0\%\;(51)$	$8.3\%\;(5)$	$8.3\%\;(5)$	$6.7\%\;(4)$
	$p=0.127$	$p=0.476$	* $\mathbf{p<0.05}$	*** $\mathbf{p<0.001}$	*** $\mathbf{p<0.001}$	*** $\mathbf{p<0.001}$

TABLE I: Benchmarking Results From Simulated Assembly Checks. We present the percentage of

60

designs that passed each assembly check followed by the number of designs in parentheses.

	Gap/ Concurrence	Structural Tipping	Sufficient Overlap	Thickness	Overall Screwable
Ours	$93.3\%\;(56)$	$96.7\%\;(58)$	$\mathbf{98.3\%}\;(59)$	$\mathbf{86.7\%}\;(52)$	$\mathbf{86.7\%}\;(52)$
MLP+KD-Tree Cost	$21.6\%\;(13)$	$71.7\%\;(43)$	$63.3\%\;(38)$	$36.7\%\;(22)$	$11.7\%\;(7)$
Ours w/o GenAI	$95.0\%\;(57)$	$86.7\%\;(52)$	$93.3\%\;(56)$	$81.7\%\;(49)$	$66.7\%\;(40)$
Ours w/o Perturbations	$93.3\%\;(56)$	$96.7\%\;(58)$	$96.7\%\;(58)$	$\mathbf{86.7\%}\;(52)$	$\mathbf{86.7\%}\;(52)$
Ours w/o GAT (Blox-Net)	$\mathbf{100\%}\;(60)$	$\mathbf{100\%}\;(60)$	$85.0\%\;(51)$	$8.3\%\;(5)$	$8.3\%\;(5)$

TABLE II: Ablation Results From Simulated Assembly Checks. We present the percentage of

60

designs that passed each assembly check followed by the number of designs in parentheses. For the condition without GenAI, only one candidate design is created (as opposed to

8

Fig. 4: Remaining Assembly Failures. A. For the castle , the body of the gripper collides with the remaining assembly. B. The bicycle handlebar post remains longer than the available screws after optimization. C. After assemblability modifications for the wishing well and stairs , the designs are no longer visually recognizable.

Explore similar work

End-to-End Autonomous Generation of Human Assembly Plans
Oct 7, 2026Faustin Arion von Arx, Millicent Schlafly, Mark D. Fuge
ETH Zurich Department of Mechanical and Process Engineering Zurich, Switzerland
Geometric 2D Scene Graph Generation
Jul 24, 2026Christoph Jahn, Urs Waldmann, Bastian GoldlueckeScene Graph Generation Graph Representation Learning
1Mercedes-Benz AG, Sindelfingen, Germany · Department of Computer and Information Science, University of Konstanz, Konstanz, Germany
AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents
Sep 30, 2026Jiahao Zhang, Yeying Fan, Moitreya Chatterjee +5Robot Manipulation Benchmarks Robotic Assembly
The Australian National University · Tsinghua University · Mitsubishi Electric Research Laboratories (MERL) +1

Oct 7, 2026, cs.ROJ/K move · Enter open · S save

End-to-End Autonomous Generation of Human Assembly Plans

Faustin Arion von Arx, Millicent Schlafly, Mark D. Fuge

ETH Zurich Department of Mechanical and Process Engineering Zurich, Switzerland

Open paper AskPDF arXiv

Turning a CAD design into an assembly plan is still largely done by hand, requiring engineers to reason about geometric feasibility, tool access, stability, and the ergonomics of human assembly. In this work, we encode long-established design for assembly (DfA) principles into a contained, end-to-end approach for generating assembly plans. Our approach takes only a mesh assembly and produces either a step-by-step assembly manual or a structured failure report, requiring no joint metadata, fastener annotations, or additional information. Four major components of a manufacturing plan are addressed autonomously: an assembly tool list, the assembly sequence and subassemblies, an assembly manual, and design feedback for improving assemblability. For determining the sequence plan, we systematically disassemble the object in a physics simulator and apply a cost function that encodes DfA principles. Manual generation, tool labelling, and assembly feedback rely primarily on multimodal large language models. Compared with a baseline that always removes the outermost part first from Tian et al., DfA-aware sequence planning reduces simulated assembly time, measured with a robot-arm assembly-time proxy, by 35% on 136 assemblies of 5 to 30 parts. The correct tool is selected for 88.6% of assembly steps. A vision-language model judge compares the generated manuals against ablated variants, identifying which page elements carry the information a reader needs. The presented approach and open-source code are available for use by engineers or AI agents looking to rapidly accelerate the creation of manufacturing plans for a given product design.

Abstract

Explore similar work

End-to-End Autonomous Generation of Human Assembly Plans

Geometric 2D Scene Graph Generation

AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents