Design for Robotic Assembly (DfRA) remains largely dependent on manual planning and heuristic simulation, limiting scalability and robustness in complex industrial settings. Although large language models (LLMs) show promise for semantic reasoning and task planning, most approaches remain tightly coupled to pre-built simulators that assume an accurate world model. We introduce Iterative Design for Robotic Assembly (IDfRA), a closed-loop framework that combines an LLM for plan generation with a vision-language model (VLM) for execution assessment. Given a target structure and a partial environmental signature, the LLM proposes an assembly plan, the robot executes it once at test time, and the VLM evaluates the resulting state to provide feedback for replanning. Through this iterative planning-execution-verification loop, the system progressively improves semantic fidelity and physical feasibility. Crucially, IDfRA does not require an accurate a priori world model before deployment. Instead, physical constraints are discovered online through interaction, enabling adaptation to under-specified environments. Empirical evaluation demonstrates that IDfRA attains 73.3% top-1 accuracy in semantic recognisability, surpassing the baseline on this metric. Moreover, the resulting assembly plans exhibit robust physical feasibility, achieving an overall 86.9% construction success rate, with design quality improving across iterations, albeit not always monotonically. Pairwise human evaluation further corroborates the advantages of IDfRA relative to alternative approaches. By integrating self-verification with context-aware adaptation, the framework evidences strong potential for deployment in unstructured manufacturing scenarios.
Figures & tables
Fig. 1: The overall IDfRA framework. Top : Given a target assembly (e.g., “house”) and a signature of the environment in .json format, IDfRA iteratively generates a detailed assembly plan specifying each available block’s position and pose. Bottom : Renders of the 10 iterations of the “house” assembly.
Fig. 3: IDfRA system architecture. A GIF of the initial design execution, any missing blocks, and the original inputs are passed to the VLM Judge . Based on the Judge ’s feedback and previous plans, the Replanner produces an updated design which is then executed. This process repeats for 10 iterations and a VLM Selector chooses the final assembly.
Fig. 4: IDfRA design generation. (a) Multiple runs of IDfRA – each comprising 10 iterations – produce different design evolutions given the same blocks and input “giraffe”. (b) 10 candidate designs produced in parallel by BloxNet – each design is produced independently without influence from the previous one. (c) Designs generated by multiple full runs of IDfRA for “dining table set”. (d) Designs generated by IDfRA (left) and BloxNet (right) for “robot”, “Taj Mahal”, “Eiffel Tower”, “burger”, “bridge”, “ceiling fan”.
Fig. 5: Human evaluation of final designs . Pairwise voting scores from 100 survey responses comparing IDfRA vs BloxNet across 15 assemblies (best viewed zoomed-in.)
Fig. 6: Iterative design improvement . For each structure, the Y-axis shows the number of times an iteration matched or outperformed previous iterations. The mean trend is indicated by a black dashed line with standard deviation shaded in grey.
N
Method
Top-1 Acc.
Average Ranking
Relative Ranking
5
Iterative DfRA
64.44%
1.60
32.00 %
BloxNet
62.22%
1.62
32.44%
10
Iterative DfRA
73.33%
2.24
22.44%
BloxNet
57.78%
2.38
23.78%
15
Iterative DfRA
68.89 %
2.71
18.07 %
BloxNet
51.11%
3.36
22.37%
TABLE I: VLM-based design recognisability : Top-1 accuracy, average ranking, and relative ranking averaged across three runs. Computed using VLM rankings of N objects chosen from a pool of 200 objects based on semantic resemblance.
Assembly Design
% Blocks Correctly Placed
% Assemblies Successful
Giraffe
97.78
80
Taj Mahal
100
100
House
100
100
Shark
95
70
Burger
100
100
Jungle
95.71
70
TABLE II: Physical feasibility analysis : Percentage of blocks correctly placed by the robot and percentage of successful assemblies across 10 trials for six assembly designs.
VLM Recognisability (N = 10)
Containing Missing Blocks
Top-1 Acc.
Average Rank
# targets
# iterations
Proposed IDfRA
94.44 %
1.05
2
10
PNG instead of GIF
72.22%
1.61
2
9
No missing blocks
66.67%
2.22
3
11
No history
83.33%
2.05
4
5
TABLE III: Ablation studies : VLM-based design recognisability and frequency of missing blocks across 10 iterations for each of the 6 representative targets (“bridge”, “burger”, “ceiling fan”, “couch”, “giraffe”, “robot”).
Fig. 7: Limitations . IDfRA occasionally generates designs which contain overhanging blocks or include additional unnecessary details (highlighted in red) such as for “ship” (a) and “ceiling fan” (b). Though rare, it is also possible that no iteration produces a ‘good’ design such as this run of “giraffe” (c). Such failures largely stem from the VLM misidentifying unstable structures as stable.
Fig. 8: Physical Experiments . Four generated designs–“sheep”, “jungle”, “burger”, and “Taj Mahal”– assembled using 3D printed blocks by an AGILEX PiPER arm and suction gripper for manipulation, and an Intel RealSense Depth Camera for localisation.
Multi-pair robotic assembly in unstructured environments faces spatial interference and contact uncertainties. Existing paradigms fail to bridge cognitive decision-making and physical execution, as they either encounter state-space explosion and knowledge bottlenecks or suffer from logical hallucinations and topological conflicts. We propose an end-to-end neuro-symbolic framework that solves the challenge hierarchically: generating optimal subgraphs for each pair, decoupling generality from edge cases, and then resolving cross-pair interferences. Given an eye-on-hand RGB-D assembly scene, the framework extracts semantic instance identity and state while quantifying the scene for divergence calculation. For each pair, optimal subgraph is generated via LLM using barely basic actions to mitigate hallucinations. Supportive actions for edge cases are reasoned and inserted with a lightweight discriminator. Driven by the divergence between the quantified baseline and current scene, it is easily extensible at low cost. Augmented subgraphs are topologically coordinated into global sequences while preserving internal behavioral coherence. Dynamic behavior trees embedding atomic skills close the force-aware execution loop. Offline evaluation on 100 real-world scenes achieves 97.00% global executability, outperforming classical and state-of-the-art planners. Real-robot deployment on a UR3 arm attains 90% success rate with 0.5 mm tolerance under strong interference, demonstrating a unified and verifiable solution for complex autonomous assembly.
Xinyi Li, Aiguo Song, Linhu Wei +1
School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China.
Human-robot collaboration (HRC) in structured assembly requires reliable state estimation and adaptive task planning under noisy perception and human interventions. To address these challenges, we introduce a design-grounded human-aware planning framework for human-robot collaborative structured assembly. The framework comprises two coupled modules. Module I, Perception-to-Symbolic State (PSS), employs vision-language models (VLMs) based agents to align RGB-D observations with design specifications and domain knowledge, synthesizing verifiable symbolic assembly states. It outputs validated installed and uninstalled component sets for online state tracking. Module II, Human-Aware Planning and Replanning (HPR), performs task-level multi-robot assignment and updates the plan only when the observed state deviates from the expected execution outcome. It applies a minimal-change replanning rule to selectively revise task assignments and preserve plan stability even under human interventions. We validate the framework on a 27-component timber-frame assembly. The PSS module achieves 97% state synthesis accuracy, and the HPR module maintains feasible task progression across diverse HRC scenarios. Results indicate that integrating VLM-based perception with knowledge-driven planning improves robustness of state estimation and task planning under dynamic conditions.
Yanyi Chen, Min Deng
Department of Civil, Environmental, and Construction Engineering, Texas Tech University, Lubbock, TX 79409, USA
We introduceWorkBenchMark, a LEGO Duplo-based robotic assembly benchmark motivated by the RoboCup Smart Manufacturing League. Robotic assembly couples low-level manipulation with task-level symbolic reasoning under physical constraints, a combination that current end-to-end learning methods do not yet solve reliably. The benchmark provides 400 tasks across four complexity tiers. We provide an open-vocabulary perception, Assembly-by-Disassembly baseline solution. Our planning-based pipeline outperforms a modern vision-language-action approach across all tiers. The benchmark, simulation environment, and baseline implementation will be released openly to support the broader robotic assembly community.
Wenbo Ma, Daniel Swoboda, Matteo Tschesche +1
Chair of Machine Learning and Reasoning (i6), RWTH Aachen University, Aachen, Germany · MASCOR Institute, FH Aachen University of Applied Sciences, Aachen, Germany