Organizations: Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan 250014, China · Shandong Provincial Key Laboratory of Industrial Network and Information System Security, Shandong Fundamental Research Center for Computer Science, Jinan 250014, China · Quan Cheng Laboratory, Jinan, China · Northeastern University, China · Shandong University, China · Tencent, China · University of Chinese Academy of Sciences, Beijing, China
Large language models have demonstrated strong capabilities in structured reasoning and code generation, yet identifying an assembly fault does not guarantee the ability to generate executable rework programs. In interlocking assemblies, a locally plausible repair program may violate blocking dependencies, omit required recovery operations, or terminate in an incorrect configuration. We identify this challenge as fault-conditioned executable rework generation, where models must jointly reason about structural failures and generate programs that restore constrained assembly states. To benchmark this capability, we introduce FORGE-Bench, containing 4,000 tasks across 26 interlocking assemblies with five assemblies held out for evaluating generalization to unseen structures. We further propose FORGE (Fault-Oriented Rework Code Generation for Executable Interlocking Assembly), a structured state-to-program framework that decomposes rework generation into Rework Structuring and Action Grounding. Rework Structuring captures fault-induced recovery dependencies, while Action Grounding maps these dependencies into state-conditioned executable actions. Experiments show that locally reasonable repair code is insufficient for reliable assembly recovery, and that connecting structural recovery reasoning with executable action realization is essential for verified target restoration.
Figures & tables
Figure 1: Motivation of FORGE. Existing methods overlook structural dependencies during assembly rework, while FORGE enables executable recovery through dependency-aware reasoning.
Figure 2: Overview of FORGE. Rework Structuring identifies recovery obligations, while Action Grounding converts them into executable actions.
Figure 3: Construction pipeline of FORGE-Bench . Executable assembly structures are converted into public task representations, augmented with controlled fault and no-fault states, paired with reference rework programs, and filtered through symbolic execution and target-state restoration.
Figure 4: Composition of FORGE-Bench .
Dataset statistics
Rework statistics
Statistic
Count
Share
Statistic
Count
Share
Total tasks
4,000
100.0%
No-fault cases
1,115
27.9%
Train / Test
3,000 / 1,000
75.0 / 25.0%
Single-fault cases
2,353
58.8%
Assembly families
26
–
Multi-fault cases
532
13.3%
Train assemblies
21
–
Counterfactual pairs
160
–
Test assemblies
5
–
Fault/no-fault instances
320
–
Table 1: Key statistics of FORGE-Bench .
Model
Core Metrics
Overall-Core ↑
LLM-as-Judge
SSE ↑
SAPF ↑
VMRE ↑
DGTR ↑
MVE ↑
HRQ ↑
Score ↑
Open-source Models
Qwen3.5-9B Base ( Qwen Team, 2026a )
20.00
11.04
22.50
25.83
22.50
20.00
20.31
2.55
Gemma-3-12B-IT ( Team et al., 2025 )
6.21
50.00
2.50
5.83
2.50
—
11.17
1.91
GLM-4-9B-0414-Chat ( GLM Team, 2025 )
8.42
38.75
3.33
3.33
3.33
—
9.53
1.87
InternLM3-8B-Instruct ( InternLM Team, 2024 )
16.00
22.22
20.00
20.00
20.00
20.00
19.70
1.79
Table 2: Main results on FORGE-Bench . Models are grouped into open-source models, closed-source models, prior methods, and our method. The best results are highlighted in bold, and the second-best results are underlined.
Method
Core Metrics
Overall-Core ↑
LLM-as-Judge
SSE ↑
SAPF ↑
VMRE ↑
DGTR ↑
MVE ↑
HRQ ↑
Score ↑
Vanilla SFT
69.23
61.11
63.23
71.56
62.60
77.44
67.53
4.03
w/o Rework Structuring
62.38
41.67
62.19
68.19
60.10
73.60
61.35
4.12
w/o Action Grounding
65.43
66.67
64.05
67.38
63.11
80.67
67.89
4.30
FORGE (Ours)
71.54
61.11
72.60
75.93
72.13
83.17
72.75
4.37
Table 3: Ablation study of FORGE components. The best results are highlighted in bold with a blue background.
Figure 5: Qualitative comparison of executable rework generation. FORGE preserves recovery dependencies and restores the target assembly, while baseline predictions exhibit missing dependency handling or unnecessary intervention.
Figure 6: Consistency between automatic evaluation and human judgment.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
File
Role
initial_state.json
Initial assembly state.
target_state.json
Target assembly configuration.
actual_state.json
Observed state supplied to the model.
assembly_graph.json
Assembly relations between parts.
blocking_graph.json
Blocking and access dependencies.
contact_graph.json
Contact or adjacency relations.
Appendix
Table 4: Public files in a FORGE-Bench task.
Action
Recorded precondition
remove(part_id)
Part is present and not blocked for removal.
insert(part_id)
Part is absent, target pose is free, and insertion path is collision-free.
reset_pose(part_id)
Part is present and reset path is collision-free.
rotate(part_id, axis, angle_deg)
Part is present and rotation path is collision-free.
rotate_to_unlock(part_id, axis, angle_deg)
Part is present and the rotation path is collision-free.
rotate_translate_to_unlock (part_id, after_pose)
Part is present and the coupled unlock path is collision-free.
Appendix
Table 5: Recorded action interfaces and symbolic preconditions.
Statistic
Count
Share / note
Total tasks
4,000
100.0%
Train / test tasks
3,000 / 1,000
75.0 / 25.0%
Assembly families
26
21 train, 5 held-out
No-fault cases
1,115
27.9%
Single-fault cases
2,353
58.8%
Multi-fault cases
532
13.3%
Appendix
Table 6: FORGE-Bench composition under the 4,000-task protocol.
Setting
Recorded value
Base model
Qwen3.5-9B
Checkpoint source
/home/models/Qwen3.5-9B
Trainable modules
All linear modules
LoRA rank / scale / dropout
8 / 32 / 0.05
Precision
bfloat16
Per-device batch size
1
Appendix
Table 7: Recorded training and inference configuration.
Field family
Recorded rule
Position / translation
Tolerance-aware comparison.
Orientation / rotation
Tolerance-aware comparison.
Scalar parameter
Tolerance-aware comparison.
Part identity
Exact identifier equality.
Action type
Exact action-name equality.
Target state
Pose, presence, and tolerance-aware symbolic comparison.
Appendix
Table 8: Evaluator comparison rules.
Failure category
Description
Diagnostic signal
Fault misdiagnosis
The predicted fault type or affected part does not match the observed assembly state.
D^i=Di∗
Dependency omission
A required unlock, clearing, restoration, or relocking operation is missing from the recovery program.
Missing dependency
Illegal grounding
The generated action or parameter is unavailable in the current symbolic state.
¬Legal
Parameter error
The action type is appropriate, but its pose, rotation, or other parameters fall outside the evaluator tolerance.
Invalid parameter
Incomplete restoration
The program executes partially but the terminal state does not match the target configuration.
ST≡S∗
Over-disassembly
The program moves additional components that are not required by the recovery route.
Excessive actions
Appendix
Table 9: Failure categories used in the qualitative analysis.
Large language models (LLMs) increasingly generate structured layouts and executable scene programs from natural-language instructions. However, executability does not imply engineering feasibility: an industrial scene program may execute successfully while its final state still violates coupled boundary, collision, clearance, aisle, or structural constraints. We identify this mismatch as the executability-engineering-feasibility gap, where locally plausible placements can remain globally infeasible because corrections to one device may introduce violations elsewhere. To make this gap measurable in a controlled setting, we construct an execution-grounded industrial scene-program dataset with 1,749 constructible scenarios, complete executable programs, and 94 final-state geometric checks per scenario. We further propose VIGOR, which combines supervised scene-program generation with execution-grounded verification and staged geometric refinement. Supervised fine-tuning establishes complete program generation, local projection resolves independently correctable violations, and joint conflict-graph repair coordinates residual multi-object conflicts while preserving structural relations. Experiments show that VIGOR achieves 96.86% Complete Feasibility and an Overall score of 91.20, outperforming the strongest evaluated baseline by 10.42 points in Overall. These results show that, under controlled industrial layout conditions, reliable scene-program generation requires not only executable programs, but also explicit verification and coordinated resolution of coupled engineering constraints.
Fuqiang Wang, Zheng Guo, Xiaoming Wu +5
Qilu University of Technology, Jinan, China · Tianjin University, Tianjin, China · Shandong University, Jinan, China +3
A GUI command can be correct at the interface level yet still produce an incorrect assembly. An agent must identify the relevant components, respect their structural relationships, and determine which parts should move during recovery. We introduce MAGAR-Bench, a benchmark of 4,000 multimodal samples from 26 assembly families that separates Assembly Decision, Atomic Action, and Minimal Rework. For rework, the objective counts distinct moved parts within the validated recovery candidate set for each state, rather than the number of actions. We also introduce MAGAR, a selective correction framework built on a frozen policy obtained by supervised fine-tuning. Its key design matches correction granularity to output dependencies. Structured candidates revise assembly fields and rework scope, while complete action candidates keep native GUI primitives coupled with their parameters. A learned selector retains the original prediction unless a candidate passes a task-specific intervention threshold. On recorded states from held-out assembly families, MAGAR improves Overall, the task-balanced aggregate, from 37.50 to 41.39, with gains on all three tasks. Across 209 interventions, 70 correct an error and 22 introduce one. These results support task-specific correction while highlighting the remaining tradeoff between fixing errors and preserving reliable predictions.
Jingxuan Wei, Jiahao Fu, Fuqiang Wang +5
University of Chinese Academy of Sciences, Beijing, China · Qilu University of Technology (Shandong Academy of Sciences), Shandong Computer Science Center, Jinan, China · China University of Petroleum (East China), China +2
Large language models (LLMs) can translate natural-language problem descriptions into optimization code, but the code is prone to silent failures: it executes and returns a solver-feasible solution while encoding a semantically incorrect formulation. On compositional problems, the resulting feasibility-correctness gap reaches 90 percentage points. We introduce ReLoop, which combines two mechanisms. Structured generation decomposes code production into a four-stage reasoning chain (understand, formalize, synthesize, verify) to reduce formulation errors during generation. Behavioral verification detects the errors that remain by testing whether the formulation responds correctly to solver-based parameter perturbation, a signal that comes from the solver rather than from LLM self-review and requires no ground truth. The two mechanisms address different error structures: structured generation gives the largest gain on compositional problems (+8.5pp accuracy on RetailOpt-190 with Claude Opus 4.6), and behavioral verification gives its largest gain on localized defects (+4.4pp on MAMO-ComplexLP). With diagnostic execution recovery, ReLoop reaches 100% executable code on Claude Opus 4.6, and relative to direct generation it raises or preserves every reported metric of the three chat-tuned foundation models on all three benchmarks. For the narrowly fine-tuned SFT model we test, the chain-of-thought prompt conflicts with its learned output format and lowers its accuracy on MAMO-ComplexLP; we document and analyze this interaction. We release RetailOpt-190, 190 compositional retail optimization scenarios in which several constraints interact.
Junbo Jacob Lian, Yujun Sun, Huiling Chen +3
Institute of Operations Research and Analytics, National University of Singapore · McCormick School of Engineering, Northwestern University · Wenzhou Buyi Pharmacy Chain Co., Ltd. +2