DiagGen: Agentic Generation of Deformable Assets with Sim-based Diagnostics for Robotic Simulation
Organizations: University of British Columbia, Vancouver, Canada. · National University of Singapore, Singapore. · University of Cambridge, Cambridge, United Kingdom. · University of Wisconsin–Madison, Madison, WI, USA.
Abstract
While simulation-ready deformable assets are essential for in-silico robotic manipulation tasks, existing generation frameworks typically assess physical plausibility after generation, leaving an object's simulated response unused as feedback for repairing upstream errors. We present DiagGen, an agentic framework that turns a single in-the-wild image into a simulation-ready deformable asset through a generate--simulate--diagnose--refine loop. DiagGen constructs part-aware geometry and material parameters, then uses a VLM (vision-language model)-based agent to select semantically informative regions, probe them in a physics simulator, observe material responses, and route evidence-backed repair cues to the responsible generation stage. Experiments on 40 assets show that diagnostics provides useful repair cues and can moderately improve the quality of generated deformable assets. Finally, we show that unlike assets generated from visual foundation models which may not be simulatable, DiagGen-generated deformables can be directly dropped into a high-fidelity physical simulator for the planning and simulation of contact-rich pick-and-place tasks. The project's website is https://diaggen.github.io/.
Figures & tables
| Asset | Simulation observation, corroboration, and diagnosis | Ground-truth stage and issue | Correct issue and route? |
|---|---|---|---|
| Curling iron | During probe, release, and settling, the cord/plug moved as a “compact stiff assembly with little visible local bending”. The agent judged that “resistance to local bending conflicts with the semantic prior for a power-cord component”. With targeting confirmed, it recommended material_inference to restore plausible cord flexibility during subsequent probe and release. | Material inference: barrel-like stiffness assigned to the flexible cord. | Yes; issue and route |
| Massager | The agent judged that the probe lay on the coil, separate from the head-side anchor. It expanded and shifted the target along the “visible coil span”, yet three attempts still failed to isolate the spring. It diagnosed a “realized-region/foreground-resolution problem” and requested segmentation to recover “every visible helical turn and both coil-to-body boundaries” as one continuous foreground spring region. | Segmentation: incomplete part mask omitted most of the metal coil spring. | Yes; issue and route |
| Kitchen tongs | During the distal-tip probe, the mesh stayed connected while the “distal arm bent substantially”. After release, two additional default-length unforced settling windows showed “essentially no visible return toward the initial straight-arm configuration”. The agent judged the “persistent post-release bend” unsuitable for acceptance. The corroborating scale record identified an oversized mesh, so it recommended mesh_processing before reassessing material response. | Mesh processing: metric-scale target set to 2.8 m instead of 0.28 m. | Yes; issue and route |
| Asset | Diagnostic cue excerpt | Routed stage | Repair-agent strategy-change excerpt |
|---|---|---|---|
| USB hub | From the static triptych, the agent judged the terminal outside the pinned cable-root support. Expanding the target still failed to isolate the plug. It diagnosed a “missing visual-ownership/foreground-assignment defect” and requested segmentation to restore the silver terminal as a distinct visible region. | Segmentation | The agent replaced generic USB-part descriptions with explicit regions for the housing, cable, connector, and metal plug. After regeneration, the plug appeared as a distinct component that the simulator could target reliably. |
| Juice box | At the start of the diagnostic episode, Genesis rejected the asset before reset, so no probe could run. The runtime log reported local self-clearance below the assigned shell-thickness threshold; the material record then confirmed the incompatible uniform shell specification, and diagnostics routed the failure to material inference. | Material inference | The agent reduced the per-part shell thicknesses and replaced a single 3.0 GPa stiffness with distinct values of 0.8, 0.7, 0.6, and 0.3 GPa for the body, top, seam, and flap. The rebuilt asset initialized, completed probe and release tests, and was accepted. |
| Medical bag | During probing and release, the agent saw “visually stable triptychs” of the surviving pouch body and front patch. Yet the expected straps, handles, and hardware were unavailable for interaction, leaving an incomplete simulated bag despite its stable body motion. Grounding confirmed the missing regions, prompting repair through mesh processing. | Mesh processing | The agent increased the meshing fidelity of the missing parts. The rebuilt connected mesh restored all five part labels with 13,318 tetrahedra and passed topology validation. |
3D Asset Generation
3D asset generation creates and edits objects, characters, materials, and scenes from text, images, video, or geometric constraints.
See all 170 papers