ElasticFit: Fit-Aware 3D Object Insertion via VLM Reasoning and Generative Adaptation

Tzu-Hsin Hsieh; Ricardo Marroquim

cs.CVOct 5, 2026

ElasticFit: Fit-Aware 3D Object Insertion via VLM Reasoning and Generative Adaptation

Authors: Tzu-Hsin Hsieh, Ricardo Marroquim

Organizations: Delft University of Technology Delft, Netherlands

Abstract

Inserting objects into existing 3D scenes requires more than selecting a plausible location: the inserted object must also fit local geometry while preserving semantic intent and physical plausibility. Although recent Vision-Language Models (VLMs) and generative models enable semantic reasoning and visual content creation, they offer limited 3D grounding and geometric control when an inserted object must fit into constrained local spaces. We introduce ElasticFit, a VLM-guided framework for fit-aware object insertion centered on a novel scene-grounded representation. Given a language instruction and rendered scene observations, ElasticFit infers structured fitting cues that specify where the object should be grounded, what volume it should occupy, how it should be oriented, and its adaptation mode (rigid placement, uniform scaling, or elastic fitting). These cues convert high-level VLM reasoning into explicit 3D constraints that condition object generation and guide downstream geometric fitting. ElasticFit then generates a scene-conditioned object prior, reconstructs it in 3D, and refines the mesh through mode-specific fitting while enforcing collision avoidance, contact consistency, and physical grounding. In fixed-asset baseline comparisons, ElasticFit improves spatial relation success from 50.8% to 69.7% and support success from 48.3% to 91.7% over the strongest baseline, while providing novel support for generative "make-it-fit" insertions in complex scenarios.

Figures & tables

Figure 2: Overview of ElasticFit. Given a user instruction $L$ and an existing 3D scene $S$ , ElasticFit infers structured fitting cues $G$ that encode the target region, orientation, constraints, and adaptation behavior. Stage 1 performs semantic-spatial grounding, Stage 2 generates a scene-conditioned object prior $O_{0}$ . Stage 3 refines it through geometry- and physics-aware fitting with rigid, uniform, or elastic adaptation. The result is a stable final insertion $O^{\ast}$ .

Figure 3: Visualization of the three adaptation modes in Stage 3. Each row shows the target ghost box, generated initial object, and fitted result. Rigid mode preserves geometry, uniform mode applies isotropic scaling, and elastic mode applies anisotropic deformation to fit the target region.

Table 3

Figure 4: Qualitative comparison on representative Common Placement Set tasks. Baselines often struggle with physical refinement, ambiguous references, viewpoint-dependent relations, or irregular support geometry. ElasticFit produces more spatially and physically plausible placements.

Subset	SRS $\uparrow$	CFR $\uparrow$	SupR $\uparrow$	CPS $\uparrow$
All tasks	60.8%	71.7%	83.3%	4.57
Pose-and-contact tasks	73.3%	76.7%	80.0%	4.50
Gap-fitting tasks	48.3%	66.7%	86.7%	4.63

Table 3: Performance on the Fit-Critical Set.

Setting	SRS $\uparrow$	CFR $\uparrow$	SupR $\uparrow$	CPS $\uparrow$	Pen. $\downarrow$
ElasticFit	47.8%	76.7%	83.3%	4.63	1.08%
w/o physics settle	46.7%	76.7%	80.0%	4.60	1.45%
w/o geom.-phys. fitting	47.8%	36.7%	83.3%	4.23	6.09%

Table 4: Ablation study on gap-fitting tasks.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Diagnostic category	Count / 25	Explanation
SRS not satisfied, but visually plausible ( $\mathrm{CPS}\geq 4$ )	24 (96.0%)	These cases do not fully satisfy strict geometric predicates, but their rendered placements remain contextually plausible according to CPS.
parallel_to check not satisfied	19 (76.0%)	These cases contain an unsatisfied parallel_to relation, indicating that orientation constraints are a common source of strict SRS misses in constrained regions.

Appendix

Table 5: Diagnostic breakdown of SRS-unsatisfied cases on the gap-fitting subset. The gap-fitting subset contains 30 tasks; this table analyzes the 25 cases with

\mathrm{SRS}<1

. Categories are diagnostic and not mutually exclusive.

Figure 5: Representative failure cases of ElasticFit. Top: directional fitting errors arise from ambiguous reference geometry or fill directions. Middle: generation–adaptation mismatch occurs when the object prior and subsequent deformation do not match the target fitting region. Bottom: concave support ambiguity occurs in sink-like geometry, where rim or surface support can be mistaken for the intended interior placement.

Failure source	Count / 60	Description
Directional fitting error	2 (3.3%)	Incorrect yaw, pitch, alignment, or gap-fill direction.
Generation / adaptation mismatch	6 (10.0%)	Object prior or deformation mismatches the target fitting region.
Concave support ambiguity	1 (1.7%)	Sink-like concave geometry can confuse inside placement with rim or surface support.

Appendix

Table 6: Failure attribution on the Fit-Critical Set. Each failed task is assigned to its dominant failure source. Percentages are computed over all 60 Fit-Critical tasks.

Aspect	FirePlace	ElasticFit
Target object	Pre-existing fixed asset	Generated or fixed object
Main objective	Place a given object	Generative “make-it-fit” insertion
Adaptation	Pose-level placement	Rigid, uniform, or elastic fitting
Constrained spaces	Limited by fixed-asset geometry	Designed for gaps, corners, wall-side, under/inside spaces
Grounding	Fine-grained geometric placement constraints	Grounded fitting representation $G$ with support, ghost box, orientation, mode, scale, and prompt
Physical refinement	Geometry-aware collision/support refinement	Geometry-aware fitting plus physics-guided settling

Appendix

Table 7: Design-level comparison between FirePlace and ElasticFit.

Method	Target Object	Placement Strategy	Adaptation	Constrained-Space Insertion
LayoutGPT	Fixed / retrieved	Layout pose direct prediction	Rigid only	No
Holodeck	Fixed / retrieved	Layout constraints / box solver	Rigid only	No
LayoutVLM	Fixed / retrieved	Visual layout / box solver	Rigid only	No
FirePlace	Fixed asset	Fine-grained geom. constraints	Rigid only	Fixed-asset only
ElasticFit	Generated / fixed	Grounded rep. $G$ + geom.-phys. fitting	Rigid / Uniform / Elastic	Yes

Appendix

Table 8: Design-level comparison of object insertion and layout methods.

Field	Type	Description
anchor_object_id	int	Main reference object for spatial reasoning.
support_object_id	int	Object expected to support the inserted object.
target_uv	$[u,v]$	Image-space target point for the object’s bottom center.
look_at_uv	$[u,v]$	Optional point that the object front should face.
top_uv	$[u,v]$	Optional top contact point for leaning or tilted poses.
scale	$[w,d,h]$	Object size in the default upright pose.

Appendix

Table 9: Structured JSON output predicted by Stage 1.

Prompt block	Purpose
Role and task	Defines the VLM as a 3D spatial reasoning stage for placing a new object in an existing 3D scene.
Scene context and inputs	Provides image resolution, normalized image coordinates, visible object IDs, object metadata, and the user instruction.
Reference and scale rules	Selects a valid anchor object and predicts the target object dimensions in its default upright pose.
Coordinates and placement rules	Predicts the target support point, optional facing point, and optional top-contact point for leaning or lying poses.
Physical attributes and orientation	Predicts adaptation mode, pose tag, orientation relation, yaw offset, pitch offset, and orientation reference object.
Snapping and gap filling	Handles contact-heavy instructions such as against or flush with , and fill-space instructions such as fill , occupy , or fit .

Appendix

Table 10: Prompt structure for Stage 1 VLM reasoning.

Mode	Scale	Main variable	Solver
Rigid	$(1,1,1)$	Translation near ghost box	Heuristic search
Uniform	$(\alpha,\alpha,\alpha)$	Translation and global scale	Gradient optimization
Elastic	$(s_{x},s_{y},s_{z})$	Box-local extents	Box-aligned expansion

Appendix

Table 11: Summary of the three adaptation modes in Stage 3.

Task set	Subset / category	Primary constraint types
Common Placement	Distance	near, far
	Directional	left of, right of, in front of, behind
	Facing	face to
	Alignment	aligned with, parallel to
	Against wall	against wall
Fit-Critical	Pose-and-contact	user-defined rotation, leaning, lying

Appendix

Table 12: Task taxonomy used in the evaluation. All tasks include a base support condition specifying the intended placement surface; the listed types indicate the primary additional spatial or fit-critical constraint.

Method	Spatial semantics	Physical plausibility
LayoutGPT	Directly predicts target position and yaw from a CSS-style scene description.	No explicit collision or support refinement; placements are only clamped to scene bounds.
Holodeck	Generates Holodeck-style spatial constraints and searches over discrete placement candidates.	Uses AABB-level collision, support, wall, and relation checks during candidate scoring.
LayoutVLM	Initializes object pose from vision-language reasoning and semantic constraints.	Uses bounding-box-level differentiable optimization for feasible placement refinement.
ElasticFit	Uses view-aligned structured grounding with support validation and local scene cues.	Uses geometry-aware local mesh grounding and physics-guided placement refinement.

Appendix

Table 13: Summary of adapted baselines for the single-object insertion setting.

Prompt block	Purpose
Task instruction	Specifies single-object insertion into a fixed existing 3D scene.
Coordinate system	Defines left , depth , top , and yaw orientation.
Scene bounds	Provides the valid 3D range of the scene in centimeters.
Output template	Enforces one CSS-style block for the target object.
Rules	Predicts only position and yaw while keeping the target asset fixed.
Scene objects	Lists context objects with captions, size, position, and orientation.

Appendix

Table 14: Prompt structure for the adapted LayoutGPT baseline.

Method	Avg. total time (s)	Notes
LayoutGPT	42.97	End-to-end runtime
Holodeck	16.71	End-to-end runtime
LayoutVLM	100.03	End-to-end runtime
ElasticFit	11.53	Placement-only; 10.50 s VLM + 1.03 s fitting

Appendix

Table 15: Average runtime comparison on the Common Placement Set.

Figure 7: Additional qualitative results on the Common Placement Set. Each example shows the final fitted placement produced by ElasticFit, with red boxes highlighting the inserted object. The examples include diverse spatial relations such as left/right, near/far, facing, alignment, and against-wall placement.

Figure 8: Additional qualitative results on the Fit-Critical Set. The examples show final fitted insertions produced by ElasticFit, including pose-and-contact cases and gap-fitting cases in constrained local scene regions.

Assets
Alarm clock	Book	Chair
Backpack	Bowl	Coffee machine
Bike	Camera bag	Fan
Camera	Cap	Glasses
Lamp	Laptop	Monitor
Mug	Picture frame	Pillow

Appendix

Table 16: Third-party 3D assets used in the experiments.

Explore similar work

Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generation
Nov 18, 2025Weimin Bai, Yubo Li, Weijian Luo +43D Generative Models 3D Spatial Reasoning
Peking University, Beijing · hi-lab, Xiaohongshu Inc. · MMLab, CUHK +1
ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation
Jul 7, 2026Tianjiao Yu, Xinzhuo Li, Yifan Shen +43D Generative Models 3D Representation
University of Illinois Urbana-Champaign
Global Graph-Validated Optimization for VLM-based 3D Indoor Scene Generation
Aug 4, 2026Jialu Huang, Yingxuan You, Fei Wang +13D Scene Generation 3D Scene Understanding
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, China · 2 École Polytechnique Fédérale de Lausanne (EPFL), CVLab, Switzerland · School of Electronics and Information, Northwestern Polytechnical University and Shaanxi Key Laboratory of Information Acquisition and Processing, China

Nov 18, 2025, cs.CVJ/K move · Enter open · S save

Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generation

Weimin Bai, Yubo Li, Weijian Luo, Zeqiang Lai+3

Peking University, Beijing · hi-lab, Xiaohongshu Inc. · MMLab, CUHK+1

3D Generative Models 3D Spatial Reasoning 3D Generation Geometric Constraints Critic-Free Reinforcement Learning

Open paper AskPDF arXiv

Text-to-3D generation has advanced rapidly, yet state-of-the-art models, encompassing both optimization-based and feed-forward architectures, still face two fundamental limitations. First, they struggle with coarse semantic alignment, often failing to capture fine-grained prompt details. Second, they lack robust 3D spatial understanding, leading to geometric inconsistencies and catastrophic failures in part assembly and spatial relationships. To address these challenges, we propose VLM3D, a general framework that repurposes large vision-language models (VLMs) as powerful, differentiable semantic and spatial critics. Our core contribution is a dual-query critic signal derived from the VLM's Yes or No log-odds, which assesses both semantic fidelity and geometric coherence. We demonstrate the generality of this guidance signal across two distinct paradigms: (1) As a reward objective for optimization-based pipelines, VLM3D significantly outperforms existing methods on standard benchmarks. (2) As a test-time guidance module for feed-forward pipelines, it actively steers the iterative sampling process of SOTA native 3D models to correct severe spatial errors. VLM3D establishes a principled and generalizable path to inject the VLM's rich, language-grounded understanding of both semantics and space into diverse 3D generative pipelines.

Abstract

Explore similar work

Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generation

ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation

Global Graph-Validated Optimization for VLM-based 3D Indoor Scene Generation