Deformable Contact-Aware 3D Object Placement

Songhe Wang; David Miller; Charles A. Kamhoua

cs.CVNov 5, 2026

Deformable Contact-Aware 3D Object Placement

Authors: Songhe Wang, David Miller, Charles A. Kamhoua

Abstract

Current language-guided 3D object placement typically predicts a rigid 6-DoF pose, ignoring the physical reality that indoor placements are dictated by contact. Consequently, predicted poses often float, interpenetrate supports, or fail to deform soft surfaces. We propose Deformable Contact-Aware Placement (DCAP), a framework that reframes placement as a drop-to-equilibrium process. Rather than predicting a static pose, DCAP constructs a semantic release state that specifies the drop location, object scale, and deformable components, allowing a coupled rigid-deformable simulator to settle the object under gravity and friction. This explicitly separates semantic intent from physical realization. Evaluated on a benchmark of 2,063 prompt-asset-scene cases, DCAP significantly outperforms static baselines. On standard placements, it improves VLM Physics & Naturalness by 21.2 points and human preference by 26.5 points over the strongest raw rigid baseline, while reducing mean penetration depth by 72.3%. Compared to simulated rollout baselines, DCAP reduces residual speed by 72.1%. Ultimately, our results prove that physically accurate 3D placement requires modeling semantic intent and physical realization as strictly separate processes.

Figures & tables

Component	Module	Role
$b,s$	Support grounding and scale prior	Infer the intended support category and a physically meaningful object scale
$S_{b},P_{b}$	Segmentation and parts	Identify the support instance and decompose it into usable semantic parts
$R_{0}$	Three-candidate VLM orientation	Select a coarse release orientation from prompt-conditioned rendered candidates
$\pi,a,\mathbf{t}_{0}$	Ray-cast anchoring	Convert the prompt into a release point above the target region
$\theta,\mathcal{R},\mathcal{D}$	Materials and split	Assign discrete material values and rigid/deformable roles
$X_{0}$	Thin-aware particle filling	Initialize a contact-ready deformable particle set

Table 1: DCAP modules for release-state construction and physical rollout.

Figure 2: Thin-aware particle filling. DCAP estimates local thickness and adapts particle spacing so thin compliant regions retain enough through-volume samples for contact and stress resolution.

Aggregate quality

Controlled physics view

Method

VLM

Human

Pen.

Contact

Spd.

Conv.

Contrast

Change

VLM P/N gain

Pen. drop

Final conv.

Final time

P/N

\uparrow

P/N

\uparrow

\downarrow

\uparrow

cm/s

\downarrow

\uparrow

\uparrow

\uparrow

\uparrow

\downarrow

FirePlace raw

64.0

50.1

1.48

0.65

n.a.

FirePlace raw

static ref.

ref.

n.a.

14.6

PlaceIt3D raw

60.6

n.a.

1.70

0.62

n.a.

PlaceIt3D raw

static ref.

ref.

n.a.

LayoutGPT raw

49.7

n.a.

2.31

0.47

n.a.

FP raw

\to

+rigid

rigid rollout

+2.7

0.38

0.61

38.2

Holodeck raw

54.2

n.a.

2.05

0.53

n.a.

FP raw

\to

+def.

deform. rollout

+8.7

0.73

0.71

86.9

Table 2: Baseline comparison on the standard split (1,923 cases). Left: aggregate placement quality. Right: controlled ablations; VLM P/N gains report

B-A

for each

A\!\to\!B

contrast, with “ref.” rows defining block references.

Figure 3: Representative settling rollouts. A backpack and a plush toy settle under gravity while preserving the intended support and scale.

Figure 4: Qualitative comparison. DCAP produces stable contact, deformation, and infeasibility handling, while rigid baselines often float, interpenetrate, or miss soft-support deformation.

Human forced-choice results					Objective diagnostics
Comparison	Axis	DCAP	Baseline	Diff.	Method	Pen.	Contact	Speed	Conv.
						cm $\downarrow$	$\uparrow$	cm/s $\downarrow$	$\uparrow$
DCAP vs FP raw	RP	61.2	56.4	+4.8*	FirePlace raw	1.48	0.65	n.a.	n.a.
DCAP vs FP raw	P/N	76.6	50.1	+26.5*	FP+rigid	1.10	0.70	6.8	0.61
DCAP vs FP+def.	RP	61.2	55.6	+5.6*	FP+def.	0.75	0.79	4.1	0.71
DCAP vs FP+def.	P/N	76.6	63.6	+13.0*	DCAP static	0.86	0.58	n.a.	n.a.

Table 3: Human preference and objective diagnostics. Bradley–Terry–Luce (BTL) estimates from forced-choice evaluations strongly align with automated objective metrics. Asterisks indicate that the paired bootstrap confidence interval for the human BTL difference excludes zero after Holm correction; significance markers apply only to human forced-choice differences.

Contact-regime placement quality						Stress feasibility
Regime	Cases	FP raw	FP+def.	DCAP rigid	DCAP full	Stress statistic	Infeas.	Feas.	Summary
		P/N	P/N	P/N	P/N / Conv.
Soft	869	60.4	70.4	74.7	88.2 / 0.89	Pred. reject	62	12	74
Mixed	550	64.8	74.0	78.0	84.5 / 0.84	Pred. feasible	14	52	66
Rigid	504	69.2	75.2	80.0	80.9 / 0.80	Reject P/R/F1	0.84	0.82	0.83
Standard avg.	1,923	64.0	72.7	77.0	85.2 / 0.85	OOD true accept	–	0.81	52/64

Table 4: Performance across contact regimes and stress feasibility. Left: DCAP outperforms baselines on soft supports. Bold indicates best P/N score within each contact regime; convergence is reported after the slash. Right: robustness on extreme stress cases, measured by discrete rejection/acceptance counts rather than continuous diagnostics.

Ablation and sensitivity results					Runtime and sensitivity
Variant / setting	P/N	Pen.	Conv.	Hard fail	Stage / setting	Value	Note
DCAP full	84.1	0.42	0.84	0.039	Full pipeline	107.5 s	total mean
No thin-aware fill	76.3	0.58	0.75	0.094	Median pipeline	88.4 s	median
No material-specific map	78.0	0.55	0.77	0.081	Simulation	54.2 s	main bottleneck
Rigid support only	74.5	0.64	0.72	0.103	Segmentation + parts	18.6 s	scene processing
Rigid object only	79.0	0.57	0.78	0.073	Thin-aware filling	12.8 s	particle count

Table 5: System ablations, runtime, and sensitivity. Left: Module-level ablations evaluated via VLM and objective metrics. Right: Pipeline runtime breakdown and hyperparameter sensitivity. Note: The left and right blocks are independent and not row-paired.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting
LLM parsing, target-part selection, material mapping	gpt-5.2 ; reasoning effort low; max output 512 tokens; structured JSON output; seed 0 where supported; two schema-repair retries before fallback
Orientation VLM inside DCAP	gpt-5.2 ; reasoning effort low; max output 256 tokens; three rendered candidates; no access to final evaluation scores
VLM evaluator	gemini-3-pro-preview ; temperature 0; top-p 1.0; max output 512 tokens; randomized method order; separate prompts from the orientation module
Segmentation backend	OpenScene mesh features, Segment Anything ViT-H masks, and Mask2Former panoptic labels fused from 48 RGB-D render views; fallback to geometric support extraction when confidence is low
Renderer	Blender 4.2.3 with Eevee Next rasterization; 1024 $\times$ 1024 PNGs for VLM scoring and 768 $\times$ 768 panels for human trials; fixed focal length, exposure, and environment light across methods
Simulator	Genesis 0.2.1-compatible rigid–MPM backend with custom two-way frictional contact; Python 3.11, CUDA 12.4, single-precision GPU kernels

Appendix

Table 6: Concrete implementation settings for the reported experiments.

Object category	Min (m)	Max (m)
Ball	0.04	0.30
Pillow	0.30	0.80
Plush toy	0.12	0.65
Towel	0.35	1.60
Shirt / clothing	0.45	1.20
Dumbbell	0.20	0.55

Appendix

Table 7: Size sanity ranges.

Material ID	Density	Young’s modulus	Poisson ratio	Static friction	Dynamic friction
soft foam	35	0.08	0.30	0.70	0.55
firm foam	80	0.35	0.30	0.75	0.60
cotton fabric	250	0.02	0.35	0.65	0.50
wool fabric	180	0.015	0.35	0.70	0.55
leather	860	12	0.35	0.60	0.45
rubber	1100	5	0.49	0.95	0.80

Appendix

Table 8: Curated material library. Density is in kg/m 3 and Young’s modulus is in MPa.

Base category	Base found	Parts found	Fallback rate	Common failure
Bed	90.5%	83.7%	7.1%	Blanket/mattress boundary
Sofa	87.8%	79.4%	9.5%	Cushion vs. seat merging
Chair	85.2%	75.6%	11.0%	Thin legs and back occlusion
Shelf	88.6%	72.9%	8.3%	Small boards and clutter
Table / desk	92.1%	80.8%	5.6%	Merged tabletop objects
Floor / carpet	94.4%	84.0%	4.2%	Carpet/floor boundary

Appendix

Table 9: Segmentation and decomposition reporting table.

Module	Trigger	Recovery rule
LLM JSON output	Invalid schema or missing required key	Retry twice with parser error; if still invalid, use category prior and mark low confidence
Support grounding	Confidence $<0.6$ or no support mask above 0.55	Expand synonyms, rerun support query, then choose largest upward-facing compatible support if available
Scale prior	Predicted longest dimension outside category range	Clamp to Table 7 ; if clamping changes size by $>2\times$ , mark hard upstream failure
Target part / anchor	Invalid part, unavailable anchor, or confidence $<0.6$	Choose largest upward-facing compatible part and center anchor
Orientation	VLM confidence $<0.55$ or tied top candidates	Use canonical stable orientation with lowest estimated center of mass
Material mapping	Confidence $<0.6$ or unknown label	Use dictionary match if possible; otherwise default unknown and record material warning

Appendix

Table 10: Fallback triggers and recovery rules.

Submodule	Metric	Raw	Post-fallback
Base support selection (LLM)	Top-1 accuracy	91.8%	96.4%
Target sub-part selection (LLM)	Accuracy	84.6%	91.2%
Size prior sanity (LLM)	Within valid range	93.2%	98.1%
Material mapping (LLM)	Human-agreement / safe default	81.4%	94.7%
Orientation (VLM)	Accuracy / consistency	88.9%	92.1%
Segmentation (base instance)	Success rate	86.7%	93.8%

Appendix

Table 11: Submodule evaluation. Raw rates measure the first-pass module output; post-fallback rates include recovery and confidence gating.

Parameter	Value
Simulator / version	Genesis 0.2.1-compatible rigid–MPM backend with custom two-way contact kernels
Global timestep	1/240 s
Substeps	8
Gravity	9.81 m/s 2 along the negative vertical axis
Initial clearance $\delta$	0.03 m
Rigid–deformable coupling	Two-way frictional contact with velocity projection

Appendix

Table 12: Simulation configuration.

Diagnostic	Tolerance
Mean penetration depth	$\eta_{\mathrm{pen}}=1.0$ cm
Target drift	$\eta_{\mathrm{drift}}=0.15$ of the support OBB diagonal
Support contact ratio	$\eta_{\mathrm{contact}}=0.50$
Residual speed	$\eta_{\mathrm{res}}=3.0$ cm/s, with component-specific convergence thresholds in Table 12

Appendix

Table 13: Diagnostic tolerances used for feasibility decisions.

Item	Setting
Reported statistic	Mean wall-clock time over the 1,923 standard placement cases
Included stages	LLM/VLM API latency, scene rendering, segmentation lookup, target anchoring, material mapping, particle filling, rollout, diagnostics, and final rendering
Excluded stages	One-time TSDF/Poisson reconstruction, dataset download, manual benchmark annotation, and cached asset normalization
Parallelism	One case per GPU during simulation; CPU preprocessing uses 16 workers; model calls are issued synchronously per case for deterministic logs
Timing source	Python wall-clock timers around each pipeline stage, with CUDA synchronization before and after simulation kernels

Appendix

Table 14: Runtime measurement protocol.

Item	Setting / outcome
Source scenes	214 candidate indoor reconstructions
Accepted scenes	186 simulation-ready scenes
Rendered views	120 RGB-D views per scene on a hemispherical indoor camera path
Mesh extraction	TSDF fusion followed by screened Poisson meshing at 2 cm nominal resolution
Hole repair	Fill bounded holes below 0.15 m diameter
Component filtering	Remove disconnected components below 0.5% of scene area

Appendix

Table 15: Mesh conversion and quality-control log.

Support group	Cases	Scenes	Assets	Coverage focus
Bed / mattress	462	42	31	Soft supports; heavy rigid and plush objects
Sofa / cushion	407	37	29	Soft cushions, arms, backs, and seated supports
Chair / seat	310	31	23	Mixed rigid–soft seats and backs
Shelf / table / desk	504	42	34	Rigid supports, narrow anchors, and tabletop controls
Floor / carpet	240	24	20	Mixed carpet and open-floor placement
OOD / infeasible	140	10	18	Unusual feasible prompts and annotated impossible cases

Appendix

Table 16: Benchmark support-category split. Counts are prompt–asset–scene cases.

Split / regime	Cases	Use in evaluation
Standard placement split	1,923	Feasible in-distribution cases used for aggregate placement quality, VLM scoring, human preference, and objective physical diagnostics
Soft-contact regime	869	Cases dominated by compliant support or object behavior, such as mattresses, cushions, blankets, towels, clothing, and plush objects
Mixed-contact regime	550	Cases involving both rigid and compliant contact, such as chairs, carpeted floors, and partially soft supports
Rigid-contact regime	504	Rigid-support controls such as shelves, tables, desks, books, balls, and stable tabletop placement
Stress split	140	OOD or infeasible cases evaluated separately for feasibility behavior
Feasible OOD	64	Unusual but physically satisfiable prompts; returned feasible placements receive diagnostics

Appendix

Table 17: Reporting splits and prompt-regime composition.

Item	Setting
Renderer	Blender 4.2.3, Eevee Next rasterization, ambient occlusion enabled
Resolution	1024 $\times$ 1024 PNGs for VLM scoring; 768 $\times$ 768 panels for human trials; lossless PNG storage
Camera set	Five fixed canonical views per case: overview, left oblique, right oblique, elevated front, and target close-up
Lighting	Constant HDRI-style environment light plus weak area fill; no method-specific shadows or retouching
Normalization	Same object scale, material color, target support mesh, and camera intrinsics across all compared methods
Blinding	Human and VLM comparison panels randomize left/right order and omit method names

Appendix

Table 18: Rendering settings for method visualization and evaluation.

Method	Main adaptation	Collision handling	Notes
FirePlace reimplementation	Same object scale as DCAP; upward-surface sampling and plausibility scoring	Reject candidates with SDF penetration above 2 cm; no deformable settling	Strongest rigid placement baseline
LayoutGPT	Abstract layout coordinates mapped to scene OBB anchors	Project to nearest valid support; shallow collision rejection	Good semantic intent, weak contact modeling
Holodeck	Scene-generation priors adapted to existing reconstructions	Nearest-support projection and AABB-overlap rejection	Strong indoor commonsense but sensitive to coordinate mismatch

Appendix

Table 19: Baseline adaptation settings.

Item	Value
Number of participants	42
Total trials	2940
Preference trials	2520
Attention-check trials	420
Trials per participant	70, including 60 preference trials and 10 attention checks
Attention-check rate	14.3% of total trials (420 total checks; 10 per participant)

Appendix

Table 20: Human study reporting fields.

Metric	Contrast	Delta	95% CI
VLM P/N	DCAP full minus FirePlace raw	+21.2	[19.6, 22.8]
Human P/N	DCAP full minus FirePlace raw	+26.5	[22.1, 30.8]
Penetration depth	FirePlace raw minus DCAP full	+1.07 cm	[0.96, 1.19]
Support contact	DCAP full minus FirePlace raw	+0.24	[0.21, 0.27]
Residual speed	FP + rigid minus DCAP full	+4.9 cm/s	[4.2, 5.6]
VLM P/N	DCAP full minus FP + def.	+12.5	[10.9, 14.0]

Appendix

Table 21: 95% bootstrap confidence intervals for key standard-split deltas. Positive penetration and speed drops mean lower values for DCAP.

Video ID	Scene / prompt	Compared methods	Notes
V6589283	Bed: dumbbell on the center of the mattress	DCAP / FirePlace	Indentation and stable heavy-object contact
V3459802	Sofa: plush toy on the left cushion	DCAP / FirePlace / Holodeck	Soft support with mild plush deformation
V2349234	Shelf: book on the top shelf	DCAP / LayoutGPT	Rigid-support control case
V9097692	Chair: towel draped over the seat and back	DCAP / Holodeck	Soft-on-rigid settling and folding behavior
V4204730	OOD: oversized ball on a narrow shelf	DCAP / all baselines	Infeasibility and drift diagnostics

Appendix

Table 22: Supplementary video index.

3D Scene Layout Generation

3D scene layout generation arranges objects and structures into coherent, controllable scenes from text, images, plans, or functional goals.

See all 84 papers

ReRef-3D: A Benchmark for Spatial Referring Expression-Guided 3D Object Placement
Aug 17, 2026Mary Lynn Martin, Yifei Zhang, Martha Palmer +13D Spatial Reasoning 3D Scene Editing
University of Colorado Boulder
GALA: Geometry-Aware Language Model for Controllable Object Arrangement
Oct 20, 2026Chucheng Xiang, Runze Wang, Zhi Deng +103D Scene Layout Generation 3D VLMs
University of Science and Technology of China, Hefei, China · Tencent, Shenzhen, China · Tsinghua University, Shenzhen, China +1
presto: Efficient, Training-free, and Open-world Object Placement via Imaginary Search
Aug 21, 2026Weixuan Ding, Shang Liu, Hanyu Pei +1MLLM Reasoning Compositional Image Generation
Wuhan University, Wuhan Hubei Province, China · University of Louisville, Louisville KY, USA

Aug 17, 2026J/K move · Enter open · S save

ReRef-3D: A Benchmark for Spatial Referring Expression-Guided 3D Object Placement

Mary Lynn Martin, Yifei Zhang, Martha Palmer, Maria Leonor Pacheco

University of Colorado Boulder

3D Spatial Reasoning 3D Scene Editing Referring Expression Comprehension 3D Visual Grounding

Open paper AskPDF arXiv

We introduce ReRef-3D, a benchmark for language-guided single-object placement in controlled 3D tabletop scenes. It contains 33,826 instructions across 998 CLEVR-derived scenes, spanning 16 placement types and direct, one-hop, and two-hop references. Given that an instruction defines a region of acceptable placements rather than one coordinate, predictions are evaluated by relation satisfaction and physical validity. Each instruction also includes a verified natural rephrasing. We evaluate LLaVA-3D, 3D-LLM, 3D-LLaVA, LL3DA, and PlaceIt3D, with PlaceIt3D evaluated under both point-based and region-based supervision. Region-supervised PlaceIt3D achieves the highest placement accuracy at about 80% and LLaVA-3D reaches about 70%. Performance varies by relation type, but nearest and aligned remain the most difficult. Rephrasing has little effect on performance overall.

Abstract

3D Scene Layout Generation

Explore similar work

ReRef-3D: A Benchmark for Spatial Referring Expression-Guided 3D Object Placement

GALA: Geometry-Aware Language Model for Controllable Object Arrangement

presto: Efficient, Training-free, and Open-world Object Placement via Imaginary Search