Less Than Meets the Eye: Action-Set Identifiability and Principled Abstention for 3D Physical Geometry
Organizations: BRAC University · Deutsche Telekom AG
Abstract
A corridor, its mirror reflection, and its image on a display can look identical, yet their physical surfaces lie at different depths. We test whether bounded camera motions distinguish these explanations. Intervene3D takes an image and an initial geometry estimate, predicts the view under each hypothesis and candidate motion, and selects a motion by maximin separability. It reports the mechanism and first physical surface only if the available motions separate the hypotheses still in contention. Otherwise it abstains. An adequacy test also rejects observations that no hypothesis explains. On a matched synthetic benchmark with exact landmarks, disabling abstention gives confident answers on 76.4% of unresolvable scenes. With abstention, none receives a confident answer under exact or noisy landmarks. At 100% accuracy on answered scenes, maximin selection answers 61.0% of scenes and summed selection 43.1%. The adequacy test lowers wrong answers on strongly curved mirror scenes from 56.8% to 12.4%. Real display experiments compare two hypotheses using one frozen depth-network estimate. The second image supplies photometric evidence and is never passed through the depth network. One executed motion reduces the pooled median surface error of twelve networks by 31.5%. These networks have larger error inside the annotated panel than outside on 33.4% of illusion-benchmark images. They place the first physical surface behind the glass on 36.9% of Mirror3D frames.
Figures & tables
| Protocol | Contending hypothesis pairs | When equation 5 is applied |
|---|---|---|
| Synthetic | after the belief update; if none remain, the maximum-posterior hypothesis against each alternative | Select and execute one motion, update the belief, then test the remaining alternatives |
| Executed real motion | Median relative first-surface depth difference exceeds , determined before motion | Test predicted separability over registered candidate poses before using the selected target frame |
| Fixed stereo | The two hypotheses on the annotated region | Test photometric separability using the already captured right image |
| Every accepted motion | Within the envelope | |
| Region–checkpoint combinations | ||
| Abstention | ||
| Median surface error, answer / network | / | — |
| Answer worse than the network | of combinations | — |
| Checkpoints with lower median error | of | — |
| Mechanism accuracy, contending regions | of | — |
| Surface error (%), answer / network | ||||||
|---|---|---|---|---|---|---|
| Category | Images | Fooled, % (range) | Abstain, % | Committed acc., % | all answers | answers |
| Printed sheet on a table | ( – ) | / | / | |||
| Poster beside an object | ( – ) | / | / | |||
| Monitor in the frame † | ( – ) | / | / | |||
| Monitor filling the frame ‡ | ( – ) | / | — | |||
| All admissible | ( – ) | — | — | |||
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning |
|---|---|
| Initial image and camera pose | |
| Geometry source and initial estimate; a frozen depth network on real images, supplied landmarks in simulation | |
| Direct content; flat emissive panel; planar mirror; transparent pane | |
| Hypothesis family; the real illusion protocols use | |
| Bounded set of candidate camera motions and one motion | |
| Closed-form transition and geometry predicted under after |
| Parameter | Value |
|---|---|
| Base scenes, seeds | per seed with three mechanisms, with four; seeds, benchmark redrawn per seed; splits by base scene, |
| Camera | px, px; interface center offset m laterally and vertically relative to the camera, pointing jitter |
| Interface | distance m; half-width of the image half-width at that distance; aspect ; tilt up to |
| Content | landmarks, jittered stratified sampling over of the interface extent in a random subset of a grid; nearest depth behind the interface m, depth spread m |
| Display | static, or view-tracked with probability |
| Operator rig | markers; rig offset m laterally and m vertically, jitter m, forward offset m; resampled until the reflection is out of view at |
| Metric | Definition | Question answered |
|---|---|---|
| CEA | how often the mechanism is named correctly, abstention counted as an error | |
| Mechanism accuracy | over the stated evaluation subset | mechanism correctness over all regions or the contending subset, as specified |
| Committed acc. | how often a given answer is correct | |
| Abstention rate | how often an answer is withheld | |
| Abst. on resolvable | how often an answer is withheld that some action would warrant | |
| FPCR | how often a confident answer is given where none is warranted |
| Method | CEA | Committed | Abstain | Abst. res. | AUROC | FPCR |
|---|---|---|---|---|---|---|
| Single-view classifier | ||||||
| Passive multi-view | ||||||
| Next-best-view proxy | — | |||||
| Random intervention | ||||||
| Largest-translation | ||||||
| Ablations | ||||||
| Mirror acc. | Four mechanisms | |||||
|---|---|---|---|---|---|---|
| Objective | 3 mechanisms | 4 mechanisms | committed | abstains | abst. res. | CEA |
| Single-view classifier | — | — | ||||
| Passive multi-view | — | — | ||||
| Largest-translation | — | |||||
| Summed equation 6 | ||||||
| Maximin equation 7 | ||||||
| Maximin | Summed | Largest- translation | Forced maximin | ||||
|---|---|---|---|---|---|---|---|
| Budget | Resolvable | abstains | abst. res. | abstains | abst. res. | abst. res. | CEA |
| m, actions | |||||||
| m, actions | |||||||
| m, actions | |||||||
| Transmitting pane | Dielectric pane | |||
| Mechanism | resolvable | abstains | resolvable | abstains |
| Direct | ||||
| Emissive | ||||
| Reflection | ||||
| Transmission | ||||
| All, CEA / abst. res. | / | / | ||
| Exact encoder | Noisy encoder | ||||
|---|---|---|---|---|---|
| Held-out mechanism | Test fires | test off | test on | test off | test on |
| Spherical mirror, sag | |||||
| Spherical mirror, full sag range † , sag | |||||
| Cylindrical display, sag | |||||
| Water-like surface, reflection and refraction | |||||
| Three primary mechanisms, pooled | |||||
| Method | Clean CEA | Perturbed CEA | Change | |
|---|---|---|---|---|
| Passive multi-view | ||||
| Next-best-view proxy | ||||
| Random intervention | ||||
| Largest-translation | ||||
| Intervene3D-sum |
| Maximin | Summed | Random | Largest- translation | First frame | |
|---|---|---|---|---|---|
| Every accepted motion | |||||
| Within the envelope |
| Category | Fooled pairs | , fooled | , correct | Surface error, answer / predictor | Answer lower |
|---|---|---|---|---|---|
| Printed sheet on a table | / | ||||
| Poster beside an object | / | ||||
| Monitor in the frame | / | ||||
| Monitor filling the frame ‡ | / |
| Category | Error in / out | Median | |
|---|---|---|---|
| Printed sheet on a table | [ ] | [ ] | |
| Poster beside an object | [ ] | [ ] | |
| Monitor in the frame | [ ] | [ ] | |
| Monitor filling the frame | [ ] | [ ] | |
| All admissible | [ ] | [ ] |
| Illusion | Mirror3D | Executed motion | AUROC | |||||
| Checkpoint | Family | Params | Fooled | Behind glass | Predictor | Answer | Learned | Single |
| DA-V2-Small | Depth-Anything-V2 | M | ||||||
| DA-V2-Base | Depth-Anything-V2 | M | ||||||
| DA-V2-Large | Depth-Anything-V2 | M | ||||||
| DA3-Small | Depth-Anything-3 | M | ||||||
| DA3-Base | Depth-Anything-3 | M | ||||||
| Checkpoint | Family | Fooled | Error inside / outside | Surface behind glass |
|---|---|---|---|---|
| DA-V2-Small | Depth-Anything-V2 | |||
| DA-V2-Base | Depth-Anything-V2 | |||
| DA-V2-Large | Depth-Anything-V2 | |||
| DA3-Small | Depth-Anything-3 | |||
| DA3-Base | Depth-Anything-3 | |||
| DA3-Large | Depth-Anything-3 |
| Checkpoint | Single-layer | Multi-layer | Gap | First surface | Through glass |
|---|---|---|---|---|---|
| DA-V2-Small | |||||
| DepthPro |
| Confidently wrong surfaces | |||
|---|---|---|---|
| Checkpoint | Abstention | Network | Intervene3D |
| DA-V2-Small | |||
| DA-V2-Base | |||
| DA-V2-Large | |||
| DA3-Small | |||
| DA3-Base | |||
| Held out | Evaluation pool | Classifier | Inside-plane gap |
|---|---|---|---|
| Checkpoint | All categories | ||
| Checkpoint and images | All categories | — | |
| Checkpoint and scenes | All categories | — | |
| Scenes, within category | Printed sheet on a table | ||
| Scenes, within category | Poster beside an object | – | |
| Checkpoint | Exclude monitor in the frame | — |
| Benchmark | Resampled unit | Quantity | Network or detector | Intervene3D |
|---|---|---|---|---|
| Illusion, executed motion | scenes | median surface error | [ , ] | [ , ] |
| Illusion, stereo | photographs | surface made worse | [ , ] | [ , ] |
| ScanNet++ mirrors | scenes | confidently wrong | [ , ] | [ , ] |
| Method | Stereo | Executed motion |
|---|---|---|
| OWLv2 ( Minderer et al., 2023 ) | ||
| Grounding DINO | ||
| HetNet (PMD weights), then OWLv2 | ||
| HetNet (MSD weights), then OWLv2 | ||
| GDNet, then OWLv2 | ||
| Intervene3D |
| Motions | Maximin | Summed | Gap in points [min, max] |
|---|---|---|---|
| [ , ] | |||
| [ , ] | |||
| [ , ] | |||
| [ , ] | |||
| [ , ] | |||
| [ , ] |
| Committed | Reflection committed | CEA | Abstention | |
|---|---|---|---|---|
| Configuration | CEA | Abstention | Passive multi-view |
|---|---|---|---|
| Belief temperature , paired | |||
| (reported) | |||
3D Scene Understanding
3D scene understanding builds geometric and semantic representations of environments for spatial reasoning, perception, and interaction.
See all 311 papers