SwitchSplat: Learning Discrete Object States from Casually Captured Scenes

Yizheng Xie; Yalda Foroutan; George Shramko; Daniel Rebain; Shrisudhan Govindarajan; Andrea Tagliasacchi

cs.CVOct 17, 2026

SwitchSplat: Learning Discrete Object States from Casually Captured Scenes

Authors: Yizheng Xie, Yalda Foroutan, George Shramko, Daniel Rebain, Shrisudhan Govindarajan, Andrea Tagliasacchi

Organizations: Simon Fraser University, Canada · University of British Columbia, Canada · Wayve Labs, Canada · University of Toronto, Canada

Abstract

3D Gaussian Splatting has made high-fidelity 3D reconstruction fast and accessible, yet existing representations assume a static world. This assumption breaks down in casual captures, where a user naturally interacts with the environment -- opening doors, toggling switches, or moving objects -- introducing discrete state changes that no current framework can model. Time-conditioned dynamic methods such as 4DGS conflate when a state was observed with what state the scene is in, limiting them to replaying seen sequences rather than composing arbitrary object configurations. We present SwitchSplat, the first 3DGS representation to address this gap via a base+slot decomposition: a shared static base captures the invariant scene, while each slot independently holds the two binary states (on/off) of one interactable object. Because the assignment of Gaussians to slots is fully differentiable, the model discovers the geometry of each state directly from video, without manual annotation. At inference, the user can freely combine slot states and render the resulting scene from any viewpoint. We validate SwitchSplat on a new casual-capture dataset and show that it outperforms static and dynamic baselines in novel-view synthesis quality, while uniquely supporting rendering of state combinations never seen during capture. Project page: https://switchsplat.github.io

Figures & tables

Figure 1. SwitchSplat learns a state-aware scene representation from a casual capture ( top : input video frames showing the user interacting with the car). The scene is decomposed into a static base and a set of independently switchable slots, one per interactable object (here: right door, hood, trunk). Each slot models two binary states, enabling SwitchSplat to render any combination of object configurations from novel viewpoints at inference time ( bottom : four of the $2^{3}=8$ possible slot configurations, labeled by their on/off state vector). The top row is a filmstrip of a camera moving around a blue car while a person successively opens its side door, hood, and trunk. The bottom row contains four novel-view renderings of the car, each paired with labeled switches for the trunk, door, and hood. The examples show all three parts closed, only the hood open, only the trunk open, and both the trunk and hood open, illustrating independent composition of the three binary object states.

Figure 2. Pipeline – From a casual capture in which the operator interacts with objects (a), we fit a differentiable slot table (b) whose learnable endpoints $(t_{l},t_{r})$ and temperature ${rgb(0bp)=(0,0,0);rgb(25bp)=(0,0,0);rgb(75bp)=(0.5,0.5,0.5);rgb(100bp)=(0.5,0.5,0.5)}_{k}$ define a per-frame interaction probability $\pi_{k}(t)$ for each object. Each 3D Gaussian primitive carries a state-logit vector $\ell$ over a static base channel together with an off and an on channel per slot, yielding per-state probabilities (c). At render time (d), these probabilities are fused with $\pi_{k}(t)$ into a frame-conditional opacity $\tilde{\alpha}(t)$ that drives standard volumetric composition. All Gaussians, state logits, and slot-table parameters are optimized end-to-end under photometric losses, producing a state-aware scene that renders arbitrary object configurations from novel viewpoints. From left to right, the diagram shows filmstrips from a casual kitchen capture; four colored temporal slot windows aligned with icons for different interactable objects; Gaussian ellipses separated into dark-blue off, gray base, and light-blue on groups; projected Gaussian layers selected by the requested state; and kitchen renderings with an appliance closed or open. Arrows indicate that the slot probabilities and categorical Gaussian assignments determine state-conditioned opacity and the final rendering.

Figure 3. State-aware Gaussian – (left) Each Gaussian carries a learnable logit vector $\boldsymbol{\ell}$ that defines a categorical distribution $\mathbf{p}$ that softly assigns each primitive to the static base or “on”/“off” object states. (right) At render time, user specifies list of states that are active or inactive , and the probability of each gaussian being present in the scene $\omega(\tilde{\mathbf{p}})$ is computed via Eq. ( 1 ). An image of an open refrigerator is decomposed into three soft Gaussian-assignment layers representing the static base, the refrigerator's off state, and its on state. State-dependent binary weights of one, zero, and one retain the base and on-state layers while suppressing the off-state layer. The retained layers are summed to render the refrigerator in its open state.

Figure 4. Slot window function – The slot table

\pi_{k}(t)

is parameterized as a normalized product of two sigmoids controlled by learnable endpoints

(t_{l},t_{r})

and temperature

{rgb(0bp)=(0,0,0);rgb(25bp)=(0,0,0);rgb(75bp)=(0.5,0.5,0.5);rgb(100bp)=(0.5,0.5,0.5)}

. (left) Sweeping

{rgb(0bp)=(0,0,0);rgb(25bp)=(0,0,0);rgb(75bp)=(0.5,0.5,0.5);rgb(100bp)=(0.5,0.5,0.5)}

interpolates between sharp rectangular windows (low

{rgb(0bp)=(0,0,0);rgb(25bp)=(0,0,0);rgb(75bp)=(0.5,0.5,0.5);rgb(100bp)=(0.5,0.5,0.5)}

) and broad, soft transitions (high

{rgb(0bp)=(0,0,0);rgb(25bp)=(0,0,0);rgb(75bp)=(0.5,0.5,0.5);rgb(100bp)=(0.5,0.5,0.5)}

). (right) The resulting per-frame Bernoulli PMF over slot

k

’s state,

P(\mathcal{S}_{k,t})

. Green markers denote the two learnable endpoints

t_{l},t_{r}

, which can shift along the time axis during optimization. Two plots share vertical markers for the left endpoint, midpoint, and right endpoint of a temporal slot. The left plot overlays slot-window curves for temperatures 0.7, 2, and 6: the lowest temperature produces nearly rectangular transitions, while higher temperatures produce progressively broader, smoother shoulders. The right plot shows complementary Bernoulli probabilities, with the active-state probability near one inside the endpoints and the dashed inactive-state probability near one outside. Green markers and arrows identify the learnable endpoints.

Method	Outdoor		Indoor					Concurrent Indoor
	Balcony	Car	Kitchen 1	Flat 1	Flat 2	Flat 3	Chair	M Kitchen 1	M Kitchen 2	M Kitchen 3	Average
Static method
FullCircle (3DGS)	25.70	26.86	29.32	29.26	28.46	28.92	29.88	30.16	30.13	30.10	28.88
FullCircle (3DGUT)	25.77	26.77	29.35	29.27	28.62	29.14	30.06	30.38	30.23	30.35	28.99
Dynamic method
DeGauss	25.86	27.31	29.69	29.36	30.11	28.91	34.10	32.72	32.29	32.68	30.30

Table 1. Quantitative comparison on the main benchmark scenes with standard held-out views. We report PSNR (dB) on ten scenes grouped by capture setting. Concurrent indoor scenes contain multiple interactions occurring simultaneously. Best and second-best results are shown in bold and underlined , respectively.

Panel of Figure 5 — Figure 5. Qualitative comparison across three evaluation settings : standard held-out views, novel-camera views and novel state combinations. Columns show FullCircle, DeGauss, 4DGS, SwitchSplat, and the corresponding ground truth. A three-row, five-column comparison grid. Columns show FullCircle, DeGauss, 4DGS, SwitchSplat, and ground truth. The rows show a standard held-out kitchen view with an open drawer, a novel-camera view of an open refrigerator, and a novel-state kitchen configuration with multiple open compartments. The baseline renderings contain varying amounts of blur, ghosting, or incorrect object-state content, whereas the SwitchSplat renderings more closely preserve the sharp geometry and selected states visible in the ground truth.

	Kitchen 2			Oslo
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Novel camera frames
DeGauss †	32.38	0.965	0.174	29.40	0.960	0.207
4DGS †	32.05	0.967	0.168	29.93	0.966	0.194
SwitchSplat (3DGS)	33.90	0.974	0.155	31.82	0.972	0.186
SwitchSplat (3DGUT)	34.26	0.975	0.154	32.55	0.973	0.185

Table 2. Generalization to novel cameras and state combinations. † Dynamic baselines use a per-image oracle over time, selecting the highest-PSNR rendering from a set of sampled time; these results are upper bounds.

Slots	Slots opt.	$\mathcal{L}_{\text{compactness}}$	$\mathcal{L}_{\text{sparsity}}$	PSNR $\uparrow$
✗	–	–	–	31.98
✓	✗	✓	✓	34.25
✓	✓	✗	✓	32.88
✓	✓	✓	✗	34.29
✓	✓	✓	✓	34.31

Table 3. Ablation Study. We evaluate the impact of various components in our method by systematically excluding them and analyzing the reconstruction quality (PSNR) on the “Kitchen 2” scene. We report average PSNR across the 3 evaluation splits and 5 independent runs.

Panel of Figure 6 — Figure 6. Qualitative comparison. We compare FullCircle, DeGauss, 4DGS, and SwitchSplat against ground-truth images from standard held-out test views. Scenes are shown from top to bottom: Balcony, Balcony, Car, Flat 2, Flat 2, and Flat 3. A six-row, five-column image grid. Columns show FullCircle, DeGauss, 4DGS, SwitchSplat, and ground truth. From top to bottom, the rows contain two views of an outdoor grill on a balcony, a blue car with an open side door, an open closet, an open microwave, and an open kitchen cabinet. In the interacting object regions, the baseline renderings often contain blur, floating artifacts, or incomplete geometry; the SwitchSplat renderings more closely match the crisp object states and surrounding scene in the ground truth.

Panel of Figure 7 — Figure 7. Qualitative comparison. We compare FullCircle, DeGauss, 4DGS, and SwitchSplat against ground-truth images from standard held-out test views. Scenes are shown from top to bottom: Kitchen 1, Kitchen 1, Oslo, Flat 1, Flat 1, and Flat 3. A six-row, five-column image grid. Columns show FullCircle, DeGauss, 4DGS, SwitchSplat, and ground truth. From top to bottom, the rows show an open refrigerator, a countertop appliance, another open refrigerator, an open coat closet, an open laundry closet, and an open interior door. The baseline renderings frequently blur, duplicate, or omit parts of the interacting objects, while the SwitchSplat renderings more closely reproduce the object configurations and scene structure shown in the ground truth.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8. Golden-set benchmark construction. (a) The training sequence contains one interaction at a time, repeatedly returning to the base state. (b) We additionally recapture the scene under controlled object states using new camera trajectories. States observed during training evaluate novel camera trajectories, while previously unseen joint states evaluate novel state combinations.

Panel of Figure 9 — Figure 9. VLM prompt used for interaction-slot initialization. The prompt template is reproduced verbatim; only line wrapping is introduced for typesetting. Scene-dependent quantities, such as the valid frame range, are inserted at runtime.

PSNR $\uparrow$
Method	Outdoor		Indoor					Concurrent indoor			Avg.
Method	Balcony	Car	Kitchen 1	Flat 1	Flat 2	Flat 3	Chair	M Kitchen 1	M Kitchen 2	M Kitchen 3
FullCircle (3DGS)	26.87	27.88	31.68	29.30	30.40	30.72	30.57	29.76	28.32	27.30	29.28
FullCircle (3DGUT)	26.89	27.73	31.62	30.33	30.74	31.52	31.15	30.59	28.54	28.14	29.72
DeGauss	26.95	27.86	31.95	30.54	31.12	30.82	34.51	33.39	31.31	31.87	31.03
4DGS	26.70	29.19	31.37	31.58	30.66	30.48	32.64	32.65	31.81	31.69	30.88

Appendix

Table 4. Full per-scene quantitative results on the standard benchmark. We report PSNR, SSIM, and LPIPS on (a) base-state test frames outside interaction intervals and (b) all held-out test frames. SwitchSplat with 3DGUT is our canonical configuration; the 3DGS variant evaluates effect of the rendering backend.

Figure 10. VLM performance across reasoning-effort levels. We evaluate 10 independent predictions at each reasoning-effort level. Top: event-set Jaccard measures whether the VLM recovers the correct set of interaction slots; bottom: endpoint MAE measures temporal localization error. Dots denote individual runs and diamonds denote the mean at each effort level. Increasing reasoning effort generally improves both event recovery and endpoint localization. At maximum reasoning effort, 6 of 10 runs recover the event set perfectly (

100\%

Jaccard), with predicted transition endpoints localized to within only a few frames.

Figure 11. VLM slot-prediction failure modes. We aggregate approximately 3,000 predictions across five reasoning-effort levels, with 10 independent runs per level. Each bar reports the total number of occurrences of a failure mode, with colored segments indicating contributions from different reasoning-effort levels. Most errors arise from missing slots or over-segmentation, while incorrect merging is less common and false positives are rare. At higher reasoning effort, only a small number of failures remain, indicating that the VLM provides a highly reliable initialization in practice.

	Missing slots			Exact slots	Extra slots
Number of slots ( $K$ )	0	1	2	3	4	5	6
PSNR $\uparrow$	33.31	34.05	35.11	35.98	35.92	35.98	35.96
SSIM $\uparrow$	0.97	0.97	0.98	0.98	0.98	0.98	0.98
LPIPS $\downarrow$	0.19	0.18	0.18	0.17	0.17	0.17	0.17

Appendix

Table 5. Sensitivity to VLM-predicted slot count on the Kitchen 2 scene. The VLM predicts three interaction slots (

K{=}3

). Performance degrades when slots are removed, while adding up to three extra slots has little effect.

Figure 12. Sensitivity of slot initialization on Kitchen 2. The heatmaps report final temporal IoU and test PSNR across endpoint shifts and scales. The bottom rows compare the ground truth, initialization, and final learned intervals for six marked settings. Cases A–C recover from substantial perturbations, while D–F show failures under more extreme perturbations.

	Kitchen 2			Oslo
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Base-state test frames
FullCircle (3DGS)	33.82	0.972	0.175	33.20	0.976	0.190
FullCircle (3DGUT)	33.88	0.972	0.175	32.84	0.975	0.189
DeGauss	34.37	0.971	0.184	34.97	0.980	0.184
4DGS	33.79	0.975	0.171	34.68	0.981	0.178

Appendix

Table 6. Full quantitative evaluation on the Golden scenes. We report PSNR, SSIM, and LPIPS on Kitchen 2 and Oslo across base-state, standard-test, novel-camera, and novel state-combination frames. For the two OOD splits, DeGauss † and 4DGS † are evaluated using the oracle-over-time upper bound described in Section A.2 , while SwitchSplat is evaluated directly from the requested state code. Best results are shown in bold .

	Balcony			Car
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Full evaluation
DeGauss	25.86	0.833	0.320	27.31	0.851	0.285
4DGS	26.36	0.846	0.291	29.14	0.891	0.236
SwitchSplat (3DGS)	26.89	0.851	0.286	28.92	0.894	0.237
SwitchSplat (3DGUT)	26.97	0.852	0.286	29.03	0.895	0.237

Appendix

Table 7. Evaluation on outdoor scenes excluding moving backgrounds. We compare the original full-frame evaluation against an evaluation that excludes temporally varying background regions, including foliage, sky, clouds, and street activity. Results are reported on the full test set.

Scene	Articulated GS	SplArt	Independent 3DGUT	SwitchSplat (3DGUT)
USB	41.68	45.70	49.18	49.72
Blade	41.34	46.15	50.85	50.06
Folding chair	42.26	43.98	50.22	50.42
Fridge	37.26	41.40	43.10	44.18
Laptop	36.23	40.90	45.55	45.84
Oven	36.66	41.59	43.03	44.62

Appendix

Table 8. Quantitative comparison on the PARIS ( Liu et al., 2023 ) dataset. We report PSNR (dB) across all ten articulated objects. For ArticulatedGS ( Guo et al., 2025 ) , we average the published start- and end-state PSNR for each object; SplArt ( Lin et al., 2025 ) reports the corresponding per-object results directly. For 3DGUT ( Wu et al., 2025a ) we report the average of models trained on separate states. Best and second-best results are shown in bold and underline , respectively.

	Tested on pinhole			Tested on fisheye
Trained on	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Base-state test frames
Pinhole	36.17	0.979	0.167	34.88	0.979	0.090
Fisheye	35.66	0.978	0.167	37.09	0.983	0.084
All standard test frames
Pinhole	35.98	0.978	0.173	34.12	0.976	0.094

Appendix

Table 9. Pinhole vs. fisheye training on Kitchen 2. Rows give the training camera, and columns give the test camera. Compare numbers within a column only, because the two columns score different images. Best result per column in bold . On fisheye views, both models sort Gaussians by distance from the camera. Each row is a single training run.

Panel of Figure 13 — Figure 13. Pinhole vs. fisheye training, qualitative example. A novel state-combination view of Kitchen 2, with all three objects switched at once. The top row shows the ground truth, the other rows give the training camera, and the columns give the test camera. The pinhole-trained model blurs the fridge door and leaves floaters in front of the oven and on the floor, most visibly on the fisheye view. The fisheye-trained model is cleaner overall and closer to the ground truth. A three-row, two-column image grid of a kitchen. The left column shows pinhole images and the right column shows circular fisheye images on a white background. The top row is the ground truth, the middle row is the pinhole-trained model, and the bottom row is the fisheye-trained model.

	Without mask fine-tuning			With mask fine-tuning
Split	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Base state	37.09	0.983	0.084	36.92	0.983	0.084
Standard test	36.59	0.982	0.088	36.34	0.981	0.089
Novel camera	35.74	0.982	0.076	35.73	0.982	0.076
Novel combination	34.69	0.980	0.079	35.23	0.981	0.078

Appendix

Table 10. Mask fine-tuning improves novel state combinations. Kitchen 2, fisheye test views. The model without mask fine-tuning is the fisheye-trained model of Table 9 . Best result per split in bold . Each column group is a single training run.

Panel of Figure 14 — Figure 14. Automatic object mask. The open drawer of Kitchen 2 in one training frame. (a) Photo. (b) Render with the drawer switched back to its closed state. (c) $1-\mathrm{SSIM}$ between (a) and (b). (d) Pseudo-labels from thresholding (c): object (red) and background (blue). (e) Mask predicted by the MLP. (f) Final mask after SAM 2.1 propagation. Reflections on the steel fridge, including one of the person who holds the camera, also appear as change and produce wrong object pseudo-labels. The MLP inherits these errors and spills onto the fridge and the floor, whereas the propagated mask stays on the drawer. A two-row, three-column grid of circular fisheye images of a kitchen with an open freezer drawer, on a white background. The top row shows the photo, a render with the drawer closed, and a heat map of image differences that is bright on the drawer and on reflections of the steel fridge. The bottom row shows red and blue pseudo-label pixels, a yellow MLP mask that covers the drawer but also part of the fridge door and the floor, and a magenta final mask that covers only the drawer region.

Panel of Figure 15 — Figure 15. Mask fine-tuning removes haze and floaters. Kitchen 1 with all nine objects switched at once, seen from a new viewpoint with no ground truth. (Top) Without mask fine-tuning, haze and floaters appear in the scene. (Bottom) With mask fine-tuning, the scene is cleaner. Two renders of the same kitchen view, one above the other, without and with mask fine-tuning. A fridge, cabinets, and several appliances are open.

Panel of Figure 16 — Figure 16. Qualitative comparison. We compare FullCircle, DeGauss, 4DGS, and SwitchSplat against ground-truth images from standard held-out test views. Scenes are shown from top to bottom: Car, Chair, Flat2, Kitchen 1, Kitchen 1, Flat 3.

Panel of Figure 17 — Figure 17. Qualitative comparison. We compare FullCircle, DeGauss, 4DGS, and SwitchSplat against ground-truth images from standard held-out test views. Scenes are shown from top to bottom: Flat2, Flat2, Kitchen 1, Kitchen 1, Kitchen 1, Flat 3.

Figure 18. SwitchSplat dataset interaction events across 12 scenes featuring indoor, outdoor, rigid and non-rigid scenarios. Scenes contain 3–9 object-interaction episodes with various temporal structure: interactions may occur sequentially or overlap in time. Sementic labels shown are obtained through VLM predictions.

3D Scene Representation

3D scene representation studies how to encode geometry, appearance, semantics, and motion for reconstruction, rendering, generation, and interaction.

See all 240 papers

ZipSplat: Fewer Gaussians, Better Splats
Jun 3, 2026Alexander Veicht, Sunghwan Hong, Dániel Baráth +13D Gaussian Splatting Multi-View 3D Reconstruction
ETH Zürich · Microsoft
Accelerated Spatio-Temporal Gaussian Rendering via Kinematic and Semantic Priors
May 25, 2026Denis Gridusov, Maxim Popov, Sergey Kolyubin3D Gaussian Splatting 4D Reconstruction
ITMO University, St. Petersburg, Russia
StreamSplat: Streaming Feed-Forward 3D Gaussian Splatting
Aug 3, 2026Changhao Song, Yuxuan Wang, Qibiao Li +23D Gaussian Splatting Streaming 3D Reconstruction
University of Science and Technology of China

Jun 3, 2026J/K move · Enter open · S save

ZipSplat: Fewer Gaussians, Better Splats

Alexander Veicht, Sunghwan Hong, Dániel Baráth, Marc Pollefeys

ETH Zürich · Microsoft

3D Gaussian Splatting Multi-View 3D Reconstruction Feed-Forward 3D Reconstruction

Open paper AskPDF arXiv

Feed-forward 3D Gaussian Splatting reconstructs scenes from images in seconds. Pixel-aligned methods keep every Gaussian grounded in an observation, but produce redundant, potentially misaligned Gaussians across views. Token-based methods instead aggregate all views into a fixed set of learned tokens but lack explicit observation support and use a fixed reconstruction budget. We introduce ZipSplat, a method that predicts a compact, observation-grounded scene representation whose size is set at inference. It derives each scene token from an image patch rather than a learned query, and decodes it into freely placed 3D Gaussians. Tokens across views are merged into a compact set whose size is adjustable at inference. However, without geometric supervision, decoded Gaussians can drift from their source patch, and merging can mix tokens from different surfaces. We therefore introduce two geometric losses to ground and shape this representation. A cone loss ties each token's Gaussians to its source patch, and GeoCE aligns token similarity with 3D proximity so that tokens observing the same surface merge together. Building on these observation--geometry associations, we further introduce TokenBA, a token-level bundle adjustment that refines scene tokens and camera poses in a few seconds. ZipSplat requires neither camera poses nor intrinsics, yet outperforms pixel-aligned and token-based baselines on DL3DV and RealEstate10K, even those given ground-truth cameras. It further generalizes zero-shot to Mip-NeRF360 and ScanNet++. Our project page is at https://veichta.com/zipsplat.

Abstract

3D Scene Representation

Explore similar work

ZipSplat: Fewer Gaussians, Better Splats

Accelerated Spatio-Temporal Gaussian Rendering via Kinematic and Semantic Priors

StreamSplat: Streaming Feed-Forward 3D Gaussian Splatting