Topology-Aware 4D Gaussian Splatting for Egocentric Videos
Organizations: Bocconi University · Technical University of Munich · Google
Abstract
Reconstructing dynamic scenes and objects from monocular egocentric video is crucial for embodied AI, yet existing 4D Gaussian Splatting (4DGS) approaches struggle with everyday interactions involving topological changes, such as cutting, assembly, opening, or closing. Since 4DGS methods rely on deforming a fixed representation, events that split, merge, or reveal surfaces cause severe visual artifacts. To address this challenge, we introduce TopoEgo, the first benchmark dedicated to topological changes in egocentric video (covering fission, fusion, open, and close), alongside Topo-4DGS, a topology-aware reconstruction method. Topo4DGS jointly models part-level motion and time-varying surface support by assigning independent rigid transformations to interacting parts and dynamically activating or deactivating Gaussians as surfaces appear, disappear, or separate. Extensive experiments demonstrate that Topo-4DGS significantly outperforms state-of-the-art 4DGS approaches across topological changes, especially on post-event frames. Code and data will be made publicly available.
Figures & tables
| TopoEgo (EPIC-KITCHENS) | TopoEgo (Ego4D) | Rigid Objects Control | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | fission | fusion | open | close | fission | fusion | open | close | PSNR | SSIM | LPIPS |
| EgoGaussian [ 29 ] | 20.48 | 25.17 | 18.32 | 22.20 | 22.97 | 20.76 | 17.70 | 19.69 | 27.34 | 0.804 | 0.186 |
| DeGauss [ 24 ] | 21.39 | 25.60 | 18.58 | 23.24 | 24.37 | 22.23 | 19.51 | 20.99 | 25.10 | 0.742 | 0.232 |
| DeGauss (w/o hands) | 20.15 | 21.70 | 16.70 | 19.40 | 21.03 | 19.92 | 17.20 | 18.64 | 23.07 | 0.651 | 0.317 |
| Def-3DGS [ 28 ] | 21.93 | 26.03 | 18.93 | 23.58 | 24.64 | 22.54 | 19.67 | 21.31 | 24.37 | 0.732 | 0.243 |
| Def-3DGS (w/o hands) | 22.70 | 26.70 | 19.05 | 23.35 | 24.45 | 23.10 | 19.40 | 22.05 | 26.44 | 0.764 | 0.197 |
| Full Scene | Topo. Object | Post-Event Object | Other Dyn. Objects | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| EgoGaussian [ 29 ] | 23.82 | 0.729 | 0.249 | 20.91 | 0.638 | 0.247 | 16.79 | 0.464 | 0.300 | 20.51 | 0.602 | 0.302 |
| DeGauss [ 24 ] | 26.64 | 0.837 | 0.264 | 21.99 | 0.698 | 0.257 | 18.76 | 0.591 | 0.310 | 23.65 | 0.754 | 0.224 |
| DeGauss (w/o hands) | 23.44 | 0.844 | 0.271 | 19.34 | 0.704 | 0.264 | 16.55 | 0.598 | 0.318 | 20.81 | 0.760 | 0.231 |
| Def-3DGS [ 28 ] | 27.15 | 0.846 | 0.256 | 22.33 | 0.705 | 0.256 | 19.31 | 0.600 | 0.312 | 23.97 | 0.752 | 0.223 |
| Def-3DGS (w/o hands) | 27.80 | 0.858 | 0.263 | 22.60 | 0.711 | 0.266 | 19.83 | 0.611 | 0.320 | 24.69 | 0.764 | 0.230 |
| Config | Topo. Object | Post-Event | |||||||
|---|---|---|---|---|---|---|---|---|---|
| MO | MB | BC | TA | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| ✗ | ✗ | ✗ | ✗ | 21.11 | 0.664 | 0.278 | 18.45 | 0.579 | 0.286 |
| ✓ | ✗ | ✗ | ✗ | 21.72 | 0.681 | 0.270 | 19.07 | 0.607 | 0.274 |
| ✓ | ✓ | ✗ | ✗ | 23.84 | 0.745 | 0.190 | 23.05 | 0.698 | 0.222 |
| ✓ | ✓ | ✓ | ✗ | 23.92 | 0.755 | 0.179 | 23.27 | 0.709 | 0.199 |
| ✓ | ✓ | ✓ | ✓ | 25.32 | 0.786 | 0.158 | 26.77 | 0.780 | 0.151 |
| Topo. Object | Post-Event | |||||
|---|---|---|---|---|---|---|
| Variant | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| UT-MLP | 23.18 | 0.701 | 0.270 | 19.41 | 0.622 | 0.277 |
| Fixed-KM | 22.95 | 0.703 | 0.267 | 19.59 | 0.633 | 0.263 |
| w/o TR | 21.60 | 0.662 | 0.274 | 18.26 | 0.596 | 0.275 |
| w/o JSR | 21.61 | 0.665 | 0.277 | 18.57 | 0.607 | 0.272 |
| w/o SR | 23.25 | 0.693 | 0.293 | 20.80 | 0.628 | 0.293 |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Main paper section | Extended material | Sec. |
|---|---|---|
| Sec. 5.1 Motion Bifurcation | Spatial and Temporal Branch normalization and parameterization | Sec. D.3 |
| Sec. 5.1 Motion Bifurcation | Spatial Branch initialization and learned partition | Sec. D.4 |
| Sec. 5.3 Optimization | Objectives, masking, and spatial regularization | Sec. D.5 |
| Sec. 5.2 Topological Boundary Resolution | Time Refinement | Sec. F.1 |
| Sec. 5.3 Optimization | Missing-pose interpolation and its ablation | Secs. E and 9 |
| Sec. 5.2 Topological Boundary Resolution | Boundary construction and branch binding | Sec. F.2 |
| Category | #Actions | #Videos | Coverage |
|---|---|---|---|
| Ego4D ( videos) | |||
| Topological (four target families) | |||
| fission ( : cut/break/tear) | |||
| fusion ( : attach/sew/weld) | |||
| open (separable containers) | |||
| close (separable containers) | |||
| Event family | Inclusion criterion | Effect on represented surface support |
|---|---|---|
| Fission | A manipulated object separates into independently moving pieces through cutting, fracture, tearing, detachment, or disassembly. | Newly exposed cut or separation surfaces may become visible and require representation. |
| Fusion | Independently moving pieces are joined into a single coupled object. | Contacting surfaces become sealed or occluded and may be removed from the active representation. |
| Open | Parts of a separable assembly move apart, exposing an interior (e.g., removing a lid or opening a container). | Previously concealed interior surfaces become visible and enter the active representation. |
| Close | Parts of a separable assembly are rejoined, concealing its interior. | Interior and contact surfaces become concealed and leave the active representation. |
| Clips | |||||
|---|---|---|---|---|---|
| Event family | EPIC-K. | Ego4D | Frames | Objects | Events |
| fission | 3 | 3 | 1,922 | 12 | 11 |
| fusion | 3 | 3 | 939 | 8 | 7 |
| open | 3 | 3 | 810 | 6 | 7 |
| close | 3 | 3 | 947 | 10 | 9 |
| Total | 12 | 12 | 4,618 | 36 | 34 |
| PSNR | LPIPS | |||||
| Method | ||||||
| Baselines | ||||||
| Def-3DGS [ 28 ] | 20.27 | 16.43 | 12.46 | 0.252 | 0.267 | 0.389 |
| DeGauss [ 24 ] | 19.62 | 16.12 | 11.34 | 0.265 | 0.276 | 0.432 |
| EgoGaussian [ 29 ] | 20.12 | 16.53 | 15.71 | 0.238 | 0.263 | 0.392 |
| 4DGS [ 26 ] | 17.25 | 13.20 | 9.45 | 0.385 | 0.491 | 0.546 |
| Topo. Object (region) | ||||
|---|---|---|---|---|
| PSNR | SSIM | LPIPS | Rate (it/s) | |
| 1 | 20.62 | 0.741 | 0.198 | 43.83 |
| 2 (default) | 23.06 | 0.781 | 0.173 | 37.24 |
| 4 | 22.86 | 0.773 | 0.176 | 34.13 |
| 8 | 23.43 | 0.786 | 0.176 | 33.62 |
| 16 | 23.31 | 0.778 | 0.173 | 33.48 |
| Full scene | Topo. Object (region) | |||||
| Event times | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| Ground-truth | 26.92 | 0.842 | 0.249 | 26.66 | 0.815 | 0.125 |
| Automatically predicted | 26.87 | 0.838 | 0.253 | 26.59 | 0.805 | 0.139 |
| Offset frames | 26.15 | 0.859 | 0.272 | 26.19 | 0.789 | 0.178 |
| Offset frames | 26.17 | 0.859 | 0.261 | 26.52 | 0.796 | 0.141 |
| Offset frames | 26.19 | 0.860 | 0.263 | 26.58 | 0.798 | 0.149 |
| Full scene | Topo. Object (region) | |||||
|---|---|---|---|---|---|---|
| Training input | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| Default training set | 27.42 | 0.817 | 0.205 | 26.78 | 0.812 | 0.132 |
| training frames | 26.43 | 0.842 | 0.269 | 22.86 | 0.760 | 0.202 |
| training frames | 25.34 | 0.758 | 0.234 | 22.43 | 0.657 | 0.183 |
| Source | Event | Method | PSNR | SSIM | LPIPS | -IoU |
|---|---|---|---|---|---|---|
| EPIC-K. | fission | EgoGaussian | 19.46 | 0.597 | 0.364 | 0.465 |
| Topo-4DGS | 21.92 | 0.650 | 0.355 | 0.488 | ||
| fusion | EgoGaussian | 13.75 | 0.218 | 0.441 | 0.066 | |
| Topo-4DGS | 25.21 | 0.734 | 0.309 | 0.507 | ||
| open | EgoGaussian | 18.60 | 0.608 | 0.309 | 0.514 | |
| Topo-4DGS | 21.68 | 0.643 | 0.293 | 0.574 |
| Full scene | Topo. Object (region) | Other Dyn. Objects (region) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Clip | Method | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| P01_105 † | HyperNeRF | 19.46 | 0.707 | 0.337 | 12.78 | 0.414 | 0.357 | 11.09 | 0.224 | 0.364 |
| Topo-4DGS | 30.37 | 0.946 | 0.102 | 24.17 | 0.888 | 0.108 | 21.07 | 0.815 | 0.120 | |
| P28_105 † | HyperNeRF | 19.28 | 0.550 | 0.340 | 16.92 | 0.513 | 0.450 | – | – | – |
| Topo-4DGS | 35.37 | 0.966 | 0.071 | 31.50 | 0.907 | 0.137 | – | – | – | |
| P03_120 † | HyperNeRF | 23.11 | 0.771 | 0.244 | 16.51 | 0.508 | 0.392 | 14.35 | 0.176 | 0.471 |
| Topo-4DGS [Ours] | EgoGaussian [ 29 ] | 4DGS [ 26 ] | Def-3DGS [ 28 ] | DeGauss [ 24 ] | ||||||||||||
| Clip | Type | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| EPIC-KITCHENS | ||||||||||||||||
| P02_108 | fusion | 33.27 | 0.964 | 0.045 | 29.05 | 0.748 | 0.119 | 31.19 | 0.807 | 0.196 | 32.34 | 0.893 | 0.125 | 31.70 | 0.860 | 0.139 |
| P02_122 | fission | 32.67 | 0.961 | 0.057 | 28.76 | 0.759 | 0.136 | 29.15 | 0.753 | 0.280 | 30.83 | 0.843 | 0.169 | 31.24 | 0.865 | 0.160 |
| P04_04 | fission | 33.85 | 0.964 | 0.086 | 22.02 | 0.641 | 0.094 | 32.17 | 0.939 | 0.097 | 32.19 | 0.945 | 0.090 | 30.29 | 0.915 | 0.127 |
| P04_118 | close | 26.08 | 0.889 | 0.302 | 20.91 | 0.696 | 0.447 | 22.94 | 0.736 | 0.531 | 23.96 | 0.821 | 0.458 | 23.39 | 0.787 | 0.479 |
| Topo-4DGS [Ours] | EgoGaussian [ 29 ] | 4DGS [ 26 ] | Def-3DGS [ 28 ] | DeGauss [ 24 ] | ||||||||||||
| Clip | Type | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| EPIC-KITCHENS | ||||||||||||||||
| P02_108 | fusion | 28.17 | 0.866 | 0.075 | 25.85 | 0.764 | 0.077 | 26.00 | 0.752 | 0.083 | 27.12 | 0.828 | 0.078 | 26.65 | 0.796 | 0.080 |
| P02_122 | fission | 20.91 | 0.769 | 0.113 | 19.10 | 0.681 | 0.123 | 18.85 | 0.668 | 0.149 | 20.02 | 0.737 | 0.128 | 19.55 | 0.708 | 0.136 |
| P04_04 | fission | 25.78 | 0.862 | 0.219 | 18.50 | 0.560 | 0.280 | 18.20 | 0.550 | 0.410 | 21.20 | 0.760 | 0.292 | 20.20 | 0.680 | 0.335 |
| P04_118 | close | 21.96 | 0.701 | 0.301 | 19.20 | 0.611 | 0.418 | 19.35 | 0.620 | 0.451 | 20.18 | 0.665 | 0.429 | 19.82 | 0.640 | 0.440 |
| Topo-4DGS [Ours] | EgoGaussian [ 29 ] | 4DGS [ 26 ] | Def-3DGS [ 28 ] | DeGauss [ 24 ] | |||||||||||||
| Clip | Type | Object | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| EPIC-KITCHENS | |||||||||||||||||
| P02_122 | fission | compost_bin | 36.58 | 0.945 | 0.074 | 29.20 | 0.820 | 0.145 | 29.75 | 0.842 | 0.132 | 32.95 | 0.905 | 0.092 | 31.90 | 0.885 | 0.105 |
| P04_04 | fission | knife | 27.82 | 0.834 | 0.193 | 17.10 | 0.340 | 0.430 | 18.05 | 0.455 | 0.380 | 21.45 | 0.676 | 0.285 | 21.20 | 0.661 | 0.295 |
| P04_118 | close | cutting_board | 24.20 | 0.798 | 0.257 | 19.85 | 0.662 | 0.477 | 20.55 | 0.690 | 0.464 | 24.08 | 0.842 | 0.400 | 23.82 | 0.834 | 0.403 |
| P06_110 | fission | knife | 17.83 | 0.705 | 0.151 | 15.85 | 0.595 | 0.250 | 16.15 | 0.625 | 0.225 | 17.05 | 0.668 | 0.178 | 16.88 | 0.690 | 0.171 |
| Clip | Type | Topo-4DGS [Ours] | EgoGaussian [ 29 ] | 4DGS [ 26 ] | Def-3DGS [ 28 ] | DeGauss [ 24 ] | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | ||
| EPIC-KITCHENS | ||||||||||||||||
| P02_108 | fusion | 26.04 | 0.809 | 0.105 | 21.16 | 0.571 | 0.132 | 21.70 | 0.597 | 0.160 | 23.34 | 0.737 | 0.137 | 22.38 | 0.661 | 0.144 |
| P02_122 | fission | 26.94 | 0.824 | 0.083 | 21.14 | 0.578 | 0.146 | 20.75 | 0.564 | 0.171 | 23.19 | 0.737 | 0.143 | 22.07 | 0.655 | 0.151 |
| P04_04 | fission | 26.28 | 0.872 | 0.351 | 13.52 | 0.356 | 0.388 | 13.24 | 0.426 | 0.553 | 20.78 | 0.580 | 0.417 | 20.20 | 0.535 | 0.426 |
| P04_118 | close | 22.10 | 0.718 | 0.388 | 17.47 | 0.487 | 0.462 | 17.57 | 0.503 | 0.546 | 19.20 | 0.562 | 0.474 | 18.59 | 0.569 | 0.477 |
| Topo-4DGS [Ours] | EgoGaussian [ 29 ] | Def-3DGS [ 28 ] | DeGauss [ 24 ] | 4DGS [ 26 ] | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Clip | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| P03_03 | 32.24 | 0.939 | 0.139 | 32.73 | 0.940 | 0.125 | 31.92 | 0.931 | 0.146 | 31.55 | 0.925 | 0.153 | 31.70 | 0.928 | 0.150 |
| P17_01 | 22.06 | 0.767 | 0.372 | 18.67 | 0.685 | 0.431 | 21.74 | 0.752 | 0.384 | 21.48 | 0.743 | 0.392 | 21.31 | 0.738 | 0.398 |
| 35fa3 | 30.11 | 0.938 | 0.176 | 31.53 | 0.946 | 0.169 | 29.86 | 0.941 | 0.099 | 30.71 | 0.946 | 0.097 | 22.49 | 0.755 | 0.316 |
| P18_06 | 38.21 | 0.976 | 0.107 | 37.58 | 0.975 | 0.106 | 35.30 | 0.973 | 0.133 | 33.63 | 0.963 | 0.144 | 32.37 | 0.916 | 0.284 |
| P32_01 | 29.57 | 0.890 | 0.197 | 30.49 | 0.922 | 0.172 | 32.33 | 0.947 | 0.132 | 29.30 | 0.922 | 0.163 | 33.92 | 0.944 | 0.151 |
| Topo-4DGS [Ours] | EgoGaussian [ 29 ] | Def-3DGS [ 28 ] | DeGauss [ 24 ] | 4DGS [ 26 ] | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Clip | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| P03_03 | 29.13 | 0.900 | 0.120 | 29.71 | 0.902 | 0.109 | 28.31 | 0.881 | 0.139 | 28.82 | 0.892 | 0.128 | 28.55 | 0.886 | 0.134 |
| P17_01 | 21.79 | 0.657 | 0.269 | 21.58 | 0.583 | 0.285 | 21.18 | 0.631 | 0.289 | 21.42 | 0.642 | 0.280 | 20.96 | 0.623 | 0.296 |
| 35fa3 | 25.36 | 0.765 | 0.214 | 26.24 | 0.773 | 0.188 | 23.87 | 0.658 | 0.253 | 23.67 | 0.634 | 0.237 | 20.29 | 0.486 | 0.378 |
| P18_06 | 33.68 | 0.934 | 0.079 | 32.72 | 0.928 | 0.084 | 19.20 | 0.589 | 0.318 | 19.37 | 0.599 | 0.320 | 20.63 | 0.590 | 0.369 |
| P32_01 | 27.93 | 0.851 | 0.240 | 28.82 | 0.879 | 0.218 | 25.11 | 0.823 | 0.283 | 29.00 | 0.878 | 0.241 | 31.05 | 0.897 | 0.214 |
4D Reconstruction
4D reconstruction recovers changing 3D scenes, shapes, and motion over time from images, videos, or other sparse measurements.
See all 149 papers