NeuSE: Neural SE(3)-Equivariant Embedding for Long-Term Object-Based Simultaneous Localization and Mapping

Jiahui Fu; Yilun Du; Kurran Singh; Joshua B. Tenenbaum; John J. Leonard

cs.ROMar 13, 2023

NeuSE: Neural SE(3)-Equivariant Embedding for Long-Term Object-Based Simultaneous Localization and Mapping

Authors: Jiahui Fu, Yilun Du, Kurran Singh, Joshua B. Tenenbaum, John J. Leonard

Organizations: MIT CSAIL

Abstract

We present NeuSE, a novel Neural SE(3)-Equivariant Embedding for objects, and illustrate how it supports object-based Simultaneous Localization and Mapping (SLAM) for consistent spatial understanding with long-term scene changes. NeuSE is a set of latent object embeddings created from partial object observations. It serves as a compact point cloud surrogate for complete object models, encoding the full shape, scale, and transform information about an object. In addition, the inferred latent code is both SE(3) and scale equivariant, enabling strong generalization to objects of both unseen sizes and different SE(3) poses. This makes NeuSE particularly effective in real-world scenarios where objects may vary in size or spatial configuration. With NeuSE, relative frame transforms can be directly derived from inferred latent codes. Our proposed SLAM paradigm, using NeuSE for object shape, size, and pose characterization, can operate independently or in conjunction with typical SLAM systems. It directly infers SE(3) camera pose constraints that are compatible with general SLAM pose graph optimization, while maintaining a lightweight, object-centric map that adapts to real-world changes. Our evaluation is conducted on synthetic and real-world sequences with changes in both controlled and uncontrolled settings, featuring multi-category objects of various shapes and sizes. Our approach demonstrates improved localization capability and change-aware mapping consistency when working either independently or as a complement to common SLAM pipelines. Additional results are presented in the supplementary video: https://youtu.be/fZ4_2P1fQBQ

Figures & tables

Figure 1: Schematic of consistent spatial understanding with NeuSE . An object-centric map of multiple object categories constructed from real-world data is shown for illustration. (a) NeuSE acts as a compact point cloud surrogate for objects, encoding full object shapes and sizes and transforming $\text{SE}(3)$ -equivariantly with the object in the physical world. Latent codes of bottles and mugs from different frames can be effectively associated (dashed line) for direct computation of inter-frame transforms, which are then added to constrain camera pose ( ${T}_{i}$ ) optimization both locally ( $T_{Li}$ ) and globally ( $T_{Gi}$ ). (b) The system performs change-aware object-level mapping, where changed objects (highlighted in orange ) are updated alongside unchanged ones with full shape reconstructions in the object-centric map. (c) With $\text{SE}(3)$ and scale equivariance, NeuSE effectively handles multiple instances of identically shaped objects (dark bottles $d_{1}$ and $d_{2}$ ), as well as objects with diverse shapes (bottles, mugs, and cans) and sizes (big white bottle $w_{1}$ and small white bottle $w_{2}$ ).

Figure 2: System overview . The input to the system is a depth image and segmentation masks of objects of interest, generated from the corresponding RGB images, which together create partial object point clouds. The system’s six modules then process these partial point clouds to interact with and update object and frame information in the database. The output of the system can be retrieved from the database as camera pose estimates for input frame sequences and the latest object-centric map of the environment, consisting of full shape reconstructions of the objects of interest.

Figure 3: Illustration of $\text{SE}(3)$ and scale equivariance. Equivariance means that the latent representation, though not visually resembling the physical object, transforms in the latent space in the same way as the object does in the real world. (a) With $\text{SE}(3)$ equivariance, the latent code undergoes the same $\text{SE}(3)$ transform $\mathbf{T}$ applied to the mug in the physical world. (b) With scale equivariance, the latent code scales by the same factor $s$ as the bottle in the physical world.

Figure 4: (a) Breaking pose ambiguity with covisible ambiguous objects . Motions around a bottle’s axis of symmetry result in seemingly identical observations, making it impossible to determine inter-frame transforms. However, with two covisible bottles, the intersection ( green ) of their camera pose distributions ( yellow and blue ) for the current observation reveals the true camera pose, where inter-frame transforms can then be determined without ambiguity. (b) Latent symmetry . The canonicalized latent embedding should be invariant to camera motion ( ${\bf T}_{1i}$ ) around the object’s axis of symmetry, inducing consistently small Chamfer distance between the transformed bottle ( ${{\bf T}_{1,2}\bf P}_{1i}$ ) and the target point cloud .

Figure 5: Data association. With NeuSE as the object representation, we utilize semantic labels, scale, shape, and spatial proximity for effective data association. After associating the unambiguous mug observation with the correct purple mug in the database, the bottle observation undergoes four steps to eliminate incorrect candidates in the dashed gray box and identify the correct green bottle : (a) Semantic matching selects the bottle-labeled instances from the object library; (b) Scale comparison rules out the smaller orange bottle ; (c) Shape comparison eliminates the yellow bottle with a different shape; and (d) spatial proximity comparison identifies the correct green bottle , as the transform derived from the combination of the purple mug and green bottle results in minimal center difference for their partial point clouds.

Figure 6: Pose graph optimization . With objects observed in periods of consecutive frames, we derive from corresponding latent codes (1) short-range odometry constraints ( gray ) within a local

K

-frame sliding window, and (2) global loop closure constraints (black) between the current (

{\bf T}_{N}

) and the first frame of each of its previous consecutive observable periods (

{\bf T}_{1}

and

{\bf T}_{M}

), which are working jointly to constrain the pose graph optimization.

Figure 7: Object layout comparison through graph matching . Object graphs are constructed for the current frame ( $G$ ) and the library ( $G^{\prime}$ ). For object $a$ and $b$ , which are similar in shape to the blue mug and pink bottle in the library, respectively, the inter-object distances between them and the anchor objects in the four corners are computed and compared. (a) All corresponding edges (dashed and solid lines) with anchor objects have similar oriented lengths, indicating that the mug is unchanged but was seen with an occluded handle, leading to a false ambiguous transform by the latent code. (b) There are no similar edges, indicating a different layout with the bottle moved.

Figure 8: Equivariance error distribution. Scatter plots showing the error distributions from the testing data across five object categories in three equivariance experiments. (a) Most normalized RMSE values across the full rotation range $(0^{\circ},180^{\circ})$ are bounded within a range close to zero, indicating small differences between the latent codes of the original and rotated objects, thus demonstrating strong $\text{SO}(3)$ -equivariance. (b) The RMSE values between predicted and ground truth translations are concentrated below 0.01 m, reflecting NeuSE’s high translation encoding accuracy. (c) For scale changes ranging from one to nearly seven times the original size, the errors between the estimated and ground truth scale ratios average around 0.011, with most error values ranging between -0.01 and 0.03, suggesting NeuSE’s scale equivariance across object sizes.

Figure 9: Robust reconstruction with equivariance. Examples of robust object reconstructions (Ours) compared to the ground truth (GT). Yellow points represent partial point cloud observations, and the colored reconstructions are shown from viewing angles that emphasize the occluded parts completed by NeuSE. Across the three transformations applied to the example objects: (a) $\text{SO}(3)$ rotation on the mug , (b) 3D translation on the bowl , and (c) scaling on the bottle , NeuSE effectively encodes and reconstructs the full object shape, consistently aligning with partial observations from both the original and transformed states.

Figure 10: Synthetic data overview . Table layouts with object changes and the ground truth camera trajectories of (a) the mug-bottle, 10-table setting, and (b) the multi-category, 12-table setting.

	Planar				Non-planar
	Mug-only	Mug + Bottle	Multi ( $\text{SE}(3)$ -Only)	Multi	Mug-only	Mug + Bottle	Multi ( $\text{SE}(3)$ -Only)	Multi
RMSE of ATE (m)
$1^{\text{st}}$ traversal	0.072	0.043	0.034	0.021	0.058	0.045	0.028	0.026
$2^{\text{nd}}$ traversal	0.096	0.071	0.049	0.041	0.077	0.033	0.041	0.029
Full	0.116	0.065	0.052	0.047	0.091	0.053	0.042	0.036
RMSE of Trans RPE (m/f)

Table 1: RMSE of ATE and translational RPE on synthetic sequences. Best results are marked in bold.

Figure 11: (a)–(c): Comparison of estimated and ground truth (GT) trajectories on synthetic sequences. (a) Mug + Bottle planar sequences. (b) Mug + Bottle non-planar sequences. (c) Multi-category planar (left) and non-planar (right) sequences. Trajectory color variation indicates ATE distribution. The decreasing ATE values, shown by lighter trajectory colors from (a) to (c) , demonstrate the efficacy of NeuSE’s $\text{SE}(3)$ -equivariance in handling randomly oriented objects from (a) to (b) and highlight the benefit of including ambiguous objects for improved inter-frame transform prediction. (d) and (e): Ambiguous objects for inter-frame transform prediction. In (d) , object point clouds are transformed from the orange frame to the green frame using transforms derived from the pink mugs alone, or together with green bottles . The improved point cloud alignment in (e) , using all the objects (Mug + Bottle) compared to Mug-only, demonstrates the effectiveness of using covisible ambiguous objects to enhance transform prediction accuracy.

	Planar		Non-planar
	PC-based	Ours	PC-based	Ours
${1^{\text{st}}}$ traversal	83.6%	91.2%	79.7%	90.3%
Full	37.1%	88.6%	36.6%	89.1%

Table 2: Data association accuracy comparison with point clouds (PC-based) and NeuSE (Ours). Best results are in bold.

Figure 12: Comparison of estimated trajectories on synthetic sequences with point cloud-based (PC-based) and NeuSE-based (Ours) data association. (a) Results on planar multi-category sequence. (b) Results on non-planar multi-category sequence. Trajectory color variation indicates ATE distribution. Our approach with NeuSE-based data association consistently improves external noisy odometry by encoding full object shapes to handle partial observations, outperforming the point cloud-based method in both sequences. The point cloud-based approach struggles more on the non-planar sequence due to the more complex occlusion patterns and the reduced observation overlap from a mix of upright and laid-down objects.

	Planar			Non-planar
	Raw Odometry	PC-based	Ours	Raw Odometry	PC-based	Ours
RMSE of ATE (m)
$1^{\text{st}}$ traversal	0.406	0.301	0.236	0.412	0.416	0.164
$2^{\text{nd}}$ traversal	0.453	0.629	0.398	0.456	0.628	0.339
Full	1.064	0.917	0.389	1.083	1.01	0.348
RMSE of Trans RPE (m/f)

Table 3: RMSE of ATE and translational RPE of camera trajectory estimates with point cloud-based and NeuSE-based data association. Best results are marked in bold.

Figure 13: (a) Ground truth scene setup: A mug is centered with six bottles of increasing size around it. The camera orbits from the largest purple bottle to the smallest yellow bottle , creating varying mug visibility along the trajectory. (b) Translational RPE curves (b.1) against mug visibility ratio (b.2) , plotted with a shared x-axis of frame index. NeuSE exhibits robustness to occlusion at lower noise levels, while a higher 8 mm noise level produces a U-shaped RPE curve, indicating increased sensitivity to occlusion under noisier depth data.

Figure 14: Visualization of estimated trajectories against ground truth (GT) for the original and corrupted data at varying noise levels. Trajectory color variation indicates ATE distribution. (a) also shows object reconstructions, illustrating the camera’s approximately circular path around the object setup. From (a) to (d) , NeuSE remains robust at lower noise levels (2 mm and 5 mm), recovering camera trajectories that closely align with the ground truth. At the higher noise level (8 mm), the estimated trajectory becomes jagged, especially near the start and end where mug visibility is low, indicating NeuSE’s sensitivity to occlusion under high depth noise.

	RMSE of ATE (m)	RMSE of Trans RPE (m/f)
Original	0.019	0.003
2 mm	0.021	0.004
5 mm	0.055	0.009
8 mm	0.136	0.034
1 cm	37.398	7.697

Table 4: RMSE of ATE and translational RPE on synthetic sequences of different depth noise levels.

Figure 15: Multi-view mug reconstruction with varying visibility and noise levels . Ground truth views are arranged along the camera path, including frames with maximum, minimum, and intermediate mug visibility to highlight the visibility variation along the way. Yellow points represent the observed partial point clouds. NeuSE demonstrates strong robustness at lower noise levels (2 mm and 5 mm), accurately reconstructing the full mug shape in alignment with partial observations, regardless of visibility changes. At the 8 mm noise level, NeuSE’s sensitivity to occlusion increases, resulting in artifacts like a falsely shaped square handle and residual points around the mug body in low-visibility frames (first and last columns). At the 1 cm level, where NeuSE fails to recover the camera trajectory, reconstructions degrade further, producing entirely incorrect mug shapes, as the impact of noise dominates over that of occlusion.

Figure 16: Real-world data overview . (a) Object layout and the ground truth camera trajectory of the controlled environment. Object changes occur at each junction of the colored trajectory segments. (b) Camera setup and open lab (second row) and indoor office (third row) environments for the hand-held sequences.

Method	Type	4-Round: $1^{\text{st}}-2^{\text{nd}}$ round	4-Round: $2^{\text{nd}}-3^{\text{rd}}$ round	4-Round: $3^{\text{rd}}-4^{\text{th}}$ round	4-Round: Full	Triple-Infinity
CubeSLAM ( Yang and Scherer, 2019 )		0.208	0.156	0.164	0.206	0.334
EM-Fusion ( Strecke and Stueckler, 2019 )	All Objects Detected	0.302	0.315	0.219	0.304	0.281
VOOM ( Wang et al., 2024 )		0.461	0.395	0.236	0.538	0.160
Obj-only	Raw Odometry	0.628	0.904	1.022	1.248	0.591
	Mug-only	0.170	0.142	0.155	0.163	0.232
	All-object ( $\text{SE}(3)$ -only)	0.142	0.167	0.160	0.174	0.194

Table 5: RMSE (m) of the Absolute Trajectory Error on real-world sequences. Best results for each trajectory are marked in bold.

Figure 17: Baseline trajectory estimation results against ground truth (GT) on the 4-Round (left) and Triple-Infinity loops (right). (a) EM-Fusion; (b) CubeSLAM; (c) VOOM. All chosen baselines suffer from either localization drift in the 4-Round trajectory or failure to close the loop in the Triple-Infinity trajectory.

Figure 18: Visualization of estimated trajectories against ground truth (GT) on controlled sequences. Trajectory color variation indicates ATE distribution (color bar on the right). Top: Estimated trajectories of the 4-Round loop. The integration of our strategy (columns 3 and 5) helps smooth unstable trajectory estimates and improves general estimation accuracy, as showcased by the absence of small spikes (upper right) seen in column 2 and the consistently lighter colors of lower ATE values along the way. Bottom: Estimated trajectories of the Triple-Infinity loop. Our strategy (column 5) successfully corrects the accumulated start and end point drift for ORB3-PW (column 4), leading to an improved and consistent trajectory estimate when revisiting the leftmost table.

Figure 19: Trajectory estimation and object mapping artifacts in selected baselines. (a) EM-Fusion undergoes heavy out-of-plane drift and intermittent loss of tracking in the Triple-Infinity loop due to the rapid rotations around corners. (b) CubeSLAM fails to handle object changes effectively, resulting in data association errors, missed detections, drift, and overlapping cuboids, which affect the joint optimization of cuboid and camera trajectory estimates. (c) VOOM produces an object map with incorrect object count, scale, and pose estimates, including a green object falsely intersecting with the camera trajectory. Its use of dual quadrics as the object representation struggles to capture the full size of objects from partially occluded point clouds, particularly in cluttered scenes.

Figure 20: Trajectory estimation against ground truth (GT) of NeuSE’s $\text{SE}(3)$ -only variant on controlled sequences. The $\text{SE}(3)$ -only variant has difficulty handling objects with sizes outside the training data distribution, leading to less accurate camera pose constraints, such as in regions with the larger trash bins. This limitation is highlighted by the cyan blocks in (a) for the 4-Round loop and (b) for the Triple-Infinity loop.

Figure 21: Comparison of estimated trajectories against ground truth on hand-held sequences. Column 1: Top-down views of the full Indoor sequence. Column 2: Side views of the start and end segments of the Freeform sequence, where ground truth camera poses are available. (a) VOOM completes the Indoor loop but suffers from significant drift and loses tracking early in the Freeform loop, indicating its limited ability to handle more drastic variations in viewing angles. (b)–(c) NeuSE-derived camera pose constraints consistently enhance ORB-SLAM3’s localization performance across both sequences, despite the more complex hand-held motion and varied viewing angles. The stronger alignment between ORB3 + Ours and the ground truth, particularly in the Freeform loop in (c) , highlights our method’s effectiveness in facilitating relocalization after tracking loss, such as when the camera re-enters the office space following the corridor “blackout.”

	VOOM	ORB3	ORB3 + $\text{SE}(3)$ -Only	ORB3 + Ours
Indoor
Before	0.724	0.180	0.174	0.130
After	0.179	0.201	0.183	0.168
Full	0.696	0.192	0.181	0.151
Freeform (partial)
Before	Tracking Lost	0.032	0.041	0.031

Table 6: RMSE (m) of the Absolute Trajectory Error on hand-held sequences. Best results for each trajectory are marked in bold.

Figure 22: Trajectory estimation against ground truth (GT) for NeuSE’s $\text{SE}(3)$ -only variant on handheld sequences. The $\text{SE}(3)$ -only variant offers less noticeable improvements in camera pose estimation accuracy compared to the full $\text{SE}(3)$ - and scale-equivariant version. Specifically, unlike NeuSE, this variant fails to process observations of the white square jar, an instance intentionally included under the trash bin category with a similar shape but significantly smaller. The jar is used to test the approach’s ability to handle objects of varying sizes. This failure to correctly handle objects of out-of-distribution sizes results in persistent drift in areas where the jar is observed, as highlighted by the cyan blocks in (a) for the Indoor loop and (b) for the Freeform loop.

Figure 23: Consistent multi-view object reconstruction. We present our reconstruction (Ours) of the table with the highest object density across the four real-world testing sequences, including the trash bin visible in the first side view. Across all the five object categories present, the reconstruction maintains shape and spatial consistency, closely aligning with the ground truth (GT), as confirmed by the two side views.

Figure 24: Object reconstruction across scales . We present NeuSE’s reconstructions of the largest and smallest objects from real-world sequences across each of the five categories: mug, bottle, bowl, can, and trash bin. Reconstructions are visualized to preserve the approximate relative size ratios of the objects within each pair. (a) For object pairs without reference for relative scale ratio in ground truth images, the rough volumes from left to right are: beer mug (600 mL) and espresso cup (60 mL); trash bin (26.5 L) and small jar (0.3 L); yellow can (494 mL) and blue can (188 mL). From (a) through (b) , despite the considerable intra-category scale variation, NeuSE maintains consistent reconstruction performance across both categories and object sizes.

Figure 25: Complete object reconstructions of the synthetic multi-category sequences with planar and non-planar object layouts . (a) Planar layout. (b) Non-planar layout. Tables are rendered for visual clarity, with points back-projected to the world using camera pose estimates from NeuSE-predicted constraints, demonstrating the effectiveness of our localization strategy.

	TP	FP	FN	Pr	Re
Synthetic: Mug + Bottle
PMT	7	2	2	77.8%	77.8%
Ours	9	0	0	100.0%	100.0%
Synthetic: Multi
PMT	11	2	0	84.6%	100.0%
Ours	11	0	0	100.0%	100.0%

Table 7: Change detection results on the synthetic and real-world sequences. Best results are marked in bold.

Figure 26: Results of change-aware mapping for real-world controlled sequences. (a) (a.1) presents the reconstructed object-centric map with camera trajectory estimates, illustrating the spatial consistency between object reconstructions and actual camera motion. (a.2) and (a.3) display full object reconstructions and estimated trajectories for the 4-Round and Triple-Infinity sequences. (b) Comparison of non-tabletop object reconstructions, i.e., various trash bins on the floor, across ground truth scenes (GT), from Panoptic Multi-TSDF Mapping (PMT), and from our object-centric maps (Ours). (d)–(e) Reconstructed objects and layouts of all the five tables in the controlled environment, shown before (left) and after (right) changes, comparing GT, PMT, and Ours. Our approach provides a lightweight, object-centric map that better captures object changes and outperforms PMT in both reconstruction quality and change detection accuracy. (c) Failures in PMT’s change detection and reconstruction are highlighted in cyan blocks. Top: The orange bowl on table 3 is falsely labeled as changed by PMT during the Triple-Infinity loop due to the little overlap between the nearly opposite camera views when the bowl is visible. Bottom: PMT fails to correctly detect the objects that switch locations, resulting in overlapping reconstructions of the white and green bottles (left) and the black mug in the red bowl(right).

Figure 27: Overview of the reconstructed object-centric map and camera trajectory in the uncontrolled environment. The reconstruction and the camera trajectory estimates cover two regions, the indoor office area and the open lab space, connected by a blackout corridor, and demonstrate spatial consistency both within each region and across them, well aligned with the overall floor plan. (b)–(e)–(f) show the reconstructed object layout and camera trajectory estimates for the Indoor and Freeform loops, with a gray segment in the corridor to be the “blackout” part of the Freeform loop, where no objects of interest are available and ORB-SLAM3 tracking fails to recover the accurate trajectory. (a) shows the reconstructions for the unchanged objects while (d) presents the full estimated camera trajectory with the object layout in the open lab space. For the indoor office area, (c) outlines the Indoor trajectory estimate with the reconstructed object-centric map, and (g) exhibits the unchanged objects in the indoor office room.

Figure 28: Object reconstruction and change detection in the indoor office area. The map progression of the five tables is shown before and after the object changes (the left figure is the original state of the table, and the right figure is the new state of the table). Our approach maintains a clean and up-to-date object-centric map with all changes correctly detected. While for PMT, as its TSDF-based mapping method is limited by the parts of objects it has observed and the quality of camera trajectory estimates, the reconstructions can be incomplete and noisy, for example, the map for table 1 before changes. Moreover, as highlighted by the cyan blocks, PMT fails to remove the orange bowl on table 3 (right figure) after it is moved to the right side of the table. PMT also leaves a residual volume of the large blue bottle on table 5 at its previous location, and makes it overlap with the blue can after it is moved to the right side of the windowsill.

Figure 29: Object reconstruction and change detection in the open lab space. The map transition of the five tables is shown before and after the object changes (the left figure is the original state of the table, and the right figure is the new state of the table). Our method accurately captures object changes and recovers complete object shapes across categories. In contrast, as highlighted by the cyan blocks, PMT produces incomplete reconstructions, such as the missing blue can on table 3, and fails to remove the dark bottle from table 1 after it is moved.

Simultaneous Localization and Mapping

Simultaneous localization and mapping estimates a robot’s motion while building a map of an unknown environment.

See all 197 papers

Aug 10, 2026

Multi-Submap Implicit Neural SLAM with Local-to-Global Loop Closure for Large-Scale Scene Reconstruction

Neural Radiance Fields (NeRF)-based SLAM has demonstrated impressive results in small-scale scene reconstruction, yet scaling these methods to extensive, complex environments remains challenging due to catastrophic forgetting and accumulated trajectory drift. This paper presents a robust, large-scale neural SLAM system featuring a multi-submap architecture and a dual-tier loop closure mechanism. Specifically, we propose a progressive mapping strategy that dynamically allocates neural submaps to maintain high-fidelity representations without memory explosion. For robust pose estimation, an optical-flow-based tracking module is integrated to handle aggressive motions. To address global consistency, we introduce a local-to-global loop closure framework leveraging the foundation model for high-performance global descriptor extraction, significantly enhancing relocalization accuracy under varying viewpoints. Furthermore, an inter-submap online distillation algorithm is designed during back-end optimization to enforce geometric and appearance consistency across overlapping submap boundaries. To validate the system, we developed a customized handheld mechatronic platform and conducted extensive evaluations on both public benchmarks and our large-scale indoor-outdoor datasets. Experimental results, including direct deployment on an onboard computing unit, demonstrate that our approach outperforms state-of-the-art neural SLAM methods in reconstruction quality and localization robustness, providing a scalable solution for real-world robotic perception and digital twinning. We will release the code publicly on https://github.com/dtc111111/MSN-SLAM .

Tianchen Deng, Chongdi Wang, Nailin Wang +6

School of Automation and Intelligent Sensing, Shanghai Jiao Tong university and State Key Laboratory of Avionics Integration and Aviation System-of-Systems Synthesis, Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai 200240, China · Carnegie Mellon University, Pittsburgh, PA, United States · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore

Simultaneous Localization and Mapping 3D Reconstruction Loop Closure Detection Large-Scale 3D Reconstruction Neural Radiance Fields

Ask PDF

Nov 20, 2025

LEGO-SLAM: Language-Embedded Gaussian Optimization SLAM

Recent advances in 3D Gaussian Splatting (3DGS) have enabled Simultaneous Localization and Mapping (SLAM) systems to build photorealistic maps. However, these maps lack the open-vocabulary semantic understanding required for robotic interaction. Integrating language features into SLAM remains a significant challenge, as storing high-dimensional features incurs excessive memory and rendering overhead, while existing methods with static models lack adaptability for novel environments. We propose LEGO-SLAM (Language-Embedded Gaussian Optimization SLAM), a framework that achieves real-time, open-vocabulary mapping within a 3DGS-based SLAM system. At the core of our method is a scene-adaptive autoencoder that distills high-dimensional language embeddings into a compact 16-dimensional feature space, reducing the memory per Gaussian and accelerating rendering. Unlike static approaches, our encoder adapts online to unseen scenes. These compact features also enable a language-guided pruning strategy that identifies semantic redundancy, reducing the map's Gaussian count by up to 58% while maintaining rendering quality. Furthermore, we introduce a language-based loop detection approach that reuses the language features already extracted for mapping, eliminating the need for a separate detection model. Experiments demonstrate that LEGO-SLAM achieves competitive mapping quality and tracking accuracy, all while providing open-vocabulary capabilities at 15 FPS. Our project page is available at https://lab-of-ai-and-robotics.github.io/LEGO-SLAM/

Sibaek Lee, Seongbo Ha, Kyeongsu Kang +3

Department of Intelligent Robotics, Sungkyunkwan University, Suwon, South Korea

3D Gaussian Splatting 3DGS SLAM Visual SLAM Semantic Mapping Loop Closure Detection Open-Vocabulary Semantic Mapping

Ask PDF

Oct 15, 2026

LIN-SLAM: Lightweight LiDAR--Inertial Neural SLAM for Real-Time Onboard Navigation

Geometric light detection and ranging (LiDAR)--inertial simultaneous localization and mapping (SLAM) provides efficient state estimation but lacks the continuous map representation and global refinement capabilities of neural implicit SLAM, while neural implicit methods remain computationally expensive. We propose Lightweight LiDAR--Inertial Neural SLAM (LIN-SLAM), a hybrid method that couples high-rate geometric LiDAR--inertial estimation with neural implicit tracking, continual mapping, and global refinement for accurate real-time onboard SLAM. Experiments on twelve representative sequences from three datasets spanning 32-, 64-, and 128-channel LiDAR configurations show that LIN-SLAM achieves the lowest overall mean absolute trajectory error (ATE) among the evaluated state-of-the-art geometric and neural implicit methods, while substantially reducing runtime compared with existing neural implicit SLAM methods. On a Unitree Go2 equipped with an NVIDIA Jetson Orin NX and a 16-channel LiDAR, LIN-SLAM sustains real-time localization and neural mapping over a 1.47-km indoor trajectory through challenging repetitive corridor environments, with an endpoint closure error of only 0.141~m. To the best of our knowledge, LIN-SLAM is the first LiDAR--inertial SLAM system with a neural implicit backend to achieve real-time online operation on a mobile robotic platform while maintaining state-of-the-art localization accuracy. The code will be released upon acceptance.

Shiqi Che, Yijie Liu, Shuo Wang +1

School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China

Simultaneous Localization and Mapping Robot Localization LiDAR Odometry

Ask PDF

Abstract

Simultaneous Localization and Mapping

Explore similar work

Multi-Submap Implicit Neural SLAM with Local-to-Global Loop Closure for Large-Scale Scene Reconstruction

LEGO-SLAM: Language-Embedded Gaussian Optimization SLAM

LIN-SLAM: Lightweight LiDAR--Inertial Neural SLAM for Real-Time Onboard Navigation