cs.CVAug 10, 2026

End-to-End Visual Odometry with RNNs and Attention

Authors: Ruiyu Li, Yinjia Liu, Alexander Yu

Organizations: Carnegie Mellon University, Pittsburgh, PA, United States.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Video Odometry (VO) is the process of estimating the ego-motion of an object by analyzing visual information such as a sequence of frames from one or multiple cameras. It has been a popular research topic in computer vision and robotics, and its applications include mobile robotic systems as well as autonomous driving. In this project, we investigate existing end-to-end deep-learning approaches to VO, and propose a novel temporal attention-based model to improve upon the baseline. In addition, while the vast majority of existing deep-learning-based approaches to VO are trained on driving data, we investigate the performance of deep-learning-based VO to the more dynamic and complex problem of hand-held cameras.

Figures & tables

Figure 1: An example trajectory in the TUM RGB-D dataset generated from a 3D reconstruction by Mur-Atal in Mur-Artal and Tardos ( 2017 )
Figure 2: Model architectures of the baseline models. (a) The architecture used by the original DeepVO model, Baseline Model 1, and Baseline Model 2. The number of input channels, and trainability of convolutional layers varies between models. (b) The architecture of Baseline Model 3. The convolutional head responsible for RGB images is frozen during training, while the head responsible for depth images is trainable.
Figure 3: Architectures of the attention-based VO model. (a) The combined LSTM and attention model. (b) The transformer encoder based model. Note the reduced number of filters in the transformer based-model.
Figure 4: Training and validation losses of the original DeepVO model and the three additional baseline models on the TUM RGB-D dataset. Note the significant difference in Y-axis range between training and validation losses.
Figure 5: Representative time-series predictions generated by the best performing baseline model: Baseline Model 1. Note the approximately correct velocities predicted by the model in validation, especially from times 15s to 20s.
Figure 6: Training and validation losses of the attention based models compared against the best-performing baseline model.
Figure 7: Representative time-series predictions generated by the transformer-based model.
More in this topic

Visual Odometry

Visual odometry estimates camera or robot motion from successive images, enabling navigation and mapping when external positioning is unavailable.

See all 56 papers

Explore similar work

CardsList

  1. OpenVO: Open-World Visual Odometry with Temporal Dynamics Awareness

    Feb 22, 2026Phuc D. A. Nguyen, Anh N. Nhu, Ming C. LinVisual Odometry

    University of Maryland, College Park, USA

  2. DAPEVO: Deep Adaptive Patch Frame-Event Visual Odometry

    Sep 25, 2026Luca Gandolfi, Simone Nascivera, Roberto Pellerito +3Visual OdometryRGB-Event Sensor Fusion

    Robotics and Perception Group, University of Zurich, Switzerland · Bocconi University, Italy

  3. FFVO: A Feedforward Pose Decoder for Long-Horizon Visual Odometry

    Sep 12, 2026Meng-Li Shih, Shih-Yang Su, Yuliang Zou +7Camera Pose EstimationVisual Odometry

    Waymo LLC · University of Washington · Google DeepMind