Camera extrinsic calibration is a fundamental task in computer vision. However, precise relative pose estimation in constrained, highly distorted environments, such as in-cabin automotive monitoring (ICAM), remains challenging. We present InCaRPose, a Transformer-based architecture designed for robust relative pose prediction between image pairs, which can be used for camera extrinsic calibration. By leveraging frozen backbone features such as DINOv3 and a Transformer-based decoder, our model effectively captures the geometric relationship between a reference and a target view. Unlike traditional methods, our approach achieves absolute metric-scale translation within the physically plausible adjustment range of in-cabin camera mounts in a single inference step, which is critical for ICAM, where accurate real-world distances are required for safety-relevant perception. We specifically address the challenges of highly distorted fisheye cameras in automotive interiors by training exclusively on synthetic data. Our model is capable of generalization to real-world cabin environments without relying on the exact same camera intrinsics and additionally achieves competitive performance on the public 7-Scenes dataset. Despite having limited training data, InCaRPose maintains high precision in both rotation and translation, even with a ViT-Small backbone. This enables real-time performance for time-critical inference, such as driver monitoring in supervised autonomous driving. We release our real-world In-Cabin-Pose test dataset consisting of highly distorted vehicle-interior images and our code at https://github.com/felixstillger/InCaRPose.
Figures & tables
Figure 1: Our InCaRPose predicts the relative camera pose between a reference and a target view (shown as camera frustums). Trained exclusively on synthetic images, the model generalizes to real-world cabin environments and enables camera extrinsic calibration.
Figure 2: Camera coordinate system of the standard view compared to the vehicle’s coordinate system.
Figure 3: Standard view comparison: (left) real-world image and (right) synthetic image from the simulation environment.
Figure 4: InCaRPose’s architecture overview. Two images are encoded by a frozen ViT backbone and fused by a cross-attention Transformer decoder. A prediction head outputs the relative camera pose between the views, optionally in both directions.
Model
Error
Rot. ( ∘ )
Trans. (m)
Dir. ( ∘ )
InCaRPoseSmall224
Mean
6.11
0.11
42.55
Median
4.43
0.08
37.74
InCaRPoseBase224
Mean
4.91
0.12
53.67
Median
3.55
0.09
42.45
InCaRPoseLarge224
Mean
4.15
0.10
36.17
Median
2.75
0.07
23.46
Table 1: Model Performance Comparison: Rotation and Translation Errors (Trans. = translation error in m, Dir. = translation direction error in °) on our In-Cabin-Pose real-world test set.
Figure 5: Qualitative results on real-world inference. All translation vectors are normalized to a common scale for visualization.
Model
Error
Rot. ( ∘ )
Trans. (m)
Dir. ( ∘ )
Reloc3r224 [ 17 ] 2 2 2 Trained on additional data
Mean
7.96
–
39.43
Median
2.45
–
33.51
RelPoseNet [ 35 ] 1 1 1 Values are taken from the authors’ paper
Mean
–
–
–
Median
9.30
0.21
–
Relformer [ 28 ] 1 1 1 Values are taken from the authors’ paper
Mean
–
–
–
Median
6.27
0.18
–
Table 2: Model performance: rotation and translation errors on 7-Scenes. Not available values are indicated with “-”. Bold indicates best performance.
Scope
Scene
Model
Rot. Err ( ∘ )
Trans. Err (m)
Mean
Median
Mean
Median
Overall
All
InCaRPoseSmall224
6.99
3.44
2.31
1.45
InCaRPoseBase224
6.53
3.01
2.29
1.35
InCaRPoseLarge224
6.49
2.66
2.19
1.21
Reloc3r224 [ 17 ] 2 2 footnotemark: 2
9.95
1.13
–
–
Scene
KingsCollege
InCaRPoseSmall224
2.74
2.67
1.20
1.05
Table 3: Validation results on Cambridge Landmarks.
Representation
Error
Rot. ( ∘ )
Trans. (m)
Dir. ( ∘ )
Rotation Vector
Mean
6.35
0.14
81.24
Median
4.80
0.11
70.62
Euler Intrinsic
Mean
7.61
0.14
99.14
Median
5.77
0.10
106.76
Euler Extrinsic
Mean
6.39
0.14
108.73
Median
4.67
0.11
110.92
Table 4: Output rotation representations ablation on In-Cabin-Pose with InCaRPoseBase224.
Backbone
Error
Rot. ( ∘ )
Trans. (m)
Dir. ( ∘ )
Dune-Base
Mean
4.52
0.13
55.35
Median
3.00
0.09
53.40
DUST3R-Large
Mean
5.23
0.14
81.18
Median
3.99
0.10
77.63
DINOv2-Base
Mean
5.98
0.12
45.78
Median
3.95
0.09
40.43
Table 5: Backbone ablation on In-Cabin-Pose with InCaRPoseBase224.
Backbone
Res.
Config
Latency (ms)
FPS
Speedup
Small
224
Baseline (FP32)
14.65
68.28
1.00
FP16
14.57
68.64
1.01
torch.compile
14.00
71.41
1.05
FP16 + torch.compile
14.48
69.06
1.01
Small
512
Baseline (FP32)
24.75
40.40
1.00
FP16
16.49
60.65
1.50
Table 6: Detailed inference runtime measurements on a single NVIDIA RTX 4090 GPU. We report average per-frame latency (ms), frames per second (FPS), and relative speedup with respect to the FP32 baseline at the corresponding backbone and resolution.
Backbone
Metric
Rot. Err ( ∘ )
Dir. Err ( ∘ )
Dune-Base
Mean
6.26
52.11
Median
3.88
50.90
DUST3R-Large
Mean
6.98
81.52
Median
5.29
73.93
DINOv2-Base
Mean
8.11
45.60
Median
5.30
39.58
Table 7: Backbone ablation on COLMAP ground truth (distorted images and 224 resolution).
Figure 6: Comparison of preprocessing methods. (a) and (b): images are cropped to the center. (c) and (d): images are zero-padded to a square aspect ratio. This adjustment is necessary to handle varying input resolutions within the dataset while maintaining a consistent model input.
Metric
Value
Max rotation error
19.86 ∘
Mean rotation error
3.08 ∘
Median rotation error
2.27 ∘
Max translation direction error
55.83 ∘
Mean translation direction error
5.75 ∘
Median translation direction error
4.60 ∘
Table 8: Rotation and translation direction errors. Translation direction error is only evaluated when the relative ArUco ground truth translation exceeds 0.1 m.
Figure 7: COLMAP fails to estimate translation along the y -axis. In the first view the camera is to the side of the steering wheel. In the second view the camera moved upwards (y) above the steering wheel. Translation is normalized.
Figure 8: Rotation error in degrees versus the number of parameters. Evaluated on image resolution of 224 on the In-Cabin-Pose dataset.
Figure 9: ArUco (orange) vs. COLMAP (blue) camera trajectories for intervals focused on specific transformations. Each sequence predominantly captures motion along the target axis, though residual degrees-of-freedom are also present.
Figure 10: Inference on frames with physically occluded ArUco markers. We also changed the reference image, in which a different object (a high-contrast checkerboard on the left seat) is now visible in only one image to additionally challenge the model.
Accurate camera intrinsic calibration is fundamental to robot perception, and the accuracy depends on the quality of the collected images. However, existing target-based calibration methods often require the practitioner to manually filter out high-quality images and to specify an appropriate radial distortion order. This paper presents a fully automatic intrinsic calibration pipeline that determines both from the collected data. We adopt an iterative rejection scheme that estimates parameters on a candidate image set and removes views whose mean residual exceeds a multiple of the median. Crucially, this process runs independently under each candidate distortion order, so that the retained image set is consistent with the residual scale of that order. Further, the distortion order is selected on held-out images, with the intrinsics and distortion fixed and only the board pose re-estimated, ensuring that an added coefficient is supported by independent observations. Finally, we integrate both steps into an interactive calibration tool that supports full-pipeline data inspection and parameter estimation. Experiments on our own camera data and five public real-world datasets show that image filtering reduces the held-out reprojection error by 25%, the order selection further by 5%, achieving the lowest held-out mean among four compared configurations without manual image selection. We will release the code and data to facilitate future research.
Xiangcheng Hu
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China
We present DRS-VPT, a feed-forward transformer architecture for foundational image-to-scan registration. Given query images and a reference 3D point cloud, the model predicts the scan pose and point map alongside the poses and point maps of each camera, all expressed in the first camera's frame. It additionally predicts a coarse-to- fine pyramid of per-point and per-pixel features for direct reprojective alignment of the scan to the first image. This formulation unifies downstream tasks such as camera-LiDAR calibration in autonomous driving and indoor camera-to-map relocalization. A single DRS-VPT model achieves state-of-the-art performance for image-to-LiDAR registration in autonomous driving, competitive indoor relocalization without training map-specific weights, and strong zero-shot transfer to unseen environments. We also show qualitatively that the model learns complex scan-to-image projection properties such as occlusion of back-facing points.
Lanke Frank Tarimo Fu, Maurice Fallon
University of Oxford, Oxford OX2 6NN, United Kingdom
Accurate camera-LiDAR calibration is an essential prerequisite for robust multi-modal perception in robotics. Recent target-less methods based on learned pixel-point correspondences achieve high extrinsic calibration accuracy and generalize across environments. However, they operate on rectified images with known camera intrinsics, restricting their applicability to systems whose cameras have already been individually calibrated. We present a fully automatic, target-less framework that jointly estimates camera intrinsics, radial-tangential distortion, and camera-LiDAR extrinsics from raw images and LiDAR scans. Our method generalizes deep correspondence-based calibration from rectified images to raw cameras through intrinsic- and distortion-aware correspondence learning via synthetic augmentations. During calibration, we iteratively couple correspondence prediction and nonlinear optimization to jointly refine the camera parameters and relative sensor transformation. We evaluate our approach on an unseen camera-LiDAR configuration from the KITTI dataset and on three real robotic platforms in a zero-shot manner. The results demonstrate that joint calibration with our generalized pixel-point correspondence network improves extrinsic accuracy over correspondence-based methods using pre-calibrated cameras, while additionally recovering accurate intrinsics.
Simon Bultmann, Daniele Cattaneo, Abhinav Valada
Department of Computer Science, University of Freiburg, Germany