MultiCam: On-the-fly Multi-Camera Pose Estimation Using Spatiotemporal Overlaps of Known Objects

Shiyu Li; Hannah Schieber; Kristoffer Waldow; Benjamin Busam; Julian Kreimeier; Daniel Roth

cs.CVMar 24, 2026

MultiCam: On-the-fly Multi-Camera Pose Estimation Using Spatiotemporal Overlaps of Known Objects

Authors: Shiyu Li, Hannah Schieber, Kristoffer Waldow, Benjamin Busam, Julian Kreimeier, Daniel Roth

Organizations: Technische Universtität München, Human-Centered-Computing and Extended Reality Lab, Klinikum rechts der Isar, Orthopedics and Sports Orthopedics, Munich Institute of Robotics and Machine Intelligence (MIRMI), Germany. · TH Köln, Computer Graphics Group · Technische Universität München, Human-Centered-Computing and Extended Reality Lab, Germany. · Technische Universtität München, Photogrammetry and Remote Sensing Lab, Munich, Germany

Abstract

Multi-camera dynamic Augmented Reality (AR) applications require a camera pose estimation to leverage individual information from each camera in one common system. This can be achieved by combining contextual information, such as markers or objects, across multiple views. While commonly cameras are calibrated in an initial step or updated through the constant use of markers, another option is to leverage information already present in the scene, like known objects. Another downside of marker-based tracking is that markers have to be tracked inside the field-of-view (FoV) of the cameras. To overcome these limitations, we propose a constant dynamic camera pose estimation leveraging spatiotemporal FoV overlaps of known objects on the fly. To achieve that, we enhance the state-of-the-art object pose estimator to update our spatiotemporal scene graph, enabling a relation even among non-overlapping FoV cameras. To evaluate our approach, we introduce a multi-camera, multi-object pose estimation dataset with temporal FoV overlap, including static and dynamic cameras. Furthermore, in FoV overlapping scenarios, we outperform the state-of-the-art on the widely used YCB-V and T-LESS dataset in camera pose accuracy. Our performance on both previous and our proposed datasets validates the effectiveness of our marker-less approach for AR applications. The code and dataset are available on https://github.com/roth-hex-lab/IEEE-VR-2026-MultiCam.

Figures & tables

Panel of Table 1 — Table 1: Dataset Overview. Comparison of existing instance-level 6D pose datasets with our proposed dataset in terms of camera setup, field-of-view (FOV) overlap, and object complexity.

Figure 1: Architecture of our proposed graph-based pose estimation. We generate object candidates from a set of captured RGB images using our 6D pose estimation module. Based on object candidates from different camera views, we select the matching candidates and calculate camera poses. Finally, we implement object and camera optimization using our novel spatiotemporal scene graph. We build our spatiotemporal scene graph using object information from different spaces and times. The icons represent camera nodes, i.e., hmd , camera $C_{1}$ , and camera $C_{n}$ . The circles represent objects. If there are overlaps between two cameras, we calculate camera poses based on the hmd SLAM coordinate system.

Figure 2: Objects included in our Femoral Nailing dataset.

Approach	One-stage			Two-stage
Method	PoseCNN	SegDriven	YOLOPose	GDR-Net
	[ 60 ]	[ 17 ]	( Ours )	[ 55 ]
P.E.	1	1	1	1	$N$
002_master_chef_can	3.6	33.0	70.2	51.7	41.5
003_cracker_box	25.1	44.6	85.4	45.1	83.2
004_sugar_box	40.3	75.6	73.8	83.9	91.5

Table 2: Single-view object pose evaluation on YCB-V dataset [ 60 ] with ADD(-S)-0.1d. P.E. means whether the method is trained with 1 pose estimator for the whole dataset or 1 per object (

N

objects in total). (

*

) denotes symmetric objects and (

-

) denotes unavailable results.

Panel of Figure 3 — Figure 3: Multi-view visualization of object poses in YCB-V dataset: Our proposed approach, MultiCam, (bottom) is able to detect and optimize object poses in a more accurate way compared to the state-of-the-art CosyPose (center) [ 24 ] in multi-view setups. The ground truth is displayed on top. Differences are notable on the clamp right next to the can, and on the other clamp, a drift in different views becomes apparent.

Panel of Figure 4 — Figure 4: Multi-view visualization from our Femoral Nailing dataset in near distance (top two rows) and far distance (bottom two rows): We compared YOLOPose + CosyPose [ 24 ] (second left), CNOS [ 32 ] + Megapose [ 25 ] + CosyPose [ 24 ] (second right), and our approach (MultiCam) (right) with the ground truth (left). Our approach has smaller visualization errors than the other approaches. MultiCam also integrates more objects using spatiotemporal information, leveraging shared and non-shared fov information. Calibration markers presented in the scene are used to compare with our proposed marker-less approach.

Panel of Table 3 — Table 3: Quantitative multi-view 6D object pose results on YCB-V dataset [ 60 ] . The best results per category (known vs. unknown camera poses) are printed in bold .

Panel of Table 4 — Table 4: Evaluation of camera pose estimation error before and after object-level bundle adjustment on the YCB-V dataset [ 60 ] : We report pose rotation error (in mm) and translational error(in degrees) for the camera poses before and after bundle adjustment. Our proposed bundle adjustment helps improve both object and camera accuracy.

Panel of Table 5 — Table 5: Effect of the 3D loss in YCB-V dataset [ 60 ] . The proposed 3D loss outperforms the 2D key point loss. The larger the distance between the object and the camera is, the more obvious the advantage of the 3D loss.

Methods	4 views		8 views
	$e_{trans}\downarrow$	$e_{rot}\downarrow$	$e_{trans}\downarrow$	$e_{rot}\downarrow$
CosyPose [ 24 ]	44.38	4.19	65.93	8.11
ARToolKitPlus marker [ 52 ]	64.27	14.01	61.97	13.48
MultiCam (ours)	38.22	3.25	38.96	3.33

Table 6: Evaluation of camera pose estimation error on the T-LESS dataset [ 15 ] . We report pose rotation error (in mm) and translational error(in degrees) for the camera poses with state-of-the-art calibration and pose estimation methods.

Methods	Object-level									Calibration board
Conditions	YOLOpose + CosyPose [ 24 ]			CNOS [ 32 ] + Megapose [ 25 ] + CosyPose [ 24 ]			MultiCam (ours)			Charuco marker
Conditions	$e_{trans}\downarrow$	$e_{rot}\downarrow$	runtime	$e_{trans}\downarrow$	$e_{rot}\downarrow$	runtime	$e_{trans}\downarrow$	$e_{rot}\downarrow$	runtime	$e_{trans}\downarrow$	$e_{rot}\downarrow$	runtime
Near	64.54	9.81	135.05	47.17	6.84	84831.93	45.54	6.48	43.37	34.98	3.40	21.72
Far	94.69	8.90	194.70	70.47	6.91	79163.96	52.79	5.53	47.65	81.68	6.76	18.56
Mean	79.62	9.36	164.9	58.8	6.88	81997.95	49.17	6.01	45.51	58.33	5.08	20.14

Table 7: Camera pose estimation/Calibration results on our Femoral Nailing dataset: The translation error is in millimeters (mm) (

e_{ave\_{trans}}

denoted as

e_{trans}

\downarrow

), the rotation error (

e_{ave\_rot}

denoted as

e_{rot}

\downarrow

) is in degrees and the average runtime is in ms. The best results among all methods are labeled in bold.

Methods	YOLOpose + CosyPose [ 24 ]			MultiCam (ours)
Methods	$ADD(S)AUC\uparrow$	$e_{trans}\downarrow$	$e_{rot}\downarrow$	$ADD(S)AUC\uparrow$	$e_{trans}\downarrow$	$e_{rot}\downarrow$
Aiming arm	0.00	158.62	20.53	30.90	85.57	19.00
Bar $\star$	0.00	244.59	-	37.51	109.23	-
Gear $\star$	33.33	156.18	-	47.65	127.31	-
Insertion handle	0.00	215.67	13.01	56.29	46.62	9.13
Nail	0.00	261.23	140.67	46.59	75.46	83.61

Table 8: Multiview object pose evaluation on Femoral Nailing ADD(-S)-AUC, translation error in mm and rotation error in degrees with near and far conditions. Symmetric objects are denoted by

\star

. Rotational errors are only evaluated for unsymmetrical objects.

Figure 5: Drift error correction with MultiCam: The figure shows the increasing sensor drift (orange) which can be reduced when on-the-fly camera pose estimation is performed (blue).

Augmented Reality

See all 18 papers

Dense Dynamic Scene Reconstruction and Camera Pose Estimation from Multi-View Videos
Mar 12, 2026Shuo Sun, Unal Artan, Malcolm Mielle +2Multi-View Depth Estimation Multi-View 3D Reconstruction
Örebro University, Sweden · Schindler EPFL Lab, Lausanne, Switzerland · Technical University of Munich, Germany
Efficient Online 3D Multi-Camera Multi-Object Tracking and Pose Estimation
Apr 16, 2026Linh Van Ma, Tran Thien Dat Nguyen, Juhua Hu +2Visual Object Tracking Object Pose Estimation
School of Electrical Engineering and Computer Science at GIST, Gwangju, Korea · School of Electrical Engineering, Computing and Mathematical Sciences, Curtin University, Australia · School of Engineering and Technology, University of Washington, USA
QuacamFM: Quaternion-Constrained Flow Matching for Camera Pose Estimation
Sep 26, 2026Bao-Long Tran, Cuong Le, Tahereh Dehdarirad +2Camera Pose Estimation Flow Matching
Linköping University, Sweden

Mar 12, 2026J/K move · Enter open · S save

Dense Dynamic Scene Reconstruction and Camera Pose Estimation from Multi-View Videos

Shuo Sun, Unal Artan, Malcolm Mielle, Achim J. Lilienthal+1

Örebro University, Sweden · Schindler EPFL Lab, Lausanne, Switzerland · Technical University of Munich, Germany

Multi-View Depth Estimation Multi-View 3D Reconstruction Camera Pose Estimation Dynamic Scene Reconstruction Visual SLAM

Open paper AskPDF arXiv

We address the challenging problem of dense dynamic scene reconstruction and camera pose estimation from multiple freely moving cameras -- a setting that arises naturally when multiple observers capture a shared event. Prior approaches either handle only single-camera input or require rigidly mounted, pre-calibrated camera rigs, limiting their practical applicability. We propose a two-stage optimization framework that decouples the task into robust camera tracking and dense depth refinement. In the first stage, we extend single-camera visual SLAM to the multi-camera setting by constructing a spatiotemporal connection graph that exploits both intra-camera temporal continuity and inter-camera spatial overlap, enabling consistent scale and robust tracking. To ensure robustness under limited overlap, we introduce a wide-baseline initialization strategy using feed-forward reconstruction models. In the second stage, we refine depth and camera poses by optimizing dense inter- and intra-camera consistency using wide-baseline optical flow. Additionally, we introduce MultiCamRobolab, a new real-world dataset with ground-truth poses from a motion capture system. Finally, we demonstrate that our method significantly outperforms state-of-the-art feed-forward models on both synthetic and real-world benchmarks, while requiring less memory.

Abstract

Augmented Reality

Explore similar work

Dense Dynamic Scene Reconstruction and Camera Pose Estimation from Multi-View Videos

Efficient Online 3D Multi-Camera Multi-Object Tracking and Pose Estimation

QuacamFM: Quaternion-Constrained Flow Matching for Camera Pose Estimation