Kineo: Calibration-Free Metric Motion Capture From Sparse RGB Cameras

Charles Javerliat; Pierre Raimbaud; Guillaume Lavoué

cs.CVOct 28, 2025

Kineo: Calibration-Free Metric Motion Capture From Sparse RGB Cameras

Authors: Charles Javerliat, Pierre Raimbaud, Guillaume Lavoué

Organizations: École Centrale de Lyon, CNRS, INSA Lyon, Université Claude Bernard Lyon 1, Université Lumière Lyon 2, LIRIS, UMR5205, ENISE, Saint-Etienne, France

Abstract

Markerless multi-view motion capture is often constrained by the need for precise camera calibration, limiting accessibility for non-experts and in-the-wild captures. Existing calibration-free approaches mitigate this requirement but suffer from high computational cost and reduced reconstruction accuracy. We present Kineo, a fully automatic, calibration-free pipeline for markerless motion capture from videos captured by unsynchronized, uncalibrated, consumer-grade RGB cameras. Kineo leverages 2D keypoints from off-the-shelf detectors to simultaneously calibrate cameras, including Brown-Conrady distortion coefficients, and reconstruct 3D keypoints and dense scene pointmaps at metric scale. A confidence-aware spatio-temporal keypoint subsampling strategy, combined with graph-based global optimization, enables robust calibration at a fixed computational cost independent of sequence length. We further introduce a pairwise reprojection consistency score, a heuristic measure of the agreement between views, which we use to weight bone lengths in metric scale recovery. Evaluations on EgoHumans and Human3.6M demonstrate substantial improvements over prior calibration-free methods. Compared to previous state-of-the-art approaches, Kineo reduces camera translation error by approximately 83-85%, camera angular error by 86-92%, and world mean-per-joint position error (W-MPJPE) by 83-91%. Kineo is also efficient in real-world scenarios, processing multi-view sequences faster than their duration in a specific configuration (e.g., 36 min to process 1 h 23 min of footage). The full pipeline and evaluation code are openly released to promote reproducibility and practical adoption at: https://liris-xr.github.io/kineo/

Figures & tables

Figure 1. Overview of the Kineo pipeline for markerless motion capture from uncalibrated and unsynchronized multi-camera videos. Starting from raw video inputs, the system first performs audio-based temporal synchronization to align unsynchronized streams on a shared timeline. Next, automatic camera calibration estimates extrinsic and intrinsic parameters, including Brown-Conrady lens distortion, via a graph-based optimization over 2D keypoint correspondences selected by a confidence-aware subsampling strategy. Using the recovered cameras, 3D keypoints and scene pointmaps are reconstructed, and each triangulated keypoint is assigned a pairwise reprojection consistency score to measure the agreement between views. Finally, metric-scale recovery is achieved either through a human body prior using the SMPL model or a monocular metric depth estimator for subject-agnostic scaling. The modular design enables robust and scalable reconstruction of human subjects across long, multi-view sequences, and can extend to other subjects given a suitable 2D keypoint detector.

Figure 2. Example subsampling for a pair of views from Human3.6M. Colors indicate the pair confidence score $w_{ij,k}$ , solid lines represent selected correspondences, and dashed lines represent correspondences that were not selected. Only correspondences whose pair score exceeds $\tau$ can be selected, which discards the keypoints the 2D keypoint detector does not locate confidently (e.g., occluded or poorly localized keypoints).

Figure 3. Example of camera loop closure with incorrect (left) and correct (right) relative scale factors. When the relative scales

\lambda_{ij}

are incorrect, traversing the cycle

C_{i}\!\to\!C_{j}\!\to\!C_{k}\!\to\!C_{i}^{\prime}

does not return to the original camera position, resulting in

C_{i}\neq C_{i}^{\prime}

. In contrast, when the correct scales are used, the composition of the relative transformations yields the identity, and the loop closes consistently with

C_{i}=C_{i}^{\prime}

Panel of Figure 4 — Figure 4. Comparison of scene pointmaps produced by two geometry estimation models. (a) shows the pointmap generated with MoGe-2, which uses single-view images and camera intrinsics to produce metric pointmaps. (b) shows the pointmap generated with VGGT, which leverages multiple views to predict a globally consistent pointmap and camera poses, aligned to our camera system. VGGT provides more spatially consistent pointmaps across views, while MoGe-2 relies on single-view depth estimation.

Panel of Figure 5 — Figure 5. Examples of different output formats with different 2D keypoint detectors used by Kineo. (a–c) show keypoint-based representations in various formats, while (d) illustrates a full parametric SMPL body reconstruction. For the SMPL model, NLF surface keypoints are first detected and subsequently fitted using the SMPLFitter module from NLF.

Figure 6. Visualization of the proposed pairwise reprojection consistency score on 3D human skeleton reconstructions from the EgoHumans dataset, obtained using Kineo. The low score on the hands shows that it flags the joints on which the views disagree. Beyond the bone weighting of Sec. 7.1 , it could be leveraged for downstream filtering, which we leave for future work.

Figure 7. Overview of SMPL-based global scaling. The SMPL body model serves as a prior for metric scale, providing plausible human body proportions that guide the estimation of the global scale factor $\alpha$ , which converts the scene from arbitrary units to metric units. Given an estimated skeleton in arbitrary units, we optimize $\alpha$ while updating the SMPL shape parameters $\beta$ so that the SMPL skeleton closely matches the bone lengths of the scaled input skeleton, enforced by $\mathcal{L}_{\text{bones}}$ while remaining close to the mean shape, enforced by $\mathcal{L}_{\beta}$ . The resulting optimized scale $\alpha^{*}$ is then applied to rescale the entire scene, including the skeletons, cameras, and scene pointmaps, producing a metric-scale representation.

Figure 8. Overview of metric depth-based global scaling. Unlike SMPL-based scaling, which relies on a parametric human body model, metric depth-based scaling is subject-agnostic, as the metric depth map predicted directly from the RGB image by MoGe-2 ( Wang et al., 2025b ) is independent of the subjects present in the scene. The global scale factor $\alpha$ is computed as the ratio between the joint depths in metric units obtained from the depth map and the corresponding joint depths in arbitrary units. The resulting scale $\alpha$ is then applied to rescale the entire scene, including the skeletons, cameras, and scene pointmaps, producing a metric-scale representation.

Figure 9. Automatic calibration runtimes for different keypoint detectors across sequences of varying durations. The comparison includes three keypoint sources: NLF (batch size 1, full precision), RTMPose (batch size 1, full precision), and RTMPose (batch size 32, half precision). Sequence sizes are small (S, 7 min 10 s total across 4 views), medium (M, 43 min 03 s total across 4 views), and long (L, 1 h 23 min total across 4 views). Each stacked bar breaks down the runtime by processing stage: Intrinsics Initialization (yellow), Extrinsics Initialization (orange), Bundle Adjustment (pink), Global Scaling (purple), Keypoints Pairs Sampling (blue), and Keypoints Detection (gray). Total runtime is indicated above each bar. RTMPose (B32 Half) consistently achieves the fastest processing times, completing all stages in less time than the total duration of the original video sequences. Importantly, the calibration runtime scales efficiently, showing only moderate increases as sequence duration grows.

Figure 10. Estimated distortion on three EgoHumans ( Khirodkar et al., 2023 ) views. (a) Input frame with a grid distorted by the ground-truth camera. (b) Frame and grid undistorted with the estimated camera (red), overlaid on the ground-truth grid (black). (c) Error of the estimated distortion after focal length rescaling (Sec. 8.2.3 ), with arrows showing its direction and pink contours outlining the observed region.

Figure 11. Mean PA-MPJPE (in millimeters) against the number of views on the EgoHumans ( Khirodkar et al., 2023 ) sequences, with estimated and ground-truth intrinsics. Bands give the 95% confidence interval of the mean over sequences. Camera sets are nested from two to eight views; All uses every static camera of the sequence (8 to 19).

Panel of Figure 12 — Figure 12. Qualitative comparison between HSfM ( red ) and Kineo ( blue ) on their respective best-performing sequences from Human3.6M. Ground-truth cameras and skeletons are shown in black . Visualization is scale-aligned.

Panel of Figure 13 — Figure 13. Qualitative results on sequences from the EgoHumans dataset. Kineo estimations are visualized in blue , while ground-truth cameras and skeletons are shown in black . The scene pointmap is predicted as a downstream task of our method using MoGe-2 ( Wang et al., 2025b ) .

Panel of Figure 14 — Figure 14. Examples of craftsmen recorded at an experimental archaeology worksite. Both sequences were captured using six uncalibrated and unsynchronized GoPro Hero11 Black cameras, with no calibration procedure necessary.

Symbol	Description
Temporal synchronization (Sec. 4 )
$\gamma_{k}$	Temporal lag of camera $k$ , relative to a reference camera
$\bar{\gamma}_{k}$	Ground-truth temporal lag of camera $k$ (AIST++)
$H$	Hop length used during the Short-Time Fourier Transform
$f_{s}$ , $f_{r}$	Audio sample rate and video frame rate
$d_{1}$ , $d_{2}$	Distance from each camera to the sound source

Table S1. Notation used in the main paper.

Parameter	Value
Temporal synchronization (Sec. 4 )
Reference camera	First camera; the other audio tracks are resampled to its sample rate
Audio features	13 MFCCs over 128 HTK mel bands (log-mel in dB), 2048-point FFT, 40 ms Hann window, 5 ms hop ( $H=240$ at 48 kHz)
Feature normalization	Each coefficient is standardized to zero mean and unit variance
Lag estimation	Maximum of the full cross-correlation, computed by FFT, without interpolation
2D keypoints

Table S2. Implementation details and hyperparameters of each stage of the pipeline, shared by all experiments unless stated otherwise.

Experiment	2D keypoints	$\tau$	$P$	$\kappa$	$M$
Tables 3 , 4 , 1 , 6 and 7 , Sec. 8.2.3 , Figs. 9 and 11 ; Tables S6 , S8 and S9 , Fig. S1	NLF	0.6	3000	0.6	500
Tables 11 , 6 , 8 and 9 , Fig. 9 ; Table S7	RTMPose	0.5	4000	0.5	400
Table 6 ; Table S8	DWPose	0.5	4000	0.5	400
Tables 5 and 6 , Human3.6M	Ground truth	0.4	4000	0.4	200
Tables 5 and 6 , EgoHumans	Ground truth	0.4	4000	0.5	200

Table S3 . Confidence thresholds and sampling budgets used in the experiments of the main paper and of these supplementary materials: threshold

\tau

and budget

P

of the pairwise calibration (Sec. 5.2 ), threshold

\kappa

and budget

M

of bundle adjustment (Sec. 5.6 ). Experiments that take the cameras from the ground truth do not run calibration and are omitted. Ground-truth keypoints all have a confidence of 1, so that their thresholds only discard missing or non-interpolated keypoints.

Panel of Figure 18 — Figure S1 . Estimated distortion on EgoHumans, one view per scene, with the same panels as Fig. 10 of the main paper. The error in (c) is shown on a linear color scale saturated at 50 px, with its direction as unit arrows (dots below 1 px); hatched samples are excluded (see Protocol). Each row shows a representative sequence of its scene from the camera whose keypoints cover the largest share of the image, among the cameras of the rig’s majority focal length.

Figure S2 . Correspondences selected by (a) farthest-point sampling, (b) confidence-only sampling (Top-P) and (c) our random spatio-temporal subsampling ( $P=3000$ ) from the same confidence-filtered candidates, for the camera pair of an EgoHumans tennis sequence where the coverage of random sampling falls furthest below FPS. Whereas the convex hull of the farthest-point selection (solid) nearly matches that of the candidates (dashed), random sampling leaves the edges of the candidate region uncovered (shaded), and confidence-only sampling covers only its center.

Panel of Figure 20 — Figure S3 . Thumbnails of all supplementary video files provided alongside this document. Each thumbnail represents a single frame from the corresponding video; full videos are provided separately.

Panel of Figure 21 — Figure S4 . Illustration of the procedure to visualize the interactive visualization file for the S11_Greeting sequence from Human3.6M. ( a ) Opening the .rrd file using the online rerun viewer available at https://rerun.io/viewer . ( b ) Interactive 3D visualization, showing the scene hierarchy with ground-truth and Kineo estimates.

Markerless Motion Capture

Markerless motion capture estimates 3D body movement from video or other sensors without attached tracking markers.

See all 70 papers

Sep 17, 2026

DirtyMoCap: Robust Motion Capture from Unconstrained Markers

Optical motion capture delivers high-fidelity human motion, but its reliance on strict marker layouts and clean trajectories severely limits its real-world applicability. In practice, tracking systems frequently output unconstrained markers: sparse, noisy, and unordered point clouds with unknown or varying configurations. To bridge the gap between corrupted raw markers and parametric human models, we introduce DirtyMoCap, a robust, marker-layout-free framework. Our core insight is to map unordered marker observations to a fixed set of "proxy anchors" comprising skeletal joints and body surface points, which serve as a stable intermediate representation. We first initialize and track these anchors over long sequences using a recurrent sliding-window architecture. Then, a custom differentiable Gauss-Newton solver fits the SMPL-H model to the tracked anchors to recover full-body pose, translation, and shape. By explicitly deriving geometric residuals, our solver learns adaptive observation confidence, smoothness, and prior weights end-to-end, adapting dynamically to the reliability of the input data. Extensive experiments on diverse, noisy marker configurations demonstrate that DirtyMoCap successfully generalizes across arbitrary layouts using only a single trained model. It consistently outperforms state-of-the-art configuration-specific baselines in both joint and vertex reconstruction accuracy, while our custom CUDA solver achieves up to a 100x speedup over standard PyTorch implementations. We further apply DirtyMoCap to heterogeneous raw optical MoCap recordings of traditional Chinese martial arts, yielding a Kung Fu motion dataset of temporally coherent SMPL-H reconstructions. Code and data are available at https://wanglongzju.github.io/DirtyMoCap-Project-Page.

Long Wang, Shuting Zhao, Shen Yan +5

Zhejiang University, Hangzhou, China · Westlake University, Hangzhou, China · Fudan University, Shanghai, China +3

Gauss-Newton Optimization 3D Human Pose Estimation 3D Human Reconstruction

Ask PDF

Mar 2, 2026

Stereo-Inertial Poser: Towards Metric-Accurate Shape-Aware Motion Capture Using Sparse IMUs and a Single Stereo Camera

Recent advancements in visual-inertial motion capture systems have demonstrated the potential of combining monocular cameras with sparse inertial measurement units (IMUs) as cost-effective solutions, which effectively mitigate occlusion and drift issues inherent in single-modality systems. However, they are still limited by metric inaccuracies in global translations stemming from monocular depth ambiguity, and shape-agnostic local motion estimations that ignore anthropometric variations. We present Stereo-Inertial Poser, a real-time motion capture system that leverages a single stereo camera and six IMUs to estimate metric-accurate and shape-aware 3D human motion. By replacing the monocular RGB with stereo vision, our system resolves depth ambiguity through calibrated baseline geometry, enabling direct 3D keypoint extraction and body shape parameter estimation. IMU data and visual cues are fused for predicting drift-compensated joint positions and root movements, while a novel shape-aware fusion module dynamically harmonizes anthropometry variations with global translations. Our end-to-end pipeline achieves over 200 FPS without optimization-based post-processing, enabling real-time deployment. Quantitative evaluations across various datasets demonstrate state-of-the-art performance. Qualitative results show our method produces drift-free global translation under a long recording time and reduces foot-skating effects.

Tutian Tang, Xingyu Ji, Yutong Li +3

School of Computer Science, Shanghai Jiao Tong University · Meta Robotics Institute, SJTU · Shanghai Innovation Institute

3D Human Pose Estimation Markerless Motion Capture

Ask PDF

Jul 22, 2026

Calibration-Free 3D Multi-Camera People Tracking for Indoor Environment

Multi-Camera People Tracking (MCPT) traditionally relies on precise intrinsic and extrinsic camera calibration to project 2D detections into a unified 3D world coordinate system.However, manual calibration constitutes a major bottleneck in large-scale dataset generation from unconstrained video archives. This work proposes a unified calibration-free 3D MCPT framework that infers geometric structure directly from visual data using deep foundation models. The system integrates anchor-free detection (YOLOX), robust tracking (BoT-SORT), omni-scale appearance embedding (OsNet), pose estimation (HRNet via MMPose), and transformer-based geometric reconstruction using the Visual Geometry Grounded Transformer (VGGT). A pose-guided 3D lifting strategy projects head keypoints onto a reconstructed manifold, eliminating dependence on ground-plane homography. Global identity association is formulated as hierarchical agglomerative clustering under a joint appearance-geometry cost with strict velocity gating. Evaluation on the AI City Challenge 2024 demonstrates a HOTA score of 53.13% without access to ground-truth calibration matrices, establishing a strong baseline for purely vision-based 3D tracking.

Ponleur Veng, Dominique Vaufreydaz, Phutphalla Kong

CADT, M-PSI · Cambodia Academy of Digital Technology · Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France +2

3D MOT Multi-Object Tracking

Ask PDF

Abstract

Markerless Motion Capture

Explore similar work

DirtyMoCap: Robust Motion Capture from Unconstrained Markers

Stereo-Inertial Poser: Towards Metric-Accurate Shape-Aware Motion Capture Using Sparse IMUs and a Single Stereo Camera

Calibration-Free 3D Multi-Camera People Tracking for Indoor Environment