Volley3D: Unified 3D Full-Sequence Reconstruction from Monocular Broadcasts

Noâm Boussouf; Adrien Maglo; Mohamed Chaouch; Hideo Saito

cs.CVOct 28, 2026

Volley3D: Unified 3D Full-Sequence Reconstruction from Monocular Broadcasts

Authors: Noâm Boussouf, Adrien Maglo, Mohamed Chaouch, Hideo Saito

Organizations: Keio University, Japan · ´Ecole Centrale de Nantes, France · Universit´e Paris-Saclay, CEA-LIST, F-91120, Palaiseau, France

Abstract

3D reconstruction from single-view sports broadcast is a pivotal capability for automated analytics and immersive AR/VR replays, yet notoriously ill-posed due to difficult-to-calibrate moving cameras and depth ambiguity. Unlike widely studied racket sports, team sports such as volleyball feature unpredictable multi-contact sequences and high altitude jumps. These dynamics categorically invalidate the standard planar projection assumptions relied upon by existing methods. To resolve these limitations without 3D training supervision, we propose a complete, physics-grounded pipeline tailored to monocular broadcast volleyball. First, we include a correction module to the perspective drift induced by ground-plane projection by coupling temporal jump spotting with Newtonian interpolation for 3D player placement. Second, for 3D ball trajectory estimation, we design a dedicated sub-trajectory segmentation architecture alongside a collision-matching algorithm. By anchoring physical equations to reconstructed 3D hand vertices, we lift noisy ball 2D detections into a spatiotemporally coherent 3D scene. Evaluations on multi-camera benchmarks demonstrate our approach significantly reduces 3D spatial drift compared to 2D-reprojection baselines.

Figures & tables

(b) Reconstructed 3D Scene (Broadcast POV)

Figure 2: Overview of our proposed two-stage monocular 3D reconstruction pipeline. In the first stage, we extract and position the athletes in 3D world-space, combining smoothed homographic projections with a pose-based jump spotting module to handle aerial phases. In the second stage, we segment 2D ball tracks and explicitly anchor them to the reconstructed athletes, using the predicted 3D collision points to physically constrain the final 3D ballistic trajectory. The blue modules highlight the contributions introduced in this paper.

Figure 3: ButterWorth Filtering and Perspective Drift Correction . The raw 2D homographic projection of a player’s position during a sequence (grey) exhibits high-frequency jitter. While temporal filtering (blue dashed line) stabilizes ground locomotion, it fails during airborne phases (yellow zones). Our physics-informed interpolation (solid red line) overrides the ground-plane.

Table 1: Quantitative Evaluation of 3D Player Reconstruction Components.

Method	$\delta=1$	$\delta=2$	$\delta=3$
Using Oracle 2D Ball Trajectory
Baseline 1 ( $Y(t)$ )	0.801	0.840	0.843
Baseline 2 ( $X(t)+Y(t)$ )	0.794	0.872	0.884
TT3D (Poly. only) [ 12 ]	0.713	0.751	0.764
TT3D (Poly. + PCA Blur) [ 12 ]	0.762	0.807	0.810
HitNet (MonoTrack) [ 21 ]	0.836	0.874	0.877

Table 2: 2D Ball Trajectory Segmentation (F1-Score). Performance across temporal tolerances

\delta\in\{1,2,3\}

frames.

Theoretical Bounds (Using 3D GT Start/End Points)
Method	3D RMSE (m)	X/Y/Z RMSE (m)	2D Reproj. (px)
MonoTrack [ 21 ]	1.56	1.42 / 0.62 / 0.12	3.89
Pure Ballistics	0.38	0.28 / 0.10 / 0.23	19.2
With Drag	0.29	0.14 / 0.09 / 0.24	17.9
Our Pipeline (Using 3D Hand Bounds)
Pure Ballistics	0.38	0.31 / 0.14 / 0.18	19.4

Table 3: 3D Ball Trajectory Estimation on VolliQ. Evaluation of 3D spatial accuracy and 2D reprojection error. Our holistic physics-informed approach drastically reduces 3D spatial drift compared to reprojection-based optimization.

Figure 4: Qualitative comparison of 3D trajectory reconstruction on a short rally from Volliq [ 37 ] . Our drag model (green, err: 0.19 m, 12.7 px ) accurately follows the ground truth (red) compared to MonoTrack [ 21 ] (purple, err: 1.69 m, 4.32 px).

Panel of Figure 5 — Figure 5: Qualitative 3D reconstruction results across diverse game scenarios. Top: 2D input broadcast frames from different rallies. Bottom: corresponding unified 3D scene reconstruction.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Split	# Frames	Avg. Points / Frame
Train	923	5.64
Validation	350	5.32
Test	164	4.53

Appendix

Table S1: Court Calibration Dataset. We report the number of annotated frames per split and the average number of visible court keypoints annotated per frame.

Split	Clips	Frames	Avg.	Clips	Frames	Avg.
	Original [ 17 ]			Merged (Ours)
Train	2,152	88,232	41.0	805	69,320	86.1
Val	1,341	54,981	41.0	541	43,331	80.1
Test	1,337	54,817	41.0	548	43,628	79.6
Total	4,830	198,030	41.0	1,894	156,279	82.5

Appendix

Table S2: Trajectory Dataset Merging Statistics. Comparison between the original 41-frame video clips from Saad et al . [ 17 ] and our continuous merged rally sequences.

Split	Annotated Clips	Sub-trajectories	Collisions
Train	135	613	0
Val	87	353	0
Test	155	655	654
Total	377	1,621	654

Appendix

Table S3: Manual Trajectory and Collision Annotations. Statistics of our custom annotations across the subset of merged rally sequences.

Figure S1 : Agent Collision Distribution. Frequency of each physical source across our annotated dataset.

Dataset	Rallies	Frames per View	Sub-trajectories
VolliQ	10	3066	101

Appendix

Table S4: VolliQ 3D Benchmark Statistics. Overview of the synchronized four-camera evaluation dataset, reporting the number of rallies, duration in frames per viewpoint, total multi-view frame count, and evaluated sub-trajectories.

Figure S2 : 2D Court Template with Keypoint IDs. Distribution of the 65 annotated semantic keypoints defined across the metric court space ( $1800\text{ cm}\times 900\text{ cm}$ ). The non-linear distribution along the $Y$ -axis increases keypoint density near the foreground to counteract broadcast perspective foreshortening.

Method	Accuracy (%) $\uparrow$	F1-Score (%) $\uparrow$
DeepBall †	50.7	64.4
DeepBall-Large †	57.5	70.4
BallSeg †	17.5	19.5
TrackNetV2 †	73.8	83.6
ResTrackNetV2 †	74.7	84.2
MonoTrack †	75.9	85.1

Appendix

Table S5: 2D Ball Tracking Performance on the Volleyball Dataset. Evaluation with a distance threshold

\tau=4\text{ px}

. Methods marked with ( † ) are directly reported from WASB [ 33 ] . TrackNetV3 integrates both TrackNet and InPaintNet.

Theoretical Bounds (Using 3D GT Start/End Points)
	2D Reproj. RMSE (px) $\downarrow$				3D RMSE (m) $\downarrow$
Method	AN01	AN02	AN03	AN04	VolliQ (3D)
MonoTrack [ 21 ]	90.6	3.89	131.0	53.7	1.55
Pure Ballistics	26.3	19.2	26.6	29.1	0.380
With Drag	24.9	17.9	24.5	26.9	0.293
Our 3D Pipeline (Using Triangulated 3D Hand Bounds)

Appendix

Table S6: Cross-View Reprojection vs. Metric 3D Accuracy on VolliQ. MonoTrack [ 21 ] optimizes its trajectory via 2D reprojection on camera AN02 (its best-performing viewpoint among all 4 views) using GT triangulated foot positions and GT camera calibration. Our physics-informed methods operate purely in metric 3D space using 3D boundary constraints. The resulting 3D trajectories are reprojected onto all four views to diagnose projective overfitting and showcase reprojection methods flaws.

Panel of Figure 19 — Figure S3: Qualitative 2D Reprojection and 3D Trajectory Comparisons on VolliQ. Visual comparison between MonoTrack [ 21 ] (purple), our physics-informed lifting formulation with aerodynamic drag (green), and Ground Truth (red) across two distinct rally sequences.

Panel of Figure 20 — Figure S4: Step-by-Step Visualization of Intermediate Pipeline Stages. Visual walkthrough on the teaser rally frame: (a) raw input feed; (b) 2D bounding boxes and tracklet IDs; (c) homography estimation and projected court grid; (d) SMPL body mesh recovery; (e) jump event spotting; (f) localized 2D ball position; (g) temporal TCN deflection probabilities across the rally (pink vertical bar indicates the query frame); (h) heuristic wrist proximity and velocity scoring attributing the collision; (i, j) final 3D scene reconstructed in Blender.

Panel of Figure 21 — Figure S5: Qualitative 3D Reconstruction Visualizations (Part 1/2). Paired comparisons between monocular broadcast input frames (left) and the complete metric 3D scene reconstructed in Blender (right).

Panel of Figure 22 — Figure S6: Qualitative 3D Reconstruction Visualizations (Part 2/2). Additional paired comparisons demonstrating cross-camera stability, player grounding, and aerial spike reconstructions from uncalibrated monocular broadcast feeds.

Monocular 3D Reconstruction

Monocular 3D reconstruction infers scene geometry, objects, and motion from images or video captured by a single camera.

See all 116 papers

TT4D: A Pipeline and Dataset for Table Tennis 4D Reconstruction From Monocular Videos
May 2, 2026Nima Rahmanian, Daniel Kienzle, Thomas Gossard +3Monocular 3D Reconstruction Sports Video Analysis
University of Tübingen Tübingen, Germany
Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels
Mar 3, 2026Jiahao Lu, Jiayi Xu, Wenbo Hu +5Scene Flow Estimation 3D Point Tracking
The Hong Kong University of Science and Technology · ARC Lab, Tencent PCG
PAS3R: Pose-Adaptive Streaming 3D Reconstruction for Long Video Sequences
Mar 22, 2026Lanbo Xu, Liang Guo, Caigui Jiang +1Streaming 3D Reconstruction Monocular 3D Reconstruction
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, China · University of East Anglia, Norwich, NR47TJ, United Kingdom

May 2, 2026J/K move · Enter open · S save

TT4D: A Pipeline and Dataset for Table Tennis 4D Reconstruction From Monocular Videos

Nima Rahmanian, Daniel Kienzle, Thomas Gossard, Dvij Kalaria+2

University of Tübingen Tübingen, Germany

Monocular 3D Reconstruction Sports Video Analysis Dynamic Scene Reconstruction

Open paper AskPDF arXiv

We present TT4D, a large-scale, high-fidelity table tennis dataset. It provides

140+

hours of reconstructed singles and doubles gameplay from monocular broadcast videos, featuring multimodal annotations like high-quality camera calibrations, precise 3D ball positions, ball spin, time segmentation, and 3D human meshes over time. This rich data provides a new foundation for virtual replay, in-depth player analysis, and robot learning. The dataset's combination of scale and precision is achieved through a novel reconstruction pipeline. Prior methods first partition a game sequence into individual shot segments based on the 2D ball track, and only then attempt reconstruction. However, 2D-based time segmentation collapses under occlusion and varied camera viewpoints, preventing reliable reconstruction. We invert this paradigm by first lifting the entire unsegmented 2D ball track to 3D through a learned lifting network. This 3D trajectory then allows us to reliably perform time segmentation. The learned lifting network also infers the ball's spin, handles unreliable ball detections, and successfully reconstructs the ball trajectory in cases of high occlusion. This lift-first design is necessary, as our pipeline is the only method capable of reconstructing table tennis gameplay from general-view broadcast monocular videos. We demonstrate the dataset's fidelity through two downstream tasks: estimating the racket's pose & velocity at impact, and training a generative model of competitive rallies.

Abstract

Monocular 3D Reconstruction

Explore similar work

TT4D: A Pipeline and Dataset for Table Tennis 4D Reconstruction From Monocular Videos

Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels

PAS3R: Pose-Adaptive Streaming 3D Reconstruction for Long Video Sequences