Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports

Yuchen Yang; Yuqing Shao; Duxiu Huang; Linfeng Dong; Yifei Liu; Suixin Tang; Xiang Zhou; Yuanyuan Gao

cs.CVMar 10, 2026

Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports

Authors: Yuchen Yang, Yuqing Shao, Duxiu Huang, Linfeng Dong, Yifei Liu, Suixin Tang, Xiang Zhou, Yuanyuan Gao,

Organizations: Fudan University · Shanghai Artificial Intelligence Laboratory · East China University of Science and Technology · Southeast University ·

Abstract

Sports have long attracted broad attention as they push the limits of human physical and cognitive capabilities. Amid growing interest in spatial intelligence for vision-language models (VLMs), sports provide a natural testbed for understanding high-intensity human motion and dynamic object interactions. To this end, we present CourtSI, the first large-scale spatial intelligence dataset tailored to sports scenarios. CourtSI contains over 1M QA pairs, organized under a holistic taxonomy that systematically covers spatial counting, distance measurement, localization, and relational reasoning, across representative net sports including badminton, tennis, and table tennis. Leveraging well-defined court geometry as metric anchors, we develop a semi-automatic data engine to reconstruct sports scenes, enabling scalable curation of CourtSI. In addition, we introduce CourtSI-Bench, a high-quality evaluation benchmark comprising 3,686 QA pairs with rigorous human verification. We evaluate 25 proprietary and open-source VLMs on CourtSI-Bench, revealing a remaining human-AI performance gap and limited generalization from existing spatial intelligence benchmarks. These findings indicate that sports scenarios expose limitations in spatial intelligence capabilities captured by existing benchmarks. Further, fine-tuning Qwen3-VL-8B on CourtSI improves accuracy on CourtSI-Bench by 23.5 percentage points. The adapted model also generalizes effectively to CourtSI-Ext, an evaluation set built on a similar but unseen sport, and demonstrates enhanced spatial-aware commentary generation. Together, these findings demonstrate that CourtSI provides a scalable pathway toward advancing spatial intelligence of VLMs in sports.

Figures & tables

Figure 1: Overview. We introduce a semi-automatic data engine that reconstructs sports scenes in 3D with court, player, and ball locations. Built upon this pipeline, we present CourtSI and CourtSI-Bench, the first large-scale spatial intelligence dataset and benchmark for sports scenarios. In addition, we provide extra evaluation protocols to validate applicability on an unseen sport and spatial-aware commentary.

Figure 2: Overview of the data engine. It consists of court annotation for metric-aware camera parameter estimation, ball annotation, and player mesh recovery. By leveraging court geometry and incorporating human-in-the-loop supervision, the system enables accurate and world-grounded reconstruction in sports scenarios.

Figure 3: Taxonomy and examples of CourtSI. The questions are categorized into: spatial counting , distance measurement , localization , and relational reasoning . Cnt. denotes counting. Obj. refers to object, including the ball and players. Cam. denotes camera. Ego. and Allo. denote to ego-centric and allo-centric views.

Figure 4: Distribution of CourtSI and CourtSI-Bench. Obj. refers to object, including the ball and players. Cam. denotes camera.

Camera		Ball			Player
$f_{x}$	$f_{y}$	$X$	$Y$	$Z$	Pelvis	MPJPE
$2.2\%$	$2.4\%$	$22\mathrm{cm}$	$9\mathrm{cm}$	$9\mathrm{cm}$	$23\mathrm{cm}$	$17\mathrm{cm}$

Table 1: Quantitative error analysis of the data engine. MPJPE denotes Mean Per Joint Position Error for human skeletons.

Models	Dist. Meas.				Cnt.		Loc.	Rel. Reasoning						Overall
Models	Cam.-Obj.	Height	Obj.-Line	Obj.-Obj.	Ball	Player	Obj.	Ball-Zone	Ball-Player	Cam.-Player	Player-Zone	Player-Player	Player-Line
Baseline
[5 $\%$ Set] Human	64.4	92.7	67.8	70.0	100	100	11.9	85.7	75.0	100	83.3	90.3	88.9	73.6
Proprietary Models
GPT-5.2	27.9	78.4	31.8	49.2	32.1	100	1.1	68.2	67.0	75.4	50.0	67.7	77.4	53.7
Gemini-3-Pro	0.0	8.7	0.0	0.0	21.4	97.1	0.0	10.6	3.0	37.5	34.1	20.4	5.3	8.7

Table 2: Quantitative results on CourtSI-Bench. Dark orange and light orange highlight the best and second-best results within each group of models (proprietary and open-source). —parsed denotes results obtained by using a LLM to extract answers from the original model outputs. Dist. Means., Cnt., Loc., and Rel. denote Distance Measurement, Counting, Localization, and Relational tasks, respectively.

Figure 5: Error Analysis. The VLMs are prompted to provide detailed step-by-step reasoning. Correct and incorrect reasoning steps are highlighted in green and red , respectively. Questions and VLM’s explanations are simplified for demonstration.

Figure 6: The impact of perspective ambiguity.

Models	Dist. Meas.				Cnt.		Loc.	Rel. Reasoning						Overall
Models	Cam.-Obj.	Height	Obj.-Line	Obj.-Obj.	Ball	Player	Obj.	Ball-Zone	Ball-Player	Cam.-Player	Player-Zone	Player-Player	Player-Line
GPT-5.2	61.8	70.0	44.2	53.6	33.3	100	0.0	72.7	66.7	90.0	21.4	73.5	76.2	55.0
Gemini-3-Pro	0.0	15.4	0.0	0.0	33.3	100	0.0	9.1	11.1	30.0	64.3	17.6	0.0	13.5
—parsed	75.5	83.1	66.3	56.4	83.3	100	0.0	90.9	100	90.0	85.7	70.6	76.2	66.8
Qwen3-VL-235B-A22B	0.0	50.8	30.4	27.6	50.0	100	0.0	63.6	88.9	100	64.3	67.6	71.4	47.9
LLaVA-OneVision-72B	19.1	53.8	28.3	21.8	50.0	100	0.0	45.5	55.6	90.0	71.4	61.8	57.1	43.3

Table 3: Evaluation on CourtSI-Ext. Annotation conventions follow table 2 .

Figure 7: Evaluation on the spatial-aware commentary generation. Comparison between Qwen3-VL-8B fine-tuned on CourtSI and its base model. The left panel shows user study results assessing the quality of generated commentaries in both linguistic and spatial-awareness dimensions. The right panel presents an illustrative example.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Calibration examples. A 3D court box with real-world dimensions is reprojected onto the image using the estimated camera parameters. The close alignment between the projected court structure and the image clues indicates strong reprojection consistency, validating the accuracy of the calibration.

Figure 9: Illustration of player depth estimation using PromptHMR, DepthAnythingV3, and our method. All baselines take metric-scale camera intrinsics as input.

Baseline	Badminton		Tennis		Table Tennis
Baseline	$e_{fx}$ (%)	$e_{fy}$ (%)	$e_{fx}$ (%)	$e_{fy}$ (%)	$e_{fx}$ (%)	$e_{fy}$ (%)
WildCamera	67.20	70.86	17.48	12.16	13.66	15.20
DepthAnythingV3	38.89	38.24	5.42	4.16	0.85	0.90
Ours	0.55	0.72	4.23	4.31	0.01	1.45

Appendix

Table 4: Quantitative error analysis of camera intrinsic parameters.

Figure 10: Original broadcast frames (top) and depth maps estimated by DepthAnythingV3 (bottom). Yellow arrows indicate the ball positions.

Baseline	Badminton			Tennis			Table Tennis
Baseline	$X$	$Y$	$Z$	$X$	$Y$	$Z$	$X$	$Y$	$Z$
DepthAnythingV3 ∗	$1227\mathrm{cm}$	$252\mathrm{cm}$	$241\mathrm{cm}$	$3168\mathrm{cm}$	$2045\mathrm{cm}$	$1833\mathrm{cm}$	$249\mathrm{cm}$	$199\mathrm{cm}$	$26\mathrm{cm}$
Ours	$10\mathrm{cm}$	$4\mathrm{cm}$	$6\mathrm{cm}$	$29\mathrm{cm}$	$11\mathrm{cm}$	$9\mathrm{cm}$	$0.3\mathrm{cm}$	$0.3\mathrm{cm}$	$0.5\mathrm{cm}$

Appendix

Table 5: Quantitative error analysis of ball localization. ∗ denotes using ground truth 2D ball locations and camera parameters.

Baseline	Pelvis
Baseline	Badminton	Tennis	Table Tennis
PromptHMR	$134\mathrm{cm}$	$1144\mathrm{cm}$	$33\mathrm{cm}$
DepthAnythingV3	$2191\mathrm{cm}$	$2744\mathrm{cm}$	$62\mathrm{cm}$
Ours	$21\mathrm{cm}$	$27\mathrm{cm}$	$16\mathrm{cm}$

Appendix

Table 6: Quantitative error analysis of player localization.

Category Name		CourtSI-Bench				CourtSI
Category Name		Count	B	T	TT	Count	B	T	TT
Distance Measurement	Camera-Object	277	27.80%	35.74%	36.46%	75,783	33.93%	24.41%	41.66%
	Height	229	23.58%	37.12%	39.30%	51,154	31.43%	25.00%	43.56%
	Object-Line	317	24.61%	44.16%	31.23%	102,054	31.09%	25.20%	43.71%
	Object-Object	663	25.34%	41.18%	33.48%	178,878	29.95%	25.55%	44.50%
Spatial Counting	Ball	28	25.00%	42.86%	32.14%	23,015	31.02%	25.10%	43.88%

Appendix

Table 7: Detailed Data Distribution. B, T, and TT denote badminton, tennis, and table tennis, respectively.

Sports Analytics

See all 27 papers

ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning
Apr 27, 2026Yiming Zhang, Jiacheng Chen, Jiaqi Tan +3VLM Evaluation Spatial Reasoning Benchmarks
1Simon Fraser University · 2Hong Kong University of Science and Technology · University of Waterloo +1
ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Models
May 20, 2026Qirui Shen, Wenda Wang, Jiachen Lu +5VLM Evaluation Spatial Reasoning Benchmarks
School of Architecture, Tsinghua University
IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning
Jul 3, 2026Yuening Cai, Junwei Zhou, Youran Qu +1VLM Evaluation Spatial Reasoning Benchmarks
Department of Computer Science Dartmouth College

Apr 27, 2026J/K move · Enter open · S save

ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning

Yiming Zhang, Jiacheng Chen, Jiaqi Tan, Yongsen Mao+2

1Simon Fraser University · 2Hong Kong University of Science and Technology · University of Waterloo+1

VLM Evaluation Spatial Reasoning Benchmarks 3D Spatial Reasoning 3D Scene Understanding

Open paper AskPDF arXiv

Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings. First, many benchmarks derive question-answer (QA) pairs from point-cloud-based 3D annotations originally curated for traditional 3D perception. When such annotations are treated as ground truth for video-based evaluation, reconstruction and annotation artifacts can miss objects that are clearly visible in the video, mislabel object identities, or corrupt geometry-dependent answers (e.g., size), yielding incorrect or ambiguous QA pairs. Second, evaluations often assume full-scene access, while many VLMs operate on sparsely sampled frames (e.g., 16-64), making many questions effectively unanswerable under the actual model inputs. We improve evaluation validity by introducing ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs. To this end, we re-annotate objects and geometry across 381 scenes from 5 datasets to improve data quality, and regenerate all QA pairs with rigorous bias mitigation and human verification using professional 3D annotation tools. We further enhance evaluation controllability by providing variants across multiple frame budgets (16/32/64/all) and fine-grained object visibility metadata, enabling controlled diagnostic analyses. Evaluations of general and domain-specific VLMs on ReVSI reveal systematic failure modes that are obscured by prior benchmarks, yielding a more reliable and diagnostic assessment of spatial intelligence.

Abstract

Sports Analytics

Explore similar work

ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning

ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Models

IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning