Interpretable Pre-Release Baseball Pitch Type Anticipation from Broadcast 3D Kinematics

Jerrin Bright; Michelle Lu; John Zelek

cs.CVMar 5, 2026

Interpretable Pre-Release Baseball Pitch Type Anticipation from Broadcast 3D Kinematics

Authors: Jerrin Bright, Michelle Lu, John Zelek

Organizations: Vision and Image Processing Lab · University of Waterloo, Canada

Abstract

How much can a pitcher's body reveal about the upcoming pitch? We study this question at scale by classifying eight pitch types from monocular 3D pose sequences, without access to ball-flight data. Our pipeline chains a diffusion-based 3D pose backbone with automatic pitching-event detection, groundtruth-validated biomechanical feature extraction, and gradient-boosted classification over 229 kinematic features. Evaluated on 119,561 professional pitches, the largest such benchmark to date, we achieve 80.4% accuracy using body kinematics alone. A systematic importance analysis reveals that upper-body mechanics contribute 64.9% of the predictive signal versus 35.1% for the lower body, with wrist position (14.8%) and trunk lateral tilt emerging as the most informative joint group and biomechanical feature, respectively. We further show that grip-defined variants (four-seam vs.\ two-seam fastball) are not separable from pose, establishing an empirical ceiling near 80% and delineating where kinematic information ends and ball-flight information begins.

Figures & tables

Figure 1: Overview of the proposed system. The input broadcast video is first processed by a 2D pose estimator [ 31 ] and a VLM [ 29 ] to obtain the action prompt and per-frame 2D poses. These are passed to DreamPose3D [ 4 ] to reconstruct the pitcher’s 3D pose sequence. From the 3D poses, we perform handedness estimation and event localization, and use these features together with the 3D pose sequence to classify the pitch type.

Pitch Type	Count	Percentage
Four-seam Fastball (FF)	38,560	32.3%
Two-seam Fastball (FT)	18,570	15.5%
Slider (SL)	16,221	13.6%
Changeup (CH)	12,170	10.2%
Curveball (CB)	10,456	8.7%
Sweeper (SW)	10,225	8.6%

Table 1: Pitch type distribution in the filtered dataset.

Configuration	Feats.	RF	XGB
Poses Only	154	63.1%	76.5%
Poses + Biomechanics	229	73.2%	80.4%
Poses + Ball Flight †	166	91.9%	94.0%

Table 2: Classification accuracy across feature configurations. † Uses post-release ball-flight data (not a valid predictive setting).

Pitch	Prec.	Rec.	F1	Support
CH	91%	81%	85%	2,434
CB	88%	80%	84%	2,091
SP	87%	71%	78%	723
SW	84%	78%	81%	2,045
SL	83%	78%	81%	3,245
FF	78%	90%	84%	7,712

Table 3: Per-class performance of the proposed model (XGBoost, Poses + Biomechanics) on the test set (

n

= 23,913).

Figure 2: Feature importance decomposition across four complementary views: (a) top-10 individual features ranked by XGBoost gain, (b) joint-level importance aggregated over all features per joint, (c) feature category breakdown showing the relative contribution of raw poses, biomechanical metrics, temporal deltas, and handedness, and (d) anatomical body-part grouping with upper- vs. lower-body totals. All panels share a consistent blue palette; darker shades indicate higher importance or upper-body / biomech associations, lighter shades indicate lower-body or raw-pose features.

Figure 3: Normalized confusion matrix (row-normalized; values are per-class recall in %). Diagonal cells indicate correct classification rates per pitch type; off-diagonal cells indicate systematic confusions. Red boxes highlight the dominant fastball ambiguity (FF $\leftrightarrow$ FT).

Configuration	Features	Accuracy	$\Delta$
Raw Poses only	154	76.5%	–
+ Biomech Metrics	199	78.9%	+2.4%
+ Temporal Deltas	229	80.4%	+1.5%

Table 4: Feature addition and corresponding accuracy gains.

Approach	Features	Accuracy
10 frames (evenly spaced)	510	64.1%
3 frames (evenly spaced)	154	63.2%
3 key events (FP/MER/REL)	154	76.5%

Table 5: Sampling Strategy Comparison

Event	Mean Error	Median Error	Within $\pm$ 10 ms
FP	46.9 ms	40.2 ms	22.1%
MER	46.5 ms	34.2 ms	31.4%
REL	9.6 ms	7.5 ms	90.7%

Table 6: Event detection accuracy evaluated on 66 broadcast pitches against ground truth event timestamps. Error is the absolute timing difference between our detected event and labeled event.

Group	Count	Accuracy
Right-Handed (RHP)	82,453	80.6%
Left-Handed (LHP)	37,108	79.9%
Overall	119,561	80.4%

Table 7: Accuracy by Handedness

Figure 4: Feature importance aggregated by pitching event. All three events contribute within a 5% range, indicating that the full delivery, not just the release point, encodes pitch-type information.

Figure 5: Hand pose estimation with ball interaction. As future work, we plan to incorporate hand pose estimation in conjunction with the interaction object (baseball) to capture grip-specific contextual cues, enabling more analysis of pitch-type indicators.

Explore similar work

Scalable Injury-Risk Screening in Baseball Pitching From Broadcast Video
Mar 5, 2026Jerrin Bright, Justin Mende, John ZelekMarkerless Motion Capture Sports Video Analysis
Vision and Image Processing Lab · University of Waterloo, Canada
A 3D Pose-Based Ensemble Framework for Cricket Shot Classification and Automated Biomechanical Analysis
Sep 22, 2026Sourav Shome, M. D. Ashiquzzaman Rahad, Rameswar DebnathSports Video Analysis Image Classification
Computer Science and Engineering Discipline Khulna University
Throwing4: Phase-Aware Pose Modeling for Throw-Distance Prediction in Elite Athletics
Oct 12, 2026Bikash Kumar Badatya, Kartike Tiwari, Jyotirmoy Amin +1Sports Outcome Prediction Sports Video Analysis
Department of Electrical Engineering, Indian Institute of Technology Gandhinagar, Gandhinagar, India. · Department of Electronics and Communication Engineering, Madan Mohan Malaviya University of Technology, Gorakhpur, India. · Department of Mathematical Sciences, IISER Berhampur, Berhampur, India.

Mar 5, 2026J/K move · Enter open · S save

Scalable Injury-Risk Screening in Baseball Pitching From Broadcast Video

Jerrin Bright, Justin Mende, John Zelek

Vision and Image Processing Lab · University of Waterloo, Canada

Markerless Motion Capture Sports Video Analysis 3D Human Pose Estimation

Open paper AskPDF arXiv

Injury prediction in pitching depends on precise biomechanical signals, yet gold-standard measurements come from expensive, stadium-installed multi-camera systems that are unavailable outside professional venues. We present a monocular video pipeline that recovers 18 clinically relevant biomechanics metrics from broadcast footage, positioning pose-derived kinematics as a scalable source for injury-risk modeling. Built on DreamPose3D, our approach introduces a drift-controlled global lifting module that recovers pelvis trajectory via velocity-based parameterization and sliding-window inference, lifting pelvis-rooted poses into global space. To address motion blur, compression artifacts, and extreme pitching poses, we incorporate a kinematics refinement pipeline with bone-length constraints, joint-limited inverse kinematics, smoothing, and symmetry constraints to ensure temporally stable and physically plausible kinematics. On 13 professional pitchers (156 paired pitches), 16/18 metrics achieve sub-degree agreement (MAE

< 1^{\circ}

). Using these metrics for injury prediction, an automated screening model achieves AUC 0.811 for Tommy John surgery and 0.825 for significant arm injuries on 7,348 pitchers. The resulting pose-derived metrics support scalable injury-risk screening, establishing monocular broadcast video as a viable alternative to stadium-scale motion capture for biomechanics.

Abstract

Explore similar work

Scalable Injury-Risk Screening in Baseball Pitching From Broadcast Video

A 3D Pose-Based Ensemble Framework for Cricket Shot Classification and Automated Biomechanical Analysis

Throwing4: Phase-Aware Pose Modeling for Throw-Distance Prediction in Elite Athletics