Automatic video reframing is increasingly important for adapting landscape content to heterogeneous display formats. Existing methods typically predict a single deterministic crop, despite the subjective nature of reframing and the presence of multiple plausible regions of interest. We present BayesCrop, a real-time probabilistic framework for multi-crop video reframing. BayesCrop represents crop preferences using a two-component spatially isotropic Gaussian mixture over crop position and scale. Each component defines a representative reframing mode together with its local uncertainty, enabling both dual-view prediction and sampling of additional plausible crop candidates. For efficient video processing, we introduce an anchor-and-propagate strategy that performs neural prediction only on sparse anchor frames and propagates crop trajectories using motion vectors available from the compressed video bitstream. The resulting trajectories can be combined into a single output using a lightweight periodic view-switching strategy. Experiments on LIVE-YT-MC, LIVE-YT-VC++, and a multi-target long-video benchmark demonstrate competitive single-output localization, high multi-crop reference coverage, diverse yet coherent crop hypotheses, and stable perceptual quality. BayesCrop achieves real-time throughput of 112 FPS at the default operating point, while sampling from the learned distributions further improves reference coverage beyond the two representative crop means.
Figures & tables
Figure 1: Probabilistic ROI prediction on LIVE-YT-VC++ [ 15 ] . Green boxes denote predicted GMM means; red and blue boxes are samples from the two crop modes.
Figure 2: Overview of BayesCrop. Given a landscape-orientation (LO) video, BayesCrop predicts two probabilistic crop modes on anchor frames and propagates them using compressed-domain motion vectors. Gaussian temporal smoothing produces two stable crop trajectories, which are alternated by a lightweight Periodic View Switch Agent to generate the final mixed-view portrait-orientation (PO) output.
Figure 3: Pseudo-label generation pipeline for LIVE-YT-MC within a sliding temporal window. The three steps are: (1) temporal warping of human-opinion crops (red), (2) aggregation of warped candidates within the window (blue), and (3) two-component GMM fitting, where green boxes denote representative crop means. For visualization, the figure uses a window size of 5, while the actual data-generation pipeline uses a window size of 15.
Figure 4: GMM-based ROI modeling in LIVE-YT-MC. Elliptical contours show the original fitted spatial covariance, while circular contours show the spatially isotropic representation used for supervision. The examples illustrate both separated crop modes and local scale variation.
Figure 5: Architecture of Probabilistic Multi-Crop Predictor. A frozen DINOv2 backbone and multi-scale feature aggregator feed two learnable crop queries that predict the parameters (π,μ,σ) of a two-component spatially isotropic GMM.
Method
π -mIoU ↑
π Acc. ↑
W2(x,y)↓
W2(x,y,r)↓
CenterCrop
46.6
27.1
0.13
0.20
BayesCrop w/o Aggregator
54.8
62.4
0.10
0.14
BayesCrop w/o IoU Loss
49.8
57.6
0.11
0.16
BayesCrop w/o KL Loss
56.6
62.2
0.10
0.13
BayesCrop (Full)
58.3
63.4
0.09
0.13
Table 1: Ablation study of the BayesCrop predictor on LIVE-YT-MC. We evaluate spatial localization, mixture-weight prediction, and distribution matching. Lower Wasserstein distance indicates better distributional agreement.
Refresh Interval
mIoU ↑
Aesthetic ↑
Temp. Smooth. ↑
FPS ↑
R=1
52.2
45.4
96.0
73
R=5
52.7
45.7
95.9
112
R=10
52.4
45.4
96.0
116
R=15
52.0
45.5
96.0
120
Table 2: Ablation study of the periodic refresh interval R on LIVE-YT-VC++. We evaluate the trade-off among localization accuracy, perceptual quality, temporal smoothness, and system throughput.
Method
# Params ↓
Single mIoU ↑
Best-of-2 ↑
FPS ↑
CenterCrop
–
47.7
–
–
SmartVidCrop [ 1 ]
8.33M
48.6
–
16.9
STCAT [ 12 ]
207.5M
58.2
–
44.1
CG-STVG [ 8 ]
238.9M
59.1
–
31.9
BayesCrop ( π^ -selected, R=5 )
26.0M
55.2
–
112
BayesCrop (Two-View Set, R=5 )
26.0M
–
68.4
112
Table 3: Comparison with state-of-the-art methods on LIVE-YT-VC++. Single-output mIoU evaluates one selected crop per frame. Because BayesCrop explicitly predicts two plausible crop modes, we additionally report Best-of-2 IoU as a set-level reference coverage metric. Best-of-2 and conventional single-output mIoU measure different properties and should not be interpreted as identical evaluation protocols.
Output / Selection Strategy
Evaluation
IoU ↑
Always Left View
Single-output
52.7
Always Right View
Single-output
53.3
Random Selection
Single-output
53.1
π^ -based Selection
Single-output
55.2
Best-of-2
Set Coverage
68.4
Table 4: Analysis of BayesCrop view-selection strategies on LIVE-YT-VC++. The first four rows evaluate single-output policies, while Best-of-2 evaluates the complete two-view candidate set.
Figure 6: Qualitative comparison on MiraData [ 13 ] . Rows correspond to CenterCrop (top), Gemini 3 Flash (middle), and BayesCrop (bottom). BayesCrop provides more varied viewpoints while maintaining stable framing around relevant content over time.
Method
Aesthetic ↑
Temp. Smooth. ↑
Text-Video Align. ↑
Preference ↑
CenterCrop
54.2
96.5
24.0
33.3%
Gemini 3 Flash
53.8
96.6
23.4
27.2%
BayesCrop + Periodic Agent
54.7
96.4
24.2
39.4%
Table 5: Evaluation on the Multi-Target Long Video Benchmark using VBench and a three-way human preference study (180 total votes). BayesCrop uses S=3 temporal segments for periodic view switching.
Candidates
Best-of- N↑
Avg. IoU ↑
Intra IoU
Cross IoU ↓
Means (2)
68.4
53.0
–
42.6
Samples (2)
66.5
51.6
–
41.2
Samples (6)
70.9
51.3
85.3
41.3
Samples (10)
72.5
51.4
85.5
41.3
Table 6: GMM sampling analysis on LIVE-YT-VC++. Best-of- N measures set-level reference coverage, Avg. IoU measures average candidate quality against the single reference, and Intra/Cross IoU characterize the compactness within each mode and separation between the two modes.
Figure 7: Details of Model Design
Dataset
Aes. ↑
Temp. ↑
LIVE-YT-VC [ 14 ]
46.7
90.4
LIVE-YT-VC++ [ 14 ]
46.8
92.8
LIVE-YT-MC (Left View)
45.3
94.8
LIVE-YT-MC (Right View)
45.3
94.8
Table 7: Comparison of Dataset Quality across LIVE-YT benchmarks. We report the VBench Aesthetic Quality (Aes.) and Temporal Smoothness (Temp.) scores to evaluate the baseline characteristics of each corpus.
Figure 10: Box area analysis of LIVE-YT-MC
Figure 11: Box overlap analysis of LIVE-YT-MC
Method
IoU ↑
AutoFlip
50.8
SmartVidCrop (w/o Clustering) [ 1 ]
46.1
SmartVidCrop [ 1 ]
49.9
FEVR [ 7 ]
54.9
BayesCrop (Left View, R=5 )
43.1
BayesCrop (Right View, R=5 )
45.8
Table 8: Quantitative comparison on the RetargetVid [ 1 ] benchmark. Deterministic baselines are evaluated using standard single-output mIoU. For BayesCrop, we additionally report π^ -based single-view selection and Best-of-2 IoU, where the latter measures set-level reference coverage over the two predicted crop hypotheses. Best-of-2 IoU is therefore not directly comparable to conventional single-output mIoU. All outputs are standardized to the benchmark setting using a fixed 1:3 aspect ratio while preserving the original frame height.
View-Switch Strategy
Aesthetic ↑
Temp. Smooth. ↑
Text–Video Align. ↑
Periodic View Switch Agent
54.7
96.4
24.2
MLLM-Based View Switch Agent
54.7
96.4
24.0
Table 9: Comparison of BayesCrop view-switching strategies on the Multi-Target Long Video Benchmark. Both methods use the same two BayesCrop trajectories and differ only in the temporal view-selection policy.
Figure 12: Qualitative comparison of ROI extraction methods on the MiraData dataset [ 13 ] (Part 1 of 3). For each subfigure ( a – b ), representing individual video samples stacked vertically, the rows correspond to the Center Crop (first), the Baseline (second), Ours (third), and Final Crop (bottom).
Figure 12: Qualitative comparison of ROI extraction methods on the MiraData dataset [ 13 ] (Part 2 of 3). Subfigures ( c – d ) demonstrate further examples including our Final Crop outputs on the bottom row.
Figure 12: Qualitative comparison of ROI extraction methods on the MiraData dataset [ 13 ] (Part 3 of 3). Subfigure ( e ) demonstrates the final example of tracking and framing performance.
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China · Warwick Manufacturing Group, University of Warwick, Coventry CV1 7AL, U.K. · Tapall.ai, Shenzhen 518055, China +3