cs.CVNov 6, 2026

BayesCrop: Real-Time Probabilistic Multi-Crop Video Reframing

Authors: Cheng-Han Lee, Rajesh Sureddi, Neil Birkbeck, Yilin Wang, Balu Adsumilli, Alan Bovik

Organizations: The University of Texas at Austin · YouTube, Google · University of Colorado Boulder

Abstract

Automatic video reframing is increasingly important for adapting landscape content to heterogeneous display formats. Existing methods typically predict a single deterministic crop, despite the subjective nature of reframing and the presence of multiple plausible regions of interest. We present BayesCrop, a real-time probabilistic framework for multi-crop video reframing. BayesCrop represents crop preferences using a two-component spatially isotropic Gaussian mixture over crop position and scale. Each component defines a representative reframing mode together with its local uncertainty, enabling both dual-view prediction and sampling of additional plausible crop candidates. For efficient video processing, we introduce an anchor-and-propagate strategy that performs neural prediction only on sparse anchor frames and propagates crop trajectories using motion vectors available from the compressed video bitstream. The resulting trajectories can be combined into a single output using a lightweight periodic view-switching strategy. Experiments on LIVE-YT-MC, LIVE-YT-VC++, and a multi-target long-video benchmark demonstrate competitive single-output localization, high multi-crop reference coverage, diverse yet coherent crop hypotheses, and stable perceptual quality. BayesCrop achieves real-time throughput of 112 FPS at the default operating point, while sampling from the learned distributions further improves reference coverage beyond the two representative crop means.

Figures & tables

Explore similar work

Apr 27, 2026

Subjective Portrait Region Cropping in Landscape Videos with Temporal Annotation Smoothing

With the rise of mobile video consumption on diverse handheld display resolutions and orientation modes, altering videos to aspect ratios poses challenges. Static cropping and border padding often compromises visual quality, while warping may distort a video's intended meaning. Here we advocate for a more effective approach: cropping significant regions within video frames in a temporal manner, while minimizing distortion and preserving essential content. One barrier to solving this problem is the lack of sufficiently large-scale database devoted to informing these tasks. Towards filling this gap, we introduce the LIVE-YouTube Video Cropping (LIVE-YT VC) database, featuring 1800 videos, annotated by 90 human subjects. Using videos sourced from the YouTube-UGC and LSVQ Databases, this new resource is the largest publicly-available subjective video portrait region cropping database. We also introduce a post-processed version of the database, called LIVE-YT VC++, whereby a novel intra-frame temporal filter was deployed to smooth subjective annotations within each video. We demonstrate the usefulness of this new data resource using the SmartVidCrop algorithm and state-of-the-art video grounding models, in hopes of establishing our subjective dataset as a benchmark for future research. Our contributions offer a resource for advancing video aspect ratio transformation models towards ensuring that reshaped mobile-friendly video content retains its quality and meaning. Since our labels bear resemblances to video saliency annotations, we also conducted an additional analysis to explore the similarity between our labels and video saliency predictions. Finally, we repurposed state-of-the-art video grounding models for aspect ratio change tasks, and fine-tuned them on our dataset. As a service to the research community, we plan to open source the project.
Mar 15, 2026

Show Me When and Where: Towards Referring Video Object Segmentation in the Wild

Referring video object segmentation (RVOS) has recently generated great popularity in computer vision due to its widespread applications. Existing RVOS setting contains elaborately trimmed videos, with text-referred objects always appearing in all frames, which however fail to fully reflect the realistic challenges of this task. This simplified setting requires RVOS methods to only predict where objects, with no need to show when the objects appear. In this work, we introduce a new setting towards in-the-wild RVOS. To this end, we collect a new benchmark dataset using Youtube Untrimmed videos for RVOS - YoURVOS, which contains 1,120 in-the-wild videos with 7 times more duration and scenes than existing datasets. Our new benchmark challenges RVOS methods to show not only where but also when objects appear in videos. To set a baseline, we propose Object-level Multimodal TransFormers (OMFormer) to tackle the challenges, which are characterized by encoding object-level multimodal interactions for efficient and global spatial-temporal localisation. We demonstrate that previous VOS methods struggle on our YoURVOS benchmark, especially with the increase of target-absent frames, while our OMFormer consistently performs well. Our YoURVOS dataset offers an imperative benchmark, which will push forward the advancement of RVOS methods for practical applications.
Jul 30, 2026

RefCaptioner: Multi-Reference Image-Grounded Video Captioning

Existing video captioning models generate natural descriptions of video content but cannot explicitly ground local visual elements to multiple reference images. We introduce multi-reference image-grounded video captioning, a new task requiring factual video descriptions with phrase-level reference grounding, and propose RefCaptioner, a two-stage post-training framework for this task. RefCaptioner combines mixed-data SFT with Hierarchical Coverage-Discounted GRPO to jointly improve reference selection, phrase-level binding, distractor rejection, and cross-reference consistency while preserving general video-captioning ability. To support training, we construct a corpus containing 20,00020,000 videos and 171,354 reference images. We further introduce MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding on both real-world and AI-generated videos. Experiments show that RefCaptioner achieves the best overall performance among the open-source models while remaining competitive on standard video captioning benchmarks. Human evaluation further confirms that its captions are preferred by annotators and enable more source-faithful video reconstruction with both open-source and proprietary video generators.