Bitstream-corrupted Harsh Visual Understanding (BcHVU) aims to understand harshly degraded videos originally decoded from a severely corrupted bitstream in real-world multimedia communication. The ill-posed nature of BcHVU poses a major challenge for existing vision models, as even subtle bitstream corruption can lead to irreversible pixel distortion and significant semantic loss. To address these challenges in BcHVU, we propose Bitstream Language Modeling as Robust Semantic Priors (BLMSP), a framework for learning and injecting bitstream-native semantic cues. Our proposed BLMSP framework learns to extract bitstream-native semantic cues by bitstream language modeling, and leverages them as priors by injecting into off-the-shelf vision models of BcHVU tasks. Specifically, we present a Video Bitstream Byte Model (VBBM) that integrates byte-level modeling and cross-codec semantic distillation, enabling it to interpret robust semantics from byte sequences in multiple corrupted bitstream formats. The learned bitstream semantics are leveraged as robust priors and fused into BcHVU model backbones for improving the quality of video restoration, captioning, and human pose estimation. To train BLMSP, we construct a large-scale multi-source Corrupted-bitstream Harsh-video Paired (CHP) dataset containing 607k corrupted bitstream segments and 287k paired harsh video clips. Extensive experimental results show that the learned bitstream priors improve video restoration, captioning, and human pose estimation by 2.51 dB in PSNR, 0.20 in CIDEr, and 0.18 in [email protected] on average, respectively. These results demonstrate that corrupted bitstream can serve as robust semantic priors in solving pixel distortion and semantic loss in BcHVU.
Figures & tables
Figure 1: Comparison between downstream processing without and with bitstream-native semantic priors.
Figure 2: Overview of the proposed BLMSP framework for Bitstream-corrupted Harsh Visual Understanding (BcHVU). BLMSP learns semantic representations directly from corrupted video bitstreams and injects them as robust priors into downstream BcHVU-specific models. A cross-codec semantic distillation scheme further transfers knowledge from MJPEG to H.264 bitstreams through feature alignment and distribution-level supervision, enabling robust video captioning, human pose estimation, and video restoration under severe visual degradation.
Figure 3: Overview of the CHP dataset. (a) Sample distribution across corruption scenarios and source codecs. (b) Examples of clean, bit-flipped, byte-loss, and mixed corruption. (c) Decoded video frames with different corruption severities.
Scenario
S
p
q
α
Light-Flip
512
0.05
0.15
1.0
Medium-Flip
256
0.10
0.25
1.0
Heavy-Flip
128
0.15
0.35
1.0
Light-Loss
512
0.05
0.15
0.0
Medium-Loss
256
0.10
0.25
0.0
Heavy-Loss
128
0.15
0.35
0.0
Table 1: Bitstream corruption parameters used to construct CHP. S is the segment length in bytes; p is the segment-selection probability; q is the per-byte perturbation probability; and α controls the corruption type.
Figure 4: Top-1 action recognition accuracy of byte-level models on clean and corrupted UCF101 H.264 bitstreams. Existing methods are evaluated on clean bitstreams, while our model is evaluated under both clean and corrupted bitstream conditions.
Figure 5: Qualitative examples under bitstream corruption. (a) Video captioning compares ground-truth descriptions, captions without priors, and captions with priors. (b) Human pose estimation uses white, red, and green skeletons to denote ground truth, without priors, and with priors, respectively. (c) Video restoration compares ground-truth images, corrupted inputs, restorations without priors, and restorations with priors.
Bitstream-corrupted image restoration (BCIR) aims to reconstruct faithful images from corrupted compressed bitstreams. However, corruption occurs in bitstream-domain may lead to unpredictable error propagation and distortion amplification in pixel-domain, and even result in complete undecodability by standard decoders, posing significant challenge to existing image restoration paradigms. To address this challenge, we propose Bitstream Native Restorer (BiNar), a generative restoration framework that leverages fine-grained bitstream-native semantics to guide bitstream-corrupted image restoration. In BiNar, Bitstream Semantic Modeling (BSeM) extracts robust informative semantics and models sequential bytes directly from corrupted bitstream, enabling to generate fine-grained bitstream descriptions without relying on pixel decoding. Based on the descriptions,the Corruption-adaptive Multimodal Semantic Aggregation (CMSA) further extracts adaptive visual modal cues in decoded corrupted images and aggregates to bitstream descriptions. The multimodal aggregated fine-grained descriptions can serve as coherent restoration guidance. Finally, Blind Corruption Localization (BCoL) exploits the segmentation prior of visual foundation models to identify and locate error regions that requires recovery in corrupted images. The located error regions and the fine-grained descriptions jointly serve as visual prompt and language prompt of diffusion models image restoration, respectively. Extensive experiments under diverse bitstream corruption conditions show that BiNar outperforms all State-of-the-arts image restoration models regardless of perceptual quality or objective quality. Ablation studies demonstrate the effectiveness of fine-grained bitstream-native semantics in BCIR task.
This paper reports on the NTIRE 2026 Challenge on Bitstream-Corrupted Video Restoration (BSCVR). The challenge aims to advance research on recovering visually coherent videos from corrupted bitstreams, whose decoding often produces severe spatial-temporal artifacts and content distortion. Built upon recent progress in bitstream-corrupted video recovery, the challenge provides a common benchmark for evaluating restoration methods under realistic corruption settings. We describe the dataset, evaluation protocol, and participating methods, and summarize the final results and main technical trends. The challenge highlights the difficulty of this emerging task and provides useful insights for future research on robust video restoration under practical bitstream corruption.
Bitstream-corrupted video recovery aims to restore realistic content degraded during video storage or transmission. Existing methods typically assume that predefined masks of corrupted regions are available, but manually annotating these masks is labor-intensive and impractical in real-world scenarios. To address this limitation, we introduce a new blind video recovery setting that removes the reliance on predefined masks. This setting presents two major challenges: accurately identifying corrupted regions and recovering content from extensive and irregular degradations. We propose a Metadata-Guided Diffusion Model (M-GDM) to tackle these challenges. Specifically, intrinsic video metadata are leveraged as corruption indicators through a dual-stream metadata encoder that separately embeds motion vectors and frame types before fusing them into a unified representation. This representation interacts with corrupted latent features via cross-attention at each diffusion step. To preserve intact regions, we design a prior-driven mask predictor that generates pseudo masks using both metadata and diffusion priors, enabling the separation and recombination of intact and recovered regions through hard masking. To mitigate boundary artifacts caused by imperfect masks, a post-refinement module enhances consistency between intact and recovered regions. Extensive experiments demonstrate the effectiveness of our method and its superiority in blind video recovery. Code is available at: https://github.com/Shuyun-Wang/M-GDM.
Shuyun Wang, Hu Zhang, Xin Shen +2
The University of Queensland · Data61, CSIRO, Australia