Authors: Fangcheng Li, Chaoran Huang, Tianyi Liu, Wenyang Liu, Kejun Wu, Qiong Liu, You Yang, Zhengguo Li
Organizations: School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · School of Electrical and Electronics Engineering, Nanyang Technological University, Singapore · VI Department, Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR), Singapore
Conventional action recognition typically relies on successful pixel decoding of the bitstream. However, bitstream corruption during storage or transmission may cause severe visual artifacts or even decoding failure, posing a significant challenge to reliable action recognition. Bitstream Action Recognition (BAR) aims to overcome the dependency on decoding and the vulnerability to corruption. In this paper, we propose a novel BAR framework, Bitstream Recognition via Anchoring Corrupted Embeddings (BRACE). BRACE is a dual-branch byte-modeling architecture that treats a corrupted bitstream and its intact counterpart as two byte realizations of the same action. This guides the generation of rich and stable representations for robustness to corruption through Intact-Anchored Representation Alignment (IARA). The intact representation serves as a stable anchor, and the corrupted one is aligned to it at the embedding and decision levels under Unreliable-Anchor Suppression (UAS), entirely in representation space and without repairing the bitstream. To address the scarcity of corrupted bitstreams in practice, we introduce the Real-world Bitstream Corruption Simulator (RBCS), a four-parameter simulator that reproduces bit-flip and byte-loss errors arising in transmission and storage. Building on RBCS, we construct the first large-scale BAR dataset (BAR-D), which comprises the BAR-Stanford40 and BAR-PPMI subsets and spans diverse corruption types and severity levels. Finally, we build a large benchmark on BAR-D involving 14 action recognition methods from the pixel, compressed, and bitstream domains. Extensive experiments demonstrate that BRACE has superior robustness to bitstream corruption than all comparison methods. Ablation studies further validate the effectiveness of the proposed RBCS augmentation and IARA.
Figures & tables
Fig. 1 : Real-world corruption of image bitstream during transmission and storage may cause severe visual artifacts or decoding failure, which can hardly be solved by standard image codecs and further affect image action recognition. Bitstream Action Recognition (BAR) enables the direct understanding of corrupted bitstream, solving the challenge of dependency on decoding and vulnerability to corruption.
Fig. 2 : The illustration of our proposed Real-world Bitstream Corruption Simulator (RBCS). (a) The corruption pipeline of bitstream. Bitstream is partitioned into segments and each segment is independently corrupted by Bit-Flip or Byte-Loss. (b) RBCS applied to a real JPEG bitstream block. Red marks a bit-flipped byte and blue marks a lost byte.
Figure 3
Fig. 4 : Overview of our proposed BRACE framework. (a) BRACE pairs two byte-modeling branches. The intact branch is frozen and encodes the clean bitstream into anchor representations. The corrupted branch is trainable and encodes the corrupted bitstream. Intact-Anchored Representation Alignment (IARA) pulls the corrupted feature and prediction toward these anchors through embedding alignment (MSE loss) and decision alignment (KL loss). A cross-entropy loss on the ground-truth label trains the corrupted branch. (b) The bitstream encoder adopts the ByteFormer architecture [ 15 ] . It embeds the raw bytes and reduces the token count with a convolution. Positional encoding is then added before a stack of windowed self-attention blocks. These blocks merge tokens at selected depths to shorten the sequence. Mean pooling yields the final representation. (c) Unreliable-Anchor Suppression (UAS) aims to suppress unreliable intact samples whose confidence is below a threshold. IARA aligns only to reliable anchors.
Method
Domain
Clean
Light Avg
Medium Avg
Heavy Avg
Extreme Avg
Corrupt Avg
Acc
mAP
Acc
mAP
Acc
mAP
Acc
mAP
Acc
mAP
Acc
mAP
BAR-Stanford40
ResNet-50
Pixel
85.30
90.09
0.02
2.57
0.00
2.52
0.00
2.50
0.00
2.50
0.01
2.52
ViT-B/16
Pixel
83.33
86.23
0.00
2.59
0.00
2.51
0.00
2.50
0.00
2.50
0.00
2.52
Swin-Tiny
Pixel
85.52
90.08
0.01
2.58
0.00
2.52
0.00
2.50
0.00
2.50
0.00
2.52
ConvNeXt-Tiny
Pixel
87.55
92.66
0.02
2.58
0.00
2.52
0.00
2.50
0.00
2.50
0.00
2.52
TABLE II : Model Comparisons on BAR-Stanford40 and BAR-PPMI subsets. Accuracy (%) and mAP (%) are used for measuring the performance on Clean data and corrupted data at 4 severity levels. These metric values are computed by averaging the Flip, Loss, Mixed corruption. “Corrupt Avg” is the mean over all scenarios. The best results are in bold .
Fig. 5 : Per-scenario comparison of bitstream-domain methods on BAR-Stanford40 in terms of Accuracy (%) and mAP (%). Each axis represents one of the 12 corruption scenarios. Our BRACE outperforms all comparison methods across all scenarios.
Fig. 6 : Comparison of bitstream-domain methods by corruption type on BAR-Stanford40. The results report the average Accuracy and mAP for four severity levels on different corruption types. Our BRACE outperforms all comparison methods across all corruption types.
Fig. 7 : Illustration of Qualitative comparison on BAR-Stanford40. Each row is one image and each column represents its severity. The ConvNeXt-Tiny, the Transform-Bitstream Classifier (TBC), and our BRACE from three domains give their predictions. The correct predictions are in green and wrong or failed predictions are in red. Pixel-domain and compressed-domain methods fail once the bitstream decoding fails, while BRACE keeps recognizing the action from the raw bytes.
Fig. 8 : Ablation on the effectiveness of RBCS augmentation on BAR-Stanford40.
Components
BAR-Stanford40
BAR-PPMI
Lce
Lfeat
Lpred
UAS
Clean
Corrupt Avg
Clean
Corrupt Avg
Acc
mAP
Acc
mAP
Acc
mAP
Acc
mAP
✓
60.59
63.15
29.86
29.70
48.83
50.80
21.59
22.10
✓
✓
61.98
65.83
35.40
35.68
50.06
52.11
25.07
26.18
✓
✓
61.61
63.02
35.68
36.06
50.12
52.04
24.26
25.25
✓
✓
✓
62.02
65.97
37.23
37.65
50.36
52.31
25.80
26.94
TABLE III : Ablation on the training objectives of Intact-Anchored Representation Alignment (IARA). UAS denotes Unreliable-Anchor Suppression under γ=0.5 . All variants are trained with RBCS augmentation. Acc and mAP report Accuracy and mean Average Precision. Corrupt Avg is the result averaged over all corruption scenarios. These results validate the effectiveness of IARA components.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 9 : Bitstream length statistics on BAR-PPMI. (a) Distribution of bitstream lengths over all samples. (b) Average bitstream length per action class. The patterns match those on BAR-Stanford40 in the main text, and length offers no shortcut for recognition.
Fig. 10 : Ablation on the effectiveness of RBCS augmentation on BAR-PPMI.
Fig. 11 : Per-scenario comparison of bitstream-domain methods on BAR-PPMI in terms of Accuracy (%) and mAP (%). Each axis represents one of the 12 corruption scenarios. Our BRACE outperforms all comparison methods across all scenarios.
Fig. 12 : Comparison of bitstream-domain methods by corruption type on BAR-PPMI. The results report the average Accuracy and mAP for four severity levels on different corruption types. Our BRACE outperforms all comparison methods across all corruption types.
Byte-space Action Recognition (BAR) aims to recognize human actions directly from compressed image bitstreams without any pixel decoding. By operating entirely in byte space, BAR is inherently independent of file integrity and pixel-level reconstruction, making it naturally applicable to privacy-sensitive scenarios and robust against bitstream corruption. In this paper, we propose ByteAction, a BAR foundation model that achieves accurate action recognition on corrupted image bitstreams. ByteAction follows a dual-view byte-level recognition framework. It constructs weakly and strongly corrupted bitstream views, which are augmented by Bitstream Pattern Augmentation (BPA) and encoded with a shared ByteFormer backbone. The model is optimized with both classification and corruption consistency objectives. Specifically, we propose Bitstream Pattern Augmentation (BPA), which reshapes one-dimensional byte sequences into two-dimensional byte matrix and applies region-level erasure to encourage the model to learn robust cross-region byte dependencies. We further propose a Corruption Consistency Training strategy that constrains the model to maintain stable predictions across different corruption severities through bidirectional KL divergence. Experiments on the image bitstream from Stanford40, PPMI, and PASCAL VOC 2012 Action demonstrate that ByteAction achieves state-of-the-art corruption robustness across all scenarios while maintaining competitive intact bitstream performance.
Fangcheng Li, Zhen Yu, Kejun Wu +2
School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, 430074, China
Bitstream-corrupted Harsh Visual Understanding (BcHVU) aims to understand harshly degraded videos originally decoded from a severely corrupted bitstream in real-world multimedia communication. The ill-posed nature of BcHVU poses a major challenge for existing vision models, as even subtle bitstream corruption can lead to irreversible pixel distortion and significant semantic loss. To address these challenges in BcHVU, we propose Bitstream Language Modeling as Robust Semantic Priors (BLMSP), a framework for learning and injecting bitstream-native semantic cues. Our proposed BLMSP framework learns to extract bitstream-native semantic cues by bitstream language modeling, and leverages them as priors by injecting into off-the-shelf vision models of BcHVU tasks. Specifically, we present a Video Bitstream Byte Model (VBBM) that integrates byte-level modeling and cross-codec semantic distillation, enabling it to interpret robust semantics from byte sequences in multiple corrupted bitstream formats. The learned bitstream semantics are leveraged as robust priors and fused into BcHVU model backbones for improving the quality of video restoration, captioning, and human pose estimation. To train BLMSP, we construct a large-scale multi-source Corrupted-bitstream Harsh-video Paired (CHP) dataset containing 607k corrupted bitstream segments and 287k paired harsh video clips. Extensive experimental results show that the learned bitstream priors improve video restoration, captioning, and human pose estimation by 2.51 dB in PSNR, 0.20 in CIDEr, and 0.18 in [email protected] on average, respectively. These results demonstrate that corrupted bitstream can serve as robust semantic priors in solving pixel distortion and semantic loss in BcHVU.
Bitstream-corrupted image restoration (BCIR) aims to reconstruct faithful images from corrupted compressed bitstreams. However, corruption occurs in bitstream-domain may lead to unpredictable error propagation and distortion amplification in pixel-domain, and even result in complete undecodability by standard decoders, posing significant challenge to existing image restoration paradigms. To address this challenge, we propose Bitstream Native Restorer (BiNar), a generative restoration framework that leverages fine-grained bitstream-native semantics to guide bitstream-corrupted image restoration. In BiNar, Bitstream Semantic Modeling (BSeM) extracts robust informative semantics and models sequential bytes directly from corrupted bitstream, enabling to generate fine-grained bitstream descriptions without relying on pixel decoding. Based on the descriptions,the Corruption-adaptive Multimodal Semantic Aggregation (CMSA) further extracts adaptive visual modal cues in decoded corrupted images and aggregates to bitstream descriptions. The multimodal aggregated fine-grained descriptions can serve as coherent restoration guidance. Finally, Blind Corruption Localization (BCoL) exploits the segmentation prior of visual foundation models to identify and locate error regions that requires recovery in corrupted images. The located error regions and the fine-grained descriptions jointly serve as visual prompt and language prompt of diffusion models image restoration, respectively. Extensive experiments under diverse bitstream corruption conditions show that BiNar outperforms all State-of-the-arts image restoration models regardless of perceptual quality or objective quality. Ablation studies demonstrate the effectiveness of fine-grained bitstream-native semantics in BCIR task.