Bitstream Action Recognition is Byte Modeling

Fangcheng Li; Chaoran Huang; Tianyi Liu; Wenyang Liu; Kejun Wu; Qiong Liu; You Yang; Zhengguo Li

cs.CVAug 16, 2026

Bitstream Action Recognition is Byte Modeling

Authors: Fangcheng Li, Chaoran Huang, Tianyi Liu, Wenyang Liu, Kejun Wu, Qiong Liu, You Yang, Zhengguo Li

Organizations: School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · School of Electrical and Electronics Engineering, Nanyang Technological University, Singapore · VI Department, Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR), Singapore

Abstract

Conventional action recognition typically relies on successful pixel decoding of the bitstream. However, bitstream corruption during storage or transmission may cause severe visual artifacts or even decoding failure, posing a significant challenge to reliable action recognition. Bitstream Action Recognition (BAR) aims to overcome the dependency on decoding and the vulnerability to corruption. In this paper, we propose a novel BAR framework, Bitstream Recognition via Anchoring Corrupted Embeddings (BRACE). BRACE is a dual-branch byte-modeling architecture that treats a corrupted bitstream and its intact counterpart as two byte realizations of the same action. This guides the generation of rich and stable representations for robustness to corruption through Intact-Anchored Representation Alignment (IARA). The intact representation serves as a stable anchor, and the corrupted one is aligned to it at the embedding and decision levels under Unreliable-Anchor Suppression (UAS), entirely in representation space and without repairing the bitstream. To address the scarcity of corrupted bitstreams in practice, we introduce the Real-world Bitstream Corruption Simulator (RBCS), a four-parameter simulator that reproduces bit-flip and byte-loss errors arising in transmission and storage. Building on RBCS, we construct the first large-scale BAR dataset (BAR-D), which comprises the BAR-Stanford40 and BAR-PPMI subsets and spans diverse corruption types and severity levels. Finally, we build a large benchmark on BAR-D involving 14 action recognition methods from the pixel, compressed, and bitstream domains. Extensive experiments demonstrate that BRACE has superior robustness to bitstream corruption than all comparison methods. Ablation studies further validate the effectiveness of the proposed RBCS augmentation and IARA.

Figures & tables

Fig. 1 : Real-world corruption of image bitstream during transmission and storage may cause severe visual artifacts or decoding failure, which can hardly be solved by standard image codecs and further affect image action recognition. Bitstream Action Recognition (BAR) enables the direct understanding of corrupted bitstream, solving the challenge of dependency on decoding and vulnerability to corruption.

Panel of Figure 2 — Fig. 2 : The illustration of our proposed Real-world Bitstream Corruption Simulator (RBCS). (a) The corruption pipeline of bitstream. Bitstream is partitioned into segments and each segment is independently corrupted by Bit-Flip or Byte-Loss. (b) RBCS applied to a real JPEG bitstream block. Red marks a bit-flipped byte and blue marks a lost byte.

Fig. 4 : Overview of our proposed BRACE framework. (a) BRACE pairs two byte-modeling branches. The intact branch is frozen and encodes the clean bitstream into anchor representations. The corrupted branch is trainable and encodes the corrupted bitstream. Intact-Anchored Representation Alignment (IARA) pulls the corrupted feature and prediction toward these anchors through embedding alignment (MSE loss) and decision alignment (KL loss). A cross-entropy loss on the ground-truth label trains the corrupted branch. (b) The bitstream encoder adopts the ByteFormer architecture [ 15 ] . It embeds the raw bytes and reduces the token count with a convolution. Positional encoding is then added before a stack of windowed self-attention blocks. These blocks merge tokens at selected depths to shorten the sequence. Mean pooling yields the final representation. (c) Unreliable-Anchor Suppression (UAS) aims to suppress unreliable intact samples whose confidence is below a threshold. IARA aligns only to reliable anchors.

BAR-Stanford40
Method	Domain	Clean		Light Avg		Medium Avg		Heavy Avg		Extreme Avg		Corrupt Avg
Method	Domain	Acc	mAP	Acc	mAP	Acc	mAP	Acc	mAP	Acc	mAP	Acc	mAP
ResNet-50	Pixel	85.30	90.09	0.02	2.57	0.00	2.52	0.00	2.50	0.00	2.50	0.01	2.52
ViT-B/16	Pixel	83.33	86.23	0.00	2.59	0.00	2.51	0.00	2.50	0.00	2.50	0.00	2.52
Swin-Tiny	Pixel	85.52	90.08	0.01	2.58	0.00	2.52	0.00	2.50	0.00	2.50	0.00	2.52
ConvNeXt-Tiny	Pixel	87.55	92.66	0.02	2.58	0.00	2.52	0.00	2.50	0.00	2.50	0.00	2.52

TABLE II : Model Comparisons on BAR-Stanford40 and BAR-PPMI subsets. Accuracy (%) and mAP (%) are used for measuring the performance on Clean data and corrupted data at 4 severity levels. These metric values are computed by averaging the Flip, Loss, Mixed corruption. “Corrupt Avg” is the mean over all scenarios. The best results are in bold .

Fig. 5 : Per-scenario comparison of bitstream-domain methods on BAR-Stanford40 in terms of Accuracy (%) and mAP (%). Each axis represents one of the 12 corruption scenarios. Our BRACE outperforms all comparison methods across all scenarios.

Fig. 6 : Comparison of bitstream-domain methods by corruption type on BAR-Stanford40. The results report the average Accuracy and mAP for four severity levels on different corruption types. Our BRACE outperforms all comparison methods across all corruption types.

Fig. 7 : Illustration of Qualitative comparison on BAR-Stanford40. Each row is one image and each column represents its severity. The ConvNeXt-Tiny, the Transform-Bitstream Classifier (TBC), and our BRACE from three domains give their predictions. The correct predictions are in green and wrong or failed predictions are in red. Pixel-domain and compressed-domain methods fail once the bitstream decoding fails, while BRACE keeps recognizing the action from the raw bytes.

Fig. 8 : Ablation on the effectiveness of RBCS augmentation on BAR-Stanford40.

Components				BAR-Stanford40				BAR-PPMI
$\mathcal{L}_{ce}$	$\mathcal{L}_{feat}$	$\mathcal{L}_{pred}$	UAS	Clean		Corrupt Avg		Clean		Corrupt Avg
$\mathcal{L}_{ce}$	$\mathcal{L}_{feat}$	$\mathcal{L}_{pred}$	UAS	Acc	mAP	Acc	mAP	Acc	mAP	Acc	mAP
$\checkmark$				60.59	63.15	29.86	29.70	48.83	50.80	21.59	22.10
$\checkmark$	$\checkmark$			61.98	65.83	35.40	35.68	50.06	52.11	25.07	26.18
$\checkmark$		$\checkmark$		61.61	63.02	35.68	36.06	50.12	52.04	24.26	25.25
$\checkmark$	$\checkmark$	$\checkmark$		62.02	65.97	37.23	37.65	50.36	52.31	25.80	26.94

TABLE III : Ablation on the training objectives of Intact-Anchored Representation Alignment (IARA). UAS denotes Unreliable-Anchor Suppression under

\gamma=0.5

. All variants are trained with RBCS augmentation. Acc and mAP report Accuracy and mean Average Precision. Corrupt Avg is the result averaged over all corruption scenarios. These results validate the effectiveness of IARA components.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 11 — Fig. 9 : Bitstream length statistics on BAR-PPMI. (a) Distribution of bitstream lengths over all samples. (b) Average bitstream length per action class. The patterns match those on BAR-Stanford40 in the main text, and length offers no shortcut for recognition.

Fig. 10 : Ablation on the effectiveness of RBCS augmentation on BAR-PPMI.

Fig. 11 : Per-scenario comparison of bitstream-domain methods on BAR-PPMI in terms of Accuracy (%) and mAP (%). Each axis represents one of the 12 corruption scenarios. Our BRACE outperforms all comparison methods across all scenarios.

Fig. 12 : Comparison of bitstream-domain methods by corruption type on BAR-PPMI. The results report the average Accuracy and mAP for four severity levels on different corruption types. Our BRACE outperforms all comparison methods across all corruption types.

Video Action Recognition

Video action recognition identifies and classifies human or animal activities by learning from visual appearance, motion, and temporal context.

See all 111 papers

ByteAction: Byte-space Action Recognition Foundation Model
Aug 24, 2026Fangcheng Li, Zhen Yu, Kejun Wu +2Corruption Robustness Visual Representation Learning
School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, 430074, China
Towards Bitstream-corrupted Harsh Visual Understanding: Through Bitstream Language Modeling as Robust Semantic Priors
Aug 22, 2026Chaoran Huang, Fangcheng Li, Tianyi Liu +2Video Restoration Video Captioning
BiNar: Bitstream Native Restorer for Bitstream-corrupted Image Restoration
Oct 16, 2026Fangcheng Li, Zhen Yu, Kejun Wu +1Image Restoration Degradation-Aware Image Restoration

Aug 24, 2026J/K move · Enter open · S save

ByteAction: Byte-space Action Recognition Foundation Model

Fangcheng Li, Zhen Yu, Kejun Wu, Qiong Liu+1

School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, 430074, China

Corruption Robustness Visual Representation Learning

Open paper AskPDF arXiv

Byte-space Action Recognition (BAR) aims to recognize human actions directly from compressed image bitstreams without any pixel decoding. By operating entirely in byte space, BAR is inherently independent of file integrity and pixel-level reconstruction, making it naturally applicable to privacy-sensitive scenarios and robust against bitstream corruption. In this paper, we propose ByteAction, a BAR foundation model that achieves accurate action recognition on corrupted image bitstreams. ByteAction follows a dual-view byte-level recognition framework. It constructs weakly and strongly corrupted bitstream views, which are augmented by Bitstream Pattern Augmentation (BPA) and encoded with a shared ByteFormer backbone. The model is optimized with both classification and corruption consistency objectives. Specifically, we propose Bitstream Pattern Augmentation (BPA), which reshapes one-dimensional byte sequences into two-dimensional byte matrix and applies region-level erasure to encourage the model to learn robust cross-region byte dependencies. We further propose a Corruption Consistency Training strategy that constrains the model to maintain stable predictions across different corruption severities through bidirectional KL divergence. Experiments on the image bitstream from Stanford40, PPMI, and PASCAL VOC 2012 Action demonstrate that ByteAction achieves state-of-the-art corruption robustness across all scenarios while maintaining competitive intact bitstream performance.

Abstract

Video Action Recognition

Explore similar work

ByteAction: Byte-space Action Recognition Foundation Model

Towards Bitstream-corrupted Harsh Visual Understanding: Through Bitstream Language Modeling as Robust Semantic Priors

BiNar: Bitstream Native Restorer for Bitstream-corrupted Image Restoration