Reference-Efficient Multimodal Bitstream-Corrupted Video Recovery via Depth-Conditioned Adaptation
Authors: Tianyi Liu, Jiaojiao Wang, Ye Lu, Kejun Wu, Chengtao Cai, Zhengguo Li, Chia-Wen Lin
Organizations: Nanyang Technological University, Singapore · School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · College of Intelligent Systems Science and Engineering, Harbin Engineering University, Harbin 150001, China · VI Department, Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR), Singapore · Department of Electrical Engineering and the Institute of Communications Engineering, National Tsing Hua University, Hsinchu 30013, Taiwan
Recovering visually faithful content from damaged video bitstreams is important for reliable multimedia communication and storage. Existing methods exploit local residuals and non-local RGB references of the input video sequence, but both sources can become unreliable after decoding, while processing additional references increases receiver computation and memory. This paper studies multimodal bitstream-corrupted video recovery (MBSCVR) using RGB and depth information, where pre-transmission depth-based geometry aids reconstruction when the received appearance is corrupted. We propose a novel MBSCVR framework that selects and effectively utilizes a single selected non-local RGB reference per recovery window to achieve higher performance. It integrates a reliability-aware multimodal selector that combines valid appearance with depth compatibility before reference encoding. Depth-Conditioned Reference Adaptation (DCRA) introduces selected RGB evidence into feature completion and content generation, while frame-aligned depth conditions completion queries. On the complete BSCV benchmark with oracle masks, MBSCVR uses one selected reference to achieve 32.30 dB on YouTube-VOS and 28.93 dB on DAVIS, surpassing full-reference B2SCVR by 0.27 and 0.33 dB and improving SSIM on both subsets. Matched receiver-forward measurements show 36.8% lower latency and 40.8% lower peak memory. These results demonstrate how compact geometric guidance and adapted appearance processing improve the quality--resource trade-off of multimedia recovery.
Figures & tables
Fig. 1: Motivation and overview of MBSCVR. (a) Auxiliary depth obtained before RGB video corruption is independently coded and delivered with stronger protection, then aligned with the received RGB frames by media time. This provides geometric evidence when local appearance is damaged. (b) RGB-only recovery draws on multiple references to enrich appearance context, increasing reference encoding and temporal interaction costs. (c) MBSCVR uses geometry to select and condition a compact set of RGB references. RAMRS selects relevant appearance using valid RGB context, depth compatibility, and reference reliability, while DCRA introduces the selected appearance and geometric guidance into recovery.
Fig. 2: Architecture of MBSCVR, illustrated with a single selected RGB reference ( K=1 ). RAMRS ranks candidates before reference encoding. DCRA integrates the selected RGB features through a cross-stage bridge and conditions feature completion through frame-aligned depth queries. The bridge produces a completion residual Δtc before feature completion and a generation residual Δtg after flow-guided propagation. Selected RGB features also retain their direct path to content generation. The depth branch preserves temporal alignment and matches the RGB attention windows, producing ΔQ=αqPq(Z) through a rank-four projection. The conditioned query is Q′=Q(F)+ΔQ , while keys and values remain RGB-derived. DCRA interfaces and the completion module are jointly adapted; snowflakes indicate frozen modules. The attention diagram omits the scaling factor and bias term B for clarity.
Fig. 3: Cross-stage RGB reference bridge within DCRA. Nonnegative spatial weights wtj combine selection evidence, reliability, and visibility to aggregate selected reference features into a normalized RGB prior Pt . The support map St controls how this prior enters the bridge input Ut , concentrating reference guidance on supported damaged regions. A shared lightweight trunk and two zero-initialized output heads produce separate residuals for feature completion and content generation. Here, Hc and Hg each denote the shared trunk followed by the corresponding output head. Summations run over selected references j∈RK , and the spatial coordinate p is omitted for clarity.
YouTube-VOS
DAVIS
Method
Venue
Refs
PSNR ↑
SSIM ↑
LPIPS ↓
VFID ↓
PSNR ↑
SSIM ↑
LPIPS ↓
VFID ↓
STTN [ 9 ]
ECCV’20
Multiple
29.30
0.9150
0.0472
0.0370
26.21
0.8603
0.0634
0.1868
STTN ∗ [ 9 ]
29.94
0.9283
0.0420
0.0350
26.61
0.8731
0.0584
0.1808
FuseFormer [ 10 ]
ICCV’21
28.74
0.9010
0.0556
0.0411
26.19
0.8592
0.0662
0.1933
FuseFormer ∗ [ 10 ]
29.77
0.9284
0.0421
0.0351
26.71
0.8801
0.0557
0.1761
E2FGVI-HQ [ 11 ]
CVPR’22
29.69
0.9228
0.0469
0.0363
26.79
0.8765
0.0600
0.1806
TABLE I: Complete-test-set BSCV comparison on YouTube-VOS and DAVIS. ∗ denotes BSCV retraining. Bold indicates the best result within the single-reference group.
Fig. 4: Qualitative comparison on representative BSCV DAVIS examples. From left to right: corrupted input with the corruption mask overlaid, the RGB reference selected by RAMRS, BSCVR with multiple references, B2SCVR with multiple references, B2SCVR with a single reference, MBSCVR with a single reference, and ground truth. Highlighted boxes and enlarged insets show a common region within each example. MBSCVR combines selected appearance with geometric guidance to recover structural boundaries and local details under a single-reference budget.
Reference budget
Corr. PSNR ↑
Corr. SSIM ↑
K=1
21.3143
0.67805
K=3
21.3797
0.68054
TABLE II: Reference-budget comparison on the 125-video DAVIS component set using the same MBSCVR checkpoint. A budget of three uses all available candidates when fewer than three are present.
Configuration
Refs
ms
MiB
TFLOPs
B2SCVR Full
9
431.7
3202.3
4.265
B2SCVR K=1
1
270.0
1886.6
2.405
MBSCVR K=1
1
272.9
1895.8
2.409
MBSCVR K=3
3
313.4
2086.2
2.845
TABLE III: Receiver-forward measurements on one shared window with six local frames, using float32 on an RTX 3090, two warmups, and ten repeats. Columns report median latency, peak allocated memory, and FLOPs for supported profiler operators. Latency includes device preparation and output transfer.
Configuration
Corr. PSNR ↑
Corr. SSIM ↑
A. Reference selection
Fixed stride
21.1716
0.67417
RAMRS
21.3143
0.67805
B. DCRA optimization
No DCRA
20.6456
0.65272
Interfaces only
20.9957
0.66107
TABLE IV: Reference selection and DCRA optimization on the DAVIS subset. Panel A uses the same trained model, candidate pool, and single-reference budget. Panel B compares gains from a basic no-DCRA pipeline, with DCRA but interface only variant, and joint interface–completion optimized version.