Deep Learning-based Filtering for Video Coding: A Survey on Architectures, Algorithms, and Complexity Analysis
Organizations: Department of Computer and Electronics Convergence Engineering, Sunmoon University, Asan 31460, South Korea · Division of Artificial Intelligence Engineering/ICT Convergence Research Institute, Sookmyung Women’s University, Seoul 04310, South Korea
Abstract
As Ultra-High-Definition (UHD) displays and immersive media services become ubiquitous in the Internet of Things (IoT) and Consumer Electronics (CE) sectors, including 8K display and mobile devices, the demand for high-efficiency video coding is unprecedented. While Deep Learning-based Filtering (DLF) has emerged as a promising solution to mitigate compression artifacts inherent in standards like High Efficiency Video Coding (HEVC/H.265) and Versatile Video Coding (VVC/H.266), its deployment in CE devices is severely constrained by computational complexity, memory bandwidth, and power consumption. To bridge the gap between academic research and practical deployment, this paper presents a comprehensive, hardware-oriented survey of DLF techniques. We propose a systematic three-dimensional taxonomy classifying methods into (1) Integration Scheme within the Video Coding, (2) Coding Information Utilization, and (3) Network Design Strategy. Unlike prior reviews, this work critically analyzes the trade-offs between Rate-Distortion (RD) performance and hardware feasibility, highlighting the evolution from heavy, performance-oriented models to lightweight, hardware-friendly architectures targeting Neural Processing Units (NPUs). Furthermore, we incorporate the latest standardization activities from the Joint Video Experts Team (JVET) on Neural Network-based Video Coding (NNVC) to provide realistic guidelines. We also identify open challenges such as real-time inference latency and error propagation, providing a roadmap toward robust, low-power intelligent video coding in next-generation CE vision endpoints.
Figures & tables
| Ref. | Year | Std. / SW | Primary Technique |
|---|---|---|---|
| [ 9 ] | 2017 | HM-16.0 | Variable filter size, Residual learning |
| [ 10 ] | 2017 | x265 | Dynamic Metadata |
| [ 11 ] | 2017 | HM-16.0 | Deep CNN (20 layers) |
| [ 12 ] | 2017 | HM-16.0 | Scalable Architecture |
| [ 13 ] | 2018 | HEVC | Global Residual Learning |
| [ 14 ] | 2018 | HM-16.0 | Input: CU Partition Mask |
| Domain | Input Signal/Prior | Scheme Usage | Granularity | Availability Mode | Signaling Overhead | ||
|---|---|---|---|---|---|---|---|
| ILF | FIF | PPF | |||||
| Pixel | Reconstructed | Frame/Patch | Implicit(Recon.) | None(Derived) | |||
| Prediction Signal | Block(PU) | Implicit(Inter/Intra) | None(Derived) | ||||
| Coding | QP | Frame/Slice/CU | Implicit(Parsed) | None(But Parsing Dep.) | |||
| Partition | CTU/Block | Implicit(Parsed) | None(But Parsing Dep.) | ||||
| Prediction Mode(I/P/B) | Block(PU) | Implicit(Parsed) | None(But Parsing Dep.) | ||||
| Category | Constraint / Metric | Target Requirement (Standardization) | Engineering Implication |
|---|---|---|---|
| Implementation & Determinism | Fixed-point Support | Integer-only Inference (e.g., INT8/INT16) Floating-point operations are strictly prohibited. | Ensures bit-exact consistency across different CPU/GPU/NPU platforms. |
| Operation Set | Limited Ops (Add, Shift, Clip, Mul) Avoid complex activations (e.g., Sigmoid, Tanh). | Reduces hardware logic gate count and latency; Implementation for fixed-function hardware. | |
| Computational Complexity | MACs / Pixel | Bounded Complexity (e.g., 1k 2k MACs) Strict worst-case guarantee required. | Determines real-time decoding feasibility (e.g., 4K@60fps) within power budget. |
| Parameter Count | Compact Models (e.g., 64k parameters) Or shared weights across components. | Minimizes on-chip SRAM usage and external DRAM bandwidth for weight fetching. | |
| Memory & Bandwidth | Line Buffer | Minimized Vertical Receptive Field Avoid full-frame buffering. | Critical for pipelined hardware architecture; Affects chip die size directly. |
| Granularity | Block/CTU-level Processing Independence between blocks preferred. | Enables parallel processing and reduces dependency-induced latency. |
Video Compression
Video compression reduces the bits needed to store or transmit moving images while balancing visual quality, computational cost, and downstream usefulness.
See all 80 papers