Towards Seconds-level Video Representation: Quadratically Gated Feature Grid with Adaptive, Prediction-free Sampling
Authors: Tie-Yun Li, Hao-Ran Wang, Chao Wang, Zhiyong Yin, Yiming Xu, Zhenwei Cai, Li Li, Rongzhi Lin, Feng-Lei Fan
Organizations: School of Mathematics, Harbin Institute of Technology, Harbin 150001, China. · Frontier of Artificial Networks (FAN) Lab, Department of Data Science, City University of Hong Kong, Hong Kong, China SAR. · China Construction Third Engineering Bureau (Shenzhen) Co., Ltd., Shenzhen, Guangdong 518110, China. · MoE Key Laboratory of Brain-Inspired Intelligent Perception and Cognition, University of Science and Technology of China, Hefei 230027, China.
Implicit neural video representations are notoriously slow to build, which does not cross the basic practical threshold for most industrial applications. Here, we present Q-GAS, a coordinate-queryable video representation that couples multi-resolution explicit 3D feature grids with a compact SwiGLU-style quadratic-gating (QG) decoder and an error-aware, prediction-free sampler. The explicit grids avoid hash-collision coefficient sharing at the cost of modestly higher storage yet greatly boost the expressive capacity; the QG decoder introduces low-rank local second-order interactions to improve the nonlinear feature response. During fitting, a global error map periodically refreshes a high-error pool, and each batch mixes pool samples with uniform samples, which is free from the auxiliary model to predict difficult samples, thereby further escalating the efficiency. On UVG Beauty/Bosphorus/Jockey, Q-GAS fits and quantizes under 50 s on one RTX PRO 6000, decodes at over 370 FPS with mixed precision and about 146 FPS in FP32. In our PyTorch implementation, Q-GAS trains and quantizes 5.5-7.2 times faster than the evaluated Instant-NGP baseline and 19-28 times faster than RNeRV. Under a 90 s budget, it improves PSNR/MS-SSIM/LPIPS over the evaluated NeRV-family pipelines at comparable BPP over the measured overlap. We have shared our code at https://github.com/MillenRosen/Q-GAS for readers' free use.
Figures & tables
Fig. 1: An overview of Q-GAS. Multi-Resolution 3D Feature Grids: At each coordinate (x,y,t) , trilinear interpolation retrieves features. The schematic illustrates the multiscale construction. QG Decoder: A linear branch and a SiLU-gated branch quadratically interact through a Hadamard product, which serves as a compact RGB decoder. Error-aware Sampling: Reconstruction errors constitute the global error map during training, and Adam optimizes the representation with uniform coordinates and coordinates from the current high-error pool.
Fig. 2: Explicit and hash-constrained 3D feature grids at the same resolution and feature dimension. Both use trilinear interpolation. Feature grid: Each vertex has an independent feature vector. Hash encoding: Hash collisions force distinct vertices to share a feature vector, coupling their representations.
Fig. 3: Error-aware coordinate sampling. Coordinates from two branches are combined into a batch for training. One branch samples uniformly from the entire observed sequence, while the other draws from a pool of difficult coordinates constructed using reconstruction errors (dynamic sampling) or differences between adjacent frames (static sampling).
Method
Params (M)
Beauty Sequence
Bosphorus Sequence
PSNR ↑
MS-SSIM ↑
Time (s) ↓
BPP ↓
FPS ↑
PSNR ↑
MS-SSIM ↑
Time (s) ↓
BPP ↓
FPS ↑
H.265 [ 39 ]
0.000
33.30
0.8943
10.23
0.01849
143.03
36.58
0.9676
11.61
0.02054
150.91
Small-Model Regime ( ∼ 3M Parameters)
HNeRV [ 6 ]
3.001
31.69
0.8686
5378.78
0.01478
366.26
29.41
0.8758
4796.08
0.01474
360.52
HiNeRV [ 7 ]
3.173
33.36
0.8913
7345.86
0.01899
16.91
35.87
0.9629
7647.15
0.01919
16.92
FFNeRV [ 13 ]
3.033
32.38
0.8809
7265.61
0.01351
184.62
31.89
0.9245
6698.77
0.01398
181.83
TABLE I: Comparison of different models at approximately 3M and 5M parameters. Time includes training and quantization for neural methods and encoding for H.265. H.265 uses the medium preset at CRF 28. HNeRV uses 8-bit decoder weights and 6-bit embeddings. FFNeRV uses its 8-bit weight transform. The remaining neural methods use 8-bit quantization. Our variants use chunked decoding with AMP FP16. Bold-faced values indicate the better result between our static and dynamic variants.
Fig. 4: Rate–distortion performance after 90 s of training. Rows: Beauty and Bosphorus . Columns: PSNR, MS-SSIM, and LPIPS. Quality and BPP use the same 8-bit quantized model. Points show means, with horizontal and vertical error bars indicating two-sided 95% Student’s t confidence intervals.
Fig. 5: Quality versus decoding throughput after 90 s of training. Panel order and statistical summaries follow Figure 4 . The FPS axis is logarithmic, and size of marker area indicates mean BPP. Q-GAS methods use uncompiled FP32 full-frame decoding.
Fig. 6: Mean frame-wise PSNR after 8-bit quantization versus synchronized optimization time for approximately 3M- and 5M-parameter models. Each budget is evaluated independently over 5 runs. Points show means, and shaded bands show two-sided 95% Student’s t confidence intervals.
Fig. 7: Full-frame reconstructions of Bosphorus (frame 382) and corresponding enlarged regions after 90 s training, using 8-bit quantized models. Red and blue boxes mark regions A and B, which are enlarged in the second and third rows, respectively. For each method, we select the configuration with the lowest mean BPP at or above 0.03. Five baselines are then arranged from left to right in ascending order of full-frame PSNR.
Decoder
PSNR (dB) ↑
Time (s) ↓
GPU FPS ↑
Beauty
Linear
32.557±0.069
34.559±0.636
1486.6±2.1
FAN
32.673±0.112
20.353±0.625
631.7±0.2
QG
32.737±0.039
18.270±0.354
898.9±1.5
Bosphorus
Linear
33.339±0.180
34.736±1.162
1484.7±4.7
TABLE II: Comparison of compact decoders in models with approximately 11.40M parameters. Results are reported as means ± sample standard deviations over completed runs. The best mean for each metric within each sequence is bold-faced.
Fig. 8: Early fitting performance of the decoders in Table II over the first 1,000 updates. Markers show mean frame-wise PSNR every 50 updates. The two training windows use different PSNR ranges to reveal fitting differences.
Method
Params (M)
Time (s) ↓
FPS ↑
PSNR (dB) ↑
MS-SSIM ↑
Beauty
FG + MLP
5.05
32.48
400.43
32.39
0.8833
FG + QG
5.06
40.48
181.37
32.52
0.8841
Jockey
FG + MLP
5.05
32.91
407.66
28.20
0.8632
FG + QG
5.06
39.53
181.66
28.55
0.8688
TABLE III: Decoder comparison in the two-hidden-layer architecture with 64 neurons at each layer, using the same feature grids and uniform sampling. All variants use a common 20,000-step schedule and unquantized FP32 evaluation. FG denotes the 3D feature grid. Green shading highlights the QG configurations. The best result for each metric is bold-faced.
Fig. 9: Global video PSNR over 20,000 updates under the settings of Tables III and IV . Panels (a–b) show the full trajectory and late-stage detail for Beauty , and (c–d) show those for Jockey . The baseline uses uniform sampling with an MLP decoder, whereas + QG replaces the decoder with QG while retaining the same feature grids.
Method
Params (M)
Time (s) ↓
FPS ↑
PSNR (dB) ↑
MS-SSIM ↑
Beauty
FG + MLP
5.05
32.48
400.43
32.39
0.8833
FG + Dynamic + MLP
5.05
36.82
405.86
32.49
0.8839
FG + Static + MLP
5.05
32.16
406.62
32.38
0.8829
FG + Dynamic + QG
5.06
42.29
181.22
32.59
0.8849
FG + Static + QG
5.06
39.06
180.51
32.52
0.8841
TABLE IV: Comparison of sampling strategies and their combinations with the QG decoder. Apart from the sampling strategies, the training and evaluation settings follow those in Table III . FG denotes 3D feature grid. Rows without Dynamic or Static use uniform sampling. Green shading highlights the QG configurations. The best result for each metric and sequence within each decoder group is bold-faced.
Fig. 10: Visual comparison on Beauty . Left: the ground-truth frame and enlarged eye region linked by red dashed lines. Right: reconstructed eye regions. FG denotes 3D feature grids, and D/S denote dynamic/static sampling. Configurations without D/S use uniform sampling. These examples are from earlier runs, separate from Tables III and IV .
Fig. 11: Dynamic error-aware sampling on Jockey . The latest error map E guides the prioritized samples in B2 toward high-error regions, including the horse and rider, while B1 provides uniform samples from the full sequence.
Gaussian Splatting has emerged as an effective representation for video, but existing methods rely on per-video optimization. This leads to slow encoding and limits generalization across videos. To amortize this optimization, we propose HyperGS, a feedforward, optimization-free approach that directly predicts Gaussian representations from any video in a single forward pass, speeding up encoding and decoding by orders of magnitude while generalizing to out-of-distribution videos at higher resolutions. In HyperGS, we design a factorized spatiotemporal Transformer to extract tokens from video, and a learnable query-based Transformer to obtain 8-parameter Gaussian representations for each video frame. We find that naively predicting Gaussians across diverse videos induces a needle-like degeneration that collapses training, and address this with a rank-based geometric regularizer whose strength adapts dynamically to stabilize optimization. HyperGS achieves encoding at 104--105× the speed of per-video Gaussian optimization at matched reconstruction quality while generalizing zero-shot to 720p video, enabling higher-resolution rendering without re-encoding. HyperGS improves PSNR by +2.9--3.1 dB over the prior video encoders on K400, SSv2, and UCF101 at a smaller video representation size. By predicting explicit 2D Gaussians in a single forward pass, HyperGS combines the fast, flexible rendering of Gaussian Splatting with the speed and generalization of feedforward prediction, advancing Gaussians as a practical direction for fast and generalizable video representation.
Fatimah Zohra, Chen Zhao, Shuming Liu +2
1King Abdullah University of Science and Technology (KAUST) · 2Harbin Institute of Technology, Shenzhen
Implicit Neural Representations (INR) provide a compact and content-adaptive paradigm for video compression, typically representing a video through shared network parameters and frame-indexed embeddings. Compared to conventional or autoencoder-based codecs, these approaches exploit temporal redundancy within videos in an implicit manner, which potentially results in sub-optimal compression performance. In this paper, we propose G-NeRV, a graph-inspired INR that explicitly improves temporal redundancy exploitation in the implicit latent space. Motivated by the total correlation principles in information theory, we construct a temporal neighborhood over frame embeddings and perform message passing to aggregate reusable information from neighboring frames through an adaptive gate controlling the injection of neighboring information. Inspired by the reference frame buffer in conventional video coding, a memory bank mechanism has been further designed to enable efficient temporal-neighbor retrieval under random frame-index sampling in INR training. This new representation model has been integrated into an advanced representation compression framework and compared with existing conventional and neural video codecs. The results show that the G-NeRV codec outperforms the state-of-the-art INR-based codec, NVRC, and the latest standard video codec, VVC VTM, by 8.86% and 14.68% (in BD-rate), respectively, measured by PSNR on the UVG dataset.
Changqi Wang, Ge Gao, Fan Zhang +4
University of Bristol · Bytedance Inc., San Diego, USA
2D Gaussian Splatting (2DGS) has recently become a promising paradigm for high-quality video representation. However, existing methods employ content-agnostic or spatio-temporal feature overlapping embeddings to predict canonical Gaussian primitive deformations, which entangles static and dynamic components in videos and prevents modeling their distinct properties effectively. These result in inaccurate predictions for spatio-temporal deformations and unsatisfactory representation quality. To address these problems, this paper proposes a Spatio-Temporal hash encoding framework for Gaussian-based Video representation (STGV). By decomposing video features into learnable 2D spatial and 3D temporal hash encodings, STGV effectively facilitates the learning of motion patterns for dynamic components while maintaining background details for static elements. In addition, we construct a more stable and consistent initial canonical Gaussian representation through a key frame canonical initialization strategy, preventing from feature overlapping and a structurally incoherent geometry representation. Experimental results demonstrate that our method attains better video representation quality (+0.98 PSNR) against other Gaussian-based methods and achieves competitive performance in downstream video tasks.
Jierun Lin, Jiacong Chen, Qingyu Mao +4
College of Electronics and Information Engineering, Shenzhen University · College of Applied Technology, Shenzhen University · School of Artificial Intelligence, Shenzhen Technology University +1