CoANeRV: Coordinate-Aware Token-Space Neural Video Representation

Jialong Guo; Ke Liu; Mengxuan Li; Jiajun Bu; Haishuai Wang

cs.CVAug 14, 2026

CoANeRV: Coordinate-Aware Token-Space Neural Video Representation

Authors: Jialong Guo, Ke Liu, Mengxuan Li, Jiajun Bu, Haishuai Wang

Organizations: Zhejiang Key Laboratory of Accessible Perception and Intelligent Systems College of Computer Science, Zhejiang University, China

Abstract

Neural representations for videos (NeRV) have shown strong reconstruction fidelity by storing video-specific information in network weights. However, existing formulations typically require either costly per-video optimization or video-specific weight generation, making it difficult to scale to efficient amortized video representation. We propose CoANeRV, a coordinate-aware token-space framework that adapts the broader token-conditioned neural-field paradigm to amortized video representation. CoANeRV forms compact video tokens in one feed-forward pass and uses a shared coordinate-conditioned decoder to reconstruct continuous spatio-temporal queries, avoiding per-video decoder optimization or generation while retaining coordinate-level reconstruction flexibility. To make token-space reconstruction effective, CoANeRV introduces a coordinate-aware decoding architecture that aligns spatio-temporal queries with video tokens through axis-adaptive positional encoding and temperature-modulated cross-attention. Block-wise coordinate querying further reduces peak attention memory, making high-resolution reconstruction practical. Experiments on diverse video datasets show that CoANeRV consistently improves reconstruction quality over prior feed-forward NeRV and INR baselines, reduces peak memory compared with attention-based coordinate decoders, and provides efficient amortized encoding without per-video optimization. These results support the proposed video-specific combination of feed-forward token formation, spatio-temporal coordinate retrieval, and memory-bounded dense querying. The code is available at https://github.com/jialong2023/CoANeRV.

Figures & tables

Figure 1: Framework comparison and encoding speed. (a) Representative weight-space video approaches store instance-specific information in optimized, generated, or modulated decoder parameters, whereas CoANeRV stores video-specific information in compact tokens and reconstructs pixels with a shared coordinate-conditioned decoder. (b) CoANeRV enables feed-forward token inference, while the decoder-only variant freezes the shared decoder and optimizes only video tokens to isolate the cost of per-video token fitting.

Figure 2: CoANeRV framework. CoANeRV encodes each video through convolutional patch tokenization and a six-layer token former, shown as the “Transformer Encoder Block” in the schematic, and reconstructs pixels by querying the resulting compact video tokens with continuous spatio-temporal coordinates. The decoder is shared across videos, so video-specificity is stored in token space rather than decoder weight space.

Methods	F	Encoder size	INR size $\downarrow$	$\#\hat{\theta}^{\prime}$ $\downarrow$	Epoch	GPU hrs $\downarrow$	PSNR $\uparrow$				SSIM $\uparrow$
Methods	F	Encoder size	INR size $\downarrow$	$\#\hat{\theta}^{\prime}$ $\downarrow$	Epoch	GPU hrs $\downarrow$	Train	K400	SthV2	UCF101	Train	K400	SthV2	UCF101
TransINR Chen and Wang (2022)	4	48.0M	99k	25k	150	63	23.7	22.1	24.6	22.1	0.659	0.631	0.728	0.622
GINR Kim et al. (2023)	4	47.6M	139.4k	25.6k	150	65	24.5	23.2	25.9	23.1	0.685	0.66	0.744	0.66
FastNeRV Chen et al. (2024)	4	47.6M	85.6k	24.1k	150	9	26.6	26.6	29.4	26.1	0.756	0.754	0.816	0.752
CoANeRV(ours)	4	45.5M	86.4k	27k	20	5	31.4	31.5	31.9	31.5	0.924	0.922	0.925	0.923
CoANeRV(ours)	4	45.5M	86.4k	27k	150	39	35.3	34.8	36.0	35.8	0.955	0.951	0.956	0.955

Table 1: CoANeRV vs. feed-forward INR baselines. ‘F’ denotes the number of frames. “INR size” reports the size of the reconstruction network, while

\#\hat{\theta}^{\prime}

reports the video-specific representation size: generated video-specific weights for prior methods and video tokens for CoANeRV. For each frame setting, we report two CoANeRV rows: a 20-epoch checkpoint to highlight convergence speed, and the full 150-epoch result for final accuracy. Training time is measured in GPU hours.

Figure 3: Qualitative comparison. Visualizations for feed-forward INR methods: TransINR Chen and Wang (2022) (top row), GINR Kim et al. (2023) (second row), FastNeRV Chen et al. (2024) (third row), and CoANeRV (bottom row, ours). CoANeRV reconstructs finer details and cleaner structures. Best viewed digitally and zoomed in.

Methods	F	Token length $N$	$\#\hat{\theta}^{\prime}$	Epoch	Peak memory / batch (GB) $\downarrow$	GPU hrs $\downarrow$	PSNR $\uparrow$				SSIM $\uparrow$
Methods	F	Token length $N$	$\#\hat{\theta}^{\prime}$	Epoch	Peak memory / batch (GB) $\downarrow$	GPU hrs $\downarrow$	Train	K400	SthV2	UCF101	Train	K400	SthV2	UCF101
ANR-V-S	4	384	27k	150	18	75	31.9	29.9	32.0	31.0	0.933	0.919	0.934	0.929
CoANeRV-S	4	384	27k	150	2.6	39	35.3	34.8	36.0	35.8	0.955	0.951	0.956	0.955
ANR-V-M	4	512	36k	150	23	85	32.2	30.8	33.1	32.2	0.942	0.930	0.942	0.939
CoANeRV-M	4	512	36k	150	2.7	41	35.8	35.2	35.6	35.4	0.960	0.952	0.962	0.958
ANR-V-L	4	768	54k	150	34	132	33.6	31.2	33.4	32.9	0.947	0.934	0.946	0.946

Table 2: Comparison with video-adapted ANR. ANR-V extends ANR Zhang et al. (2024) from spatial queries

(x,y)

to video queries

(x,y,t)

using localized attention layers. CoANeRV uses temperature-modulated cross-attention with block-wise coordinate querying. Both methods are compared under matched token lengths and video-specific representation sizes. Memory is the recorded peak GPU footprint for one minibatch and is reported in GB, using

1\,\mathrm{GB}=1024\,\mathrm{MiB}

throughout the paper.

RGB to YUV	Block Query	Convolution Tokenizer	Axis-Adaptive- Embedding	Temperature-scaled fused SDPA	Peak memory / batch (GB) $\downarrow$	GPU hrs $\downarrow$	PSNR $\uparrow$				SSIM $\uparrow$
RGB to YUV	Block Query	Convolution Tokenizer	Axis-Adaptive- Embedding	Temperature-scaled fused SDPA	Peak memory / batch (GB) $\downarrow$	GPU hrs $\downarrow$	Train	K400	SthV2	UCF101	Train	K400	SthV2	UCF101
					35	178	26.9	25.3	26.3	26.2	0.668	0.579	0.612	0.602
✓					35	178	27.8	26.5	27.4	27.2	0.726	0.693	0.708	0.701
✓	✓				18	178	27.8	26.5	27.4	27.2	0.726	0.693	0.708	0.701
✓	✓	✓			18	172	28.2	27.7	28.4	28.5	0.821	0.793	0.813	0.802
✓	✓		✓		18	180	28.8	27.8	28.5	28.8	0.825	0.804	0.816	0.812

Table 3: Ablation study. Progressive analysis from an ANR-V/LAL attention decoder baseline. The temperature-attention column denotes temperature-scaled standard attention executed through fused SDPA; the reported peak memory reflects the full decoding path, including spatial block querying where enabled. All memory entries are the recorded peak GPU footprint for one minibatch and are reported in GB, using

1\,\mathrm{GB}=1024\,\mathrm{MiB}

Figure 4: Attention pattern analysis. Token probability distributions for (a) pre-training and (b) post-training at the same coordinate, and (c)–(d) post-training at different coordinates from the same video. The figure illustrates the emergence of sharper and more coordinate-specific attention patterns.

	AV1 CRF 60	H.264			FastNeRV					CoANeRV (ours)
	AV1 CRF 60	CRF 35	CRF 40	CRF 45	8 bits	7 bits	6 bits	5 bits	4 bits	8 bits	7 bits	6 bits	5 bits	4 bits
Size(KB) $\downarrow$	21.9	20.4	13.1	8.7	23.7	20.7	17.7	14.7	11.6	26.9	23.6	20.2	16.9	13.5
PSNR $\uparrow$	32.4	32.8	30.0	27.3	28.4	28.3	28.1	27.5	25.6	33.0	32.9	32.5	31.2	27.8
SSIM $\uparrow$	0.910	0.912	0.860	0.788	0.808	0.807	0.802	0.784	0.712	0.938	0.937	0.933	0.919	0.866
VPS $\uparrow$	313	447	460	485	583					125

Table 4: Per-video representation-storage comparison. Rate-distortion comparison with traditional codecs and FastNeRV under a simple per-video storage protocol. For CoANeRV and FastNeRV, size reports the quantized method-specific video payload—video tokens for CoANeRV and generated video-specific parameters for FastNeRV—while excluding shared or video-agnostic weights amortized across a video collection. VPS means videos per second.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Method	New-instance path	Query domain	Retrieval/decoder	Video-specific distinction
LAG-INR	Feed-forward Transformer tokens	2D coordinates; 6D rays	Local aggregation and multi-band modulation	Temporal modeling and exact dense-video tiling not evaluated
Tokenized Neural Fields	Gradient-fitted tokens; frozen decoder	1D/2D coordinates; 6D rays	Standard cross-attention and shared MLP	No amortized encoder or evaluated temporal axis
LH-NeF	Hierarchical feed-forward tokenizer	General metric domains	Group routing, Gaussian cross-attention, and FiLM	No evaluated video temporal modeling
CoANeRV	Feed-forward video tokenizer and token former	Continuous $(x,y,t)$	Temperature-scaled retrieval and shared MLP	Axis-adaptive encoding and exact full-token-bank video tiling

Appendix

Table 5: Technical positioning relative to closely related token-conditioned neural fields. “Not evaluated” describes the scope of the cited paper and does not assert that the method could not be extended.

Stage	Shape	Example
Input video	$T\times C\times H\times W$	$4\times 3\times 256\times 256$
After Conv2D	$T\times D\times H^{\prime}\times W^{\prime}$	$4\times 768\times 16\times 16$
Patch tokens	$N_{p}\times D$	$1024\times 768$
Video tokens	$N\times d$	$384\times 72$

Appendix

Table 6: Data shapes in the convolutional video tokenizer.

Figure 5: Encoding time comparison between CoANeRV and gradient-based NeRV optimization.

Method	Frames	Iter.	Time (s)	PSNR $\uparrow$			SSIM $\uparrow$
Method	Frames	Iter.	Time (s)	K400	SthV2	UCF101	K400	SthV2	UCF101
NeRV	8	300	122	24.78	24.13	24.75	0.73	0.70	0.72
NeRV	8	600	230	27.55	27.25	27.11	0.81	0.80	0.79
NeRV	8	900	373	29.82	29.89	29.29	0.85	0.87	0.86
NeRV	8	2400	942	33.64	33.73	33.17	0.90	0.91	0.89
CoANeRV	8	300	254	24.94	24.91	24.89	0.78	0.77	0.75

Appendix

Table 7: Standalone token optimization analysis under the 8-frame setting. For CoANeRV, the decoder is frozen and only randomly initialized video tokens are optimized for each test video. Time denotes per-video optimization time.

Figure 6: Temperature Coefficient. PSNR/SSIM results across training and test sets for different $\tau$ values. Both metrics peak at $\tau=0.4$ , indicating optimal attention sharpness. Lower $\tau$ causes over-concentration; higher $\tau$ leads to diffuse attention.

Figure 7: Attention Weight Distribution with Different Temperature Parameters. Distribution of attention weights when querying spatiotemporal coordinates through video tokens before training. Red curve ( $\tau=0.4$ ) shows a sharper distribution over tokens, while blue curve ( $\tau=1.0$ ) exhibits smoother and more dispersed attention weights.

Figure 8: RGB vs. YUV distributions. RGB channels occupy the full normalized range, whereas the YUV chrominance channels are strongly concentrated around $0.5$ .

Method	Frames	MS-SSIM $\uparrow$			LPIPS $\downarrow$
Method	Frames	K400	SthV2	UCF101	K400	SthV2	UCF101
TransINR	4	0.787	0.868	0.843	0.565	0.361	0.411
GINR	4	0.829	0.895	0.876	0.494	0.304	0.331
FastNeRV	4	0.921	0.954	0.941	0.267	0.139	0.167
CoANeRV	4	0.981	0.983	0.984	0.052	0.044	0.034

Appendix

Table 8: Perceptual reconstruction quality under the 4-frame setting. CoANeRV consistently improves MS-SSIM and reduces LPIPS across all evaluated datasets.

Method	FLOPs (G)	MACs (G)	Architecture Type
NeRV	1.30	0.65	Convolutional
ANR-V	239.73	119.45	LAL-based attention
CoANeRV (Ours)	31.92	15.95	Temperature-modulated attention

Appendix

Table 9: Computational complexity comparison for single-frame reconstruction. FLOPs and MACs are measured in billions (G).

Figure 9: Sigma parameter. Reconstruction performance across training and test sets under different $\sigma$ values.

Figure 10: Encoder depth. Performance improves as the encoder depth increases.

Figure 11: MLP depth. A deeper decoder improves the non-linear coordinate-to-signal mapping.

Figure 12: Cross-domain qualitative comparison. Ground truth (top), FastNeRV (second row), ANR-V (third row), and CoANeRV (bottom).

Dataset	Method	PSNR (dB) $\uparrow$	SSIM $\uparrow$	PSNR Gain	SSIM Gain
EchoCP	FastNeRV	20.53	0.607	-	-
	ANR-V	24.40	0.784	+3.87	+0.177
	CoANeRV	26.85	0.864	+6.32	+0.257
EchoNet-LVH	FastNeRV	22.88	0.684	-	-
	ANR-V	27.45	0.835	+4.57	+0.151
	CoANeRV	29.64	0.916	+6.76	+0.232

Appendix

Table 10: Comparative performance on medical imaging datasets.

Model A100 A6000 4090 2080Ti DCVC-DC 3.3 / 4.3 1.7 / 2.2 2.3 / 2.9 0.8 / 1.4 DCVC-FM 5.0 / 5.9 3.1 / 3.8 3.7 / 4.4 1.9 / 2.3 DCVC-RT 125.2 / 112.8 70.4 / 63.8 118.8 / 105.3 39.5 / 34.1 CoANeRV 247.0 / 7.7 168.4 / 6.5 236.8 / 7.4 94.6 / 3.4 (a) Coding speed on

1920\times 1080

videos.

Model A100 A6000 4090 2080Ti DCVC-DC 6.5 / 7.9 3.5 / 4.3 5.5 / 6.7 2.1 / 2.9 DCVC-FM 8.5 / 9.4 5.9 / 6.6 9.3 / 10.4 4.0 / 4.7 DCVC-RT 173.9 / 149.2 147.3 / 132.5 225.1 / 185.2 73.3 / 67.0 CoANeRV 250.0 / 12.7 202.3 / 8.5 240.6 / 12.4 98.5 / 5.4 (b) Coding speed on

1280\times 720

videos.

Appendix

Table 11: Speed analysis. Encoding / decoding speed (frames per second) across multiple devices.

Figure 13: Rate–distortion curves on HEVC Class D sequences.

Methods	F	Frame Resolution	$\#\hat{\theta}^{\prime}$	Epoch	Peak memory / batch (GB) $\downarrow$	GPU hrs $\downarrow$	PSNR $\uparrow$				SSIM $\uparrow$
Methods	F	Frame Resolution	$\#\hat{\theta}^{\prime}$	Epoch	Peak memory / batch (GB) $\downarrow$	GPU hrs $\downarrow$	Train	K400	SthV2	UCF101	Train	K400	SthV2	UCF101
ANR-V-S	4	512	27k	50	64	275	27.9	27.2	28.0	27.0	0.833	0.819	0.834	0.829
CoANeRV-S	4	512	27k	50	5.2	62	29.3	29.5	29.0	29.8	0.855	0.846	0.856	0.855
ANR-V-M	4	512	36k	50	76	285	28.2	27.8	28.1	28.2	0.842	0.830	0.842	0.839
CoANeRV-M	4	512	36k	50	5.3	66	29.8	29.7	29.6	29.9	0.860	0.852	0.860	0.860
ANR-V-L	4	512	54k	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM

Appendix

Table 12: CoANeRV vs. ANR-V with different resolutions. CoANeRV achieves better quality with much lower memory and training cost. Memory is the recorded peak GPU footprint for one minibatch in GB, using

1\,\mathrm{GB}=1024\,\mathrm{MiB}

Bits	PSNR $\uparrow$			SSIM $\uparrow$
Bits	K400	SthV2	UCF101	K400	SthV2	UCF101
32 (Full)	33.02	33.33	33.52	0.938	0.942	0.947
8	32.99	33.30	33.49	0.938	0.942	0.947
7	32.89	33.20	33.39	0.937	0.941	0.946
6	32.51	32.82	33.01	0.933	0.937	0.942
5	31.20	31.51	31.70	0.919	0.923	0.928

Appendix

Table 13: Ablation study on token quantization. Decoder weights remain in 32-bit floating point.

Parameter	Value
CoANeRV Architecture
Video-token dimension ( $d$ )	72
Video-token length ( $N$ )	384
Patch-token self-encoder depth	0
Number of token-former layers ( $L$ )	6
Token-former width ( $D$ )	768

Appendix

Table 14: Complete Hyperparameter Settings

Dataset split	Before filtering	4F	8F	16F
K400 train	10,000	10,000	10,000	9,997
K400 validation	13,732	13,732	13,732	13,731
Something-Something V2 evaluation	20,000	20,000	20,000	19,947
UCF101 evaluation	3,500	3,500	3,500	3,500

Appendix

Table 15: Dataset split sizes before and after deterministic minimum-frame filtering.

Video Representation Learning

Video representation learning develops features that capture appearance, motion, and temporal structure for recognition, prediction, and control.

See all 143 papers

TeCoNeRV: Leveraging Temporal Coherence for Compressible Neural Representations for Videos
Feb 18, 2026Namitha Padmanabhan, Matthew Gwilliam, Abhinav ShrivastavaNeural Video Compression Implicit Neural Representations
Department of Computer Science, University of Maryland, College Park, USA.
TinyNeRV: Compact Neural Video Representations via Capacity Scaling, Distillation, and Low-Precision Inference
Apr 10, 2026Muhammad Hannan Akhtar, Ihab Amer, Tamer ShanablehNeural Video Compression Neural Network Compression
Department of Computer Science & Engineering, American University of Sharjah, Sharjah, United Arab Emirates
LRConv-NeRV: Low Rank Convolution for Efficient Neural Video Compression
Mar 18, 2026Tamer ShanablehVideo Representation Learning Neural Video Compression
American University of Sharjah, UAE, Department of Computer Science and Engineering

Feb 18, 2026J/K move · Enter open · S save

TeCoNeRV: Leveraging Temporal Coherence for Compressible Neural Representations for Videos

Namitha Padmanabhan, Matthew Gwilliam, Abhinav Shrivastava

Department of Computer Science, University of Maryland, College Park, USA.

Neural Video Compression Implicit Neural Representations Hypernetworks

Open paper AskPDF arXiv

Implicit Neural Representations (INRs) have recently demonstrated impressive performance for video compression. However, since a separate INR must be overfit for each video, scaling to high-resolution videos while maintaining encoding efficiency remains a significant challenge. Hypernetwork-based approaches predict INR weights (hyponetworks) for unseen videos at high speeds, but with low quality, large compressed size, and prohibitive memory needs at higher resolutions. We address these fundamental limitations through three key contributions: (1) an approach that decomposes the weight prediction task spatially and temporally, by breaking short video segments into patch tubelets, to reduce the pretraining memory overhead by 20

\times

; (2) a residual-based storage scheme that captures only differences between consecutive segment representations, significantly reducing bitstream size; and (3) a temporal coherence regularization framework that encourages changes in the weight space to be correlated with video content. Our proposed method, TeCoNeRV, achieves substantial improvements of 2.47dB and 5.35dB PSNR over the baseline at 480p and 720p on UVG, with 36% lower bitrates and 1.5-3

\times

faster encoding speeds. With our low memory usage, we are the first hypernetwork approach to demonstrate results at 480p, 720p and 1080p on UVG, HEVC and MCL-JCV. Our project page is available at https://namithap10.github.io/teconerv/ .

Abstract

Video Representation Learning

Explore similar work

TeCoNeRV: Leveraging Temporal Coherence for Compressible Neural Representations for Videos

TinyNeRV: Compact Neural Video Representations via Capacity Scaling, Distillation, and Low-Precision Inference

LRConv-NeRV: Low Rank Convolution for Efficient Neural Video Compression