DepthTCM: High Efficient Depth Compression via Physics-aware Transformer-CNN Mixed Architecture

Young-Seo Chang; Yatong An; Jae-Sang Hyun

cs.CVMar 22, 2026

DepthTCM: High Efficient Depth Compression via Physics-aware Transformer-CNN Mixed Architecture

Authors: Young-Seo Chang, Yatong An, Jae-Sang Hyun

Abstract

We propose DepthTCM, a physics-aware end-to-end framework for depth map compression. In our framework of DepthTCM, the high-bit depth map is first converted to a conventional 3-channel image representation losslessly using a method inspired by a physical sinusoidal fringe pattern based profiliometry system, then the 3-channel color image is encoded and decoded by a recently developed Transformer-CNN mixed neural network architecture. Specifically, DepthTCM maps depth to a smooth 3-channel using multiwavelength depth (MWD) encoding, then globally quantized the MWD encoded representation to 4 bits per channel to reduce entropy, and finally is compressed using a learned codec that combines convolutional and Transformer layers. Experiment results demonstrate the advantage of our proposed method. On Middlebury 2014, DepthTCM reaches 0.307 bpp while preserving 99.38% accuracy, a level of fidelity commensurate with lossless PNG. We additionally demonstrate practical efficiency and scalability, reporting average end-to-end inference times of 41.48 ms (encoder) and 47.45 ms (decoder) on the ScanNet++ iPhone RGB-D subset. Ablations validate our design choices: relative to 8-bit quantization, 4-bit quantization reduces bitrate by 66% while maintaining comparable reconstruction quality, with only a marginal 0.68 dB PSNR change and a 0.04% accuracy difference. In addition, Transformer--CNN blocks further improve PSNR by up to 0.75 dB over CNN-only architectures.

Figures & tables

Figure 1: DepthTCM framework architecture. The framework integrates multiwavelength depth (MWD) encoding, 4-bit quantization, and a hybrid Transformer–CNN compression backbone into a fully differentiable end-to-end system for efficient depth map compression and reconstruction.

Figure 2: Visualization of Multiwavelength Depth Encoding. (a) Input 32-bit depth map, (b) 8-bit Quantized MWD Encoded Map, (c) 4-bit Quantized MWD Encoded Map, (d) Red channel: sinusoidal encoding, $\sin(2\pi Z/P)$ . (e) Green channel: sinusoidal encoding, $\cos(2\pi Z/P)$ , (f) Blue channel: normalized depth for long-wavelength variation.

Figure 3: Rate–distortion curves on the Middlebury 2014 dataset. NRMSE (%) is plotted against bitrate (bpp) measured per original depth pixel. The error is computed following the evaluation protocol of N-DEPTH, where metrics are calculated over the shared intersection of valid recovery regions. Rate–distortion operating points of our method are obtained by varying the Lagrangian multiplier $\lambda$ .

Method	Codec	Bitrate $\downarrow$ (bpp)	PSNR $\uparrow$ (dB)	NRMSE $\downarrow$ (%)	CR $\uparrow$
MWD †	JPEG 90	1.116	N/A	1.012	28.66:1
N-DEPTH †	JPEG 90	0.774	N/A	0.068	41.35:1
Ours (8-bit)	Learned	0.525	50.66	0.059	26.39:1
Ours (4-bit)	Learned	0.307	49.89	0.063	44.29:1

Table 1: Quantitative comparison of depth compression methods on the Middlebury 2014 [ 21 ] benchmark. Results marked with

\dagger

are reported from the original publication [ 26 ] , where JPEG(Q=90) is used as the entropy codec. Our method employs end-to-end learned entropy coding. All bitrate (bpp) values are computed per original depth pixel.

Method	Codec	Bitrate $\downarrow$ (bpp)	PSNR $\uparrow$ (dB)	RMSE $\downarrow$	Accuracy $\uparrow$ (%)
MWD (4-bit)	PNG	6.896	75.56	0.102	99.97
MWD (4-bit)	JPEG 90	2.191	26.87	4.774	98.63
MWD (4-bit)	JPEG-LS	4.287	57.47	0.340	99.86
Ours (8-bit)	Learned	1.075	44.99	0.509	99.85
Ours (4-bit)	Learned	0.363	44.31	0.535	99.84

Table 2: Performance comparison with standard image codecs on the DIODE [ 29 ] dataset. Our method achieves a competitive accuracy comparable to the near-lossless JPEG-LS but at a significantly reduced bitrate.

Method	Codec	Bitrate $\downarrow$ (bpp)	PSNR $\uparrow$ (dB)	RMSE $\downarrow$	Accuracy $\uparrow$ (%)
MWD (4-bit)	PNG	10.40	57.69	0.109	99.86
MWD (4-bit)	JPEG 90	4.401	12.45	20.08	74.56
Ours (8-bit)	Learned	1.422	24.20	5.204	93.41
Ours (4-bit)	Learned	0.580	23.77	5.465	93.08

Table 3: Performance comparison with standard image codecs on the KITTI Depth Completion benchmark [ 28 ] .

Method	Codec	Bitrate $\downarrow$ (bpp)	PSNR $\uparrow$ (dB)	RMSE $\downarrow$	Accuracy $\uparrow$ (%)
MWD (4-bit)	PNG	9.153	78.45	0.030	99.68
MWD (4-bit)	JPEG 90	2.428	35.39	4.334	53.89
Ours (8-bit)	Learned	0.655	53.61	0.531	94.35
Ours (4-bit)	Learned	0.602	53.26	0.553	94.12

Table 4: Performance comparison with standard image codecs on ScanNet v2 [ 4 ] .

Method	Codec	Bitrate $\downarrow$ (bpp)	PSNR $\uparrow$ (dB)	RMSE $\downarrow$	Accuracy $\uparrow$ (%)
MWD (4-bit)	PNG	11.528	78.19	0.013	99.98
MWD (4-bit)	JPEG 90	2.554	34.68	0.086	97.09
Ours (8-bit)	Learned	0.901	42.74	0.036	98.76
Ours (4-bit)	Learned	0.638	42.63	0.037	98.75

Table 5: Performance comparison on the ScanNet++ [ 32 ] iPhone RGB-D subset.

Method	Codec	Bitrate $\downarrow$ (bpp)	PSNR $\uparrow$ (dB)	RMSE $\downarrow$	Accuracy $\uparrow$ (%)
MWD (4-bit)	PNG	3.803	75.12	0.076	99.98
MWD (4-bit)	JPEG 90	1.058	38.29	5.300	98.60
Ours (8-bit)	Learned	0.543	49.14	1.653	99.57
Ours (4-bit)	Learned	0.217	48.45	1.743	99.55

Table 6: Performance comparison on the UnrealStereo4K dataset [ 34 ] (downsampled to

1920\times 1080

for testing).

Dataset	Input	Enc. (ms)	Dec. (ms)	Enc./Dec. FPS
DIODE	1024x768	164.40	194.80	6.08 / 5.13
ScanNet v2	640x480	82.78	96.46	12.08 / 10.37
ScanNet++	256x192	41.48	47.45	24.10 / 21.08
N-DEPTH [ 26 ]	1280x720	236.40	237.20	4.23 / 4.22

Table 7: Average inference time (Ours, 4-bit) at benchmark input depth resolutions. Throughput is

1000/t

(FPS).

Method	Bitrate $\downarrow$ (bpp)	PSNR $\uparrow$ (dB)	RMSE $\downarrow$	Accuracy $\uparrow$ (%)
8-bit	1.075	44.99	0.509	99.21
5-bit	0.397	44.61	0.544	99.16
4-bit	0.363	44.31	0.535	99.17
3-bit	0.441	40.94	0.769	98.83
2-bit	0.427	20.42	5.540	88.47
PNG	6.896	75.56	0.103	99.98

Table 8: Ablation study of quantization bit-depths on the DIODE dataset.

	Transformer-CNN		CNN-only
Method	PSNR	Bpp	PSNR	Bpp
Ours (8-bit)	50.66	0.525	49.91	0.532
Ours (4-bit)	49.89	0.307	49.22	0.259

Table 9: Ablation study of architecture choices on Middlebury 2014 [ 21 ] .

Quantization	Bit Depth	PSNR	RMSE	Bitrate
Adaptive	2-6 bits	47.82	0.018	0.412
Fixed (Global)	4 bits	48.05	0.017	0.307

Table 10: Ablation study on quantization strategies.

Method	Bitrate $\downarrow$ (bpp)	PSNR $\uparrow$ (dB)	RMSE $\downarrow$	Accuracy $\uparrow$ (%)
MWD (4-bit) + PNG	6.896	75.56	0.102	99.97
MWD (4-bit) + JPEG 90	2.191	26.87	4.774	98.63
MWD (4-bit) + JPEG LS	4.287	57.47	0.340	99.86
Ours (8-bit)	1.075	44.99	0.509	99.85
Ours (4-bit)	0.363	44.31	0.535	99.84
Only-TCM	1.443	22.38	1.213	92.42

Table 11: Comparison between our MWD-based approach and the standalone TCM backbone (Only-TCM) on the DIODE dataset. The “Only-TCM” baseline fails to capture geometric structures, resulting in high bitrates and low PSNR compared to our method.

Method	Inputs	Rate Control	Architecture Features	Notes
Texture-Guided DMC [ 18 ]	Depth+RGB	Fixed	$\bullet$ RGB guidance	$\bullet$ Requires paired RGB (not always available).
Lossless High-Precision [ 31 ]	Depth only	N/A	$\bullet$ Pseudo-residual	$\bullet$ Focused only on archival lossless coding (not RD trade-offs).
Variable-Rate DMC [ 23 ]	Depth only	Variable ( $\lambda$ )	$\bullet$ Wedgelet filters $\bullet$ VGG19 encoder	$\bullet$ Synthesized-view RD (original texture + compressed depth); limited cross-dataset reproducibility.
DepthTCM (Ours)	Depth only	Variable ( $\lambda$ )	$\bullet$ Physics-based MWD $\bullet$ Transformer–CNN	$\bullet$ Depth-only; physics-based transform improves RD efficiency.

Table 12: Comparison with end-to-end depth compression methods. Direct metric comparison is omitted due to different datasets and evaluation protocols.

Learned Image Compression

Learned image compression uses neural encoders, quantization, and entropy models to reduce image bitrates while balancing fidelity, perceptual quality, and task utility.

See all 120 papers

Motion-Compensated Weight Compression
May 23, 2026Ismail LamaakalPredictive Coding Weight-Only Quantization
Multidisciplinary Faculty of Nador · Mohammed Premier University · Oujda 60000, Morocco
One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts
Oct 8, 2026Adrian Bulat, Yassine Ouali, Georgios TzimiropoulosEfficient ViTs Mixture of Experts
Samsung AI Cambridge · Technical University of Iasi · Queen Mary University of London
Deep Microcompression: Structured Pruning and Bit-packed Quantization for Microcontrollers
Sep 4, 2026Opegbemi Matthias Busoye, Tolulope Matthew Busoye, Eghonghon-aye EigbeNeural Network Compression On-Device Inference
PowerLabs Technologies, Lagos, Nigeria

May 23, 2026J/K move · Enter open · S save

Motion-Compensated Weight Compression

Ismail Lamaakal

Multidisciplinary Faculty of Nador · Mohammed Premier University · Oujda 60000, Morocco

Predictive Coding Weight-Only Quantization Neural Network Compression

Open paper AskPDF arXiv

Neural network weights are increasingly a bottleneck for deployment, yet most compression pipelines treat layers independently and overlook cross-layer redundancy induced by function-preserving symmetries. We propose Motion-Compensated Weight Compression (MCWC), a weight-only codec that aligns permutation-symmetric blocks (e.g., hidden units and attention heads) to maximize cross-layer correspondence, turning depth into a predictable sequence. In the aligned coordinate system, MCWC uses a lightweight layer-sequential predictor with periodic keyframes and encodes only quantized prediction residuals using a learned entropy model trained under a rate distortion objective. A simple decoder reconstructs deployable weights by entropy decoding, dequantization, predictor-driven reconstruction, and inverse alignment, enabling fast weight materialization for inference. Across Transformer language modeling and vision classification, MCWC improves the rate accuracy Pareto frontier over strong quantization and learned weight-codec baselines, while maintaining competitive decode time. Ablations confirm that alignment, prediction, entropy modeling, and keyframe scheduling are each necessary for the full gains. Our code is available via https://github.com/Ism-ail11/MCWC.

Abstract

Learned Image Compression

Explore similar work

Motion-Compensated Weight Compression

One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts

Deep Microcompression: Structured Pruning and Bit-packed Quantization for Microcontrollers