PAT-VCM: Plug-and-Play Auxiliary Tokens for Video Coding for Machines

Wei Jiang; Wei Wang

cs.CVApr 14, 2026

PAT-VCM: Plug-and-Play Auxiliary Tokens for Video Coding for Machines

Authors: Wei Jiang, Wei Wang

Organizations: Futurewei Technologies Inc. San Jose, CA 95131

Abstract

Existing video coding for machines is often trained for a specific downstream task and model. As a result, the compressed representation becomes tightly coupled to the end task, making it difficult to scale across multiple tasks or adapt to model updates. We propose PAT-VCM, a plug-and-play auxiliary-token framework for video coding for machines. PAT-VCM keeps a shared baseline compressed stream and augments it with lightweight task-aware auxiliary tokens, allowing different downstream tasks to recover the information they need without retraining a separate codec for each task. The framework supports three forms of auxiliary information: visual residual tokens, prompt/control tokens, and semantic tokens. We evaluate PAT-VCM on segmentation, depth estimation, and semantic recognition. A shared detection-oriented auxiliary branch provides a reusable first refinement, task-specific visual branches improve segmentation and depth, prompt tokens provide further segmentation gains at negligible bitrate, and semantic tokens achieve strong recognition performance with extremely low overhead. These results suggest that a shared compressed representation, combined with lightweight task-aware auxiliary tokens, is a practical and scalable alternative to tightly task-coupled VCM design.

Figures & tables

Figure 1: Overview of PAT-VCM. A shared baseline stream provides a common compressed representation and coarse reconstruction. On top of this baseline, auxiliary streams can be attached to preserve or recover information needed by different downstream tasks. These auxiliary streams may take different forms, including visual residual tokens, text-based tokens, and prompt/control tokens.

Dataset	Orig	Cosmos	Det-Aux	+ Seg-Aux	+ Seg-Aux + 1-pt	+ Seg-Aux + FG+BG
DAVIS	0.774	0.677	0.686	0.709	0.751	0.764
MOSE	0.713	0.693	0.694	0.707	0.743	0.753
VIPSeg	0.731	0.721	0.723	0.727	0.751	0.754

Table 1: Full segmentation results. Metric is mean IoU.

Figure 2: Qualitative segmentation results. From left to right: original frame, compressed reconstruction, segmentation on the compressed reconstruction, result after Seg-Aux, and result after Seg-Aux plus prompt tokens. Examples marked with “(>Orig)” exceed the segmentation quality obtained from the uncompressed original frame under the same frozen SAM-based evaluation.

Variant	Full AbsRel	ROI AbsRel	ROI $\delta<1.25$
Cosmos	2.576	3.764	0.828
+ Det-Aux	2.151	3.057	0.834
+ Det-Aux + Depth-Aux	1.290	1.554	0.840

Table 2: Depth auxiliary results on DAVIS. AbsRel is lower-is-better,

\delta<1.25

is higher-is-better.

Figure 3: Qualitative depth estimation results. From left to right: original frame, compressed reconstruction, depth from the compressed reconstruction, result after Depth-Aux, error map for the compressed reconstruction, and error map for the refined result. Examples marked with “(>Orig)” exceed the depth quality obtained from the uncompressed original frame.

Task	System	bpp	Metric	Value
Segmentation	Cosmos DV4	0.0833	IoU $\uparrow$	0.677
	PAT-VCM Det-Aux+Seg-Aux	0.1242	IoU $\uparrow$	0.687
	PAT-VCM Det-Aux+Seg-Aux+FG+BG	0.1242	IoU $\uparrow$	0.764
	Cosmos CV4 (nearest higher-rate)	0.1671	IoU $\uparrow$	0.509
	MPA q=1	0.0908	IoU $\uparrow$	0.739
Detection	Cosmos DV4	0.0833	matched IoU $\uparrow$	0.475

Table 3: Rate-matched evaluation on DAVIS. Segmentation uses SAM IoU, detection uses matched IoU, and depth uses AbsRel. For Cosmos CV4 we report the nearest bitrate operating point.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	No. GT	Cosmos	+ Det-Aux	Orig	% gap closed
DAVIS-2017 val	59	0.390	0.475	0.610	39%
MOSE val	386	0.233	0.249	0.267	47%
VIPSeg val	528	0.127	0.133	0.161	18%

Appendix

Table 4: Cross-dataset detection recall for the shared detection-oriented branch. Higher is better.

Method	Matched IoU	[email protected]
Cosmos	0.475	0.516
+ Det-Aux	0.512	0.549

Appendix

Table 5: Matched-protocol detection results on DAVIS. Higher is better.

Bin	No. instances	Cosmos+SAM	+ Text token	$\Delta$
Hard ( $<0.30$ )	26	0.098	0.246	+0.149
Medium ( $[0.30,0.75)$ )	160	0.592	0.586	-0.006
Easy ( $\geq 0.75$ )	191	0.853	0.836	-0.017
Overall	377	0.682	0.652	-0.030

Appendix

Table 6: Text-token segmentation on MOSE val. Metric is mean IoU.

Figure 4: Qualitative hard-case segmentation examples with text-based semantic tokens. From left to right: original frame, compressed reconstruction, segmentation on the compressed reconstruction, and result after adding text tokens. In these cases, the compressed visual signal is too weak for reliable mask selection, while the encoder-side text token provides semantic guidance that helps recover the correct object mask.

Policy	Avg bits/ROI	Avg bits/clip	$\Delta$ bpp	Hard-bin IoU	Overall IoU
No text	0	0	0	0.098	0.682
Uniform text	154	231	0.000088	0.246	0.652
Adaptive text	10.4	15.6	0.000006	0.246	0.692

Appendix

Table 7: Uniform versus adaptive text-token policies on MOSE val.

Variant	Mean IoU	Gain over Cosmos
Cosmos	0.677	+0.000
+ Det-Aux	0.687	+0.010
+ Det-Aux + 1-pt	0.740	+0.063
+ Det-Aux + Seg-Aux	0.710	+0.033
+ Det-Aux + Seg-Aux + 1-pt	0.751	+0.074
Orig	0.774	+0.097

Appendix

Table 8: Token-type ablation for segmentation on DAVIS. Metric is mean IoU.

Difficulty	No. instance	Cosmos	+ Det-Aux	+ Det-Aux + Seg-Aux	+ Det-Aux + Seg-Aux + 1-pt
easy	23	0.8742	0.8780	0.8715	0.9010
medium	29	0.6281	0.6443	0.6823	0.7236
hard	6	0.1661	0.1551	0.2208	0.3382

Appendix

Table 9: Segmentation performance by difficulty bin on DAVIS. Bin are defined by Cosmos IoU.

Size bin	No. instance	Cosmos	+ Det-Aux	+ Det-Aux + Seg-Aux	+ Det-Aux + Seg-Aux + 1-pt
Small	20	0.6010	0.5953	0.6131	0.6278
Medium	32	0.7263	0.7366	0.7611	0.8209
Large	6	0.6762	0.7222	0.7562	0.8188

Appendix

Table 10: Segmentation ROI-size sensitivity on DAVIS. Metric is mean IoU.

Difficulty	No. instance	Cosmos	+ Det-Aux	+ Det-Aux + Depth-Aux
easy	4	0.0341	0.0332	0.0394
medium	31	0.1243	0.1270	0.1320
hard	23	4.2539	3.5733	2.2300

Appendix

Table 11: Depth performance by difficulty bin on DAVIS. AbsRel is lower-is-better.

Dataset	Variant	AbsRel	RMSE	$\delta<1.25$
DAVIS full-frame	Cosmos	2.576	0.315	0.810
DAVIS full-frame	+ Det-Aux	2.151	0.310	0.807
DAVIS ROI-only	Cosmos	3.764	0.364	0.828
DAVIS ROI-only	+ Det-Aux	3.057	0.357	0.834
VIPSeg full-frame	Cosmos	0.484	0.187	0.893
VIPSeg full-frame	+ Det-Aux	0.476	0.187	0.895

Appendix

Table 12: Baseline depth characterization. Metrics are reported after per-frame scale–shift alignment. Lower AbsRel and RMSE are better; higher

\delta<1.25

is better.

Pipeline	Latency (ms / clip)	Throughput
Cosmos	101.6	88.6 fps
Encoder: Cosmos + DETR + Det-Aux	488.3	18.4 fps
Decoder: Cosmos + Det-Aux + DETR	530.5	17.0 fps
Encoder: Cosmos + DETR + Det-Aux + Seg-Aux	640.0	14.1 fps
Decoder: Cosmos + Det-Aux + Seg-Aux + SAM	692.5	13.0 fps
Encoder: above + adaptive LLaVA text	704.0	12.8 fps

Appendix

Table 13: Overall runtime by task and pipeline on a single NVIDIA L40S GPU.

Variant	Extra	Total	Bits / frame	bpp
Cosmos DV baseline	0	196,608	21,845	0.08333
+ Det-Aux (measured ROI-masked)	40,960	237,568	26,396	0.10069
+ Det-Aux (full-frame upper bound)	147,456	344,064	38,229	0.14583
+ Det-Aux + Seg-Aux	96,460	293,068	32,563	0.12422
+ Text + box	228	196,836	21,871	0.08343
+ 1-pt	5	196,615	21,846	0.08334

Appendix

Table 14: Bitrate counts for a 9-frame

512\times 512

clip.

Task	System	bpp	Metric	Value
Segmentation	Cosmos DV4 baseline	0.0833	IoU $\uparrow$	0.6761
	PAT-VCM (Det-Aux + Seg-Aux + FG+BG)	0.1242	IoU $\uparrow$	0.7640
	Cosmos CV4 (nearest higher-rate)	0.1671	IoU $\uparrow$	0.5087
Detection	Cosmos DV4 baseline	0.0833	matched IoU $\uparrow$	0.4751
	PAT-VCM (Det-Aux)	0.1007	matched IoU $\uparrow$	0.5116
	Cosmos CV4 (nearest higher-rate)	0.1671	matched IoU $\uparrow$	0.0309

Appendix

Table 15: Rate-matched comparison on DAVIS against the Cosmos CV4 quantization sweep. For CV4, we report the nearest higher-rate operating point from the sweep.

System	bpp	IoU $\uparrow$
Cosmos DV4	0.0833	0.6761
Cosmos DV4 + Det-Aux + Seg-Aux	0.1242	0.6874
Cosmos CV4 N=1	0.0838	0.5460
Cosmos CV4 N=2	0.1671	0.5087
Cosmos CV4 N=3	0.2504	0.6328
Cosmos CV4 N=10	0.8338	0.6764

Appendix

Table 16: Full DAVIS segmentation sweep for Cosmos CV4.

System	bpp	matched IoU $\uparrow$
Cosmos DV4	0.0833	0.4751
Cosmos DV4 + Det-Aux	0.1007	0.5116
Cosmos CV4 N=1	0.0838	0.0093
Cosmos CV4 N=2	0.1671	0.0309
Cosmos CV4 N=3	0.2504	0.2717
Cosmos CV4 N=10	0.8338	0.5229

Appendix

Table 17: Full DAVIS detection sweep for Cosmos CV4.

System	bpp	AbsRel $\downarrow$
Cosmos DV4	0.0833	2.296
Cosmos DV4 + Det-Aux	0.1007	2.412
Cosmos CV4 N=1	0.0838	23.55
Cosmos CV4 N=2	0.1671	19.86
Cosmos CV4 N=3	0.2504	2.361
Cosmos CV4 N=10	0.8338	1.448

Appendix

Table 18: Full DAVIS depth sweep for Cosmos CV4.

System	bpp	SAM IoU $\uparrow$
MPA q=1	0.0908	0.7387
MPA q=2	0.1320	0.7315
MPA q=3	0.1914	0.7454
MPA q=8	1.0189	0.7656
Cosmos baseline	0.0833	0.6761
Det-Aux + Seg-Aux	0.1242	0.6874

Appendix

Table 19: Comparison with MPA on DAVIS under matched per-frame SAM evaluation.

Method	Class agreement (%) $\uparrow$	Bits / ROI	No. instances
Cosmos (no token)	64.9	0	57
+ Det-Aux (no token)	63.2	0	57
+ Det-Aux + Class-label token	100.0	7	57

Appendix

Table 20: Semantic-token results for CLIP recognition on DAVIS.

Model	ROI MAE ∘
Cosmos	5.13
+ Det-Aux	5.07
+ Normal-Aux	5.06
+ Depth-Aux	5.17

Appendix

Table 21: Surface normal estimation on DAVIS ROI regions. MAE is mean angular error in degrees; lower is better.

Method	MKE (px) $\downarrow$	Bits / person
Cosmos (no token)	33.62	0
+ Det-Aux (no token)	31.99	0
+ Skeleton token	7.33	102

Appendix

Table 22: Pose estimation on DAVIS. MKE is mean keypoint error in pixels; lower is better.

Branch	Training dataset	In-domain held-out
Det-Aux	YTVIS 2021 splits 0–6	YTVIS split_7
Seg-Aux feat-distill	YTVIS 2021 splits 0–6	YTVIS split_7
Seg-Aux SAM-aware	MOSE train	MOSE val
Depth-Aux	VIPSeg train	VIPSeg val
Normal-Aux	VIPSeg train	–
Pose-Aux (primary)	VIPSeg train	–

Appendix

Table 23: Training datasets for auxiliary branches.

Video Compression

Video compression reduces the bits needed to store or transmit moving images while balancing visual quality, computational cost, and downstream usefulness.

See all 73 papers

Visual Token Coding for Video Multimodal Large Language Models
Aug 28, 2026Chenxin Fang, Tao Chen, JunChao You +3Multimodal Token Compression Video Token Compression
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China
Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding
Aug 9, 2026Donghui Feng, Fengxi Zhang, Changsheng Gao +6Entropy Modeling Visual Token Compression
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China · College of Computing and Data Science, Nanyang Technological University, Singapore · Pengcheng Laboratory, Shenzhen, Guangdong, China +1
Video Codec-Guided Training-Free Token Compression for Video Large Language Models
Oct 9, 2026Xihua Sheng, Ruijie Liu, Chang Wen ChenVideo Token Compression Video-Language Models
Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China

Aug 28, 2026J/K move · Enter open · S save

Visual Token Coding for Video Multimodal Large Language Models

Chenxin Fang, Tao Chen, JunChao You, Jun Peng+2

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China

Multimodal Token Compression Video Token Compression Visual Token Compression Multimodal Large Language Models

Open paper AskPDF arXiv

In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC). Inspired by classical video coding principles, e.g., HEVC, VTC performs structured compression by predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy. Based on this baseline framework, we also enhance VTC with a set of novel dynamic designs, such as Dynamic Resolution Input (DyRSO), Dynamic Token Allocation (DyTA), and Spatial Coverage Top-K (SC-TopK), and term this new approach

VTC_{Dy}

. To validate VTC, we apply it to three MLLMs and conduct experiments on multiple video understanding benchmarks. The experimental results show that VTC

_{\mathrm{Dy}}

achieves an average performance retention of 100.1% with a 50% token budget for Qwen3-VL, while still retaining 97.8% of the average performance when the token budget is reduced to 25%. Moreover, as a plug-and-play design, VTC requires no additional tuning of MLLMs for token coding. Our code is available at https://github.com/Msr233/VTC.

Abstract

Video Compression

Explore similar work

Visual Token Coding for Video Multimodal Large Language Models

Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding

Video Codec-Guided Training-Free Token Compression for Video Large Language Models