cs.CVOct 30, 2026

A Patch is More Than a Token

Authors: Zixiong Wang, Haiping Ma

Organizations: Shaoxing University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision Transformers represent each image patch through a canonical token produced under one observation condition. That token is a powerful interface, but it is still one realized state of the patch. We introduce PMTT, which establishes two foundations: a patch representation object can extend from one canonical state to a canonical-centered Representation State Family, and identity-preserving variation opens Patch-Level Local Feature Space. We develop four parts of this representation program. Patch Information Acquisition and Dense Orbital Views expose lawful same-patch states. Visible-to-Hidden prediction tests whether observed states reveal structure beyond themselves. Intrinsic analysis identifies low-dimensional, energy-concentrated, patch-specific, predictive geometry. Representation Organization separates acquiring more information from organizing that information, with an analytic Manifold as the central structured representation. We realize PMTT on frozen DINOv3 S/B/L/H+ backbones without neural post-training of the representation. Across two independent 4,096-anchor cohorts, Manifold explains 70.8-79.6% of Hidden variation, compared with 55.0-72.3% for directly Observed states, and wins on more than 95% of anchors at every scale. Effective rank falls from about 2.5 to 2.0 while dominant energy concentration rises with backbone scale. Across six visual capabilities and four scales, Observed exceeds the canonical Center in 21 of 24 conditions, Manifold exceeds Center in 23, and Manifold exceeds Observed in 16. The strongest current consequence appears in physical correspondence, where MegaDepth satisfies Center < Observed < Manifold at every scale. These results show that pretrained visual backbones contain richer patch information than the standard single-token interface directly exposes. Code and experimental assets are available at https://github.com/zixiongwang-ai/PMTT.

Figures & tables

Figure 1: The scientific structure of PMTT . A patch retains a fixed spatial identity while lawful local observations produce a canonical-centered Representation State Family. The family opens Patch-Level Local Feature Space, which motivates four scientific questions: information acquisition, Visible-to-Hidden prediction, intrinsic laws, and representation organization. A clean analytic realization on frozen DINOv3 separates discrete Observed states from continuous Manifold organization and reveals low-dimensional, energy-concentrated, patch-specific, predictive geometry. The resulting representation is richer than the standard single-token interface.
Figure 2: From Dense Orbital Views to Manifold organization. (a) A frozen DINOv3 example preserves Patch Identity across the original image and Canonical State. (b) Eight orbital directions illustrate Identity-Preserving Local Observation. The full construction uses 16 Visible states and 16 disjoint Hidden states. (c) Observed retains the acquired states as discrete points. (d) Manifold organizes the same information into a bounded continuous local state space. Triangles denote Visible states, open circles Hidden states, and the square the Canonical State.
Figure 3: Intrinsic Laws and predictive Manifold geometry. Manifold predicts Hidden states substantially better than Observed across both independent cohorts and all four frozen DINOv3 scales. Supporting panels show decreasing effective rank and increasing energy concentration. Solid lines denote the Independent Holdout Cohort and dashed lines the Original Cohort.
Table 1: Intrinsic Laws across backbone scales. Each cohort contains 4,096 anchors. Coverage measures the reduction in total Hidden squared error relative to the Canonical State. Full-32 uses Visible and Hidden states and is a diagnostic ceiling. Energy percentages average within-anchor normalized spectra.
Figure 4: Capability gains from acquisition and Manifold organization. Observed exceeds Center in 21 of 24 task-scale conditions, Manifold exceeds Center in 23, and Manifold exceeds Observed in 16. The full matrix separates Information Acquisition Value, Representation Organization Value, and Total Representation Extension in task-native metrics.
Table 2: Primary capability results. Displayed values are percentages except NYUv2 AbsRel. Bold marks the best primary representation within each task-scale condition. Controls and supplementary pose conditions do not enter the 72-cell matrix.
Figure 5: Visible correspondence gains from Manifold organization. The largest case shows outdoor MegaDepth correspondence, followed by indoor ScanNet and semantic SPair correspondence. Each case improves over both Center and Observed . Score fields use a shared within-case normalization, so localization sharpness and peak position are directly comparable. The examples visualize the mechanism, while Fig. 4 and Table 2 report complete outcomes.
Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Table A1 : Evaluation substrate and statistical units. The table records official units, primary metrics, paired units, uncertainty procedures, and control availability.
Table A2 : Analytic realization configuration. Representation formation is analytic and uses no neural post-training. Task readers and dense axes use the fixed train-side configurations listed here.
Figure A1 : Independent intrinsic replication. The Original Cohort and Independent Holdout Cohort each contain 4,096 anchors and independently reproduce the Manifold ordering at every scale.
Figure A2 : Spectrum, energy, and effective-rank scaling. Mean normalized spectra, effective rank, Top-4 energy, and overlap with the Full-32 Diagnostic Ceiling are shown separately for both cohorts.
Figure A3 : Patch specificity, direction, and diagnostic ceiling. Correct same-patch geometry predicts substantially more Hidden variation than Random, Donor, or Shared geometry. Supporting panels report tangential variation, boundedness, cap saturation, and the Full-32 ceiling.
Figure A4 : High-coverage real orbital state families, page 1 of 4. Each page shows the original image, Canonical State, all 16 Visible and 16 Hidden observation crops, and the corresponding factor projection. Yellow squares locate the same underlying image point.
Figure A5 : High-coverage real orbital state families, page 2 of 4.
Figure A6 : High-coverage real orbital state families, page 3 of 4.
Figure A7 : High-coverage real orbital state families, page 4 of 4.
Table A3 : Complete intrinsic summary. The table separates pooled and anchor-mean quantities, paired organization effects, spectra, overlap, and scale comparisons.
Table A4 : Intrinsic controls and diagnostics. All coverage values are pooled percentages. Random, Donor, and Shared isolate direction and patch identity, while Full-32 provides a non-inference ceiling.
Figure B1 : Paired capability effects with uncertainty. Columns show Manifold −- Center, Observed −- Center, and Manifold −- Observed with saved 95% paired bootstrap intervals. NYUv2 uses AbsRel reduction in units of 10−310^{-3} ; other tasks use percentage points.
Figure B2 : Downstream specificity controls. DonorObserved substitutes donor Visible states. DestroyedManifold preserves the factor spectrum while destroying orientation. The primary reader is fixed.
Figure B3 : Semantic correspondence and video propagation details. SPair category and rank structure are shown with DAVIS region, boundary, interior, and temporal metrics.
Figure B4 : Physical correspondence and pose details. Primary 32-anchor P@5/P@10 results, difficulty strata, pose AUC, failure, and the separate 64-anchor B/H+ pose supplement are reported.
Figure B5 : Dense prediction metrics and strata. ADE20K semantic organization is shown with boundary, interior, rare, and common partitions. NYUv2 remains in native depth metrics and strata.
Table B1 : Detailed 72-cell results and paired effects, page 1 of 3. All primary scores and all three paired effects are retained in the accompanying exact CSV.
Table B2 : Detailed 72-cell results and paired effects, page 2 of 3.
Table B3 : Detailed 72-cell results and paired effects, page 3 of 3.
Table B4 : Downstream controls. Only evaluated conditions are shown. Physical correspondence includes all scales; other capabilities include B and H+.
Table B5 : SPair and DAVIS secondary results, page 1 of 2. Exact category, difficulty, rank, region, boundary, interior, and temporal results remain available in the CSV.
Table B6 : SPair and DAVIS secondary results, page 2 of 2.
Table B7 : MegaDepth and ScanNet secondary results, page 1 of 2. All descriptive metrics use the complete 1,500-pair sets; paired inference uses the fixed untouched subsets.
Table B8 : MegaDepth and ScanNet secondary results, page 2 of 2.
Table B9 : NYUv2 metrics and strata. Lower is better for depth error metrics and higher is better for δ\delta accuracy. All values retain their original image-level aggregation.
Table B10 : ADE20K metrics and strata. Dataset-level mIoU, pixel accuracy, mean class accuracy, boundary/interior, and rare/common partitions are reported from summed 150-class confusion matrices.
Figure C1 : Extended positive correspondence gallery, page 1 of 4. Additional cases extend the visible correspondence gains beyond Fig. 5 without repeating its examples.
Figure C2 : Extended positive correspondence gallery, page 2 of 4.
Figure C3 : Extended positive correspondence gallery, page 3 of 4.
Figure C4 : Extended positive correspondence gallery, page 4 of 4.
Figure C5 : Extended positive video and dense-semantics gallery, page 1 of 5. ADE20K and DAVIS examples show cases in which Manifold improves over both Observed and Center . Green pixels become correct, orange pixels become incorrect, and gray pixels are unchanged or ignored.
Figure C6 : Extended positive video and dense-semantics gallery, page 2 of 5.
Figure C7 : Extended positive video and dense-semantics gallery, page 3 of 5.
Figure C8 : Extended positive video and dense-semantics gallery, page 4 of 5.
Figure C9 : Extended positive video and dense-semantics gallery, page 5 of 5.
More in this topic

Visual Representation Learning

Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.

See all 230 papers

Explore similar work

CardsList
Apr 29, 2026

Learning Dynamic Evidence Routes for Vision Transformer Probing

Probing frozen vision transformers typically uses permutation-invariant aggregation (GAP or [CLS]\texttt{[CLS]}), treating patch tokens as an unstructured set. Content-dependent probes such as self-attention are useful accuracy controls, but they do not expose a fixed token schedule or fixed position weights for auditing. We introduce SSMProbe, a probe of frozen vision transformers that learns a Sinkhorn patch route and reads it with a diagonal S4 decoder. The decoder's causal linear time-invariant (LTI) recurrence yields fixed position coefficients, exposing how patch arrangement interacts with sequential weighting. Held-out spatial predictability measures route organization beyond a location-bias null. On CUB and Cars, MAE, BEiT, DINOv2, and supervised ViT yield spatially organized routes with distinct readout profiles: DINOv2 is [CLS]\texttt{[CLS]}-dominant, while MAE benefits from learned routing over fixed scans. On MAE/ImageNet, Sinkhorn reaches 70.33% accuracy, 5.55 percentage points above the strongest fixed scan.

Zice Wang, Zhenyu Zhang

School of Future Technology, Northeastern University, Shenyang, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China

Vision Transformer
AskPDF
May 22, 2026

Vision Transformers Need Better Token Interaction

Vision Transformers (ViTs) can learn strong image-level representations while their patch representations become less effective for dense prediction during prolonged training. We revisit this dense degradation phenomenon and argue that it is not fully explained by high-norm artifacts alone. Instead, we characterize semantic diffusion: an optimization shortcut in which global semantic information spreads through patch tokens beyond what is locally justified. Our analysis shows that dense representation quality is not captured by locality alone: shallow features can remain better aligned with foreground regions yet underperform deeper features, and [CLS] features remain complementary for dense prediction. These observations suggest that the goal should not be to remove global context, but to make token interactions more selective. We therefore study sparse attention as a minimal intervention, replacing softmax attention with entmax-1.5 while preserving global token connectivity. On DINOv1 ViT-S/16 trained for 200 epochs on ImageNet-1K, this change preserves ImageNet linear probing accuracy and substantially improves semantic segmentation performance: VOC mIoU increases from 42.80 to 48.78, ADE20K from 19.85 to 21.97, and Cityscapes from 36.79 to 37.87. These results suggest that selective token mixing is a simple and effective bias for improving dense ViT representations.

Linxiang Su

University of Szeged

Visual Representation LearningVision TransformerSparse AttentionSemantic Segmentation
AskPDF
Oct 2, 2026

From Patching to Pruning Visual Computation in Vision Language Models

Vision language models (VLMs) incur substantial inference cost because every visual token is processed by the attention and MLP projections of every decoder layer, even when token-specific visual computation is unnecessary at many depths. We introduce Patch-to-Prune (P2P), inspired by Mechanistic Interpretability, a training-free framework that converts activation patching from a diagnostic tool into an inference-time computation bypass. P2P performs validation-guided forward and backward layer sweeps to identify decoder regions whose visual-token projection outputs can be replaced by fixed neutral proxy activation vectors within a user-specified accuracy tolerance. Unlike conventional token-pruning methods, P2P preserves the sequence length, token order, positional information, attention mask, and residual pathways, thereby pruning computation without removing tokens or modifying the pretrained model weights. We evaluate P2P on four VLMs from the Qwen2.5-VL and LLaVA families across seven multi-modal benchmarks using mutually disjoint calibration, validation, and test partitions. P2P at a 3% tolerance retains around 94% of dense accuracy while reducing FLOPs by 55%. Beyond these efficiency gains, our layer-wise analysis suggests that visual processing in VLMs is non-uniformly distributed across decoder depth: early and late layers often require little token-specific visual computation, whereas intermediate layers appear to perform most task-relevant visual integration, enabling later reasoning to rely largely on visual information already embedded in shared residual and textual representations. This makes P2P both an efficient inference framework and a causal lens into visual information processing in VLMs.

Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen +3

Northeastern University, USA · EmbodyX Inc., USA · Zhejiang University, China +1

Efficient VLM InferenceVisual Token PruningVLM InterpretabilityActivation Patching
AskPDF