cs.CVOct 30, 2026

A Patch is More Than a Token

Authors: Zixiong Wang, Haiping Ma

Organizations: Shaoxing University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision Transformers represent each image patch through a canonical token produced under one observation condition. That token is a powerful interface, but it is still one realized state of the patch. We introduce PMTT, which establishes two foundations: a patch representation object can extend from one canonical state to a canonical-centered Representation State Family, and identity-preserving variation opens Patch-Level Local Feature Space. We develop four parts of this representation program. Patch Information Acquisition and Dense Orbital Views expose lawful same-patch states. Visible-to-Hidden prediction tests whether observed states reveal structure beyond themselves. Intrinsic analysis identifies low-dimensional, energy-concentrated, patch-specific, predictive geometry. Representation Organization separates acquiring more information from organizing that information, with an analytic Manifold as the central structured representation. We realize PMTT on frozen DINOv3 S/B/L/H+ backbones without neural post-training of the representation. Across two independent 4,096-anchor cohorts, Manifold explains 70.8-79.6% of Hidden variation, compared with 55.0-72.3% for directly Observed states, and wins on more than 95% of anchors at every scale. Effective rank falls from about 2.5 to 2.0 while dominant energy concentration rises with backbone scale. Across six visual capabilities and four scales, Observed exceeds the canonical Center in 21 of 24 conditions, Manifold exceeds Center in 23, and Manifold exceeds Observed in 16. The strongest current consequence appears in physical correspondence, where MegaDepth satisfies Center < Observed < Manifold at every scale. These results show that pretrained visual backbones contain richer patch information than the standard single-token interface directly exposes. Code and experimental assets are available at https://github.com/zixiongwang-ai/PMTT.

Figures & tables

Figure 1: The scientific structure of PMTT . A patch retains a fixed spatial identity while lawful local observations produce a canonical-centered Representation State Family. The family opens Patch-Level Local Feature Space, which motivates four scientific questions: information acquisition, Visible-to-Hidden prediction, intrinsic laws, and representation organization. A clean analytic realization on frozen DINOv3 separates discrete Observed states from continuous Manifold organization and reveals low-dimensional, energy-concentrated, patch-specific, predictive geometry. The resulting representation is richer than the standard single-token interface.
Figure 2: From Dense Orbital Views to Manifold organization. (a) A frozen DINOv3 example preserves Patch Identity across the original image and Canonical State. (b) Eight orbital directions illustrate Identity-Preserving Local Observation. The full construction uses 16 Visible states and 16 disjoint Hidden states. (c) Observed retains the acquired states as discrete points. (d) Manifold organizes the same information into a bounded continuous local state space. Triangles denote Visible states, open circles Hidden states, and the square the Canonical State.
Figure 3: Intrinsic Laws and predictive Manifold geometry. Manifold predicts Hidden states substantially better than Observed across both independent cohorts and all four frozen DINOv3 scales. Supporting panels show decreasing effective rank and increasing energy concentration. Solid lines denote the Independent Holdout Cohort and dashed lines the Original Cohort.
Table 1: Intrinsic Laws across backbone scales. Each cohort contains 4,096 anchors. Coverage measures the reduction in total Hidden squared error relative to the Canonical State. Full-32 uses Visible and Hidden states and is a diagnostic ceiling. Energy percentages average within-anchor normalized spectra.
Figure 4: Capability gains from acquisition and Manifold organization. Observed exceeds Center in 21 of 24 task-scale conditions, Manifold exceeds Center in 23, and Manifold exceeds Observed in 16. The full matrix separates Information Acquisition Value, Representation Organization Value, and Total Representation Extension in task-native metrics.
Table 2: Primary capability results. Displayed values are percentages except NYUv2 AbsRel. Bold marks the best primary representation within each task-scale condition. Controls and supplementary pose conditions do not enter the 72-cell matrix.
Figure 5: Visible correspondence gains from Manifold organization. The largest case shows outdoor MegaDepth correspondence, followed by indoor ScanNet and semantic SPair correspondence. Each case improves over both Center and Observed . Score fields use a shared within-case normalization, so localization sharpness and peak position are directly comparable. The examples visualize the mechanism, while Fig. 4 and Table 2 report complete outcomes.
Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Table A1 : Evaluation substrate and statistical units. The table records official units, primary metrics, paired units, uncertainty procedures, and control availability.
Table A2 : Analytic realization configuration. Representation formation is analytic and uses no neural post-training. Task readers and dense axes use the fixed train-side configurations listed here.
Figure A1 : Independent intrinsic replication. The Original Cohort and Independent Holdout Cohort each contain 4,096 anchors and independently reproduce the Manifold ordering at every scale.
Figure A2 : Spectrum, energy, and effective-rank scaling. Mean normalized spectra, effective rank, Top-4 energy, and overlap with the Full-32 Diagnostic Ceiling are shown separately for both cohorts.
Figure A3 : Patch specificity, direction, and diagnostic ceiling. Correct same-patch geometry predicts substantially more Hidden variation than Random, Donor, or Shared geometry. Supporting panels report tangential variation, boundedness, cap saturation, and the Full-32 ceiling.
Figure A4 : High-coverage real orbital state families, page 1 of 4. Each page shows the original image, Canonical State, all 16 Visible and 16 Hidden observation crops, and the corresponding factor projection. Yellow squares locate the same underlying image point.
Figure A5 : High-coverage real orbital state families, page 2 of 4.
Figure A6 : High-coverage real orbital state families, page 3 of 4.
Figure A7 : High-coverage real orbital state families, page 4 of 4.
Table A3 : Complete intrinsic summary. The table separates pooled and anchor-mean quantities, paired organization effects, spectra, overlap, and scale comparisons.
Table A4 : Intrinsic controls and diagnostics. All coverage values are pooled percentages. Random, Donor, and Shared isolate direction and patch identity, while Full-32 provides a non-inference ceiling.
Figure B1 : Paired capability effects with uncertainty. Columns show Manifold −- Center, Observed −- Center, and Manifold −- Observed with saved 95% paired bootstrap intervals. NYUv2 uses AbsRel reduction in units of 10−310^{-3} ; other tasks use percentage points.
Figure B2 : Downstream specificity controls. DonorObserved substitutes donor Visible states. DestroyedManifold preserves the factor spectrum while destroying orientation. The primary reader is fixed.
Figure B3 : Semantic correspondence and video propagation details. SPair category and rank structure are shown with DAVIS region, boundary, interior, and temporal metrics.
Figure B4 : Physical correspondence and pose details. Primary 32-anchor P@5/P@10 results, difficulty strata, pose AUC, failure, and the separate 64-anchor B/H+ pose supplement are reported.
Figure B5 : Dense prediction metrics and strata. ADE20K semantic organization is shown with boundary, interior, rare, and common partitions. NYUv2 remains in native depth metrics and strata.
Table B1 : Detailed 72-cell results and paired effects, page 1 of 3. All primary scores and all three paired effects are retained in the accompanying exact CSV.
Table B2 : Detailed 72-cell results and paired effects, page 2 of 3.
Table B3 : Detailed 72-cell results and paired effects, page 3 of 3.
Table B4 : Downstream controls. Only evaluated conditions are shown. Physical correspondence includes all scales; other capabilities include B and H+.
Table B5 : SPair and DAVIS secondary results, page 1 of 2. Exact category, difficulty, rank, region, boundary, interior, and temporal results remain available in the CSV.
Table B6 : SPair and DAVIS secondary results, page 2 of 2.
Table B7 : MegaDepth and ScanNet secondary results, page 1 of 2. All descriptive metrics use the complete 1,500-pair sets; paired inference uses the fixed untouched subsets.
Table B8 : MegaDepth and ScanNet secondary results, page 2 of 2.
Table B9 : NYUv2 metrics and strata. Lower is better for depth error metrics and higher is better for δ\delta accuracy. All values retain their original image-level aggregation.
Table B10 : ADE20K metrics and strata. Dataset-level mIoU, pixel accuracy, mean class accuracy, boundary/interior, and rare/common partitions are reported from summed 150-class confusion matrices.
Figure C1 : Extended positive correspondence gallery, page 1 of 4. Additional cases extend the visible correspondence gains beyond Fig. 5 without repeating its examples.
Figure C2 : Extended positive correspondence gallery, page 2 of 4.
Figure C3 : Extended positive correspondence gallery, page 3 of 4.
Figure C4 : Extended positive correspondence gallery, page 4 of 4.
Figure C5 : Extended positive video and dense-semantics gallery, page 1 of 5. ADE20K and DAVIS examples show cases in which Manifold improves over both Observed and Center . Green pixels become correct, orange pixels become incorrect, and gray pixels are unchanged or ignored.
Figure C6 : Extended positive video and dense-semantics gallery, page 2 of 5.
Figure C7 : Extended positive video and dense-semantics gallery, page 3 of 5.
Figure C8 : Extended positive video and dense-semantics gallery, page 4 of 5.
Figure C9 : Extended positive video and dense-semantics gallery, page 5 of 5.
More in this topic

Visual Representation Learning

Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.

See all 230 papers

Explore similar work

CardsList

  1. Learning Dynamic Evidence Routes for Vision Transformer Probing

    Apr 29, 2026Zice Wang, Zhenyu ZhangVision Transformer

    School of Future Technology, Northeastern University, Shenyang, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China

  2. Vision Transformers Need Better Token Interaction

    May 22, 2026Linxiang SuVisual Representation LearningVision Transformer

    University of Szeged

  3. From Patching to Pruning Visual Computation in Vision Language Models

    Oct 2, 2026Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen +3Efficient VLM InferenceVisual Token Pruning

    Northeastern University, USA · EmbodyX Inc., USA · Zhejiang University, China +1