Learning Hierarchical Patch Splitting Policies for Faster Vision Transformers
Organizations: Carnegie Mellon University
Abstract
Vision Transformers (ViTs) are bottlenecked by the length of their input token sequence. Standard patchification fixes this length with a single patch size applied across the image, regardless of regional content. Prior work on adaptive patch sizing reduces token count by assigning coarser patches to visually simple regions, but relies on noisy pixel-level heuristics such as local entropy to make patchification decisions. We argue that this compute allocation should be grounded in the model's interpretation of task relevant information, not in raw pixel statistics. We introduce Split Policy Learned Image Tokenization which learns to allocate token resolution based on what the model finds meaningful, producing shorter token sequences that are better matched to task-relevant content. SPLIT matches the accuracy of full-resolution ViT classification baselines while using 30% fewer tokens. The same gains hold on dense prediction: SPLIT achieves competitive accuracy on COCO and ADE20K, demonstrating that semantically aligned patchification generalises well beyond classification.
Figures & tables
| Method | % Tokens | Object recall | Boundary recall |
|---|---|---|---|
| Random | 0.700 | 0.697 | 0.698 |
| APT (entropy) | 0.719 | 0.808 | 0.833 |
| SPLIT (Ours) | 0.714 | 0.907 | 0.945 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyper-parameter (default) | Values swept | Test acc (%) |
|---|---|---|
| PPO clip ratio ( ) | / / | 86.49 / 86.49 / 86.48 |
| Rollouts ( ) | / / | 86.49 / 86.49 / 86.33 |
| PPO epochs ( ) | / / | 86.50 / 86.49 / 83.28 |
| Warm-start balancer rate ( ) | / / | 86.44 / 86.49 / 86.54 |
| Backbone LR ( ) | / / / | 86.55 / 86.49 / 85.97 / 85.73 |
| Policy learning rate | |||
|---|---|---|---|
| 85.2 / 0.72 | 86.3 / 0.81 | 86.2 / 0.84 | |
| 86.1 / 0.62 | 86.4 / 0.58 | 86.1 / 0.51 | |
| 86.3 / 0.55 | 86.1 / 0.52 | 86.1 / 0.51 | |
| RL iterations | (warm-start only) | |||
|---|---|---|---|---|
| COCO mAP | 58.06 | 58.20 | 58.52 | 58.84 |
| Warm-start iters | RL iters | Total iters | Final mIoU |
|---|---|---|---|
| 59.34 | |||
| 59.39 | |||
| 59.38 |
| Dataset | Dense | SPLIT (tok. frac.) | APT fixed- (tok. frac.) | APT recal. |
|---|---|---|---|---|
| CIFAR-100 (probe) | 89.87 | 89.26 (0.795) | 89.60 (0.831) | 89.49 |
| ImageNet-V2 | 79.13 | 79.07 (0.90) | 78.93 (0.89) | 78.93 |
| ImageNet-Sketch | 61.14 | 61.22 (0.74) | 56.82 (0.50) | 60.73 |
| ImageNet-R | 77.19 | 77.06 (0.83) | 75.71 (0.74) | 76.42 |
| ImageNet-A | 64.45 | 64.37 (0.90) | 63.28 (0.89) | 63.60 |
| Noise | SPLIT acc @ tok. frac. | APT acc @ tok. frac. |
|---|---|---|
| 0.00 | 87.7 @ 0.53 | 87.5 @ 0.53 |
| 0.05 | 86.5 @ 0.53 | 86.7 @ 0.62 |
| 0.10 | 84.9 @ 0.54 | 85.3 @ 0.75 |
| 0.15 | 83.0 @ 0.56 | 82.8 @ 0.92 |
| 0.20 | 80.1 @ 0.59 | 79.3 @ 0.95 |
| Config | ms/iter | ms/img | Iters | Total |
| Vanilla (dense) | 963 | 60 | 3000 | 48.2 min |
| APT finetune | 809 | 51 | 3000 | 40.45 min |
| SPLIT warm-start | 910 | 57 | 1500 | 22.8 min |
| SPLIT PPO phase | 1186 | 74 | 1500 | 29.6 min |
| SPLIT total | — | — | 3000 | 52.4 min |
| Method | Token fraction | Thing recall | Boundary recall |
|---|---|---|---|
| Random | 0.700 | 0.697 | 0.698 |
| APT (entropy) | 0.719 | 0.808 | 0.833 |
| SPLIT, warm-start only | 0.708 | 0.887 | 0.934 |
| SPLIT, full | 0.714 | 0.907 | 0.945 |
| Decision Network Architecture | Param Count | % Tokens | Val Acc (%) |
|---|---|---|---|
| 1.115M | 89.59 | 87.69 | |
| 1.116M | 90.08 | 87.74 | |
| Cross Attention 3.2 | 1.115M | 89.53 | 87.81 |
Vision Transformer
Vision Transformers (ViTs) use self-attention over image patches to learn visual representations for recognition, segmentation, and other tasks.
See all 386 papersVision Transformers Need Better Token Interaction
[CLS] features remain complementary for dense prediction. These observations suggest that the goal should not be to remove global context, but to make token interactions more selective. We therefore study sparse attention as a minimal intervention, replacing softmax attention with entmax-1.5 while preserving global token connectivity. On DINOv1 ViT-S/16 trained for 200 epochs on ImageNet-1K, this change preserves ImageNet linear probing accuracy and substantially improves semantic segmentation performance: VOC mIoU increases from 42.80 to 48.78, ADE20K from 19.85 to 21.97, and Cityscapes from 36.79 to 37.87. These results suggest that selective token mixing is a simple and effective bias for improving dense ViT representations.