Learning Hierarchical Patch Splitting Policies for Faster Vision Transformers
Organizations: Carnegie Mellon University
Abstract
Vision Transformers (ViTs) are bottlenecked by the length of their input token sequence. Standard patchification fixes this length with a single patch size applied across the image, regardless of regional content. Prior work on adaptive patch sizing reduces token count by assigning coarser patches to visually simple regions, but relies on noisy pixel-level heuristics such as local entropy to make patchification decisions. We argue that this compute allocation should be grounded in the model's interpretation of task relevant information, not in raw pixel statistics. We introduce Split Policy Learned Image Tokenization which learns to allocate token resolution based on what the model finds meaningful, producing shorter token sequences that are better matched to task-relevant content. SPLIT matches the accuracy of full-resolution ViT classification baselines while using 30% fewer tokens. The same gains hold on dense prediction: SPLIT achieves competitive accuracy on COCO and ADE20K, demonstrating that semantically aligned patchification generalises well beyond classification.
Figures & tables
| Method | % Tokens | Object recall | Boundary recall |
|---|---|---|---|
| Random | 0.700 | 0.697 | 0.698 |
| APT (entropy) | 0.719 | 0.808 | 0.833 |
| SPLIT (Ours) | 0.714 | 0.907 | 0.945 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyper-parameter (default) | Values swept | Test acc (%) |
|---|---|---|
| PPO clip ratio ( ) | / / | 86.49 / 86.49 / 86.48 |
| Rollouts ( ) | / / | 86.49 / 86.49 / 86.33 |
| PPO epochs ( ) | / / | 86.50 / 86.49 / 83.28 |
| Warm-start balancer rate ( ) | / / | 86.44 / 86.49 / 86.54 |
| Backbone LR ( ) | / / / | 86.55 / 86.49 / 85.97 / 85.73 |
| Policy learning rate | |||
|---|---|---|---|
| 85.2 / 0.72 | 86.3 / 0.81 | 86.2 / 0.84 | |
| 86.1 / 0.62 | 86.4 / 0.58 | 86.1 / 0.51 | |
| 86.3 / 0.55 | 86.1 / 0.52 | 86.1 / 0.51 | |
| RL iterations | (warm-start only) | |||
|---|---|---|---|---|
| COCO mAP | 58.06 | 58.20 | 58.52 | 58.84 |
| Warm-start iters | RL iters | Total iters | Final mIoU |
|---|---|---|---|
| 59.34 | |||
| 59.39 | |||
| 59.38 |
| Dataset | Dense | SPLIT (tok. frac.) | APT fixed- (tok. frac.) | APT recal. |
|---|---|---|---|---|
| CIFAR-100 (probe) | 89.87 | 89.26 (0.795) | 89.60 (0.831) | 89.49 |
| ImageNet-V2 | 79.13 | 79.07 (0.90) | 78.93 (0.89) | 78.93 |
| ImageNet-Sketch | 61.14 | 61.22 (0.74) | 56.82 (0.50) | 60.73 |
| ImageNet-R | 77.19 | 77.06 (0.83) | 75.71 (0.74) | 76.42 |
| ImageNet-A | 64.45 | 64.37 (0.90) | 63.28 (0.89) | 63.60 |
| Noise | SPLIT acc @ tok. frac. | APT acc @ tok. frac. |
|---|---|---|
| 0.00 | 87.7 @ 0.53 | 87.5 @ 0.53 |
| 0.05 | 86.5 @ 0.53 | 86.7 @ 0.62 |
| 0.10 | 84.9 @ 0.54 | 85.3 @ 0.75 |
| 0.15 | 83.0 @ 0.56 | 82.8 @ 0.92 |
| 0.20 | 80.1 @ 0.59 | 79.3 @ 0.95 |
| Config | ms/iter | ms/img | Iters | Total |
| Vanilla (dense) | 963 | 60 | 3000 | 48.2 min |
| APT finetune | 809 | 51 | 3000 | 40.45 min |
| SPLIT warm-start | 910 | 57 | 1500 | 22.8 min |
| SPLIT PPO phase | 1186 | 74 | 1500 | 29.6 min |
| SPLIT total | — | — | 3000 | 52.4 min |
| Method | Token fraction | Thing recall | Boundary recall |
|---|---|---|---|
| Random | 0.700 | 0.697 | 0.698 |
| APT (entropy) | 0.719 | 0.808 | 0.833 |
| SPLIT, warm-start only | 0.708 | 0.887 | 0.934 |
| SPLIT, full | 0.714 | 0.907 | 0.945 |
| Decision Network Architecture | Param Count | % Tokens | Val Acc (%) |
|---|---|---|---|
| 1.115M | 89.59 | 87.69 | |
| 1.116M | 90.08 | 87.74 | |
| Cross Attention 3.2 | 1.115M | 89.53 | 87.81 |
Vision Transformer
Vision Transformers (ViTs) use self-attention over image patches to learn visual representations for recognition, segmentation, and other tasks.
See all 386 papers