Split inference partitions a Vision Transformer between a resource-constrained edge device and a server. However, transmitting intermediate tokens across the communication channel is costly. Deploying fine-tuned backbones to every endpoint becomes expensive as well. We study token reduction for split inference under a frozen backbone, where only a small adapter is shipped. We observe that the CLS token plays a dual role: it acts as the global summary the receiver needs for classification, and it serves as a signal on which patches to keep. We propose Consommé, a method utilizing both characteristics. There, an information bottleneck compresses the CLS token into a compact latent which is transmitted in place of the full CLS token and conditions a learned patch gate merging patches into exactly k tokens via soft assignment. It reaches any budget in a single step, thus escaping the per-layer cap of bipartite matching (BPM). On DeiT-S/T, it leads all methods at aggressive token budgets and remains operable at shallow splits where BPM cannot operate, while matching fine-tuned methods at gentle budgets at a fraction of their deployment cost.
Figures & tables
Tier
Method
Requires
Additional
Fine-Tuning
Parameters
1
ToMe [ 4 ]
No
No
ToFu [ 19 ]
No
No
2
(S/L-)HAC-ToMe [ 14 ]
No
Yes
DiffRate [ 6 ]
No
Yes
Consommé (Ours)
No
Yes
Table 1: Methods and their respective tiers. The increased footprint describes methods which have an additional overhead due to additional parameters.
Figure 1: Overview of Consommé. By leveraging the CLS token’s information via an IB, it serves as a selection mechanism for the soft-assignment merging gate in order to reduce the number of tokens to be transmitted.
Figure 2: Consommé’s visualizations of different token budgets performed on DeiT-S on different split layers s∈{3,6,9} on samples from the ImageNet-1K validation set. The later the token reduction is performed, the more the tokens focus on the actual content and not on the background, thus identifying the more informative tokens which also aligns with the human-visual distinctive features. The biggest group which is non-colored corresponds semantically to a background group.
Tier
Method
Top-1 Acc (%)
k =8
k =16
k =32
k =64
DeiT-S
79.8
1
ToMe
70.39 ∗
75.39 ∗
77.3 ∗
78.41 ∗
ToFu
68.18 ∗
73.47 ∗
75.75 ∗
77.4 ∗
2
S-HAC-ToMe
70.83 ∗
75.72 ∗
77.54 ∗
78.56 ∗
L-HAC-ToMe
72.2 ∗
76.2 ∗
77.9 ∗
78.7 ∗
Table 2: Results of different token reduction methods on DeiT-S and DeiT-T applied to split layer 9. ∗ indicates that the exact token count was not reached. We report the performance of the tier-3 methods only on DeiT-S as a cost-agnostic reference point given their substantially higher deployment cost for SI settings and thus omit them for DeiT-T. Consommé achieves superior results for reduced token counts.
Figure 3: We evaluate the classification accuracy on DeiT-S and DeiT-T for a token budget of k=64 and k=8 , while varying the split layer. Consommé exhibits much better accuracy retention at shallow splits.
Model
k
Additional Param (M)
Backbone
Edge
Server
Total
%
DeiT-S
8
0.59
0.22
0.82
22
3.7
DeiT-S
64
1.97
0.22
2.2
22
10
DeiT-T
8
0.21
0.06
0.26
5
5.2
ViT-B
8
1.1
0.53
1.63
86
2
Table 3: Consommé’s deployment parameter counts on different token budgets (k) on different models compared to the backbone’s parameter count.
Ablation
k =8
k =64
Consommé (full)
78.4
79.2
No-Reconstruction
75.93
79.09
CLS Conditioning
76.06
79.02
Static Anchor
74.74
78.23
Table 4: Results of incremental component removal on DeiT-S.
Tier
Method
Top-1 Acc (%)
k =8
k =16
k =32
k =64
ViT-B
85.1
1
ToMe
60.88 ∗
75.98 ∗
80.7 ∗
83.03 ∗
ToFu
70.96 ∗
78.7 ∗
81.79 ∗
83.18 ∗
2
S-HAC-ToMe
65.1 ∗
77.23 ∗
81.41 ∗
83.22
L-HAC-ToMe
69.81 ∗
78.61 ∗
82.07 ∗
83.7 ∗
Table 5: Results of different token reduction methods on ViT-B applied to split layer s=9 . ∗ indicates that the exact token count was not reached. Consommé achieves superior results on all reported token budgets k .
Split s
Transmitted patch tokens
k =8
k =16
k =32
k =64
3
-
-
32
64
6
0 8
16
34
64
9
0 9
18
34
61
Table 6: Number of patch tokens actually transmitted by the BPM-based methods (ToMe, ToFu, S-HAC, L-HAC). The CLS token is not counted. Because bipartite matching removes at most half the tokens per layer, the reachable count is higher than the token budget k on s=3 . We report the nearest reachable count. - marks budgets that are unreachable at that split.
Tier
Method
Top-1 Acc (%)
k =8
k =16
k =32
k =64
DeiT-S
79.8
1
ToMe
19.63
53.22
70.72 ∗
76.27
ToFu
42.39
62.49
73.27 ∗
77.01
2
S-HAC-ToMe
26.42
57.4
71.95 ∗
76.69
L-HAC-ToMe
44.7
63.3
73.9 ∗
77.5
Table 7: Split 6 results of different token reduction methods on DeiT-S, DeiT-T, and ViT-B. ∗ indicates that the exact token count was not reached by the method. On all models, methods from all three tiers suffer from heavy performance degradation, with some even reaching single-digit top-1 accuracy. Consommé achieves superior results when reducing the token budget.
Tier
Method
Top-1 Acc (%)
k =8
k =16
k =32
k =64
DeiT-S
79.8
1
ToMe
-
-
19.1
61.57
ToFu
-
-
52.18
72.63
2
S-HAC-ToMe
-
-
35.58
67.54
L-HAC-ToMe
-
-
51.67
71.9
Table 8: Results on split layer 3. ∘ marks that the model did not converge. - shows that the token count was not reached by the method. Consommé satisfies all token budgets k on split layer s=3 , whereas all BPM methods, as well as some training-based methods, fail.
DeiT-T
DeiT-S
ViT-B
Token dimension D / latent d
192 / 96
384 / 192
768 / 256
Per image (values)
Full activations (N+1)D
37,824
75,648
151,296
Consommé d+kD
1,632
3,264
6,400
Compressed Baseline Method (1+k)D
1,782
3,456
6,912
Consommé deployment (parameters)
Table 9: Transmission and deployment cost at token budget k=8 . The Break-even point is the one-time shipping cost divided by the per-image saving against the stated reference, assuming that parameters and activations are shipped at equal numerical precision. Consommé’s decoder size and cost are split-layer invariant, whereas the tier-3 cost grows as the split layer moves earlier.
Tokens Kept
k
τ
Acc (%)
Mean
Min
Max
8
0.0
78.40
8.00±0.00
8
8
0.1
78.40
7.97±0.19
5
8
0.5
78.40
7.77±0.58
3
8
1.0
78.36
7.43±0.91
2
8
2.0
78.03
6.62±1.29
2
8
Table 10: Token pruning on DeiT-S at split s=9 . Anchor groups with a summed assignment weight less than τ tokens are pruned and not transmitted.