Aperture: Merge-Consistent Rotary States for Compressed Tokens

Yuhao Du; Shunian Chen

cs.AISep 29, 2026

Aperture: Merge-Consistent Rotary States for Compressed Tokens

Authors: Yuhao Du, Shunian Chen

Organizations: The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute

Abstract

Token compression combines content from several positions, yet rotary position embeddings usually assign the merged token one coordinate. We ask what positional information must survive later merges. Aperture stores Fourier moments of the token's weighted support at the model's rotary frequencies. We prove that these moments have minimal real dimension among continuous states sufficient for the selected expected rotary interactions. Represented mass makes updates additive; attention normalisation remains a separate readout choice. Uniform intervals give a centre rotation times a sinc gain. We characterise when centres determine interval widths and construct matched examples where they do not. Numerical checks verify the weighted-support implementation. In trained temporal readers, compression transfer varies with gain calibration and feature placement. In a prespecified native video question-answering comparison, stored support reaches $65.63\%$ accuracy versus $67.12\%$ for the deployed merging rule. These results separate exact positional preservation under compression from downstream benefit.

Figures & tables

Figure 1: Position follows the represented support. A token merger combines content and positional moments with the same represented masses. (a) Different merge trees produce the same state for the same weighted support. (b) A centre summary is also independent of the merge tree, but generally differs from averaging the original rotary states; the marked chord is the information a centre discards.

Figure 2: From token support to rotary gain. (a) Retention leaves intervals of different widths and gaps after eviction; centres alone omit their extent (schematic). (b) A uniform interval contributes the signed gain

\operatorname{sinc}(\theta w/2)

with envelope

\min(1,2/(|\theta|w))

. The dot marks the hard-cutoff control’s threshold

\theta w=\pi

, where the exact gain is still

2/\pi

. (c) Positional kernels

K(\Delta)=\sum_{m}g_{m}(w_{i})g_{m}(w_{j})\cos(\theta_{m}\Delta)

, normalised by

K(0)

, for narrow, wide and mixed supports. Panels (b)–(c) use the model’s bank.

Readout	Tree discrepancy	Affine defect
Centre RoPE	$<10^{-12}$	$1.347$
exact Aperture	$<10^{-12}$	$<10^{-12}$
Aperture RMS	$<10^{-12}$	$0.432$
MN-sinc	$<10^{-12}$	$1.831$
Hard cutoff	$<10^{-12}$	$0.577$

Table 1: Tree equality and affinity are different. Float64 checks on four adjacent intervals of widths

(0.2,0.7,0.8,1.3)

, with mass proportional to width. Tree discrepancy recomputes each readout from the final associative summary. Affine defect compares the parent code with the weighted child codes; it is a numerical witness, not an accuracy score.

Observation	$N{=}8$	$N{=}9$
Relative offsets only	1	1
Absolute centres	1	1
Absolute centres, left boundary	0	0
Absolute centres, total span	1	0

Table 2: Nullity of the width equations under four observation models (anchor audit of section B.2 ).

Figure 3: A width change invisible to centre observations. Two valid tilings have the same centres and different boundaries (

\delta=\pm 0.3

on the alternating mode). The marked query point belongs to the first interval only in the lower chain; content can be identical in this constructed pair. Which observations break the ambiguity is algebraic: Proposition 1 states the boundary and parity dependence, and Table 3.1 tabulates the nullity of the width equations for

N=8

and

N=9

Figure 4: Retention conditions and transfer under content deletion. (a) The four conditions: coarsen (C) merges adjacent supports into a plain tiling; dilate (D) widens them into overlap; mask (M) zeroes payloads in place (hatched); evict (E) deletes slots (gaps). (b) Coarsen-trained, coarsen-tested accuracy per arm on the five retained checkpoints; bars are 95%

t

intervals, and the

y

axis is truncated to

[0.40,0.58]

. (c) Accuracy minus the centre-only arm in the six train

\to

eval cells whose evaluation deletes content (cell codes are the panel-a conditions, train

\to

eval), paired within seed; bars are one seed standard deviation, and a black rim marks cells below centre on every seed (width feature 3 of 6; Aperture RMS 2 of 6; exact Aperture 0 of 6). Full cells and sample sizes are in section D.10 .

Paired accuracy contrast	3B	7B
exact Aperture $-$ centre only	$+0.0259\;[-0.0001,+0.0518]$	$+0.0018\;[-0.0214,+0.0249]$
exact Aperture $-$ width feature	$+0.0114\;[-0.0134,+0.0362]$	$+0.0256\;[+0.0004,+0.0508]$
exact Aperture $-$ MN-sinc	$+0.0597\;[+0.0294,+0.0900]$	$+0.0549\;[+0.0298,+0.0800]$
Matched-RMS $-$ MN-sinc	$+0.0214\;[-0.0140,+0.0568]$	$+0.0073\;[-0.0198,+0.0345]$

Table 3: Large-model contrasts at the primary held-out budget. Mean paired difference and 95%

t

interval on 24 matched seeds per comparison. Intervals are unadjusted and in accuracy units; family-wise conclusions are stated in the text. Matched-RMS rescales exact Aperture to the temporal gain RMS of MN-sinc.

Comparison	Difference	95% seed interval	95% crossed bootstrap
Qwen2.5-Omni-3B: unseen policy	$+0.0001$	$[-0.0068,+0.0070]$	$[-0.0145,+0.0153]$
Qwen2.5-Omni-3B: unseen rate	$+0.0007$	$[-0.0078,+0.0092]$	$[-0.0138,+0.0172]$
Qwen2.5-Omni-3B: width feature	$-0.0007$	$[-0.0062,+0.0049]$	$[-0.0150,+0.0168]$
Qwen2.5-Omni-7B: unseen policy	$+0.0007$	$[-0.0028,+0.0042]$	$[-0.0125,+0.0153]$
Qwen2.5-VL-7B: spatial / PPE	$-0.0149$	$[-0.0175,-0.0123]$	$[-0.0228,-0.0067]$

Table 4: Native interface contrasts, in accuracy units. Stored support minus the named reference (centre-only unless specified). Onset uses ten paired seeds and 1,024 items; spatial question answering uses five seeds and 4,000 items. Seed

t

intervals condition on the test set; crossed bootstrap intervals resample seeds and population clusters. Both 95% intervals are unadjusted; Holm correction covers the five seed tests.

Figure 5: Native interface comparisons. (a) The five contrasts of the frozen manifest (stored-support accuracy minus the named reference): effect, 95% paired-seed

t

interval, and per-seed differences as ticks. The filled marker (

\star

) is the only contrast significant after Holm correction (adjusted

p=4.7\times 10^{-4}

). (b) Native merging per arm – MVBench at the deployed selector (primary cell), MVBench

2{\times}2

grid, Video-MME – as seed means with 95%

t

intervals over five shared seeds;

y

axis truncated to

[0.605,0.708]

. Hollow markers: unadapted 3B references (PPE at retention

0.45

; dense, complete media), not adapted-7B comparators.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Code	A1: affinity	A2: translation	Violated
Aperture	$1.1\times 10^{-14}$	$4.9\times 10^{-13}$	—
Centre only	$2.0$	$5.7\times 10^{-13}$	A1
Hard cutoff	$6.5\times 10^{-1}$	$2.8\times 10^{-14}$	A1
Aperture RMS	$1.6$	$5.3\times 10^{-13}$	A1
MN-sinc	$1.5\times 10^{2}$	$5.3\times 10^{-13}$	A1
Width-capped phase	$1.5$	$1.7$	A1+A2

Appendix

Table 5: Numerical checks of support affinity and translation covariance. Largest residual in float64 over sampled supports. A1 tests merging equal-width adjacent intervals; A2 tests a common translation. Values below

10^{-9}

are numerical zero. A1 is a necessary special case of affinity, not a test over all measures. The point-support limit holds for all tested codes to

1.1\times 10^{-16}

. MN-sinc and the width-capped phase control are defined in Appendix F .

Support domain	Tree / quadrature error	Interval-summary error
1D mixtures	$<10^{-12}$	$0.764$
2D mixtures	$<10^{-12}$	$0.699$

Appendix

Table 6: General-support implementation audit. Maximum absolute raw-moment tree/quadrature errors over 32 mixtures per dimension, seven weighted atoms and seven frequencies per mixture. Interval-summary error is measured on normalised moments using one uniform box over the enclosing bounds.

Crossing / payload	Cell accuracies	Paired effects
Overlap / metadata Centre	TR: 0.526 [0.502, 0.550] TM: 0.646 [0.628, 0.665] GR: 0.526 [0.502, 0.550] GM: 0.646 [0.628, 0.665]	G: 0.000 [0.000, 0.000] M: 0.121 [0.113, 0.128] I: 0.000 [0.000, 0.000]
Overlap / metadata Stored support	TR: 0.521 [0.507, 0.536] TM: 0.636 [0.622, 0.650] GR: 0.523 [0.508, 0.537] GM: 0.640 [0.623, 0.656]	G: 0.002 [0.000, 0.005] M: 0.116 [0.110, 0.122] I: 0.002 [-0.002, 0.005]
Gap / metadata Centre	TR: 0.527 [0.506, 0.548] TM: 0.651 [0.631, 0.670] GR: 0.527 [0.506, 0.548] GM: 0.651 [0.631, 0.670]	G: 0.000 [0.000, 0.000] M: 0.124 [0.117, 0.131] I: 0.000 [0.000, 0.000]
Gap / metadata Stored support	TR: 0.524 [0.507, 0.542] TM: 0.641 [0.626, 0.657] GR: 0.523 [0.505, 0.541] GM: 0.642 [0.625, 0.659]	G: 0.000 [-0.002, 0.001] M: 0.118 [0.112, 0.125] I: 0.002 [-0.001, 0.005]
Overlap / physical Centre	TR: 0.521 [0.506, 0.537] TM: 0.651 [0.635, 0.666] GR: 0.524 [0.510, 0.539] GM: 0.656 [0.642, 0.670]	G: 0.004 [0.001, 0.006] M: 0.130 [0.124, 0.136] I: 0.002 [-0.002, 0.006]
Overlap / physical Stored support	TR: 0.524 [0.514, 0.534] TM: 0.646 [0.630, 0.662] GR: 0.529 [0.519, 0.540] GM: 0.652 [0.636, 0.668]	G: 0.006 [0.003, 0.009] M: 0.122 [0.113, 0.131] I: 0.001 [-0.006, 0.007]

Appendix

Table 7: Real-video crossings. Four-cell accuracy and paired geometry (G), mask (M) and interaction (I) effects with 95% seed intervals over ten prespecified seeds and 2,048 fixed test items. Masks retain half the slots and protect onset-bearing evidence. TR/TM denote tiling with retained/masked content; GR/GM denote changed geometry. The compact display uses centre and stored-support readers; the full grid is exported separately.

Arm	Overall	80-cell range	Policy cell	Rate cell
G2: Qwen2.5-Omni-3B (13 arms)
centre	0.084	0.083–0.084	0.084 [0.081, 0.087]	0.083 [0.080, 0.087]
stored_support	0.083	0.083–0.084	0.084 [0.079, 0.090]	0.084 [0.078, 0.090]
exact_interval †	0.088	0.087–0.088	0.088 [0.080, 0.096]	0.088 [0.080, 0.096]
learned_interval	0.084	0.084–0.085	0.084 [0.082, 0.087]	0.085 [0.083, 0.087]
start_end	0.083	0.083–0.084	0.083 [0.079, 0.088]	0.083 [0.078, 0.088]

Appendix

Table 8: Complete G2/G3 adaptation grid. Per-arm accuracy over the full prespecified ladder (five policies

\times

four retention fractions

\times

four frame rates

=80

cells; 1,024 fixed test items per cell; ten seeds). Overall averages all 800 cell-seed accuracies; 80-cell range spans the per-cell seed means. The policy and rate columns are the prespecified primary cells ( noncontiguous keep

1/4

@ 8 fps; contiguous keep

1/2

@ 16 fps) with 95% seed

t

intervals; for G3 only the policy cell is a prespecified contrast. Twelve-way chance is 0.0833. The dagger marks the exact-Aperture seed-9 checkpoint analysed with the full distributions in Figure 6 and the accompanying text.

Figure 6: Onset adaptation remains near chance. (a) Accuracy of all 16,800 evaluation cells (13 3B arms and 8 7B arms, 10 seeds, 80 cells per checkpoint: five policies $\times$ four retained fractions $\times$ four frame rates) against twelve-way chance (dashed line, $1/12$ ). The blue subset marks the exact-Aperture seed-9 checkpoint; the text reports its above-chance accuracy and the unresolved prespecified contrasts. (b) Per-arm means over all cells, with the range of the ten per-seed means as whiskers. A training-batch probe at learning rate $10^{-3}$ memorises 16 items within 325 steps; this verifies fitting that batch, not learning the full task.

arm	final train loss	test mean	$\times$ chance
centre-only ( centre )	$2.4950$	$0.0866$	$1.0395$
stored support ( stored_support )	$2.4898$	$0.0862$	$1.0342$
exact Aperture ( exact_interval )	$2.4903$	$0.0848$	$1.0172$
learned interval ( learned_interval )	$2.4922$	$0.0850$	$1.0202$
PPE-ranked ( ppe_ranked )	$2.4956$	$0.0860$	$1.0324$

Appendix

Table 9: Exploratory budget probe: per-arm endpoints. Means over 10 seeds at learning rate

10^{-3}

; the protocol is otherwise identical to the frozen 3B grid; this analysis is outside the prespecified family. Final train loss is the per-seed last logged value, averaged;

\ln 12=2.4849

is the twelve-way-chance loss. The test mean is over all 80 cells of the frozen test inventory.

Model / retention / rate	Readout	Median / p90	Peak	State
3B / 0.5 / 16 fps	Centre-only	6.61 / 6.87	24.08	48
3B / 0.5 / 16 fps	Stored support	6.60 / 6.88	24.08	17
3B / 0.25 / 8 fps	Centre-only	5.19 / 5.46	22.48	48
3B / 0.25 / 8 fps	Stored support	5.20 / 5.44	22.48	4.5
7B / 0.25 / 8 fps	Centre-only	5.19 / 5.42	34.71	48
7B / 0.25 / 8 fps	Stored support	5.17 / 5.42	34.71	4.5

Appendix

Table 10: Classification request cost on EPIC onset. Descriptive pooled timings for ten seeds and 1,280 requests per arm in each cell; observed peak allocated memory. All models are Qwen2.5-Omni on A800 hardware. Charged state is a 48 KiB position-ID bound for centre-only and retained membership/moment buffers for stored support. Medians and p90 are seconds, peak is GiB, and state is KiB.

Method	Accuracy	Med./p90	Req./s	GPU	KV/state
Centre	0.676	930.8/4144.2	0.46	25.51	320.41/0.527
Stored support	0.692	1333.8/8622.2	0.20	25.51	320.41/0.527
PPE	0.777	984.1/3194.9	0.30	25.51	320.41/4.237

Appendix

Table 11: Generative execution cost. The spatial panel compares Centre, Stored support and native PPE at one-quarter retention. Timing begins after media preprocessing and covers model execution through autoregressive decoding. Accuracy uses the first 256 items of the fixed evaluation order, paired across arms. Latency is median/p90 in milliseconds, throughput is requests/s, GPU is observed peak allocated GiB, and KV/state use MiB. Raw traces also retain time to first token (TTFT), generated-token throughput and reserved memory. Timing and memory summaries are descriptive.

Readout	Mean tokens	Median tokens	Requests at cap
Centre-only	4.36	2	8
Stored support	12.86	3	79
PPE	8.95	3	56

Appendix

Table 12: Generated lengths in the MVBench timing subset. Descriptive summaries of five seeds on the same 256 items (1,280 requests per arm), with a 128-token cap. Repeated item/seed requests are not treated as independent inferential replicates.

Code	Contrast	Estimate	95% interval
Aperture	$L_{1}$	$-0.0677$	$[-0.0808,-0.0546]$
Aperture	$L_{2}$	$-0.0077$	$[-0.0208,+0.0054]$
Aperture	$D-M$	$+0.0600$	$[+0.0391,+0.0798]$
Aperture RMS	$L_{1}$	$-0.0939$	$[-0.1069,-0.0808]$
Aperture RMS	$L_{2}$	$-0.0072$	$[-0.0203,+0.0058]$
Aperture RMS	$D-M$	$+0.0867$	$[+0.0663,+0.1070]$

Appendix

Table 13: Uncertainty in four-policy contrasts. The

L_{1}

and

L_{2}

intervals use cell-wise Welch–Satterthwaite variances;

D-M

intervals use 20,000 independent seed resamples within each condition. All contrasts are in accuracy units, with 15 runs per cell.

Figure 7: Sampling-rate and retention transfer. Test accuracy of the same checkpoints under token eviction and changes in video frame rate. Points show seed means and bars one seed standard deviation; every arm has

n=15

seeds except token index (

n=5

). Arm colours match Figure 4 ; token index denotes the per-modality index arm. Exact values and all arms are in Table 14 and Table 15 .

Code	$n$	8 fps	32 fps	Minimum at 32 fps
Token index	5	$0.967\;(0.068)$	$0.238\;(0.077)$	0.174
Interleaved index	5	$1.000\;(0.000)$	$0.292\;(0.079)$	0.195
Centre only	15	$1.000\;(0.002)$	$0.932\;(0.117)$	0.677
Width feature	15	$1.000\;(0.001)$	$0.845\;(0.228)$	0.333
Aperture	15	$1.000\;(0.002)$	$0.951\;(0.092)$	0.662
Aperture RMS	15	$1.000\;(0.000)$	$0.997\;(0.006)$	0.981

Appendix

Table 14: Synthetic sampling-rate transfer. Mean test accuracy (seed s.d.) from the same eviction-trained checkpoints. The training rate is 8 fps; 32 fps is held out. The last column reports the lowest seed accuracy. Chance is 0.125.

Trained on tilings
Code	$n$	Retain 1.00	Retain 0.50	Retain 0.25
Token index	5	$0.999\;(0.002)$	$0.370\;(0.048)$	$0.354\;(0.048)$
Interleaved index	5	$0.999\;(0.002)$	$0.347\;(0.065)$	$0.339\;(0.068)$
Centre only	5	$1.000\;(0.000)$	$0.442\;(0.067)$	$0.423\;(0.061)$
Width feature	5	$1.000\;(0.000)$	$0.442\;(0.062)$	$0.422\;(0.064)$
Aperture	5	$1.000\;(0.000)$	$0.431\;(0.069)$	$0.411\;(0.066)$

Appendix

Table 15: Synthetic retention evaluation. Mean test accuracy (seed s.d.) at 8 fps, grouped by training regime. Each row reports its own seed count. Training on tilings produces near-ceiling accuracy on the matching evaluation, which limits the sensitivity of that comparison.

Training	Code	$n$	32 fps	64 fps	128 fps
Single	Aperture	15	$0.979\;[0.147,1.000]$	$0.885\;[0.152,1.000]$	$0.750\;[0.131,1.000]$
Single	Learned gain	15	$0.971\;[0.290,1.000]$	$0.762\;[0.108,1.000]$	$0.574\;[0.201,1.000]$
Single	Width feature	15	$0.905\;[0.213,1.000]$	$0.824\;[0.134,0.991]$	$0.544\;[0.089,0.981]$
Covered	Aperture	15	$1.000\;[0.999,1.000]$	$0.994\;[0.465,1.000]$	$0.938\;[0.403,1.000]$
Covered	Learned gain	15	$1.000\;[0.981,1.000]$	$0.999\;[0.553,1.000]$	$0.988\;[0.184,1.000]$
Covered	Width feature	15	$1.000\;[0.997,1.000]$	$1.000\;[0.967,1.000]$	$0.909\;[0.641,1.000]$

Appendix

Table 16: Training coverage and rate transfer. Median test accuracy [minimum, maximum] across seeds. Single-rate training uses 8 fps; covered training samples 2, 4, 8, 16 and 32 fps. Both 64 and 128 fps are outside these training sets. The substantial ranges motivate the paired rank comparisons in the text.

Code	$n$	$b=1$	$b=2$	$b=4$	$b=8$
Centre only	24	$0.669\;(0.077)$	$0.739\;(0.059)$	$0.708\;(0.045)$	$0.310\;(0.025)$
Width feature	24	$0.695\;(0.062)$	$0.762\;(0.050)$	$0.722\;(0.038)$	$0.317\;(0.020)$
Aperture	48	$0.687\;(0.067)$	$0.753\;(0.056)$	$0.731\;(0.044)$	$0.346\;(0.013)$
MN-sinc	24	$0.677\;(0.080)$	$0.746\;(0.070)$	$0.674\;(0.055)$	$0.317\;(0.029)$
Width-capped phase	24	$0.696\;(0.083)$	$0.757\;(0.069)$	$0.733\;(0.053)$	$0.302\;(0.023)$
Aperture RMS	24	$0.668\;(0.082)$	$0.738\;(0.068)$	$0.714\;(0.052)$	$0.324\;(0.020)$

Appendix

Table 17: 3B budget evaluation. Bin accuracy, mean (seed s.d.); training budgets are

b=1,2

, the primary held-out budget is

b=4

, and

b=8

is supplementary. MN-sinc and width-capped phase are controls defined in Appendix F . Contrasts use matched seeds; row means use the stated

n

Code	$n$	$b=1$	$b=2$	$b=4$	$b=8$
Centre only	24	$0.732\;(0.047)$	$0.794\;(0.038)$	$0.754\;(0.028)$	$0.315\;(0.024)$
Width feature	24	$0.697\;(0.071)$	$0.761\;(0.060)$	$0.730\;(0.046)$	$0.320\;(0.033)$
Aperture	24	$0.709\;(0.073)$	$0.777\;(0.062)$	$0.756\;(0.048)$	$0.356\;(0.015)$
MN-sinc	24	$0.711\;(0.048)$	$0.771\;(0.046)$	$0.701\;(0.042)$	$0.337\;(0.022)$
Aperture RMS	24	$0.720\;(0.065)$	$0.780\;(0.058)$	$0.750\;(0.043)$	$0.321\;(0.026)$
Aperture, matched RMS	24	$0.714\;(0.061)$	$0.775\;(0.053)$	$0.708\;(0.039)$	$0.337\;(0.026)$

Appendix

Table 18: 7B budget evaluation. Bin accuracy, mean (seed s.d.); training budgets are

b=1,2

, the primary held-out budget is

b=4

, and

b=8

is supplementary. MN-sinc and width-capped phase are controls defined in Appendix F . Contrasts use matched seeds; row means use the stated

n

Budget	$b{=}1$	$b{=}2$	$b{=}4$	$b{=}8$
Aperture $-$ clock, $3$ B	$+0.0047$	$+0.0096$	$+0.0259$	$+0.0376$
Aperture $-$ clock, $7$ B	$-0.0218$	$-0.0176$	$+0.0021$	$+0.0197$
Aperture RMS $-$ clock, $3$ B	$-0.0026$	$-0.0021$	$+0.0063$	$+0.0064$

Appendix

Table 19: Supplementary budget comparison. Mean paired token-accuracy difference from the centre-only clock on 24 matched seeds. Aperture RMS normalises temporal gain energy and is a separate control from matching MN-sinc. Budgets 1 and 2 are used in training, 4 is the primary held-out budget, and 8 is an exploratory endpoint. Adjusted tests are reported in Appendix D.5 .

(a) Width reconstruction error / $\bar{w}$
Policy	Streams	Recursion	Linear, $L=1$	Linear, $L=16$
Contiguous tiling	2048	$1.78\times 10^{-14}$	$0.33$	$0.25$
Unanchored window	2048	$0.63$	$0.34$	$0.27$
Overlap, $25\%$	2048	$0.98$	$0.42$	$0.31$
Overlap, $50\%$	2048	$1.95$	$0.50$	$0.37$
Retain $75\%$	2048	$7.10$	$0.42$	$0.42$

Appendix

Table 20: Geometry and width reconstruction. (a) Width errors normalised by the original mean width. From 2,048 random 32-part streams, conditions retain the stated valid-stream counts. Recursion reports the median maximum error per stream using absolute centres. Linear probes report median token error from relative offsets within

L

hops, using disjoint stream partitions. (b) Jacobian ranks within fixed families of two-dimensional tilings; full rank gives local identification within that family. (c) Conditioning of the anchored recovery operator. These quantities describe different estimators and information sets.

	kernel of the width $\,\to\,$ offset map			minimax floor / $\bar{w}$
$N$	$\dim\ker D$	$\dim\ker E$	Identifiable	$L{=}1$	$L{=}4$	$L{=}16$	Full span
4	1	3.9	0.000	0.880	—	—	0.480
8	1	8.4	0.000	0.813	0.318	—	0.246
16	1	16.5	0.000	0.781	0.269	—	0.121
32	1	32.6	0.000	0.767	0.249	0.085	0.062
64	1	64.4	0.000	0.759	0.237	0.073	0.030

Appendix

Table 21: Identified sets from relative centre offsets.

D

maps the widths of an

N

-part tiling to adjacent offsets;

E

maps a

2N

-part tiling to offsets among its retained tokens. The reported nullity of

E

averages 194, 254, 256, 256 and 256 valid half-retention draws for increasing

N

. Identifiable is the fraction of surviving widths fixed by those offsets. Floors are mean identified-set half-diameters, normalised by

\bar{w}

. Dashes denote spans covered by the full-span column.

Regime	$N$	$L{=}1$	$L{=}2$	$L{=}4$	$L{=}8$	$L{=}16$	$R^{2}_{\max}$	$L_{\mathrm{sat}}$
Tiling	4	0.518	0.554	—	—	—	0.554	2
	8	0.580	0.660	0.676	—	—	0.676	4
	16	0.618	0.726	0.758	0.759	—	0.759	4
	32	0.648	0.764	0.812	0.815	0.813	0.815	4
	64	0.656	0.783	0.849	0.860	0.860	0.860	8
Eviction	4	0.120	0.131	—	—	—	0.131	2

Appendix

Table 22: Linear width prediction from nearby centre offsets. Held-out

R^{2}

on the segmentation ensemble, with offset span

L

in token hops. Eviction starts from

2N

parts and retains half on average.

L_{\rm sat}

is the smallest tested span within 0.01 of the maximum

R^{2}

. Dashes denote spans at or beyond the stream length. These are fitted prediction scores, distinct from the worst-case floors in Table 21 .

Policy	Support	Content	Aperture	Aperture RMS
Coarsen	Tiling	Retained	$+0.1696\;(0.0220)$	$+0.2049\;(0.0233)$
Dilate	Overlap	Retained	$+0.1833\;(0.0260)$	$+0.2132\;(0.0188)$
Mask	Tiling	Masked	$+0.1232\;(0.0322)$	$+0.1265\;(0.0378)$
Evict	Gaps	Removed	$+0.0942\;(0.0188)$	$+0.1038\;(0.0123)$

Appendix

Table 23: Width sensitivity in the geometry/content design. Test accuracy minus width-permuted accuracy, mean (seed s.d.);

n=15

per condition and code. The primary audit includes padding slots. The live-slot control gives the same ordering (Appendix D.11 ).

Coarsen
Code	$n$	Accuracy	$\Delta_{w}$
Token index	15	0.4434 (0.0210)	+0.0000 (0.0000)
Centre only	15	0.4643 (0.0166)	+0.0000 (0.0000)
Width feature	15	0.5214 (0.0242)	+0.2626 (0.0298)
Hard cutoff RMS	5	0.5213 (0.0309)	+0.1678 (0.0305)
Aperture	15	0.5174 (0.0210)	+0.1696 (0.0220)

Appendix

Table 24: Positional-code comparison within each retention policy. Test accuracy and within-model width sensitivity

\Delta_{w}

, mean (seed s.d.). Each row reports its own completed-run count. The RMS aperture and width-feature accuracy contrast is averaged over the four policies in the text.

Retained	Masked tokens	Centre only	Aperture RMS	$\Delta_{w}$
1	0.00	0.4643	0.5238	+0.2049 (0.0233)
0.875	1.34	0.4168	0.4748	+0.1316 (0.0394)
0.75	3.33	0.4137	0.4250	+0.0990 (0.0245)
0.5	7.14	0.4627	0.4932	+0.1265 (0.0378)
0.25	10.58	0.3750	0.3969	+0.1033 (0.0282)

Appendix

Table 25: Sensitivity across content-retention levels. Mean test accuracy for the centre-only and Aperture RMS readers; width sensitivity is mean (seed s.d.). Retained fractions below one use masking, with unchanged support geometry. There are 15 seeds at fractions 1 and 0.5, and five at each other level.

(a) Width sensitivity $\Delta_{w}$
Code	Coarsen	Dilate	Mask	Evict	Raw
Token index	$+0.000$ †	$+0.000$	$+0.000$	$+0.000$	—
Centre only	$+0.000$ †	$+0.000$	$+0.000$	$+0.000$	—
Width feature	$+0.266$ †	$+0.009$	$+0.054$	$+0.070$	—
Hard cutoff RMS	$+0.168$ †	$+0.011$	$+0.060$	$+0.040$	—
Aperture	$+0.162$ †	$-0.008$	$+0.024$	$+0.042$	—

Appendix

Table 26: Transfer from coarsen training. Each row uses five saved checkpoints evaluated on 1,024 test items.

\dagger

marks the training policy. Width sensitivity is a within-model mean; accuracy differences are paired by seed. Raw denotes unmerged tokens, for which the width-permutation contrast is not reported.

(a) Width sensitivity $\Delta_{w}$
Code	Coarsen	Dilate	Mask	Evict	Raw
Centre only	$+0.000$	$+0.000$ †	$+0.000$	$+0.000$	—
Width feature	$+0.273$	$+0.206$ †	$+0.058$	$+0.082$	—
Aperture	$+0.148$	$+0.168$ †	$+0.041$	$+0.034$	—
Aperture RMS	$+0.201$	$+0.217$ †	$+0.049$	$+0.038$	—
(b) Accuracy minus centre only, mean (seed s.d.)

Appendix

Table 27: Transfer from dilate training. Each row uses five saved checkpoints evaluated on 1,024 test items.

\dagger

marks the training policy. Width sensitivity is a within-model mean; accuracy differences are paired by seed. Raw denotes unmerged tokens, for which the width-permutation contrast is not reported.

(a) Width sensitivity $\Delta_{w}$
Code	Coarsen	Dilate	Mask	Evict	Raw
Token index	$+0.000$	$+0.000$	$+0.000$ †	$+0.000$	—
Centre only	$+0.000$	$+0.000$	$+0.000$ †	$+0.000$	—
Width feature	$+0.166$	$+0.010$	$+0.184$ †	$+0.101$	—
Hard cutoff RMS	$+0.085$	$-0.004$	$+0.106$ †	$+0.046$	—
Aperture	$+0.118$	$-0.005$	$+0.121$ †	$+0.052$	—

Appendix

Table 28: Transfer from mask training. Each row uses five saved checkpoints evaluated on 1,024 test items.

\dagger

marks the training policy. Width sensitivity is a within-model mean; accuracy differences are paired by seed. Raw denotes unmerged tokens, for which the width-permutation contrast is not reported.

(a) Width sensitivity $\Delta_{w}$
Code	Coarsen	Dilate	Mask	Evict	Raw
Centre only	$+0.000$	$+0.000$	$+0.000$	$+0.000$ †	—
Width feature	$+0.164$	$+0.003$	$+0.067$	$+0.148$ †	—
Aperture	$+0.113$	$-0.010$	$+0.047$	$+0.088$ †	—
Aperture RMS	$+0.125$	$-0.006$	$+0.069$	$+0.109$ †	—
(b) Accuracy minus centre only, mean (seed s.d.)

Appendix

Table 29: Transfer from evict training. Each row uses five saved checkpoints evaluated on 1,024 test items.

\dagger

marks the training policy. Width sensitivity is a within-model mean; accuracy differences are paired by seed. Raw denotes unmerged tokens, for which the width-permutation contrast is not reported.

(a) Padding and live-slot controls
Permutation	Moved	$\lVert\Delta w\rVert_{\mathrm{rms}}$	$\lVert\Delta g\rVert/\lVert g\rVert$	Coarsen	Dilate	Mask	Evict
All slots	0.855	1.036	0.325	+0.2033	+0.2168	+0.1234	+0.1092
Live only	0.809	0.890	0.266	+0.1746	+0.1975	+0.0891	+0.0670
(b) Fraction of live slots permuted
One swap	0.146	0.368	0.047	+0.0330	+0.0451	+0.0105	+0.0156
$25\%$	0.149	0.381	0.049	+0.0424	+0.0480	+0.0244	+0.0178

Appendix

Table 30: Width-permutation controls on fixed checkpoints. Aperture RMS, five checkpoints per policy; perturbation magnitudes average all 20 checkpoints. Moved is the fraction of live slots with a changed width; RMS width change is in seconds, and relative gain change is dimensionless. Policy columns report Aperture RMS width sensitivity on five seeds per policy. All-slot permutation includes padding; other modes operate on live slots. Extent-blind controls have zero sensitivity in every mode.

(a) Per-condition means
Cell	Sham gap	Real gap	Sham acc.	Real acc.	$\Delta$ acc.	Positive
Coarsen	$-0.0012$	$+0.2033$	0.4398	0.5199	$+0.0801$	5/5
Dilate	$-0.0061$	$+0.2168$	0.4197	0.5080	$+0.0883$	5/5
Mask	$-0.0031$	$+0.1234$	0.4318	0.4766	$+0.0447$	3/5
Evict	$+0.0031$	$+0.1092$	0.4004	0.4369	$+0.0365$	5/5
(b) Four-policy contrasts and raw $p$ -values

Appendix

Table 31: Training with informative and permuted widths. Top: Aperture RMS models with real widths or freshly permuted live widths, five matched seeds per condition. The last column counts seeds with a positive accuracy difference. Bottom: four-policy contrasts and raw

p

-values; the real-width sensitivity uses the full 15-seed grid, while sham sensitivity and accuracy differences use five seeds per cell. These comparisons assess width dependence but do not universally bound sensitivity to out-of-distribution inputs.

(a) Matched-policy evaluation
Policy	Avail.	$\Delta_{w}$	S.d.	Accuracy	Clock	Error	No query
Coarsen	0.0000	+0.2096	0.0093	0.5379	0.4463	0.00 s	0.036
Similarity merge	0.0000	+0.3113	0.0134	0.4900	0.4141	0.00 s	0.046
Evict	0.2363	+0.1084	0.0053	0.4646	0.4209	1.75 s	0.054
Salience evict	0.2227	+0.2473	0.0077	0.6480	0.5703	0.00 s	0.214
(b) Test accuracy under policy transfer

Appendix

Table 32: Retention-policy controls. (a) Aperture RMS, five seeds per policy; the centre-only accuracy uses one seed. Availability is the geometric width advantage defined in Appendix D.9 ; reconstruction error is the median anchored-recursion error at the answer token. No-query accuracy diagnoses content-selection shortcuts. (b) Mean Aperture RMS test accuracy under policy transfer; underlining marks the training condition. Similarity merging and salience eviction are ToMe- and H2O-inspired controls, respectively. Salience-based selection permits shortcuts, limiting comparisons of absolute accuracy.

Policy	Quantity	1 layer	2 layers	4 layers	8 layers
Coarsen	Clock accuracy	$0.3913$	$0.4753$	$0.4557$	$0.4551$
Coarsen	Width sensitivity	$0.2552$	$0.2256$	$0.1787$	$0.1755$
Evict	Clock accuracy	$0.4167$	$0.4176$	$0.4268$	$0.4528$
Evict	Width sensitivity	$0.0661$	$0.0498$	$0.1084$	$0.1328$

Appendix

Table 33: Depth with full-prefix attention. Mean test accuracy or within-model width sensitivity; at least

3

seeds per cell. Width sensitivity is measured on Aperture RMS.

Policy	Code	$W=4$	$W=8$	$W=16$	$W=32$
Coarsen	Aperture RMS	$0.505\;(0.016)$ [5]	$0.501\;(0.040)$ [5]	$0.512\;(0.012)$ [5]	$0.524\;(0.025)$ [5]
Coarsen	Centre only	$0.453\;(0.014)$ [5]	$0.471\;(0.045)$ [5]	$0.440\;(0.009)$ [5]	$0.460\;(0.023)$ [5]
Evict	Aperture RMS	$0.427\;(0.020)$ [5]	$0.405\;(0.023)$ [5]	$0.435\;(0.020)$ [5]	$0.460\;(0.031)$ [5]
Evict	Centre only	$0.375\;(0.031)$ [5]	$0.392\;(0.036)$ [5]	$0.430\;(0.017)$ [5]	$0.415\;(0.018)$ [5]

Appendix

Table 34: Attention-window sweep. Mean test accuracy (seed s.d.) [number of runs]. Four layers each attend within a window of

W

sequence positions. Variable token supports and interleaved modalities prevent a fixed conversion from this window to elapsed time. Width sensitivities in the text use only matching test audits.

Explore similar work

Progressive Cramming: Reliable Token Compression and What It Reveals
Jul 23, 2026Dmitrii Tarasov, Timofei Lashukov, Elizaveta Goncharova +1Token Compression Data Compression Methods
FusionBrain Lab, AXXX, Moscow, Russia · HSE University, Moscow, Russia · Artificial Intelligence Institute, Innopolis University, Innopolis, Russia
Allocation Before Ranking: Decoupled Token Compression for OmniLLMs
Aug 3, 2026Zhenghui Guo, Yilin Yang, Yuanbin Man +5Token Compression Data Compression Methods
Department of Computer Science, University of Houston · Department of Computer Science and Engineering, The University of Texas at Arlington
CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models
Aug 3, 2026Yu Chen, Xiaohong Li, Xiaole Wang +3Token Compression Video-Language Models
1Evermind · 2Shanda Group · 3Peking University

Jul 23, 2026, cs.CLJ/K move · Enter open · S save

Progressive Cramming: Reliable Token Compression and What It Reveals

Dmitrii Tarasov, Timofei Lashukov, Elizaveta Goncharova, Andrey Kuznetsov

FusionBrain Lab, AXXX, Moscow, Russia · HSE University, Moscow, Russia · Artificial Intelligence Institute, Innopolis University, Innopolis, Russia

Token Compression Data Compression Methods Token Embeddings Token Budget Allocation

Open paper AskPDF arXiv

Token cramming compresses sequences into learned embeddings with near-perfect reconstruction, but fixed token budgets and 99% accuracy thresholds leave it unclear whether residual errors reflect optimization failures or fundamental limits. We introduce progressive cramming, which grows the target prefix token-by-token, stopping only when reconstruction is no longer achievable within a fixed optimization budget. Progressive trajectories occupy low-dimensional structure in embedding space. Prepending a crammed embedding causes a moderate but consistent accuracy drop on multiple-choice benchmarks even with the original prefix in context, and collapses capability almost entirely under generative evaluation. Causal attention-knockout interventions trace this degradation to the embedding's interactions in the model's early layers. These results position progressive cramming as a tool for studying compression limits and show that perfect reconstruction - achievable through brittle steering rather than transferable semantics - is insufficient for meaningful compression.

Abstract

Explore similar work

Progressive Cramming: Reliable Token Compression and What It Reveals

Allocation Before Ranking: Decoupled Token Compression for OmniLLMs

CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models