Budget-Constrained Embodied Perception: Four Resource Walls and a Pre-Registered Evaluation of Access-Structured Perception on Open Models at less than 31B

Defu Lin; Wenhui Chen; Ziyao Lin; Jianlin Chen; Peiji Long; Chi Man Vong

cs.AIAug 24, 2026

Budget-Constrained Embodied Perception: Four Resource Walls and a Pre-Registered Evaluation of Access-Structured Perception on Open Models at less than 31B

Authors: Defu Lin, Wenhui Chen, Ziyao Lin, Jianlin Chen, Peiji Long, Chi Man Vong

Organizations: University of Macau, Macao SAR, China · South China University of Technology, Guangzhou, China · independent researcher

Abstract

Embodied multimodal agents must answer from growing observation streams under a fixed per-decision token budget. We formalize this constraint through four resource walls: a perceptual Shannon wall for bounded state, a horizon wall for query-independent frame selection, a round wall for non-adaptive retrieval, and a conditional composition wall for fixed-depth inference. We introduce ASP, a training-free wrapper for frozen multimodal models that combines a capped structured state, a verbatim episodic index, and query-conditioned budget allocation with iterative access. Following a pre-registered protocol, we evaluate seven open-weight models from 3B to 31B on SEW-Bench, a license-free synthetic long-horizon walkthrough benchmark constructed to instantiate these walls. The registered natural-video benchmarks were not run because their frames require dataset agreements; our evidence therefore concerns access mechanisms, not natural-scene perception. Under a 4,096-token decision budget, ASP reaches 75 to 94% episodic retrieval accuracy, compared with 3 to 19% for equal-budget query-independent sampling, and budget reallocation outperforms quadrupling the sampling budget on every backbone. However, the full three-component architecture does not validate channel duality: removing the compressive state raises the flagship mean from 35.4 to 58.0, ASP does not outperform the verbatim-only baseline on any backbone, and two of four pre-registered falsification criteria fire. These results show that query-conditioned access, rather than parameter count or context growth alone, is decisive under a fixed budget, while prompted online compression does not earn its cost in this setting.

Figures & tables

Panel of Figure 1 — Fig. 1 : Asp overview. A shared keyframe gate feeds two channels that are different in kind: a compressive $O(1)$ -token structured scene state (blue), which holds the running reductions retrieval cannot cheaply rebuild, and a verbatim episodic index (red), which keeps every keyframe losslessly off-GPU and returns $k$ of them per query. A query-conditioned router (green) water-fills the fixed per-decision budget $B=B_{s}{+}B_{r}{+}B_{c}$ over the three wall prices of § III , and the decision may take up to $R$ adaptive rounds, each round’s read depends on what the previous one returned, which is what Theorem 4 shows a single retrieval cannot simulate. Each budget component is written inside the component it pays for. Everything inside the dashed box is Asp , the gate included; it is shared verbatim by every baseline so that the comparison isolates access structure; the backbone sits outside the box and is never touched. Asp changes only what the model is allowed to read per decision.

Class (representatives)	$\mathcal{C}_{\mathrm{s}}$	$\mathcal{C}_{\mathrm{v}}$	adapt. $R$	expl. $B$	Wall left unpaid
Compressive / streaming memory [ 62 , 24 , 80 ]	$\bullet$	–	–	–	Shannon (Thm. 1 ): episodic detail
Query- independent selection [ 67 , 79 , 37 ]	–	$\circ$	–	–	Horizon (Thm. 2 ): coverage
In-pass token reduction [ 7 , 57 , 65 ]	–	$\circ$	–	–	Shannon: lossy at rest , not just in context
Query-conditioned retrieval [ 66 , 1 , 54 , 58 ]	–	$\bullet$	$\circ$	–	Tracking (Lem. 3 ): running reductions
Structured scene memory [ 19 , 52 , 22 ]	$\circ$	–	$\circ$	–	Shannon; and no certified cap $L_{s}$
Context growth [ 35 , 34 , 81 ]	–	$\circ$	–	–	Composition (Thm. 3 – 4 ): width $\neq$ depth

TABLE I: Related work, read for the channel each class commits to.

\bullet

= present and unbounded in

N

;

\circ

= present but lossy or bounded; – = absent. “Explicit

B

” means a per-decision token budget that all reads are charged against, including reasoning. The last column names the wall the class cannot pay, from § III . Classification is by the commitment each system’s published design makes, not by measured performance; borderline cases are marked

\circ

rather than argued.

Panel of Figure 3 — Fig. 2 : When a bound is worth stating. Both panels plot the closed forms of Theorems 1 – 2 ; nothing is fitted. (a) The Shannon ceiling $\kappa_{s}=\frac{m/(Nd)+1}{\log_{2}K}$ is a constraint only where it falls below $1$ , and whether it does depends on the attribute count $Nd$ against the state’s $m$ bits – not on the keyframe count $N$ , which is the substitution that makes the bound look tighter than it is. The marked points are three corpora at $L_{s}{=}512$ : an earlier version of ours at $Nd{=}1{,}350$ , where $\kappa_{s}{=}1.85$ and the theorem says nothing (§ V-H ); SEW-Bench at $Nd{=}9{,}600$ , $\kappa_{s}{=}0.547$ ; and the registered EW-Bench at $Nd{=}43{,}200$ , $\kappa_{s}{=}0.381$ . Note the direction of the $L_{s}$ family: a larger compressive state weakens the wall, exactly as it should, and the wall is a statement about the ratio rather than about compression being bad. (b) The horizon ceiling $\frac{w}{N}+(1-\frac{w}{N})\frac{1}{K}$ decays to chance for any fixed window, which is why a longer context defers this wall rather than removing it: an egocentric mission stream grows without bound while $w$ is bought in tokens. At $N{=}300$ and the $w{=}12$ that $B{=}4$ k actually affords, the ceiling is $16.0\%$ .

Panel of Figure 4 — Fig. 3 : Width is not depth. The two panels differ in exactly one respect: whether a read may depend on what an earlier read returned. (a) A single retrieval must commit to its $k$ frames before seeing any of them. On a depth- $d_{q}$ dependent chain it can cover hop 1, whose location the query names, but hop 2’s location is a function of hop 1’s content , so no query-independent selection can target it, and the bound does not improve with $k$ until $k$ approaches $N$ , which the per-decision budget forbids. That is Theorem 4 (i) , and it holds against unbounded compute; part (ii) extends the obstruction to locating the answer frame when the selector does carry state. (b) The three marked positions are the same three as in (a). $R$ adaptive rounds read one frame each and resolve the chain in $d_{q}$ steps, at a token cost of $d_{q}$ single-frame reads rather than $N$ . Rounds are therefore a resource distinct from context width, priced separately as $B_{c}$ in Definition 3 and bought by Asp ’s access loop. Two things worth separating here: this panel is the only wall statement that needs no complexity-theoretic assumption, whereas Theorem 3 extends the same conclusion to a single wide forward pass, one that sees all $N$ frames at once, and that extension does rest on $\mathsf{TC}^{0}\neq\mathsf{NC}^{1}$ . The case for an access loop does not depend on it.

Panel of Figure 5 — Fig. 4 : Embodied wall-exclusion pattern. Each wall eliminates one architecture class on a witness family; only an agent paying all three prices out of the per-decision budget $B$ crosses all walls. Green: passes; red $\times$ : provably blocked.

Panel of Figure 6 — Fig. 5 : The argument, as a dependency graph. Three of the four walls, and the tracking lemma that replaces a previously deferred reduction, are unconditional and hold against unbounded compute; only the composition wall (dashed) rests on $\mathsf{TC}^{0}\neq\mathsf{NC}^{1}$ and on naturalistic items realising the $S_{5}$ encoding, and the case for Asp survives without it, Theorem 4 carries the same architectural conclusion unconditionally. Note where the two composition results diverge: super-additivity in additive form needs a witness that obstructs both channels at once, which the uniform mixture is not (§ III ), so the mixture claim is dominance instead. Everything to the right of Corollary 2 is registered before measurement and has a stated failure mode.

Class $z$	$B_{s}$	$B_{r}$	$B_{c}$	Wall paid
ret (episodic detail)	0.10	0.65	0.25	Shannon $+$ horizon (via $\mathcal{C}_{\mathrm{v}}$ )
trk (cumulative state)	0.55	0.10	0.35	Shannon (via $\mathcal{C}_{\mathrm{s}}$ reductions)
cmp (compositional)	0.25	0.35	0.40	Composition (rounds)
now (immediate percept)	0.05	0.20	0.75	—

TABLE II: Default router allocation profiles (fractions of

B-B_{\rho}

), and the wall each profile pays.

R^{\max}=3

Panel of Figure 8 — Fig. 6 : What the router buys, and what a decision actually spends. (a) Each bar is one query class, partitioned into the three wall prices by the default profile of Table II ; the water-filling of § IV-D reduces to choosing a row. The classes differ almost entirely in which wall consumes the budget, not in how much is available: ret pours $65\%$ into verbatim retrieval because Theorems 1 – 2 put episodic detail out of reach any other way, trk inverts that because Lemma 3 makes retrieval useless for running reductions, and cmp is the only class obliged to pay all three. (b) The allocation is a cap , not a target. On the worked $d_{q}{=}3$ item the loop halts at answer after three rounds having spent $997$ of $4{,}096$ tokens ( $214$ against a state cap of $1{,}014$ and $742$ against a retrieval cap of $1{,}419$ ), so the budget that matters is the one the walls force it to spend, not the one it is handed. Panel (b) is constructed from the protocol, not measured $\dagger$ .

Panel of Figure 9 — Fig. 7 : Why the three subtests are different problems. Each strip is one episode’s frame stream, and the green bar at the top is what a single $4$ k-token decision can actually read, $12$ of $300$ frames at the measured $302$ tokens/frame, drawn to scale. ret hides the answer in one or two frames, so the only way in is query-conditioned verbatim retrieval; every query-independent selection of $12$ frames misses it with probability $1-12/300=96\%$ (Theorem 2 ). trk spreads the answer over every marked event, so no subset of $12$ frames suffices at any budget and the reduction must instead be accumulated online in $\mathcal{C}_{\mathrm{s}}$ (Lemma 3 ). cmp makes each hop’s location depend on the previous hop’s content, so the reads must be sequenced , not merely numerous (Theorems 3 – 4 ). The three failure modes are orthogonal by construction, which is what lets the collapse pattern of Figure 9 be diagnostic. Marks are illustrative of the construction protocol (Appendix B ); the counts and the scale are the corpus’s real ones, the individual positions are not.

Backbone	Method	SEW-RET	SEW-TRK	SEW-CMP	Mean	$w$
Ministral 3B	Blind	0.0	8.3	24.0	10.8
	Uniform- $w$	18.8	17.7	43.8	26.7	8
	Socratic	10.9	12.5	12.5	12.0
	Retr-only	65.6	27.1	41.7	44.8
	State-only	6.2	16.7	8.3	10.4
	EGAgent (reimpl.)	0.0	53.1	0.0	17.7

TABLE III : SEW-Bench at a fixed per-decision budget

B{=}4{,}096

tokens. Accuracy (%) by exact match on the closed answer alphabet (§ V-A );

n{=}32

24

24

questions for RET / TRK / CMP over four

N{=}300

-frame episodes. Mean is the unweighted mean of the three witness subtests. Uniform-

w

uses the largest

w

the budget affords at that backbone’s measured tokens/frame (rightmost column); no backbone affords the

w{=}16

of the pre-registration. Chance is

12.5\%

on RET and TRK (

K{=}8

) but

25.0\%

on CMP , which mixes sixteen

K{=}8

colour items with eight binary ones whose golds are balanced

4

–

4

by construction. The CMP cells here are a re-run against the corrected corpus, after eight of those items were found to carry a degenerate gold in the corpus first measured; Table V gives both runs and § V-F the account. Every value is measured – the three registered benchmarks with no licence-free data path stay in Table X as predictions.

Panel of Figure 11 — Fig. 8 : Measured budget and scale curves. (a) Whether restructuring a small budget beats enlarging an unstructured one, read as a curve rather than a single contrast: the Asp point at $B{=}4$ k is to be compared with the Uniform- $w$ point at $B{=}16$ k. (b) Score against parameter count for the two ends of the comparison. The rank correlation between parameter count and score falls from $\rho_{\mathrm{scale}}{=}-0.58$ under Uniform- $w$ to $0.00$ under Asp . Missing points are budgets or backbones with no cell, left absent rather than interpolated.

Variant	SEW-RET	SEW-TRK	SEW-CMP	Mean
Asp (full)	93.8	4.2	20.8	39.6
$-\ \mathcal{C}_{\mathrm{v}}$ (no episodic index)	9.4 $\downarrow$	4.2	25.0	12.8
$-\ \mathcal{C}_{\mathrm{s}}$ (no scene state)	90.6	12.5 $\downarrow$	83.3	62.2
$-\ \rho$ (fixed 1/3 split)	90.6	8.3	62.5	53.8
$R{=}1$ (single pass)	50.0	4.2	20.8 $\downarrow$	25.0
$R{=}5$	93.8	4.2	20.8	39.6

TABLE IV: Ablations on Qwen3.8-27B,

B{=}4{,}096

. Each variant is the same access loop with one component disabled, so a drop is attributable to the removed component and not to a different code path.

\downarrow

marks the cell the pre-registration predicts will collapse:

-\mathcal{C}_{\mathrm{v}}

on RET (Thm. 1 ),

-\mathcal{C}_{\mathrm{s}}

on TRK (Lem. 3 ),

R{=}1

on CMP (Thm. 4 ).

	Asp		Retr-only		Blind
Backbone	pre	post	pre	post	pre	post
Ministral 3B	37.5	33.3	41.7	41.7	36.5	24.0
Qwen3-VL-8B	25.0	29.2	29.2	29.2	34.4	16.7
Gemma 3 12B	45.8	33.3	37.5	29.2	37.5	20.8
Ministral 14B	41.7	45.8	58.3	54.2	8.3	16.7
Qwen3.8-27B	8.3	20.8	37.5	71.9	0.0	16.7

TABLE V: What the corpus defect was worth: the CMP column before and after the fix. Eight of the twenty-four cmp items carried the gold before in the corpus this paper first measured, so a constant answer scored

8/8

on them (D-43). pre : the superseded run, retained in results/cells_pre_cmpfix/ . post : the same cells re-run against the corrected corpus, in which the naming order is chosen to give an exactly

4

–

4

gold balance. Both columns are

24

items at chance

25.0\%

– the fix removes an exploitable prior, not a chance-level difference, since cmp mixes sixteen

K{=}8

items with eight

K{=}2

ones. The Blind pair is the diagnostic to read first: an agent that sees no frames at all should sit at chance, and only the post column is entitled to. Every cmp figure elsewhere in this paper is a post figure.

Panel of Figure 14 — Fig. 9 : Measured collapse pattern. Each cell is the change in score when one channel or resource is removed from Asp , against full Asp on the flagship (SEW-RET/TRK/CMP $=93.8/4.2/20.8$ ). The theory predicts a shape , not merely that ablations hurt: removing $\mathcal{C}_{\mathrm{v}}$ should cost most on the subtest witnessing the Shannon and horizon walls while costing little on tracking, and symmetrically for $\mathcal{C}_{\mathrm{s}}$ and for rounds. Boxes mark where the pre-registration placed each column’s largest drop; they are not moved to follow the data, which is what makes criterion F3 (§ V-G ) able to fire.

Backbone	Asp @ 4k	Uniform- $w$ @ 16k	$w$ @ 16k	$\Delta$
Ministral 3B ‡	46.9	24.0	8	22.9
Qwen3-VL-8B	43.8	23.2	53	20.6
Gemma 3 12B ‡	40.3	18.1	30	22.2
Ministral 14B ‡	48.3	18.8	8	29.5
Qwen3.8-27B	39.6	29.5	53	10.1
Qwen3-VL-30B-A3B	39.9	17.7	53	22.2

TABLE VI: P3: reallocation versus growth. Asp at

B{=}4{,}096

against Uniform-

w

given four times the budget,

B{=}16{,}384

. A positive

\Delta

means restructuring access beats buying context.

Result	Caps the capability of	Beyond Def. 1 – 2 it assumes	Value here	Binds	Instantiated by
Thm. 1	any $m$ -bit compressive state, on RET	nothing (unbounded compute)	$\kappa_{s}=54.7\%$	yes	State-only , RET
Thm. 2	any query- independent $w$ -frame selection, on RET	nothing	$16.0\%$	yes	Uniform- $w$ , RET
Lem. 3	verbatim-only access reading $k_{\mathrm{tot}}{<}N$ , on TRK	read-only retrieval	$12.5\%$	yes	Retr-only , TRK
Thm. 4 (i)	one non-adaptive round, no cross-frame state, on CMP	read-only retrieval	$14.8\%$	yes	Retr-only , Socratic , CMP
Thm. 4 (ii)	localisation by one round with an $m$ -bit state	read-only retrieval	$411\%$	no	Asp $R{=}1$ ablation
Thm. 4 (iii)	one round, $m$ -bit state, read-grounded answer	read-groundedness	$423\%$	no	—

TABLE VII: Every bound, evaluated at the parameters we actually ran (

N{=}300

d{=}32

Nd{=}9{,}600

K{=}8

m{=}6{,}144

bits,

k{=}8

w{=}12

k_{\mathrm{tot}}{\in}[9,15]

d_{q}^{\max}{=}3

). “Binds” means the bound lies strictly between chance and

1

, so that it forbids something a method could otherwise have done. Two rows do not: Theorem 4 (ii) – (iii) are true but vacuous here, because

m{=}6{,}144

bits against

N{=}300

frames is a large state per pointer, and Theorem 3 is asymptotic in

d_{q}

and yields no number at

d_{q}\leq 3

. What carries the empirical argument on this corpus is therefore Theorems 1 and 2 , Lemma 3 , and Theorem 4 (i) , the four unconditional, non-vacuous rows. Every number in this table is recomputed from the corpus metadata by analysis/check_bounds.py , which exits non-zero if the paper and the corpus disagree; the two errors it was written after are recorded in notes/deviations.md .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 17 — Fig. 10 : What rests on what. Tools (left) yield the walls (centre), which combine with Assumptions 1 – 2 into the composite claims (right). Two readings the prose cannot deliver at a glance. One: only Theorem 3 inherits a complexity-theoretic hypothesis, so a reader who disbelieves $\mathsf{TC}^{0}\neq\mathsf{NC}^{1}$ loses one box and keeps the argument, Theorem 4 (i) carries the same architectural conclusion unconditionally. Two: every arrow into the right-hand column passes through the assumption node, so the composite claims are exactly as strong as $\epsilon_{0},\epsilon_{1},\delta_{r},\epsilon_{u}$ , and $\epsilon_{u}$ enters multiplied by $N$ . The greyed box is a bound that is true but says nothing at SEW-Bench parameters.

	SEW draft	SEW-Bench	EW-Bench
	(superseded)	(built, measured)	(registered, not built)
stream
keyframes $N$	$150$	$300$	${\approx}450$
queryable attributes per frame $d$	$9$	$32$	${\approx}96$
attribute values $Nd$	$1{,}350$	$9{,}600$	${\approx}43{,}200$
alphabet $K$	$8$	$8$	$8$

Appendix

TABLE VIII: Corpus parameters and the ceilings they imply. Left: an earlier version of this generator, kept because it is the case where the theory is arithmetically silent . Centre: what was built and measured. Right: the registered corpus, not built. Every ceiling is computed from the row above it, not fitted. The load-bearing row is

Nd

: it is what Theorem 1 divides by, and moving it from

1{,}350

9{,}600

is what turned a vacuous ceiling into one a method can be measured against.

Panel of Figure 19 — Fig. 11 : One real SEW-Bench episode, on a time axis (ep1 of the shipped corpus; positions recovered by replaying the generator at its fixed seed). Reading upward: the door trace is a TRK reduction that only an online accumulator can follow, since its value at $t{=}300$ depends on all seven toggles; RET marks the eight classes visible in exactly one frame each; CMP marks one ordinal chain, whose question asks for an attribute of the second sighting and so cannot be answered without first ordering all three; TRK marks the nine sightings of one counted class. Reading downward: what the two query-independent policies of Theorem 2 are allowed to look at. The room band is the reason the two are different policies rather than the same one, Lv and St each recur three or more times, so the most recent frames are not the relevant ones, and a last- $w$ window sees only the end of the walkthrough while every one of the eight RET targets sits earlier than frame $275$ . Across all four episodes the two policies cover $2/32$ and $0/32$ of the RET targets respectively, against the $w/N$ rates of $5.3\%$ and $4.0\%$ the theorem predicts: the horizon wall is not an asymptotic remark about this corpus, it is where the targets are.

Input	Value used	Source	Enters	Ours?
OpenEQA, GPT-4V	$49.6$	[ 37 ]	OpenEQA column	no
OpenEQA, blind LLM	${\approx}33$	[ 37 ]	Blind row	no
AlanaVLM-7B	$46.7$	[ 64 ]	$7$ – $8$ B rows	no
Structured-memory delta	$+3$ at $5\times$ fewer frames	[ 19 ]	State-only rows	no
VSI-Bench open-model band	$32$ – $40$	[ 71 ]	VSI-Bench column	no
EgoSchema ${\leq}31$ B band	published band	[ 38 , 66 ]	EgoSchema column	no

Appendix

TABLE IX: Provenance of every input to a registered prediction. One row per number that enters the arithmetic of § C , so that a reader can audit the registration without our worksheet. The last column is the one that matters: exactly one input is our own prior work, it enters the Asp rows only, and § C already names it as the step most likely to be wrong. A reader who discounts it can strike the Asp rows of Table X and the baseline rows still stand on published anchors alone.

Backbone	Method	OpenEQA	VSI-Bench	EgoSchema
Qwen3.8-27B	Blind	34.1 $\dagger$ $\pm$ 2	24.0 $\dagger$ $\pm$ 2	29.5 $\dagger$ $\pm$ 3
	Uniform- $w$	52.6 $\dagger$ $\pm$ 3	38.5 $\dagger$ $\pm$ 3	62.0 $\dagger$ $\pm$ 3
	Retr-only	57.1 $\dagger$ $\pm$ 3	39.8 $\dagger$ $\pm$ 3	63.4 $\dagger$ $\pm$ 3
	State-only	55.4 $\dagger$ $\pm$ 3	41.2 $\dagger$ $\pm$ 3	61.1 $\dagger$ $\pm$ 3
	EGAgent (reimpl.)	58.9 $\dagger$ $\pm$ 3	42.1 $\dagger$ $\pm$ 3	64.0 $\dagger$ $\pm$ 3
	Asp (ours)	62.3 $\dagger$ $\pm$ 3	46.0 $\dagger$ $\pm$ 3	66.8 $\dagger$ $\pm$ 3

Appendix

TABLE X : Registered but not run. The three natural-video benchmarks of the pre-registration, with their frozen predictions

\dagger

and registered 90% prediction intervals. They are not measured here: OpenEQA, VSI-Bench and EgoSchema need episode-frame dumps we have no licence-free path to, and the registered EW-Bench itself needs habitat-sim plus an HM3D/ScanNet licence (§ VI ). No value in this table is claimed as a result.

Panel of Figure 22 — Fig. 12 : Equal budget, drawn as a ledger (the $d_{q}{=}3$ item worked through in § B-C , at the flagship’s measured $302$ tokens per frame). Three things this makes concrete. One: the router’s $(B_{s},B_{r},B_{c})$ is a per-channel ceiling , not a quota; this item finished three rounds on $997$ tokens, $24\%$ of $B$ , and the unspent remainder is not transferable to another item because $B$ is per decision. Two: the round structure is visible as three separate debits against two different channels, which is what Theorem 4 says a single-round agent cannot do at any $B$ . Three: Uniform- $16$ as specified does not fit inside $B$ : sixteen frames cost more than the whole budget before the prompt or the answer is counted, which is why baselines.py sends the largest $w$ the budget affords and logs the realised value (§ V-A ). Comparing against a Uniform- $16$ that silently overran $B$ by $18\%$ would have been the easiest available way to manufacture the result this paper is testing.

Backbone	$1-\epsilon_{0}$ (attribute)	$1-\epsilon_{1}$ (one step)
Ministral 3B	1.00 (32)	0.43 (14)
Qwen3-VL-8B	0.94 (32)	0.81 (16)
Gemma 3 12B	0.71 (31)	0.44 (16)
Ministral 14B	1.00 (32)	0.50 (16)
Qwen3.8-27B	1.00 (32)	0.75 (16)
Qwen3-VL-30B-A3B	0.94 (32)	0.50 (16)

Appendix

TABLE XI : Representational sufficiency (Assumption 1 ), measured. Left: single-keyframe attribute accuracy

1-\epsilon_{0}

, the RET question asked with the target frame handed to the model, which is the best case the assumption describes (chance

1/K=12.5\%

). Right: one composition step

1-\epsilon_{1}

, two in-context frames of the same class, which was seen earlier (chance

50\%

); scored items in parentheses. Attribute reading is comfortably sufficient at this scale,

0.71

–

1.00

against a

0.125

floor, so a near-chance grid cell is not explained by an inability to read a frame that is in context. The one-step column is the weak one: 4 of 7 backbones sit at or below chance ( underlined ), so for those the CMP family is limited by composition and not only by access, and Proposition 1 ’s

1-d_{q}\epsilon_{1}

bound is vacuous there. Two caveats on the right-hand column: it is binary, so it cannot separate inability from confusion over the words first / second , and each frame renders its own index in its header, so the task is legible in principle, since one backbone scores

1.00

on it. Measured by scripts/probe_assumption.py ; per-item records released.

Efficient Multimodal Inference

Efficient multimodal inference reduces the compute, memory, latency, and energy needed to process image, video, audio, text, and sensor inputs.

See all 227 papers

Oct 19, 2026

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: our controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains. Such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps. We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory. The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and converting their outputs into model-readable observations. Persistent multimodal memory retains the corresponding images, structured data, and web references as addressable artifacts throughout each query. A coding interface executes generated programs over stored data, enabling computations that combine results from multiple tools. Together, these mechanisms support an iterative process of acquiring, inspecting, and composing multimodal evidence. Evaluated on 22 benchmarks with eight open and proprietary backbones, OpenVLHarness improves average scores by 9.7-12.6 points over matched no-tool baselines, including +11.2 with Qwen3-VL-8B, +9.7 with Kimi K3, and +9.9 with GPT-6 Astra. With GPT-6 Luna at medium reasoning effort, OpenVLHarness outperforms the base model, Codex, and Visual Sketchpad at maximum effort by 7.7-8.5 points while incurring at most 37% of their model-token cost.

Yu Zhou, Cheng-Fu Yang, Rui Sun +8

University of California, Los Angeles · Apple

Tool Use in VLMs Multimodal Reasoning VLM Reasoning MLLM Reasoning Memory-Augmented VLMs

Ask PDF

Oct 13, 2026

Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs

Multimodal large language models have advanced rapidly, but most remain optimized for single-image inputs. Multi-image reasoning is costly because each image adds many visual tokens, increasing encoding and attention demands. Existing techniques that address the above issue, including visual token pruning, token merging, visual token compression, learned resampling, hierarchical compression, retrieval-augmented visual reasoning, and memory-based architectures, have largely been designed for single-image scenarios, where redundancy is primarily spatial and local. Multi-image reasoning introduces additional challenges: redundancy may occur across images, task-relevant evidence may be sparse or distributed, and aggressive per-image compression can remove details needed for comparison, counting, temporal reasoning, retrieval, or evidence grounding. Moreover, existing token-reduction methods are often either (i) training-free but inference-inefficient, typically relying on accessing attention matrices to get token-importance that may be impractical to access with memory-efficient attention implementations such as FlashAttention, or (ii) training-dependent, requiring the base MLLM to be retrained or adapted to a new compressed-token paradigm. In this work, we study scalable multi-image reasoning and understanding without retraining the base model and without accessing attention matrices at inference time. To this end, we propose mmVB, a lightweight plug-and-play module that assigns query-conditioned per-image token budgets via cross-image context, adding only 1.6M trainable parameters while reducing visual input tokens by 9x. This reduction yields 5x faster time-to-first-token and 7x fewer prefill FLOPs, while maintaining 94.4% of full-resolution accuracy across eight multi-image reasoning benchmarks.

Aimon Rahman, Joymallya Chakraborty, Vidit Bansal +1

Amazon

Multi-Image Reasoning Efficient Multimodal Inference Multimodal Large Language Models Visual Reasoning Visual Token Compression

Ask PDF

Sep 22, 2026

BAS-OPD: Budget-Aware Selective On-Policy Self-Distillation for Fine-Grained Multimodal Perception

Multimodal large language models (MLLMs) often struggle with fine-grained visual perception when processing complete images, as critical evidence may only appear in local regions. On-policy self-distillation (OPD) enables transferring privileged visual knowledge from informative views to full-image policies, but querying the teacher for every rollout introduces substantial supervision costs. In this work, we propose BAS-OPD, a budget-aware selective OPD framework that allocates teacher supervision under limited query budgets. Instead of querying all rollouts, BAS-OPD selects informative samples while maintaining full-batch student generation. We explore random, uncertainty-based, and learned utility-based selection strategies, where the learned selector estimates query value from detached rollout statistics and online utility signals derived from student--teacher agreement and teacher confidence without additional student forward passes. BAS-OPD only changes training-time supervision allocation and preserves single-pass full-image inference. Experiments on fine-grained multimodal perception benchmarks demonstrate that BAS-OPD achieves strong performance while substantially reducing teacher supervision costs, highlighting the effectiveness of selective OPD under constrained budgets.

Zihan Chen, Hengguang Zhou, Yuan Kang +5

HFIPS, Chinese Academy of Sciences · University of Science and Technology of China · University of California, Los Angeles

VLM Distillation Multimodal Large Language Models On-Policy Distillation Knowledge Distillation Fine-Grained Visual Perception

Ask PDF

Abstract

Efficient Multimodal Inference

Explore similar work

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs

BAS-OPD: Budget-Aware Selective On-Policy Self-Distillation for Fine-Grained Multimodal Perception