Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems

Keyuan Luo; Yi Xie; Wanyun Ling; Yuanzhe Li; Ziyue Li; Zhuoqi Zeng

cs.ROOct 16, 2026

Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems

Authors: Keyuan Luo, Yi Xie, Wanyun Ling, Yuanzhe Li, Ziyue Li, Zhuoqi Zeng

Organizations: Hainan Bielefeld University of Applied Sciences (BiUH) · Technical University of Munich · Technische Universität Berlin

Abstract

When we say that a robot uses memory, does that memory actually play a role? Success-rate ablations alone cannot establish the use of task-relevant facts: they change information and computation together, while aggregate rates can hide offsetting gains and losses. We propose an Interventional Memory Evaluation Protocol (IMEP) for vision-language-action (VLA) systems to address four questions: (1) Does memory provide a net benefit? (2) Under what conditions does memory help? (3) Do changes to memory alter task outcomes? (4) Are these effects specific to task-relevant facts? At the retrieval interface, we compare target-fact substitutions with fact-preserving replacements, minimal numerical perturbations, and exact replicas, while assessing decision correctness separately from motor execution success. We evaluate within-episode, cross-episode, and agentic memory. In cross-episode learned-bank experiments, a one-unit-in-the-last-place (ULP) intervention, including its installation procedure, changes 11.9% of seed-paired outcomes, while changing key facts produces no clear additional effect over fact-preserving replacements. To examine when memory helps, we introduce ChainBench, where correct decisions require historical information unavailable in current observations. The full memory configuration raises task success from 39.1% to 74.3%, matching the oracle-decision condition under shared recorded executions. These results show that correct history can support robot decisions, but establishing whether a memory system realizes this benefit requires fact-specific evidence beyond aggregate success rates.

Figures & tables

Figure 1: Success rate alone cannot show that memory content mattered. (a) Higher task success alone does not establish use of stored content. (b) IMEP separates four questions. (c) Shell Game ( $200$ pairs per contrast), Remember Color ( $388$ pairs), and ChainBench ( $202$ chains). Shell Game’s memory-read gain has an upper bound below the registered 10-pp criterion; Remember Color operates near chance. ChainBench uses no decision-time LLM; shared recorded executions yield equal success when full memory and the oracle choose correctly. The initial-fact baseline ignores updates. Equal $U/T$ rates and initial-fact/no-memory equivalence are not established (Sections 4.2 – 4.4 ).

Figure 2: Overview of IMEP. (a) Paired evaluation follows the memory–decision–execution workflow, with separate decision (where observable) and task-success readouts. (b) Access and oracle comparisons (

A,N,O

), interventions and controls (

C,U,S,T

). (c) Information, delivery, and record-validity checks guide interpretation. The drawer task is illustrative. Implementation limits, including installation effects and unverified semantic edits, are detailed in Section 3.2 .

Setting	Intervention $X$	$n$	$\mathrm{SR}_{A}$ (%)	$\mathrm{SR}_{X}$ (%)	$D$ (pp)	Verdict
Layer 1, within-episode learned banks
Shell Game, camera blackout	read blocked	200	73.5	69.5	${+4.0}^{\scalebox{0.8}{$ \scriptstyle+4.4 $}}_{\scalebox{0.8}{$ \scriptstyle-4.7 $}}$	below the 10-pp criterion
Shell Game, view available	read blocked	200	94.0	93.5	${+0.5}^{\scalebox{0.8}{$ \scriptstyle+3.7 $}}_{\scalebox{0.8}{$ \scriptstyle-3.8 $}}$	not detected
Shell Game blackout ladder, released	both reads blocked	400	67.25	63.00	${+4.25}^{\scalebox{0.8}{$ \scriptstyle+3.39 $}}_{\scalebox{0.8}{$ \scriptstyle-3.51 $}}$	detected
Shell Game blackout ladder, structured PCMB	both reads blocked	400	66.50	63.00	${+3.50}^{\scalebox{0.8}{$ \scriptstyle+3.25 $}}_{\scalebox{0.8}{$ \scriptstyle-3.35 $}}$	detected
SimplerEnv Fractal, 12 cells	perceptual bank removed	300	91.0	92.7	${-1.7}^{\scalebox{0.8}{$ \scriptstyle+4.6 $}}_{\scalebox{0.8}{$ \scriptstyle-4.4 $}}$	not detected

Table 1: Net effects of memory access on task success.

A

: memory on;

X

: listed intervention;

n

: paired outcomes.

D=\mathrm{SR}_{A}-\mathrm{SR}_{X}

(percentage points, pp) favors memory when positive. Superscripts/subscripts give distances to upper/lower confidence limits:

90\%

CIs for unmarked rows,

95\%

for

\dagger

. The conservative CP–Bonferroni

90\%

CIs are post hoc additions except for Bridge grid. The

95\%

CIs retain study-specific methods (Appendices A, D, and F). Verdicts retain the original decision rules.

Panel of Figure 3 — Figure 3: Outcome sensitivity and fact specificity across layers. (a) Shell Game blackout: released (filled) and structured-PCMB (hollow) markers, with $400$ pairs per condition. (b) Remember Color: $388$ pairs with swaps evaluated as delivered (coverage $0.70$ – $0.83$ ). The policy is near the single-choice chance benchmark. Outcome-change intervals in (a,b) are exact $95\%$ Clopper–Pearson CIs. The black interval is a task-stratified paired-bootstrap $90\%$ CI for the task-averaged different-fact minus same-fact sensitivity $\bar{\theta}$ . Shading denotes the preregistered $\pm 7.5$ -pp equivalence band. (c) Scripted ChainBench reader on curated records ( $401$ chains): only $T$ changes decisions. This single experiment provides both Q4 evidence for that reader and the decision-readout control, not independent evidence.

Condition	Decision accuracy (%)	Task success (%)
No memory	${53.0}^{\scalebox{0.8}{$ \scriptstyle+7.0 $}}_{\scalebox{0.8}{$ \scriptstyle-7.2 $}}$	${39.1}^{\scalebox{0.8}{$ \scriptstyle+7.1 $}}_{\scalebox{0.8}{$ \scriptstyle-6.8 $}}$
Shuffled records	${51.5}^{\scalebox{0.8}{$ \scriptstyle+7.1 $}}_{\scalebox{0.8}{$ \scriptstyle-7.1 $}}$	${38.6}^{\scalebox{0.8}{$ \scriptstyle+7.1 $}}_{\scalebox{0.8}{$ \scriptstyle-6.7 $}}$
Initial-fact baseline	${52.0}^{\scalebox{0.8}{$ \scriptstyle+7.0 $}}_{\scalebox{0.8}{$ \scriptstyle-7.1 $}}$	${39.1}^{\scalebox{0.8}{$ \scriptstyle+7.1 $}}_{\scalebox{0.8}{$ \scriptstyle-6.8 $}}$
Full memory	${100.0}^{\scalebox{0.8}{$ \scriptstyle+0.0 $}}_{\scalebox{0.8}{$ \scriptstyle-1.8 $}}$	${74.3}^{\scalebox{0.8}{$ \scriptstyle+5.8 $}}_{\scalebox{0.8}{$ \scriptstyle-6.6 $}}$
Oracle decision	${100.0}^{\scalebox{0.8}{$ \scriptstyle+0.0 $}}_{\scalebox{0.8}{$ \scriptstyle-1.8 $}}$	${74.3}^{\scalebox{0.8}{$ \scriptstyle+5.8 $}}_{\scalebox{0.8}{$ \scriptstyle-6.6 $}}$

Table 2: ChainBench results.

202

chains,

K=2

(chance accuracy

50.0\%

). Superscripts/subscripts give distances to upper/lower exact

95\%

Clopper–Pearson confidence limits. Full memory uses Section 3.5 ’s configuration. The initial-fact baseline ignores updates. Conditions share two recorded physical executions per chain. Full memory and the oracle choose correctly on every chain, making their task success rates equal by construction.

Panel of Figure 4 — Figure 4: ChainBench chain construction. A write episode reveals the target drawer, and same-scene distractors then write other records. At query time, closed drawers give identical observations across target locations, requiring recall of the earlier object–drawer relation. Simulator frames illustrate the construction, not a rollout from the 202-chain study.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Study	Status	App.
Detectability: what effect can each readout detect?
Post-cue view removal (Shell Game, released banks)	preregistered; holds ( $D=+20.5\,\mathrm{pp}$ )	B.1
Scripted-reader positive control (ChainBench) a	holds	B.2
Graded synthetic wrapper (R-1, RC3/RC5)	holds (synthetic post-policy injection)	B.3
Grid resolution (GRID-1a, SimplerEnv-Bridge)	preregistered; unresolved (below announceable effect)	B.4
Repeat checks (exact replicates)	holds (determinism only)	B.5

Appendix

Table 3: Evidence ledger, ordered by question (detectability, then Q2 (only in memory), Q2 (delivery/validity), Q3/Q4 and Q1; criteria in Sections 3.3 and 3.4 ). The status column uses the verdict words of Appendix A.2 . Withdrawn and excluded rows support no claim. “App.” is the section that reports the row. a The scripted reader is both the detectability control for the decision readout and the Q4 evidence for curated records, so these rows are not independent evidence. b These analyses are post hoc. No survival gate was registered for these campaigns. c Threshold gate against the default token merge, 250 seed-paired units, attempt 1:

b{=}3

c{=}1

(exact McNemar

p=0.625

). Remember Color 5 is near chance (Appendix C.2 ). Whether the rule acts within or across attempts was not verified. The study has no closeout record, although its sealed root holds the frozen design note and the launch record.

Legacy name	Frozen requirement	Question under which it is now reported
wiring gate	$C$ matches $A$ bitwise; two adjudicators agree; cross-machine recompute	Detectability: exact replicate $C$ (App. B.5 )
path gate	probe: $A$ reads memory; $N$ ’s forced-empty reads equal its reads	Q2 (delivery): forced reads over reads (App. D.2 )
competence gate	Wilson $95\%$ interval of $\mathrm{SR}(A)$ inside $(\text{floor}+0.05,\,1-0.05)$	Q2 (only in memory) floor and ceiling (App. C.2 )
survival gate ( $G_{\rm surv}$ ; blackout ladders only)	substituted content present at the decision step in every substituting-condition trial	Q2 delivery (App. D.1 ); elsewhere post hoc coverage
utility readout	$D$ against a criterion registered before launch	Q1 (Section 3.3 )
sensitivity readout	$\rho$ vs. frozen 5% activity threshold; $U$ as reference	Q3 numerical control $\rho_{U}$ (Section 3.3 )

Appendix

Table 4: Legacy gates and readouts. Each gate or readout named in the frozen protocols, the requirement it froze, and the question under which the paper now reports it.

Benchmark family	checkpoint	embodiment; action interface	implementation / lines
SimplerEnv–Fractal	MemoryVLA Fractal checkpoints for visual matching and variant aggregation	Google Robot (static); 7-D EEF delta	Fractal overlay / 3251 ∗
SimplerEnv–Bridge	MemoryVLA Bridge checkpoint	WidowX, sink camera; 7-D EEF delta	grid: sealed release / 903; carry: carry overlay / not established †
MIKASA–Robo	MemoryVLA MIKASA checkpoint	ManiSkill Panda, wrist camera; 7-D end-effector delta pose	MIKASA build / 3322 ‡

Appendix

Table 5: Per-family checkpoint lineage. Checkpoint files, action interfaces and implementation versions. SHA-256 checksums of all weights, implementations and asset snapshots are listed in the archived manifests. ∗†‡ See the text.

Setting (Section 4.1 )	method	evaluator / implementation
Within-episode learned banks
Shell Game view removal	released MemoryVLA	$e_{1}$ / MIKASA build
Shell Game read, Q2-XA	released MemoryVLA	not recorded
SG-blackout ladder, released	released MemoryVLA	ladder runner / MIKASA build ∗
SG-blackout ladder, structured	episode-local PCMB	ladder runner / MIKASA build ∗
Fractal (completion and eight-cell runs)	perceptual bank ablated	Fractal overlay, dated after the run

Appendix

Table 6: Campaign binding. Method, evaluator revision and MemoryVLA implementation version carried by each campaign’s archived records.

e_{1}

–

e_{4}

label distinct evaluator checksums without implying order, and implementation names follow Table 5 . “Not recorded” means the archived evidence unit carries no such checksum. Full checksums are in the archived manifests.

Contrast (attempt 1)	SR base	SR int.	$b/c$	$D$ (pp)	bound	McNemar $p$
view removed, reads on	0.940	0.735	45/4	$+20.5$	LB $+13.21$	$8.2\times 10^{-10}$
reads blocked, view visible	0.940	0.935	3/2	$+0.5$	UB $3.65$	$1.0$
reads blocked, under blackout	0.735	0.695	9/1	$+4.0$	UB $7.70$	$0.0215$
interaction (row 3 $-$ row 2)				$+3.5$	below MDE $5.78$	—

Appendix

Table 7: Shell Game view removal and read contrasts, attempt 1.

n=200

seed-paired scopes per condition. SR base and SR int. are success without and with the intervention, and

D=(b-c)/n

is positive when the intervention lowers success. LB and UB are exact bounds on

D

, and

p

values are exact McNemar tests. The interaction lies below the minimum detectable effect (MDE) and supports no claim.

$\delta$ ( $L_{2}$ /step)	$\theta=\rho_{T}-\rho_{S}$	One-sided 95% LB	Verdict
$10^{-4}$	$-2.6\,\mathrm{pp}$	$-5.9\,\mathrm{pp}$	not detected
$10^{-3}$	$+2.3\,\mathrm{pp}$	$-0.8\,\mathrm{pp}$	not detected
$10^{-2}$	$+2.1\,\mathrm{pp}$	$-1.6\,\mathrm{pp}$	not detected
$2\!\times\!10^{-2}$	$+7.2\,\mathrm{pp}$	$+3.4\,\mathrm{pp}$	detected ( $\delta^{*}$ )
$5\!\times\!10^{-2}$	$+8.8\,\mathrm{pp}$	$+4.9\,\mathrm{pp}$	detected

Appendix

Table 8: Graded detectability ladder. R-1 wrapper on the common

n_{\mathrm{joint}}=388

cohort at each injected amplitude

\delta

. LB is the one-sided 95% lower bound on

\theta

(chain-level paired bootstrap).

\delta^{*}

is the smallest detected amplitude.

True $\bar{\theta}$	TOST $\pm 7.5$ accept	95% LB $>0$	positive	inconclusive
$0\,\mathrm{pp}$	0.913	0.023	0.011	0.075
$2.5\,\mathrm{pp}$	0.703	0.191	0.139	0.159
$5.0\,\mathrm{pp}$	0.292	0.568	0.504	0.203
$7.5\,\mathrm{pp}$	0.052	0.866	0.839	0.110

Appendix

Table 9: Retrospective operating characteristics of the realized-floor decision grid. Settings:

\rho_{S}=0.13

n=193+195

\phi=0.3

. Each entry is the fraction of simulations reaching the column’s decision at the given true increment

\bar{\theta}

. This is a post-outcome diagnostic.

Condition (baseline $C$ )	Task success	$b$	$c$	$\rho$
$C$ sealed reference	0.7723	—	—	—
$U$ one-byte edit	0.8267	20	31	0.2525
$R$ identical-instruction replay floor	0.7426	30	24	0.2673
$T$ wrong fact	0.0000	156	0	0.7723

Appendix

Table 10: Instruction-level manipulation check on the public executor (first of two full

n=202

runs). The identical-instruction replay floor is

26.7

–

28.2\%

across the two runs, and the one-byte increment (

-1.49\,\mathrm{pp}

, one-sided 95% lower bound

-7.92\,\mathrm{pp}

, chain-level paired bootstrap) is not detected above it. Because

T

scores exactly

0.0000

\rho_{T}

is identically

\mathrm{SR}_{C}

: the wrong-instruction contrast only restates that an incorrect instruction fails deterministically, is not a measurement of content sensitivity, and is not the latent-matched

S

T

contrast of Section 3.2 .

Campaign	Condition	Passing / total	Rate
Cross-episode substitution, installed once at attempt-2 start (post hoc audit)
Remember Color 3, fixed-path substitution	$S$	$135/193$	0.70
	$T$	$150/193$	0.78
Remember Color 5, fixed-path substitution	$S$	$161/195$	0.83
	$T$	$154/195$	0.79
Shell Game full-vision ladder, band A b	$S$	$134/198$	0.68

Appendix

Table 11: Decision-time survival by campaign. Top: post hoc audit of the cross-episode campaigns, which registered no survival gate. A retry counts when every attempt-2 decision query selects at least one unmodified substituted record, so the rate is a lower bound on delivery. The audit covered the donor-swap conditions

S

and

T

. Bottom: the registered survival gate of the Shell Game blackout ladders, which checks that each decision step of a substituting trial carries substituted content, over 800 trials per condition. a Not reported by the audit. b Withdrawn as Q3/Q4 evidence (Appendix G.1 ).

		released			structured PCMB
Condition	Operation	$b/c$	$\rho$	$D$ (pp)	$b/c$	$\rho$	$D$ (pp)
$C_{\rm det}$	anchor replayed, no substitution	0/0	0.00%	$0.00$	0/0	0.00%	$0.00$
$U$	one ULP on every stored float	3/1	1.00%	$+0.50$	6/4	2.50%	$+0.50$
$S_{\pi}$	permuted within each latent	23/4	6.75%	$+4.75$	18/6	6.00%	$+3.00$
$T_{\mu}$	resampled from own mean and s.d.	24/5	7.25%	$+4.75$	23/5	7.00%	$+4.50$
$N_{\rm blk}$	both reads blocked	21/4	6.25%	$+4.25$	18/4	5.50%	$+3.50$

Appendix

Table 12: Shell Game blackout ladders. Seeds

610000

–

610399

n=400

per condition, paired on (seed, attempt 1).

b

counts pairs in which the anchor

A

succeeds and the condition fails and

c

the reverse, so

\rho=(b+c)/n

and

D=(b-c)/n

, positive when the condition lowers success.

C_{\rm det}

and

N_{\rm blk}

are scored against the wave-2 anchor, the other conditions against the wave-1 anchor.

b^{\prime}

and

c^{\prime}

count pairs in which only

T_{\mu}

or only

S_{\pi}

changes the outcome. The conditional exact bound (each endpoint one-sided 95%) holds given the observed

b^{\prime}+c^{\prime}

and is not an unconditional interval for

\theta

Task	$n$	$n_{11}$	$n_{10}$	$n_{01}$	$n_{00}$	$\rho_{T}$	$\rho_{S}$	$\theta_{t}$
RC3	193	9	22	19	143	16.06%	14.51%	$+1.55\,\mathrm{pp}$
RC5	195	11	13	12	159	12.31%	11.79%	$+0.51\,\mathrm{pp}$

Appendix

Table 13: Seed-level joint tables. Flip indicators are

I_{T}

and

I_{S}

. Here,

n_{10}

denotes true-only discordance and

n_{01}

sham-only discordance. Donor availability removes 7 RC3 and 5 RC5 scopes, below the frozen 5% failure budget. All rates hold for the swap as delivered (Appendix D.1 ).

Study	Condition	$n$	Task success	Decision accuracy	Wrong-open rate
E1	curated records (full memory)	143	0.9510	0.9580	0.0000
E1	no memory	143	0.3287	0.3287	0.1888
E1	oracle decision	143	0.9930	1.0000	0.0000
E1	wrong decision	143	0.0000	0.0000	1.0000
E1	shuffled memory	143	0.4895	0.4965	0.1329
E1	initial-fact baseline	143	0.4895	0.4965	0.1119

Appendix

Table 14: Complete E1/E1B baseline morphology. “Curated records” denotes the full-memory condition, whose sealed decisions are replayed through the motor bridge. The initial-fact baseline selects the answer specified by the initial fact and ignores later updates. The distinct stale-system condition is defined only on rewritten chains, with study-specific denominators (E1B: all 97 of 202; E1: 72 of 143). The stale-system condition answers incorrectly on every rewritten E1B chain (

0/97

correct), providing the ChainBench validity control of Section 3.4 .

Condition vs. fresh	Fresh SR	Condition SR	$\Delta$	$b$	$c$	exact McNemar $p$	Holm $p$
Token-merge carry	31.5%	29.0%	$-2.5\,\mathrm{pp}$	33	28	0.609	1.000
Structured/versioned carry	31.5%	31.0%	$-0.5\,\mathrm{pp}$	22	21	1.000	1.000
Hybrid carry	31.5%	32.5%	$+1.0\,\mathrm{pp}$	14	16	0.856	1.000

Appendix

Table 15: Frozen deep-chain endpoint. Attempts 4–5 give 200 paired outcomes per comparison. The baseline is fresh reset,

\Delta

is condition minus fresh, and

b

counts pairs where fresh succeeds and the condition fails.

Task	released SR	structured SR	$\Delta$	$\rho$	exact McNemar $p$
Intercept Memory (IM)	20.5%	25.5%	$+5.0\,\mathrm{pp}$	18.0%	0.133
Remember Color 3 (RC3)	34.0%	32.5%	$-1.5\,\mathrm{pp}$	17.5%	0.736
Remember Color 5 (RC5)	23.5%	23.0%	$-0.5\,\mathrm{pp}$	18.5%	1.000
Remember Color 9 (RC9)	13.0%	12.5%	$-0.5\,\mathrm{pp}$	12.5%	1.000
Shell Game (SG)	89.5%	89.5%	$0.0\,\mathrm{pp}$	3.0%	1.000
Pooled, 5 tasks ( $n{=}1{,}000$ )	36.1%	36.6%	$+0.5\,\mathrm{pp}$	13.9%	0.735

Appendix

Table 16: MIKASA released versus structured-PCMB configuration contrast. Each condition and task has 200 episodes (attempts 1–2 pooled).

\Delta

is structured minus released. No task shows a detected net effect, although four non-ceiling tasks have double-digit paired discordance. The pooled row is descriptive (one backbone and checkpoint, not independent replicates).

Held-out diffusion loss	true memory	empty	mismatched
MemoryVLA MIKASA checkpoint	0.658	0.655	0.656
trained components	0.123	0.228	0.148

Appendix

Table 17: Offline mutation gate (held-out diffusion loss). The trained components separate true, empty and mismatched memory offline, but this did not predict closed-loop success (Table 18 ).

Task	trained SR †	community SR †	exact $\Delta$ (pp)	paired test
Shell Game	10%	90%	$-79.0\,\mathrm{pp}$	$p<10^{-6}$
Remember Color 5	8%	24%	$-15.5\,\mathrm{pp}$	$p=0.0003$
Remember Color 3	24%	34%	$-10.5\,\mathrm{pp}$	$p=0.051$
Intercept Memory	24%	20%	$+3.5\,\mathrm{pp}$	n.s.
Remember Color 9	14%	13%	$+1.0\,\mathrm{pp}$	n.s.
Pooled	16.0%	36.1%	$-20.1\,\mathrm{pp}$	$p<10^{-6}$

Appendix

Table 18: Closed-loop result for the trained memory plugin. 200 paired episodes per task (zero execution failures). Community is the released MemoryVLA condition of Table 16 , a cross-launch archived condition.

\Delta

is trained minus community. The paired-test method is not recorded. † Rates are rounded as exported.

\Delta

uses unrounded counts.

Study	Why excluded	Consequence
Payload-swap v1/v2	path or admissibility gates invalid	excluded before v3
Payload-swap v3 (incl. stage B)	no matched control; outcome-exposed seeds; third-task assignment failed closed under the donor rule	replaced by the formal five-condition RC3/RC5 campaign; not pooled
Five-task configuration contrast, first run	aborted: the structured persistent clock was rewound by the episode-local timestep, so newly written slots could be evicted before action attachment	partial runs preserved, none used; the reported run is the second run, launched after the clock fix passed a full-horizon smoke; no claim from v1
Same-function adapter toggle (1,000 episodes)	zero-initialized path absent from checkpoint; agreement by construction	determinism control only; not evidence on whether bank content is used
Version-conflict precursor (1,200 rollouts)	stale condition vs. unmodified $+1.7\,\mathrm{pp}$ , vs. no-conflict control $+4.0\,\mathrm{pp}$ ; could not resolve a few-point effect	superseded by the frozen 690-pair test; no claim
Earlier E1B export	provenance gaps; verdict retired	not pooled with the E1B rerun; no claim

Appendix

Table 19: Excluded and superseded studies. None supports a claim in this paper.

Item	What was done; effect on reported numbers	App.
Post hoc analyses
Four-question framework and criteria	The detectability precondition, Q1–Q4, their criteria and the first-failing-condition rule were written after every campaign had closed; a criterion that no campaign registered is post hoc.	3.1
Decision-time survival audit	Coverage per cross-episode content-substitution campaign; none registered a survival gate (the blackout ladders did), so coverage is a lower bound on delivery.	D.1
All- $K{=}2$ floor (E1B)	The closeout used a mixed- $K$ floor; all 202 chains have $K{=}2$ , so the floor is exactly $0.500$ and the gain $+0.470$ is reported against it.	C.1
Stratified re-scoring (RC fixed path)	$\bar{\theta}$ re-scored by one-ULP fragility and by step-0 perturbation magnitude; task-stratified paired bootstrap.	E.2
First-step $L_{2}$ (RC fixed path)	First-step $L_{2}$ distances of $U$ , $S$ and $T$ (means; medians for $S$ and $T$ ); not preregistered endpoints.	E.2

Appendix

Table 20: Post hoc analyses and amendments. Post hoc analyses ran on frozen records after the campaign’s closeout. None adds rollouts or feeds a registered gate. “After data” means that outcome or gate data of the amended campaign existed when the amendment was written. Unless a row says otherwise, no number reported in this paper moved.

Run A
Contrast	Cohort	$n$	Difference (pp)	95% CI (pp)
Retrieval $-$ no memory	all	403	$+5.71$	$[-0.74,+12.16]$
Retrieval $-$ shuffled records	all	403	$+5.21$	$[-1.49,+12.16]$
Oracle memory $-$ retrieval	all	403	$+43.67$	$[+38.71,+48.64]$
Latest write $-$ read-time gate	rewritten	204	$+11.27$	$[+7.35,+15.69]$
Superseded write $-$ no memory	rewritten	204	$-47.06$	$[-53.92,-40.20]$

Appendix

Table 21: Memory-state and retrieval contrasts (non-confirmatory). Each contrast is the first condition minus the second. Differences and interval endpoints are absolute percentage-point changes in decision accuracy. Intervals are reported 95% chain-bootstrap percentile intervals with 10,000 draws and seed 20260723. “Rewritten” restricts the cohort to chains with a superseded fact. Run A includes the E1B executed subset; Run B is a distinct cohort.

Memory-Augmented Agents

Memory-augmented agents use stored experience and knowledge to act coherently across long tasks, changing environments, and repeated interactions.

See all 336 papers

MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation
Sep 30, 2026Egor Cherepanov, Nikita Kachaev, Aleksandr I. Panov +1Memory-Augmented VLMs Long-Horizon Robotic Manipulation
AXXX, Moscow, Russia · MIRIAI, Moscow, Russia · HSE University, Moscow, Russia
MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation
Sep 7, 2026Haiyang Sun, Haoxiao Wang, Junming Chen +6Memory Agent Benchmarks Robot Manipulation Benchmarks
Zhejiang University, Hangzhou, China
ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies
Sep 29, 2026Yaxin Zhao, Dianye Huang, Chenwei Wang +2Memory-Augmented VLMs Long-Horizon Robotic Manipulation
Harbin Institute of Technology, Harbin, China. · Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China

Sep 30, 2026J/K move · Enter open · S save

MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation

Egor Cherepanov, Nikita Kachaev, Aleksandr I. Panov, Alexey K. Kovalev

AXXX, Moscow, Russia · MIRIAI, Moscow, Russia · HSE University, Moscow, Russia

Memory-Augmented VLMs Long-Horizon Robotic Manipulation Language-Conditioned Robot Manipulation Vision-Language-Action Models Robot Manipulation Benchmarks

Open paper AskPDF arXiv

Vision-language-action policies often see only one or a few recent frames, which makes it difficult to evaluate how they use information that disappears during a task. We introduce MIKASA-Robo-VLA, a benchmark of 90 language-conditioned manipulation tasks. All but 10 hide the cue an action depends on. Those 10 are reactive controls. MIKASA-Robo, the suite it rebuilds, has 32 tasks and uses language only in a representative VLA subset. Here every task provides an instruction, while memory-dependent tasks hide a task-relevant cue and reactive controls keep it available. For 70 tasks, environment phase timings specify an information gap, and for 28 of them the gap exceeds the 16-frame window of the widest fixed-context VLA we survey. The gap counts only the interval the cue is provably absent, not the full duration a policy must retain it, so every memory-dependent task still requires memory by construction, including the ones whose measured gap is short. We release 22,500 oracle trajectories across 10 memory types in RLDS and LeRobotDataset v3. A reference

π_{0.5}

baseline with current images and proprioception, but no observation history or explicit memory module, is fine-tuned on 14 tasks and achieves 0.211

\pm

0.044 mean task success. Its lower success on the evaluated Long-split tasks is confounded by open-loop chunking and the memory types represented in that subset. Project page: https://mikasarobo.github.io/

Abstract

Memory-Augmented Agents

Explore similar work

MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation

MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation

ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies