Memory That Changes Action Is Not Memory That Guides It: Counterfactual Auditing of History-Conditioned Robot Policies

Jiajie Zhang; Yankai Xiang; Changhao Chen

cs.ROSep 23, 2026

Memory That Changes Action Is Not Memory That Guides It: Counterfactual Auditing of History-Conditioned Robot Policies

Authors: Jiajie Zhang, Yankai Xiang, Changhao Chen

Organizations: Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · ShanghaiTech University, Shanghai, China

Abstract

A robot returning a block to its origin tray may encounter two task-consistent pasts that reconverge to the same current input but warrant different actions. Yet memory-policy evaluations often rely on task success or action change under memory perturbation, neither of which establishes that memory guides the decision. We propose the Counterfactual Memory Audit (CMA), an evaluation protocol that crosses two histories at a verified-identical present, queries a frozen policy under common randomness, and evaluates each saved action under both pasts. This separates memory sensitivity, warranted choice, matched-world physical value, and per-pair reliability. On Mem-0, every audited Put Back pair changes action, but only $20/64$ pairs are fully reliable; at a later Swap decision, all paired actions change while both memories select the same branch. Native interventions further show closed-loop influence: replacing the history bank redirects behavior toward the replaced content, while restoring a 4096-byte protected anchor recovers $38.9$ points of Swap success lost to injected bank faults. On a dual-arm physical platform, memory changes saved actions, yet five of nine completed Put Back manipulations reach the wrong target. These results show that a robot can remember and react without reliably using memory to choose the behavior its past warrants. CMA provides a decision-level audit for distinguishing these cases.

Figures & tables

Fig. 1: The audited decision: two lawful pasts, one identical present. Frames from one frozen Put Back pair (illustrative). Each task-consistent history ( $\mathcal{H}_{0}$ , $\mathcal{H}_{1}$ ) updates its native memory ( $\mathcal{M}_{0}$ , $\mathcal{M}_{1}$ ) before reconverging to the same present. Stacked outlines denote the stored multi-frame executed history. Solid rings mark each history’s origin mat; dashed rings mark the same mats at the present, both empty, making the warranted branch invisible from the current observation alone.

Fig. 2: CMA crosses lawful memories at one fixed present. (A) Two lawful histories from one illustrative later-Swap pair differ only in which mats hold a block and update two native memories (stacked outlines: stored history). (B) Both queries receive the verified-exactly-equal shared current $\mathcal{X}_{t}$ ; the frozen policy is queried once per memory under common randomness. (C) The saved actions cross the evaluator-only boundary (dashed) and are scored in both worlds, which never enter the policy; the four scores aggregate within pair before pair-cluster inference.

Cohort	Units	Within each unit
RQ1: Constructed decision audits
M0, PB	64 pairs	2 dir. $\times$ 3 seeds
M0, later Swap	24 pairs	2 dir. $\times$ 2 seeds
RQ2: Native-rollout interventions
Native event (B++, PB)	24 pairs	2 dir. $\times$ 1 query
Native event (M0, PB)	12 pairs	2 dir. $\times$ 2 seeds

Table 1: Selected simulation cohorts and independent units. Directions, histories, conditions, and seed repeats are within-unit observations; cohorts are never pooled. Units are pair clusters unless marked as episodes. M0: Mem-0; B++: BridgeVLA++ (deterministic queries); PB: Put Back. Calibration and natural-rollout cohorts: Sec. IV-D ; physical cohorts: Table 3 .

Fig. 3: Choice, physical value, and pair reliability describe the same saved actions differently. (A) Put Back pairs at the same correct-choice rate differ in crossed utility; horizontal jitter is for visibility only. (B) Per-pair bidirectional correctness across the three registered action seeds, over the same 64 pair clusters. (C) Two saved 31-point paths from one later-Swap pair (magnified from the marked region) illustrate action change without branch change; cohort statistics come from the 24 pairs, not this single pair.

Endpoint	Put Back	later Swap
$S_{\mathrm{e}xec}$	0.054 $[0.051,0.057]$	0.059 $[0.053,0.065]$
$p_{\mathrm{c}orrect}$	0.729 $[0.677,0.781]$	0.5
$F$	0.529 $[0.422,0.633]$	0
$U_{1}$	0.185 $[0.155,0.215]$	$-0.019$ $[-0.031,-0.008]$
Absolute regret	0.442	0.772
Fully reliable	20/64	0/24

Table 2: Registered endpoints of the dissociation. Constructed cohorts of Table 1 ; preregistered 95% pair-cluster bootstrap intervals.

S_{\mathrm{exec}}

: executed action divergence;

F

: signed faithfulness (

\neq 2p_{\mathrm{correct}}-1

under nonzero ambiguity); regret: shortfall from the reference value

1

; fully reliable: correct in both directions under all seeds. Blank: structural values or endpoints without registered intervals.

Fig. 4: Replacing the history bank at native events redirects behavior to the replaced content. (A) Official success under own-history reconstruction, opposite-target history, and cleared banks, each policy analyzed separately; the hatched row is the share of opposite-history continuations committing to the donor target. Whiskers: 95% cluster intervals; denominators follow Table 1 . (B) Native-chunk readout composition (12 fresh Mem-0 Put Back pairs per stage): every early readout is ambiguous, although commitment later follows the content; bank-cleared success is 5/24 early and 6/24 late.

Fig. 5: Restoring a 4096-byte protected anchor recovers closed-loop success lost to injected bank faults. (A) Official Swap success over the 36-episode assigned mixture after the injected whole-bank fault; the dashed line is the unperturbed native reference, and the strip shows the no-restore (gray) versus anchor (blue) contrast per ordered layout (six episodes each). (B) Protected state and measured restoration cost on log axes: 4096 bytes is the numeric tensor payload (serialized protection file: 5321 bytes); timings cover the restoration operation only. Dots: medians; lines: observed min–max over the 32 primary recoveries. For storage reference, the online shadow bank holds 126,976 bytes.

Fig. 6: The dual-arm physical audit: storyboards of both tasks. Frames from two recorded demonstration runs per task (illustrative; all counts and intervals are in Table 3 ). Colours mark the two histories, stacked outlines stored history, and dashed rings the decision-relevant objects and trays. (A) Put Back: the restored scene hides the origin, so the warranted return differs with the history. (B) Repick: the warranted selection differs with the lifted object. Equipment asset tags are blurred for double-blind review.

Readout	Put Back	Repick
Target correct	6/32	8/32
Wrong target	6/32	6/32
No expressed selection	19/32	17/32
Manipulation completed	9/32	10/32
thereof at the wrong target	5/9	5/10
Full success	4/32	5/32

Table 3: Physical audit on the dual-arm PiPER cell: core readouts over eight pairs / 32 executions per task. Counts use operator-provided labels, not independent blinded annotation; one further execution per task is a mixed/other case. Direction is an auxiliary movement readout. Independent units are the eight pair clusters per task; physical repeats and action seeds are within-unit observations.

Memory-Augmented Agents

Memory-augmented agents use stored experience and knowledge to act coherently across long tasks, changing environments, and repeated interactions.

See all 349 papers

Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems
Oct 16, 2026Keyuan Luo, Yi Xie, Wanyun Ling +3Memory-Augmented Agents Long-Term Memory for Embodied Agents
Hainan Bielefeld University of Applied Sciences (BiUH) · Technical University of Munich · Technische Universität Berlin
MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation
Sep 7, 2026Haiyang Sun, Haoxiao Wang, Junming Chen +6Memory Agent Benchmarks Robot Manipulation Benchmarks
Zhejiang University, Hangzhou, China
Simple Agentic Memory for Generalist Robot Policies
Sep 29, 2026Yuyou Zhang, Yunbei Zhang, Miao Li +4Agent Memory Robotic Manipulation
Carnegie Mellon University · Tulane University · New York University +2

Oct 16, 2026J/K move · Enter open · S save

Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems

Keyuan Luo, Yi Xie, Wanyun Ling, Yuanzhe Li+2

Hainan Bielefeld University of Applied Sciences (BiUH) · Technical University of Munich · Technische Universität Berlin

Memory-Augmented Agents Long-Term Memory for Embodied Agents Embodied Agent Benchmarks

Open paper AskPDF arXiv

When we say that a robot uses memory, does that memory actually play a role? Success-rate ablations alone cannot establish the use of task-relevant facts: they change information and computation together, while aggregate rates can hide offsetting gains and losses. We propose an Interventional Memory Evaluation Protocol (IMEP) for vision-language-action (VLA) systems to address four questions: (1) Does memory provide a net benefit? (2) Under what conditions does memory help? (3) Do changes to memory alter task outcomes? (4) Are these effects specific to task-relevant facts? At the retrieval interface, we compare target-fact substitutions with fact-preserving replacements, minimal numerical perturbations, and exact replicas, while assessing decision correctness separately from motor execution success. We evaluate within-episode, cross-episode, and agentic memory. In cross-episode learned-bank experiments, a one-unit-in-the-last-place (ULP) intervention, including its installation procedure, changes 11.9% of seed-paired outcomes, while changing key facts produces no clear additional effect over fact-preserving replacements. To examine when memory helps, we introduce ChainBench, where correct decisions require historical information unavailable in current observations. The full memory configuration raises task success from 39.1% to 74.3%, matching the oracle-decision condition under shared recorded executions. These results show that correct history can support robot decisions, but establishing whether a memory system realizes this benefit requires fact-specific evidence beyond aggregate success rates.

Abstract

Memory-Augmented Agents

Explore similar work

Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems

MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation

Simple Agentic Memory for Generalist Robot Policies