Gamma: Grounded Multi-Agent Memory with Mechanical Audit for Memory-Intensive Manipulation

Yudong Liu; Yueqian Lin; Zijia Tang; Yuxi Zheng; Yuan Li; Shuai Zhang; Yi Li; Taotao Jing

cs.ROOct 8, 2026

Gamma: Grounded Multi-Agent Memory with Mechanical Audit for Memory-Intensive Manipulation

Authors: Yudong Liu, Yueqian Lin, Zijia Tang, Yuxi Zheng, Yuan Li, Shuai Zhang, Yi Li, Taotao Jing,

Organizations: Duke University · Qualcomm AI Research · University of Florida

Abstract

Pretrained vision-language-action (VLA) policies fail on memory tasks, behaviour that depends on events observed long before the current frame, because their training data is overwhelmingly Markov and mapping episodic memory directly to action chunks is hard. Yet the same policy, finetuned once to follow a privileged oracle's grounded-subgoal text, nearly solves these tasks: the difficulty lies entirely in mapping memory to a grounded symbolic subgoal. Existing work entrusts this stage to a single VLM that selects keyframes and reasons over raw frames, and falls far short of the oracle: key identities and dynamics are buried in noisy pixels, the reasoning spans hundreds of frames in an unbounded context, and hallucination corrupts the memory itself. We instead present GAMMA (Grounded multi-Agent Memory with Mechanical Audit), which builds the stage as a collaborating two-agent memory system: a detector tool (SAM-3) for spatial grounding, a VLM writer that distils each replan window into one grounded text line, an append-only text memory bank, and a VLM reasoner that reads the bank and emits the next grounded subgoal at the oracle's cadence, wrapped in a propose-verify harness that admits, defers, corrects, or rejects every agent claim against mechanical visual evidence, at training-corpus construction and at deployment alike. On a sixteen-task memory-manipulation suite GAMMA recovers a large fraction of the oracle ceiling with the policy held fixed, reaching the ceiling on several tasks, and transfers with unchanged contracts to a second benchmark.

Figures & tables

Figure 1: The memory medium is the key difference. Top: a MemER-style single-VLM keyframe pipeline remembers pixels: an opaque selector drops decisive frames and the VLM emits a stale, wrong subgoal. Middle: Gamma remembers verified grounded text: the propose–verify harness gates every writer line into the append-only bank (lines from a real episode) and the reasoner looks up exactly the line it needs. Both feed the same fixed subgoal-conditioned $\pi_{0.5}$ : what differs is the memory system in between. Bars: suite-average success vs. the privileged oracle.

Figure 2: Overview of Gamma . Each replan tick, the SAM-3 detector grounds the window’s frames; the writer (Agent-1) distils them into at most one grounded event line; the reasoner (Agent-2) reads the append-only bank, without pixels, and emits the next grounded subgoal. The agents’ only interface is the bank; every claim must pass the propose–verify harness against visual evidence.

Configuration	Memory mechanism	Success
$\pi_{0.5}$ end-to-end	none	$0.179$
$\pi_{0.5}$ + memory module (FrameSamp+Modul)	latent frame memory	$0.445$
MemER-style keyframe pipeline	VLM keyframe selection	$0.424$
$\pi_{0.5}$ + oracle ungrounded subgoals	privileged, no coordinates	$0.496$
$\pi_{0.5}$ + oracle grounded subgoals	privileged	$0.841$

Table 1: The policy is not the bottleneck, and grounding sets the ceiling: oracle grounded subgoals take

\pi_{0.5}

from

0.179

0.841

; the same oracle without coordinates reaches

0.496

, memory injected into the policy

0.445

. Our task is to supply the grounded stream without privilege (all rows RoboMME-reported).

Panel of Figure 3 — Figure 3: The harness on real claims. Left (ButtonUnmaskSwap), the claim “completed: press the first button”: the button surviving every window frame refutes it— defer , and the policy retries (top); its disappearance under the gripper registers the press— admit (bottom). Right (RouteStick, stroke 3): the writer says “right target, clockwise”; from the start pad $A$ , the next pad $B$ and the stroke’s ink, the reader finds the ink bulging counterclockwise (61 of 65 points) with $B$ to the left. Both words are contradicted: the writer’s line is reject ed, the reader’s admitted.

		Temporal				Spatial				Object				Procedural
	Method	PickX	BinF	SwingX	StopC	VU	BU	VUS	BUS	PH	VRP	VPB	VPO	MC	IP	PL	RS	Avg
e2e	$\pi_{0.5}$	42.9	30.0	35.6	6.7	20.4	22.2	18.7	6.7	11.3	0.4	31.1	25.8	26.0	1.6	2.9	4.7	17.9
	SAM2Act+	76.0	40.0	25.3	0.0	27.3	32.0	18.0	26.7	17.3	5.3	24.7	20.0	29.3	0.0	0.0	0.0	21.4
mem-VLA	TokenDrop+Modul	83.6	34.4	86.0	5.3	28.2	29.3	28.4	21.3	21.3	22.0	59.6	36.0	62.0	7.1	32.4	51.6	38.0
	FrameSamp+Modul	87.3	39.6	92.0	42.0	32.7	25.1	24.4	18.2	22.9	30.4	60.0	32.0	77.8	7.6	53.6	66.7	44.5
VLM-mem	MemER	79.3	56.7	59.3	0.0	81.3	72.0	38.0	21.3	70.7	25.3	30.0	26.0	82.7	6.7	16.7	12.0	42.4

Table 2: Closed-loop success (%) on the sixteen memory tasks, grouped as in RoboMME Table 3: end-to-end VLAs, memory-VLA architectures, VLM symbolic-memory pipelines, privileged oracles (gray). Baseline and oracle rows are RoboMME-reported; both Gamma rows are means over three serving seeds at 30 episodes/task/seed behind the oracle rows’ own fixed

\pi_{0.5}

(per-seed numbers in Appendix E ). The 0.8B agents row swaps only the agents’ base model (Section 5.3 ). Bold: best non-privileged result per task, excluding the 0.8B row.

	Temporal				Spatial				Object				Procedural
Configuration	PickX	BinF	SwingX	StopC	VU	BU	VUS	BUS	PH	VRP	VPB	VPO	MC	IP	PL	RS	Avg
Writer / reasoner base: Qwen3.5-9B
Gamma (full harness)	98.9	71.1	68.9	35.5	86.7	84.4	75.6	35.6	71.1	65.6	75.5	66.7	62.2	3.3	98.9	56.7	66.0
w/o defer (press-verify gate)	95.9	70.1	67.2	36.8	86.1	56.7	75.4	13.1	71.6	67.9	73.2	65.6	64.4	6.5	97.8	55.3	62.7
w/o reject (extractor-owned claims, duplicates)	95.6	71.3	66.7	13.5	85.7	83.2	76.1	36.1	70.4	65.3	74.8	63.1	67.2	3.3	3.3	6.7	55.1
w/o correct (grounding rebinding)	98.3	72.3	66.7	36.1	85.9	86.2	76.4	35.1	70.9	61.3	71.4	60.2	61.2	3.3	97.5	57.3	65.0

Table 3: Harness ablation on two axes, in the layout of Table 2 . Top : one verdict disabled at a time at 9B (three seeds, 30 episodes/task/seed); each coloured cell loses at least four points, in the removed verdict’s colour— defer blue, reject orange, correct violet. Bottom : only the base model is replaced; grey marks the same four-point loss on the no harness rows.

Figure 4: The audit loop: every tick is logged as text (left, a real ButtonUnmaskSwap tick whose press claim is deferred because the button is still detected); each failed episode is attributed mechanically to one defect class (middle); the defects become the findings on the right, each turned into a predicate validated by its closed-loop delta.

RoboMemArena, task / subtask success (%)	Transferring	Occlusion	Counting	Sequence	Average
$\pi_{0.5}$ (no subgoals)	20.0 / 42.8	12.7 / 17.2	14.3 / 50.9	60.0 / 71.6	21.5 / 38.7
MemoryVLA ( Shi et al., 2025 )	15.0 / 37.2	7.3 / 13.1	14.3 / 55.1	37.5 / 65.2	15.0 / 35.3
MemER ( Sridhar et al., 2025 )	20.0 / 36.1	16.4 / 33.2	27.1 / 65.1	65.0 / 79.1	27.3 / 49.1
PrediMem ( Lei et al., 2026 )	22.5 / 45.2	27.3 / 38.4	45.7 / 69.3	72.5 / 89.5	38.5 / 55.2
Gamma (ours)	27.4 / 37.6	19.2 / 50.1	52.3 / 67.5	80.7 / 90.3	38.8 / 59.0
ground-truth subtask feed (privileged)	32.5 / 54.8	33.6 / 49.8	51.4 / 75.6	85.0 / 92.3	46.1 / 64.8

Table 4: RoboMemArena by task family (the benchmark’s Table 2 layout): baseline and privileged rows as reported; Gamma on held-out layouts, 50 episodes per task. Bold: best non-privileged result per column.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The propose–verify harness (companion to Algorithm 1 ), one example verdict per claim type: admit commits the claim and advances the plan; defer holds it with no effect—the same subgoal keeps being issued until its evidence appears; correct rebinds a mis-grounded coordinate to the evidence-consistent detection ( reject omitted for clarity). The same contract generates the training corpus.

Figure 6: Three more harness cases from real episodes (9B agents, serving seed 7), in the grammar of Figure 3 . A , StopCube: the press is an extractor-owned claim—the cube’s arrivals at the target are counted from detections, the next one is predicted from the observed period, and the press fires at the measured oracle lead; the reasoner’s own press claims are rejected until then. B , PatternLock: the dot-route reader takes the lit-dot sequence as stroke endpoints; the writer’s words are wrong on both strokes it wrote and it never wrote the third, so its lines are rejected and the reader’s admitted. C , VideoUnmaskSwap: the shuffle is written as a chain of six move lines, each admitted only with a measured container displacement in its own window, plus the summary; the reasoner’s lookup at execution matches the oracle’s coordinate to within one pixel.

Task (seed 7, 30 episodes)	strokes reached	sense correct	side correct	both	episodes fully correct
RouteStick	79	69	73	66	23/30
PatternLock	85	–	–	80	27/30

Appendix

Table 5: Reader record vs. the live oracle on the held-out evaluation demonstrations: strokes are counted once the oracle issues them; for PatternLock a stroke is correct when its eight-way word matches. On the development annotations the readers score

33/33

strokes and

10/10

episodes respectively.

Guard	Claim type	Evidence	Verdict	Triggers (seed 7)
Press-verify gate	progress (press)	pressed object still detected in the window	defer	not logged
Press latch	progress (press)	object occluded under the arm for consecutive frames	admit	not logged
Arrival-timed press (StopCube)	progress (press)	counted arrivals + predicted period; writer’s press rejected until then	reject / extractor	not logged
Stale-coordinate re-snap	grounding	cited coordinate matches a demo location but no current detection; nearest class-compatible detection	correct	not logged
Container-coordinate authority	grounding	deterministic fold of the admitted move chain disagrees with the cited container	correct	not logged
Direction words (RouteStick)	demo record	reader’s stroke sense/side supersede the writer’s	reject writer	1,640 words / 30 eps

Appendix

Table 6: Serve-side guards of the released server (9B configuration). Word counts are the number of writer/reasoner direction words overridden by the reader over the 30 seed-7 episodes of each task (every tick re-issues the stroke, so one stroke accrues many overrides).

Task	v1	v2	v3	v4	v5	v6	v7	v8
PickXtimes	9	6	8	8	9	10	9	10
BinFill	9	2	7	6	8	8	6	7
SwingXtimes	7	3	6	1	6	5	8	8
StopCube	3	3	1	1	2	5	5	5
VideoUnmask	10	8	10	10	10	10	10	8
ButtonUnmask	5	8	6	7	9	7	7	9

Appendix

Table 7: Closed-loop screening by round: successes out of 10 episodes per task (seed 7). v1 introduced the two-agent system with the SAM-3 detector; v2 moved completion registration to the reasoner (reverted in v3); v4 recorded shuffles as move chains; v5 exposed the train/serve geometry mismatch ( † stopped after 5 and 3 episodes); v6 added the arrival predictor and the fold authority; v7 is the released corpus and checkpoints; v8 added the deterministic demonstration readers and the press latch. The v8 column is the first 10 episodes of the final seed-7 run (Table 14 ).

Task	Ticks	Content%	Gnd med px	${>}14$ px%	Trans.	Adopted%	Lag	Premature%
ButtonUnmask	237	60.3	5.7	37.8	9	100.0	2	10.0
ButtonUnmaskSwap	680	37.4	7.8	41.3	17	100.0	1	48.3
StopCube	154	72.1	5.0	0.0	20	50.0	0	0.0
VideoUnmaskSwap	77	100.0	38.2	90.9	0	–	–	0.0
VideoRepick	114	93.9	25.6	69.5	6	100.0	1	0.0
VideoPlaceOrder	124	98.4	7.1	13.1	10	100.0	0	0.0

Appendix

Table 8: Tick-level diagnosis of GroundSG+QwenVL (10 episodes/task, one seed). Binding tasks: near-perfect content, off-instance grounding (VUS:

91\%

of coordinates beyond the radius—and its episodes never reach the first oracle transition). Direction tasks: content collapse. Press-counting: missed (StopCube adopts half the transitions) and premature switches (BUS:

48\%

premature despite the implementation’s hard-coded hold periods).

Figure 7: Three-way tick-level diagnosis (10 episodes/task, seed 7). (a) Grounding error on content-matched ticks for the three binding tasks: both single-VLM pipelines put most cited coordinates beyond the

14

px wrong-instance radius (dashed)—the right words about the wrong object—while Gamma ’s coordinates are detector readings carried by admitted bank lines and land within a few pixels. (b) ButtonUnmaskSwap episode 1, each system against the live oracle of its own run (Table 11 ): MemER declares the second press five ticks before its oracle registers the first and never returns; Gamma ’s writer claims the first press at step 128, the claim is deferred while the button is still detected and admitted one tick later, and the stream follows the oracle thereafter. (c) The family profile of Table 9 : binding fails by grounding, press-counting by timing, direction by content.

Family (tasks)	Tick-level metric	GroundSG+QwenVL	MemER	Gamma
Binding (VUS, VRP, VPO)	coordinates beyond wrong-instance radius (%) $\downarrow$	51.7	36.2	13.5
Press-counting (BU, BUS)	premature subgoal switches (%) $\downarrow$	38.5	67.4	17.0
Direction (PL, RS)	subgoal text matches oracle (%) $\uparrow$	6.5	19.5	100.0

Appendix

Table 9: Tick-level diagnosis of the single-VLM pipelines and Gamma on their shared tasks (10 episodes/task, seed 7): each fails at the stage a harness verdict owns.

	GroundSG+QwenVL			MemER			Gamma
Task	content	${>}14$ px	prem.	content	${>}14$ px	prem.	content	${>}14$ px	prem.
ButtonUnmask	60.3	37.8	10.0	45.6	7.7	64.9	70.6	0.0	28.6
ButtonUnmaskSwap	37.4	41.3	48.3	50.5	35.9	69.2	57.7	2.5	12.8
StopCube	72.1	0.0	0.0	–	–	–	89.1	0.0	23.8
VideoUnmaskSwap	100.0	90.9	0.0	100.0	81.2	0.0	82.6	1.0	40.0
VideoRepick	93.9	69.5	0.0	74.4	24.4	5.3	83.1	11.9	8.3

Appendix

Table 10: Per-task tick-level diagnosis of the three systems on the tasks each was run on (10 episodes/task, seed 7): per-tick subgoal content match with the live oracle (%), share of content-matched coordinates beyond the

14

px wrong-instance radius (%), and share of predicted switches that are premature (%). Content is a strict per-tick string match: Gamma ’s deferred switches lower it on the binding tasks while it adopts

91

–

100\%

of oracle transitions with zero median lag. “–”: not run or no coordinate in the subgoal.

GroundSG+QwenVL
step	live oracle (this run)	predicted subgoal	writer claim $\to$ verdict
0	press the first button at $\langle 64,154\rangle$	press the first button at $\langle 62,146\rangle$
128	press the second button at $\langle 61,103\rangle$	press the second button at $\langle 58,99\rangle$
192	pick up the container at $\langle 109,84\rangle$ that hides the red cube	press the second button at $\langle 58,99\rangle$
288	pick up the container at $\langle 109,84\rangle$ that hides the red cube	pick up the container at $\langle 94,118\rangle$ that hides the red cube
outcome: failure (episode ends at step 1312)

Appendix

Table 11: Case study, press-counting: ButtonUnmaskSwap episode 1 (seed 7) as seen by the three systems. Live oracle and prediction per tick; for Gamma , the writer’s claims and their verdicts.

GroundSG+QwenVL
step	live oracle (this run)	predicted subgoal	writer claim $\to$ verdict
0	pick up the correct cube at $\langle 83,101\rangle$ for the first time	pick up the correct cube at $\langle 82,123\rangle$ for the first time
outcome: failure (episode ends at step 128)
MemER
0	pick up the correct cube at $\langle 83,101\rangle$ for the first time	pick up the correct cube at $\langle 89,124\rangle$ for the first time
outcome: failure (episode ends at step 112)

Appendix

Table 12: Case study, binding: VideoRepick episode 8 (seed 7). Both single-VLM pipelines pick a cube at the demonstration-time position; Gamma ’s bank tracks the correct cube’s two moves.

GroundSG+QwenVL
step	live oracle (this run)	predicted subgoal	writer claim $\to$ verdict
0	move to the nearest right target by circling around the stick clockwise	move to the nearest right target by circling around the stick counterclockwise
outcome: failure (episode ends at step 48)
MemER
0	move to the nearest right target by circling around the stick clockwise	move to the nearest right target by circling around the stick clockwise
48	move to the nearest right target by circling around the stick counterclockwise	move to the nearest right target by circling around the stick clockwise

Appendix

Table 13: Case study, direction: RouteStick episode 0 (seed 7). The stroke sense is read at chance by the VLM pipelines and mechanically by Gamma ’s reader; progress is advanced by admitted completion claims.

Figure 8: A dropped decisive event, on one held-out VideoUnmaskSwap episode (seed 7, episode 9) seen by both systems. Top: the demonstration shuffles the three containers six times over frames 73–109 (SAM-3 container detections in red). Bottom left: MemER’s keyframe memory holds only execution frame 0 at every tick and never a demonstration frame (the demonstration is attached to each query as a raw video clip instead); its prediction points at the container standing where the red cube’s container started , now hiding the blue cube, 34 px from the oracle. Bottom right: Gamma ’s bank carries the six moves as individually admitted lines and the resulting state; the reasoner’s lookup lands within 2 px of the oracle.

Seed	PickX	BinF	SwingX	StopC	VU	BU	VUS	BUS	PH	VRP	VPB	VPO	MC	IP	PL	RS	Avg
7	100.0	63.3	66.7	43.3	86.7	83.3	80.0	36.7	70.0	76.7	73.3	70.0	60.0	6.7	96.7	53.3	66.7
8	96.7	66.7	73.3	36.7	83.3	93.3	66.7	40.0	73.3	63.3	76.7	63.3	63.3	3.3	100.0	60.0	66.3
9	100.0	83.3	66.7	26.7	90.0	76.7	80.0	30.0	70.0	56.7	76.7	66.7	63.3	0.0	100.0	56.7	65.2
mean	98.9	71.1	68.9	35.5	86.7	84.4	75.6	35.6	71.1	65.6	75.5	66.7	62.2	3.3	98.9	56.7	66.0

Appendix

Table 14: Gamma per-task success (%) by serving seed, 30 episodes/task each.

Seed	PickX	BinF	SwingX	StopC	VU	BU	VUS	BUS	PH	VRP	VPB	VPO	MC	PL	RS	Avg
7	73.3	46.7	63.3	46.7	66.7	76.7	46.7	30.0	53.3	73.3	90.0	76.7	46.7	96.7	60.0	59.2
8	76.7	60.0	43.3	43.3	63.3	80.0	43.3	23.3	53.3	63.3	83.3	83.3	40.0	100.0	63.3	57.5
9	80.0	56.7	50.0	30.0	66.7	73.3	40.0	23.3	46.7	70.0	90.0	73.3	36.7	100.0	53.3	55.6
mean	76.7	54.4	52.2	40.0	65.6	76.7	43.3	25.6	51.1	68.9	87.8	77.8	41.1	98.9	58.9	57.4

Appendix

Table 15: Gamma with the 0.8B agents: per-task success (%) by serving seed, 30 episodes/task each.

Our protocol: held-out layouts (seeds 50–99), 50 episodes per task unless marked
Row (task / subtask success, %)	Transferring	Occlusion	Counting	Sequence	Average
$\pi_{0.5}$ executor, plan fed on benchmark stage predicates (privileged)	3.5 / 23.6	2.4 / 40.7	11.4 / 36.5	35.0 / 68.1	10.0 / 41.1
$\pi_{0.5}$ executor, plan fed on segment-completion conditions (privileged, 10 ep.)	7.5 / 24.6	4.5 / 41.9	22.9 / 43.3	65.0 / 83.8	19.2 / 46.1
Gamma full pipeline: writer + harness, claims settled two ticks	27.4 / 37.6	19.2 / 50.1	52.3 / 67.5	80.7 / 90.3	38.8 / 59.0
Gamma harness predicates only, no writer	27.5 / 37.3	18.9 / 49.2	52.0 / 67.2	81.0 / 90.7	38.7 / 58.6
Gamma writer + harness, claims admitted on the tick they are verified (10–20 ep.)	1.2 / 20.2	3.2 / 45.1	24.3 / 42.1	72.5 / 86.2	19.2 / 46.8

Appendix

Table 16: RoboMemArena by task family (task / subtask success, %). Family sizes: sequence 4 tasks, counting 7, occlusion 11, transferring 4. The upper block is our protocol; the lower block reproduces the authors’ Table 2 under their own protocol for reference.

	Writer (autoregressive rollout)				Reasoner
Task	exact%	recall%	precision%	known%	ticks	exact%
PickXtimes	99.2	97.6	97.6	97.5	260	80.0
BinFill	94.4	76.4	98.6	94.3	396	68.2
SwingXtimes	95.4	87.0	95.3	99.1	262	94.7
StopCube	80.0	65.2	91.9	96.3	240	88.7
VideoUnmask	100.0	100.0	100.0	100.0	126	100.0

Appendix

Table 17: Open-loop quality of the released agents on the held-out validation split (

5{,}904

ticks,

1{,}211

event ticks). The direction tasks are where the writer’s own lines are weakest, which is why their demonstration records are reader-owned at deployment (Appendix A ).

Class	Text prompt	Class	Text prompt
red_cube	a small red block on the table	target	a flat square target marker on the table
green_cube	a small green block on the table	bin	a white open box
blue_cube	a small blue block on the table	peg	a long cylindrical peg on the table
container	white container box	stick	a thin stick standing on the table
button	a grey push button on the table	arm	robot arm

Appendix

Table 18: SAM-3 text prompts per object class (score threshold

0.35

Figure 9: Frames behind the PickXtimes bank above (episode 3, agent view; markers are the coordinates the admitted line cites). Each completed: pick up line is admitted at the tick that shows the cube leaving the table in the gripper, and each place line at the tick the cube reaches the target; counting the fourth pick is then a count over admitted lines.

Figure 10: Frames behind the ButtonUnmaskSwap bank above (episode 1). Left: the containers are admitted once they are detected over the cubes. Middle: after the swaps, the summary line records which container hides which cube, and its three coordinates are the detections in this window. Right: the press completion is admitted at the tick the button is occluded under the gripper—the evidence the defer gate waits for.

Figure 11: Frames behind the VideoRepick bank above (episode 1); the marked cube is the same physical object in all three. Left: the cube the demonstration picks, at $\langle 106,136\rangle$ . Middle: the end of the demonstration, after the shuffle has moved it to $\langle 135,114\rangle$ , the coordinate the tracker lines carry. Right: the admitted pick at execution, on that coordinate rather than on the demonstration-time one.

Figure 12: Frames behind the RouteStick bank above (episode 13). Left and middle: the two demonstration strokes whose turning sense the deterministic reader extracts from the ink the stick leaves (the pale arc), and on which the writer’s own words measured at chance. Right: the arm retracing stroke 1 at execution, with the arc-tip detection the writer records.

Memory-Augmented Agents

Memory-augmented agents use stored experience and knowledge to act coherently across long tasks, changing environments, and repeated interactions.

See all 348 papers

AGM: Achievement-Grounded Memory for Closed-Loop Agents with Frozen VLA Policies
Aug 30, 2026Hongbo Gao, Zeyu Ni, Xin Wen +4Memory-Augmented VLMs Robotic Manipulation
Harbin Institute of Technology · The University of Sydney · StellarEdge AI
ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies
Sep 29, 2026Yaxin Zhao, Dianye Huang, Chenwei Wang +2Memory-Augmented VLMs Long-Horizon Robotic Manipulation
Harbin Institute of Technology, Harbin, China. · Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China
When Plans Change, Remember: Replanning-Guided Visual Memory for Vision-Language-Action Models
Oct 15, 2026Victoria Smelova, Daniil Kazachkov, Georgiy Kazantsev +6Memory-Augmented VLMs Vision-Language-Action Models
MIRAI · Independent Researcher · Central University +2

Aug 30, 2026J/K move · Enter open · S save

AGM: Achievement-Grounded Memory for Closed-Loop Agents with Frozen VLA Policies

Hongbo Gao, Zeyu Ni, Xin Wen, Siyu Xu+3

Harbin Institute of Technology · The University of Sydney · StellarEdge AI

Memory-Augmented VLMs Robotic Manipulation Vision-Language-Action Models Embodied AI

Open paper AskPDF arXiv

Frozen vision-language-action (VLA) policies offer broad manipulation skills but execute open-loop action chunks without tracking task progress, so the agent cannot reliably decide whether to continue, retry, or terminate. External memory is a natural remedy, yet we find it can be harmful when attempted actions are recorded as completed progress: transient execution failures become persistent task-state errors, and such a memory can underperform no progress memory at all. We propose Achievement-Grounded Memory (AGM), a lightweight closed-loop framework for frozen VLA policies. AGM represents a task as a static subgoal sequence with a dynamic progress pointer and advances the pointer on physically verified achievement rather than on attempts. Proprioceptive gripper-load cues decide when to verify; coherent point tracking verifies grasps, and language-conditioned cross-view comparison, read by a single trained 2.43M-parameter verification head, verifies placements. The policy, tracker, and encoder remain frozen, the head is the only trained component, and deployment needs no auxiliary vision-language model. On the RoboMME Counting benchmark, AGM reaches 100.0% on PickXTimes and 84.0% on BinFill, surpassing the strongest memory-augmented baseline by 7.7 points on the four-task average, and the gains carry over to a physical robot, where AGM reaches 100.0% and 82.0%. These results suggest that reliable embodied memory depends more on disciplined state updates than on memory capacity.

Abstract

Memory-Augmented Agents

Explore similar work

AGM: Achievement-Grounded Memory for Closed-Loop Agents with Frozen VLA Policies

ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies

When Plans Change, Remember: Replanning-Guided Visual Memory for Vision-Language-Action Models