Long-horizon robotic operation requires memory that scales with essential information rather than execution time. We present a parameter-space memory framework where each episode is stored as a separable object. A neural implicit representation is fitted to episode video from a shared, meta-learned initialization; the resulting weight delta constitutes the memory. Crucially, this delta remains a fixed size regardless of whether the episode spans a hundred or a hundred thousand frames. A learned projector maps the weight delta into token embeddings, conditioning a language model for question answering and a decoder for joint command regression, without frame retrieval at query time. On RoboMME, the language model exceeds chance across all question types, and the decoder achieves a mean R2 of 0.51. Replacing the delta with matched noise drops performance by 11.4 points, while random-origin weights are non-decodable, proving the meta-learned initialization is essential for embedding episode content. By representing individual experiences as fixed-size weight deltas in a common parameter space, this approach decouples memory cost from time horizon and provides a basis for scalable streaming memory.
Figures & tables
Fig. 1: The overall system using weights as memory. An episode is encoded by fitting an INR from a meta-learned initialization θ ; the weight delta Δ is the memory and reconstructs the frames. With no frame access, the same Δ is decoded into the next joint command and a natural-language answer.
Fig. 2: Overall System Pipeline. (1) Each episode is fitted as an INR from the shared meta-learned initialization θ , yielding a weight delta Δ . (2) Each tensor of Δ is projected independently and pooled into 8 tokens, giving 56 weight-tokens. (3) The tokens condition either the recall head (TinyLlama-1.1B) or the action head (Pythia-400M), with no access to frames.
Fig. 3: RoboMME ground truth at a single frame, illustrated on an example PickXtimes episode. Each frame is paired with the label fields read out when it serves as the cut frame Tcut : the gripper state ( gripper ), the atomic action in progress (choice_action), the current sub-step ( subgoal ), the instruction governing the episode ( task_goal ), and the commanded joint vector ( joint_action ).
Fig. 4: t-SNE of INR weight space, meta (Flavor 2) versus random initialization (Flavor 1), colored by memory suite. Both embeddings (PCA-100, perplexity 30) use the same 8 tasks, 2 per suite, 100 samples each. Axes are arbitrary.
Question Type
Chance (%)
Stage 1 (%)
Stage 2 (%)
Gain (pp)
κ
choice_action
32.0
33.9±0.6
48.1±2.8
+16.1
0.444
gripper
50.0
62.4±8.2
69.1±4.5
+19.1
0.371
subgoal
25.0
29.2±0.9
34.5±2.7
+9.5
0.326
task_goal
29.5
29.1±0.6
33.0±0.7
+3.5
0.309
Overall
34.6
39.6±1.8
47.2±2.6
+12.6
0.446
TABLE I: End-to-end QA performance (meta-learned initialization, pixel-only fitting). Debiased accuracy (mean ± std across 2 seeds) after subtracting the LLM prior NLL (Eq. 6 ); chance is the per-item random-guess baseline; gain is Stage 2 accuracy minus chance in percentage points.
Fig. 5: Choice-scoring at inference , illustrated on an example item from Flavor 2. Every candidate answer is scored by its prior-debiased negative log-likelihood (Eq. 6 ). Scores are shown beside each candidate and the prediction is the minimum. A subgoal item answered correctly, with the true candidate separated from the distractors by a clear margin.
Fig. 6: Decoding actions from weight-space memory. Left: predicted against true actions pooled over the seven continuous joints, each z-scored, with density on a log scale; pooled R2 is 0.53. Right: full predicted profiles (orange) against ground truth (black) for episodes at fixed percentiles of mean absolute error, selected by rank rather than by hand.
Accuracy by Question Type (%)
Flavor
Config.
choice_action
gripper
subgoal
task_goal
Overall (%)
Δnoise (pp)
(32.0)
(50.0)
(25.0)
(29.5)
(34.6)
1
R, P
41.5±0.7
50.2±0.5
28.6±1.3
29.4±0.4
38.0±0.5
0.4±0.1
2
M, P
48.1±2.8
69.1±4.5
34.5±2.7
33.0±0.7
47.2±2.6
11.4±0.8
3
M, D
37.6±0.6
52.5±1.8
25.3±1.0
30.2±1.9
37.0±0.8
2.5±0.8
4
M, P+D
46.1±4.7
65.6±0.9
31.6±2.0
32.8±0.6
45.0±2.1
9.7±1.8
TABLE II: Ablation over INR configurations, Stage 2 debiased accuracy on the test split, mean ± std over 2 seeds. Init. is meta-learned (M) or random (R); heads are pixel (P), DINO (D), and CLIP (C). Δ noise is the overall accuracy lost when Δ is replaced by matched Gaussian noise.
Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.
Marcel Torne, Karl Pertsch, Homer Walke +14
Physical Intelligence · Stanford University · UC Berkeley +1
Embodied agents accumulate experience over time. We study how accumulated experience can be formed into persistent memory for future reasoning and action. We formulate Embodied Action Memory (EAM) as the capability to form and use memory over embodied experience, together with the persistent memory state produced by that process. We introduce MEMORA, a framework that instantiates EAM through a formation-consolidation-retrieval lifecycle and a multi-store world-memory architecture. MEMORA organizes experience into participant-specific Environment, Entity, Activity, and Inferred Knowledge stores: online editing revises memory as new evidence arrives, while offline consolidation abstracts repeated experience into reusable routines, habits, and preferences. We evaluate MEMORA with MEMORA-Bench, a 45-hour egocentric-video suite that measures both retrospective memory faithfulness and prospective memory-grounded planning. Across four open-weight answer models, MEMORA achieves the strongest aggregate planning performance among the evaluated memory interfaces, with its largest gains on out-of-distribution planning. On these tasks, MEMORA improves Robot-Grounded Plan score by up to 16.6 percent, suggesting that memory formed and consolidated across experience can support planning for new goals beyond directly observed episodes. A physical-robot demonstration further shows that memory formed solely from human egocentric video can ground high-level robot plans in participant-specific objects and preferences. Project website: https://github.com/yuzihaowashu/MEMORA
A robot may lose sight of an object it must later retrieve, need to recall what a person demonstrated earlier, or track which steps of a task it has already completed. Current vision-language-action (VLA) policies often fail once the information needed for action disappears from the current observation, making memory critical for long-horizon robot behavior. Existing approaches typically provide longer histories or learn implicit memory from observation-action trajectories. But action supervision tells a policy how to act, not what to remember: it does not specify which past facts should persist or how they should change as new evidence arrives. We therefore separate maintaining an evidence-grounded account of the past from learning how to act on it. This insight motivates Explicit Concept Memory (ECoMEM), which represents task-relevant history with a reusable library of grounded concepts. An evidence-based Writer selects and updates these records, while a learned Reader turns them into memory tokens that directly condition the VLA. Across 16 RoboMME tasks, ECoMEM leads the evaluated robot policies on 15 tasks. On two new real-robot tasks, the same memory library either transfers directly or requires only one new concept, achieving 86.1% success versus 8.6% for a no-memory VLA. These results show that explicit concepts provide a reusable and extensible memory interface for robot control. Project website: https://ecomem.github.io/