inr2action: Encoding Robot Episodes as Neural Implicit Representations for Recall and Action

Leann Alhashishi; Christos Margadji; Sebastian William Pattinson

cs.ROOct 20, 2026

inr2action: Encoding Robot Episodes as Neural Implicit Representations for Recall and Action

Authors: Leann Alhashishi, Christos Margadji, Sebastian William Pattinson

Organizations: Institute for Manufacturing, Department of Engineering, University of Cambridge · Matta Labs, 77 East Road, London, United Kingdom

Abstract

Long-horizon robotic operation requires memory that scales with essential information rather than execution time. We present a parameter-space memory framework where each episode is stored as a separable object. A neural implicit representation is fitted to episode video from a shared, meta-learned initialization; the resulting weight delta constitutes the memory. Crucially, this delta remains a fixed size regardless of whether the episode spans a hundred or a hundred thousand frames. A learned projector maps the weight delta into token embeddings, conditioning a language model for question answering and a decoder for joint command regression, without frame retrieval at query time. On RoboMME, the language model exceeds chance across all question types, and the decoder achieves a mean $R^2$ of 0.51. Replacing the delta with matched noise drops performance by 11.4 points, while random-origin weights are non-decodable, proving the meta-learned initialization is essential for embedding episode content. By representing individual experiences as fixed-size weight deltas in a common parameter space, this approach decouples memory cost from time horizon and provides a basis for scalable streaming memory.

Figures & tables

Fig. 1: The overall system using weights as memory. An episode is encoded by fitting an INR from a meta-learned initialization $\theta$ ; the weight delta $\Delta$ is the memory and reconstructs the frames. With no frame access, the same $\Delta$ is decoded into the next joint command and a natural-language answer.

Fig. 2: Overall System Pipeline. (1) Each episode is fitted as an INR from the shared meta-learned initialization $\theta$ , yielding a weight delta $\Delta$ . (2) Each tensor of $\Delta$ is projected independently and pooled into 8 tokens, giving 56 weight-tokens. (3) The tokens condition either the recall head (TinyLlama-1.1B) or the action head (Pythia-400M), with no access to frames.

Fig. 3: RoboMME ground truth at a single frame, illustrated on an example PickXtimes episode. Each frame is paired with the label fields read out when it serves as the cut frame $T_{\text{cut}}$ : the gripper state ( gripper ), the atomic action in progress (choice_action), the current sub-step ( subgoal ), the instruction governing the episode ( task_goal ), and the commanded joint vector ( joint_action ).

Fig. 4: t-SNE of INR weight space, meta (Flavor 2) versus random initialization (Flavor 1), colored by memory suite. Both embeddings (PCA-100, perplexity 30) use the same 8 tasks, 2 per suite, 100 samples each. Axes are arbitrary.

Question Type	Chance (%)	Stage 1 (%)	Stage 2 (%)	Gain (pp)	$\kappa$
choice_action	32.0	$33.9\pm 0.6$	$\mathbf{48.1\pm 2.8}$	+16.1	0.444
gripper	50.0	$62.4\pm 8.2$	$\mathbf{69.1\pm 4.5}$	+19.1	0.371
subgoal	25.0	$29.2\pm 0.9$	$\mathbf{34.5\pm 2.7}$	+9.5	0.326
task_goal	29.5	$29.1\pm 0.6$	$\mathbf{33.0\pm 0.7}$	+3.5	0.309
Overall	34.6	$39.6\pm 1.8$	$\mathbf{47.2\pm 2.6}$	+12.6	0.446

TABLE I: End-to-end QA performance (meta-learned initialization, pixel-only fitting). Debiased accuracy (mean

\pm

std across 2 seeds) after subtracting the LLM prior NLL (Eq. 6 ); chance is the per-item random-guess baseline; gain is Stage 2 accuracy minus chance in percentage points.

Fig. 5: Choice-scoring at inference , illustrated on an example item from Flavor 2. Every candidate answer is scored by its prior-debiased negative log-likelihood (Eq. 6 ). Scores are shown beside each candidate and the prediction is the minimum. A subgoal item answered correctly, with the true candidate separated from the distractors by a clear margin.

Fig. 6: Decoding actions from weight-space memory. Left: predicted against true actions pooled over the seven continuous joints, each z-scored, with density on a log scale; pooled $R^{2}$ is 0.53. Right: full predicted profiles (orange) against ground truth (black) for episodes at fixed percentiles of mean absolute error, selected by rank rather than by hand.

		Accuracy by Question Type (%)
Flavor	Config.	choice_action	gripper	subgoal	task_goal	Overall (%)	$\Delta_{\text{noise}}$ (pp)
		(32.0)	(50.0)	(25.0)	(29.5)	(34.6)
1	R, P	$41.5\pm 0.7$	$50.2\pm 0.5$	$28.6\pm 1.3$	$29.4\pm 0.4$	$38.0\pm 0.5$	$0.4\pm 0.1$
2	M, P	$\mathbf{48.1\pm 2.8}$	$\mathbf{69.1\pm 4.5}$	$34.5\pm 2.7$	$33.0\pm 0.7$	$\mathbf{47.2\pm 2.6}$	$\mathbf{11.4\pm 0.8}$
3	M, D	$37.6\pm 0.6$	$52.5\pm 1.8$	$25.3\pm 1.0$	$30.2\pm 1.9$	$37.0\pm 0.8$	$2.5\pm 0.8$
4	M, P+D	$46.1\pm 4.7$	$65.6\pm 0.9$	$31.6\pm 2.0$	$32.8\pm 0.6$	$45.0\pm 2.1$	$9.7\pm 1.8$

TABLE II: Ablation over INR configurations, Stage 2 debiased accuracy on the test split, mean

\pm

std over 2 seeds. Init. is meta-learned (M) or random (R); heads are pixel (P), DINO (D), and CLIP (C).

\Delta

noise is the overall accuracy lost when

\Delta

is replaced by matched Gaussian noise.

Long-Term Memory for Embodied Agents

See all 35 papers

MEM: Multi-Scale Embodied Memory for Vision Language Action Models
Mar 4, 2026Marcel Torne, Karl Pertsch, Homer Walke +14Memory-Augmented Agents Long-Term Memory for Embodied Agents
Physical Intelligence · Stanford University · UC Berkeley +1
MEMORA: Embodied Action Memory from Egocentric Videos for Reasoning and Planning
Jul 15, 2026Zihao Yu, Xiu Yuan, Chongjie ZhangLong-Term Memory for Embodied Agents Embodied AI
Washington University in St. Louis
ECoMEM: Explicit Concept Memory for Memory-Dependent Robot Control
Sep 30, 2026Yize Liu, Ke Wang, Mac Schwager +2Robotic Control Long-Horizon Robotic Manipulation
Stanford University

Mar 4, 2026J/K move · Enter open · S save

MEM: Multi-Scale Embodied Memory for Vision Language Action Models

Marcel Torne, Karl Pertsch, Homer Walke, Kyle Vedder+13

Physical Intelligence · Stanford University · UC Berkeley+1

Memory-Augmented Agents Long-Term Memory for Embodied Agents Robotic Control Long-Horizon Robotic Manipulation Vision-Language-Action Models ICL for Robotics

Open paper AskPDF arXiv

Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.

Abstract

Long-Term Memory for Embodied Agents

Explore similar work

MEM: Multi-Scale Embodied Memory for Vision Language Action Models

MEMORA: Embodied Action Memory from Egocentric Videos for Reasoning and Planning

ECoMEM: Explicit Concept Memory for Memory-Dependent Robot Control