Industrial ReMEmbR: Scaling Spatiotemporal Memory to Robot Fleets

Jad Yammine; Malek Wahidi; Abrar Anwar; Yan Chang; Mihir Acharya; Anthony Rizk

cs.ROOct 20, 2026

Industrial ReMEmbR: Scaling Spatiotemporal Memory to Robot Fleets

Authors: Jad Yammine, Malek Wahidi, Abrar Anwar, Yan Chang, Mihir Acharya, Anthony Rizk

Organizations: Idealworks · NVIDIA

Abstract

Warehouse robot fleets continuously observe operational conditions, but preserving these observations as reusable semantic memory incurs substantial computation, communication, and storage costs. For example, explaining recurring aisle blockages can require reviewing video from multiple robots across several shifts. We present IndustrialRemembr, a shared spatiotemporal memory system that extends NVIDIA ReMEmbR to robot fleets and reduces server-side computation by generating scene descriptions only on demand. On each robot, an NVIDIA Jetson encodes video clips into retrieval embeddings with Cosmos-Embed1, vision tokens with Cosmos3-Edge, and reconstruction latents with the Wan2.2 VAE, without uploading raw camera frames. The encoding models are re-used to detect people in image patches, zeroing them out to support PII abstraction for GDPR-compliant deployments. We split Cosmos3-Edge at its vision-language boundary, placing the vision encoder on the robot and the language decoder on an on-premises server. A Nemotron agent retrieves memories by content, position, and time and queries the decoder using cached vision tokens. This design eliminates caption generation at write time, while cached-token decoding reproduces the unsplit model's output in all 450 tested decodes.

Figures & tables

Panel of Figure 1 — Figure 1: Industrial ReMEmbR moves encoding onto the robots and defers text generation to the server, until a question arrives. Each robot encodes its camera clips on board, and only the resulting embedding, vision tokens and video latent are uploaded to the fleet memory. On the server, incoming embeddings are scored continuously to keep a live semantic traffic map up to date for the fleet manager. When a question arrives, the agent searches the memory and has the retrieved clips described on demand, and an operator can reconstruct any scene for inspection.

Panel of Figure 2 — Figure 2: Upgraded ReMEmbR indexes video directly instead of caption text, so retrieval is no longer limited to what the captioner chose to say. Original ReMEmbR is shown on top and Upgraded ReMEmbR at the bottom, with the replaced components in green. In the original, the index is built from caption text, so retrieval can find only what the captioner chose to say. Upgraded ReMEmbR sends each clip down two independent paths: Cosmos-Embed1 produces the video embedding that serves as the index key, and Cosmos3-Edge produces the caption the agent reads once an observation is retrieved. Both workflows still caption every segment in advance. Bidirectional arrows denote agent tool calls and returned evidence.

Panel of Figure 3 — Figure 3: Industrial ReMEmbR defers text generation until a question arrives, whereas Upgraded ReMEmbR (top) fixes the text an answer can draw on when the clip is recorded. Upgraded ReMEmbR runs the full Cosmos3-Edge captioner on every clip; Industrial ReMEmbR (bottom) runs only encoders at that point and keeps their outputs in fleet memory. The language half of Cosmos3-Edge runs once a question has arrived, on the memories the agent retrieves, and the Wan2.2 decoder turns a stored latent back into video when someone wants to see the scene. The dark tag marks the step that generates text. Green boxes are models, blue boxes are storage.

Panel of Figure 4 — Figure 4: Cached vision tokens let the server describe a scene on demand, without the source frames. The robot runs only the vision half of Cosmos3-Edge and stores its vision tokens (green). When a question arrives, the server’s language model receives the usual chat template and the embedded question (gray), with the cached vision tokens spliced into the slot where encoded video would normally go, after checking that the token counts match. The same cached tokens can answer any number of different questions.

Figure 5: Robots send only a vector, pose and time, and the server turns them into a live traffic map. (1) Each robot embeds its camera clip with Cosmos-Embed1 and sends only the vector, its pose and the time. (2) The vector is compared with cached prompt vectors for an occupied and a clear aisle, and the closer one becomes the verdict stored with the memory. (3) Every 10 s, the map service asks the agent whether there is traffic ahead of each robot. (4) Each answer colors the robot’s zone, and the pose of the memory the agent retrieved is drawn beside the robot. Green boxes are NVIDIA models, blue boxes are storage.

Panel of Figure 6 — Figure 6: Zone colors encode the agent’s latest answer per zone, and an operator can request a description of the memory behind it. Each zone takes the color of the agent’s latest answer for the robot inside it, and each robot is drawn at its VDA5050 pose beside the memory the agent retrieved for it. Selecting a robot shows its answer and lets the operator request a description, which the Cosmos3-Edge decoder writes from the vision tokens already in memory. The zone layout and robots are illustrative.

Figure 7: Operators can request an on-demand description from the Industrial ReMEmbR frontend. The popover window allows users to generate on-demand descriptions using vision tokens.

Result	Compared	Baseline	This work	Section
First-attempt parse failures (NaVQA)	Original vs. Upgraded ReMEmbR	20.7%	13.8%	6.1
Unscorable answers (NaVQA)	Original vs. Upgraded ReMEmbR	4.5%	1.9%	6.1
Retrieval recall (NaVQA)	Caption-text vs. video index	20.8%	24.6%	6.1
Description GPU equiv. (100 robots)	Every clip vs. 10% described	36.25	1.04	6.3
Traffic balanced accuracy	Best caption vs. embedding	65.3%	71.3%	6.4

Table 1: Main results and their baselines.

System	Swapped in	First-try	Unscorable	Recall	Position
		fail. (%) $\downarrow$	(%) $\downarrow$	(%) $\uparrow$	err. (m) $\downarrow$
Original ReMEmbR	–	20.7	4.5	22.9	69.1
One swap	Agent: Nemotron	16.2	0.9	18.7	71.0
	Captioner: Cosmos3-Edge	22.4	3.6	26.7	74.6
	Index: Cosmos-Embed1	23.5	5.5	29.7	79.4
Upgraded ReMEmbR	All three	13.8	1.9	24.6	66.6

Table 2: Swapping one component of original ReMEmbR at a time on full NaVQA, three trials per configuration. Original ReMEmbR uses Llama-3.1-8B, VILA-1.5-13B and MxBai. Each one-swap row replaces one of them and keeps the other two. Bold marks the best single swap in each column where it beats original ReMEmbR. Tab. 5 reports all eight configurations.

Operation	Reads from memory	Output	Tokens	Time (s)
Describe	Vision tokens, 247 kB	One-word answer	2	0.03
		List of obstacles	43	0.38
		Scene description	146	1.25
Inspect scene	Latent, 300 kB	Five-frame scene		1.76

Table 3: Time to elaborate one memory on the server. Descriptions use the Cosmos3-Edge language decoder and scene inspection the Wan2.2 decoder. Description times and token counts are means over the 150 frames of that run; the inspection time is a median.

Figure 8: One encoding answers many questions: the cached vision tokens serve every description without the frame. (1) A frame is encoded once into vision tokens, assigned to the robot in deployment. (2) The server decodes the logged answers on the right from the same cached tokens, without receiving the frame. The scene description is excerpted; its token count and time refer to the full response.

Panel of Figure 9 — Figure 9: A stored latent is enough to reproduce a scene on the server for inspection. Shown are three observations of the traffic run. Top: reference observations used in the fidelity evaluation. Middle: the scene reconstructed on the server from each memory’s stored latent. Bottom: absolute pixel differences, with full brightness marking an error of at least 10% of the intensity range. The examples cover a forklift ahead, a robot ahead and a clear aisle and were selected for scene content variety.

Panel of Figure 10 — Figure 10: The three iw.encode services fit in 6.55 GiB of resident GPU memory, against a 26.3 GiB sampled process peak for the original server-side captioner. The figure shows GPU-memory profiles for both (Tab. 8 ); Cosmos3-Edge loads only its vision encoder.

Panel of Figure 11 — Figure 11: Measured GPU time for captioning, extrapolated at one memory per robot every 12 s. Always-on captions every memory as written (4.35 s). Describe-on-demand captions a memory only when asked (1.25 s), for 10% and 1% of memories. Squares mark 100 robots.

Panel of Figure 12 — Figure 12: The caption-free embedding verdict (green) has the highest balanced accuracy among the reported representations. Results are on simulated warehouse traffic over 914 frames, by what each frame writes to memory; caption arms use reasoning. Scores exclude frames without usable context; coverage is listed at the right. The dashed line marks chance, where always answering “traffic” sits.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Use	Encoder	Input ( $T\times H\times W$ )	Stored tensor	Size (kB)
Retrieval	Cosmos-Embed1	$6\times 448\times 448$	$768$ , FP32	3.1
Description	Cosmos3-Edge vision	$1\times 192\times 320$	$60\times 2048$ , BF16	246.6
Inspection	Wan2.2 VAE	$5\times 480\times 832$	$48\times 2\times 30\times 52$ , BF16	299.5

Appendix

Table 4: Memory record for one three-second clip. Each memory retains 546.1 kB of objects and uploads 549.2 kB with the search vector; its index row is 3.2 kB. The vision-token size includes the token-grid and prompt-template metadata the decoder reads.

Agent	Captioner	Index	First-try	Unparsable	Recall	Position	Time	Yes/no	System
			fail. (%) $\downarrow$	(%) $\downarrow$	(%) $\uparrow$	err. (m) $\downarrow$	err. (min) $\downarrow$	ok $\uparrow$
Llama	VILA	MxBai	20.7	4.5	22.9	69.1	1.64	87/201	Original
Llama	VILA	Cosmos-Embed1	23.5	5.5	29.7	79.4	2.17	94/201
Llama	Cosmos3-Edge	MxBai	22.4	3.6	26.7	74.6	3.38	109/201
Llama	Cosmos3-Edge	Cosmos-Embed1	22.0	4.3	26.9	76.2	2.19	106/201
Nemotron	VILA	MxBai	16.2	0.9	18.7	71.0	2.21	81/201

Appendix

Table 5: Full seven-sequence NaVQA component ablation: three trials, 201 yes/no responses per configuration. Agents are Llama-3.1-8B and Nemotron-3-Nano-4B; VILA is VILA-1.5-13B. “First-try fail.” is the first-attempt parse-failure rate. Position and time errors are medians over parsed answers. “Original” is original ReMEmbR and “Upgraded” is Upgraded ReMEmbR .

Metric	BF16 latent	Change vs. FP32	Relative change
PSNR (dB) $\uparrow$	39.7	$-0.003$	0.008%
SSIM $\uparrow$	0.974	$-1.3\times 10^{-5}$	0.001%
LPIPS $\downarrow$	0.0052	$+4\times 10^{-6}$	0.08%

Appendix

Table 6: Reconstruction fidelity of the stored BF16 latent over 150 traffic frames, each repeated five times, and its change from decoding the FP32 latent.

Encoder	Input ( $T\times H\times W$ )	Mean latency (s)	P95 latency (s)	Runs
Cosmos-Embed1	$6\times 448\times 448$	11.76	11.78	10
Cosmos3-Edge vision	$1\times 192\times 320$	0.196	0.254	16
Wan2.2 VAE (FP32)	$5\times 480\times 832$	6.46	6.51	10

Appendix

Table 7: Mean and P95 latency of warmed TensorRT 8.5.2 engines on a Jetson AGX Xavier with random inputs, including host–device transfers. Cosmos-Embed1 uses custom TensorRT plugins.

Process	GPU memory (GiB)
Robot (iw.encode)
Cosmos-Embed1	2.96
Cosmos3-Edge vision encoder	1.61
Wan2.2 VAE encoder	1.98
All three loaded	6.55
Server

Appendix

Table 8: GPU memory by process. Encoder values are resident memory with the services loaded; server and captioner values are peaks during inference; the NIM value excludes the key-value cache that NIM reserves.

Panel of Figure 13 — Figure 13: NIM (FP8) scales to 1217 tokens/s at 8 concurrent requests, while Ollama (Q8_0) stays near 235. Throughput is aggregate output for Nemotron-3-Nano-4B at 8k context on one RTX PRO 6000 Blackwell Server Edition.

Concurrency	Output tok/s	Mean TPOT (ms)	Mean TTFT (s)	P99 TTFT (s)
1	155.8	5.96	0.097	0.120
2	297.3	6.11	0.149	0.167
4	570.2	6.42	0.272	0.307
8	953.3	7.21	0.447	0.541
16	1574.7	8.71	0.715	1.032
32	2026.9	12.52	1.002	1.674

Appendix

Table 9: NIM FP8 synthetic-request sweep. TTFT is time to first token; TPOT is time per output token.

Robots	1 GPU	2 GPUs
8	8.1	7.1
16	11.2	8.2
20	13.3	8.9
24	17.0	9.5
28	23.6	11.4
32	37.2	11.6

Appendix

Table 10: P95 map-update time (s) for robots polling every 10 s, with one NIM instance per GPU. A setup serves a fleet when 95% of updates finish within 20 s; 48 robots is the largest fleet tested.

Panel of Figure 14 — Figure 14: GPU exact search’s throughput advantage over CPU exact search grows across the tested index sizes, from parity at $10^{4}$ rows to 3.3 $\times$ at $10^{5}$ and 19 $\times$ at $10^{6}$ . Results are for 32 concurrent queries in one Milvus instance over synthetic 768-dimensional vectors; the GPU index uses NVIDIA cuVS ( RAPIDS Team, 2024 ) . Both searches return identical neighbours. The axis is logarithmic, so the length of each green bar is the relative speedup.

Event	Input	Rate	Per robot
Caption a memory (original)	$t_{\mathrm{cap}}=4.35$ s	$r$	$0.3625$ GPU eq.
Describe a memory	$t_{\mathrm{desc}}=1.25$ s	$p\,r$	$0.104\,p$ GPU eq.
Inspect a scene	$t_{\mathrm{insp}}$	$q\,r$	$q\,t_{\mathrm{insp}}/12$ GPU eq.
Update the map	$u_{a}\approx 2.4$ updates/s/GPU	$1/\tau$	$0.042$ GPU eq.
Retain objects	$b=546.1$ kB	$r$	$1.31\,D$ GB
Index rows	3.2 kB	$r$	$7.68\,D$ MB

Appendix

Table 11: Per-robot demand and storage. Sources are Tab. 3 , Tab. 4 and Tab. 10 .

Scope	Period	Memories	Uploaded	Objects	Index
1 robot	1 day	2,400	1.32 GB	1.31 GB	7.68 MB
100 robots	1 day	240,000	131.81 GB	131.07 GB	0.768 GB
100 robots	30 days	7.2 million	3.95 TB	3.93 TB	23.04 GB
100 robots	365 days	87.6 million	48.11 TB	47.84 TB	280.32 GB

Appendix

Table 12: Totals over each period at one memory every 12 s over an 8-hour day.

Memory per frame	Coverage	Accuracy	Balanced	Traffic recall			Clear
			acc.	All	Low	High	recall
Embedding verdict	914/914	72.1	71.3	73.8	68.5	78.6	68.7
Detection + reasoning	844/914	72.7	65.3	88.4	87.0	89.6	42.3
Description + reasoning	902/914	48.7	59.1	27.1	25.1	28.9	91.1
Always “traffic”	914/914	66.4	50.0	100.0	100.0	100.0	0.0

Appendix

Table 13: Simulated warehouse traffic over 914 frames (%). The embedding verdict is read from the Cosmos-Embed1 vector; the detection and description arms use Cosmos3-Edge with reasoning enabled. Coverage counts frames with usable context; the other columns exclude dropped frames. Bold marks the best of the three representations in each column. Rows are ordered by balanced accuracy.

Panel of Figure 15 — Figure 15: Fleet memory supplies Cosmos3 fine-tuning without rerunning the encoders. Cached vision tokens $\mathbf{H}$ feed the vision-language reasoner, and cached Wan2.2 latents $\mathbf{z}_{0}$ , noised to level $t$ , feed the generator. Semantic filtering selects high-priority memories, and validated text labels and aligned action targets supply supervision. With everything upstream of the cache boundary frozen, the language, diffusion or action components can be adapted (Appendix G ).

Robotic Spatial Memory

See all 29 papers

inr2action: Encoding Robot Episodes as Neural Implicit Representations for Recall and Action
Oct 20, 2026Leann Alhashishi, Christos Margadji, Sebastian William PattinsonLong-Term Memory for Embodied Agents Memory-Augmented Agents
Institute for Manufacturing, Department of Engineering, University of Cambridge · Matta Labs, 77 East Road, London, United Kingdom
MEM: Multi-Scale Embodied Memory for Vision Language Action Models
Mar 4, 2026Marcel Torne, Karl Pertsch, Homer Walke +14Memory-Augmented Agents Long-Term Memory for Embodied Agents
Physical Intelligence · Stanford University · UC Berkeley +1
RAVEN: Long-Horizon Reasoning & Navigation with a Visuo-Spatio-Temporal Memory
Jun 23, 2026Yixun Hu, Zhicheng Zheng, Lihan Zha +5Robot Navigation Embodied QA
Princeton University · University of Pennsylvania

Oct 20, 2026J/K move · Enter open · S save

inr2action: Encoding Robot Episodes as Neural Implicit Representations for Recall and Action

Leann Alhashishi, Christos Margadji, Sebastian William Pattinson

Institute for Manufacturing, Department of Engineering, University of Cambridge · Matta Labs, 77 East Road, London, United Kingdom

Long-Term Memory for Embodied Agents Memory-Augmented Agents Robot Learning Episodic Memory Implicit Neural Representations

Open paper AskPDF arXiv

Long-horizon robotic operation requires memory that scales with essential information rather than execution time. We present a parameter-space memory framework where each episode is stored as a separable object. A neural implicit representation is fitted to episode video from a shared, meta-learned initialization; the resulting weight delta constitutes the memory. Crucially, this delta remains a fixed size regardless of whether the episode spans a hundred or a hundred thousand frames. A learned projector maps the weight delta into token embeddings, conditioning a language model for question answering and a decoder for joint command regression, without frame retrieval at query time. On RoboMME, the language model exceeds chance across all question types, and the decoder achieves a mean

R^2

of 0.51. Replacing the delta with matched noise drops performance by 11.4 points, while random-origin weights are non-decodable, proving the meta-learned initialization is essential for embedding episode content. By representing individual experiences as fixed-size weight deltas in a common parameter space, this approach decouples memory cost from time horizon and provides a basis for scalable streaming memory.

Abstract

Robotic Spatial Memory

Explore similar work

inr2action: Encoding Robot Episodes as Neural Implicit Representations for Recall and Action

MEM: Multi-Scale Embodied Memory for Vision Language Action Models

RAVEN: Long-Horizon Reasoning & Navigation with a Visuo-Spatio-Temporal Memory