Warehouse robot fleets continuously observe operational conditions, but preserving these observations as reusable semantic memory incurs substantial computation, communication, and storage costs. For example, explaining recurring aisle blockages can require reviewing video from multiple robots across several shifts. We present IndustrialRemembr, a shared spatiotemporal memory system that extends NVIDIA ReMEmbR to robot fleets and reduces server-side computation by generating scene descriptions only on demand. On each robot, an NVIDIA Jetson encodes video clips into retrieval embeddings with Cosmos-Embed1, vision tokens with Cosmos3-Edge, and reconstruction latents with the Wan2.2 VAE, without uploading raw camera frames. The encoding models are re-used to detect people in image patches, zeroing them out to support PII abstraction for GDPR-compliant deployments. We split Cosmos3-Edge at its vision-language boundary, placing the vision encoder on the robot and the language decoder on an on-premises server. A Nemotron agent retrieves memories by content, position, and time and queries the decoder using cached vision tokens. This design eliminates caption generation at write time, while cached-token decoding reproduces the unsplit model's output in all 450 tested decodes.
Figures & tables
Figure 1: Industrial ReMEmbR moves encoding onto the robots and defers text generation to the server, until a question arrives. Each robot encodes its camera clips on board, and only the resulting embedding, vision tokens and video latent are uploaded to the fleet memory. On the server, incoming embeddings are scored continuously to keep a live semantic traffic map up to date for the fleet manager. When a question arrives, the agent searches the memory and has the retrieved clips described on demand, and an operator can reconstruct any scene for inspection.
Figure 2: Upgraded ReMEmbR indexes video directly instead of caption text, so retrieval is no longer limited to what the captioner chose to say. Original ReMEmbR is shown on top and Upgraded ReMEmbR at the bottom, with the replaced components in green. In the original, the index is built from caption text, so retrieval can find only what the captioner chose to say. Upgraded ReMEmbR sends each clip down two independent paths: Cosmos-Embed1 produces the video embedding that serves as the index key, and Cosmos3-Edge produces the caption the agent reads once an observation is retrieved. Both workflows still caption every segment in advance. Bidirectional arrows denote agent tool calls and returned evidence.
Figure 3: Industrial ReMEmbR defers text generation until a question arrives, whereas Upgraded ReMEmbR (top) fixes the text an answer can draw on when the clip is recorded. Upgraded ReMEmbR runs the full Cosmos3-Edge captioner on every clip; Industrial ReMEmbR (bottom) runs only encoders at that point and keeps their outputs in fleet memory. The language half of Cosmos3-Edge runs once a question has arrived, on the memories the agent retrieves, and the Wan2.2 decoder turns a stored latent back into video when someone wants to see the scene. The dark tag marks the step that generates text. Green boxes are models, blue boxes are storage.
Figure 4: Cached vision tokens let the server describe a scene on demand, without the source frames. The robot runs only the vision half of Cosmos3-Edge and stores its vision tokens (green). When a question arrives, the server’s language model receives the usual chat template and the embedded question (gray), with the cached vision tokens spliced into the slot where encoded video would normally go, after checking that the token counts match. The same cached tokens can answer any number of different questions.
Figure 5: Robots send only a vector, pose and time, and the server turns them into a live traffic map. (1) Each robot embeds its camera clip with Cosmos-Embed1 and sends only the vector, its pose and the time. (2) The vector is compared with cached prompt vectors for an occupied and a clear aisle, and the closer one becomes the verdict stored with the memory. (3) Every 10 s, the map service asks the agent whether there is traffic ahead of each robot. (4) Each answer colors the robot’s zone, and the pose of the memory the agent retrieved is drawn beside the robot. Green boxes are NVIDIA models, blue boxes are storage.
Figure 6: Zone colors encode the agent’s latest answer per zone, and an operator can request a description of the memory behind it. Each zone takes the color of the agent’s latest answer for the robot inside it, and each robot is drawn at its VDA5050 pose beside the memory the agent retrieved for it. Selecting a robot shows its answer and lets the operator request a description, which the Cosmos3-Edge decoder writes from the vision tokens already in memory. The zone layout and robots are illustrative.
Figure 7: Operators can request an on-demand description from the Industrial ReMEmbR frontend. The popover window allows users to generate on-demand descriptions using vision tokens.
Result
Compared
Baseline
This work
Section
First-attempt parse failures (NaVQA)
Original vs. Upgraded ReMEmbR
20.7%
13.8%
6.1
Unscorable answers (NaVQA)
Original vs. Upgraded ReMEmbR
4.5%
1.9%
6.1
Retrieval recall (NaVQA)
Caption-text vs. video index
20.8%
24.6%
6.1
Description GPU equiv. (100 robots)
Every clip vs. 10% described
36.25
1.04
6.3
Traffic balanced accuracy
Best caption vs. embedding
65.3%
71.3%
6.4
Table 1: Main results and their baselines.
System
Swapped in
First-try
Unscorable
Recall
Position
fail. (%) ↓
(%) ↓
(%) ↑
err. (m) ↓
Original ReMEmbR
–
20.7
4.5
22.9
69.1
One swap
Agent: Nemotron
16.2
0.9
18.7
71.0
Captioner: Cosmos3-Edge
22.4
3.6
26.7
74.6
Index: Cosmos-Embed1
23.5
5.5
29.7
79.4
Upgraded ReMEmbR
All three
13.8
1.9
24.6
66.6
Table 2: Swapping one component of original ReMEmbR at a time on full NaVQA, three trials per configuration. Original ReMEmbR uses Llama-3.1-8B, VILA-1.5-13B and MxBai. Each one-swap row replaces one of them and keeps the other two. Bold marks the best single swap in each column where it beats original ReMEmbR. Tab. 5 reports all eight configurations.
Operation
Reads from memory
Output
Tokens
Time (s)
Describe
Vision tokens, 247 kB
One-word answer
2
0.03
List of obstacles
43
0.38
Scene description
146
1.25
Inspect scene
Latent, 300 kB
Five-frame scene
1.76
Table 3: Time to elaborate one memory on the server. Descriptions use the Cosmos3-Edge language decoder and scene inspection the Wan2.2 decoder. Description times and token counts are means over the 150 frames of that run; the inspection time is a median.
Figure 8: One encoding answers many questions: the cached vision tokens serve every description without the frame. (1) A frame is encoded once into vision tokens, assigned to the robot in deployment. (2) The server decodes the logged answers on the right from the same cached tokens, without receiving the frame. The scene description is excerpted; its token count and time refer to the full response.
Figure 9: A stored latent is enough to reproduce a scene on the server for inspection. Shown are three observations of the traffic run. Top: reference observations used in the fidelity evaluation. Middle: the scene reconstructed on the server from each memory’s stored latent. Bottom: absolute pixel differences, with full brightness marking an error of at least 10% of the intensity range. The examples cover a forklift ahead, a robot ahead and a clear aisle and were selected for scene content variety.
Figure 10: The three iw.encode services fit in 6.55 GiB of resident GPU memory, against a 26.3 GiB sampled process peak for the original server-side captioner. The figure shows GPU-memory profiles for both (Tab. 8 ); Cosmos3-Edge loads only its vision encoder.
Figure 11: Measured GPU time for captioning, extrapolated at one memory per robot every 12 s. Always-on captions every memory as written (4.35 s). Describe-on-demand captions a memory only when asked (1.25 s), for 10% and 1% of memories. Squares mark 100 robots.
Figure 12: The caption-free embedding verdict (green) has the highest balanced accuracy among the reported representations. Results are on simulated warehouse traffic over 914 frames, by what each frame writes to memory; caption arms use reasoning. Scores exclude frames without usable context; coverage is listed at the right. The dashed line marks chance, where always answering “traffic” sits.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Use
Encoder
Input ( T×H×W )
Stored tensor
Size (kB)
Retrieval
Cosmos-Embed1
6×448×448
768 , FP32
3.1
Description
Cosmos3-Edge vision
1×192×320
60×2048 , BF16
246.6
Inspection
Wan2.2 VAE
5×480×832
48×2×30×52 , BF16
299.5
Appendix
Table 4: Memory record for one three-second clip. Each memory retains 546.1 kB of objects and uploads 549.2 kB with the search vector; its index row is 3.2 kB. The vision-token size includes the token-grid and prompt-template metadata the decoder reads.
Agent
Captioner
Index
First-try
Unparsable
Recall
Position
Time
Yes/no
System
fail. (%) ↓
(%) ↓
(%) ↑
err. (m) ↓
err. (min) ↓
ok ↑
Llama
VILA
MxBai
20.7
4.5
22.9
69.1
1.64
87/201
Original
Llama
VILA
Cosmos-Embed1
23.5
5.5
29.7
79.4
2.17
94/201
Llama
Cosmos3-Edge
MxBai
22.4
3.6
26.7
74.6
3.38
109/201
Llama
Cosmos3-Edge
Cosmos-Embed1
22.0
4.3
26.9
76.2
2.19
106/201
Nemotron
VILA
MxBai
16.2
0.9
18.7
71.0
2.21
81/201
Appendix
Table 5: Full seven-sequence NaVQA component ablation: three trials, 201 yes/no responses per configuration. Agents are Llama-3.1-8B and Nemotron-3-Nano-4B; VILA is VILA-1.5-13B. “First-try fail.” is the first-attempt parse-failure rate. Position and time errors are medians over parsed answers. “Original” is original ReMEmbR and “Upgraded” is Upgraded ReMEmbR .
Metric
BF16 latent
Change vs. FP32
Relative change
PSNR (dB) ↑
39.7
−0.003
0.008%
SSIM ↑
0.974
−1.3×10−5
0.001%
LPIPS ↓
0.0052
+4×10−6
0.08%
Appendix
Table 6: Reconstruction fidelity of the stored BF16 latent over 150 traffic frames, each repeated five times, and its change from decoding the FP32 latent.
Encoder
Input ( T×H×W )
Mean latency (s)
P95 latency (s)
Runs
Cosmos-Embed1
6×448×448
11.76
11.78
10
Cosmos3-Edge vision
1×192×320
0.196
0.254
16
Wan2.2 VAE (FP32)
5×480×832
6.46
6.51
10
Appendix
Table 7: Mean and P95 latency of warmed TensorRT 8.5.2 engines on a Jetson AGX Xavier with random inputs, including host–device transfers. Cosmos-Embed1 uses custom TensorRT plugins.
Process
GPU memory (GiB)
Robot (iw.encode)
Cosmos-Embed1
2.96
Cosmos3-Edge vision encoder
1.61
Wan2.2 VAE encoder
1.98
All three loaded
6.55
Server
Appendix
Table 8: GPU memory by process. Encoder values are resident memory with the services loaded; server and captioner values are peaks during inference; the NIM value excludes the key-value cache that NIM reserves.
Figure 13: NIM (FP8) scales to 1217 tokens/s at 8 concurrent requests, while Ollama (Q8_0) stays near 235. Throughput is aggregate output for Nemotron-3-Nano-4B at 8k context on one RTX PRO 6000 Blackwell Server Edition.
Concurrency
Output tok/s
Mean TPOT (ms)
Mean TTFT (s)
P99 TTFT (s)
1
155.8
5.96
0.097
0.120
2
297.3
6.11
0.149
0.167
4
570.2
6.42
0.272
0.307
8
953.3
7.21
0.447
0.541
16
1574.7
8.71
0.715
1.032
32
2026.9
12.52
1.002
1.674
Appendix
Table 9: NIM FP8 synthetic-request sweep. TTFT is time to first token; TPOT is time per output token.
Robots
1 GPU
2 GPUs
8
8.1
7.1
16
11.2
8.2
20
13.3
8.9
24
17.0
9.5
28
23.6
11.4
32
37.2
11.6
Appendix
Table 10: P95 map-update time (s) for robots polling every 10 s, with one NIM instance per GPU. A setup serves a fleet when 95% of updates finish within 20 s; 48 robots is the largest fleet tested.
Figure 14: GPU exact search’s throughput advantage over CPU exact search grows across the tested index sizes, from parity at 104 rows to 3.3 × at 105 and 19 × at 106 . Results are for 32 concurrent queries in one Milvus instance over synthetic 768-dimensional vectors; the GPU index uses NVIDIA cuVS ( RAPIDS Team, 2024 ) . Both searches return identical neighbours. The axis is logarithmic, so the length of each green bar is the relative speedup.
Event
Input
Rate
Per robot
Caption a memory (original)
tcap=4.35 s
r
0.3625 GPU eq.
Describe a memory
tdesc=1.25 s
pr
0.104p GPU eq.
Inspect a scene
tinsp
qr
qtinsp/12 GPU eq.
Update the map
ua≈2.4 updates/s/GPU
1/τ
0.042 GPU eq.
Retain objects
b=546.1 kB
r
1.31D GB
Index rows
3.2 kB
r
7.68D MB
Appendix
Table 11: Per-robot demand and storage. Sources are Tab. 3 , Tab. 4 and Tab. 10 .
Scope
Period
Memories
Uploaded
Objects
Index
1 robot
1 day
2,400
1.32 GB
1.31 GB
7.68 MB
100 robots
1 day
240,000
131.81 GB
131.07 GB
0.768 GB
100 robots
30 days
7.2 million
3.95 TB
3.93 TB
23.04 GB
100 robots
365 days
87.6 million
48.11 TB
47.84 TB
280.32 GB
Appendix
Table 12: Totals over each period at one memory every 12 s over an 8-hour day.
Memory per frame
Coverage
Accuracy
Balanced
Traffic recall
Clear
acc.
All
Low
High
recall
Embedding verdict
914/914
72.1
71.3
73.8
68.5
78.6
68.7
Detection + reasoning
844/914
72.7
65.3
88.4
87.0
89.6
42.3
Description + reasoning
902/914
48.7
59.1
27.1
25.1
28.9
91.1
Always “traffic”
914/914
66.4
50.0
100.0
100.0
100.0
0.0
Appendix
Table 13: Simulated warehouse traffic over 914 frames (%). The embedding verdict is read from the Cosmos-Embed1 vector; the detection and description arms use Cosmos3-Edge with reasoning enabled. Coverage counts frames with usable context; the other columns exclude dropped frames. Bold marks the best of the three representations in each column. Rows are ordered by balanced accuracy.
Figure 15: Fleet memory supplies Cosmos3 fine-tuning without rerunning the encoders. Cached vision tokens H feed the vision-language reasoner, and cached Wan2.2 latents z0 , noised to level t , feed the generator. Semantic filtering selects high-priority memories, and validated text labels and aligned action targets supply supervision. With everything upstream of the cache boundary frozen, the language, diffusion or action components can be adapted (Appendix G ).
Long-horizon robotic operation requires memory that scales with essential information rather than execution time. We present a parameter-space memory framework where each episode is stored as a separable object. A neural implicit representation is fitted to episode video from a shared, meta-learned initialization; the resulting weight delta constitutes the memory. Crucially, this delta remains a fixed size regardless of whether the episode spans a hundred or a hundred thousand frames. A learned projector maps the weight delta into token embeddings, conditioning a language model for question answering and a decoder for joint command regression, without frame retrieval at query time. On RoboMME, the language model exceeds chance across all question types, and the decoder achieves a mean R2 of 0.51. Replacing the delta with matched noise drops performance by 11.4 points, while random-origin weights are non-decodable, proving the meta-learned initialization is essential for embedding episode content. By representing individual experiences as fixed-size weight deltas in a common parameter space, this approach decouples memory cost from time horizon and provides a basis for scalable streaming memory.
Leann Alhashishi, Christos Margadji, Sebastian William Pattinson
Institute for Manufacturing, Department of Engineering, University of Cambridge · Matta Labs, 77 East Road, London, United Kingdom
Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.
Marcel Torne, Karl Pertsch, Homer Walke +14
Physical Intelligence · Stanford University · UC Berkeley +1
Long-term robot deployment requires a compact and scalable memory that preserves fine-grained visual semantics, grounds observations in space and time, and enables efficient storage and retrieval. In this paper, we propose RAVEN, an agentic memory system for long-horizon robotic question answering and navigation. RAVEN stores visual embeddings with pose and time in a vector database, and grounds retrieval in a spatial map to answer queries and navigate to goals. By operating directly on visual embeddings, RAVEN avoids lossy image-to-text captioning and enables accurate semantic, spatial, and temporal retrieval at scale. Across several simulated and real-world video question-answering benchmarks, RAVEN consistently surpasses caption-based memory systems and matches frontier VLMs on long-horizon tasks at 10× lower retrieval cost. Finally, we instantiate RAVEN on a Unitree Go1 robot for the task of long-horizon navigation for natural language goal-reaching, and show successful deployment over several large indoor environments.