Many robot tasks require attending to the history of past observations. For example, finding an item in a room requires remembering which places have already been searched. However, the best-performing robot policies typically condition only on the current observation, limiting their applicability to such tasks. Naively conditioning on past observations often fails due to spurious correlations: policies latch onto incidental features of training histories that do not generalize to out-of-distribution trajectories upon deployment. We analyze why policies latch onto these spurious correlations and find that this problem stems from limited coverage over the space of possible histories during training, which grows exponentially with horizon. Existing regularization techniques provide inconsistent benefits across tasks, as they do not fundamentally address this coverage problem. Motivated by these findings, we propose Big Picture Policies (BPP), an approach that conditions on a minimal set of meaningful keyframes detected by a vision-language model. By projecting diverse rollouts onto a compact set of task-relevant events, BPP substantially reduces distribution shift between training and deployment, without sacrificing expressivity. We evaluate BPP on four challenging real-world manipulation tasks and three simulation tasks, all requiring history conditioning. BPP achieves 70% higher success rates than the best comparison on real-world evaluations. Videos are available at https://bigpicturepolicies.github.io/
Figures & tables
Figure 2: Benchmark tasks. We evaluate on 4 real-world tasks (A–D) and 3 simulation tasks (E–G). All tasks require history conditioning for success.
Figure 3: Predicting longer action chunks significantly decreases spurious correlations in naïve history-conditioned policies. Left: We compare how well the features learned by the policy predict the history state (in Fixed-Password , this is the number of buttons pressed so far). Both policies trained with shorter (10) and longer (50) action chunks learn features that predict history state well on in-distribution expert trajectories. However, policies trained with shorter action chunks exhibit a much larger generalization gap to out-of-distribution policy rollouts (7.2 × vs. 2.9 × decrease), indicating greater reliance on spurious correlations. Right: These representational differences translate into significant performance gaps. Shorter action chunks drastically hurt naïve history-conditioned policy performance, while having little effect on oracle policies. Notably, training with action chunk 50 but executing with chunk 10 still significantly outperforms training with chunk 10, confirming that longer chunk prediction improves feature learning for history conditioning.
Figure 4: History state prediction regularization hurts history understanding. We evaluate whether regularizing the history encoder to predict ground-truth state information (number of buttons pressed so far in Fixed-Password ) improves generalization. While this auxiliary task improves accuracy on unseen expert trajectories, it leads to worse performance on out-of-distribution rollouts, indicating it increases reliance on spurious correlations. This regularization also degrades success rate: 55.5%±3.3% to 19.0%±3.3% .
Figure 6: BPP system architecture. We condition a standard diffusion transformer policy architecture on a small set of history keyframes. These keyframes are defined by simple, task-specific criteria and are detected using a VLM. We also mask recent histories to account for detection latency.
Figure 7: Simulation results. BPP outperforms all non-oracle methods, and even surpasses the oracle in one task, suggesting keyframe-based conditioning provides a representation more amenable to learning.
Task
Current Obs
Naïve History
PTP
BPP (Ours)
Drawer Search
11.1%
0.0%
0.0%
33.3%
Marshmallows
40.0%
25.0%
35.0%
65.0%
Mug Replacement
0.0%
5.0%
40.0%
60.0%
Stacking Puzzle
6.5%
21.0%
52.0%
56.0%
Average
14.4%
12.8%
31.8%
53.6%
Table 1: Real-Robot Experiments. BPP achieves nearly 70% higher average success rate than the best prior approach ( PTP ).
Figure 8: Qualitative behavior on DrawerSearch . BPP systematically searches through drawers without revisiting them, demonstrating robust long-horizon progress tracking.
Figure 9: PTP failure on DrawerSearch . PTP fails to track progress after failed grasps or empty drawers, getting stuck in loops.
Figure 10: Data efficiency on the Mug task. With sufficient demonstrations, naïve history conditioning catches up with BPP.
Method
Success Rate (%)
BPP (w/ VLM keyframes)
60%
BPP w/ Oracle keyframes
70%
Naïve History
5%
PTP
40%
Table 2: VLM-based keyframe labeling incurs only a modest performance drop compared to oracle keyframes, demonstrating that BPP is robust to imperfect VLM detection. Even with VLM inference latency, BPP significantly outperforms prior history-conditioning approaches.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Success Rate (%)
Naïve History
55.5±3.3
Naïve History + Frozen Encoder
43.5±2.2
Appendix
Table 3: Frozen encoder ablation on the Fixed Password Entering task. Freezing the image encoder (initialized from a current-observation-only policy) reduces performance.
Figure 11: Policy Architecture. We use a ResNet34 to encode images from 4 cameras, and a transformer decoder to denoise actions.
Method
Training Time (hours)
Current Obs Only
9.95
Naïve History
21.67 ( +118% )
BPP
12.84 ( +29% )
Appendix
Table 4: Training time comparison on DrawerSearch . Naïve history conditioning significantly increases training time, while BPP adds only modest overhead relative to training on current observations only.
Task Name
Obs. Horizon
Stride
# Past Frames
Real-World Tasks
MugReplacement
12 s
3 s
4
Marshmallows
12 s
3 s
4
DrawerSearch
25 s
5 s
5
StackingPuzzle
40 s
8 s
5
Simulation Tasks
Appendix
Table 5: History parameters for Naïve History Conditioning and PTP.
Figure 12: DrawerSearch evaluation behaviors. Our method successfully maintains long-term context to search multiple drawers, while baselines fail to progress through the sequence.
Figure 13: Marshmallow evaluation behaviors. Qualitative comparison of successful and failed executions on the marshmallow task.
Figure 14: MugReplacement evaluation behaviors. Comparison of BPP success versus Naïve History failure.
Vision-Language-Action models provide a strong foundation for general-purpose robot control, yet a vast majority of policies do not preserve and leverage episode-level information beyond the current observation. This limitation is consequential in history-dependent manipulation tasks that depend on information available only in past observations. Retaining past observations in context can aid in recovering this information, but at the significant cost of ever-growing, bloated context and inference latency. We thus introduce MemBodied, a fixed-size episodic memory with two complementary components: an associative state that records interactions across policy calls and an episode anchor that preserves a compact representation of the initial scene as a reference. At each policy call, the model conditions action generation on the current input and the memory components, rather than directly using past observations. Across five evaluated RMBench tasks requiring memory, MemBodied achieves 7.81× the mean success rate of a stateless policy and 2.98× of vanilla recurrent memory, while outperforming the strongest memory-augmented baseline by 1.3× with 10× fewer added parameters. On the fully observable LIBERO-Long suite, it reached 90.6%, a 5.4% improvement over the stateless π0 policy. These findings support MemBodied as a practical alternative to expanding the policy context for history-dependent manipulation.
Tej Deep Pala, Navonil Majumder, Bryce Goh +4
Nanyang Technological University · Griffin Labs · École Centrale de Lyon
Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.
Marcel Torne, Karl Pertsch, Homer Walke +14
Physical Intelligence · Stanford University · UC Berkeley +1
Diffusion-based imitation learning has shown strong promise for robot manipulation. However, most existing policies condition only on the current observation or a short window of recent observations, limiting their ability to resolve history-dependent ambiguities in long-horizon tasks. To address this, we introduce DSSP, a history-conditioned Diffusion State Space Policy that enables efficient, full-history conditioning for robot manipulation. Leveraging the continuous sequence modeling properties of State Space Models (SSMs), our history encoder effectively compresses the entire observation stream into a compact context representation. To ensure this context preserves critical information regarding future state evolution, the encoder is optimized with a dynamics-aware auxiliary training objective. This high-level context representation is then seamlessly fused with recent state observations to form a hierarchical conditioning mechanism for action generation. Furthermore, to maintain architectural consistency and minimize GPU memory overhead, we also instantiate the diffusion backbone itself using an SSM. Extensive experiments across simulation benchmarks and real-world manipulation tasks show that DSSP achieves state-of-the-art performance with a significantly smaller model size, demonstrating superior efficiency of the hierarchical conditioning in capturing crucial information as the history length increases.