Many robot tasks require attending to the history of past observations. For example, finding an item in a room requires remembering which places have already been searched. However, the best-performing robot policies typically condition only on the current observation, limiting their applicability to such tasks. Naively conditioning on past observations often fails due to spurious correlations: policies latch onto incidental features of training histories that do not generalize to out-of-distribution trajectories upon deployment. We analyze why policies latch onto these spurious correlations and find that this problem stems from limited coverage over the space of possible histories during training, which grows exponentially with horizon. Existing regularization techniques provide inconsistent benefits across tasks, as they do not fundamentally address this coverage problem. Motivated by these findings, we propose Big Picture Policies (BPP), an approach that conditions on a minimal set of meaningful keyframes detected by a vision-language model. By projecting diverse rollouts onto a compact set of task-relevant events, BPP substantially reduces distribution shift between training and deployment, without sacrificing expressivity. We evaluate BPP on four challenging real-world manipulation tasks and three simulation tasks, all requiring history conditioning. BPP achieves 70% higher success rates than the best comparison on real-world evaluations. Videos are available at https://bigpicturepolicies.github.io/
Figures & tables
Figure 2: Benchmark tasks. We evaluate on 4 real-world tasks (A–D) and 3 simulation tasks (E–G). All tasks require history conditioning for success.
Figure 3: Predicting longer action chunks significantly decreases spurious correlations in naïve history-conditioned policies. Left: We compare how well the features learned by the policy predict the history state (in Fixed-Password , this is the number of buttons pressed so far). Both policies trained with shorter (10) and longer (50) action chunks learn features that predict history state well on in-distribution expert trajectories. However, policies trained with shorter action chunks exhibit a much larger generalization gap to out-of-distribution policy rollouts (7.2 × vs. 2.9 × decrease), indicating greater reliance on spurious correlations. Right: These representational differences translate into significant performance gaps. Shorter action chunks drastically hurt naïve history-conditioned policy performance, while having little effect on oracle policies. Notably, training with action chunk 50 but executing with chunk 10 still significantly outperforms training with chunk 10, confirming that longer chunk prediction improves feature learning for history conditioning.
Figure 4: History state prediction regularization hurts history understanding. We evaluate whether regularizing the history encoder to predict ground-truth state information (number of buttons pressed so far in Fixed-Password ) improves generalization. While this auxiliary task improves accuracy on unseen expert trajectories, it leads to worse performance on out-of-distribution rollouts, indicating it increases reliance on spurious correlations. This regularization also degrades success rate: 55.5%±3.3% to 19.0%±3.3% .
Figure 6: BPP system architecture. We condition a standard diffusion transformer policy architecture on a small set of history keyframes. These keyframes are defined by simple, task-specific criteria and are detected using a VLM. We also mask recent histories to account for detection latency.
Figure 7: Simulation results. BPP outperforms all non-oracle methods, and even surpasses the oracle in one task, suggesting keyframe-based conditioning provides a representation more amenable to learning.
Task
Current Obs
Naïve History
PTP
BPP (Ours)
Drawer Search
11.1%
0.0%
0.0%
33.3%
Marshmallows
40.0%
25.0%
35.0%
65.0%
Mug Replacement
0.0%
5.0%
40.0%
60.0%
Stacking Puzzle
6.5%
21.0%
52.0%
56.0%
Average
14.4%
12.8%
31.8%
53.6%
Table 1: Real-Robot Experiments. BPP achieves nearly 70% higher average success rate than the best prior approach ( PTP ).
Figure 8: Qualitative behavior on DrawerSearch . BPP systematically searches through drawers without revisiting them, demonstrating robust long-horizon progress tracking.
Figure 9: PTP failure on DrawerSearch . PTP fails to track progress after failed grasps or empty drawers, getting stuck in loops.
Figure 10: Data efficiency on the Mug task. With sufficient demonstrations, naïve history conditioning catches up with BPP.
Method
Success Rate (%)
BPP (w/ VLM keyframes)
60%
BPP w/ Oracle keyframes
70%
Naïve History
5%
PTP
40%
Table 2: VLM-based keyframe labeling incurs only a modest performance drop compared to oracle keyframes, demonstrating that BPP is robust to imperfect VLM detection. Even with VLM inference latency, BPP significantly outperforms prior history-conditioning approaches.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Success Rate (%)
Naïve History
55.5±3.3
Naïve History + Frozen Encoder
43.5±2.2
Appendix
Table 3: Frozen encoder ablation on the Fixed Password Entering task. Freezing the image encoder (initialized from a current-observation-only policy) reduces performance.
Figure 11: Policy Architecture. We use a ResNet34 to encode images from 4 cameras, and a transformer decoder to denoise actions.
Method
Training Time (hours)
Current Obs Only
9.95
Naïve History
21.67 ( +118% )
BPP
12.84 ( +29% )
Appendix
Table 4: Training time comparison on DrawerSearch . Naïve history conditioning significantly increases training time, while BPP adds only modest overhead relative to training on current observations only.
Task Name
Obs. Horizon
Stride
# Past Frames
Real-World Tasks
MugReplacement
12 s
3 s
4
Marshmallows
12 s
3 s
4
DrawerSearch
25 s
5 s
5
StackingPuzzle
40 s
8 s
5
Simulation Tasks
Appendix
Table 5: History parameters for Naïve History Conditioning and PTP.
Figure 12: DrawerSearch evaluation behaviors. Our method successfully maintains long-term context to search multiple drawers, while baselines fail to progress through the sequence.
Figure 13: Marshmallow evaluation behaviors. Qualitative comparison of successful and failed executions on the marshmallow task.
Figure 14: MugReplacement evaluation behaviors. Comparison of BPP success versus Naïve History failure.