Robot manipulation tasks often involve hidden state information that cannot be directly observed and must be inferred through sequential physical interactions. In such partially observable settings, conditioning an imitation learning policy directly on the recent raw observation history leads to poor performance. This is due to state aliasing, wherein identical observations may arise from different hidden states, and the policy receives conflicting action labels for the same input. To enable history-aware disambiguation capability, we propose conditioning a diffusion policy on a structured representation of the hidden states using Bayesian belief that exposes both the current most likely state estimate and the remaining uncertainty. This representation replaces raw history with a structured, compact input, enabling the policy to implicitly modulate between exploratory and exploitative behaviors based on belief uncertainty, without explicit mode switching or reward shaping. We evaluate across two domains with qualitatively different belief representations: a continuous belief for cornstalk gripper alignment via tactile sensing, and a discrete categorical distribution for latched door opening. In both domains, the belief-conditioned policy substantially outperforms the observation-only baseline and approaches privileged ground-truth performance, with ablations illustrating that the policy adapts its exploration behavior depending on the belief uncertainty at inference time.
Figures & tables
Figure 1: System diagram of the belief-conditioned policy. Varying belief guides the low-level policy.
Figure 2: Sequential contact observations reducing uncertainty for gripper alignment.
Eval Metrics
Train Belief
Infer Belief
SR (%) ↑
Ins (%) ↑
Steps ↓
Dist (cm) ↓
Align ( ∘ ) ↓
Init Est.
Init Est. (6D)
20
50
278
4.3
31.3
Belief k=1.0
Belief (27D)
50
70
231
2.0
18.0
Belief k∈[0,1] (ours)
Belief (27D)
80
100
139
1.4
11.9
Belief (transformer)
Belief (128D)
80
90
137
2.3
17.5
Privileged
Privileged (6D)
100
100
73
1.2
4.1
Table 1: Varying latent representations; from raw observations, to belief, to oracle.
Figure 3: Performance comparison of varying policies. Raw observations (exp1) acts as lower bound, while oracle acts as upper bound (exp3). BEACON (exp2) reaches upper bound performance, while taking longer steps due to exploration.
Figure 4: Ablation of exploration and exploitation based on uncertainty present.
Figure 5: Uncertainty reduced using BEACON vs random directional search.
Performance
Decision Making
Model
Features
SR ↑
Texec↓
ΔθH/ΔθL↓
Ngrasp↓
DP (baseline)
P
0.30
341
0.1 / 0.9
7.3
+ tactile, history
P,T,H
0.55
254
0.1 / 0.1
3.5
+ belief (BEACON)
P,T,H,B
0.85
225
0.1 / 0.0
3.3
+ oracle
P,T,H,B,O
0.80
191
0.0 / 0.2
2.3
− belief (oracle)
P,T,H,O
0.80
222
0.0 / 0.1
2.9
Table 2: Door domain: ablation across input features. Performance metrics mirror Domain 1; the decision-making metric captures interaction efficiency.
Figure 6: Belief over rotation direction for both latch and handle. Through observing the rotation of the locked handle and pulling on the door, the belief maintains estimate over true states to guide the policy.
Diffusion-based imitation learning has shown strong promise for robot manipulation. However, most existing policies condition only on the current observation or a short window of recent observations, limiting their ability to resolve history-dependent ambiguities in long-horizon tasks. To address this, we introduce DSSP, a history-conditioned Diffusion State Space Policy that enables efficient, full-history conditioning for robot manipulation. Leveraging the continuous sequence modeling properties of State Space Models (SSMs), our history encoder effectively compresses the entire observation stream into a compact context representation. To ensure this context preserves critical information regarding future state evolution, the encoder is optimized with a dynamics-aware auxiliary training objective. This high-level context representation is then seamlessly fused with recent state observations to form a hierarchical conditioning mechanism for action generation. Furthermore, to maintain architectural consistency and minimize GPU memory overhead, we also instantiate the diffusion backbone itself using an SSM. Extensive experiments across simulation benchmarks and real-world manipulation tasks show that DSSP achieves state-of-the-art performance with a significantly smaller model size, demonstrating superior efficiency of the hierarchical conditioning in capturing crucial information as the history length increases.
Many robot tasks require attending to the history of past observations. For example, finding an item in a room requires remembering which places have already been searched. However, the best-performing robot policies typically condition only on the current observation, limiting their applicability to such tasks. Naively conditioning on past observations often fails due to spurious correlations: policies latch onto incidental features of training histories that do not generalize to out-of-distribution trajectories upon deployment. We analyze why policies latch onto these spurious correlations and find that this problem stems from limited coverage over the space of possible histories during training, which grows exponentially with horizon. Existing regularization techniques provide inconsistent benefits across tasks, as they do not fundamentally address this coverage problem. Motivated by these findings, we propose Big Picture Policies (BPP), an approach that conditions on a minimal set of meaningful keyframes detected by a vision-language model. By projecting diverse rollouts onto a compact set of task-relevant events, BPP substantially reduces distribution shift between training and deployment, without sacrificing expressivity. We evaluate BPP on four challenging real-world manipulation tasks and three simulation tasks, all requiring history conditioning. BPP achieves 70% higher success rates than the best comparison on real-world evaluations. Videos are available at https://bigpicturepolicies.github.io/
Max Sobol Mark, Jacky Liang, Maria Attarian +4
Carnegie Mellon University · Google DeepMind · Princeton University
We propose a stronger formulation of RL on top of RWKV-style recurrent sequence models, in which the fixed-size recurrent state is explicitly interpreted as a belief state rather than an opaque hidden vector. Instead of conditioning policy and value on a single summary h_t, we maintain a compact uncertainty-aware state b_t = (μ_t, Σ_t) derived from RWKV-style recurrent statistics and let control depend on both memory and uncertainty. This design targets a key weakness of plain fixed-state policies in partially observed settings: they may store evidence, but not necessarily confidence. We present the method, a theoretical program, and a pilot RL experiment with hidden episode-level observation noise together with a test-time noise sweep. The pilot shows that belief-state policies nearly match the best recurrent baseline overall while slightly improving return on the hardest in-distribution regime and under a held-out noise shift. Additional ablations show that this simple belief readout is currently stronger than two more structured extensions, namely gated memory control and privileged belief targets, underscoring the need for richer benchmarks.