Robust Task-Aware State Estimation from Pre-trained Perception
Organizations: CLAIR Lab, Taub Faculty of Computer Science, Technion – Israel Institute of Technology, Haifa 3200003, Israel
Abstract
Agents that perform state estimation using pretrained perception models, e.g., Vision-Language Models (VLMs), face the fundamental challenge that predictions are uncertain. Still, most approaches determinize these predictions into a single best-guess state and make decisions as if it were ground truth. This is brittle because when the guess is wrong, the agent's decisions may be unsafe or infeasible. At the other extreme, maintaining a full probability distribution over symbolic states is often intractable. We propose a middle path, drawing on the underlying probabilities the perception model already produces (e.g., a VLM's output-layer token distributions) to construct a -confidence Minimal Belief Set (MBS), a minimal subset of states whose cumulative probability meets a user-specified belief-mass threshold. The minimality of an MBS ensures that the agent considers only the most likely states, enhancing robustness in downstream decision-making while maintaining computational tractability. Using a novel three-layered belief system, we efficiently approximate individual state probabilities from per-feature predictions and task-level constraints. On this belief construction, we base our MBS construction, while maintaining only a lean factored representation of the belief. Our algorithm, Probability-Ranked Valid State Search (PRaVaSS), takes advantage of the unique structure of this belief system to construct an MBS with correctness guarantees and polynomial complexity in the unconstrained case. Using a VLM as the backbone of our state estimator, we validate our method's theoretical properties, scalability with problem size, and improvement of downstream decision-making when used in a task-planning pipeline.
Figures & tables
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Domain | ( constraints ) | ( selected_groups ) | Source |
|---|---|---|---|
| GW | FD all mutex groups: object-placement and gripper-capacity | FD selected SAS + groups | FD on synthetic seed |
| PRB | FD all mutex groups: per-shape support and per-block occupancy/clearance | FD selected SAS + groups | FD on synthetic seed |
| ALF | Global holding mutex plus pairwise held-vs-contained exclusions | Global holding mutex only (when ) | Custom lifted constraints |
| PRB | GW | ALF | |||||
|---|---|---|---|---|---|---|---|
| Backbone | Method | Cov. [95% CI] | Size | Cov. [95% CI] | Size | Cov. [95% CI] | Size |
| GPT-4.1 | MBS (Ours) | 0.93 [0.92, 0.95] | 1.35 | 0.81 [0.78, 0.84] | 21.9 | 0.84 [0.83, 0.86] | 56.1 |
| Valid MAP | 0.79 [0.77, 0.81] | 1 | 0.34 [0.30, 0.38] | 1 | 0.51 [0.48, 0.53] | 1 | |
| MAP | 0.79 [0.76, 0.81] | 1 | 0.29 [0.26, 0.33] | 1 | 0.52 [0.50, 0.55] | 1 | |
| Feature Abstention | 0.86 [0.84, 0.88] | 1 | 1.00 [1.00, 1.00] | 0.80 [0.78, 0.82] | 17 | ||
| Qwen3-VL-32B | MBS (Ours) | 0.65 [0.61, 0.68] | 226 | 0.65 [0.61, 0.69] | 0.84 [0.83, 0.86] | 72.1 | |
| PRB | GW | ||||
|---|---|---|---|---|---|
| Backbone | Method | Success [95% CI] | TO | Success [95% CI] | TO |
| GPT-4.1 | Oracle | 100.0% [99.7, 100.0] | 0% | 95.0% [93.0, 96.5] | 0% |
| MAP | 16.7% [14.7, 18.9] | 0% | 71.7% [67.9, 75.1] | 0% | |
| Adaptive Abst. | 16.8% [14.7, 19.0] | 0% | 74.0% [70.3, 77.3] | 0% | |
| AMS -Loose | 16.2% [14.2, 18.4] | 0% | 74.7% [71.0, 78.0] | 1% | |
| AMS | 32.4% [29.8, 35.1] | 0% | 79.2% [75.7, 82.2] | 2% | |
| Domain | Total variation | Mass gap (pp) | Reaches | Size ratio | ||
|---|---|---|---|---|---|---|
| PRB | 0.00 | 135 | 0.000 | [ , ] | 100.0% | 1.00 |
| 0.10 | 135 | 0.146 | [ , ] | 87.4% | 1.43 | |
| 0.20 | 135 | 0.263 | [ , ] | 67.4% | 1.67 | |
| 0.30 | 135 | 0.352 | [ , ] | 75.6% | 2.00 | |
| 0.50 | 135 | 0.450 | [ , ] | 85.2% | 2.00 | |
| GW | 0.00 | 182 | 0.000 | [ , ] | 100.0% | 1.00 |
| Domain | MBS (product belief) | MBS (true joint) | Valid MAP | MAP | ||
|---|---|---|---|---|---|---|
| PRB | 0.00 | 135 | 89.6% (2) | 89.6% (2) | 67.4% (1) | 66.7% (1) |
| 0.10 | 135 | 95.6% (4) | 91.1% (3) | 67.4% (1) | 66.7% (1) | |
| 0.20 | 135 | 97.0% (5) | 91.1% (3) | 67.4% (1) | 66.7% (1) | |
| 0.30 | 135 | 97.8% (6) | 91.1% (3) | 67.4% (1) | 66.7% (1) | |
| 0.50 | 135 | 94.8% (8) | 91.9% (3) | 32.6% (1) | 32.6% (1) | |
| GW | 0.00 | 182 | 84.1% (22) | 84.1% (22) | 25.8% (1) | 19.2% (1) |
| Mutex-coupled block | Least-decided block | ||||
|---|---|---|---|---|---|
| Domain | Mass gap (pp) | Reaches | Mass gap (pp) | Reaches | |
| PRB | 0.10 | [ , ] | 87% | [ , ] | 74% |
| 0.20 | [ , ] | 67% | [ , ] | 55% | |
| 0.30 | [ , ] | 76% | [ , ] | 65% | |
| 0.50 | [ , ] | 85% | [ , ] | 93% | |
| GW | 0.10 | [ , ] | 100% | [ , ] | 90% |
| Solved instances | Max search features solved | Median runtime (s) | Median states visited | |
| 0.50 | 3,789 (99.71%) | 61 | 16.00 | |
| 0.70 | 3,779 (99.45%) | 48 | 36.00 | |
| 0.80 | 3,774 (99.32%) | 47 | 45.00 | |
| Method | PRB | GW |
|---|---|---|
| MAP | 16.7% | 70.7% |
| [14.4%, 18.8%] | [67.2%, 73.8%] | |
| Adaptive Abst. | 16.8% | 73.3% |
| [14.7%, 19.1%] | [69.8%, 76.8%] | |
| AMS -Loose | 16.2% | 73.2% |
| [14.1%, 18.0%] | [69.5%, 76.3%] |
| PRB | GW | |||
|---|---|---|---|---|
| Method | Success [95% CI] | TO | Success [95% CI] | TO |
| MAP | 16.7% [14.7, 18.9] | 0% | 71.7% [67.9, 75.1] | 0% |
| Adaptive Abst. | 16.8% [14.7, 19.0] | 0% | 74.0% [70.3, 77.3] | 0% |
| AMS -Loose | 16.2% [14.2, 18.4] | 0% | 74.7% [71.0, 78.0] | 1% |
| AMS (strict, relaxed ) | 32.4% [29.8, 35.1] | 0% | 79.2% [75.7, 82.2] | 2% |
| AMS (strict, exact ) | 32.3% [29.7, 35.0] | 0% | 79.8% [76.4, 82.8] | 0% |
| Domain | Contrast | Difference | 95% CI |
|---|---|---|---|
| PRB | AMS (strict, relaxed ) MAP | 15.8 | [ 13.7, 17.9] |
| AMS (strict, relaxed ) Adaptive Abst. | 15.7 | [ 13.6, 17.8] | |
| AMS (strict, relaxed ) AMS -Loose | 16.2 | [ 14.2, 18.4] | |
| AMS (strict, exact ) AMS (strict, relaxed ) | 0.1 | [ 0.2, 0.0] | |
| GW | AMS (strict, relaxed ) MAP | 7.5 | [ 5.3, 9.8] |
| AMS (strict, relaxed ) Adaptive Abst. | 5.2 | [ 3.2, 7.3] |
| Method | Before observation | After observation | Change |
|---|---|---|---|
| MAP | 83/115 (72.2%) | 92/115 (80.0%) | 7.8 |
| Adaptive Abst. | 104/115 (90.4%) | 107/115 (93.0%) | 2.6 |
| AMS | 107/115 (93.0%) | 111/115 (96.5%) | 3.5 |
| Paired contrast | Before observation | After observation | |
| AMS MAP | [ , ] | [ , ] | |
| Adaptive Abst. MAP | [ , ] | [ , ] |
| Round | Method | Belief-set size | Probability mass | Search time | Attempts |
|---|---|---|---|---|---|
| Before observation | AMS | 14.0 (20) | 46.7% (45.5%) | 154.8 s (186.4 s) | 6.4 |
| Adaptive Abst. | 9.5 (12) | 34.6% (25.0%) | 70.5 s (73.2 s) | 3.6 | |
| After observation | AMS | 14.7 (20) | 75.6% (100.0%) | 152.0 s (174.6 s) | 6.5 |
| Adaptive Abst. | 10.0 (12) | 66.6% (100.0%) | 70.1 s (72.9 s) | 3.7 |
Robust Estimation
Robust estimation infers reliable parameters and states despite outliers, heavy-tailed noise, data contamination, or model uncertainty.
See all 95 papers