Beyond Semantic Classes: Online Physical Parameter Estimation via Embedding-Conditioned Gaussian Mixtures
Organizations: Rensselaer Polytechnic Institute, USA
Abstract
Enabling robots to infer interaction-relevant physical properties from vision before contact would improve planning and control in unstructured environments. Vision provides useful pre-contact information, but appearance alone is often ambiguous: visually similar regions can induce different friction, stiffness, or sinkage responses upon contact. Existing approaches often collapse this ambiguity to a point estimate or aggregate experience through fixed semantic classes, limiting their ability to preserve distinct physical regimes. Instead, we propose a real-time estimator that replaces semantic class indexing with an interaction-grounded Gaussian-mixture posterior over latent physical parameters, conditioned directly on continuous visual embeddings. Our approach maintains interaction-derived Gaussian parameter estimates, typically from inverse modeling, as mixture components while using embedding similarity to retrieve and weight relevant past interactions for each new query region. A support-aware fusion mechanism incorporates a conditional generative prior model when interaction history is sparse, while a learned embedding refinement improves retrieval relevance without discretizing the representation. Our experiments show that the estimator recovers multimodal physical beliefs, rejects unrelated interaction history, avoids spatial map discretization, and evaluates at controller rates. In a high-fidelity autonomous racing simulation with unknown low- and high-friction surface regions, the posterior enables the controller to adapt over repeated laps as interaction evidence accumulates, improving performance while maintaining friction-aware behavior.
Figures & tables
| 0.51 / 0.68 | 0.54 / 0.68 | 0.81 / 0.74 | |
| 3.62 / 2.06 | 4.11 / 2.07 | 4.01 / 2.07 | |
| 51.61 / 15.39 | 51.45 / 15.36 | 49.84 / 15.22 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Experiment | Region | Physical interpretation | ||||
|---|---|---|---|---|---|---|
| E1 (bimodal concrete) | grass | moderate friction | 0.86 | 0.09 | 30 | 25 |
| dirt | moderate friction | 0.77 | 0.08 | 30 | 25 | |
| concrete (hi) | dry concrete, high | 0.95 | 0.018 | 30 | 25 | |
| concrete (lo) | ice patch, very low | 0.12 | 0.035 | 30 | 25 | |
| E2 (OOV concrete) | grass | moderate friction | 0.86 | 0.09 | 30 | 25 |
| dirt | moderate friction | 0.77 | 0.08 | 30 | 25 |
| Group | Parameters | Role |
|---|---|---|
| Sideslip residual dynamics | Residual correction to . | |
| Slip recentering | Shift slip centers. | |
| Force gains | Scale tire forces. | |
| Slip scaling/weighting | Tune slip sensitivity. | |
| Damping strength | Scale slip damping. | |
| Tanh sharpness | Set ExpTanh sharpness. |
| Module | Role | Relative rate |
|---|---|---|
| Vision-Estimator | Infer preview tune vectors | Lowest ( ) |
| Trajectory optimizer | Generate surface-aware reference | Intermediate ( ) |
| MPC | Track latest reference in closed loop | Highest ( ) |
Robotic Perception
Robotic perception enables robots to interpret scenes, objects, motion, and contact from cameras and other sensors to guide action.
See all 338 papers