cs.RONov 6, 2026

Beyond Semantic Classes: Online Physical Parameter Estimation via Embedding-Conditioned Gaussian Mixtures

Authors: Alexander Borca-Tasciuc, Prayag Sharma, Franck Djeumou

Organizations: Rensselaer Polytechnic Institute, USA

Abstract

Enabling robots to infer interaction-relevant physical properties from vision before contact would improve planning and control in unstructured environments. Vision provides useful pre-contact information, but appearance alone is often ambiguous: visually similar regions can induce different friction, stiffness, or sinkage responses upon contact. Existing approaches often collapse this ambiguity to a point estimate or aggregate experience through fixed semantic classes, limiting their ability to preserve distinct physical regimes. Instead, we propose a real-time estimator that replaces semantic class indexing with an interaction-grounded Gaussian-mixture posterior over latent physical parameters, conditioned directly on continuous visual embeddings. Our approach maintains interaction-derived Gaussian parameter estimates, typically from inverse modeling, as mixture components while using embedding similarity to retrieve and weight relevant past interactions for each new query region. A support-aware fusion mechanism incorporates a conditional generative prior model when interaction history is sparse, while a learned embedding refinement improves retrieval relevance without discretizing the representation. Our experiments show that the estimator recovers multimodal physical beliefs, rejects unrelated interaction history, avoids spatial map discretization, and evaluates at controller rates. In a high-fidelity autonomous racing simulation with unknown low- and high-friction surface regions, the posterior enables the controller to adapt over repeated laps as interaction evidence accumulates, improving performance while maintaining friction-aware behavior.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. PhyPush: One Push is All You Need for Sensorless Physical Property Estimation with Physics-Guided Transformers

    May 25, 2026Koyo Fujii, Luis Figueredo, Praminda Caleb-Solly +4Physical ReasoningRobotic Manipulation

  2. PhysVGGT: Feed-Forward Dense Physical Property Estimation from A Single Image

    Sep 16, 2026Sneha Paul, Guile Wu, Bingbing Liu +1Materials Property PredictionRobotic Perception