TACOS: Terrain-Aware COntact Supervision for Visual Footstep Planning

Jean-Michel Fortin; William Larrivée-Hardy; Gabriel Jeanson; Olivier Gamache; Daniele De Martini; François Pomerleau; Philippe Giguère

cs.ROOct 27, 2026

TACOS: Terrain-Aware COntact Supervision for Visual Footstep Planning

Authors: Jean-Michel Fortin, William Larrivée-Hardy, Gabriel Jeanson, Olivier Gamache, Daniele De Martini, François Pomerleau, Philippe Giguère

Organizations: Northern Robotics Laboratory, Université Laval, Canada · Mobile Robotics Group, University of Oxford, United Kingdom

Abstract

Humans navigate their environment based on their vision system, anticipating how the ground will respond before committing to a step. Self-supervised learning could provide legged robots the same ability, but the supervisory signal is typically chosen on intuition, and its effect on what the model learns is rarely examined. We present Terrain-Aware COntact Supervision (TACOS), which proposes a meaningful and consistent footstep instability score, based on the residual of a factor graph in legged state estimation. It projects the continuous score into preceding images to supervise a dense visual predictor. We propose two variants, a hard score that exploits hindsight and a causal soft score available immediately. Both metrics are compared against state-of-the-art alternatives in two independent evaluations that share no data or labels. On 1716 manually annotated contacts, both variants outperform every evaluated alternative at detecting instability and ranking its severity, improving pooled severity ranking by 28 % over the strongest baseline. On three outdoor test runs, our metric also trains the most accurate visual predictor, improving rank correlation by 58 % over the strongest alternative, and by a further 25 % when supplemented with onboard context. In a real-world experiment over a 50 m forest trajectory, the trained predictor handles non-geometric terrain properties such as mud and vegetation, while stepping around geometric obstacles like fallen branches. Code is available for reproducibility at https://github.com/norlab-ulaval/TACOS.

Figures & tables

Fig. 1 : Quadruped robot traversing unstructured forest terrain, where roots, branches, and leaves create candidate footholds whose supportiveness cannot be inferred from geometry alone. Close-ups show, on the right, the GoPro view of the foot getting stuck, wedged among exposed roots, and on the left, the resulting robot fall.

Fig. 2 : Offline label-generation pipeline. (Left) Stances are extracted from the onboard contact detector, where F and H denote front and hind, and L and R left and right. (Center) Contact residuals $d_{i,k}$ from a factor graph leg odometry are aggregated over each stance into the instability score $m_{i,\mathrm{hard}}$ . (Right) Footstep placements are computed in the world frame using exteroceptive odometry and FK ( FK ), then projected into an image acquired before contact and rasterized into a sparse label mask.

Fig. 3 : Visual instability predictor. A frozen DINOv3-B encoder feeds a trainable DPT decoder that outputs a dense instability map. Camera roll and pitch ( $\phi,\theta$ ), and recent instability history ( $\bar{m}_{f},\bar{m}_{h}$ ), enter as context tokens through GCA at each decoder stage. The loss is applied only at projected footstep locations.

Fig. 4 : Precision-Recall curves for unstable-foothold classification on the human-labeled dataset, with each metric’s AP ( AP ) in the legend. Our hard and soft residuals are compared against state-of-the-art alternatives. The dashed line marks the random baseline, set by the proportion of unstable footholds in the dataset.

Fig. 5 : Distribution of the proposed hard TACOS metric for stable footholds grouped by annotated instability severity. The insets show representative events recorded by the feet-facing cameras for each severity. Stable footholds concentrate at low values, and the distributions shift upward at each severity level.

Method	Logs	Ice	Slope	Pooled
ProNav [ 7 ]	0.341	0.323	0.312	0.342
GRS [ 29 ]	0.479	0.339	0.245	0.333
OCELOT [ 12 ]	0.313	0.459	0.440	0.400
[ 31 ]	0.693	0.776	0.773	0.700
[ 32 ]	0.797	0.582	0.777	0.653
[ 17 ]	0.749	0.526	0.700	0.566

TABLE I: VUS ( VUS ) per method, per test run and pooled over all runs. Higher is better; best per column in bold , second best underlined .

Fig. 6 : Network predictions on two unseen outdoor scenes for three supervisory signals, with the input image at the top. WVN is supervised at body level, whereas GRS and ours share the same per-footstep labels, while their metrics differ. Darker colors indicate higher predicted cost, each map being normalized between the 5th and 95th percentiles of its own training labels.

Fig. 7 : Footstep-level Spearman correlation between predicted and true cost labels, per label source, over five-fold cross-validation.

Fig. 8 : Real-world experiment in forest terrain. ( Left ), the geometric costmap ( Center ), and the TACOS costmap ( Right ). Corresponding locations mark \raisebox{-.3pt}{\scriptsizeA}⃝ , terrain associated with foot sinkage; \raisebox{-.3pt}{\scriptsizeB}⃝ , a vegetation barrier; and \raisebox{-.3pt}{\scriptsizeC}⃝ , fallen branches. The smaller photographs illustrate the robot’s interaction with these obstacles. Red and white curves show trajectories. The red cross marks where the geometric costmap incorrectly treats the traversable vegetation as an impassable barrier.

$\phi/\theta$	$\bar{m}_{f,h}$	SBB-2	KAB-2	ALB-3	Pooled
–	–	0.510 $\pm$ .023	0.501 $\pm$ .020	0.373 $\pm$ .026	0.557 $\pm$ .007
$\checkmark$	–	0.509 $\pm$ .027	0.522 $\pm$ .029	0.377 $\pm$ .041	0.580 $\pm$ .015
–	$\checkmark$	0.584 $\pm$ .018	0.633 $\pm$ .016	0.612 $\pm$ .018	0.694 $\pm$ .007
$\checkmark$	$\checkmark$	0.589 $\pm$ .022	0.635 $\pm$ .009	0.613 $\pm$ .017	0.694 $\pm$ .008

TABLE II: Ablation of context conditioning with GCA . All configurations use images, and check marks indicate the added roll/pitch (

\phi,\theta

) and instability history (

\bar{m}_{f,h}

). Spearman correlation between predictions and hard residual labels is reported per test run and pooled, as mean

\pm

standard deviation over 5 folds. Higher is better, with best values in bold and second best underlined .

Terrain-Aware Robot Locomotion

Terrain-aware robot locomotion studies how robots sense, predict, and adapt movement to uneven, deformable, or otherwise challenging ground.

See all 149 papers

Sep 24, 2026

TactileStep: Sole Tactile Learning for Regulating Foot-Terrain Interaction in Humanoid Locomotion

Humanoid parkour policies can traverse various terrains, but task completion may mask challenges of harsh landings, edge contacts, and unstable stance contacts. Humans naturally regulate foot-terrain interaction through tactile feedback, modulating contact compliance according to terrain stiffness. This highlights a key domain gap between humans and humanoid robots: the absence of rich tactile sensing in most humanoid systems. We address this problem with TactileStep, a deployable tactile learning framework that brings sole pressure sensing into humanoid locomotion control for softer touchdowns and more stable support. TactileStep aligns tactile simulation with the real pressure insole, allowing the policy to learn from the same contact features available on hardware. During training, we use tactile and motion cues to recognize different foot-contact phases and apply phase-aware rewards that encourage safer landing and more stable stance. Evaluated in simulation and on a Unitree G1 humanoid across diverse terrains, TactileStep reduces peak touchdown force by up to 48.8% and peak A-weighted impact noise by up to 30.1 dB over a strong perceptive baseline, while increasing stance contact area by up to 23.8%.

Zizhuo Wang, Ming-ju Lee, Shaoting Zhu +3

Tsinghua University

Humanoid Robot Locomotion Terrain-Aware Robot Locomotion Robot Locomotion RL for Legged Locomotion Robotic Tactile Sensing

Ask PDF

Feb 20, 2026

Soft Surfaced Vision-Based Tactile Sensing for Bipedal Robot Applications

Legged locomotion benefits from embodied sensing, where perception emerges from the physical interaction between body and environment. We present a soft-surfaced, vision-based tactile foot sensor that endows a bipedal robot with a skin-like deformable layer that captures contact deformations optically, turning foot-ground interactions into rich haptic signals. From a contact image stream, our method estimates contact pose (position and orientation), visualizes shear, computes center of pressure (CoP), classifies terrain, and detects geometric features of the contact patch. We validate these capabilities on a tilting platform and in visually obscured conditions, showing that foot-borne tactile feedback improves balance control and terrain awareness beyond proprioception alone. These findings suggest that integrating tactile perception into legged robot feet improves stability, adaptability, and environmental awareness, offering a promising direction toward more compliant and intelligent locomotion systems. For the supplementary video, please visit: https://youtu.be/ceJiy9q_2Aw

Jaeeun Kim, Junhee Lim, Yu She

Department of Industrial Engineering, Purdue University, West Lafayette, IN 47906, USA · Department of Computer Science, Purdue University, West Lafayette, IN 47906, USA

Robotic Tactile Sensing Vision-Based Tactile Sensing Legged Robotics Robot Locomotion Terrain-Aware Robot Locomotion

Ask PDF

Jun 3, 2026

Learning Contact Representation for Leg Odometry

Leg odometry relies on the assumption that the velocity of a foot in stance is zero with respect to the world, which makes accurate contact detection a critical subproblem. Many approaches classify contact using ground reaction force (GRF) sensors at the foot tip, yet these sensors are not available on all legged robots and remain unresponsive to disturbances such as slippage while the foot is still in contact. Approaches without GRF sensing instead rely on heuristic pseudo-labels, model-based foot-velocity assumptions, or supervised contact learning. In this study, we propose a self-supervised representation learning framework for contact detection that uses only standard proprioceptive sensing (joint encoders and IMU) and requires neither force sensors nor contact labels. A denoising autoencoder is trained on kinematic features, a two-component Gaussian Mixture Model is fitted in the learned latent space, and its posterior is used as a continuous stance belief that scales the zero-velocity measurement covariance of an error-state Kalman filter. Experiments in simulation and on real-world terrain show that our framework yields the lowest odometry error among supervised and probabilistic baselines, and that higher pseudo-label classification accuracy does not translate into better downstream odometry. Our code and dataset are publicly available.

Emre Girgin, Cagri Kilic

Department of Aerospace Engineering, Embry Riddle Aeronautical University, United States

Legged Robotics Self-Supervised Learning Robot State Estimation

Ask PDF

Abstract

Terrain-Aware Robot Locomotion

Explore similar work

TactileStep: Sole Tactile Learning for Regulating Foot-Terrain Interaction in Humanoid Locomotion

Soft Surfaced Vision-Based Tactile Sensing for Bipedal Robot Applications

Learning Contact Representation for Leg Odometry