Soft Surfaced Vision-Based Tactile Sensing for Bipedal Robot Applications
Authors: Jaeeun Kim, Junhee Lim, Yu She
Organizations: Department of Industrial Engineering, Purdue University, West Lafayette, IN 47906, USA · Department of Computer Science, Purdue University, West Lafayette, IN 47906, USA
Legged locomotion benefits from embodied sensing, where perception emerges from the physical interaction between body and environment. We present a soft-surfaced, vision-based tactile foot sensor that endows a bipedal robot with a skin-like deformable layer that captures contact deformations optically, turning foot-ground interactions into rich haptic signals. From a contact image stream, our method estimates contact pose (position and orientation), visualizes shear, computes center of pressure (CoP), classifies terrain, and detects geometric features of the contact patch. We validate these capabilities on a tilting platform and in visually obscured conditions, showing that foot-borne tactile feedback improves balance control and terrain awareness beyond proprioception alone. These findings suggest that integrating tactile perception into legged robot feet improves stability, adaptability, and environmental awareness, offering a promising direction toward more compliant and intelligent locomotion systems. For the supplementary video, please visit: https://youtu.be/ceJiy9q_2Aw
Figures & tables
Fig. 1: Foot sensor design. (a) Isometric view of the sensor. (b) Side view with partially removed housing. (c) Exploded view with labeled parts.
Fig. 2: Depth reconstruction from the sensor using a trained model.
Fig. 3: Detecting shear force with dotted grid pad. Shear force is represented with pink and red arrows.
Fig. 4: Detecting position and orientation of the contact region of different geometries. Raw images in the upper row, detected images in the lower row. (a) Corner of a cube. (b) Circular edge of a cylinder. (c) Edge of a prism. (d) Screw head.
Fig. 5: Processed image with OpenCV function. The image processing steps flow from left to right to detect where the pressure is being applied.
Fig. 6: Estimated CoP. Red and green horizontal lines represent the x -mean, while blue lines represent the geometric center of the sensor. The line turns red when it is outside of the safe range, and maintains green when within the safe range. (a) Pressure applied to the back of the sensor. (b) No pressure applied to the sensor. The green line indicating that the CoP is in safe area is coinciding with the blue line. (c) Pressure applied to the front of the sensor.
Fig. 7: Robot balancing based on the estimated CoP at 0.3 rad/s speed to 15° tilted forward. The line turns red when it is outside of the safe range, and maintains green when within the safe range. Initial position depicted in (a) t = 0 s. At (b) t = 3 s, the robot experiences disturbance, and at (c) t = 5 s, the robot reacts to stabilize. Robot adjusted at (d) t = 7 s.
Fig. 8: Summary of the experiment done on an declined (front leaning) plane. The plane was tilted with varying angles of 5°, 10°, and 15°, with different speed of 0.1 rad/s and 0.3 rad/s. The result shows that the robot was able to maintain balance with slower speed and less steep angles. For the case of 0.3 rad/s with 10°, the robot maintained its balance by harness rather than using sensor feedback, thus indicating failure.
Fig. 9: Summary of the experiment done on an inclined (back leaning) plane. The plane was tilted with varying angles of -5°, -10°, and -15°, with different speed of 0.1 rad/s and 0.3 rad/s. The result shows that the robot was able to maintain balance with all cases.
Fig. 10: Terrain classification tested with the robot walking on a field covered by a white fabric. The texts on the top of the sensor images represent the class with highest confidence and the percentage next to it denotes the confidence. Objects in the third column represent the materials on the field. (a) At t = 0 s, tile detected. (b) At t = 30 s, rock detected. (c) At t = 75 s, spike detected.
Fig. 11: Confusion matrix for terrain classification using ResNet-50. Evaluated on the test set ( N = 11881); values are row-normalized percentages (left: true labels, bottom: predictions).
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Institute for Infocomm Research, Agency for Science Technology and Research, Singapore