A Minimal Optical-Flow Representation for Vision-Based Tactile Rotation Classification in Robotic Manipulation Across Gravity Domains

Oscar Martinez-Bernal; Mario Cavero-Vidal; Francesco Grella; Carol Martinez

cs.ROOct 8, 2026

A Minimal Optical-Flow Representation for Vision-Based Tactile Rotation Classification in Robotic Manipulation Across Gravity Domains

Authors: Oscar Martinez-Bernal, Mario Cavero-Vidal, Francesco Grella, Carol Martinez

Organizations: Space Robotics Research Group (SpaceR), Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg · Eurecat, Centre Tecnològic de Catalunya, Robotics and Automation Unit, Barcelona, Spain

Abstract

Vision-based tactile sensors provide rich contact information, but processing high-resolution images can be costly for resource-constrained platforms such as space robots. This work investigates whether a compact representation of tactile motion can classify object rotation across different gravity conditions. Dense optical flow from a simulated GelSight Mini is aggregated over a 7x9 grid into 126 features and used to classify the direction of load-induced rotation under Earth, Mars, Moon, and orbital gravity. Gravity causes a small but significant shift in these features, accounting for 1.6% of their variance (R2 = 0.016). Despite its small magnitude, this shift affects models trained only on Earth data: XGBoost accuracy decreases from 94.4% on Earth to 75.9% in orbit. In contrast, a single model trained across all four gravity domains achieves 96.3% overall accuracy and 95.1%-97.0% across individual domains, without using gravity as an input. The representation can also be reduced to 40 features while retaining 95.7% accuracy, with XGBoost requiring only 0.14 ms per inference. These findings show that Earth-gravity performance alone is insufficient to establish the transferability of tactile perception for space robotic manipulation, highlighting the need to account for gravity-induced domain shifts during training and validation.

Figures & tables

Fig. 1 : Multi-gravity scenarios considered in this work and overview of the investigated transfer problem.

Fig. 2 : Processing pipeline from simulated tactile acquisition to rotation-direction classification. Consecutive tactile frames are converted to dense optical flow, spatially aggregated over a fixed grid, and used to classify load-induced rotation as clockwise (cw) or counterclockwise (ccw).

Load	Left sensor	Right sensor
Downward $F_{z}$	ccw	cw
Upward $F_{z}$	cw	ccw

TABLE I : Rotation-direction label as a function of load direction and sensor side.

Fig. 3 : The Space Robotics Bench simulation environment. (a) Gripper with integrated GelSight Mini in a Martian environment. (b) Applied force.

Regime	Set	Events	Flow fields
All	Train (four gravities)	1893	37199
All	Test set (four gravities)	473	9403
Earth	Train (Earth)	468	8935
Earth	Test, Earth	118	2469
Earth	Test, Mars	120	2336
Earth	Test, Moon	114	2242

TABLE II : Partitions of the two training regimes: number of events and of flow fields in each set.

Fig. 4 : Gravity sensitivity per cell (Kruskal–Wallis effect size

\epsilon^{2}

) for the

dx

and

dy

components, in the world frame (columns along the profile, rows along

F_{z}

Metric	XGB				RF				KNN				LOGREG
	Earth	Mars	Moon	Orbit	Earth	Mars	Moon	Orbit	Earth	Mars	Moon	Orbit	Earth	Mars	Moon	Orbit
Accuracy	0.944	0.858	0.817	0.759	0.938	0.820	0.751	0.655	0.934	0.719	0.637	0.591	0.857	0.773	0.747	0.688
Balanced acc.	0.944	0.858	0.817	0.760	0.939	0.822	0.759	0.665	0.933	0.720	0.644	0.596	0.856	0.772	0.746	0.688
Macro-F1	0.944	0.858	0.816	0.758	0.938	0.819	0.748	0.648	0.934	0.718	0.634	0.590	0.856	0.772	0.746	0.687
ROC-AUC	0.991	0.932	0.898	0.842	0.989	0.929	0.877	0.782	0.964	0.773	0.693	0.630	0.958	0.860	0.843	0.786
PR-AUC	0.991	0.939	0.897	0.843	0.989	0.933	0.877	0.769	0.949	0.719	0.622	0.580	0.958	0.870	0.838	0.785

TABLE III : Earth-only training: metrics per flow field on the Earth test events and on the three unseen gravity domains, per model, with the accuracy drop of each domain relative to Earth.

Fig. 5 : Accuracy per flow field and gravity domain for the four classifiers (XGBoost highlighted). (a) Earth-only training, where Mars, Moon, and orbit are unseen domains. (b) All-gravity training.

Model	Accuracy	Bal. acc.	Macro-F1	ROC-AUC	PR-AUC
XGB	0.963	0.963	0.963	0.996	0.996
RF	0.955	0.955	0.955	0.994	0.994
KNN	0.949	0.948	0.948	0.976	0.965
LOGREG	0.857	0.855	0.856	0.952	0.953

TABLE IV : All-gravity training: metrics per flow field of the four classifiers on the test set.

Metric	By gravity				By load magnitude			By sensor side		By rotation magnitude
	Earth	Mars	Moon	Orbit	3.0 N	4.5 N	6.0 N	Left	Right	$<5^{\circ}$	5 to $15^{\circ}$	$>15^{\circ}$
Accuracy	0.970	0.951	0.968	0.966	0.968	0.955	0.975	0.967	0.959	0.960	0.960	0.974
Balanced acc.	0.970	0.952	0.968	0.966	0.968	0.955	0.975	0.957	0.953	0.962	0.968	0.974
Macro-F1	0.970	0.951	0.968	0.966	0.967	0.955	0.975	0.962	0.956	0.958	0.940	0.974
ROC-AUC	0.997	0.994	0.997	0.996	0.997	0.994	0.998	0.995	0.995	0.995	0.996	0.998
PR-AUC	0.997	0.993	0.996	0.996	0.996	0.994	0.998	0.991	0.997	0.997	0.984	0.998

TABLE V: All-gravity training, XGBoost breakdowns by the factors excluded from the input: metrics per flow field on the test set.

Representation	# features	Accuracy	Balanced acc.	Macro-F1	ROC-AUC	PR-AUC
$dx$ only	63	0.944	0.944	0.944	0.991	0.991
$dy$ only	63	0.947	0.947	0.947	0.991	0.991
$dx+dy$ (full)	126	0.963	0.963	0.963	0.996	0.996
Grid 1 $\times$ 1	2	0.670	0.669	0.668	0.727	0.729
Grid 2 $\times$ 3	12	0.925	0.925	0.925	0.982	0.982
Grid 4 $\times$ 5	40	0.957	0.957	0.957	0.994	0.994

TABLE VI: Minimal representation (XGBoost, fixed hyperparameters): metrics per flow field on the test set against the number of features.

Fig. 6 : Minimal representation: Accuracy versus feature count for grid resolution (a) and PCA (b).

Model	Training (s)	Inference (ms)	Model size (MB)
XGB	2.07	0.14	0.4
RF	144	33.62	58.9
KNN	0.03	2.19	19.1
LOGREG	0.21	0.11	0.005

TABLE VII: Classifier cost on one CPU thread: training time (median of 5 fits), inference time per flow field (median of 200 runs), and model size.

Explore similar work

TacVerse: A Multi-Sensor Dataset and Benchmark for Cross-Sensor Vision-Based Tactile Perception
Jun 24, 2026Lan Wei, Gurmeher Khurana, Sirine Bhouri +6Vision-Based Tactile Sensing Few-Shot Learning
Imperial College London, UK. · Queen Mary University of London, UK · King’s College London, UK
TactX: Learning Shared Tactile Representations Across Diverse Sensors
Jun 30, 2026Junsung Park, Sachin Bhadang, Carmelo Sferrazza +2Cross-Modal Representation Learning Contact-Rich Robotic Manipulation
UC San Diego · Seoul National University · Amazon FAR
FTP-1: A Generalist Foundation Tactile Policy Across Tactile Sensors for Contact-Rich Manipulation
Jun 11, 2026Chengbo Yuan, Zicheng Zhang, Mingjie Zhou +14Robot Skill Learning Contact-Rich Robotic Manipulation
1Tsinghua University · 2Shanghai Qi Zhi Institute · 3Sharpa +5

Jun 24, 2026, cs.ROJ/K move · Enter open · S save

TacVerse: A Multi-Sensor Dataset and Benchmark for Cross-Sensor Vision-Based Tactile Perception

Lan Wei, Gurmeher Khurana, Sirine Bhouri, Wenhao Hong+5

Imperial College London, UK. · Queen Mary University of London, UK · King’s College London, UK

Vision-Based Tactile Sensing Few-Shot Learning Robotic Tactile Sensing Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Vision-based tactile sensors (VBTSs) enable robots to infer contact geometry and force-related cues by imaging deformation through an internal camera, yet generalisation across sensor designs remains poorly understood. We present TacVerse, a multi-sensor dataset and benchmark for cross-sensor vision-based tactile perception. The dataset contains 106,800 tactile images from seven VBTSs and supports three downstream tasks: shape classification, grating classification, and force regression. Experiments are conducted under three settings: within-sensor training, zero-shot cross-sensor transfer, and few-shot adaptation. Strong within-sensor performance across all tasks indicates that the collected tactile observations are informative for the target objectives. Direct cross-sensor transfer, however, leads to substantial degradation. Shape classification is comparatively robust, whereas grating classification and force regression are more sensitive to sensor shift. Few-shot adaptation for force regression consistently improves performance on unseen target sensors but does not fully close the gap to within-sensor upper bounds. A representation study further shows that MAE (Masked Autoencoder) pretraining provides the most consistent gains across tasks and sensors. TacVerse provides a controlled testbed for studying sensor shift, data-efficient adaptation, and self-supervised learning in tactile perception.

Abstract

Explore similar work

TacVerse: A Multi-Sensor Dataset and Benchmark for Cross-Sensor Vision-Based Tactile Perception

TactX: Learning Shared Tactile Representations Across Diverse Sensors

FTP-1: A Generalist Foundation Tactile Policy Across Tactile Sensors for Contact-Rich Manipulation