cs.CVMar 16, 2026
SaveFEEL (Force-Enhanced Egocentric Learning): A Dataset for Physical Action Understanding
Organizations: University of Maryland, College Park, USA
Abstract
We introduce FEEL (Force-Enhanced Egocentric Learning), the first large-scale dataset pairing force measurements gathered from custom piezoresistive gloves with egocentric video. Our gloves enable scalable data collection, and FEEL contains approximately 3 million force-synchronized frames of natural unscripted manipulation in kitchen environments, with 45% of frames involving hand-object contact. Because force is the underlying cause that drives physical interaction, it is a critical primitive for physical action understanding. We demonstrate the utility of force for physical action understanding through application of FEEL to two families of tasks: (1) contact understanding, where we jointly perform temporal contact segmentation and pixel-level contacted object segmentation; and, (2) action representation learning, where force prediction serves as a self-supervised pretraining objective for video backbones. We achieve state-of-the-art temporal contact segmentation results and competitive pixel-level segmentation results without any need for manual contacted object segmentation annotations. Furthermore we demonstrate that action representation learning with FEEL improves transfer performance on action understanding tasks without any manual labels over EPIC-Kitchens, SomethingSomething-V2, EgoExo4D and Meccano.
Figures & tables
Figure 1: FEEL pairs egocentric video with force measurements to capture the physical causes—not just visual effects—of hand-object interaction.
Figure 2: Forces reveal interaction dynamics. Representative sequences showing how force measurements disambiguate physical interactions. Left: Grasping and lifting a pan exhibits distinct force signatures with an onset at the end of the reach and a spike as the pan is lifted. Right: Stirring produces rhythmic force patterns reflecting cyclic manipulation.
Figure 3: Contact detection from force measurements. Despite visual similarity across frames, force measurements precisely identify contact boundaries. Top: Raw sensor forces (pinky and middle finger dominate this grasp). Middle: Consolidated force with dual thresholds—above C threshold indicates contact, below NC threshold indicates non-contact, between is ambiguous (excluded). Forces enable scalable contact supervision without manual annotation of ambiguous visual transitions.
Figure 4: Force-sensing glove hardware. Leftmost: Palm-facing views of the left and right gloves, showing the six piezoresistive sensors ( red ) mounted at each fingertip and across the palm, the Arduino microcontroller case ( blue ) mounted at the wrist, and the on/off switch ( green ). Middle and right: Side and in-use views demonstrating the glove’s low-profile form factor during natural object manipulation.
Figure 5: Learning contacts and action from forces. (a) Contact Understanding Image network trained with force-derived contact labels performs contact detection and segmentation (red mask) for each hand (L/R). (b) Action Representation Learning Video network predicts per-hand forces from clips for pretraining. The video backbone is transferred to action recognition after discarding force heads. No manual contact or action labels required during pre-training.
| Dataset | View | Scale | Real | Modalities | Contact | Forces |
|---|---|---|---|---|---|---|
| EgoHands [ 22 ] | Ego | 1.5 hrs | ✓ | V | ||
| Arctic [ 16 ] | Both | 4 hrs | ✓ | V/D | ✓ | |
| VISOR [ 10 ] | Ego | 36 hrs | ✓ | V/A | ✓ | |
| HOI4D [ 24 ] | Ego | 45 hrs | ✓ | V/D | ✓ | |
| Ego4D (FHO) [ 18 ] | Ego | 120 hrs | ✓ | V/A | 1/2 | |
| 100DOH [ 32 ] | Both | - | ✓ | V | ✓ |
Table 1: Comparison of commonly used hand-object interaction datasets. Modality codes: V=video, A=audio, D=depth, F=force, E=eye tracking. Contact annotation for Ego4D (FHO) is partial (only a single critical frame is annotated out of each sequence). FEEL is the only real-world (Real) egocentric dataset providing both contact annotations and force measurements — all other datasets with force lack contact labels or real-world capture, and all datasets with contact lack force.
| Method | Sup. | Training Set # | HOI4D | VISOR | FEEL | |||
|---|---|---|---|---|---|---|---|---|
| BC | BSC | BC | BSC | BC | BSC | |||
| Depth [ 3 ] | Self | 5 | 66.6 | 66.0 | 52.8 | 39.9 | 48.6 | 30.0 |
| HOIRef [ 2 ] | Full | 2 (inc. VISOR) | 52.1 | 50.8 | 55.4 | 53.7 | 59.6 | 57.2 |
| VISOR [ 10 ] | Full | 1 (VISOR) | 55.3 | 55.0 | 66.3 | 65.2 | 45.2 | 40.2 |
| 100DOH [ 32 ] | Full | 2 (inc. VISOR) | 90.0 | 80.5 | 81.3 | 71.9 | 49.0 | 40.0 |
| Ours | Self | 1 (inc. FEEL) | 82.2 | 81.3 | 87.3 | 79.7 | 95.6 | 88.9 |
Table 2: Temporal contact detection results evaluated on the validation sets of FEEL, EPIC-VISOR, and HOI4D. We emphasize that while our model was only trained over dataset (FEEL), the 100DOH model and VISOR model were both trained over the EPIC-VISOR dataset [ 10 ] . BC represents "Binary contact detection" - a prediction is correct iff it captures the binary contact state between the hand and the object, irrespective of which predicted hand side is involved. BSC represents "Binary Signed Contact" - a prediction is correct iff it captures both the binary contact state of between the hand and the object, as well as the hand side involved.
| Method | Sup. | Training Set # | Dataset | ||
|---|---|---|---|---|---|
| HOI4D | VISOR | FEEL | |||
| MOVES [ 19 ] | Weak | N/A | - | - | 0.309 |
| SAM3 [ 5 ] | Full | Comp. | 0.201 | 0.085 | 0.030 |
| VISOR [ 10 ] | Full | 1 (inc. VISOR) | 0.502 | 0.402 | 0.111 |
| 100DOH [ 32 ] +SAM [ 5 ] | Full | Comp. (inc. VISOR) | 0.765 | 0.443 | 0.218 |
| Ours | Weak | 1 (FEEL) | 0.751 | 0.399 | 0.625 |
Table 3: Spatial contact segmentation results evaluated on the validation sets of FEEL, EPIC-VISOR, and HOI4D. We emphasize that while our model was only trained over dataset (FEEL), the 100DOH model and VISOR model were both trained over the EPIC-VISOR dataset. As the 100DOH model produces bounding boxes (not segmentations), we employ in addition a SAM model (trained over Internet-scale data) to segment the 100DOH model predictions. Results reported as intersection-over-union (IOU) .
Figure 6: Temporal contact detection. A representative sequence showing our model’s contact predictions over time as a hand reaches toward and grasps a mug. The model correctly predicts no contact during the approach phase and transitions to contact precisely at the moment of grasp.
Figure 7: Qualitative contact segmentation results. Each row shows an input image, ground truth, 100DOH prediction, and our prediction across FEEL, HOI4D, and VISOR. Right and left hand contacts are shown in red and blue respectively. Our model correctly identifies hand side (row 1), avoids confusing proximity for contact (rows 2-3), and handles simultaneous two-hand contacts (rows 4-5). Where our model produces partial segmentations (row 3), 100DOH (combined with SAM) confidently segments the wrong object entirely.
| Model | Pretrain | EPIC-Kitchens | SSV2 | Ego-Exo4D | Meccano | ||
|---|---|---|---|---|---|---|---|
| Verb | Noun | Action | Action | Action | |||
| Frozen | Hiera-B | K710 | 41.65 | 15.12 | 36.08 | 13.11 | 26.42 |
| Hiera-B | Ours | 46.50 | 21.38 | 39.04 | 20.83 | 29.57 | |
| EgoVideo-B | Ego4D | 59.24 | 29.42 | 49.61 | 26.00 | 36.59 | |
| EgoVideo-B | Ours | 62.00 | 30.11 | 54.34 | 27.75 | 38.23 | |
| Unfrozen | Hiera-B | K710 | 68.34 | 48.18 | 68.88 | 35.25 | 38.52 |
Table 4: Action recognition results across all benchmarks. Top-1 accuracy for frozen and unfrozen settings across EPIC-Kitchens (verb/noun), SSV2, Ego-Exo4D, and Meccano. For each model, we compare standard pretraining (K710 for Hiera-B, Ego4D for EgoVideo-B) against the same weights further pretrained on FEEL’s force data. Bold denotes the better result within each model pair. Models pre-trained over FEEL’s data generally outperform the models pre-trained solely over Kinetics and Ego4D. We particularly observe this in the frozen evaluation setting compared to the unfrozen, end-to-end training.