cs.CVOct 9, 2026
SaveWhat Egocentric Video Embeddings Encode and How to Combine Them with Inertial Sensors for Human Activity Recognition
Organizations: Independent Researcher, Japan
Abstract
Egocentric video and body-worn inertial sensors are increasingly recorded together, and the video is usually encoded by pretrained models. What these embeddings represent when the camera sits on the head, and how they should be combined with inertial sensing, has received little attention. Across VideoMAEv2, CLIP, DINOv2 and the egocentric VC-1, the leading principal directions of the embedding are dominated by where the camera looks. Measured head orientation explains them on Ego-Exo4D for egocentric (R^2 0.49-0.70) but not exocentric cameras, and the scene layout explains them on WEAR. The part that the limbs' inertial signals predict separates activities across people. In a set-level probe, the rest separates exercise sets within a participant but not activities across people. This is consistent with separate roles: the video finds the clips of the same set, and the inertial classifier labels them. On WEAR, with temporal order removed and one random limb per one-second clip, we propagate inertial predictions over a within-participant video graph whose leading components are removed. This raises subject-wise cross-validated macro-F1 from 0.648 to 0.852. In our comparisons, propagation helps only over the video graph, and fusing the modalities in one classifier, or adding inertial edges to the graph, performs worse.
Figures & tables
Figure 1. What egocentric video embeddings encode and how they combine with inertial sensing, on one WEAR recording. (a) Frames at the two ends of the first and second principal components of the video embedding (VideoMAEv2). The components follow where the camera looks. The arrow is the gradient of head tilt estimated from the frames (GeoCalib ( 7 ) ). Figure 11 shows three participants. (b) The video links the clips and the sensors label them, shown for a held-out clip of a training participant (cross-validation). (1) is linked to its 30 nearest clips in the video (edges, with one photo per limb); all come from its own set and are spread over the four limbs. (2) Each clip is predicted from its own accelerometer (colour; shape: limb). As at test time, carries one limb, and its own prediction is wrong. (3) Averaging the predictions over the links corrects . Frames from WEAR ( 1 ) (CC BY-NC-SA 4.0, https://creativecommons.org/licenses/by-nc-sa/4.0/ ), downscaled. Left: scatter of one recording's video embedding on its first two principal components, with camera frames at each end, each stacked on the frames 2 and 4 s later to show that it is video. The frames show close ground, the horizon ahead, the wearer's own limbs and the sky. An arrow shows the direction in which the estimated head tilt increases, with small head icons on its extension that look down at one end and up at the other. Right, three graphs stacked from top to bottom, each with the held-out clip as a star at the centre and edges to its 30 nearest clips in the video. (1) All nodes are grey, and photos of one neighbour per limb are linked to their nodes. (2) Nodes are coloured by each clip's own inertial prediction and shaped by its limb; the star is red (null), and the accelerometer trace of its right ankle is shown. (3) After averaging over the edges, the star is green, the true class.
Figure 2. The setting on real data (240 s of participant 0, coloured by the true activity). (a) What was recorded: all four limb sensors and activity sets lasting tens of seconds. (b) What the test set retains: each second retains one randomly assigned sensor, so the clips of one set come from different limbs. The clips are given shuffled and without labels. A set still leaves many clips with similar video, and the affinity graph exploits these. Two timelines of one recording: the continuous activity sets of four sensors, and the single randomly assigned sensor per second that the test set retains.
| WEAR, training | WEAR, test | Ego-Exo4D (analysis only) | |
|---|---|---|---|
| Participants | 22 (24 recordings) | 4, unseen | 6 (79 takes) |
| Activities | 18 outdoor workouts and null | same | basketball, dance, soccer |
| Inertial | 3-axis acc. at 50 Hz on both wrists and ankles | one randomly assigned limb per clip | head trajectory (6 DoF) of the Aria glasses |
| Video | head camera: VideoMAEv2 embeddings; public raw video | VideoMAEv2 embeddings | egocentric (Aria) and one exocentric camera |
| Clips | about one-second tiles, in order | , shuffled | one-second clips |
| Labels | per second | hidden | not used |
Table 1. Datasets. WEAR is used for the method and every evaluation. Ego-Exo4D is used only for the analysis of Section 3 .
Figure 3. What the leading components of an egocentric embedding follow, on Ego-Exo4D (VideoMAEv2, centred within each take). (a) Egocentric camera of one participant (soccer), coloured by the measured head tilt. The components follow the tilt. The arrow is the direction in which the tilt increases, with length proportional to the correlation, and the frames are the clips at its two ends. (b) Exocentric camera of the same takes. Here the components do not follow the tilt. (c) of head tilt from components 1–2 for all six participants. Frames from Ego-Exo4D ( 10 ) , used under the Ego-Exo4D licence agreement, downscaled. Left: scatter of one participant's egocentric video embeddings on their first two principal components, coloured by measured head tilt. An arrow marks the direction in which the tilt increases, with egocentric frames at its two ends (looking down at the ground, looking up across the field) and small head icons. Middle: the same takes seen by the exocentric camera, where the colours are mixed and the arrow is short. Right: bars of the variance of head tilt explained by the two components for six participants. The bars are higher for the egocentric camera.
| Encoder | Data, view | Layout | Orient. | Tilt top-30 |
|---|---|---|---|---|
| VideoMAEv2 | WEAR, ego | 0.52 | — | — |
| Ego-Exo4D, ego | 0.60 | 0.49 | 0.74 (0.63) | |
| Ego-Exo4D, exo | 0.07 | 0.36 (0.27) | ||
| CLIP | WEAR, ego | 0.61 | — | — |
| Ego-Exo4D, ego | 0.64 | 0.51 | 0.78 (0.61) | |
| Ego-Exo4D, exo | 0.44 | 0.05 | 0.22 (0.20) |
Table 2. The leading components of egocentric and exocentric embeddings. Layout and Orient.: within-recording of predicting components 1–2 from the scene layout of the frame or from head tilt, height and heading (cross-validated over five time blocks). Tilt top-30: of decoding head tilt from the top-30 subspace (cross-validated over takes), with 30 random directions in brackets. WEAR has no head sensor.
Figure 4. Overview of the approach, from one test clip to its label. Left: the two inputs of a clip. Centre-left: per-clip models trained on the training participants. , and are 19-class probabilities and is the mean of the clip’s 15 video rows. Centre-right: everything inside the dashed box runs once per test participant on all of that participant’s clips and uses no labels. This includes the graph features , three propagation stages with the graph rebuilt from between stages, and the video-only probabilities pooled after every stage. Right: the decision rule yields the label. The white panels are a schematic of a dozen clips with the actual update rules applied (position = video embedding, fill = activity prediction, grey = unsure, lines = mutual -NN edges, shape = limb carrying the accelerometer). The inertial prediction has wrong and unsure clips. Stage 1 links clips that look alike in video and averages their predictions, and stages 2–3 relink using the refined predictions as well. Raw inertial features never enter the graph. Inertial information enters it only through class probabilities ( in and the refined ), and the video never enters the inertial classifier. The dashed count band is the protocol constraint of Section 8.2 and is not part of the reported system. Block diagram of the method. Per-clip inertial and video classifiers feed a per-participant video graph. Predictions are propagated on this graph over three stages with graph re-estimation, followed by prior correction.
| P | R | ||||
|---|---|---|---|---|---|
| Inertial classifier | 0.658 | 0.644 | 0.648 | 0.761 | 0.642 |
| + propagation | 0.825 | 0.840 | 0.830 | 0.861 | 0.828 |
| + video pooling | 0.836 | 0.854 | 0.843 | 0.872 | 0.841 |
| + component removal | 0.842 | 0.865 | 0.852 | 0.876 | 0.850 |
Table 3. Record-based results under subject-wise cross-validation (mean over three seeds). is macro-F1 over all 19 classes. The last two columns split it into the null class and the 18 activities.
Figure 5. Row-normalised confusion matrices before and after the transductive stages (seed 42). The null row and column are outlined. The inertial classifier’s dominant error is activity null , and propagation removes most of that column. Two row-normalised confusion matrices over 19 classes, for the inertial ensemble and for the final system. The final system is much closer to diagonal.
Figure 6. (a) The leading components contain both the most variance and by far the most activity discriminability. (b) Removing them nevertheless raises macro-F1, with an interior optimum at 30, while neighbourhood activity purity falls monotonically. Left: variance share, activity discriminability and time correlation of the leading principal components, with the top 30 shaded. Right: pipeline macro-F1 as a function of the number of removed components, peaking at 30.
| +prob. | features only | |||||
|---|---|---|---|---|---|---|
| Encoder | ||||||
| VideoMAEv2 | 0.8410 | 0.8480 | 0.8339 | 0.8455 | ||
| CLIP | 0.8342 | 0.8435 | 0.8175 | 0.8351 | ||
| DINOv2 | 0.8337 | 0.8429 | 0.8165 | 0.8332 | ||
| VC-1 | 0.8317 | 0.8462 | 0.8151 | 0.8384 | ||
Table 4. Macro-F1 on WEAR when only the graph’s features are replaced (20 participants, 61,404 clips, mean of 2 seeds). The classifiers are fixed, and the per-clip prediction without a graph scores . “+prob.”: the graph also contains the video-probability block. “features only”: it does not.
| Subtracted | VideoMAEv2 | CLIP | DINOv2 | VC-1 |
|---|---|---|---|---|
| top 30 components | ||||
| components 1–2 only | ||||
| scene layout | ||||
| visible body | ||||
| camera tilt | ||||
| camera shake |
Table 5. Change in macro-F1 on WEAR when a component is subtracted from the video features before the graph is built (features-only graph, 2 seeds). Components are the part of the embedding explained, within a participant, by the named quantity. ∗ Uses ground-truth labels and is diagnostic only.
| inertial graph | video graph | |
|---|---|---|
| inertial classifier | (same) | +0.167 (cross) |
| video classifier | (cross) | (same) |
Table 6. Change in macro-F1 due to propagation against the same classifier without it (three seeds, configuration before component removal). Modality agreement is annotated. It does not predict the sign, but the graph source does.
| Graph | Rule / fusion | |
| inertial (classifier’s features) | LAME, best | 0.6493 |
| inertial (classifier’s features) | ours | 0.6025 |
| video | LAME, best | 0.7668 |
| video | ours | 0.8065 |
| full system, first-stage graph varied | ||
| video only | — | 0.8523 |
Table 7. Graph source and fusion (single stage unless stated, two seeds). Base: the inertial classifier alone, . Gains here and in Table 6 are measured in reduced configurations and are not directly comparable with those of Table 3 .
| Graph between stages | Cross-activity edges | |
|---|---|---|
| rebuilt from refined (ours) | 0.8523 | 0.194 / 0.087 / 0.087 |
| single stage | 0.8303 | 0.194 |
| not rebuilt, three rounds | 0.8204 | 0.194 / 0.194 / 0.194 |
| not rebuilt, 15 iterations | 0.8299 | 0.194 |
| rebuilt from base | 0.7379 | 0.194 / 0.245 / 0.245 |
| rebuilt from permuted | 0.7396 | 0.194 / 0.703 / 0.604 |
Table 8. What the stages do: macro-F1 (two seeds) and the fraction of edges joining clips of different true activities at stages 1/2/3. The sharpening and pooling schedule is identical in every row.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| F1 | inertial recall | ||||
|---|---|---|---|---|---|
| class | inertial | video | final | arm | leg |
| stretching (hamstrings) | 0.57 | 0.73 | 0.84 | 0.49 | 0.71 |
| lunges (complex) | 0.51 | 0.66 | 0.81 | 0.65 | 0.36 |
| stretching (shoulders) | 0.31 | 0.39 | 0.80 | 0.47 | 0.05 |
| sit-ups | 0.57 | 0.61 | 0.86 | 0.58 | 0.60 |
| jogging (rotating arms) | 0.68 | 0.69 | 0.84 | 0.82 | 0.53 |
Table 9. Per-class F1 of the inertial-only classifier, the video-only classifier and the final system (seed 42), with the inertial classifier’s recall split by arm and leg sensors. Bold marks the stronger single modality where the gap exceeds .
Figure 7. Ground truth and each stage for one participant (chosen as the one whose gain from the last component is closest to the median). Temporal order is used for this figure only, and the method never uses it. Colour-coded label timelines for one recording: ground truth, the inertial ensemble, and three stages of the system, which become progressively closer to the ground truth.
Figure 8. Propagation on real data. Points are one participant’s clips in a 2-D projection of their video embeddings, and colour is the activity. (a) ground truth; (b) the inertial classifier; (c) after propagation. macro-F1 on this participant rises from to . Three two-dimensional projections of one participant's video embeddings coloured by true activity, base classifier prediction and refined prediction.
Figure 9. At the base position, higher standalone accuracy yields little system accuracy. Averaging the student over sensors of the same clip (blue, privileged) traces a slope of about . The four-sensor teacher (red) lies below this line. Scatter plot of standalone clip macro-F1 against pipeline macro-F1 for students averaged over one to four sensors and for a privileged four-sensor teacher.
Figure 10. Decomposition of the 106 sets predicted entirely wrong. Only are missed by both a four-sensor inertial teacher and a video-only model, and are recoverable by video alone. The split depends on the class, and video dominates on null . Left: stacked bar of which model would rescue the sets predicted entirely wrong. Right: per-class fraction of wrong sets rescued by the four-sensor teacher and by the video-only model.
| Subtracted component | Test | ||
|---|---|---|---|
| none (raw features) | 0.8448 | 0.8523 | ✓ |
| four limbs, MLP | 0.8502 | 0.8555 | |
| four limbs, linear | 0.8500 | 0.8553 | |
| four limbs, gradient boosting | 0.8491 | 0.8535 | |
| four limbs, MLP, permuted | 0.8372 | — | |
| one limb, same for every clip | 0.8475 | 0.8544 |
Table 10. Graph built on video features after subtracting a limb-predicted component (two seeds). We denote by the number of leading components removed afterwards. “Test” marks variants a test clip can supply.
| 0.719 | 0.732 | 0.732 | 0.719 | 0.703 | |
| 0.621 | 0.632 | 0.633 | 0.622 | 0.608 |
Table 11. Neighbour purity of the video graph against labels shifted by seconds.
Figure 11. Frames at the two ends of the first and second principal components of the video embedding (VideoMAEv2, one recording per participant). Frames from WEAR ( 1 ) (CC BY-NC-SA 4.0, https://creativecommons.org/licenses/by-nc-sa/4.0/ ), downscaled. Grid of camera frames for three participants: two frames at the low and high end of the first and second principal components of each recording's video embedding.