Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors. We transfer frozen V-JEPA2 features to a set of heterogeneous sensors comprised by RGB video, depth video, and radar time series. A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets. The contribution is a controlled evaluation of subset supervision, pooling capacity, and representation transfer on 39 OMuSense-23 subjects with five subject-group folds and three seeds. With matched downstream parameter counts, the final model reaches recording-level activity macro-F1 of 75.23 averaged across seven subsets. Supervising all subsets in each batch improves the matched query-pooling control by 1.923 percentage points (conditional 95% subject-bootstrap interval [0.647, 3.209]), while query pooling shows no clear advantage over matched mean pooling under balanced supervision. Radar probes support transfer beyond pooled wavelet pixels, 64-token inference approximately preserves mean activity performance, and the tested Consensus-JEPA loss adds no measured benefit against its exact control.
Figures & tables
Figure 1: (a) Shared inference architecture of the final query model and the earlier Subset-query (B3). Grey marks frozen feature extraction and blue trained pooling and fusion. The example declares depth absence, and insets show representative observations. (b) Final all-view task supervision evaluates one batch under all seven non-empty subsets, averages the pose/activity losses, and makes one update, without a latent prediction loss.
Model (ID)
Nets
Training subsets
Pooling
Auxiliary target (weight)
Act. Avg-7
Pose
Joint
Worst-7
Full-only (B0)
1
Full input only
Mean
None
57.92±2.67
78.99
48.82
22.35
Subset-mean (B2)
1
Balanced seven
Mean
None
71.88±0.74
89.80
64.15
54.04
Subset-query (B3)
1
Balanced seven
Query
None
74.05±0.36
90.09
66.64
55.22
Specialists (B1)
7
One fixed subset
Query
None
74.74±0.55
90.29
67.63
60.00
Zero-loss (A0)
1
Balanced seven
Query
Full latent branch (0)
73.24±1.56
90.07
65.92
54.26
Consensus (C1)
1
Balanced seven
Query
Full latent, predictor (0.003)
73.23±1.16
90.06
65.91
54.73
Table 1: Earlier controlled reference models and their out-of-fold (OOF) recording-level macro-F1 (%). All use frozen V-JEPA2 tokens, a fusion transformer, and supervised pose/activity heads, and universal models sample one subset per batch. Activity Avg-7 is mean ± sample SD, pose and joint are Avg-7 means, and Worst-7 averages the within-seed minimum.
Pooling / supervision
Act. Avg-7
Pose
Joint
Worst-7
Query / balanced
73.31±0.21
89.41
65.49
54.43
Mean / balanced
73.51±0.99
89.61
65.58
56.09
Mean / all-view
73.80±1.00
90.47
66.72
56.70
Query / all-view
75.23±0.60
91.25
68.85
59.59
Table 2: Completed capacity-matched comparison. All four models have 882,695 downstream parameters. Scores are recording-level macro-F1 (%), with three-seed sample SD for activity Avg-7 and means for the remaining columns.
Figure 2: (a) Activity macro-F1 per sensor subset (three-seed means) for the local query/balanced control, the final query/all-view model, and the earlier specialists, with D depth and R radar. (b) Activity Avg-7 of the four capacity-matched models with training-seed SD, not subject intervals. Lower panels give paired contrasts in pp with 95% subject-bootstrap intervals.
Radar representation
Dim.
Activity
Pose
Joint
Pooled CWT pixels
256
69.99
71.30
45.06
Spectral/statistical
135
74.17
76.00
52.91
V-JEPA2 mean + PCA
256
73.95
82.22
60.49
V-JEPA2 mean
1,024
76.31
84.58
64.17
Table 3: Fixed radar linear probes, with classifier input dimension and radar-only recording-level macro-F1 (%), not seven-subset averages. Each row pools five OOF folds without repeated seeds.
Study/stage (representation)
Full input
Seven subsets
Published, N=50 , window-level protocols
OMuSense-23 [ 10 ] (477 handcrafted)
81.0/81.0
–
Physiological SSL [ 9 ] (four 1-D)
74.0/–
–
Hybrid fusion [ 8 ] (handcrafted+deep)
87.2/–
–
Initial stage, N=39 , rolling LOSO/window
V-JEPA2 (frozen RGB, depth, radar)
77.99/76.27
–
Table 4: Published OMuSense-23 baselines and stages of this work, as activity accuracy/macro-F1 (%). Published protocols are subject fivefold, except the 30/10/10 split of physiological SSL. Stages are not matched reruns, earlier F1 aggregation differs from pooled OOF scoring, and a dash marks an untabulated result.