An Error Budget for Shuffled Multimodal Windows in the 3rd WEAR Challenge: Stacking, Count Constraints and Seam Matching

Soheil Kooklan

cs.CVOct 23, 2026

An Error Budget for Shuffled Multimodal Windows in the 3rd WEAR Challenge: Stacking, Count Constraints and Seam Matching

Authors: Soheil Kooklan

Organizations: Independent Researcher

Abstract

The 3rd WEAR Dataset Challenge asks for an activity label for every one-second window of four unseen participants. Each window has 15 VideoMAEv2 frame embeddings, the accelerometer signal of one random limb, and the windows are shuffled. This report is an empirical analysis of one system built after the challenge deadline. Window classifiers are stacked with features averaged over per-participant graphs of video neighbours and scored successor candidates, and a mean-field style decoder with a soft count constraint per activity labels all windows of a participant together. The setting is offline and transductive, and the count constraint relies on each activity being performed once. In a participant-wise simulation with participant-level bootstrap intervals, stacking added 0.051 macro F1 over stage-1 fusion and the count constraint 0.011 (95% interval 0.006 to 0.017). A quadratic seam distance ranks the true same-limb successor first among about 720 candidates in 83.9% of cases, but adding it to the link model changed macro F1 by -0.0001 (-0.0006 to 0.0005). With the decoder settings and the mixing weight chosen inside the folds and the mislabelled spans left out of training, the system scores 0.936 in simulation (0.940 for participants who performed each activity once); the submitted configuration scores 0.938 (0.943). Per-participant scores range from 0.80 to 0.97. These results come from a non-nested stacking protocol that uses unlabelled features transductively, so they are not a fully inductive estimate. An oracle that averages the final unary scores over the true segments reaches 0.973, and 79% of the remaining errors involve the null class. On the private leaderboard, which no choice used, the selected Kaggle submission scored 0.9231 (public 0.9254, with bounds partly tuned there), and a later one with bounds chosen in simulation scored 0.9247 (public 0.9253).

Figures & tables

Part	Trained or tuned on	Labels of fold $k$
Window models	windows of the other four folds	not used
Successor models	successor pairs of the other four folds	not used
Participant graphs	no training; unlabelled features of one participant	not used
Video PCA	no training; unlabelled frames of all training and test participants	not used
Stage 2	OOF stage-1 outputs of the other four folds	indirectly a
Stage 3	OOF stage-2 outputs of the other four folds	indirectly a

Table 1. What each part of the system is trained or tuned on when the participants of fold

k

are validated. OOF: out-of-fold.

Panel of Figure 1 — Figure 1. Overview of the system. Everything after the window models works on one participant at a time. Five boxes connected by arrows from top to bottom: window models, successor candidates and rescorer, participant graphs, three stacking chains, and the decoder with a count constraint.

	All	Single
Video transformer (best video-only model)	0.5777	0.6035
IMU gradient boosting (best IMU-only model)	0.6139	0.6208
Mean of the two joint networks	0.6790	0.6956
Fused window models $p^{(1)}$ , arg max	0.7724	0.7816
$p^{(1)}$ + decoder	0.8876	0.8923
Stacked unary term, arg max	0.9236	0.9279

Table 2. Build-up, ablations and oracles in simulation (macro F1 for all participants and for single-performance participants). The indented rows are variants of the final system. Oracles use information that is not available at test time.

Seam distance	Top-1 (interval)	Top-5	Participants
Last sample of $a$ vs. first of $b$	0.595 (0.577–0.612)	0.818	0.499–0.666
End-velocity difference	0.011 (0.009–0.013)	0.048	0.005–0.021
Linear, from $a$ only	0.685 (0.666–0.703)	0.878	0.569–0.744
Linear, both sides	0.790 (0.775–0.803)	0.922	0.705–0.836
Quadratic, both sides (ours)	0.839 (0.828–0.849)	0.938	0.781–0.882
Quadratic, normalised	0.773 (0.757–0.787)	0.917	0.670–0.828

Table 3. Same-limb successor ranking. Each anchor is ranked against all other windows with the same limb in its recording (17,601 anchors, median 719 candidates). Interval: 95% participant bootstrap of the top-1 rate. Participants: range of the per-participant top-1 rate.

	Top-1			Macro F1
Links	all	same	cross	all	single
First-pass pair model	0.534	0.795	0.446	0.9357	0.9406
+ rescorer	0.568	0.812	0.485	0.9364	0.9416
+ Hungarian	0.586	0.810	0.510	0.9361	0.9412
+ video bi-encoder	0.628	0.809	0.567	0.9376	0.9425
+ IMU bi-encoder (final)	0.641	0.852	0.569	0.9381	0.9432

Table 4. Link models. Each indented row adds its features to the nearest row above it with less indentation, so the Hungarian variant and the video bi-encoder both extend the rescorer. Top-1: share of anchors whose highest-scored candidate is the true successor, for all, same-limb and cross-limb pairs. F1: macro F1 of the final system with only the decoder’s link scores changed.

		$h=90$	95	100	105	110	125
All	$l=70$	0.9348	0.9374	0.9375	0.9368	0.9358	0.9346
	$l=75$	0.9356	0.9381	0.9380	0.9373	0.9364	0.9353
	$l=80$	0.9357	0.9380	0.9381	0.9374	0.9364	0.9355
	$l=85$	0.9362	0.9381	0.9382	0.9377	0.9367	0.9360
Single	$l=70$	0.9391	0.9428	0.9429	0.9419	0.9404	0.9388
	$l=75$	0.9396	0.9432	0.9430	0.9421	0.9407	0.9392

Table 5. Sensitivity of simulated macro F1 to the count bounds

l

and

h

(

\beta=1.5

, final unary term), for all participants and for single-performance participants. Final setting in bold.

	Change	$l$ / $h$ / $\beta$	Sim		Public	Private
			all	single
1	Base	70/125/2	0.9332	0.9371	0.92154	0.92225
2	New stage 3, $l=80$	80/125/2	0.9348	0.9384	0.92129	0.92211
3	Lower $h$ and $\beta$	80/100/1.5	0.9367	0.9430	0.92317	0.92063
4	Seam links	80/100/1.5	0.9373	0.9433	0.92261	0.92191
5	$h=105$	80/105/1.5	0.9363	0.9424	0.92207	0.91948

Table 6. Kaggle submissions leading to the final system (rows 1 to 9), in the order they were made, and one later check (row 10, chain C with the mislabelled spans left out, made after the final choice). Row 1 uses the stage-3 models of chains A and B and of a chain with the video bi-encoder links; row 2 replaces the last by chain C trained with the original labels, and row 9 by chain C with pseudo-labels. Each row is evaluated in simulation with its own unary term. A step is compared with the configuration it changes: rows 2 and 3 with the row above, rows 4 to 6 with row 3, rows 7 and 8 with row 6, and row 9 with row 8. Sim: simulation macro F1 for all and for single-performance participants. Public and Private: public and private leaderboard; the private scores were seen only after the leaderboard closed.

Variant	All	Single
Final system, main protocol	0.9381	0.9432
Decoder settings and mixing weight chosen inside the folds	0.9359	0.9403
and mislabelled spans left out of training	0.9355	0.9403
Mislabelled spans left out of training	0.9374	0.9428
Stage 2 of chain C with another seed	0.9374	0.9428
with a third seed	0.9380	0.9430

Table 7. Simulation macro F1 of the final system under variants of the validation protocol, for all participants and for single-performance participants.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Part	Input	Model and training	Test	OOF
Video MLP	pooled features of the compressed embeddings (mean, std, mean of first three and of last three frames, mean $\|\Delta\|$ ), standardised per participant	MLP 1120–512–256–19, dropout 0.3 (input 0.1), Adam, learning rate $10^{-3}$ , weight decay $10^{-4}$ , batch 256, 15 epochs, label smoothing 0.05, class weights	refit on all, 2 seeds	0.543
Video transformer	15 frames, standardised per participant	2 pre-norm transformer layers (width 256, 4 heads, feed-forward 512, dropout 0.2) plus a mean/std branch; AdamW, learning rate $5\cdot 10^{-4}$ , weight decay 0.05, one-cycle schedule, batch 512, 12 epochs, label smoothing 0.05, class weights, 2 seeds	mean of fold models	0.578
IMU MLP	179 features (Section 3.1 ) and a copy standardised per participant and limb, limb one-hot; all four limbs of each tile	MLP 362–512–256–19, dropout 0.3, Adam, learning rate $10^{-3}$ , batch 512, 12 epochs, class weights	refit on all, 2 seeds	0.578
IMU gradient boosting	as IMU MLP, limb as categorical feature	scikit-learn HistGradientBoosting: 250 iterations, learning rate 0.08, 31 leaves, min. 40 samples per leaf, L2 1.0, balanced classes, random half of the rows	refit on all	0.614
IMU 1D CNN	3 axes, magnitude, standardised axes (7 channels)	6 convolution layers (64 to 256 channels, batch norm, GELU, two max-pools), mean and max pooling, limb embedding of size 32; AdamW, learning rate $2\cdot 10^{-3}$ , weight decay 0.01, 12 epochs of 300k windows, batch 1024, rotations of up to $10^{\circ}$ , 2 seeds	mean of fold models	0.557
Stage-1 fusion $p^{(1)}$	log probabilities of the five models above	weighted mean with weights 0.2, 0.2, 0.2, 0.3, 0.1 (order as above)	same	0.772

Appendix

Table 8. Settings of the trained parts. OOF: out-of-fold window macro F1 (all participants, scored tiles) or successor top-1. Test: how test predictions are formed. All parts are used in the final submission except the last two rows.

Class	Windows	Precision	Recall	F1	Counts q10 / q50 / q90	Count error without / with	Gained	Lost	Missed to null
Null	26,030	0.920	0.956	0.938	–	–	450	82	–
Jogging	2,503	0.961	0.965	0.963	81 / 96 / 107	3.9 / 2.7	7	18	0.55
Jogging (rotating arms)	2,192	0.953	0.934	0.943	86 / 93 / 101	4.6 / 3.6	50	5	0.24
Jogging (skipping)	2,275	0.964	0.949	0.957	75 / 91 / 100	2.8 / 1.5	7	21	0.40
Jogging (sidesteps)	2,468	0.988	0.970	0.979	82 / 92 / 103	2.6 / 2.4	5	20	0.77
Jogging (butt-kicks)	2,153	0.951	0.941	0.946	79 / 91 / 96	4.1 / 3.1	16	2	0.73

Appendix

Table 9. Per-class results of the final system in simulation (all participants, scored tiles). Counts: windows per activity in the 17 single-performance training participants (10th percentile, median, 90th percentile). Count error: mean absolute difference between the predicted and the true number of windows of the activity over the same participants, without and with the count constraint. Gained / lost: windows of the class that became correct and that became wrong when the constraint was added. Missed to null: share of the missed windows of the class that were predicted as null.

Participant	0*	1	2*	3*	4	5	6	7	8	9	10*
Macro F1	0.915	0.905	0.964	0.947	0.796	0.947	0.962	0.882	0.971	0.947	0.854
Participant	11	12	13	14*	15	16	17	18	19	20	21
Macro F1	0.972	0.946	0.950	0.932	0.960	0.962	0.974	0.943	0.969	0.972	0.974

Appendix

Table 10. Macro F1 of the final system for each training participant in simulation (scored tiles, over the classes that occur in the participant’s labels or predictions). Participants marked with an asterisk performed some or all activities twice.

Human Activity Recognition

Human activity recognition classifies actions and behaviors from wearable sensors, video, radio signals, and other measurements.

See all 228 papers

HARMES: A Multi-Modal Dataset for Wearable Human Activity Recognition with Motion, Environmental Sensing and Sound
May 4, 2026Robin Burchard, Pascal-André Brückner, Marius Bock +2Human Activity Recognition Wearable Sensing
University of Siegen, Siegen, Germany · University of Bonn and & Lamarr Institute for Machine Learning and Artificial Intelligence, Bonn, Germany
When Clean Signals Are Not Enough: Detecting Structural Ambiguity for Safe Wearable Stress Classification
Aug 19, 2026Saba A. Farahani, Hung Cao, Amir M. RahmaniWearable Health Monitoring Selective Prediction
University of California, Irvine Irvine, CA, USA
Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction
May 21, 2026Dinithi Dissanayake, Shaveen Silva, Ovindu Atukorala +2Multimodal Emotion Recognition Multimodal Fusion
Augmented Human Lab, National University of Singapore, Singapore · University of Moratuwa, Sri Lanka

May 4, 2026J/K move · Enter open · S save

HARMES: A Multi-Modal Dataset for Wearable Human Activity Recognition with Motion, Environmental Sensing and Sound

Robin Burchard, Pascal-André Brückner, Marius Bock, Juergen Gall+1

University of Siegen, Siegen, Germany · University of Bonn and & Lamarr Institute for Machine Learning and Artificial Intelligence, Bonn, Germany

Human Activity Recognition Wearable Sensing Multimodal Sensor Fusion

Open paper AskPDF arXiv

With each sensing modality exhibiting inherent strengths and limitations, multi-modal approaches for wearable Human Activity Recognition (HAR) are becoming increasingly relevant -- particularly for recognizing Activities of Daily Living (ADLs), where individual modalities often produce ambiguous signals for similar or complex activities. This work introduces HARMES, a multi-modal wearable dataset combining three wrist-recorded modalities: motion sensing via an Inertial Measurement Unit (IMU), atmospheric environmental sensors (humidity, temperature, and pressure), and audio. Collected from 20 participants performing household activities in their own homes, HARMES totals over 80 hours of recorded data, with approximately three hours of labeled activity data per participant across 15 ADL classes. To the best of our knowledge, HARMES is the first dataset to combine this particular sensor trio, and it is nearly six times larger than the previously largest wrist-inertial-acoustic HAR dataset. In an extensive benchmark, we evaluate cross-subject generalization and conduct an ablation study revealing that modality contributions are activity-dependent and can provide complementary value, particularly for activities that are ambiguous from motion data alone. HARMES is freely available at Zenodo, alongside example code for loading the dataset and training models on GitHub.

Abstract

Human Activity Recognition

Explore similar work

HARMES: A Multi-Modal Dataset for Wearable Human Activity Recognition with Motion, Environmental Sensing and Sound

When Clean Signals Are Not Enough: Detecting Structural Ambiguity for Safe Wearable Stress Classification

Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction