Representation Is Not Enough: Body-Localized Thermal Evidence for Contactless Stress and Craving Sensing in Opioid Use Disorder
Organizations: Arizona State University
Abstract
Removing wearables from physiological monitoring also removes their supervision: the signal indicating where and when a stress response occurred. Contactless stress sensing therefore becomes a weakly supervised evidence-localization problem, where a clip-level label must be traced to the body regions and moments that produced it. We address this with FABLE-Therm, a weakly supervised architecture that preserves localized evidence across body regions, time, and encoder-specific representations until the final decision. FABLE-Therm fuses frozen foundation-model encoders at the embedding level, with theory explaining why localized fusion can outperform feature concatenation and prediction averaging. We study this problem in opioid use disorder (OUD), where stress is a major relapse trigger and sustained wearable use can be difficult during early recovery. Using fixed thermal video, FABLE-Therm achieves 0.938 AUROC on held-out participants, and its learned representation transfers to self-reported craving, providing, to our knowledge, the first evidence that craving can be recovered from contactless thermal video. Localized evidence also enables participant-level analysis of deployment failure. We find that improving representation alone is insufficient for equitable deployment: additional data from the underserved group would recover only about half of the cohort gap, while the remainder reflects person-to-person heterogeneity. This modality-agnostic decomposition applies to models with identifiable subpopulations. Together with the first cohort-structured contactless thermal OUD benchmark, our results show that preserving localized evidence supports both accurate sensing and principled analysis of who a model fails and why.
Figures & tables
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
| Study | Signal and spatial support | Modeling and supervision | Relevance to the present work |
|---|---|---|---|
| Shastri et al. (2012) | Perinasal thermal video | Morphological/wavelet perspiration extraction; lab and surgeon-expertise comparisons | Demonstrates a contactless sympathetic channel, but relies on a tracked, prespecified facial ROI and hand-crafted dynamics. |
| Engert et al. (2014) | Six facial thermal imprints with conventional stress markers | Longitudinal statistics; multivariate phase classification | Shows regional sensitivity while documenting difficulty separating anticipation, stress, and recovery in a small healthy sample. |
| Gioia et al. (2022) | Multiple predefined facial ROIs ( ECG/EDA/resp.) | Thermal statistics, recursive feature elimination, SVM | Supports thermal-only subject-independent recognition, but fixes ROI and feature vocabulary before learning. |
| Kumar et al. (2021) | Facial thermal video with ECG/ICG reference | Heat-emission representation, spatiotemporal ISTI reconstruction | Learns video dynamics via a physiologically grounded cardiac intermediary; spatial support remains facial. |
| Xiao et al. (2023) | Body thermal video plus EDA during training | Privileged-modality co-teaching, thermal-only deployment | Establishes whole-body thermal stress sensing, but uses a concurrent wearable teacher to shape thermal learning. |
| Liu et al. (2024b) | Facial thermal video and RGB rPPG | Feature-level fusion, conventional classifiers, SHAP | Complementary camera-derived physiology and explainability; fixed-feature facial multimodal design. |
| Term | Attaches to | Failure mode addressed |
|---|---|---|
| fused logit (mixup of ) | class imbalance; hard labels treated as certain | |
| each branch logit | fusion rescuing uninformative branches | |
| branch probabilities | view-specific noise masquerading as complementarity | |
| MIL weights | attention collapse to one cell / drift to uniform | |
| fusion policy | premature encoder monopoly | |
| via GRL | identity shortcut surviving a person-disjoint split |
| Group | Age | Gender | Race/Ethnicity |
|---|---|---|---|
| Control ( ) | Female | White , Asian , | |
| Male | Black , Other | ||
| OUD ( ) | Female , | White , Black , | |
| Male , | Other , | ||
| Non-binary | Am. Indian/AK Native |
| Session profile | # sessions | craving share |
|---|---|---|
| No craving (all non-craving) | ||
| Mixed (both classes present) | – | |
| All craving | ||
| Total |
| AUROC | |||
| Baseline family (best member) | Test-Both | Ctrl OUD | |
| Frozen linear probe (Concat-3) | 4 | ||
| LoRA fine-tuning (ImageNet-ViT) | 3 | ||
| Weight-space ens. (Snapshot) | 6 | ||
| Learned fusion (TMC) | 6 | ||
| Fixed comb. rules (Median/Min) | 6 | ||
| Test-Both (within-distribution) | Control-to-OUD (cross-cohort) | |||||
| Method | Acc | F1 | AUROC | Acc | F1 | AUROC |
| Frozen linear probe (logistic regression on global token) | ||||||
| AnyThermal | 0.640 | 0.745 | 0.667 | 0.448 | 0.512 | 0.613 |
| ImageNet-ViT | 0.612 | 0.713 | 0.644 | 0.518 | 0.608 | 0.615 |
| InfMAE | 0.668 | 0.790 | 0.575 | 0.437 | 0.510 | 0.569 |
| Concat-3 | 0.638 | 0.741 | 0.675 | 0.440 | 0.496 | 0.621 |
| Test-Control | Test-OUD | |||||
| Method | Acc | F1 | AUROC | Acc | F1 | AUROC |
| Frozen linear probe | ||||||
| AnyThermal | 0.637 | 0.747 | 0.649 | 0.649 | 0.740 | 0.719 |
| ImageNet-ViT | 0.592 | 0.683 | 0.705 | 0.657 | 0.772 | 0.500 |
| InfMAE | 0.663 | 0.782 | 0.589 | 0.681 | 0.806 | 0.523 |
| Concat-3 | 0.640 | 0.749 | 0.661 | 0.632 | 0.720 | 0.728 |
| Protocol | OUD in train? | AUROC | Acc. |
|---|---|---|---|
| Test-Control | yes | ||
| Test-Both | yes | ||
| Test-OUD ( ) | yes | ||
| Control OUD ( ) | no | ||
| Control-only @ the same | no | ||
| Setting | Accuracy | AUROC | Positive-class F1 |
|---|---|---|---|
| Test-Both | 0.884 | 0.938 | 0.926 |
| Test-Control | 0.927 | 0.979 | 0.953 |
| Test-OUD | 0.784 | 0.810 | 0.862 |
| Control-to-OUD | 0.817 | 0.771 | 0.894 |
| FABLE-Therm | StressNet (ours-relabelled) | |||||||
|---|---|---|---|---|---|---|---|---|
| Protocol | subj. | win. | Acc | F1 | AUROC | Acc | F1 | AUROC |
| Test-Control | ||||||||
| Test-OUD | ||||||||
| Test-Both | ||||||||
| FABLE-Therm | StressNet (ours-relabelled) | |||||||
| Protocol | subj. | win. | Acc | F1 | AUROC | Acc | F1 | AUROC |
| Control OUD | ||||||||
| Method | Accuracy | Balanced accuracy | F1 | AUROC |
|---|---|---|---|---|
| Frozen linear probes | ||||
| AnyThermal | 0.459 | 0.452 | 0.152 | 0.597 |
| ImageNet-ViT | 0.676 | 0.670 | 0.545 | 0.863 |
| InfMAE | 0.737 | 0.733 | 0.649 | 0.902 |
| Concat-3 | 0.477 | 0.469 | 0.144 | 0.601 |
| Non-deep tabular ensembles | ||||
| Aggregator | Encoder/input | Accuracy | Balanced acc. | F1 | AUROC |
|---|---|---|---|---|---|
| ABMIL ( Ilse, Tomczak, and Welling 2018 ) | AnyThermal (gated) | 0.585 | 0.577 | 0.307 | 0.720 |
| ImageNet-ViT (gated) | 0.780 | 0.776 | 0.702 | 0.949 | |
| InfMAE (gated) | 0.715 | 0.711 | 0.603 | 0.891 | |
| Concat-3 (gated) | 0.666 | 0.660 | 0.482 | 0.830 | |
| Concat-3 (plain) | 0.659 | 0.655 | 0.580 | 0.779 | |
| TransMIL ( Shao et al. 2021 ) | AnyThermal | 0.589 | 0.582 | 0.334 | 0.775 |
| Method | Accuracy | Balanced accuracy | F1 | AUROC |
|---|---|---|---|---|
| Frozen linear probe (logistic regression on global token) | ||||
| AnyThermal | 0.488 | 0.525 | 0.644 | 0.970 † |
| ImageNet-ViT | 0.226 | 0.222 | 0.165 | 0.162 |
| InfMAE | 0.519 | 0.528 | 0.555 | 0.537 |
| Concat-3 | 0.490 | 0.527 | 0.644 | 0.967 † |
| Weight-space / randomization ensembles | ||||
| Aggregator | Instance | Views | T-Both | T-OUD |
|---|---|---|---|---|
| ABMIL | static vector | 1 | ||
| TransMIL | static vector | 1 | ||
| DSMIL | static vector | 1 | ||
| FABLE-Therm branch | reg. trajectory | 1 | ||
| Separate MIL + logit attn. | reg. trajectory | 3 sep. | ||
| FABLE-Therm | reg. trajectory | 3 shd. |
| Test-Both | Control-to-OUD | ||||||
|---|---|---|---|---|---|---|---|
| Aggregator | Encoder | Acc | F1 | AUROC | Acc | F1 | AUROC |
| ABMIL | AnyThermal | 0.681 | 0.778 | 0.676 | 0.567 | 0.669 | 0.624 |
| ImageNet-ViT | 0.609 | 0.707 | 0.712 | 0.617 | 0.701 | 0.672 | |
| InfMAE | 0.625 | 0.735 | 0.604 | 0.540 | 0.648 | 0.568 | |
| Concat-3 | 0.717 | 0.808 | 0.711 | 0.639 | 0.737 | 0.679 | |
| TransMIL | AnyThermal | 0.638 | 0.734 | 0.710 | 0.747 | 0.842 | 0.665 |
| Hyperparameter | Within-group | Control-to-OUD |
|---|---|---|
| Width | 96 | 96 |
| Attention heads | 4 | 2 |
| Learning rate | ||
| Warmup epochs | 19 | 29 |
| Batch size | 32 | 32 |
| Dropout | 0.568 | 0.447 |
| Setting | Accuracy | AUROC | F1 |
|---|---|---|---|
| 3-encoder, learned gate, Optuna-tuned (Table D.5 ) | |||
| Test-Both | 0.884 | 0.938 | 0.926 |
| Test-Control | 0.927 | 0.979 | 0.953 |
| Test-OUD | 0.784 | 0.810 | 0.862 |
| 4-encoder (+LanguageBind), uniform gate, non-Optuna defaults | |||
| Test-Both | 0.867 | 0.933 | 0.913 |
| Cohort | FPR | FNR | Sensitivity | Specificity |
|---|---|---|---|---|
| Control ( ) | ||||
| FPR 95% CI: | Sens. 95% CI: | |||
| OUD ( ) | ||||
| FPR 95% CI: | Spec. 95% CI: | |||