Healthcare professionals work in complex, high-stakes environments where effective communication is critical for care delivery, team coordination, and individual well-being. However, communication activity in everyday clinical settings remains challenging to measure and largely unexplored in human behavioral research. We present VoxCare, a scalable egocentric wearable audio sensing and computing system that captures natural communication behaviors of hospital professionals in real-world settings without storing raw audio. VoxCare performs real-time, on-device acoustic feature extraction and applies a speech foundation model-guided teacher-student framework to identify foreground speech activity. From these features, VoxCare derives interpretable behavioral measures of communication frequency, duration, and vocal arousal. Our analyses reveal how, when, and how often clinicians communicate across different shifts and working units, and suggest that communication activity reflects underlying workload and stress. By enabling continuous assessment of communication patterns in everyday contexts, this study provides data-driven approaches to understand the behaviors of healthcare providers and ultimately improve healthcare delivery.
Figures & tables
Figure 1: Overview of the VoxCare . VoxCare consists of a scalable wearable audio sensor that decodes speech signals to acoustic features in real time, an offline foreground/background speaker diarization, and a behavioral feature extractor.
Figure 2: Overview of the foreground speaker diarization process. VoxCare uses a teacher–student learning framework for foreground speaker diarization. Specifically, a teacher network based on a speech foundation model is first trained to perform foreground speaker diarization. The teacher network is then transferred to a lightweight student network through knowledge distillation.
Figure 3: Example of the FG speech sessions. The x-axis represents the time in minutes. The 1st and the 2nd sessions contain the FG speech at t=0,1 and t=4 , respectively.
Figure 4: Demographics of the primary working unit in the current dataset.
Nurse
Non-Nurse
Total
Sex
Male
49 (19.2%)
26 (10.2%)
75(29.4%)
Female
120 (47.1%)
60 (23.5%)
180 (70.6%)
Primary Shift
Day Shift
105 (41.2%)
77 (31.2%)
182(72.2%)
Night Shift
64 (25.1%)
9 (3.7%)
73(28.8%)
Table 1: Demographics of the primary work shift, sex and age in the current analysis.
Model
MMCSG
ICSI-Meeting Corpus
Miss
False
Speaker
DER
Miss
False
Speaker
DER
Detection
Alarm
Confusion
Detection
Alarm
Confusion
Teacher Model
Whisper-Tiny
9.8 ± 0.3
3.6 ± 0.2
3.3 ± 0.3
16.8 ± 0.3
10.5 ± 0.4
5.7 ± 0.3
1.4 ± 0.1
17.6 ± 0.1
Whisper-Small
3.6 ± 0.1
3.4 ± 0.1
0.8 ± 0.1
7.7 ± 0.2
6.0 ± 0.6
3.9 ± 0.5
0.2 ± 0.0
10.1 ± 0.1
Whisper-Large
3.5 ± 0.2
3.5 ± 0.6
0.8 ± 0.1
7.8 ± 0.5
4.3 ± 1.1
4.5 ± 0.5
0.2 ± 0.1
9.0 ± 0.6
Table 2: Comparison of foreground speaker diarization results in VoxCare .
Figure 5: Violin plots comparing speaking session frequency (sessions per hour, left) and average session duration (right) for day-shift and night-shift staff.
Figure 6: Violin plots comparing speaking session frequency (sessions per hour, top) and average session duration (bottom) across different hospital work units.
90th Arousal
90th Arousal
in 1st-half Shift
in 2nd-half Shift
Primary Work Shift
Day Shift
0.40 [0.38, 0.41]
0.41 [0.39, 0.42]
Night Shift
0.44 [0.41, 0.47]
0.35 [0.32, 0.38]
p-value (Work Shift)
0.001
< 0.001
F (Work Shift)
10.34
13.78
Table 3: Comparisons of speaking arousal patterns among different work units and shifts. Mean and 95% CI were reported along with the p-value and F in the three-way ANOVA test.
Figure 7: Correlations between speaking frequency and IRB scores among day-shift staff working in labs (left) and Non-ICU units (right).
Figure 8: Correlations between speaking session frequency and STAI scores among day-shift (left) and night-shift (right) staff working in Non-ICU units.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Illustration of different types of diarization errors used in model evaluation.
Figure 10: Correlations between 90th percentile speaking arousal and STAI scores among day-shift and night-shift staff working in Non-ICU and ICU units.
Figure 11: Changes in 90th Arousal across the shift between day shift and night shift staff working in nursing units.
Figure 12: Correlations between speaking frequency and IRB scores among night-shift staff working in ICU (left) and Non-ICU units (right).
Clinical speech reveals identity through both spoken identifiers and voice, yet its pauses, delivery, and speaker roles can also carry useful clinical evidence. Protecting that speech therefore requires more than preserving its words. We present AegisVox, a factorized framework that assigns consistent surrogate voices and edits identifier-bearing turns using document-consistent replacements. Across 390 public simulated or non-patient conversations totaling 63.6 hours, a matched comparison of two speech renderers separates replacement realization from retained identifiers, speaker linkage, and clinical preservation. Text-to-speech rendering realizes 535/601 planned replacements versus 453/601 for the codec-based speech editor. However, similar aggregate clinical preservation conceals more model-labeled contradictions, and conversation-wide speaker scores conceal greater edited-speech linkage under a score-direction sensitivity analysis. Fresh transcription of original audio provides a reference for interpreting the clinical measurement chain; an independent judge tests sensitivity to model-based assessment. Better replacement realization can therefore coexist with worse clinical or speaker-linkage measurements. AegisVox supports use-specific evaluation of those tradeoffs, not a guarantee of identifier removal or clinical safety.
Wearable human activity recognition (HAR) has made steady progress, yet much of this progress remains grounded in fixed-window, closed-set classification benchmarks. This formulation is poorly matched to everyday behavior, where activities are open-ended, unscripted, personalized, variable in duration, and often compositional. To address this mismatch, we introduce ActivityNarrated, an open-ended narrative paradigm for language-grounded wearable activity understanding. We formulate this setting as dense sensor signal captioning with a comprehensive benchmark protocol that measures temporal localization, caption quality, sensor-language alignment, conventional closed-set classification as a downstream diagnostic, and additional robustness measures. We further present ActNarrator, a 3-stage architecture that discretizes continuous IMU signals into reusable motion tokens and uses an external frozen small language model to generate open-vocabulary activity captions. Experiments show that our method provides high quality dense sensor captioning with superior adaptivity and robustness, enabling various downstream tasks by turning sensor-based human activity understanding into sensor-grounded text-level reasoning. This includes downstream classification where ActNarrator outperforms state-of-the-art HAR models by 3.8 - 31.6 % in Macro-F1. This paradigm also enables novel activity understanding capabilities such as complex question-answering over long time horizons.
Speech and language technologies offer valuable opportunities for supporting mental health assessment through objective and interpretable cues. We present a systematic feature-based analysis framework leveraging perceptually grounded acoustic and linguistic characteristics, including prosody, vocal quality, semantic coherence, syntactic structure, and sarcasm. Using statistical analysis and interpretable machine learning (XGBoost with SHAP and LIME), we examine associations between speech features and validated symptom measures of depression, anxiety, and ADHD. Evaluated on both controlled benchmark datasets (StressID, DAIC-WOZ, Androids, EATD) and a real-world clinical dataset, the framework reveals stable and consistent relationships between symptom severity and vocal irregularities (e.g., shimmer, jitter), lexical-syntactic patterns, and affective tone. An ablation study conducted across all datasets further identifies the most informative feature groups. This work explores a transparent and clinically interpretable approach to speech-based mental health analysis.
Vassilis Lyberatos, Edmund G. Dervakos, Eleni Adamidi +2
1National Technical University of Athens, Athens, Greece · 2PsychNow