Understanding Everyday Communication Patterns, Stress, and Productivity in Hybrid Work through Egocentric Wearable Audio Sensing
Authors: Tiantian Feng, Aditya Kommineni, Emily Zhou, Burcin Becerik-Gerber, Gale Lucas, Shawn C. Roll, Shrikanth Narayanan
Organizations: Signal Analysis and Interpretation Laboratory, University of Southern California, USA⋆ · Sonny Astani Department of Civil and Environmental Engineering, University of Southern California, USA‡ · Chan Division of Occupational Science and Occupational Therapy, University of Southern California, USA†
The hybrid work trend in the post-COVID period has transformed how people communicate while at work. However, there is a lack of quantitative understanding of how home and office environments shape communicative behavior and affective well-being. To study this, as a part of a larger study of the role of work environment on stress, we designed an egocentric wearable audio sensing system that captured interpersonal communication across both working locations. The system extracts real-time acoustic features on a wearable audio sensor and performs offline foreground speaker diarization to derive speaking load and activeness features. To improve diarization performance under low-dimensional feature constraints, we apply a teacher-student knowledge distillation framework that transfers knowledge from large-scale speech foundation models into a compact ResNet-based model, achieving consistent improvements in diarization performance over the baseline across three public egocentric speech datasets. In a four-month longitudinal deployment with 16 participants, our system revealed location-dependent differences in work communication patterns. Moreover, the mixed linear model analyses showed that office-based speaking load was significantly associated with higher self-reported stress and lower self-reported productivity, while speaking activeness during home-based work amplified both productivity and affective load in this cohort.
Figures & tables
Demographics
Female
Male
Total
( n=10 )
( n=6 )
( n=16 )
Age (Years)
μ±σ
42.6±8.7
40.5±7.6
43.5±7.9
Range
27-56
30-51
27-56
Self-report Work Time
Start Time (Range)
8-9AM
6-9AM
6-9AM
TABLE I: Participant demographic and work schedule characteristics. Office time (%) represented the self-reported proportion of total working time spent in the office.
Fig. 1: Overview of the study protocol. EMAs were distributed via text message four times daily, with the first and last surveys aligned to each participant’s self-reported work start and end times. Concurrently, participants wore a wearable audio sensor that performs on-device decoding of speech signals into acoustic features, followed by offline foreground speaker extraction and communicative feature computation.
Fig. 2: Overview of the FG/BG speaker diarization modeling that relies on teacher–student learning for improved FG/BG speaker diarization. Specifically, a teacher network based on a speech foundation model and a student model is first trained to perform FG/BG diarization. The teacher network is then transferred to the student network through knowledge distillation.
Model
MMCSG
ICSI-Meeting Corpus
EasyCom
VAD ↓
SC ↓
DER ↓
VAD ↓
SC ↓
DER ↓
VAD ↓
SC ↓
DER ↓
Teacher Model
WavLM-Large
7.7 ±0.2
0.7 ±0.1
8.4 ±0.2
13.0 ±0.3
0.3 ±0.1
13.3 ±0.4
24.7 ±1.4
0.7 ±0.3
25.4 ±0.5
Whisper-Tiny
14.5 ±0.5
4.0 ±0.1
18.5 ±0.5
17.5 ±0.8
1.9 ±0.3
19.4 ±0.5
31.7 ±1.5
2.2 ±0.4
33.9 ±3.0
Whisper-Small
7.1 ±0.1
0.8 ±0.1
7.9 ±0.1
10.0 ±0.2
0.2 ±0.1
10.2 ±0.2
23.1 ±0.4
1.0 ±0.1
24.1 ±0.5
Whisper-Large
6.5 ±0.2
0.8 ±0.1
7.3 ±0.3
8.3 ±0.2
0.1 ±0.1
8.4 ±0.3
21.6 ±0.6
1.0 ±0.1
22.6 ±0.5
TABLE II: Comparison of FG/BG speaker diarization results in our proposed work. VAD, SC, and DER represent voice activity detection error rate, speaker confusion rate, and diarization error rate, respectively. The lower number represents better performance.
Fig. 3: Evaluation of Distilled-ResNet models trained with and without office noise augmentation. We reported average DER across three datasets at Signal-to-noise ratios of 5, 15, and 25 dB.
Fig. 4: Overview of sensor wearing statistics.
Speaking Load
Speaking Activeness
Predictor
Coef.
p-value
Coef.
p-value
Intercept
1.59
0.912
-0.17
0.026
Sex (Male)
4.54
0.384
0.01
0.750
Location (Office)
12.61
< 0.001
0.12
< 0.001
Age
0.15
0.626
0.00
0.083
Group Var
103.13
-
0.01
-
TABLE III: Mixed linear model regression results for speaking load and activeness as dependent variables. Bold p-values indicated statistical significance ( p<0.05 ).
Fig. 5: Distribution of speaking time per hour across three workday periods by work location (home vs. office). Participants showed higher speaking time per hour when working in the office across all work periods.
Stress
Productivity
(n=1668)
(n=1793)
Predictor
Coef.
p-value
Coef.
p-value
Intercept
31.54
0.001
49.87
< 0.001
Sex (Male)
12.48
0.079
4.42
0.369
Survey (Mid-work EMA)
-2.10
0.263
0.04
0.983
Survey (End-work EMA)
-1.06
0.599
0.23
0.903
TABLE IV: Mixed linear model results examining the effects of location-specific slope term for communication patterns on stress and productivity. Bold indicates statistical significance ( p<0.05 ).
Healthcare professionals work in complex, high-stakes environments where effective communication is critical for care delivery, team coordination, and individual well-being. However, communication activity in everyday clinical settings remains challenging to measure and largely unexplored in human behavioral research. We present VoxCare, a scalable egocentric wearable audio sensing and computing system that captures natural communication behaviors of hospital professionals in real-world settings without storing raw audio. VoxCare performs real-time, on-device acoustic feature extraction and applies a speech foundation model-guided teacher-student framework to identify foreground speech activity. From these features, VoxCare derives interpretable behavioral measures of communication frequency, duration, and vocal arousal. Our analyses reveal how, when, and how often clinicians communicate across different shifts and working units, and suggest that communication activity reflects underlying workload and stress. By enabling continuous assessment of communication patterns in everyday contexts, this study provides data-driven approaches to understand the behaviors of healthcare providers and ultimately improve healthcare delivery.
Tiantian Feng, Kleanthis Avramidis, Anfeng Xu +3
University of Southern California, USA · The University of Memphis, USA
Automatically detecting stress in speech provides an unobtrusive way to gain insights relevant to behavioral research or clinical assessment. This study investigates the automatic differentiation between a stressful and non-stressful situation, and the prediction of physiological and affective stress responses. Speech data was collected from 50 participants who either completed the Trier Social Stress Test (TSST) or a non-stressful control condition. With a processing pipeline that included speaker diarization and machine learning models, we achieved stress detection performance significantly above a mean baseline. Moreover, relevant physiological and affective stress responses were partially predictable from acoustic-prosodic features. Feature-importance analyses identified the most informative predictors contributing to model performance. The findings demonstrate that speech can serve as a meaningful and unobtrusive indicator of multiple dimensions of the human stress response.
Hanna Drimalla, Wieland R. Cremer, Christine Kraus +1
Human-Centered Artificial Intelligence Group, Faculty of Technology, Bielefeld University, Bielefeld, Germany. · Department of Cognitive Psychology, Faculty of Psychology, Ruhr University Bochum, Bochum, Germany.
Estimating cognitive load from speech has largely been studied in controlled laboratory settings, with limited understanding of its reliability in natural collaborative conversations. We investigate whether speech and interaction dynamics predict perceived cognitive load during dyadic conversations. We analyze audio from 53 dyads performing nine collaborative tasks and extract static acoustic, dynamic, and interaction features to train a two-head Gated Recurrent Unit encoder to predict cognitive load scores. Results show conversational interaction provides useful signals for predicting cognitive load related to time pressure, mental work, effort, and task performance. Temporal demand is associated with turn-taking dynamics such as overlap and speaker switch, while mental demand is linked to imbalanced participation between speakers. These findings highlight the importance of task structure and conversational interaction for modeling cognitive load in natural collaborative settings.
Tahiya Chowdhury
Department of Computer Science, Colby College, Waterville, Maine, USA