From Sound to Occupancy: Privacy-Aware Audio Sensing for Buildings
Organizations: School of Architecture, University of Illinois Urbana-Champaign · School of Architecture, University of Kentucky
Abstract
Occupancy-based controls regulate lighting and HVAC according to whether a space is in use. Passive infrared (PIR) sensors, the most common choice, respond to motion and can report an occupied room as vacant when its occupants remain still. Sound offers an alternative cue, since seated occupants type, turn pages and cough, but microphones raise privacy concerns. We therefore study a two-stage design: an on-device encoder maps each second of audio to a short embedding, and a server-side temporal model turns the stream of embeddings into an occupancy state. To our knowledge, no publicly available dataset pairs office audio with occupancy labels, so we evaluate the two stages separately on public data. The encoder is trained on synthetic scenes that mix empty-room recordings, room impulse responses, office sounds and read speech, and tested on unseen rooms, recordings and speakers for whether a second contains a sound made by a person. On 11 real office recordings (DCASE 2013) it reaches an AUC of 0.87, against 0.76 for log-mel statistics. Attacks on the transmitted embeddings recover at most 1% of the spoken words, against 94% from log-mel frames, but still identify the speaker among 40 in 69 to 77% of the utterances. Noise-floor normalization with adversarial training, an 8-dimensional embedding, or sending only class probabilities lowers this to 38 to 55% without hurting detection. On HPDmobile, which pairs home occupancy labels with an amplitude envelope of the audio, a causal temporal model pretrained with a hierarchical contrastive loss matches hand-crafted features (AUC 0.75) and does better when only one labelled day per home is available. These results make the two-stage design a practical basis for privacy-aware acoustic occupancy sensing, with speaker identity as the one leak still to close.
Figures & tables
| Dataset | Content used | Labels | License | Role |
|---|---|---|---|---|
| HPDmobile [ 15 ] | 6 homes, 152 labelled days; 10-s clips of an 80 Hz rectified amplitude envelope from 3 to 5 hubs per home | Home occupied, every 10 s | CC0 | Temporal model |
| DCASE 2016 Task 2 [ 17 ] | 220 isolated office sounds (11 classes); 54 test mixtures of 120 s | Class, onset, offset | CC BY 3.0 / 4.0 | Event sources; test set |
| DCASE 2013 OL/OS [ 31 ] | 320 isolated office sounds (16 classes); 11 scripted recordings in real offices (1,601 s) | Class, onset, offset (two annotators) | CC BY 4.0 | Event sources; real test set |
| LibriSpeech [ 19 ] | train-clean-100 (251 speakers), dev-clean (40), test-clean (40) | Transcript, speaker, sex | CC BY 4.0 | Speech sources; attacks |
| BUT ReverbDB [ 32 ] | 9 rooms (3 offices, 2 lecture rooms, meeting room, conference room, hotel room, stairs), 31 microphones; empty-room noise (60 s) and impulse responses (1 s) | None | CC BY 4.0 | Backgrounds; room acoustics |
| Unseen rooms (synthetic) | Real offices (DCASE 2013) | DCASE 2016 test | ||||
|---|---|---|---|---|---|---|
| Representation | AUC | F1 | AUC | F1 | AUC | F1 |
| HPDmobile-style envelope | 0.699 | 0.400 | 0.621 | 0.275 | 0.726 | 0.263 |
| Log-mel statistics | 0.873 | 0.684 | 0.682 | 0.041 | 0.609 | 0.159 |
| Log-mel statistics + NF | 0.894 | 0.699 | 0.760 | 0.447 | 0.895 | 0.460 |
| Encoder | 0.959 | 0.852 | 0.834 | 0.251 | 0.950 | 0.436 |
| Encoder + adv. 1 | 0.958 | 0.751 | 0.832 | 0.170 | 0.935 | 0.365 |
| Representation | Rate size | Words (%) | STOI | Speaker ID (%) | Sex (%) |
|---|---|---|---|---|---|
| Log-mel frames (reference) | 100 Hz 64 | 94.3 | 0.96 | 98.2 | 94.3 |
| Log-mel statistics | 1 Hz 128 | 0.8 | 0.39 | 97.8 | 88.6 |
| Log-mel statistics + NF | 1 Hz 128 | 1.0 | 0.39 | 94.2 | 80.5 |
| HPDmobile-style envelope | 1 Hz 80 | 0.9 | 0.44 | 22.4 | 55.8 |
| Encoder | 1 Hz 32 | 0.9 | 0.39 | 76.8 | 81.9 |
| Encoder + adv. 1 | 1 Hz 32 | 0.4 | 0.39 | 62.3 | 82.6 |
| Strict | Transductive | |||||
|---|---|---|---|---|---|---|
| Method | Bal. acc. | F1 | AUC | Bal. acc. | F1 | AUC |
| Hand-crafted + logistic regression | 0.649 | 0.676 | 0.748 | 0.649 | 0.676 | 0.748 |
| Hand-crafted + gradient boosting | 0.654 | 0.712 | 0.743 | 0.654 | 0.712 | 0.743 |
| Same network, labels only | 0.614 .007 | 0.713 .032 | 0.708 .008 | 0.608 .028 | 0.706 .037 | 0.699 .007 |
| Pretrained + fine-tuning | 0.605 .018 | 0.704 .042 | 0.684 .033 | 0.596 .018 | 0.658 .032 | 0.686 .020 |
| Pretrained + linear probe | 0.678 .006 | 0.716 .010 | 0.752 .005 | 0.673 .008 | 0.697 .012 | 0.752 .003 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | H1 | H2 | H3 | H4 | H5 | H6 |
| Hand-crafted + logistic regression | 0.861 | 0.698 | 0.833 | 0.717 | 0.760 | 0.620 |
| Hand-crafted + gradient boosting | 0.700 | 0.745 | 0.889 | 0.702 | 0.806 | 0.616 |
| Same network, labels only | 0.622 | 0.706 | 0.871 | 0.607 | 0.748 | 0.639 |
| Pretrained + fine-tuning | 0.671 | 0.578 | 0.845 | 0.573 | 0.795 | 0.555 |
| Pretrained + linear probe | 0.826 | 0.708 | 0.815 | 0.707 | 0.761 | 0.666 |
| Labelled days | 30 | 17 | 35 | 17 | 15 | 38 |
| Representation | Real offices | DCASE 2016 test | L212 | D105 | C236 |
|---|---|---|---|---|---|
| HPDmobile-style envelope | 0.621 (0.587–0.655) | 0.726 (0.695–0.757) | 0.704 | 0.694 | 0.748 |
| Log-mel statistics | 0.682 (0.617–0.740) | 0.609 (0.587–0.631) | 0.869 | 0.855 | 0.879 |
| Log-mel statistics + NF | 0.760 (0.726–0.794) | 0.895 (0.884–0.905) | 0.907 | 0.913 | 0.885 |
| Encoder | 0.834 (0.816–0.854) | 0.950 (0.942–0.957) | 0.945 | 0.959 | 0.953 |
| Encoder + adv. 1 | 0.832 (0.817–0.847) | 0.935 (0.927–0.943) | 0.942 | 0.958 | 0.946 |
| Encoder + NF | 0.869 (0.847–0.892) | 0.961 (0.954–0.967) | 0.948 | 0.963 | 0.939 |
| Real offices (DCASE 2013) | DCASE 2016 test | Unseen rooms (synthetic) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Representation | AUC | AP | EER | AUC | AP | EER | AUC | AP | EER |
| HPDmobile-style envelope | 0.622 (0.589–0.655) | 0.564 | 0.418 | 0.713 (0.683–0.742) | 0.657 | 0.352 | 0.683 | 0.467 | 0.363 |
| Log-mel statistics | 0.660 (0.592–0.723) | 0.593 | 0.395 | 0.542 (0.519–0.566) | 0.545 | 0.479 | 0.775 | 0.598 | 0.280 |
| Log-mel statistics + NF | 0.745 (0.709–0.782) | 0.673 | 0.312 | 0.879 (0.868–0.890) | 0.841 | 0.173 | 0.832 | 0.654 | 0.214 |
| Encoder | 0.825 (0.803–0.846) | 0.804 | 0.248 | 0.941 (0.931–0.949) | 0.932 | 0.106 | 0.915 | 0.816 | 0.143 |
| Encoder + adv. 1 | 0.824 (0.807–0.838) | 0.786 | 0.260 | 0.924 (0.915–0.933) | 0.894 | 0.120 | 0.911 | 0.775 | 0.147 |
Passive Acoustic Monitoring
Passive acoustic monitoring uses unattended microphones and hydrophones to detect, classify, and track wildlife and other sounds over time.
See all 42 papers