Domain-Invariant and Language-Model-Assisted Acoustic Side-Channel Attacks on Keyboards
Organizations: Computer Science and Engineering, Indraprastha Institute of Information Technology Delhi
Abstract
Acoustic side-channel attacks (ASCA) recover typed text from the sound of keystrokes, using only a commodity microphone and no access to the victim's device. Prior work establishes that such attacks succeed under idealized laboratory conditions - a single keyboard, a fixed microphone, a quiet room, and training and testing on the same device but leaves open the question that matters for security: whether keystroke inference survives the open world of unseen keyboards, diverse users, noisy environments, and compressed transmission channels. The answer developed here is that it does, provided the acoustic model is made explicitly domain-invariant and its noisy predictions are corrected by a language model.
Figures & tables
| ASCA | Acoustic Side-Channel Attack |
| HEAR | The keystroke-acoustics dataset introduced in this thesis (the single dataset used throughout) |
| DECKER | Domain-invariant Embedding for Cross-Keyboard Extraction and Recognition |
| EvFlow-CTSSM | Event-Triggered Continuous-Time State-Space Model with Latent Flow Canonicalization |
| LF-CAN | Latent Rectified Flow Canonicalization |
| CT-SSM | Continuous-Time State-Space Model |
| CTC | Connectionist Temporal Classification |
| Raw keystroke waveform of samples for the -th event | |
| Ground-truth key label of the -th keystroke | |
| Keyboard (domain) identity of the -th keystroke | |
| Acoustic encoder mapping a waveform to an embedding | |
| Embedding of the -th keystroke, dimension | |
| Device-specific acoustic impulse response of keyboard | |
| Latent (device-free) keystroke source signal |
| Dataset | Size | MD | MU | MG | NE | NLoS | Cont. text | VoIP/phone |
|---|---|---|---|---|---|---|---|---|
| Asonov & Agrawal (2004) [ 2 ] | 2k | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Zhuang et al. (2009) [ 62 ] | 10k | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ |
| Harrison et al. (2023) [ 25 ] | 900 | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ |
| RefleXnoop (CCS’24) [ 57 ] | 5k+ | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ |
| Heimdall (NDSS’24) [ 35 ] | 10.4k | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✓ |
| Hear (this thesis) | 10k+ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| ID | Assumption |
|---|---|
| A1 | Passive recording only; no probes or malware. |
| A2 | Commodity hardware microphones. |
| A3 | Concealed placement within everyday objects. |
| A4 | Socially realistic distance (1–3 m). |
| A5 | NLoS operation enabled by multipath reflections. |
| A6 | Robustness to ambient real-world noise. |
| Attribute | Value |
|---|---|
| Participants | (near gender-balanced) |
| Laptop keyboards | (scissor, membrane, mechanical) |
| Recording channels | external microphone, on-device microphone, VoIP stream |
| Environments | quiet (library), moderate (office), high-noise (café/outdoor) |
| Labelled keystroke events | per primary channel |
| Character classes | printable + special keys (Tab, Enter, arrows, F1–F12, …) |
| Keyboard Gen. (Standard) | Keyboard Gen. (VoIP) | Codec Gen. | ||||||||||
| ID | OOD | ID | OOD | ID | OOD | |||||||
| Model | Acc | mF1 | Acc | mF1 | Acc | mF1 | Acc | mF1 | Acc | mF1 | Acc | mF1 |
| CNN Downstream | ||||||||||||
| Wav2Vec2 | 47.82 | 46.41 | 38.95 | 37.64 | 33.71 | 32.26 | 25.14 | 23.88 | 34.15 | 32.78 | 26.54 | 25.18 |
| WavLM | 58.34 | 56.92 | 49.47 | 48.16 | 44.12 | 42.73 | 35.86 | 34.54 | 45.21 | 43.86 | 37.54 | 36.21 |
| HuBERT | 36.27 | 34.84 | 27.61 | 26.18 | 25.92 | 24.71 | 18.66 | 17.42 | 26.83 | 25.52 | 19.44 | 18.21 |
| Model architecture | Seen-domain CER | Unseen-domain CER | Degradation |
| Standard ASR & fixed-window AED | |||
| Conformer-CTC | 22.4% | 54.1% | |
| CRNN (AED) | 17.8% | 48.9% | |
| ECAPA-TDNN | 18.2% | 47.6% | |
| SSL audio foundation models | |||
| wav2vec 2.0 | 14.8% | 48.2% | |
| Model | Seen-KB | Unseen-KB | Drop |
|---|---|---|---|
| wav2vec2 (audio) | 92.4% | 51.8% | |
| ECAPA-TDNN (audio) | 95.3% | 58.1% | |
| CoAtNet-S (spec) | 90.7% | 49.6% | |
| ResNet-34 (spec) | 88.5% | 47.9% |
| Fusion variant | Seen-KB | Unseen-KB |
|---|---|---|
| F1 Early Concat | 96.8% | 63.2% |
| F2 Early Projection | 96.9% | 64.1% |
| F3 Cross-Attn (A V) | 97.1% | 65.4% |
| F4 Cross-Attn (V A) | 96.7% | 64.8% |
| F5 Bi-dir Cross-Attn | 97.3% | 66.2% |
| F6 Gated Fusion | 96.4% | 62.7% |
| Model variant | Seen-KB | Unseen-KB | Gap |
|---|---|---|---|
| Decker (full) | 98.9% | 81.3% | |
| w/o GRL | 97.8% | 68.1% | |
| w/o ASR | 97.5% | 75.2% | |
| w/o KSN | 96.9% | 63.7% | |
| ECAPA + GRL (DANN-only) | 97.9% | 71.6% |
| LLM | Char Acc. | Sent. Match | Norm. Lev. Dist. |
|---|---|---|---|
| Decker (raw) | 81.3% | 42.6% | 0.124 |
| GPT-2 Small | 86.9% | 58.7% | 0.092 |
| FLAN-T5 Base | 89.4% | 62.1% | 0.078 |
| LLaMA-2 7B | 91.6% | 66.5% | 0.056 |
| GPT-4 | 93.2% | 72.8% | 0.041 |
| Condition | Top-1 Key Acc. | Sent. Acc. (raw) | Sent. Acc. (+FLAN-T5) |
|---|---|---|---|
| No jitter | 81.3% | 42.6% | 62.1% |
| ms jitter | 78.5% | 40.1% | 59.8% |
| ms jitter | 75.4% | 37.9% | 58.1% |
| ms jitter | 69.6% | 33.2% | 52.0% |
| spurious segments | 74.0% | 36.8% | 55.9% |
| spurious segments | 71.2% | 34.7% | 53.4% |
| Module | L/K | L/S |
|---|---|---|
| KSN (1-D CNN) | 0.24 ms | 12 ms |
| ASR (optional) | 0.41 ms | 20–25 ms |
| Mel-spec extraction | 0.18 ms | 9 ms |
| ECAPA-TDNN encoding | 1.42 ms | 72–81 ms |
| Key classifier head | 0.03 ms | 1.2 ms |
| LLM (GPT-2, beam=8) | — | 68–94 ms |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| Optimizer | AdamW |
| Learning rate | |
| Weight decay | |
| Batch size | |
| Training epochs | |
| KSN pretraining epochs | ( spectral loss) |
| Cross-User | Cross-Environment | |||
|---|---|---|---|---|
| Model | In | Out | In | Out |
| ECAPA-TDNN (baseline) | 94.8% | 60.4% | 93.9% | 55.1% |
| Bi-dir Cross-Attn fusion | 96.5% | 66.9% | 95.8% | 61.7% |
| Decker (full) | 98.4% | 82.7% | 98.1% | 78.9% |
| Split | Cross-KB | Cross-User | Cross-Env | Cross-Device |
|---|---|---|---|---|
| Top-1 | 81.3% | 82.7% | 78.9% | 74.6% |
| Top-5 | 95.7% | 96.2% | 94.1% | 91.8% |
Domain Generalization
Domain generalization develops models that perform reliably in new environments without training on target-domain data.
See all 392 papers