ZarrFM: A Canonical Tensor Storage Architecture for EEG Foundation-Model Pretraining
Organizations: University of Michigan, Ann Arbor · Reteena · University of Texas, Austin
Abstract
Statistical models for brain-computer interfaces (BCIs) increasingly require large, heterogeneous EEG corpora, yet existing datasets vary in sampling rates, channel layouts, window lengths, and storage formats, creating substantial preprocessing and I/O overhead. We propose ZarrFM, an execution-oriented representation that complements BIDS by storing a corpus as a single Zarr v3 hierarchy with indexed metadata, canonical channel spaces, explicit masks, hash-stable subject splits, verified event tables, and block-aligned sharding. On real PhysioNet recordings and synthetic corpora, we compare ZarrFM with EDF, FIF, NPY, HDF5, and Parquet. Across 200 recordings, its two-level index reduces metadata queries from 601 requests to 14. Preserving native signal codes reduces storage from 42.4 MB/hour in EDF to 22.7 MB/hour, while block-aligned sampling reaches 3,131 windows/s, up to 11 times faster than Parquet. Reads remain bit-identical across independent Zarr implementations, and concurrent failure-injected appends recover without data loss. These results show that ZarrFM can provide a compact, interoperable, and execution-efficient storage layer for large-scale EEG pretraining.
Figures & tables
| format | size | windows/s | MB/s out | ms/window |
|---|---|---|---|---|
| NPY float32 memmap (ceiling) | 84.8 MB | 50031 | 4713 | 0.02 |
| HDF5 float32, chunked, uncompressed | 84.8 MB | 15786 | 1487 | 0.06 |
| ZarrFM, block-merged reads | 22.7 MB | 3131 | 295 | 0.32 |
| HDF5 float32, gzip-4 | 33.9 MB | 1584 (1055–1615) | 149 | 0.63 |
| FIF (MNE, preload=False ) | 84.8 MB | 672 | 63 | 1.49 |
| EDF, original file (MNE, preload=False ) | 42.4 MB | 595 | 56 | 1.68 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| node | requirement |
|---|---|
| root attrs | format="zarrfm-dataset" , version (semver), dataset_id (UUID by default), canonical_version , n_recordings ; SHOULD: name , description , created , created_by . |
| recordings/<id>/ | one conforming recording per id; id is one path component, SHOULD read sub-X_ses-Y[_task-Z] . |
| index/ | recording_id , session , task , primary_stream , content_hash as JSON list attrs; arrays subject_idx int32, n_channels int32, n_samples int64, sfreq / duration_s float64, n_events / nbytes int64, split int8 ( unassigned), all length ; canonical_present bool. MUST agree with the recordings’ primary streams. |
| subjects/ | attr id unique; optional aligned float64 arrays or string-list attrs (minimal by policy). |
| splits/ | attr names ; attrs level {subject, session, recording}, seed , fractions , explicit . No unit at the declared level may straddle splits. |
| vocab/ | attr event_labels ; every recording’s live revision vocab MUST equal it. |
| node | type | meaning |
|---|---|---|
| channels/* attrs name , type , unit , reference | JSON lists | vendor name; type vocabulary; unit after scaling; what the channel was measured against |
| channels/*/gain, offset | float64 | physical raw gain offset; the source’s own when its codes fit int16 |
| channels/*/bad | bool | flagged bad; masked, not removed; the one node an edit rewrites in place |
| channels/*/canonical_id | int32 | index into the canonical table, if none |
| channels/*/position | float64 | head-frame metres (x right, y anterior, z up) |
| events/r /onset, duration | float64 | seconds from t0 ; onset non-decreasing, inside the primary stream |
| representation | files | requests | 0 ms (NVMe) | 1 ms (NFS-like) | 10 ms (object store) |
|---|---|---|---|---|---|
| BIDS sidecars (stdlib parsers) | 601 | 601 | 140 ms | 969 ms | 6502 ms |
| Zarr, per-recording channel tables | 4952 | 2614 | 6722 ms | 9387 ms | 28252 ms |
| Zarr, dataset index (two-level) | 4952 | 14 | 36 ms | 48 ms | 152 ms |
| recording | EDF | ZarrFM | entropy | open | epochs | windows |
|---|---|---|---|---|---|---|
| CHB-MIT, 23 ch, 256 Hz, 1 h | 42.4 MB | 22.7 MB | 8.7 bit | 4.7 /24.5/19.6 | 11 /76/61 | 49/ 15 / 14 |
| EEGBCI, 64 ch, 160 Hz, 2 min | 2.6 MB | 1.4 MB | 7.7 bit | 4.7 /9.8/6.6 | 12 /35/16 | 47/ 16 / 10 |
| Sleep-EDF | 48.3 MB | 26.3 MB | 9.3 bit | 4.5 /45.6/467.4 | 15.4 /1779.4/1768.2 | 34.5 /288.2/308.9 |
| format | size | windows/s | MB/s out | ms/window |
|---|---|---|---|---|
| NPY float32 memmap | 39.3 MB | 7240 | 1898 | 0.14 |
| HDF5 float32, chunked, uncompressed | 39.3 MB | 4066 | 1066 | 0.25 |
| ZarrFM, block-merged reads | 18.5 MB | 3221 | 844 | 0.31 |
| FIF (MNE, preload=False ) | 39.3 MB | 693 | 182 | 1.44 |
| HDF5 float32, gzip-4 | 36.3 MB | 489 | 128 | 2.05 |
| EDF (MNE export, preload=False ) | 19.7 MB | 460 | 121 | 2.17 |
| synthetic, 64 ch (bytes at 64 ch) | CHB-MIT, 23 ch | ||||||
|---|---|---|---|---|---|---|---|
| chunk | sequential | block | per-window | sequential | block | per-window | |
| 512 (64 KiB) | 475 | 490 | 168 | 617 | 588 | 186 | |
| 1024 (128 KiB) | 660 | 730 | 190 | 1031 | 995 | 209 | |
| 2048 (256 KiB) | 941 | 1001 | 183 | 1470 | 1455 | 202 | |
| 4096 (512 KiB) | 1201 | 1294 | 167 | 1847 | 1810 | 190 | |
| 8192 (1 MiB) | 1286 | 1377 | 140 | 2116 | 2131 | 165 | |
| latency | block win/s | req. | random win/s | req. |
|---|---|---|---|---|
| 0 ms | 1359 | 48 | 143 | 600 |
| 1 ms | 1340 | 48 | 119 | 600 |
| 10 ms | 1020 | 48 | 58 | 600 |
| 30 ms | 684 | 48 | 27 | 600 |
Electroencephalography
Electroencephalography (EEG) studies brain-generated electrical activity to understand cognition and support diagnosis, monitoring, and brain-computer interfaces.
See all 200 papers