Paper-Faithful Federated Learning: Scientific Checks for Practitioner and Agent Development
Organizations: Department of Mathematics and Mathematical Statistics Umeå University
Abstract
Federated learning experiments can produce reasonable-looking results even when the experiment implementation differs from the intended experimental protocol. Small errors in configuration, data partitioning, aggregation, randomness, evaluation, or checkpoint recovery may therefore affect a comparison without causing the program to fail. We present FedBrew, a research codebase for controlled and reproducible federated learning experiments. Its focus is controlled simulation with checks of specified conditions of the intended experimental protocol. FedBrew uses configuration checks, identifiable client data partitions, checkpoint recovery checks, self-describing run records, and development-time checks of scientific conditions. These checks assess, for example, whether client partitions are disjoint and exhaustive and aggregation weights match the prescribed rule; numerical and recovery checks examine non-finite client model states and interrupted runs. The same structure also provides clear extension points and automated checks for new algorithms, datasets, models, and tasks, whether they are added by researchers or AI coding agents. FedBrew therefore supports reproducible and inspectable FL experiments through checks of specified software and scientific conditions and the run records associated with reported results.
Figures & tables
| Failure mode | Condition checked or evidence recorded | Check stage or evidence |
| Ignored or incompatible configuration the run silently proceeds on a default and reports success | An unrecognized configuration setting raises an error rather than falling back to default values | Configuration load |
| Invalid client partition examples are duplicated or omitted, changing client data and aggregation weights | A generated partition is disjoint and exhaustive: no example is assigned to two clients, and none is dropped. | Development-time check |
| Incorrect aggregation weights a run may finish with a weighted mean different from the one required by its experimental protocol | For classification, a client’s aggregation weight is its full training-split size, independently of how many examples its local update processes. | Development-time check |
| Non-finite client model state the round completes, and the corruption is folded into the aggregate and written into the next checkpoint | A non-finite client model state is rejected before contributing to the aggregate. | During aggregation |
| Pseudorandom seed reuse or inconsistent checkpoint recovery the run completes although its pseudorandom streams or restored state differ from the intended experimental protocol | Pseudorandom streams for client selection and data loading are derived using the experiment seed and the relevant round, client, and execution phase. Development-time checks compare interrupted and uninterrupted runs under specified deterministic conditions; universal bit-for-bit equivalence across execution modes or platforms is not claimed. | Development-time check |
| Untraceable result the result cannot be traced to the experiment that produced it | The run record stores its resolved configuration, source-code revision when available, runtime metadata, and, for manifest-based datasets, the hash of the dataset manifest. Metrics and checkpoints are periodically saved to disk at a configurable round interval. The manifest hash identifies its description, not the contents of individual client data files; the recorded source-code revision does not identify the contents of modified tracked or untracked source files. | Run record ( run.json ), initialized before training and updated when results are saved to disk |
| Check case | Detection and stage | Check module ( tests/ ) |
| Unknown configuration key Injected fault | Configuration check rejects the unknown key before training. | test_unknown_config_keys.py |
| Partition disjointness and exhaustiveness Reference property | Development-time check compares client assignments with disjointness and exhaustiveness conditions. | test_partition_disjointness.py |
| Manifest declares an incorrect tensor type or range Injected fault | Preflight check reports a mismatch between declared and observed tensor properties. | test_manifest_input_claims.py |
| Incorrect classification aggregation weight or weighted mean Reference calculation | Development-time checks compare client training-split weights and the server aggregate with reference values. | test_aggregation_weight_is_the_train_split_size.py ; test_aggregation_correctness.py |
| Non-finite client model state Injected fault | Aggregation rejects the non-finite update and does not checkpoint the affected round. | test_non_finite_aggregation.py |
| Incomplete restoration of coupled server/client state Injected fault | Recovery check rejects incomplete restoration of coupled server and client state. | test_resume_is_all_or_nothing.py |
ML Reproducibility
Machine learning reproducibility studies whether model results and scientific claims can be independently checked and reliably repeated.
See all 153 papers