Does the Trust Boundary Hold over Time? A Long-Horizon Benchmark for Receiver-Directed Federated Learning
Authors: Ting Xu, Henry Leung
Organizations: University of Calgary Department of Biomedical Engineering Calgary, Alberta, Canada · University of Calgary Department of Electrical and Software Engineering Calgary, Alberta, Canada
In receiver-directed federated learning, each site draws a trust boundary. For every source and model component it judges, on its own data, whether the transfer helps (admit), harms (reject) or cannot yet be told (abstain). We ask whether this boundary holds over hundreds of rounds. Our benchmark has a federation of thirteen semi-synthetic sites, an exact oracle for every evaluated transfer, paired seeds, an attack protocol, fourteen methods over 600 rounds and Qwen3-1.7B LoRA sites over 500. The answer depends on which boundary is measured. Across 20 paired CPU seeds, a gate judging against a fixed reference better identified transfers useful relative to the site's starting model. Yet it admitted 29% of transfers harmful to the model the site now has. A gate judging against the current model admitted 2% of harmful transfers on that target in the CPU tier, but missed many transfers useful to the starting model. Two short propositions explain these distinct failures. We also evaluate a held out audit that leaves gate verdicts intact. After training, it decides whether each final model is deployed or the site falls back to local training, and reports admissions that have gone stale. With independent audit data its guarantees hold after any training history. At 600 rounds the audit deployed 65 to 72% of models, none with negative benefit on its target, and kept 98% of the gain. No stale report had the wrong sign in our runs. With 20,000 audit observations, a variance-adaptive bound found 99% of stale transfers that cost at least 0.01 in log loss. A common-target Qwen3 comparison on 60 selected transfers shows the same tension over 100 rounds. With 38% adversarial sites, FedFomo and CS-Gate were the only evaluated methods with positive mean gain in every honest group.
Figures & tables
Benchmark
Attack
Truth per transfer
Scored
pFL-Bench ( 22 )
no
no
final accuracy, fairness, cost
Motley ( 23 )
no
no
final personalised accuracy
Matsuda et al. ( 24 )
no
no
final accuracy across tasks
PFLlib ( 25 )
privacy
no
final accuracy, cost, leakage
FLamby ( 26 )
no
no
final task metric
Flower Hub ( 27 )
no
no
final quality, runtime, communication
Table 1. What existing federated benchmarks score.
Oracle
Metric
Frozen arm
Current arm
Frozen better
frozen
useful
0.77
0.36
20/20
frozen
harmful
0.009
0.052
20/20
frozen
ever wrong
0.07
0.20
20/20
current
useful
0.61
0.55
16/20
current
harmful
0.287
0.022
0/20
current
ever wrong
0.39
0.24
0/20
Table 2. CS-Gate at 600 rounds scored against both oracles on the same runs (logistic sites, p=0.1 , λ=0 ; mean over 20 paired seeds). The last column counts seeds in which the frozen arm is better. Test gains are given in the text. λ=0.3 gives the same pattern.
Rounds
Reference
Useful
Harmful
Wrong-sign
Abstain
150
frozen
0.61
0.005
0.036
0.41
150
current
0.40
0.023
0.102
0.54
600
frozen
0.77
0.011
0.066
0.24
600
current
0.59
0.024
0.184
0.37
Table 3. CS-Gate with logistic sites: mean over 20 seeds and a grid of re-probing rates and rebasing weights.
Figure 1. Verdict quality over 600 rounds for logistic (left) and MLP (right) sites. Bands are 95% intervals across 20 seeds. Solid: frozen reference. Dashed: current.
No attacker
38% attackers
Method
CA min
US min
Worst
CA min
US min
Worst
FedAvg
0.568
0.430
0.023
−4812
−3790
−6057
FedProx
0.571
0.440
0.022
−468
−367
−589
Krum
0.527
0.406
0.011
−1.81
−1.75
−2.74
Trimmed mean
0.660
0.478
−0.002
−14.1
−7.82
−22.5
Median
0.570
0.449
0.016
−6.09
−2.92
−11.1
Table 4. Gain in test log loss over the local model for honest sites at 600 rounds, mean over 20 seeds.
Figure 2. CS-Gate’s gain over the local model for each honest group at four attack levels over 600 rounds. The curves coincide.
No attacker
38% attackers
Method
Useful
Harmful
Useful
Harmful
Attacker
FedFomo (2021)
0.18
0.075
0.14
0.059
0.09
FLTrust (2021)
0.29
0.289
0.24
0.211
0.18
FLAME (2022)
0.58
0.504
0.71
0.530
0.20
pFedGraph (2023)
0.93
0.862
0.39
0.545
0.35
CoBo (2024)
0.55
0.393
0.70
0.752
0.94
Table 5. Implied verdicts scored against the current-model oracle at round 600 (logistic sites, 20 seeds; adapted versions, see text). Useful and harmful are the shares of truly useful and truly harmful triples admitted. The last column is the share of the five attackers’ transfers admitted.
Frozen arm
Current arm
Deployed
68%
75%
Deployed with negative target benefit
0
0
Test gain, no audit
0.677
0.854
Test gain, audited
0.746
0.936
Sites below local on test, no audit
19/65
15/65
Sites below local on test, audited
6/65
3/65
Table 6. Audit with Qwen3 LoRA sites at round 100 ( λ=0 , five seeds, 13 receivers each). Deployment uses the empirical Bernstein (EB) bound with 5,000 audit observations. Stale reports are shown with the Hoeffding (H) and EB bounds, as correct reports out of all harmful admitted imports. Gains are test log loss over local.
Frozen arm
Current arm
Deployed, 5,000 / 20,000 obs.
65% / 92%
72% / 95%
Deployed with negative target benefit
0
0
Test gain, no audit
0.286
0.298
Test gain, audited (5,000)
0.281
0.292
Sites below local on test, no audit
37/260
24/260
Sites below local on test, audited (5,000)
12/260
12/260
Table 7. Audit at round 600 (logistic sites, λ=0 , 20 seeds, 13 receivers each). Deployment uses the empirical Bernstein (EB) bound. Stale reports are shown with the Hoeffding (H) and EB bounds, as correct reports out of all harmful admitted imports and out of those with harm at least 0.01. Gains are test log loss over local; λ=0.3 gives the same pattern.
Department of Computer Science and Engineering Hong Kong University of Science and Technology Hong Kong SAR, China · Department of Electrical and Computer Engineering University of Thessaly Volos, Greece