DeepAFL: Deep Analytic Federated Learning

Jianheng Tang; Yajiang Huang; Kejia Fan; Feijiang Han; Jiaxu Li; Jinfeng Xu; Run He; Anfeng Liu

cs.LGFeb 28, 2026

DeepAFL: Deep Analytic Federated Learning

Authors: Jianheng Tang, Yajiang Huang, Kejia Fan, Feijiang Han, Jiaxu Li, Jinfeng Xu, Run He, Anfeng Liu,

Organizations: Peking University, PRC · Key Lab of High Confidence Software Technologies (PKU), Ministry of Education, PRC · Central South University, PRC · University of Pennsylvania, USA ·

Abstract

Federated Learning (FL) is a popular distributed learning paradigm to break down data silo. Traditional FL approaches largely rely on gradient-based updates, facing significant issues about heterogeneity, scalability, convergence, and overhead, etc. Recently, some analytic-learning-based work has attempted to handle these issues by eliminating gradient-based updates via analytical (i.e., closed-form) solutions. Despite achieving superior invariance to data heterogeneity, these approaches are fundamentally limited by their single-layer linear model with a frozen pre-trained backbone. As a result, they can only achieve suboptimal performance due to their lack of representation learning capabilities. In this paper, to enable representable analytic models while preserving the ideal invariance to data heterogeneity for FL, we propose our Deep Analytic Federated Learning approach, named DeepAFL. Drawing inspiration from the great success of ResNet in gradient-based learning, we design gradient-free residual blocks in our DeepAFL with analytical solutions. We introduce an efficient layer-wise protocol for training our deep analytic models layer by layer in FL through least squares. Both theoretical analyses and empirical evaluations validate our DeepAFL's superior performance with its dual advantages in heterogeneity invariance and representation learning, outperforming state-of-the-art baselines by up to 5.68%-8.42% across three benchmark datasets.

Figures & tables

Figure 1: Comparing our proposed DeepAFL with the naive approaches for representation learning. (a) Illustration of multi-analytic layers with random projections. (b) Illustration of multi-analytic layers with random projections & activation functions. (c) Illustration of our proposed DeepAFL. (d) Among these, our DeepAFL exhibits the best performance improvements with increasing layers. For all these approaches, the activation function is GELU, the random projection dimension is 1024, the pre-trained model is the ResNet-18 used in AFL, and the evaluation dataset is the CIFAR-100.

Baseline	CIFAR-100				Tiny-ImageNet
	Non-IID-1		Non-IID-2		Non-IID-1		Non-IID-2
	$\alpha=0.1$	$\alpha=0.01$	$s=10$	$s=5$	$\alpha=0.1$	$\alpha=0.01$	$s=10$	$s=5$
FedAvg ( 2017 )	$56.62_{\pm 0.12}$	$32.99_{\pm 0.20}$	$55.76_{\pm 0.13}$	$48.33_{\pm 0.15}$	$46.04_{\pm 0.27}$	$32.63_{\pm 0.19}$	$39.06_{\pm 0.26}$	$29.66_{\pm 0.19}$
FedProx ( 2020 )	$56.45_{\pm 0.22}$	$33.37_{\pm 0.09}$	$55.80_{\pm 0.16}$	$48.29_{\pm 0.14}$	$46.47_{\pm 0.23}$	$32.26_{\pm 0.14}$	$38.97_{\pm 0.23}$	$29.17_{\pm 0.16}$
MOON ( 2021a )	$56.58_{\pm 0.02}$	$33.34_{\pm 0.11}$	$55.70_{\pm 0.25}$	$48.34_{\pm 0.19}$	$46.21_{\pm 0.14}$	$32.38_{\pm 0.20}$	$38.79_{\pm 0.14}$	$29.24_{\pm 0.30}$
FedGen ( 2021 )	$56.48_{\pm 0.17}$	$33.09_{\pm 0.09}$	$60.93_{\pm 0.17}$	$48.12_{\pm 0.06}$	$46.27_{\pm 0.14}$	$32.33_{\pm 0.14}$	$38.82_{\pm 0.16}$	$29.37_{\pm 0.25}$

Table 1: Performance comparisons of the top-1 accuracy (%) among our DeepAFL and the baselines, on the CIFAR-100 and Tiny-ImageNet. The best result is highlighted in bold , and the second-best result is underlined . All the experiments were conducted three times, and the results are shown as

\mathrm{Mean}_{\pm\mathrm{Standard\;Error}}

. All the improvements of our DeepAFL were validated by Chi-squared tests. The results of all baselines are directly obtained from the given benchmark in AFL ( He et al., 2025 ) .

Panel of Figure 2 — Figure 2: Efficiency evaluations on the CIFAR-100 dataset.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Baseline	CIFAR-10
	Non-IID-1		Non-IID-2
	$\alpha=0.1$	$\alpha=0.05$	$s=4$	$s=2$
FedAvg ( 2017 )	$64.02_{\pm 0.18}$	$60.52_{\pm 0.39}$	$68.47_{\pm 0.13}$	$57.81_{\pm 0.03}$
FedProx ( 2020 )	$64.07_{\pm 0.08}$	$60.39_{\pm 0.09}$	$68.46_{\pm 0.08}$	$57.61_{\pm 0.12}$
MOON ( 2021a )	$63.84_{\pm 0.03}$	$60.28_{\pm 0.17}$	$68.47_{\pm 0.15}$	$57.72_{\pm 0.15}$
FedGen ( 2021 )	$64.14_{\pm 0.24}$	$60.65_{\pm 0.19}$	$68.24_{\pm 0.28}$	$57.02_{\pm 0.18}$

Appendix

Table 2: Performance comparisons of the top-1 accuracy (%) among our DeepAFL and the baselines, on the CIFAR-10. The best result is highlighted in bold , and the second-best result is underlined . All the experiments were conducted three times, and the results are shown as

\mathrm{Mean}_{\pm\mathrm{Standard\;Error}}

. All the improvements of our DeepAFL were validated by Chi-squared tests at the level of

p<0.05

. The results of all baselines are directly obtained from the given benchmark in AFL ( He et al., 2025 ) .

Datasets	Layers	$\lambda=0.01$	$\lambda=0.05$	$\lambda=0.1$	$\lambda=0.5$	$\lambda=1$	$\lambda=5$	$\lambda=10$
CIFAR-10	$T=5$	$85.14_{\pm 0.05}$	$85.15_{\pm 0.04}$	$85.14_{\pm 0.03}$	$85.11_{\pm 0.05}$	$85.13_{\pm 0.04}$	$85.17_{\pm 0.05}$	$\textbf{85.20}_{\pm 0.05}$
	$T=10$	$85.87_{\pm 0.08}$	$85.90_{\pm 0.09}$	$85.86_{\pm 0.08}$	$85.92_{\pm 0.08}$	$85.91_{\pm 0.08}$	$85.92_{\pm 0.10}$	$\textbf{85.93}_{\pm 0.09}$
	$T=20$	$86.34_{\pm 0.07}$	$86.31_{\pm 0.08}$	$86.37_{\pm 0.07}$	$86.27_{\pm 0.06}$	$86.33_{\pm 0.08}$	$86.39_{\pm 0.08}$	$\textbf{86.43}_{\pm 0.07}$
CIFAR-100	$T=5$	$64.61_{\pm 0.01}$	$64.62_{\pm 0.02}$	$64.59_{\pm 0.02}$	$64.61_{\pm 0.02}$	$64.62_{\pm 0.02}$	$64.64_{\pm 0.01}$	$\textbf{64.66}_{\pm 0.02}$
	$T=10$	$65.95_{\pm 0.07}$	$\textbf{65.96}_{\pm 0.06}$	$65.95_{\pm 0.08}$	$65.94_{\pm 0.06}$	$65.94_{\pm 0.06}$	$65.91_{\pm 0.04}$	$65.91_{\pm 0.05}$
	$T=20$	$66.91_{\pm 0.06}$	$66.91_{\pm 0.06}$	$66.92_{\pm 0.06}$	$\textbf{66.96}_{\pm 0.05}$	$66.94_{\pm 0.03}$	$66.87_{\pm 0.03}$	$66.95_{\pm 0.03}$

Appendix

Table 3: Accuracy of our DeepAFL under varying regularization parameters

\lambda

Datasets	Layers	$\gamma=0.01$	$\gamma=0.05$	$\gamma=0.1$	$\gamma=0.5$	$\gamma=1$	$\gamma=5$	$\gamma=10$
CIFAR-10	$T=5$	$85.06_{\pm 0.05}$	$85.14_{\pm 0.06}$	$\textbf{85.15}_{\pm 0.04}$	$85.10_{\pm 0.02}$	$85.10_{\pm 0.04}$	$85.04_{\pm 0.04}$	$84.99_{\pm 0.04}$
	$T=10$	$85.87_{\pm 0.06}$	$85.90_{\pm 0.07}$	$\textbf{85.91}_{\pm 0.26}$	$85.86_{\pm 0.10}$	$85.84_{\pm 0.08}$	$85.78_{\pm 0.08}$	$85.65_{\pm 0.07}$
	$T=20$	$86.21_{\pm 0.09}$	$86.21_{\pm 0.08}$	$86.32_{\pm 0.08}$	$\textbf{86.37}_{\pm 0.06}$	$86.33_{\pm 0.08}$	$86.26_{\pm 0.07}$	$86.23_{\pm 0.07}$
CIFAR-100	$T=5$	$\textbf{64.72}_{\pm 0.07}$	$64.66_{\pm 0.02}$	$64.63_{\pm 0.01}$	$64.58_{\pm 0.04}$	$64.49_{\pm 0.00}$	$63.77_{\pm 0.07}$	$63.21_{\pm 0.10}$
	$T=10$	$\textbf{65.96}_{\pm 0.05}$	$65.94_{\pm 0.06}$	$65.95_{\pm 0.08}$	$65.81_{\pm 0.08}$	$65.69_{\pm 0.06}$	$64.82_{\pm 0.15}$	$64.26_{\pm 0.15}$
	$T=20$	$\textbf{66.98}_{\pm 0.04}$	$66.94_{\pm 0.04}$	$66.94_{\pm 0.03}$	$66.82_{\pm 0.07}$	$66.74_{\pm 0.07}$	$66.20_{\pm 0.09}$	$65.61_{\pm 0.16}$

Appendix

Table 4: Accuracy of our DeepAFL under varying regularization parameters

\gamma

Datasets	Layers	None	GELU	ReLU	LeakyReLU	Tanh	Hardwish	Softshrink
CIFAR-10	$T=5$	$83.23_{\pm 0.10}$	$\textbf{85.13}_{\pm 0.04}$	$84.37_{\pm 0.09}$	$84.49_{\pm 0.12}$	$84.42_{\pm 0.07}$	$84.78_{\pm 0.03}$	$83.77_{\pm 0.12}$
	$T=10$	$83.08_{\pm 0.17}$	$\textbf{85.89}_{\pm 0.10}$	$85.04_{\pm 0.12}$	$84.54_{\pm 0.49}$	$84.92_{\pm 0.12}$	$85.41_{\pm 0.04}$	$84.13_{\pm 0.05}$
	$T=20$	$82.99_{\pm 0.06}$	$\textbf{86.34}_{\pm 0.05}$	$85.47_{\pm 0.17}$	$83.85_{\pm 1.48}$	$85.62_{\pm 0.05}$	$86.05_{\pm 0.06}$	$84.25_{\pm 0.07}$
CIFAR-100	$T=5$	$60.82_{\pm 0.09}$	$\textbf{64.62}_{\pm 0.01}$	$64.26_{\pm 0.02}$	$64.29_{\pm 0.01}$	$63.28_{\pm 0.12}$	$64.02_{\pm 0.02}$	$62.89_{\pm 0.13}$
	$T=10$	$60.82_{\pm 0.09}$	$\textbf{65.98}_{\pm 0.08}$	$65.39_{\pm 0.09}$	$65.48_{\pm 0.10}$	$64.27_{\pm 0.17}$	$65.07_{\pm 0.20}$	$63.91_{\pm 0.14}$
	$T=20$	$60.83_{\pm 0.09}$	$\textbf{66.95}_{\pm 0.02}$	$66.68_{\pm 0.08}$	$66.72_{\pm 0.10}$	$64.02_{\pm 1.02}$	$66.04_{\pm 0.06}$	$64.78_{\pm 0.08}$

Appendix

Table 5: Accuracy of our DeepAFL under varying activation functions

\sigma(\cdot)

Datasets	Layers	$d_{\Phi}=2^{7}$	$d_{\Phi}=2^{8}$	$d_{\Phi}=2^{9}$	$d_{\Phi}=2^{10}$	$d_{\Phi}=2^{11}$	$d_{\Phi}=2^{12}$	$d_{\Phi}=2^{13}$
CIFAR-10	$T=5$	$81.48_{\pm 0.30}$	$83.31_{\pm 0.12}$	$84.58_{\pm 0.08}$	$85.07_{\pm 0.02}$	$85.48_{\pm 0.03}$	$\textbf{85.76}_{\pm 0.7}$	$85.57_{\pm 0.33}$
	$T=10$	$82.28_{\pm 0.30}$	$84.41_{\pm 0.01}$	$85.55_{\pm 0.11}$	$\textbf{85.96}_{\pm 0.10}$	$85.94_{\pm 0.01}$	$85.94_{\pm 0.12}$	$85.77_{\pm 0.28}$
	$T=20$	$82.68_{\pm 0.27}$	$84.95_{\pm 0.13}$	$86.01_{\pm 0.01}$	$86.28_{\pm 0.15}$	$86.41_{\pm 0.01}$	$\textbf{86.48}_{\pm 0.11}$	$83.12_{\pm 0.39}$
CIFAR-100	$T=5$	$57.61_{\pm 0.23}$	$62.21_{\pm 0.14}$	$64.06_{\pm 0.07}$	$64.63_{\pm 0.02}$	$65.39_{\pm 0.05}$	$66.46_{\pm 0.12}$	$\textbf{67.46}_{\pm 0.01}$
	$T=10$	$58.20_{\pm 0.19}$	$63.45_{\pm 0.19}$	$65.29_{\pm 0.07}$	$65.93_{\pm 0.10}$	$66.36_{\pm 0.04}$	$\textbf{67.20}_{\pm 0.06}$	$67.19_{\pm 0.44}$
	$T=20$	$58.51_{\pm 0.25}$	$64.29_{\pm 0.36}$	$66.64_{\pm 0.10}$	$66.93_{\pm 0.04}$	$67.39_{\pm 0.06}$	$67.85_{\pm 0.05}$	$\textbf{67.94}_{\pm 0.02}$

Appendix

Table 6: Accuracy of our DeepAFL under varying dimensions

d_{\Phi}

. For a comprehensive and consistent analysis, all the experimental results were obtained by fixing the other dimension

d_{\text{F}}

at 1024.

Datasets	Layers	$d_{\text{F}}=2^{7}$	$d_{\text{F}}=2^{8}$	$d_{\text{F}}=2^{9}$	$d_{\text{F}}=2^{10}$	$d_{\text{F}}=2^{11}$	$d_{\text{F}}=2^{12}$	$d_{\text{F}}=2^{13}$
CIFAR-10	$T=5$	$83.48_{\pm 0.10}$	$83.76_{\pm 0.09}$	$84.24_{\pm 0.14}$	$85.14_{\pm 0.04}$	$86.16_{\pm 0.06}$	$\textbf{86.51}_{\pm 0.04}$	$85.55_{\pm 0.11}$
	$T=10$	$83.58_{\pm 0.11}$	$84.03_{\pm 0.10}$	$84.84_{\pm 0.08}$	$85.89_{\pm 0.09}$	$\textbf{86.50}_{\pm 0.13}$	$86.42_{\pm 0.04}$	$84.90_{\pm 0.09}$
	$T=20$	$83.83_{\pm 0.11}$	$84.49_{\pm 0.09}$	$85.34_{\pm 0.04}$	$86.31_{\pm 0.08}$	$86.53_{\pm 0.05}$	$\textbf{85.59}_{\pm 0.10}$	$83.40_{\pm 0.10}$
CIFAR-100	$T=5$	$61.22_{\pm 0.06}$	$61.82_{\pm 0.11}$	$63.11_{\pm 0.11}$	$64.63_{\pm 0.01}$	$66.49_{\pm 0.12}$	$\textbf{67.81}_{\pm 0.01}$	$67.38_{\pm 0.09}$
	$T=10$	$61.54_{\pm 0.10}$	$62.57_{\pm 0.10}$	$64.08_{\pm 0.04}$	$65.96_{\pm 0.08}$	$67.59_{\pm 0.06}$	$\textbf{67.92}_{\pm 0.08}$	$66.12_{\pm 0.04}$
	$T=20$	$61.92_{\pm 0.14}$	$63.41_{\pm 0.12}$	$65.23_{\pm 0.18}$	$66.94_{\pm 0.03}$	$\textbf{68.20}_{\pm 0.03}$	$67.14_{\pm 0.07}$	$64.10_{\pm 0.10}$

Appendix

Table 7: Accuracy of our DeepAFL under varying dimensions

d_{\text{F}}

. For a comprehensive and consistent analysis, all the experimental results were obtained by fixing the other dimension

d_{\Phi}

at 1024.

Datasets	Layers	$d=2^{7}$	$d=2^{8}$	$d=2^{9}$	$d=2^{10}$	$d=2^{11}$	$d=2^{12}$	$d=2^{13}$
CIFAR-10	$T=5$	$77.01_{\pm 0.31}$	$80.62_{\pm 0.17}$	$83.54_{\pm 0.09}$	$85.04_{\pm 0.12}$	$86.19_{\pm 0.03}$	$\textbf{86.54}_{\pm 0.04}$	$/$
	$T=10$	$77.96_{\pm 0.21}$	$81.62_{\pm 0.19}$	$84.24_{\pm 0.08}$	$85.77_{\pm 0.04}$	$\textbf{86.58}_{\pm 0.04}$	$86.47_{\pm 0.12}$	$/$
	$T=20$	$78.69_{\pm 0.11}$	$82.38_{\pm 0.08}$	$85.07_{\pm 0.07}$	$86.22_{\pm 0.02}$	$86.71_{\pm 0.13}$	$\textbf{84.76}_{\pm 0.34}$	$/$
CIFAR-100	$T=5$	$49.88_{\pm 0.27}$	$57.15_{\pm 0.10}$	$61.74_{\pm 0.04}$	$64.63_{\pm 0.00}$	$66.86_{\pm 0.13}$	$\textbf{68.59}_{\pm 0.08}$	$/$
	$T=10$	$51.36_{\pm 0.13}$	$58.76_{\pm 0.12}$	$63.22_{\pm 0.02}$	$65.96_{\pm 0.06}$	$68.20_{\pm 0.06}$	$\textbf{69.10}_{\pm 0.03}$	$/$
	$T=20$	$53.21_{\pm 0.10}$	$60.26_{\pm 0.07}$	$64.72_{\pm 0.11}$	$66.93_{\pm 0.03}$	$68.75_{\pm 0.06}$	$\textbf{69.12}_{\pm 0.08}$	$/$

Appendix

Table 8: Accuracy of our DeepAFL under varying dimensions

d_{\Phi}

and

d_{\text{F}}

. For a comprehensive and consistent analysis, all the experimental results were obtained by ensuring

d_{\Phi}=d_{\text{F}}=d

Layers	Training Acc	$\Delta_{\text{Training Acc}}$	Testing Acc	$\Delta_{\text{Testing Acc}}$	Time Cost	$\Delta_{\text{Time Cost}}$
AFL	$83.81_{\pm 0.00}$	$/$	$80.75_{\pm 0.00}$	$/$	$52.36_{\pm 0.24}$	$/$
$T=0$	$85.35_{\pm 0.02}$	$1.54_{\pm 0.02}$	$83.29_{\pm 0.20}$	$2.54_{\pm 0.20}$	$58.02_{\pm 0.17}$	$5.66_{\pm 0.13}$
$T=5$	$89.03_{\pm 0.03}$	$3.68_{\pm 0.02}$	$85.13_{\pm 0.08}$	$1.84_{\pm 0.08}$	$65.08_{\pm 0.39}$	$7.06_{\pm 0.22}$
$T=10$	$90.61_{\pm 0.05}$	$1.58_{\pm 0.03}$	$85.89_{\pm 0.09}$	$0.75_{\pm 0.11}$	$71.91_{\pm 0.28}$	$6.83_{\pm 0.29}$
$T=15$	$91.80_{\pm 0.04}$	$1.19_{\pm 0.04}$	$86.18_{\pm 0.08}$	$0.29_{\pm 0.04}$	$78.86_{\pm 0.24}$	$6.95_{\pm 0.15}$
$T=20$	$92.74_{\pm 0.03}$	$0.94_{\pm 0.01}$	$86.34_{\pm 0.06}$	$0.16_{\pm 0.06}$	$85.71_{\pm 0.22}$	$6.85_{\pm 0.17}$

Appendix

Table 9: Analyses of our DeepAFL’s representation learning capability on the CIFAR-10 with increasing network depth. “Training Acc” and “Testing Acc” show the accuracy (%) on the training and test sets. “Time Cost” denotes the required training time (s). The symbol

\Delta

represents the difference between two consecutive cases, which reflects the marginal effect of deepening the network.

Layers	Training Acc	$\Delta_{\text{Training Acc}}$	Testing Acc	$\Delta_{\text{Testing Acc}}$	Time Cost	$\Delta_{\text{Time Cost}}$
AFL	$61.55_{\pm 0.00}$	$/$	$58.56_{\pm 0.00}$	$/$	$50.05_{\pm 0.29}$	$/$
$T=0$	$65.66_{\pm 0.04}$	$4.11_{\pm 0.04}$	$60.81_{\pm 0.15}$	$2.25_{\pm 0.15}$	$56.90_{\pm 0.17}$	$6.85_{\pm 0.19}$
$T=5$	$74.93_{\pm 0.04}$	$9.27_{\pm 0.04}$	$64.62_{\pm 0.02}$	$3.81_{\pm 0.08}$	$66.48_{\pm 0.21}$	$9.58_{\pm 0.23}$
$T=10$	$79.41_{\pm 0.08}$	$4.48_{\pm 0.04}$	$65.98_{\pm 0.13}$	$1.36_{\pm 0.09}$	$75.25_{\pm 0.33}$	$8.77_{\pm 0.03}$
$T=15$	$82.69_{\pm 0.09}$	$3.28_{\pm 0.04}$	$66.59_{\pm 0.03}$	$0.61_{\pm 0.09}$	$83.50_{\pm 0.12}$	$8.25_{\pm 0.09}$
$T=20$	$85.15_{\pm 0.03}$	$2.45_{\pm 0.06}$	$66.95_{\pm 0.04}$	$0.36_{\pm 0.02}$	$91.74_{\pm 0.29}$	$8.24_{\pm 0.22}$

Appendix

Table 10: Analyses of our DeepAFL’s representation learning capability on the CIFAR-100 with increasing network depth. “Training Acc” and “Testing Acc” show the accuracy (%) on the training and test sets. “Time Cost” denotes the required training time (s). The symbol

\Delta

represents the difference between two consecutive cases, which reflects the marginal effect of deepening the network.

Layers	Training Acc	$\Delta_{\text{Training Acc}}$	Testing Acc	$\Delta_{\text{Testing Acc}}$	Time Cost	$\Delta_{\text{Time Cost}}$
AFL	$57.39_{\pm 0.00}$	$/$	$54.67_{\pm 0.00}$	$/$	$125.31_{\pm 0.38}$	$/$
$T=0$	$60.30_{\pm 0.02}$	$2.91_{\pm 0.02}$	$56.72_{\pm 0.14}$	$2.05_{\pm 0.14}$	$130.72_{\pm 0.22}$	$5.41_{\pm 0.08}$
$T=5$	$66.89_{\pm 0.06}$	$6.59_{\pm 0.02}$	$60.24_{\pm 0.02}$	$3.52_{\pm 0.07}$	$145.73_{\pm 0.13}$	$15.01_{\pm 0.11}$
$T=10$	$70.14_{\pm 0.03}$	$3.24_{\pm 0.02}$	$61.34_{\pm 0.13}$	$1.10_{\pm 0.06}$	$160.75_{\pm 0.23}$	$15.02_{\pm 0.03}$
$T=15$	$72.64_{\pm 0.05}$	$2.50_{\pm 0.02}$	$61.94_{\pm 0.08}$	$0.60_{\pm 0.11}$	$175.76_{\pm 0.34}$	$15.01_{\pm 0.02}$
$T=20$	$74.75_{\pm 0.02}$	$2.11_{\pm 0.03}$	$62.34_{\pm 0.02}$	$0.40_{\pm 0.03}$	$190.01_{\pm 0.09}$	$14.25_{\pm 0.11}$

Appendix

Table 11: Analyses of our DeepAFL’s representation learning capability on the Tiny-ImageNet with increasing network depth. “Training Acc” and “Testing Acc” show the accuracy (%) on the training and test sets. “Time Cost” denotes the required training time (s). The symbol

\Delta

represents the difference between two consecutive cases, which reflects the marginal effect of deepening the network.

Layers	DeepAFL	Ablation (1)	Ablation (2)	Ablation (3)	Ablation (4)
$T=0$	${\color[rgb]{0,0,0}\textbf{85.35}_{\pm 0.02}}$	$85.35_{\pm 0.02}$ $(\downarrow 0.00)$	$85.35_{\pm 0.02}$ $(\downarrow 0.00)$	$85.35_{\pm 0.02}$ $(\downarrow 0.00)$	$85.35_{\pm 0.02}$ $(\downarrow 0.00)$
$T=5$	${\color[rgb]{0,0,0}\textbf{89.03}_{\pm 0.03}}$	$85.23_{\pm 0.05}$ $(\downarrow 3.80)$	$86.86_{\pm 0.03}$ $(\downarrow 2.17)$	$85.35_{\pm 0.02}$ $(\downarrow 3.68)$	$86.13_{\pm 0.03}$ $(\downarrow 2.90)$
$T=10$	${\color[rgb]{0,0,0}\textbf{90.61}_{\pm 0.05}}$	$85.28_{\pm 0.05}$ $(\downarrow 5.33)$	$86.97_{\pm 0.03}$ $(\downarrow 3.64)$	$85.35_{\pm 0.02}$ $(\downarrow 5.26)$	$86.30_{\pm 0.04}$ $(\downarrow 4.31)$
$T=15$	${\color[rgb]{0,0,0}\textbf{91.80}_{\pm 0.04}}$	$85.32_{\pm 0.07}$ $(\downarrow 6.48)$	$87.03_{\pm 0.04}$ $(\downarrow 4.77)$	$85.35_{\pm 0.02}$ $(\downarrow 6.45)$	$86.38_{\pm 0.04}$ $(\downarrow 5.42)$
$T=20$	${\color[rgb]{0,0,0}\textbf{92.74}_{\pm 0.03}}$	$85.33_{\pm 0.07}$ $(\downarrow 7.41)$	$87.06_{\pm 0.03}$ $(\downarrow 5.68)$	$85.35_{\pm 0.02}$ $(\downarrow 7.39)$	$86.43_{\pm 0.04}$ $(\downarrow 6.31)$
$T=25$	${\color[rgb]{0,0,0}\textbf{93.49}_{\pm 0.02}}$	$85.34_{\pm 0.06}$ $(\downarrow 8.15)$	$87.07_{\pm 0.03}$ $(\downarrow 6.42)$	$85.35_{\pm 0.02}$ $(\downarrow 8.14)$	$86.45_{\pm 0.03}$ $(\downarrow 7.04)$

Appendix

Table 12: Ablation study of our DeepAFL’s training accuracy on the CIFAR-10. We evaluate four distinct ablation models to explore the contributions of our key components in the representations: (1) Ablating the residual skip connection, (2) Ablating the random projection

\mathbf{B}_{t}

, (3) Ablating the activation function

\sigma(\cdot)

, and (4) Ablating the trainable transformation

\mathbf{\Omega}_{t+1}

. The value in parentheses

(\downarrow)

indicates the performance drop compared to our full DeepAFL under identical conditions.

Layers	DeepAFL	Ablation (1)	Ablation (2)	Ablation (3)	Ablation (4)
$T=0$	$\textbf{83.29}_{\pm 0.20}$	$83.29_{\pm 0.20}$ $(\downarrow 0.00)$	$83.29_{\pm 0.20}$ $(\downarrow 0.00)$	$83.29_{\pm 0.20}$ $(\downarrow 0.00)$	$83.29_{\pm 0.20}$ $(\downarrow 0.00)$
$T=5$	$\textbf{85.13}_{\pm 0.08}$	$83.03_{\pm 0.08}$ $(\downarrow 2.10)$	$83.92_{\pm 0.21}$ $(\downarrow 1.21)$	$83.33_{\pm 0.23}$ $(\downarrow 1.80)$	$83.87_{\pm 0.16}$ $(\downarrow 1.26)$
$T=10$	$\textbf{85.89}_{\pm 0.09}$	$83.06_{\pm 0.07}$ $(\downarrow 2.83)$	$83.99_{\pm 0.21}$ $(\downarrow 1.90)$	$83.34_{\pm 0.23}$ $(\downarrow 2.55)$	$83.95_{\pm 0.14}$ $(\downarrow 1.94)$
$T=15$	$\textbf{86.18}_{\pm 0.08}$	$83.05_{\pm 0.08}$ $(\downarrow 3.13)$	$84.00_{\pm 0.24}$ $(\downarrow 2.18)$	$83.33_{\pm 0.23}$ $(\downarrow 2.85)$	$84.08_{\pm 0.28}$ $(\downarrow 2.10)$
$T=20$	$\textbf{86.34}_{\pm 0.06}$	$83.03_{\pm 0.08}$ $(\downarrow 3.31)$	$84.01_{\pm 0.25}$ $(\downarrow 2.33)$	$83.32_{\pm 0.23}$ $(\downarrow 3.02)$	$84.02_{\pm 0.22}$ $(\downarrow 2.32)$
$T=25$	$\textbf{86.40}_{\pm 0.06}$	$82.99_{\pm 0.11}$ $(\downarrow 3.41)$	$84.00_{\pm 0.29}$ $(\downarrow 2.40)$	$83.33_{\pm 0.23}$ $(\downarrow 3.07)$	$84.08_{\pm 0.27}$ $(\downarrow 2.32)$

Appendix

Table 13: Ablation study of our DeepAFL’s testing accuracy on the CIFAR-10. We evaluate four distinct ablation models to explore the contributions of our key components in the representations: (1) Ablating the residual skip connection, (2) Ablating the random projection

\mathbf{B}_{t}

, (3) Ablating the activation function

\sigma(\cdot)

, and (4) Ablating the trainable transformation

\mathbf{\Omega}_{t+1}

. The value in parentheses

(\downarrow)

indicates the performance drop compared to our full DeepAFL under identical conditions.

Layers	DeepAFL	Ablation (1)	Ablation (2)	Ablation (3)	Ablation (4)
$T=0$	$\textbf{65.66}_{\pm 0.04}$	$65.66_{\pm 0.04}$ $(\downarrow 0.00)$	$65.66_{\pm 0.04}$ $(\downarrow 0.00)$	$65.66_{\pm 0.04}$ $(\downarrow 0.00)$	$65.66_{\pm 0.04}$ $(\downarrow 0.00)$
$T=5$	$\textbf{74.93}_{\pm 0.04}$	$65.81_{\pm 0.04}$ $(\downarrow 9.12)$	$69.80_{\pm 0.11}$ $(\downarrow 5.13)$	$65.66_{\pm 0.04}$ $(\downarrow 9.27)$	$65.67_{\pm 0.07}$ $(\downarrow 9.26)$
$T=10$	$\textbf{79.41}_{\pm 0.08}$	$66.32_{\pm 0.03}$ $(\downarrow 13.1)$	$70.16_{\pm 0.10}$ $(\downarrow 9.25)$	$65.66_{\pm 0.04}$ $(\downarrow 13.8)$	$65.70_{\pm 0.06}$ $(\downarrow 13.7)$
$T=15$	$\textbf{82.69}_{\pm 0.09}$	$66.60_{\pm 0.11}$ $(\downarrow 16.1)$	$70.38_{\pm 0.09}$ $(\downarrow 12.3)$	$65.66_{\pm 0.03}$ $(\downarrow 17.0)$	$65.72_{\pm 0.08}$ $(\downarrow 17.0)$
$T=20$	$\textbf{85.15}_{\pm 0.03}$	$66.86_{\pm 0.08}$ $(\downarrow 18.3)$	$70.46_{\pm 0.08}$ $(\downarrow 14.7)$	$65.66_{\pm 0.03}$ $(\downarrow 19.5)$	$65.72_{\pm 0.09}$ $(\downarrow 19.4)$
$T=25$	$\textbf{87.21}_{\pm 0.02}$	$67.10_{\pm 0.12}$ $(\downarrow 20.1)$	$70.57_{\pm 0.08}$ $(\downarrow 16.6)$	$65.67_{\pm 0.04}$ $(\downarrow 21.5)$	$65.74_{\pm 0.10}$ $(\downarrow 21.5)$

Appendix

Table 14: Ablation study of our DeepAFL’s training accuracy on the CIFAR-100. We evaluate four distinct ablation models to explore the contributions of our key components in the representations: (1) Ablating the residual skip connection, (2) Ablating the random projection

\mathbf{B}_{t}

, (3) Ablating the activation function

\sigma(\cdot)

, and (4) Ablating the trainable transformation

\mathbf{\Omega}_{t+1}

. The value in parentheses

(\downarrow)

indicates the performance drop compared to our full DeepAFL under identical conditions.

Layers	DeepAFL	Ablation (1)	Ablation (2)	Ablation (3)	Ablation (4)
$T=0$	$\textbf{60.81}_{\pm 0.15}$	$60.81_{\pm 0.15}$ $(\downarrow 0.00)$	$60.81_{\pm 0.15}$ $(\downarrow 0.00)$	$60.81_{\pm 0.15}$ $(\downarrow 0.00)$	$60.81_{\pm 0.15}$ $(\downarrow 0.00)$
$T=5$	$\textbf{64.62}_{\pm 0.02}$	$60.91_{\pm 0.15}$ $(\downarrow 3.71)$	$62.43_{\pm 0.09}$ $(\downarrow 2.19)$	$60.80_{\pm 0.16}$ $(\downarrow 3.82)$	$60.80_{\pm 0.10}$ $(\downarrow 3.82)$
$T=10$	$\textbf{65.98}_{\pm 0.13}$	$61.17_{\pm 0.10}$ $(\downarrow 4.81)$	$62.55_{\pm 0.06}$ $(\downarrow 3.43)$	$60.81_{\pm 0.16}$ $(\downarrow 5.17)$	$60.82_{\pm 0.05}$ $(\downarrow 5.16)$
$T=15$	$\textbf{66.59}_{\pm 0.03}$	$61.48_{\pm 0.06}$ $(\downarrow 5.11)$	$62.55_{\pm 0.08}$ $(\downarrow 4.04)$	$60.82_{\pm 0.16}$ $(\downarrow 5.77)$	$60.85_{\pm 0.05}$ $(\downarrow 5.74)$
$T=20$	$\textbf{66.95}_{\pm 0.04}$	$61.82_{\pm 0.10}$ $(\downarrow 5.13)$	$62.55_{\pm 0.09}$ $(\downarrow 4.40)$	$60.82_{\pm 0.16}$ $(\downarrow 6.13)$	$60.84_{\pm 0.12}$ $(\downarrow 6.11)$
$T=25$	$\textbf{67.40}_{\pm 0.10}$	$61.95_{\pm 0.11}$ $(\downarrow 5.45)$	$62.61_{\pm 0.05}$ $(\downarrow 4.79)$	$60.82_{\pm 0.16}$ $(\downarrow 6.58)$	$60.83_{\pm 0.07}$ $(\downarrow 6.57)$

Appendix

Table 15: Ablation study of our DeepAFL’s testing accuracy on the CIFAR-100. We evaluate four distinct ablation models to explore the contributions of our key components in the representations: (1) Ablating the residual skip connection, (2) Ablating the random projection

\mathbf{B}_{t}

, (3) Ablating the activation function

\sigma(\cdot)

, and (4) Ablating the trainable transformation

\mathbf{\Omega}_{t+1}

. The value in parentheses

(\downarrow)

indicates the performance drop compared to our full DeepAFL under identical conditions.

Layers	DeepAFL	Ablation (1)	Ablation (2)	Ablation (3)	Ablation (4)
$T=0$	$\textbf{60.30}_{\pm 0.02}$	$60.30_{\pm 0.02}$ $(\downarrow 0.00)$	$60.30_{\pm 0.02}$ $(\downarrow 0.00)$	$60.30_{\pm 0.02}$ $(\downarrow 0.00)$	$60.30_{\pm 0.02}$ $(\downarrow 0.00)$
$T=5$	$\textbf{66.89}_{\pm 0.06}$	$60.84_{\pm 0.03}$ $(\downarrow 6.05)$	$63.21_{\pm 0.03}$ $(\downarrow 3.68)$	$60.30_{\pm 0.01}$ $(\downarrow 6.59)$	$60.31_{\pm 0.02}$ $(\downarrow 6.58)$
$T=10$	$\textbf{70.14}_{\pm 0.03}$	$61.87_{\pm 0.05}$ $(\downarrow 8.27)$	$63.46_{\pm 0.03}$ $(\downarrow 6.68)$	$60.30_{\pm 0.01}$ $(\downarrow 9.84)$	$60.31_{\pm 0.01}$ $(\downarrow 9.83)$
$T=15$	$\textbf{72.64}_{\pm 0.05}$	$62.53_{\pm 0.04}$ $(\downarrow 10.1)$	$63.59_{\pm 0.03}$ $(\downarrow 9.05)$	$60.31_{\pm 0.01}$ $(\downarrow 12.3)$	$60.32_{\pm 0.02}$ $(\downarrow 12.3)$
$T=20$	$\textbf{74.75}_{\pm 0.02}$	$62.97_{\pm 0.03}$ $(\downarrow 11.8)$	$63.68_{\pm 0.03}$ $(\downarrow 11.1)$	$60.31_{\pm 0.01}$ $(\downarrow 14.4)$	$60.33_{\pm 0.02}$ $(\downarrow 14.4)$
$T=25$	$\textbf{76.58}_{\pm 0.03}$	$63.30_{\pm 0.02}$ $(\downarrow 13.3)$	$63.74_{\pm 0.03}$ $(\downarrow 12.8)$	$60.31_{\pm 0.01}$ $(\downarrow 16.3)$	$60.33_{\pm 0.03}$ $(\downarrow 16.3)$

Appendix

Table 16: Ablation study of our DeepAFL’s training accuracy on the Tiny-ImageNet. We evaluate four distinct ablation models to explore the contributions of our key components in the representations: (1) Ablating the residual skip connection, (2) Ablating the random projection

\mathbf{B}_{t}

, (3) Ablating the activation function

\sigma(\cdot)

, and (4) Ablating the trainable transformation

\mathbf{\Omega}_{t+1}

. The value in parentheses

(\downarrow)

indicates the performance drop compared to our full DeepAFL under identical conditions.

Layers	DeepAFL	Ablation (1)	Ablation (2)	Ablation (3)	Ablation (4)
$T=0$	$\textbf{56.72}_{\pm 0.14}$	$56.72_{\pm 0.14}$ $(\downarrow 0.00)$	$56.72_{\pm 0.14}$ $(\downarrow 0.00)$	$56.72_{\pm 0.14}$ $(\downarrow 0.00)$	$56.72_{\pm 0.14}$ $(\downarrow 0.00)$
$T=5$	$\textbf{60.24}_{\pm 0.02}$	$57.39_{\pm 0.11}$ $(\downarrow 2.85)$	$58.16_{\pm 0.24}$ $(\downarrow 2.08)$	$56.69_{\pm 0.13}$ $(\downarrow 3.55)$	$56.69_{\pm 0.11}$ $(\downarrow 3.55)$
$T=10$	$\textbf{61.34}_{\pm 0.13}$	$58.42_{\pm 0.16}$ $(\downarrow 2.92)$	$58.22_{\pm 0.17}$ $(\downarrow 3.12)$	$56.70_{\pm 0.13}$ $(\downarrow 4.64)$	$56.69_{\pm 0.14}$ $(\downarrow 4.65)$
$T=15$	$\textbf{61.94}_{\pm 0.08}$	$59.01_{\pm 0.10}$ $(\downarrow 2.93)$	$58.20_{\pm 0.14}$ $(\downarrow 3.74)$	$56.70_{\pm 0.13}$ $(\downarrow 5.24)$	$56.69_{\pm 0.12}$ $(\downarrow 5.25)$
$T=20$	$\textbf{62.34}_{\pm 0.02}$	$59.21_{\pm 0.08}$ $(\downarrow 3.13)$	$58.25_{\pm 0.19}$ $(\downarrow 4.09)$	$56.70_{\pm 0.13}$ $(\downarrow 5.64)$	$56.68_{\pm 0.12}$ $(\downarrow 5.66)$
$T=25$	$\textbf{62.66}_{\pm 0.05}$	$59.37_{\pm 0.14}$ $(\downarrow 3.29)$	$58.25_{\pm 0.16}$ $(\downarrow 4.41)$	$56.70_{\pm 0.14}$ $(\downarrow 5.96)$	$56.70_{\pm 0.10}$ $(\downarrow 5.96)$

Appendix

Table 17: Ablation study of our DeepAFL’s testing accuracy on the Tiny-ImageNet. We evaluate four distinct ablation models to explore the contributions of our key components in the representations: (1) Ablating the residual skip connection, (2) Ablating the random projection

\mathbf{B}_{t}

, (3) Ablating the activation function

\sigma(\cdot)

, and (4) Ablating the trainable transformation

\mathbf{\Omega}_{t+1}

. The value in parentheses

(\downarrow)

indicates the performance drop compared to our full DeepAFL under identical conditions.

Layers	DeepAFL	Strategy (1)	Strategy (2)	Strategy (3)	Strategy (4)
$T=0$	${\color[rgb]{0,0,0}\textbf{85.35}_{\pm 0.02}}$	$85.35_{\pm 0.02}$ $(\downarrow 0.00)$	$85.35_{\pm 0.02}$ $(\downarrow 0.00)$	$85.35_{\pm 0.02}$ $(\downarrow 0.00)$	$85.35_{\pm 0.02}$ $(\downarrow 0.00)$
$T=5$	${\color[rgb]{0,0,0}\textbf{89.03}_{\pm 0.03}}$	$85.35_{\pm 0.02}$ $(\downarrow 3.68)$	$85.36_{\pm 0.03}$ $(\downarrow 3.67)$	$85.68_{\pm 0.01}$ $(\downarrow 3.35)$	$85.68_{\pm 0.02}$ $(\downarrow 3.35)$
$T=10$	${\color[rgb]{0,0,0}\textbf{90.61}_{\pm 0.05}}$	$85.35_{\pm 0.03}$ $(\downarrow 5.26)$	$85.36_{\pm 0.03}$ $(\downarrow 5.25)$	$85.68_{\pm 0.01}$ $(\downarrow 4.93)$	$85.67_{\pm 0.02}$ $(\downarrow 4.94)$
$T=15$	${\color[rgb]{0,0,0}\textbf{91.80}_{\pm 0.04}}$	$85.35_{\pm 0.03}$ $(\downarrow 6.45)$	$85.35_{\pm 0.03}$ $(\downarrow 6.45)$	$85.70_{\pm 0.00}$ $(\downarrow 6.10)$	$85.68_{\pm 0.03}$ $(\downarrow 6.12)$
$T=20$	${\color[rgb]{0,0,0}\textbf{92.74}_{\pm 0.03}}$	$85.35_{\pm 0.03}$ $(\downarrow 7.39)$	$85.35_{\pm 0.03}$ $(\downarrow 7.39)$	$85.70_{\pm 0.01}$ $(\downarrow 7.04)$	$85.67_{\pm 0.01}$ $(\downarrow 7.07)$
$T=25$	${\color[rgb]{0,0,0}\textbf{93.49}_{\pm 0.02}}$	$85.35_{\pm 0.03}$ $(\downarrow 8.14)$	$85.35_{\pm 0.03}$ $(\downarrow 8.14)$	$85.71_{\pm 0.01}$ $(\downarrow 7.78)$	$85.68_{\pm 0.02}$ $(\downarrow 7.81)$

Appendix

Table 18: Training accuracy comparison of our DeepAFL against the other four deepening strategies for the analytic networks on the CIFAR-10, including (1) cascades w/ random feature, (2) cascades w/ random feature & label encoding, (3) cascades w/ activated random feature, (4) cascades w/ activated random feature & label encoding. All the strategies share the same zero-layer features to ensure fairness. The value in parentheses

(\downarrow)

indicates the performance lags behind our DeepAFL.

Layers	DeepAFL	Strategy (1)	Strategy (2)	Strategy (3)	Strategy (4)
$T=0$	$\textbf{83.29}_{\pm 0.20}$	$83.30_{\pm 0.20}$ $(\downarrow 0.00)$	$83.30_{\pm 0.20}$ $(\downarrow 0.00)$	$83.30_{\pm 0.20}$ $(\downarrow 0.00)$	$83.30_{\pm 0.20}$ $(\downarrow 0.00)$
$T=5$	$\textbf{85.13}_{\pm 0.08}$	$83.30_{\pm 0.22}$ $(\downarrow 1.83)$	$83.31_{\pm 0.24}$ $(\downarrow 1.82)$	$83.62_{\pm 0.20}$ $(\downarrow 1.51)$	$83.61_{\pm 0.20}$ $(\downarrow 1.52)$
$T=10$	$\textbf{85.89}_{\pm 0.09}$	$83.29_{\pm 0.23}$ $(\downarrow 2.60)$	$83.31_{\pm 0.23}$ $(\downarrow 2.58)$	$83.63_{\pm 0.23}$ $(\downarrow 2.26)$	$83.60_{\pm 0.20}$ $(\downarrow 2.29)$
$T=15$	$\textbf{86.18}_{\pm 0.08}$	$83.32_{\pm 0.23}$ $(\downarrow 2.86)$	$83.31_{\pm 0.23}$ $(\downarrow 2.87)$	$83.62_{\pm 0.20}$ $(\downarrow 2.56)$	$83.60_{\pm 0.19}$ $(\downarrow 2.58)$
$T=20$	$\textbf{86.34}_{\pm 0.06}$	$83.29_{\pm 0.24}$ $(\downarrow 3.05)$	$83.31_{\pm 0.24}$ $(\downarrow 3.03)$	$83.62_{\pm 0.22}$ $(\downarrow 2.72)$	$83.62_{\pm 0.23}$ $(\downarrow 2.72)$
$T=25$	$\textbf{86.40}_{\pm 0.06}$	$83.29_{\pm 0.24}$ $(\downarrow 3.11)$	$83.31_{\pm 0.23}$ $(\downarrow 3.09)$	$83.63_{\pm 0.22}$ $(\downarrow 2.77)$	$83.63_{\pm 0.21}$ $(\downarrow 2.77)$

Appendix

Table 19: Testing accuracy comparison of our DeepAFL against the other four deepening strategies for the analytic networks on the CIFAR-10, including (1) cascades w/ random feature, (2) cascades w/ random feature & label encoding, (3) cascades w/ activated random feature, (4) cascades w/ activated random feature & label encoding. All the strategies share the same zero-layer features to ensure fairness. The value in parentheses

(\downarrow)

indicates the performance lags behind our DeepAFL.

Layers	DeepAFL	Strategy (1)	Strategy (2)	Strategy (3)	Strategy (4)
$T=0$	$\textbf{65.66}_{\pm 0.04}$	$65.66_{\pm 0.04}$ $(\downarrow 0.00)$	$65.66_{\pm 0.04}$ $(\downarrow 0.00)$	$65.66_{\pm 0.04}$ $(\downarrow 0.00)$	$65.66_{\pm 0.04}$ $(\downarrow 0.00)$
$T=5$	$\textbf{74.93}_{\pm 0.04}$	$65.64_{\pm 0.06}$ $(\downarrow 9.29)$	$65.58_{\pm 0.07}$ $(\downarrow 9.35)$	$66.18_{\pm 0.08}$ $(\downarrow 8.75)$	$65.30_{\pm 0.05}$ $(\downarrow 9.63)$
$T=10$	$\textbf{79.41}_{\pm 0.08}$	$65.64_{\pm 0.05}$ $(\downarrow 13.8)$	$65.58_{\pm 0.06}$ $(\downarrow 13.8)$	$66.67_{\pm 0.08}$ $(\downarrow 12.7)$	$65.39_{\pm 0.06}$ $(\downarrow 14.0)$
$T=15$	$\textbf{82.69}_{\pm 0.09}$	$65.64_{\pm 0.05}$ $(\downarrow 17.0)$	$65.58_{\pm 0.06}$ $(\downarrow 17.1)$	$67.11_{\pm 0.11}$ $(\downarrow 15.6)$	$65.51_{\pm 0.04}$ $(\downarrow 17.2)$
$T=20$	$\textbf{85.15}_{\pm 0.03}$	$65.64_{\pm 0.06}$ $(\downarrow 19.5)$	$65.58_{\pm 0.07}$ $(\downarrow 19.6)$	$67.52_{\pm 0.11}$ $(\downarrow 17.6)$	$65.60_{\pm 0.07}$ $(\downarrow 19.6)$
$T=25$	$\textbf{87.21}_{\pm 0.02}$	$65.64_{\pm 0.05}$ $(\downarrow 21.6)$	$65.57_{\pm 0.07}$ $(\downarrow 21.6)$	$67.86_{\pm 0.13}$ $(\downarrow 19.4)$	$65.70_{\pm 0.12}$ $(\downarrow 21.5)$

Appendix

Table 20: Training accuracy comparison of our DeepAFL against the other four deepening strategies for the analytic networks on the CIFAR-100, including (1) cascades w/ random feature, (2) cascades w/ random feature & label encoding, (3) cascades w/ activated random feature, (4) cascades w/ activated random feature & label encoding. All the strategies share the same zero-layer features to ensure fairness. The value in parentheses

(\downarrow)

indicates the performance lags behind our DeepAFL.

Layers	DeepAFL	Strategy (1)	Strategy (2)	Strategy (3)	Strategy (4)
$T=0$	$\textbf{60.81}_{\pm 0.15}$	$60.81_{\pm 0.15}$ $(\downarrow 0.00)$	$60.81_{\pm 0.15}$ $(\downarrow 0.00)$	$60.81_{\pm 0.15}$ $(\downarrow 0.00)$	$60.81_{\pm 0.15}$ $(\downarrow 0.00)$
$T=5$	$\textbf{64.62}_{\pm 0.02}$	$60.80_{\pm 0.13}$ $(\downarrow 3.82)$	$60.75_{\pm 0.14}$ $(\downarrow 3.87)$	$61.23_{\pm 0.07}$ $(\downarrow 3.39)$	$60.57_{\pm 0.04}$ $(\downarrow 4.05)$
$T=10$	$\textbf{65.98}_{\pm 0.13}$	$60.78_{\pm 0.10}$ $(\downarrow 5.20)$	$60.74_{\pm 0.11}$ $(\downarrow 5.24)$	$61.67_{\pm 0.09}$ $(\downarrow 4.31)$	$60.69_{\pm 0.05}$ $(\downarrow 5.29)$
$T=15$	$\textbf{66.59}_{\pm 0.03}$	$60.78_{\pm 0.12}$ $(\downarrow 5.81)$	$60.74_{\pm 0.11}$ $(\downarrow 5.85)$	$61.98_{\pm 0.03}$ $(\downarrow 4.61)$	$60.85_{\pm 0.02}$ $(\downarrow 5.74)$
$T=20$	$\textbf{66.95}_{\pm 0.04}$	$60.79_{\pm 0.10}$ $(\downarrow 6.16)$	$60.73_{\pm 0.11}$ $(\downarrow 6.22)$	$62.21_{\pm 0.02}$ $(\downarrow 4.74)$	$60.95_{\pm 0.07}$ $(\downarrow 6.00)$
$T=25$	$\textbf{67.40}_{\pm 0.10}$	$60.78_{\pm 0.10}$ $(\downarrow 6.62)$	$60.72_{\pm 0.11}$ $(\downarrow 6.68)$	$62.31_{\pm 0.04}$ $(\downarrow 5.09)$	$60.99_{\pm 0.06}$ $(\downarrow 6.41)$

Appendix

Table 21: Testing accuracy comparison of our DeepAFL against the other four deepening strategies for the analytic networks on the CIFAR-100, including (1) cascades w/ random feature, (2) cascades w/ random feature & label encoding, (3) cascades w/ activated random feature, (4) cascades w/ activated random feature & label encoding. All the strategies share the same zero-layer features to ensure fairness. The value in parentheses

(\downarrow)

indicates the performance lags behind our DeepAFL.

Layers	DeepAFL	Strategy (1)	Strategy (2)	Strategy (3)	Strategy (4)
$T=0$	$\textbf{60.30}_{\pm 0.02}$	$60.30_{\pm 0.02}$ $(\downarrow 0.00)$	$60.30_{\pm 0.02}$ $(\downarrow 0.00)$	$60.30_{\pm 0.02}$ $(\downarrow 0.00)$	$60.30_{\pm 0.02}$ $(\downarrow 0.00)$
$T=5$	$\textbf{66.89}_{\pm 0.06}$	$60.20_{\pm 0.02}$ $(\downarrow 6.69)$	$59.84_{\pm 0.02}$ $(\downarrow 7.05)$	$59.92_{\pm 0.03}$ $(\downarrow 6.97)$	$56.14_{\pm 0.08}$ $(\downarrow 10.8)$
$T=10$	$\textbf{70.14}_{\pm 0.03}$	$60.20_{\pm 0.01}$ $(\downarrow 9.94)$	$59.83_{\pm 0.01}$ $(\downarrow 10.3)$	$60.00_{\pm 0.02}$ $(\downarrow 10.1)$	$54.78_{\pm 0.07}$ $(\downarrow 15.4)$
$T=15$	$\textbf{72.64}_{\pm 0.05}$	$60.21_{\pm 0.01}$ $(\downarrow 12.4)$	$59.82_{\pm 0.04}$ $(\downarrow 12.8)$	$60.08_{\pm 0.03}$ $(\downarrow 12.6)$	$54.40_{\pm 0.07}$ $(\downarrow 18.2)$
$T=20$	$\textbf{74.75}_{\pm 0.02}$	$60.20_{\pm 0.01}$ $(\downarrow 14.5)$	$59.81_{\pm 0.01}$ $(\downarrow 14.9)$	$60.17_{\pm 0.04}$ $(\downarrow 14.6)$	$54.23_{\pm 0.11}$ $(\downarrow 20.5)$
$T=25$	$\textbf{76.58}_{\pm 0.03}$	$60.19_{\pm 0.01}$ $(\downarrow 16.4)$	$59.80_{\pm 0.02}$ $(\downarrow 16.8)$	$60.25_{\pm 0.04}$ $(\downarrow 16.3)$	$54.12_{\pm 0.11}$ $(\downarrow 22.5)$

Appendix

Table 22: Training accuracy comparison of our DeepAFL against the other four deepening strategies for the analytic networks on the Tiny-ImageNet, including (1) cascades w/ random feature, (2) cascades w/ random feature & label encoding, (3) cascades w/ activated random feature, (4) cascades w/ activated random feature & label encoding. All the strategies share the same zero-layer features to ensure fairness. The value in parentheses

(\downarrow)

indicates the performance lags behind our DeepAFL.

Layers	DeepAFL	Strategy (1)	Strategy (2)	Strategy (3)	Strategy (4)
$T=0$	$\textbf{56.72}_{\pm 0.14}$	$56.72_{\pm 0.14}$ $(\downarrow 0.00)$	$56.72_{\pm 0.14}$ $(\downarrow 0.00)$	$56.72_{\pm 0.14}$ $(\downarrow 0.00)$	$56.72_{\pm 0.14}$ $(\downarrow 0.00)$
$T=5$	$\textbf{60.24}_{\pm 0.02}$	$56.69_{\pm 0.12}$ $(\downarrow 3.55)$	$56.32_{\pm 0.10}$ $(\downarrow 3.92)$	$56.31_{\pm 0.10}$ $(\downarrow 3.93)$	$53.26_{\pm 0.12}$ $(\downarrow 6.98)$
$T=10$	$\textbf{61.34}_{\pm 0.13}$	$56.61_{\pm 0.13}$ $(\downarrow 4.73)$	$56.26_{\pm 0.11}$ $(\downarrow 5.08)$	$56.40_{\pm 0.06}$ $(\downarrow 4.94)$	$52.01_{\pm 0.06}$ $(\downarrow 9.33)$
$T=15$	$\textbf{61.94}_{\pm 0.08}$	$56.62_{\pm 0.14}$ $(\downarrow 5.32)$	$56.25_{\pm 0.13}$ $(\downarrow 5.69)$	$56.49_{\pm 0.10}$ $(\downarrow 5.45)$	$51.74_{\pm 0.12}$ $(\downarrow 10.2)$
$T=20$	$\textbf{62.34}_{\pm 0.02}$	$56.64_{\pm 0.14}$ $(\downarrow 5.70)$	$56.23_{\pm 0.08}$ $(\downarrow 6.11)$	$56.61_{\pm 0.10}$ $(\downarrow 5.73)$	$51.63_{\pm 0.09}$ $(\downarrow 10.7)$
$T=25$	$\textbf{62.66}_{\pm 0.05}$	$56.62_{\pm 0.14}$ $(\downarrow 6.04)$	$56.27_{\pm 0.09}$ $(\downarrow 6.39)$	$56.66_{\pm 0.12}$ $(\downarrow 6.00)$	$51.66_{\pm 0.16}$ $(\downarrow 11.0)$

Appendix

Table 23: Testing accuracy comparison of our DeepAFL against the other four deepening strategies for the analytic networks on the Tiny-ImageNet, including (1) cascades w/ random feature, (2) cascades w/ random feature & label encoding, (3) cascades w/ activated random feature, (4) cascades w/ activated random feature & label encoding. All the strategies share the same zero-layer features to ensure fairness. The value in parentheses

(\downarrow)

indicates the performance lags behind our DeepAFL.

Panel of Figure 4 — Figure 4: Accuracy-Efficiency balance of our DeepAFL on the CIFAR-10.

Panel of Figure 9 — Figure 9: Ablation study of our DeepAFL on the CIFAR-100.

Figure 14: The training process for each layer in our DeepAFL.

Figure 15: Layer-wise reduction of empirical risk in our DeepAFL.

Figure 16: The sandwiched least-squares process in our DeepAFL.

Baseline	CIFAR-10		CIFAR-100		Tiny-ImageNet
Baseline	$\alpha=0.1$	$\alpha=0.05$	$\alpha=0.1$	$\alpha=0.1$	$\alpha=0.1$	$\alpha=0.1$
FedAWA ( 2025 )	79.42%	76.37%	58.94%	50.48%	59.07%	52.33%
FedLPA ( 2024a )	49.57%	42.56%	40.04%	22.55%	39.48%	26.67%
AFL ( 2025 )	80.75%	80.75%	58.56%	58.56%	54.67%	54.67%
DeepAFL $(T=5)$	85.20%	85.20%	64.72%	64.72%	60.31%	60.31%
DeepAFL $(T=10)$	85.93%	85.93%	65.96%	65.96%	61.37%	61.37%

Appendix

Table 24: Performance comparisons of the top-1 accuracy (%) among our DeepAFL and two new baselines (i.e., FedAWA ( Shi et al., 2025 ) and ( Liu et al., 2024a ) ) on CIFAR-10, CIFAR-100, and Tiny-ImageNet. The best result is highlighted in bold , and the second-best result is underlined .

Layers	$T=1$	$T=2$	$T=5$	$T=10$	$T=20$
FedAvg	56.62%	55.83%	$<$ 5%	$<$ 3%	$<$ 1%
FedDyn	57.55%	57.22%	56.26%	$<$ 3%	$<$ 1%

Appendix

Table 25: Accuracy of gradient-based baselines with varying layers on CIFAR-100.

Layers	Testing Acc	$\Delta_{\text{Testing Acc}}$	$\Delta_{\text{AFL Acc}}$	Time Cost	$\Delta_{\text{Time Cost}}$	$\Delta_{\text{AFL Cost}}$
AFL	75.45%	$/$	$/$	107.19s	$/$	$/$
DeepAFL ( $T=5$ )	78.05%	2.60%	2.60%	118.97s	11.78s	11.78s
DeepAFL ( $T=10$ )	79.01%	0.96%	3.56%	126.55s	7.58s	19.37s
DeepAFL ( $T=15$ )	79.73%	0.72%	4.28%	133.79s	7.24s	26.60s
DeepAFL ( $T=20$ )	80.13%	0.40%	4.68%	141.02s	7.23s	33.83s
DeepAFL ( $T=25$ )	80.33%	0.20%	4.88%	148.56s	7.54s	41.37s

Appendix

Table 26: Analyses of DeepAFL’s scalability with the ViT-B-16-I-1K backbone on CIFAR-100.

Layers	Testing Acc	$\Delta_{\text{Testing Acc}}$	$\Delta_{\text{AFL Acc}}$	Time Cost	$\Delta_{\text{Time Cost}}$	$\Delta_{\text{AFL Time}}$
AFL	86.35%	$/$	$/$	107.19s	$/$	$/$
DeepAFL ( $T=5$ )	87.71%	1.36%	1.36%	118.97s	11.78s	11.78s
DeepAFL ( $T=10$ )	88.15%	0.44%	1.80%	126.55s	7.58s	19.37s
DeepAFL ( $T=15$ )	88.49%	0.34%	2.14%	133.79s	7.24s	26.60s
DeepAFL ( $T=20$ )	88.80%	0.31%	2.45%	141.02s	7.23s	33.83s
DeepAFL ( $T=25$ )	88.95%	0.15%	2.60%	148.56s	7.54s	41.37s

Appendix

Table 27: Analyses of DeepAFL’s scalability with the ViT-B-16-I-21K backbone on CIFAR-100.

Layers	Consistent Participation						Inconsistent Participation
Layers	$\eta=100\%$	$\eta=90\%$	$\eta=80\%$	$\eta=70\%$	$\eta=60\%$	$\eta=50\%$	$\eta=100\%$	$\eta=90\%$	$\eta=80\%$	$\eta=70\%$	$\eta=60\%$	$\eta=50\%$
$T=5$	64.72%	64.53%	64.38%	63.92%	62.85%	61.54%	64.72%	64.21%	64.03%	64.05%	62.19%	61.36%
$T=10$	65.96%	65.46%	65.26%	64.49%	64.76%	63.17%	65.96%	65.36%	65.05%	64.45%	64.09%	62.93%
$T=15$	66.56%	66.39%	66.20%	65.38%	65.17%	65.08%	66.56%	66.22%	66.17%	65.64%	64.60%	64.98%
$T=20$	66.98%	66.76%	66.66%	66.47%	66.27%	65.14%	66.98%	66.57%	66.51%	66.32%	65.50%	64.81%

Appendix

Table 28: Performance evaluation of DeepAFL on CIFAR-100 under partial client participation.

Layers	$\tau=0\%$	$\tau=10\%$	$\tau=20\%$	$\tau=30\%$	$\tau=40\%$	$\tau=50\%$
DeepAFL ( $T=5$ )	64.72%	64.20%	63.63%	63.25%	62.51%	60.91%
DeepAFL ( $T=10$ )	65.96%	65.43%	65.09%	63.96%	62.98%	60.99%
DeepAFL ( $T=15$ )	66.56%	66.11%	65.66%	64.29%	63.15%	60.97%
DeepAFL ( $T=20$ )	66.98%	66.38%	66.11%	64.69%	63.35%	60.45%

Appendix

Table 29: Performance evaluation of DeepAFL on CIFAR-100 under noisy training environments.

Layer	Training CSR	Testing CSR	Training IFS	Testing IFS	Training DM	Testing DM
DeepAFL ( $T=0$ )	1.583	1.529	3.25	3.13	3.21	3.09
DeepAFL ( $T=5$ )	1.565	1.520	3.20	3.11	3.16	3.07
DeepAFL ( $T=10$ )	1.555	1.516	3.17	3.10	3.14	3.06
DeepAFL ( $T=15$ )	1.546	1.513	3.15	3.09	3.12	3.05
DeepAFL ( $T=20$ )	1.539	1.510	3.13	3.08	3.11	3.04

Appendix

Table 30: Separability analyses of DeepAFL’s intermediate features on CIFAR-100.

Gradient-Free Optimization

See all 14 papers

Jul 9, 2026

FedOPAL: One-Shot Federated Learning via Analytic Visual Prompt Tuning

With the widespread deployment of basic models in edge intelligence, communication bandwidth has become a core bottleneck restricting the scalability of federated learning. Although one-shot federated learning alleviates this problem by minimizing communication rounds, existing iterative fine-tuning or knowledge distillation methods still face challenges such as high server-side computational costs and hyperparameter sensitivity. Analytical federated learning achieves efficient gradientfree aggregation using least-squares closed-form solutions, but in environments with non-independent and identically distributed data, its static feature assumptions fail, leading to feature manifold misalignment and severely impairing model performance. To address this contradiction, this paper proposes the FedOPAL framework. This framework adapts the visual prompts as feature rectifiers, actively correcting the feature distribution of heterogeneous data to a linearly separable space by applying local proximal constraints, thereby satisfying the theoretical assumptions of analytical federated learning. Experimental results show that FedOPAL not only significantly outperforms the original analytical methods on several benchmarks, but also achieves accuracy comparable to state-of-the-art iterative methods while maintaining zero server-side training costs, providing a new engineering paradigm for efficient collaboration of large models on the edge.

Lingyu Qiu, Daniela Annunziata, Stefano Izzo +2

Department of Mathematics and Applications “R. Caccioppoli”, University of Naples Federico II, Italy

Federated Learning Aggregation Non-IID Federated Learning Federated Learning

Ask PDF

Aug 10, 2026

FedA2L: Adaptive layer-wise learning rate adjustment in decentralized federated learning

Decentralized intelligence systems with heterogeneous devices and limited coordination increasingly rely on decentralized federated learning (DFL). However, DFL suffers from convergence inefficiency under data heterogeneity due to the use of a uniform learning rate (LR) that ignores layer-specific optimization needs. Foundational layers are responsible for maintaining network consensus, while specialized layers adapt to local data characteristics, leading to conflicting gradients and degraded performance under non-IID conditions. To address this fundamental tension, this work introduces FedA2L, a method that dynamically adjusts layer-wise LRs based on model divergence signals. By leveraging local update intensity and network consensus constraints, FedA2L seamlessly integrates into existing DFL protocols without additional communication or coordination. Extensive evaluations across DFL algorithms, various model architectures, and datasets demonstrate that FedA2L achieves up to 4.94 times faster convergence than vanilla DFL and reduces communication rounds by up to 59% compared to scheduler-based baselines. Furthermore, FedA2L exhibits resilience to severe data heterogeneity, larger network sizes, and sparse topologies, reducing communication overhead and establishing it as a versatile optimization tool for resource-constrained or large-scale distributed learning in edge and IoT deployments. The code is released at https://github.com/nclabteam/FedA2L.

Van Truong Vo, Khoa Nguyen, Taehong Kim

School of Information and Communication Engineering, Chungbuk National University, Cheongju, South Korea

Non-IID Federated Learning Federated Learning Layer-Wise Learning Rate Adaptation

Ask PDF

Aug 16, 2025

FAIRVAR: Fair Federated Learning via Variance Regularization

Federated learning (FL) allows collaborative training of machine learning models across multiple parties without sharing raw data. However, heterogeneous data can cause some clients to have disproportionate influence on the global model, leading to disparities in their performance. Fairness, understood as reducing these disparities, is therefore a crucial concern in FL and has been addressed in various ways. We studied performance equitable fairness in FL, where the goal is to minimize performance disparities across clients. We evaluated several existing fairness-aware methods and introduce here a new gradient-variance-regularized method, implemented in two variants: FairGrad (approximate) and FairGrad* (exact). We theoretically characterize the connections between these methods and, empirically, on heterogeneous benchmarks, show that FairGrad and FairGrad* consistently improve fairness by reducing variance in client accuracies, while maintaining competitive or improved mean performance compared to existing fairness-aware baselines.

Zahra Kharaghani, Ali Dadras, Tommy Löfstedt

Department of Computing Science, Umeå University, Umeå, Sweden · Department of Mathematics and Mathematical Statistics, Umeå University, Umeå, Sweden

Algorithmic Fairness Federated Learning Heterogeneous Federated Learning

Ask PDF

Abstract

Gradient-Free Optimization

Explore similar work

FedOPAL: One-Shot Federated Learning via Analytic Visual Prompt Tuning

FedA2L: Adaptive layer-wise learning rate adjustment in decentralized federated learning

FAIRVAR: Fair Federated Learning via Variance Regularization