Significant challenges remain in AI-driven educational systems in balancing privacy preservation with accurate cognitive diagnosis. To overcome this, we propose a federated inference framework in which several commercial LLM APIs collaborate without requiring access to raw student data or proprietary model internals. Using multiple federated entities, such as LLaMA-3.3-70B, GPT-4o-mini, and Claude-3-Haiku, our framework builds upon a heterogeneous multi-LLM architecture. The predictions generated by these entities are combined with epsilon-local differential privacy by adding Laplace noise locally to each entity's prediction output before aggregation, while residual-based aggregation mitigates model heterogeneity. Our approach is predicated on an honest-but-curious trust paradigm in which API providers are presumed not to abuse submitted queries, and our differential privacy mechanism shields the published diagnostic results from external inference. We conduct rigorous privacy-utility analysis showing strong privacy guarantees with minimal accuracy loss, and extensive real-world evaluations across three educational benchmarks confirm the framework's practical usability and cross-domain generalizability.
Table 1: Comparison with related work. ✓ = supported. Het. = Heterogeneous LLMs; BB-API = Black-Box API Access; Pred. = Prediction-level Aggregation.
Dataset
Domain
K
Base
Fed
Improv.
P-Cost
MAE
MAE
ASSIST09
Math
4
0.2410
0.2068
14.19%
0.19%
GSM8K
Word
5
0.2328
0.2156
7.39%
0.27%
UCI
Academic
5
0.1132
0.0969
14.40%
0.94%
Average
–
4.67
0.1956
0.1731
11.99%
0.46%
Table 2: Performance comparison across three educational benchmarks. Our heterogeneous multi-LLM federated framework with ε=2.0 achieves consistent improvements with minimal privacy cost. N=students, K=concepts, P-Cost=privacy cost vs. no privacy.
Configuration
MAE
vs. Baseline
Baseline (Single LLM)
0.2410
–
Fed (No Privacy)
0.2064
+14.35%
Fed ( ε=2.0 )
0.2068
+14.19%
Fed ( ε=1.0 )
0.2232
+7.38%
Fed ( ε=0.5 )
0.2499
− 3.69%
Table 3: Privacy–utility tradeoff on the ASSIST09 dataset. LLaMA-3.3-70B is used as the baseline model due to its superior standalone performance. Lower ε values provide stronger privacy guarantees and typically increase MAE as a result of stronger noise injection. Full multi-baseline comparison across all configurations is provided in Table 4 .
Configuration
ASSIST09
GSM8K
UCI
MAE
MAE
MAE
Full System ( ε =2.0, RC)
0.2068
0.2156
0.0969
w/o Federation (Single LLM)
0.2410
0.2328
0.1132
w/o Privacy (No DP, with RC)
0.2064
0.2150
0.0960
Simple Ensemble (no DP, no RC)
0.2072
0.2161
0.1938
Fed (DP, w/o RC) ( ε =2.0)
0.2999
0.2508
0.2997
Table 4: Ablation study validating federation necessity ( 11.99% avg. contribution), essentiality of residual correction (w/o RC degrades below single-LLM baseline), and minimal privacy cost ( 0.46% avg.) under ε=2.0 .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Overview of the privacy-preserving heterogeneous multi-LLM cognitive diagnosis framework. Student responses and the associated Q-matrix are independently processed by three LLMs (LLaMA-3.3-70B, GPT-4o-mini, Claude-3-Haiku), producing 4D knowledge state vectors. Each entity applies (ε,0) -differential privacy via Laplace noise at the output level before aggregation, preventing the centralized aggregator from inferring individual student responses. Residual-corrected aggregation combines heterogeneous predictions into a global diagnosis, leveraging complementary model behaviors while maintaining strong privacy guarantees.
Model
ASSIST09
GSM8K
UCI
Avg.
LLaMA-3.3-70B
0.2410
0.2328
0.1132
0.1956
GPT-4o-mini
0.2287
0.2295
0.2148
0.2243
Claude-3-Haiku
0.2793
0.2686
0.2159
0.2546
Our Federation
0.2068
0.2156
0.0969
0.1731
Appendix
Table 5: Per-model MAE vs. full federated system ( ε =2.0) on ASSIST09, GSM8K and UCI. Federation outperforms all individual models on three datasets.
Figure 3: Performance comparison across three educational datasets. Our heterogeneous multi-LLM federated framework with ε=2.0 consistently outperforms the single-LLM baseline: ASSIST09 achieves 14.19% improvement (MAE: 0.2068 vs. 0.2410), GSM8K achieves 7.39% improvement (MAE: 0.2156 vs. 0.2328), and UCI achieves 14.40% improvement (MAE: 0.0969 vs. 0.1132). Privacy-free federation achieves 14.35% , 7.64% , and 15.19% on ASSIST09, GSM8K, and UCI respectively, with ε=2.0 incurring only 0.46% average privacy cost. Full multi-baseline comparison across all configurations is provided in Table 4 .
Dataset
Concept
LLaMA
GPT-4
Claude
ASSIST09
Equations
0.2534
0.2170
0.1950
Percentages
0.3053
0.2305
0.3027
Integers
0.3820
0.1989
0.2590
Conversions
0.5157
0.2686
0.3607
GSM8K
Problem_Setup
0.2630
0.3440
0.2840
Arithmetic
0.2850
0.2963
0.2780
Appendix
Table 6: Concept-level MAE per model across all datasets. Bold = best per concept. LLaMA excels at Answer_Verification (0.2140); GPT-4 at Integers (0.1989) and Arithmetic (0.2963); Claude at Equations (0.1950), Multi_Step_Reasoning (0.2294), and Family_Support (0.1022), confirming domain-specific complementarity across model architectures.
Method
Per Student
Per 1K
Scale
Human Expert Annotation
$15–50
$15K–50K
Low
Crowdsourcing (MTurk)
$0.50–2.00
$500–2K
Medium
Single LLM (GPT-4o-mini)
$0.001–0.002
$1–2
High
Our Framework
$0.002–0.005
$2–5
High
LLaMA only (Groq free)
$0.000
$0
Highest
Appendix
Table 7: API cost comparison. Our framework costs 0.002–0.005 per student, substantially lower than human expert annotation ($15–50) ( Leighton and Gierl, 2007 ) .
1National University of Singapore (Chongqing) Research Institute · 2Chongqing Key Laboratory of Trusted Perception and Interaction Technology for Intelligent and Connected Vehicles, Chongqing, China · 3National University of Singapore +3