Cascaded LLM systems coordinate models of varying sizes with human experts to balance accuracy, cost, and abstention under uncertainty. However, single-model tiers at each stage often struggle with ambiguous queries, triggering premature escalations to costlier models or experts due to under-confidence and inefficient compute scaling. CascadeDebate addresses this gap by inserting multi-agent deliberation directly at each tier's escalation boundary. Confidence-based routers activate lightweight agent ensembles only for uncertain cases, enabling consensus-driven resolution of ambiguities internally without invoking higher-cost upgrades. Our unified architecture alternates single-model inference with selective multi-agent deliberation across model scales, culminating in human experts as the final fallback. This design scales test-time compute dynamically according to query difficulty. Across five benchmarks spanning science, medicine, and general knowledge, CascadeDebate outperforms strong single-model cascades and standalone multi-agent systems by up to 26.75 percent. An online threshold optimizer proves essential, boosting accuracy by 20.98 to 52.33 percent relative improvement over fixed policies and enabling elastic adaptation to real-world distributions.
Figures & tables
Figure 1: Adaptive Cascade Framework Overview. Current systems route queries through a hierarchy of solvers: from small models to large models, finally to human experts. High-confidence answers commit immediately at each stage; uncertain cases escalate to balance cost and accuracy.
Figure 2: CascadeDebate Architecture. The unified framework alternates single-model inference with multi-agent deliberation across base and large scales, culminating in human experts as final fallback. Confidence-based routers activate lightweight agent ensembles only for marginal cases at each escalation boundary, resolving ambiguities via intra-tier consensus before deferring to costlier tiers. The online threshold optimizer continuously refines deferral boundaries τ from human feedback, enabling elastic adaptation to real-world query distributions while balancing accuracy against token and expert costs.
Dataset
Baselines
Cascade
Ssingle(Mbase)
Smulti(Mbase)
Ssingle(Mlarge)
Smulti(Mlarge)
Standard
Ours
Llama-3.2 (Base: 1B, Large: 3B)
ARC-Easy
68.56
73.89
89.67
91.00
93.90
95.33 ± 0.703
ARC-Challenge
50.67
54.78
78.33
81.67
84.30
92.89 ± 0.857
MMLU
44.22
47.56
62.44
64.78
67.70
82.67 ± 1.262
MedQA
34.22
35.11
60.89
64.00
68.20
86.44 ± 1.141
Table 1: CascadeDebate vs. Baselines. Accuracy (%) of Llama-3.2-Instruct (1B/3B) and Qwen2.5-Instruct (1.5B/3B) across single-model ( Ssingle(Mbase/large) ), multi-agent ( Smulti(Mbase/large) ), standard cascade ( Ssingle(Mbase)→Ssingle(Mlarge)→Shuman ), and CascadeDebate ( Ssingle(Mbase)→Smulti(Mbase)→Ssingle(Mlarge)→Smulti(Mlarge)→Shuman ) systems. Best in bold .
Figure 3: CascadeDebate Cost-Accuracy Pareto Frontier. Top: Cost-accuracy curves across five benchmarks using Llama-3.2-Instruct model. CascadeDebate (red ⋆ ) dominates the Pareto frontier, matching or exceeding Smulti(Mlarge) accuracy at 20-35% lower token cost. Bottom: Online threshold adaptation. CascadeDebate (solid red) rapidly surpasses fixed-threshold cascade (dashed red) and static baselines as the optimizer calibrates τk to task difficulty over 1,000 samples.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Domain
Split
Choices
Samples
ARC-Easy
Science
Test
3–5
1,000
ARC-Challenge
Science
Test
3–5
1,000
MMLU
General
Test
4
1,000
MedQA
Medical
Validation
5
1,000
MedMCQA
Medical
Validation
4
1,000
Appendix
Table 2: Summary of datasets. We use 1,000 randomly sampled instances from each dataset for consistent evaluation.
Table 3: Domain-specific agent roles used in the multi-agent stages.
Figure 4: Generalizability Analysis using Qwen2.5 Models. The performance trends are consistent with the Llama-3.2 results. Top Row: The Cascade (red star ⋆ ) consistently occupies the optimal position on the Pareto frontier across all benchmarks. Bottom Row: The online threshold learning curve demonstrates rapid adaptation and stability, confirming that our proposed cost-aware optimization is robust regardless of the underlying backbone model.
Dataset
Fixed Threshold
Online Learning
Improvement
ARC-Easy
72.8%
95.3%
+22.5%
ARC-Challenge
76.8%
92.9%
+16.1%
MMLU
59.7%
82.7%
+23.0%
MedQA
59.7%
86.4%
+26.7%
MedMCQA
50.1%
76.3%
+26.2%
Appendix
Table 4: Impact of Online Threshold Optimization. Comparison of accuracy between the Fixed Threshold strategy and the Online Learning strategy.
Figure 5: Performance Analysis of the Fixed Threshold Strategy on Llama-3.2 ( τ=0.6 ). Top Row: Cost-Accuracy trade-off. The Fixed Cascade (red diamond ♢ ) generally improves upon the Single Base model but often falls short of the optimal Pareto frontier compared to the Online strategy. Bottom Row: Stage distribution ratios. Without dynamic adjustment, the system relies on static confidence scores, resulting in a rigid allocation of resources.
Figure 6: Stage-wise Sample Distribution with online learned thresholds for (a) Llama-3.2 and (b) Qwen2.5. Bars show the proportion of queries that terminate at each stage under confidence-based routing. Easier benchmarks are more likely to meet the learned acceptance thresholds in earlier stages and terminate without escalation. Harder benchmarks more frequently fall below early-stage thresholds, triggering deferral to later stages including larger models and the human expert.