Large language models struggle to maintain strict adherence to structured workflows in high-stakes domains such as healthcare triage. Monolithic approaches that encode entire decision structures within a single prompt are prone to instruction-following degradation as prompt length increases, including lost-in-the-middle effects and context window overflow. To address this gap, we present Arbor, a framework that decomposes decision tree navigation into specialized, node-level tasks. Decision trees are standardized into an edge-list representation and stored for dynamic retrieval. At runtime, a directed acyclic graph (DAG)-based orchestration mechanism iteratively retrieves only the outgoing edges of the current node, evaluates valid transitions via a dedicated LLM call, and delegates response generation to a separate inference step. The framework is agnostic to the underlying decision logic and model provider. Evaluated against single-prompt baselines across 10 foundation models using annotated turns from real clinical triage conversations. Arbor improves mean turn accuracy by 29.4 percentage points, reduces per-turn latency by 57.1%, and achieves an average 13.8x reduction in per-turn cost. These results indicate that architectural decomposition reduces dependence on intrinsic model capability, enabling smaller models to match or exceed larger models operating under single-prompt baselines.
Figures & tables
Figure 1: Overview of the proposed Arbor architecture, consisting of decision tree processing (shown at the bottom), an evaluation phase (top left), and a message generation phase (top right). The processing phase normalizes raw sources into an edge-list database, which enables the evaluation phase to dynamically retrieve outgoing edges. The evaluation phase then evaluates transitions via iterative LLM calls and updates the current node until no further transitions are taken. The message generation phase selects the appropriate prompt and produces the user-facing response.
Transitions
Transition Key
Transition unique ID
Node from key
Source node – the identifier of the originating node
Node to key
Target node – where the flow goes if answer matches
Question
The condition or prompt shown at this edge
Answer
The answer/condition required to follow transition
Extra context
Auxiliary text or business logic
Table 1: Edge list document.
Figure 2: Turn Accuracy. Bars show mean turn accuracy over five runs, error bars indicate the standard deviation.
Figure 3: Latency per Turn. Bars show mean response latency (in seconds) over five runs. The y-axis is in log scale. Absolute latencies should not be compared across models, as they reflect different deployment and inference infrastructures. However, they remain comparable within each model when contrasting Baseline and Arbor.
Figure 4: Cost per Turn. Bars show the average cost per conversational turn in US Dollars ($). The y-axis is in log scale.
Metric
Arbor (Mean ± SD)
Single-prompt (Mean ± SD)
Performance Delta
Turn Accuracy
88.23% (± 7.66%)
58.80% (± 22.59%)
+29.42 points
Cost per Turn
0.012(±0.011)
0.166(±0.125)
13.8x Cheaper
Latency per Turn
14.51s (± 8.82s)
33.84s (± 27.23s)
57.1% Faster
Table 2: Aggregate performance comparison.
Figure 5: Quality score distribution. Bars show the frequency count of assigned quality scores (1-4) for accepted messages, grouped by strategy.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Arbor transition evaluator system message (excerpt).
Figure 7: Arbor message generation system message (excerpt).
Figure 8: Single-prompt baseline system message (excerpt). The full decision tree is omitted for brevity.
Model
Input ($/1M tokens)
Output ($/1M tokens)
Proprietary models
GPT-5 (minimal / medium / high)
$1.25
$10.00
GPT-4.1
$2.00
$8.00
Claude Sonnet 4.5
$3.00
$15.00
Gemini 3 Pro
$2.00
$12.00
Gemini 3 Flash
$0.50
$3.00
Appendix
Table 3: Per-token rates assumed for cost calculations. Proprietary model prices follow standard public pricing at the time of experimentation. Open-weights model costs are normalized inference costs under a fixed deployment configuration.
Model
Strategy
Turn Accuracy
Turn Accuracy STD
GPT-5-minimal
Arbor
92.07%
1.7%
GPT-5-minimal
Baseline
62.76%
2.0%
Claude Sonnet 4.5
Arbor
91.15%
2.1%
Claude Sonnet 4.5
Baseline
43.33%
3.8%
GPT-5-medium
Arbor
90.80%
0.8%
GPT-5-medium
Baseline
80.57%
2.6%
Appendix
Table 4: Turn accuracy results used in Figure 2 .
Model
Strategy
Latency per Turn (Mean) (s)
Latency per Turn (Median) (s)
GPT-4.1
Arbor
5.08
4.97
GPT-4.1
Baseline
14.08
9.74
DeepSeek V3.1
Arbor
5.90
4.24
DeepSeek V3.1
Baseline
81.48
77.92
GPT-5-minimal
Arbor
8.40
7.07
GPT-5-minimal
Baseline
10.74
9.82
Appendix
Table 5: Latency results used in Figure 3 .
Model
Strategy
Cost per Turn (USD)
Gemini 3 Flash
Arbor
$0.0025
Gemini 3 Flash
Baseline
$0.0693
DeepSeek V3.1
Arbor
$0.00025
DeepSeek V3.1
Baseline
$0.0756
GPT-5-minimal
Arbor
$0.0078
GPT-5-minimal
Baseline
$0.1545
Appendix
Table 6: Cost results used in Figure 4 .
Strategy
Score 1
Score 2
Score 3
Score 4
Mean
STD
Arbor
1
5
35
104
3.67
0.58
Baseline
0
5
47
98
3.62
0.55
Appendix
Table 7: Quality score distribution used in Figure 5 with calculated mean and standard deviation.