Large language models struggle to maintain strict adherence to structured workflows in high-stakes domains such as healthcare triage. Monolithic approaches that encode entire decision structures within a single prompt are prone to instruction-following degradation as prompt length increases, including lost-in-the-middle effects and context window overflow. To address this gap, we present Arbor, a framework that decomposes decision tree navigation into specialized, node-level tasks. Decision trees are standardized into an edge-list representation and stored for dynamic retrieval. At runtime, a directed acyclic graph (DAG)-based orchestration mechanism iteratively retrieves only the outgoing edges of the current node, evaluates valid transitions via a dedicated LLM call, and delegates response generation to a separate inference step. The framework is agnostic to the underlying decision logic and model provider. Evaluated against single-prompt baselines across 10 foundation models using annotated turns from real clinical triage conversations. Arbor improves mean turn accuracy by 29.4 percentage points, reduces per-turn latency by 57.1%, and achieves an average 13.8x reduction in per-turn cost. These results indicate that architectural decomposition reduces dependence on intrinsic model capability, enabling smaller models to match or exceed larger models operating under single-prompt baselines.
Figures & tables
Figure 1: Overview of the proposed Arbor architecture, consisting of decision tree processing (shown at the bottom), an evaluation phase (top left), and a message generation phase (top right). The processing phase normalizes raw sources into an edge-list database, which enables the evaluation phase to dynamically retrieve outgoing edges. The evaluation phase then evaluates transitions via iterative LLM calls and updates the current node until no further transitions are taken. The message generation phase selects the appropriate prompt and produces the user-facing response.
Transitions
Transition Key
Transition unique ID
Node from key
Source node – the identifier of the originating node
Node to key
Target node – where the flow goes if answer matches
Question
The condition or prompt shown at this edge
Answer
The answer/condition required to follow transition
Extra context
Auxiliary text or business logic
Table 1: Edge list document.
Figure 2: Turn Accuracy. Bars show mean turn accuracy over five runs, error bars indicate the standard deviation.
Figure 3: Latency per Turn. Bars show mean response latency (in seconds) over five runs. The y-axis is in log scale. Absolute latencies should not be compared across models, as they reflect different deployment and inference infrastructures. However, they remain comparable within each model when contrasting Baseline and Arbor.
Figure 4: Cost per Turn. Bars show the average cost per conversational turn in US Dollars ($). The y-axis is in log scale.
Metric
Arbor (Mean ± SD)
Single-prompt (Mean ± SD)
Performance Delta
Turn Accuracy
88.23% (± 7.66%)
58.80% (± 22.59%)
+29.42 points
Cost per Turn
0.012(±0.011)
0.166(±0.125)
13.8x Cheaper
Latency per Turn
14.51s (± 8.82s)
33.84s (± 27.23s)
57.1% Faster
Table 2: Aggregate performance comparison.
Figure 5: Quality score distribution. Bars show the frequency count of assigned quality scores (1-4) for accepted messages, grouped by strategy.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Arbor transition evaluator system message (excerpt).
Figure 7: Arbor message generation system message (excerpt).
Figure 8: Single-prompt baseline system message (excerpt). The full decision tree is omitted for brevity.
Model
Input ($/1M tokens)
Output ($/1M tokens)
Proprietary models
GPT-5 (minimal / medium / high)
$1.25
$10.00
GPT-4.1
$2.00
$8.00
Claude Sonnet 4.5
$3.00
$15.00
Gemini 3 Pro
$2.00
$12.00
Gemini 3 Flash
$0.50
$3.00
Appendix
Table 3: Per-token rates assumed for cost calculations. Proprietary model prices follow standard public pricing at the time of experimentation. Open-weights model costs are normalized inference costs under a fixed deployment configuration.
Model
Strategy
Turn Accuracy
Turn Accuracy STD
GPT-5-minimal
Arbor
92.07%
1.7%
GPT-5-minimal
Baseline
62.76%
2.0%
Claude Sonnet 4.5
Arbor
91.15%
2.1%
Claude Sonnet 4.5
Baseline
43.33%
3.8%
GPT-5-medium
Arbor
90.80%
0.8%
GPT-5-medium
Baseline
80.57%
2.6%
Appendix
Table 4: Turn accuracy results used in Figure 2 .
Model
Strategy
Latency per Turn (Mean) (s)
Latency per Turn (Median) (s)
GPT-4.1
Arbor
5.08
4.97
GPT-4.1
Baseline
14.08
9.74
DeepSeek V3.1
Arbor
5.90
4.24
DeepSeek V3.1
Baseline
81.48
77.92
GPT-5-minimal
Arbor
8.40
7.07
GPT-5-minimal
Baseline
10.74
9.82
Appendix
Table 5: Latency results used in Figure 3 .
Model
Strategy
Cost per Turn (USD)
Gemini 3 Flash
Arbor
$0.0025
Gemini 3 Flash
Baseline
$0.0693
DeepSeek V3.1
Arbor
$0.00025
DeepSeek V3.1
Baseline
$0.0756
GPT-5-minimal
Arbor
$0.0078
GPT-5-minimal
Baseline
$0.1545
Appendix
Table 6: Cost results used in Figure 4 .
Strategy
Score 1
Score 2
Score 3
Score 4
Mean
STD
Arbor
1
5
35
104
3.67
0.58
Baseline
0
5
47
98
3.62
0.55
Appendix
Table 7: Quality score distribution used in Figure 5 with calculated mean and standard deviation.
Large language models (LLMs) are increasingly deployed for extended, multi-topic conversations, yet the flat, append-only structure of current conversation interfaces introduces a fundamental limitation: all context accumulates in a single unbounded window, causing topically distinct threads to bleed into one another and progressively degrade response quality. We term this failure mode logical context poisoning. In this paper, we introduce the Conversation Tree Architecture (CTA), a hierarchical framework that organizes LLM conversations as trees of discrete, context-isolated nodes. Each node maintains its own local context window; structured mechanisms govern how context flows between parent and child nodes, downstream on branch creation and upstream on branch deletion. We additionally introduce volatile nodes, transient branches whose local context must be selectively merged upward or permanently discarded before purging. We formalize the architecture's primitives, characterize the open design problems in context flow, relate our framework to prior work in LLM memory management, and describe a working prototype implementation. The CTA provides a principled foundation for structured conversational context management and extends naturally to multi-agent settings.
Agent orchestration frameworks -- LangGraph, CrewAI, Google ADK, OpenAI Agents SDK, and others -- place an external orchestrator above the LLM, tracking state and injecting routing instructions at every turn. We present a controlled comparison showing that for procedural tasks, this architecture is dominated by a simpler alternative: putting the entire procedure in the system prompt and letting the model self-orchestrate. Across three domains -- travel booking (14 nodes), Zoom technical support (14 nodes), and insurance claims processing (55 nodes) -- we evaluate 200 conversations per condition using LLM-as-judge scoring on five quality criteria. The in-context approach scores 4.53--5.00 on a 5-point scale while a LangGraph orchestrator using the same model scores 4.17--4.84. The orchestrated system fails on 24% of travel, 9% of Zoom, and 17% of insurance conversations, compared to 11.5%, 0.5%, and 5% for the in-context baseline. While external orchestration may have been necessary for earlier models, advances in frontier model capabilities have made it unnecessary for multi-turn conversations following a defined procedure.
Large Language Models (LLMs) are increasingly deployed in multi-turn dialogue settings where preserving conversational context across turns is essential. A standard serving practice concatenates the full dialogue history at every turn, which reliably maintains coherence but incurs substantial cost in latency, memory, and API expenditure, especially when queries are routed to large proprietary models. Existing approaches often struggle to balance the trade-off between response quality and efficiency. We propose a framework that exploits the early turns of a session to estimate a local response manifold and then adapt a smaller surrogate model to this local region for the remainder of the conversation. Concretely, we learn soft prompts that maximize semantic divergence between the large and surrogate small language models' responses to surface least-aligned local directions, stabilize training with anti-degeneration control, and distill the mined cases into localized LoRA fine-tuning so the surrogate runs without prompts at inference. A simple gate enables a one-time switch with rollback on drift. We further provide a theoretical analysis for key components in SOMA. Extensive experiments show the effectiveness of SOMA. The source code is provided at: https://github.com/LabRAI/SOMA.
Xueqi Cheng, Qiong Wu, Zhengyi Zhou +3
Florida State University · AT&T Chief Data Office · Louisiana State University +1