Tool-use agents pay per decision: at each hop a language model re-reads the state and thinks for seconds, though its actions are finite and decisions recur. We present NIOCIS, a routing layer that folds execution logs into a loop-collapsed action graph (forward transitions plus recovery back-edges) answering one typed question: which declared option at a state, plus stop, does the fleet take next? A small model reads a calibrated distribution over them in one pass, emitting no text. On 1,615 production security-operations episodes we measure what the layer replaces, returns and is bounded by. Replaces: the typed encoder matches the logged next hop on 32.2% of the 3,875 held-out decisions (34.9% within the declared options) in 10 ms of encoder time (19.5 ms for the typed-choice form), against 2.2 s per LLM hop; two feature readouts reach 32.8% and 32.3%, an ordering the paired test does not resolve, so the typed layer is chosen for cost, not a separated lead. Tuning: jpt-4b reaches 9.9% zero-shot and 31.9% fine-tuned, a gap an exact paired test does not resolve (p=0.68), at 244 ms. Returns little: the realised hop falls outside the declared options for 7.9% of decisions and outside the model's top twenty for 9.3%; the most confident 15% holds 91.9% accuracy. Bounded: a cross-trace leave-one-out test puts the released state's ceiling at 25.6--44.9% as the state key grows more specific, i.e. repeated states disagree more often than they agree; the option-space ablation attributes the gain to per-node candidate restriction (+3.5 points over a flat catalog) rather than graph traversal (zero within its interval); a gated self-evolution loop lifts every arm above its anchor yet ends below full supervision; a confidence-gated cascade loses agreement when the escalation arm is the zero-shot stand-in; and agreement past a third of the fleet needs a state carrying the plan and the observation.
Figures & tables
Figure 1: NIOCIS: logs are folded into a loop-collapsed action graph; a typed-decision router answers “which edge in A(v) does the fleet take next?” per state, escalating only uncertain cases to the LLM; executed walks flow back as gated updates. The step budget B and per-node revisit cap R on the execution box are full-design parameters: the corpus study of § 5 replays logged episodes teacher-forced and exercises neither bound.
Figure 2: The induced action graph G at the level on which every node can be named: a capability-family structure over the released corpus. The 154 catalog actions are grouped into 23 capability families by an ordered capability-keyword rule (a display quotient defined in the released analysis code, not a construct of the method), so the drawing covers all 154 stations and all 15,704 advance hops with no anonymous station. The families are placed on one axis in the order that minimises weighted arc span over the family pairs (51,717 → 43,864); solid arcs run with that order, warm dashed arcs against it, thickness proportional to transferred hops, and a hook on a node carries the hops that never leave its family. Node size is episode count and colour the dominant domain slice — a deliberately honest non-finding: 22 of the 23 families are detection-dominant, so colour separates one node. Two properties of G that a station-level view hides become visible content. First, the hunting loop is two-directional ( SIEM alerts↔event lists 550/465, identity↔threat-intel 178/170, host↔threat-intel 171/142). Second, 3,709 of the 14,089 advance transitions (26.3%; the 15,704 advance hops of § 3 less one opening step per episode—1,615 episodes) never leave their own family, 1,271 of them in the SIEM family alone. The 58 drawn arcs are the 50 heaviest family-to-family exchanges plus one arc for each family the budget would otherwise leave isolated, and they carry 64.2% of cross-family mass; the remaining share is counted, never implied absent, in the table, which lists every family with its station count, episodes and out/in/intra-family mass. close-out (ringed) is the graph’s only terminal group: every hunting line is resolved there.
Figure 3: Node-vocabulary convergence of the loop-collapsed action graph on real production traces (subsample quantities: lines = medians of 20 without-replacement draws at the shown size; shading = min–max over those draws, for the node and distinct-pair series of (a) only—back-edge pairs show their median lines). (a) Semantic node counts saturate quickly (77 action labels in 50-episode subsamples → 124 at 400 → 154 at the full 1,615), while observed transition volume (kept-frame pairs, E+∪E− ) keeps a long tail (distinct pairs 336 at 50 episodes → 1,267 at 400 → 2,027 at 1,000 → 2,536 at 1,615, of which 702 appear at recovery returns). (b) Loop-event rates: fold events settle at 29.2% of executed steps (4,124 retry + 2,364 back) and back returns run 146 per 100 episodes—the right-axis back-event series in (b) counts these return events, 2,364 in full, not (a)’s 702 distinct pairs. Axes, hues and y-axis labels are figure-local across the paper’s figures (see Figure 7 ’s caption).
split
episodes
completed (%)
decision steps
loop events
train
1,125
67.3
10,929
4,615
val
145
60.0
1,394
519
test
345
65.5
3,381
1,354
all
1,615
66.3
15,704
6,488
Table 1: Real SOC corpus after loop folding (§ 4 ): 22,192 tool calls grouped into 1,615 episodes over 154 actions. “decision steps” count advance (hop) steps; the router’s decision states add the kept recovery returns (train/val/test + 1,684/ + 186/ + 494 = 2,364), giving the 18,068 rows ( 12,613+1,580+3,875 ) all SOC experiments draw on. “loop events” count executed retry and back fold events (4,124 retries, which leave the decision stream, plus 2,364 returns, which stay). “completed” is the derived close-out outcome (§ 5 ). The self-evolution stream and its rollback gate for E4 are carved from the validation fold.
readout arm
acc. (%)
in-space (%)
ECE-15
AURC
± std / config
majority arm
26.8
–
–
–
–
logistic regression †
32.3
–
–
–
–
feature MLP, CE †
32.8
35.6
0.045
0.409
±0.001
feature MLP, CE + Brier †
32.7
35.5
0.048
0.409
±0.003
edge-ID readout, CE †
31.4
34.1
0.082
0.432
±0.007
edge-ID readout, CE + Brier †
31.6
34.3
0.080
0.431
±0.004
Table 2: Next-hop agreement on the held-out test decisions: top-1 accuracy over 3,875 decision states; for the trained readout arms, masked top-1 renormalized over each state’s declared options; raw top-1 for the majority and logistic rows (option-masked accuracy is computed for both baseline rows on the public streams and for the majority arm on all nine streams, App. B; it equals raw for the majority arm everywhere, while the logistic pair diverges on the two ToolBench streams Qin et al., 2023 : 35.7% masked versus 36.4% raw at 7,318 names and 34.8% masked versus 36.0% raw on the 351-name evaluation subset, § 6 ). Uncertainty: 5 training seeds for the readout arms, ± std—accuracy fractions, e.g. 0.004 —in the note column. “in-space” restricts to the 92.1% of decisions whose logged target lies inside the declared option space. ECE-15 and AURC are probabilities printed to three decimals. “–” = undefined (the majority and logistic rows carry no calibration cells, no in-space cells, and no AURC entry: the majority arm has no confidence ranking and no calibrated distribution is fitted for the logistic row; AURC is likewise not computed for the probe and zero-shot rows—their risk–coverage curves come only from coarse 7-point grids); note-column “–” = no seed spread and no config entry (the zero-shot row reports its fitted decoding temperature T instead). Probe and zero-shot fitting details are in App. B. † The paired interval against the indicated reference includes zero ( Table 3 ): the ordering among the marked rows is a point estimate, not a separated difference.
paired comparison (same states)
Δ (pts)
95% clustered CI
discordant
reading
feature MLP, CE + Brier − CE
−0.13
[−0.53,+0.26]
35 / 30
point estimate only
feature MLP, CE + Brier − majority arm
+5.70
[+4.60,+6.90]
105 / 326
separates
feature MLP, CE + Brier − logistic regression
+0.23
[−0.48,+0.93]
101 / 110
point estimate only
edge-ID − class space (features, CE + Brier)
−0.83
[−1.67,+0.03]
176 / 144
point estimate only
edge-ID − class space (probe, CE + Brier)
−3.79
[−5.01,−2.62]
212 / 359
separates
edge-ID CE + Brier − edge-ID logistic
−0.15
[−1.05,+0.76]
176 / 170
point estimate only
Table 3: Paired tests on the same held-out states (seed-0 fits; e10_paired_ci.json , e11_probe_edge_ci.json ). Δ is in accuracy points with a 95% CI from 2,000 trace-clustered bootstrap resamples; “discordant” gives the counts (first arm right / second arm right). Rows read point estimate only have a CI that includes zero: their ordering in Table 2 is a point estimate, not a separated difference.
bucket ( ∣A(v)∣ of the state)
n
share
mean ∣A(v)∣
majority
MLP CE + Brier
edge-ID
≤3
263
6.8%
1.23
92.0
93.9 (99.2)
93.2
4 – 10
139
3.6%
7.90
34.5
38.9 (54.0)
36.7
11+
3,473
89.6%
44.46
21.5
27.6 (29.8)
26.8
Table 4: The same decisions split by the branch factor ∣A(v)∣ at the state (3,875 test decisions). Only 3.6% of states fall in the middle bucket, so the headline is dominated by the 11+ bucket (89.6% of rows). “majority” is the per-node majority-vote arm; MLP and edge-ID are the feature arms, masked accuracy with in-space accuracy in parentheses; mean ∣A(v)∣ is over declared options ( stop included). e8_slices.json additionally carries a pooled train + val + test by_outdeg table with the same accuracy magnitudes over 19,375 rows; the counts and accuracies quoted here are test-split, and our seed-0 probe refit reproduces them (92.0/22.3/26.3 against the five-seed 92.2/21.2/25.9 of § 6 ).
Figure 4: (a) Readout arms on the held-out test decisions (masked top-1; raw top-1 for the majority/logistic rows; the zero-shot row counts out-of-space targets as errors, as the masked rows and the majority arm do). (b) Risk–coverage of the CE-plus-Brier arms (masked-renormalized distributions; the edge-ID probe arm is not drawn): confidence separates risk sharply at the top of the ranking in the feature arms and thinly across the bulk in both families (the probe grid starts at 50% coverage). Dense curves live in real/e2_results.json ( curve_mean ; seed 0 for the feature arms); the probe arm is drawn from its seven-point, five-seed-mean risk@coverage grid in real/e2_layaprobe_results.json .
policy
match (%)
survival (%)
gap c1
gap c3
LLM calls/dec. (%)
MIMO Pro V2.6 every hop
9.6
0.0
10.157
30.470
100.0
hidden-state probe, CE+Brier
30.7
0.6
7.788
23.365
0.0
typed+esc @ τ=0.55
20.9
0.0
8.890
26.670
74.1
typed+esc @ τ=0.75
19.1
0.0
9.084
27.252
80.2
typed+esc @ τ=0.9
16.5
0.0
9.377
28.130
85.0
Table 5: Held-out replay on the 345 test episodes (3,875 decisions, § 5 ). All policies share the declared option sets and the fleet reference behavior. Match = decision-level top-1 agreement with the logged hop (teacher-forced); survival = episodes that replay with no divergence before the log ends (the episode dies at its first divergence); gap = wrong-hop cost per episode—every diverging hop of the whole teacher-forced episode is charged—under conventions c1 (1/wrong hop) and c3 (3/wrong hop); LLM calls/dec. = escalation rate; every typed and hybrid row replays one probe fit (CE + Brier, seed 0). Gap columns come from one wrong-hop count each: c3=3c1 exactly, and displayed entries round independently.
Figure 5: End-to-end replay ( Table 5 ). (a) Decision match; (b) Episode survival under strict divergence; (c) Decision latency: MIMO Pro V2.6 in-vivo p50 2.2 s versus 19.5 ms p50 for the typed decode’s GPU typed-choice head (full typed-choice predict, encoder pass included—the typed path’s upper price; the E3 replay’s probe rows decode at the 10 ms marginal encoder pass: App. B). The accept/escalate rows spend LLM calls only where the readout is least confident—and score strictly below the pure hidden-state-probe row; on this stream the escalated hops carry the opponent’s low agreement into the route, which is consistent with the loss but is not isolated by our design (§ 6 ). The typed-family bars in (a) and (b) come from one hidden-state probe fit (CE + Brier, seed 0; probe family drawn purple); the purple bar in (c) is the typed path’s own decode price.
Figure 6: Held-out test decision accuracy across the self-evolution stream (a seeded random 1,200-row subset of the validation fold, permutation seed 7, played as 300-decision update batches; the 380 validation rows left out of the 1,200-row draw, from the 1,580-row validation fold, are the gate set). The round-3 batch carries 12% label corruption (App. A); on the seed-0 row-level chain plotted here held-out accuracy never falls across refits, rises 30.2% → 31.7%, and holds through the corrupted one (the flips are about 0.3% of the 12,613-decision base). The gate set governs commits: on this chain the gate score never regresses beyond the gate tolerance δ=0.005 . Across ten seeded chains (row- and trace-level stream splits; five chains per mode over shared seed offsets, App. A) held-out accuracy averages 30.3% → 31.0% ( ±0.5 points) and the gate refuses 13 of the 40 candidate updates (6 of the 10 chains refuse at least one), so the single chain under-reports the guardrail’s workload (point tags concatenate the round’s events with + and carry the round number: ung+acc (r1) names the raw refit ungated update and the gated commit accept at round 1, 12% corr+acc (r3) the injected round, base (r0) the base fit). The test split is never trained on.
τ=0.55/0.75/0.9→ 74.1/80.2/85.0% LLM calls, match 20.9/19.1/16.5%
Table 6: Ablation summary (A2 cells: masked acc. / ECE-15; 5 seeds where trained; A1 and A3 rows are counts and rates). A1: what loop folding removes from the decision stream. A2: loss ablation (arrows run CE → CE + Brier) on the readout arms. A3: accept/escalate threshold sweep—summary row here, full hybrid rows in Table 5 , risk–coverage curves in Figure 4 .
Figure 7: Saturation on the five process logs that carry ≥ 20 cases (statechart Leemans, 2018 is a single 17,912-event session and cannot support a subsampling curve; its census sits in Table 7 ). Twelve without-replacement subsample sizes × 20 draws each; p50 curves. (a) Distinct-node discovery climbs toward each catalog ceiling; receipt’s 27th and helpdesk’s last label appear only at their case boundaries ( Buijs, 2014 ; Polato, 2017 ) . (b) First-visit forward pairs ( E+ only; the SOC volume in Figure 3 counts kept-frame unions E+∪E− ) are still climbing at each corpus’s drawn end ( 1.4×103 – 3.2×104 cases). (c) Loop rate is flat in subsample count (the p50 curves stay level over the 12 sizes); separately, the BPIC’12 and BPIC’17 curves draw nearly coincident at 59.2/59.3 loop events per 100 logged events ( van Dongen, 2012 ; van Dongen, 2017 ) . Dashed curves in (a): the zero-parameter occupancy null at the same 12 sizes—the expected distinct labels after m of n cases without replacement, E[D(m)]=∑l(1−∏i=0m−1n−in−cl−i) , cl the number of cases carrying label l (fit discussion in the text). Semantic colour is figure-local: the panels here key corpora by hue (green names BPIC’17); Figure 10 keys sweep classes by marker and colour (axis / process logs / eval subset), Figure 4 and Figure 5 key method arms, Figure 3 key quantity series; repeated hues carry no cross-figure meaning. Panel (c)’s y-axis label echoes the left (loop-event) axis of Figure 3 b at a different scale (compare slopes, not heights).
corpus
cases
events
labels
advance
retry
back
loop mass (%)
BPIC’12
13,087
262,200
24
107,006
105,776
49,418
59.2
BPIC’17
31,509
1,202,267
26
489,408
448,205
264,654
59.3
helpdesk
4,580
21,348
14
19,443
944
961
8.9
receipt
1,434
8,577
27
8,332
6
239
2.9
roadtraffic
10,002
37,255
11
36,732
270
253
1.4
statechart
1
17,912
106
106
17,675
131
99.4
Table 7: Public corpus census and fold totals, produced by the same fold() routine as the SOC corpus’s own folding. Loop mass (%) is 100(retry+back)/events ; steps = logged events on these streams (code: fold_rules.py ).
option space at evaluation
mean ∣A∣
agreement (%)
Δ vs declared (95% CI)
declared A(v) (induced graph)
40.2
30.67
—
node-conditioned, no traversal
40.4
30.79
+0.13[0.00,+0.27]
global train-frequency, size-matched
40.2
29.60
−1.37[−1.88,−0.91]
flat catalog (all 155 classes)
155
27.07
−3.54[−4.10,−2.95]
random K=∣A(v)∣ (floor)
40.2
8.36
−22.45
Table 8: Graph-necessity ablation (E9/E9b). Only the evaluation-time option space changes; training, features, readout recipe and budget are identical across arms (5 seeds). Δ and CIs are paired against declared with a trace-clustered bootstrap (2,000 resamples, seed-0 model); “—” is the reference arm.
decision rule
acc. (%)
in-space (%)
cands
n
note
majority arm
26.8
–
1
3,875
relevance floor
feature MLP, CE †
32.8
35.6
1
3,875
strongest prior readout
typed encoder, frozen
27.7
30.1
1
3,875
parity arm
typed encoder, fine-tuned
32.2
34.9
1
3,875
3 epochs, seed 0
typed encoder, fine-tuned
31.0
–
1
200
same sample as the rows below
jpt-4b, all options
9.0
9.9
46
200
zero-shot
Table 9: What the typed interface can take over, on the held-out test fold. The two zero-shot language-model rows were drawn on a seeded 200-decision sample of the same fold and are accompanied by the typed encoder measured on that same sample, so that comparison is same-denominator; the LoRA-tuned jpt-4b row is scored on the full fold instead. “cands” is the number of options offered to the deciding model. The last row is the containment ceiling of the shortlist, not the accuracy of any model.
state given to the typed encoder
adds
acc. (%)
in-space (%)
top-20 (%)
last eight actions
published renderer
32.00
34.75
89.74
last eight actions + goal
task intent
33.08
35.93
89.24
goal + full own history
length, not intent
31.69
34.42
89.18
Table 10: State conditioning. The three arms share one training recipe, one seed and one batch size and differ only in the state string handed to the typed encoder; the published renderer is the “last eight actions” arm. The goal field is present in the released records and absent from the published renderer. “top-20” is masked recall over each state’s declared options.
key
state content
covered
agree. (%)
agree., ≥2 (%)
L1
current station
3,857
25.6
25.6
L2
one preceding action
3,400
26.5
27.6
L3
three preceding actions
1,694
38.6
43.4
L4
the full action prefix
1,234
42.8
44.9
L5
L3 plus observable features
1,301
41.2
44.7
Table 11: The information ceiling of the released state. “covered” counts held-out decisions whose key also occurs in another trace; “agree” is the leave-one-out modal agreement among those other traces; the last column requires at least two such other-trace observations. The key is made more specific from top to bottom. L4 and L5 differ in kind: L5 adds engineered features instead of history and does not improve on L4.
accepted coverage (%)
5
10
15
20
30
50
80
100
accepted accuracy (%)
96.9
93.6
91.9
82.2
67.0
50.7
37.7
32.2
Table 12: The accept-or-escalate frontier of the fine-tuned typed encoder on the held-out fold, sorted by top-1 probability. Keeping the most confident 15% of decisions holds accepted accuracy at 91.9%; the curve has no 90% point beyond 15% coverage. Temperature was fitted on the validation fold (fitted value 1.0). AURC is 0.426.
decision rule
params
acc. (%)
per decision
note
jpt-4b, all options
4B
9.9
0.22 s
zero-shot, decoder pass
jpt-4b, LoRA-tuned
4B
31.9
0.24 s
fine-tuned, full fold
jpt-4b, 20 candidates
4B
12.1
0.14 s
zero-shot, shortlist
jpt-4b, 10 candidates
4B
14.4
0.11 s
zero-shot, shortlist
jpt-4b, 5 candidates
4B
19.9
0.10 s
zero-shot, shortlist
Laya zero-shot typed choice
421M
8.6
10 ms
domain encoder, no tuning
Table 13: Same-denominator comparison on the identical held-out question: choose the realised next hop among the declared options. Every jpt-4b row but the last is zero-shot; the last is LoRA-tuned on the training fold, which makes it the like-for-like opponent for the typed encoder. The typed encoder is fine-tuned. Latency is one decoder pass for the language model and one forward pass for the typed encoder.
Figure 8: Accept-or-escalate frontier of the fine-tuned typed encoder on the held-out fold, under three confidence rules. Accepted accuracy stays above 90% up to 15% of coverage, then falls towards the accuracy of the whole fold as coverage approaches one. The three rules trace nearly the same curve, so the frontier reflects the state rather than the choice of score.
acquisition arm
admitted
edges
acc. (%)
in-space (%)
90% cov.
@20% cov.
label-free gate
2,283
1,257
28.80
31.28
0.068
71.6
random, same cap (400/round)
3,200
1,543
29.29
31.81
0.070
73.3
random, volume-matched (280/round)
2,240
1,395
27.56
29.93
0.070
72.8
no typed model (frequency backoff)
4,800
1,741
25.91
28.14
0.001
63.5
seed buffer only, no rounds
0
1,015
28.18
30.61
0.153
74.8
full supervised train fold
12,613
—
32.15
34.92
0.150
—
Table 14: The gated self-training loop over eight rounds, held-out test fold. “admitted” counts the pool rows the arm added across all rounds, so the arms differ in both rule and volume; the third row is the volume-matched random control for the first. “edges” is the number of distinct forward transitions in the induced graph at round 8. “90% cov.” is the largest fraction of decisions that can be accepted while holding accepted accuracy above 90%. The last two rows are not loop arms: they bracket it from below and above. The four loop rows are seed 0; three-seed means and the seed spread are given in the text.
Figure 9: Held-out top-1 accuracy of the typed encoder over eight rounds of gated self-training, for four acquisition rules. The three typed arms plateau between 29% and 31% and end below their own peak; the model-free arm climbs steadily from a lower floor and never reaches them. The dashed line is full supervised training on the same fold, at 32.15%, which no arm reaches. Round 0 is each arm’s own anchor set, so the starting points differ slightly (seed 0; the round-0 spread over seven runs is 1.9 points).
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Class-space margin over the edge-ID readout versus catalog budget (top- C train-frequency actions; masked accuracy, 3 of the 5 protocol seeds: both arms are refit at every budget point under pub_sweep.py ’s seeds-0/1/2 convention; the main-text reading and exception accounting are in § 6 and App. B). The 7,318-name ToolBench catalog ( Qin et al., 2023 ) provides the only multi-budget axis: the class vocabulary’s margin is non-monotone but ends wider (never crossing zero in the plotted 3-seed means; per-arm seed standard deviations stay under 0.7 points; net figures in text)—the opposite of the edge-pays-at-scale prediction. Squares mark the five process logs at their 11–27-name training vocabularies (noise-level ties); the single diamond is the evaluation subset at its 275-name training catalog, the only plotted contrary slice (2.2 points to edge IDs over 92 of its 161 held-out hops, scored in-catalog—2.4 points on the fixed-row paired rerun). API-Bank is a second small-catalog reversal (1.7 points to edge IDs under the paired convention of the fixed-row rerun—one whose 335-row training fold sits below that sweep’s own floor and whose paired gap is one-seed-driven, per-seed differences 0.0/0.0/5.2 points—and 3.5 points on unrounded means in the CE-arm pair of the main API-Bank runs, § 6 ).
LLM agent routing selects models, tools, skills, or collaboration workflows for each task or step to balance capability, cost, latency, and reliability.
Agents with smaller language-model backbones are less expensive but can drift into persistent failure modes, whereas those with larger backbones are generally more reliable but more costly. This reliability-cost trade-off motivates routing methods that decide when to invoke an agent with a larger backbone: before execution, after a fixed trajectory prefix, or locally at individual steps. Our method, TACIT-SWITCH, learns permanent handoff policies from accumulated trajectory evidence and Teacher-Annotated Censored Intervention Times (TACIT). It represents each annotation as an interval-censored observation on a cumulative-risk scale. The resulting mixture-cure threshold model estimates the probability that the paired Strong rollout succeeds and, conditional on success, the handoff threshold; no teacher is required at deployment. In a mechanism-based multi-step simulation, TACIT-SWITCH improves success by 7.4-11.1 percentage points over task-level, step-level, and fixed-prefix routing baselines at comparable cost. Within that controlled simulation, ablations show that task features and cumulative trajectory risk provide complementary information. With operating points selected on development data, TACIT-SWITCH achieves the highest held-out success among learned policies on both ALFWorld (48.5% with 4B Cheap; 45.5% with 9B Cheap) and DABench (73.1%).
Ji'an Lei, Jian Huang
School of Statistics Beijing Normal University Beijing 100875, China · Department of Applied Mathematics The Hong Kong Polytechnic University Hung Hom, Kowloon, Hong Kong SAR, China
A growing line of work equips large language model (LLM) agents with graph neural networks (GNNs) as callable tools, assuming the agent exercises judgment over when and how much to rely on such a tool. We test this directly. We expose a frozen GNN to a ReAct-style LLM agent as an explicit tool and measure, on node classification over a text-attributed graph (ogbn-arxiv, replicated on WikiCS), whether the agent uses the tool or merely obeys it. We find the agent does not exercise judgment: its predictions agree with the raw GNN's 97.6-99.2% of the time (5 seeds), collapsing into a GNN parrot that adopts the tool's output wholesale and bypasses its own reasoning. Sweeping backbone capability (Qwen2.5 0.5B-7B), the deference is not a weak-model artifact: among models able to invoke the tool, agreement rises with capability (0.60 to 0.98 from 1.5B to 7B). Crucially, the cost of deference does not shrink as capability grows and grows where alternatives emerge: a per-node oracle over the available actions beats the parrot by 0.09-0.18 at 3B and 0.12-0.22 at 7B, roughly doubling at high homophily, because the parrot is pinned to the frozen GNN while the agent's alternatives improve; at 7B a simple neighbour-label tool overtakes the GNN at high homophily (0.81 vs 0.71) yet the agent still defers. A simple selective-invocation gate recovers about half of that high-homophily gap (0.71 to 0.83) but yields no net global gain, and held-out estimates bound the best achievable gate over standard test-time features to at most a third of the oracle headroom: reliable selective invocation looks limited by available information, not merely router design. Our results are a cautionary measurement: evaluations of agent+tool systems cannot assume the agent adds judgment on top of the tool, and selective invocation must be designed in rather than expected to emerge from scale.
Model routing can reduce the cost of LLM applications by assigning simpler requests to cheaper models without sacrificing the quality of the response. However, most routers treat requests as independent. Stateful agents do not operate under this assumption: a user turn may require multiple model and tool calls, its difficulty depends on the state produced by earlier turns, and changing models can sacrifice reuse of the conversational prompt cache. We study zero-shot, cost-aware routing at the user-turn boundary, systematically varying the routing rule, access to session history, candidate models that differ in size, reasoning effort, or model provider, and prompt-cache behavior. We evaluate on multi-turn versions of SheetCopilot and SpreadsheetBench 2, where persistent workbooks and cross-turn dependencies make isolated routing decisions especially brittle. At a matched offloading rate, giving the router session history improves task pass rate over the same router without history, with gains for every model pair; history-aware routing remains within one point of always using the expensive model. We also found that cost depends on execution dynamics rather than token prices alone: the cheaper model's per-task cost advantage narrows from 6.8x to 3.2x on longer sessions. A closed form cache analysis further shows that downward switches (expensive to cheap) recover their cache cost within two LLM calls, whereas upward switches do not break even under the evaluated conditions. Together, these results show that agent routing is a sequential decision problem: effective policies must account for what happened earlier in the session and for the asymmetric cost of switching models.