cs.CROct 19, 2026
SaveNIOCIS: Typed-Decision Next-Hop Routing with Node-Conditioned Option Spaces for Tool-Use Agents
Organizations: Independent Researcher
Abstract
Tool-use agents pay per decision: at each hop a language model re-reads the state and thinks for seconds, though its actions are finite and decisions recur. We present NIOCIS, a routing layer that folds execution logs into a loop-collapsed action graph (forward transitions plus recovery back-edges) answering one typed question: which declared option at a state, plus stop, does the fleet take next? A small model reads a calibrated distribution over them in one pass, emitting no text. On 1,615 production security-operations episodes we measure what the layer replaces, returns and is bounded by. Replaces: the typed encoder matches the logged next hop on 32.2% of the 3,875 held-out decisions (34.9% within the declared options) in 10 ms of encoder time (19.5 ms for the typed-choice form), against 2.2 s per LLM hop; two feature readouts reach 32.8% and 32.3%, an ordering the paired test does not resolve, so the typed layer is chosen for cost, not a separated lead. Tuning: jpt-4b reaches 9.9% zero-shot and 31.9% fine-tuned, a gap an exact paired test does not resolve (p=0.68), at 244 ms. Returns little: the realised hop falls outside the declared options for 7.9% of decisions and outside the model's top twenty for 9.3%; the most confident 15% holds 91.9% accuracy. Bounded: a cross-trace leave-one-out test puts the released state's ceiling at 25.6--44.9% as the state key grows more specific, i.e. repeated states disagree more often than they agree; the option-space ablation attributes the gain to per-node candidate restriction (+3.5 points over a flat catalog) rather than graph traversal (zero within its interval); a gated self-evolution loop lifts every arm above its anchor yet ends below full supervision; a confidence-gated cascade loses agreement when the escalation arm is the zero-shot stand-in; and agreement past a third of the fleet needs a state carrying the plan and the observation.
Figures & tables
Figure 1: NIOCIS: logs are folded into a loop-collapsed action graph; a typed-decision router answers “which edge in does the fleet take next?” per state, escalating only uncertain cases to the LLM; executed walks flow back as gated updates. The step budget and per-node revisit cap on the execution box are full-design parameters: the corpus study of § 5 replays logged episodes teacher-forced and exercises neither bound.
Figure 2: The induced action graph at the level on which every node can be named: a capability-family structure over the released corpus. The 154 catalog actions are grouped into 23 capability families by an ordered capability-keyword rule (a display quotient defined in the released analysis code, not a construct of the method), so the drawing covers all 154 stations and all 15,704 advance hops with no anonymous station. The families are placed on one axis in the order that minimises weighted arc span over the family pairs (51,717 43,864); solid arcs run with that order, warm dashed arcs against it, thickness proportional to transferred hops, and a hook on a node carries the hops that never leave its family. Node size is episode count and colour the dominant domain slice — a deliberately honest non-finding: 22 of the 23 families are detection-dominant, so colour separates one node. Two properties of that a station-level view hides become visible content. First, the hunting loop is two-directional ( 550/465, 178/170, 171/142). Second, 3,709 of the 14,089 advance transitions (26.3%; the 15,704 advance hops of § 3 less one opening step per episode—1,615 episodes) never leave their own family, 1,271 of them in the SIEM family alone. The 58 drawn arcs are the 50 heaviest family-to-family exchanges plus one arc for each family the budget would otherwise leave isolated, and they carry 64.2% of cross-family mass; the remaining share is counted, never implied absent, in the table, which lists every family with its station count, episodes and out/in/intra-family mass. close-out (ringed) is the graph’s only terminal group: every hunting line is resolved there.
Figure 3: Node-vocabulary convergence of the loop-collapsed action graph on real production traces (subsample quantities: lines = medians of 20 without-replacement draws at the shown size; shading = min–max over those draws, for the node and distinct-pair series of (a) only—back-edge pairs show their median lines). (a) Semantic node counts saturate quickly (77 action labels in 50-episode subsamples 124 at 400 154 at the full 1,615), while observed transition volume (kept-frame pairs, ) keeps a long tail (distinct pairs 336 at 50 episodes 1,267 at 400 2,027 at 1,000 2,536 at 1,615, of which 702 appear at recovery returns). (b) Loop-event rates: fold events settle at 29.2% of executed steps (4,124 retry + 2,364 back) and back returns run 146 per 100 episodes—the right-axis back-event series in (b) counts these return events, 2,364 in full, not (a)’s 702 distinct pairs. Axes, hues and y-axis labels are figure-local across the paper’s figures (see Figure 7 ’s caption).
| split | episodes | completed (%) | decision steps | loop events |
|---|---|---|---|---|
| train | 1,125 | 67.3 | 10,929 | 4,615 |
| val | 145 | 60.0 | 1,394 | 519 |
| test | 345 | 65.5 | 3,381 | 1,354 |
| all | 1,615 | 66.3 | 15,704 | 6,488 |
Table 1: Real SOC corpus after loop folding (§ 4 ): 22,192 tool calls grouped into 1,615 episodes over 154 actions. “decision steps” count advance (hop) steps; the router’s decision states add the kept recovery returns (train/val/test 1,684/ 186/ 494 2,364), giving the 18,068 rows ( ) all SOC experiments draw on. “loop events” count executed retry and back fold events (4,124 retries, which leave the decision stream, plus 2,364 returns, which stay). “completed” is the derived close-out outcome (§ 5 ). The self-evolution stream and its rollback gate for E4 are carved from the validation fold.
| readout arm | acc. (%) | in-space (%) | ECE-15 | AURC | std / config |
|---|---|---|---|---|---|
| majority arm | 26.8 | – | – | – | – |
| logistic regression † | 32.3 | – | – | – | – |
| feature MLP, CE † | 32.8 | 35.6 | 0.045 | 0.409 | |
| feature MLP, CE Brier † | 32.7 | 35.5 | 0.048 | 0.409 | |
| edge-ID readout, CE † | 31.4 | 34.1 | 0.082 | 0.432 | |
| edge-ID readout, CE Brier † | 31.6 | 34.3 | 0.080 | 0.431 |
Table 2: Next-hop agreement on the held-out test decisions: top-1 accuracy over 3,875 decision states; for the trained readout arms, masked top-1 renormalized over each state’s declared options; raw top-1 for the majority and logistic rows (option-masked accuracy is computed for both baseline rows on the public streams and for the majority arm on all nine streams, App. B; it equals raw for the majority arm everywhere, while the logistic pair diverges on the two ToolBench streams Qin et al., 2023 : 35.7% masked versus 36.4% raw at 7,318 names and 34.8% masked versus 36.0% raw on the 351-name evaluation subset, § 6 ). Uncertainty: 5 training seeds for the readout arms, std—accuracy fractions, e.g. —in the note column. “in-space” restricts to the 92.1% of decisions whose logged target lies inside the declared option space. ECE-15 and AURC are probabilities printed to three decimals. “–” = undefined (the majority and logistic rows carry no calibration cells, no in-space cells, and no AURC entry: the majority arm has no confidence ranking and no calibrated distribution is fitted for the logistic row; AURC is likewise not computed for the probe and zero-shot rows—their risk–coverage curves come only from coarse 7-point grids); note-column “–” = no seed spread and no config entry (the zero-shot row reports its fitted decoding temperature instead). Probe and zero-shot fitting details are in App. B. † The paired interval against the indicated reference includes zero ( Table 3 ): the ordering among the marked rows is a point estimate, not a separated difference.
| paired comparison (same states) | (pts) | 95% clustered CI | discordant | reading |
|---|---|---|---|---|
| feature MLP, CE Brier CE | 35 / 30 | point estimate only | ||
| feature MLP, CE Brier majority arm | 105 / 326 | separates | ||
| feature MLP, CE Brier logistic regression | 101 / 110 | point estimate only | ||
| edge-ID class space (features, CE Brier) | 176 / 144 | point estimate only | ||
| edge-ID class space (probe, CE Brier) | 212 / 359 | separates | ||
| edge-ID CE Brier edge-ID logistic | 176 / 170 | point estimate only |
Table 3: Paired tests on the same held-out states (seed-0 fits; e10_paired_ci.json , e11_probe_edge_ci.json ). is in accuracy points with a 95% CI from 2,000 trace-clustered bootstrap resamples; “discordant” gives the counts (first arm right / second arm right). Rows read point estimate only have a CI that includes zero: their ordering in Table 2 is a point estimate, not a separated difference.
| bucket ( of the state) | share | mean | majority | MLP CE Brier | edge-ID | |
|---|---|---|---|---|---|---|
| 263 | 6.8% | 1.23 | 92.0 | 93.9 (99.2) | 93.2 | |
| – | 139 | 3.6% | 7.90 | 34.5 | 38.9 (54.0) | 36.7 |
| 3,473 | 89.6% | 44.46 | 21.5 | 27.6 (29.8) | 26.8 |
Table 4: The same decisions split by the branch factor at the state (3,875 test decisions). Only 3.6% of states fall in the middle bucket, so the headline is dominated by the bucket (89.6% of rows). “majority” is the per-node majority-vote arm; MLP and edge-ID are the feature arms, masked accuracy with in-space accuracy in parentheses; mean is over declared options ( stop included). e8_slices.json additionally carries a pooled train val test by_outdeg table with the same accuracy magnitudes over 19,375 rows; the counts and accuracies quoted here are test-split, and our seed-0 probe refit reproduces them (92.0/22.3/26.3 against the five-seed 92.2/21.2/25.9 of § 6 ).
Figure 4: (a) Readout arms on the held-out test decisions (masked top-1; raw top-1 for the majority/logistic rows; the zero-shot row counts out-of-space targets as errors, as the masked rows and the majority arm do). (b) Risk–coverage of the CE-plus-Brier arms (masked-renormalized distributions; the edge-ID probe arm is not drawn): confidence separates risk sharply at the top of the ranking in the feature arms and thinly across the bulk in both families (the probe grid starts at 50% coverage). Dense curves live in real/e2_results.json ( curve_mean ; seed 0 for the feature arms); the probe arm is drawn from its seven-point, five-seed-mean risk@coverage grid in real/e2_layaprobe_results.json .
| policy | match (%) | survival (%) | gap | gap | LLM calls/dec. (%) |
|---|---|---|---|---|---|
| MIMO Pro V2.6 every hop | 9.6 | 0.0 | 10.157 | 30.470 | 100.0 |
| hidden-state probe, CE+Brier | 30.7 | 0.6 | 7.788 | 23.365 | 0.0 |
| typed+esc @ | 20.9 | 0.0 | 8.890 | 26.670 | 74.1 |
| typed+esc @ | 19.1 | 0.0 | 9.084 | 27.252 | 80.2 |
| typed+esc @ | 16.5 | 0.0 | 9.377 | 28.130 | 85.0 |
Table 5: Held-out replay on the 345 test episodes (3,875 decisions, § 5 ). All policies share the declared option sets and the fleet reference behavior. Match = decision-level top-1 agreement with the logged hop (teacher-forced); survival = episodes that replay with no divergence before the log ends (the episode dies at its first divergence); gap = wrong-hop cost per episode—every diverging hop of the whole teacher-forced episode is charged—under conventions (1/wrong hop) and (3/wrong hop); LLM calls/dec. = escalation rate; every typed and hybrid row replays one probe fit (CE Brier, seed 0). Gap columns come from one wrong-hop count each: exactly, and displayed entries round independently.
Figure 5: End-to-end replay ( Table 5 ). (a) Decision match; (b) Episode survival under strict divergence; (c) Decision latency: MIMO Pro V2.6 in-vivo p50 2.2 s versus 19.5 ms p50 for the typed decode’s GPU typed-choice head (full typed-choice predict, encoder pass included—the typed path’s upper price; the E3 replay’s probe rows decode at the 10 ms marginal encoder pass: App. B). The accept/escalate rows spend LLM calls only where the readout is least confident—and score strictly below the pure hidden-state-probe row; on this stream the escalated hops carry the opponent’s low agreement into the route, which is consistent with the loss but is not isolated by our design (§ 6 ). The typed-family bars in (a) and (b) come from one hidden-state probe fit (CE Brier, seed 0; probe family drawn purple); the purple bar in (c) is the typed path’s own decode price.
Figure 6: Held-out test decision accuracy across the self-evolution stream (a seeded random 1,200-row subset of the validation fold, permutation seed 7, played as 300-decision update batches; the 380 validation rows left out of the 1,200-row draw, from the 1,580-row validation fold, are the gate set). The round-3 batch carries 12% label corruption (App. A); on the seed-0 row-level chain plotted here held-out accuracy never falls across refits, rises 30.2% 31.7%, and holds through the corrupted one (the flips are about 0.3% of the 12,613-decision base). The gate set governs commits: on this chain the gate score never regresses beyond the gate tolerance . Across ten seeded chains (row- and trace-level stream splits; five chains per mode over shared seed offsets, App. A) held-out accuracy averages 30.3% 31.0% ( points) and the gate refuses 13 of the 40 candidate updates (6 of the 10 chains refuse at least one), so the single chain under-reports the guardrail’s workload (point tags concatenate the round’s events with + and carry the round number: ung+acc (r1) names the raw refit ungated update and the gated commit accept at round 1, 12% corr+acc (r3) the injected round, base (r0) the base fit). The test split is never trained on.
| item | measured effect (acc. / ECE-15 where two-sided) | |
|---|---|---|
| A1 | loop folding | 22,192 positions 18,068 decision states (15,704 advances + 2,364 kept returns); 4,124 retries dropped, 6,488 loop events tagged (29.2% of steps) |
| A2 | feature MLP | 32.8 /0.045 32.7 /0.048 |
| edge-ID readout | 31.4 /0.082 31.6 /0.080 | |
| hidden-state probe | 30.3 /0.123 30.7 /0.122 | |
| A3 | escalation sweep | 74.1/80.2/85.0% LLM calls, match 20.9/19.1/16.5% |
Table 6: Ablation summary (A2 cells: masked acc. / ECE-15; 5 seeds where trained; A1 and A3 rows are counts and rates). A1: what loop folding removes from the decision stream. A2: loss ablation (arrows run CE CE Brier) on the readout arms. A3: accept/escalate threshold sweep—summary row here, full hybrid rows in Table 5 , risk–coverage curves in Figure 4 .
Figure 7: Saturation on the five process logs that carry 20 cases (statechart Leemans, 2018 is a single 17,912-event session and cannot support a subsampling curve; its census sits in Table 7 ). Twelve without-replacement subsample sizes 20 draws each; p50 curves. (a) Distinct-node discovery climbs toward each catalog ceiling; receipt’s 27th and helpdesk’s last label appear only at their case boundaries ( Buijs, 2014 ; Polato, 2017 ) . (b) First-visit forward pairs ( only; the SOC volume in Figure 3 counts kept-frame unions ) are still climbing at each corpus’s drawn end ( – cases). (c) Loop rate is flat in subsample count (the p50 curves stay level over the 12 sizes); separately, the BPIC’12 and BPIC’17 curves draw nearly coincident at 59.2/59.3 loop events per 100 logged events ( van Dongen, 2012 ; van Dongen, 2017 ) . Dashed curves in (a): the zero-parameter occupancy null at the same 12 sizes—the expected distinct labels after of cases without replacement, , the number of cases carrying label (fit discussion in the text). Semantic colour is figure-local: the panels here key corpora by hue (green names BPIC’17); Figure 10 keys sweep classes by marker and colour (axis / process logs / eval subset), Figure 4 and Figure 5 key method arms, Figure 3 key quantity series; repeated hues carry no cross-figure meaning. Panel (c)’s y-axis label echoes the left (loop-event) axis of Figure 3 b at a different scale (compare slopes, not heights).
| corpus | cases | events | labels | advance | retry | back | loop mass (%) |
|---|---|---|---|---|---|---|---|
| BPIC’12 | 13,087 | 262,200 | 24 | 107,006 | 105,776 | 49,418 | 59.2 |
| BPIC’17 | 31,509 | 1,202,267 | 26 | 489,408 | 448,205 | 264,654 | 59.3 |
| helpdesk | 4,580 | 21,348 | 14 | 19,443 | 944 | 961 | 8.9 |
| receipt | 1,434 | 8,577 | 27 | 8,332 | 6 | 239 | 2.9 |
| roadtraffic | 10,002 | 37,255 | 11 | 36,732 | 270 | 253 | 1.4 |
| statechart | 1 | 17,912 | 106 | 106 | 17,675 | 131 | 99.4 |
Table 7: Public corpus census and fold totals, produced by the same fold() routine as the SOC corpus’s own folding. Loop mass (%) is ; steps = logged events on these streams (code: fold_rules.py ).
| option space at evaluation | mean | agreement (%) | vs declared (95% CI) |
|---|---|---|---|
| declared (induced graph) | 40.2 | 30.67 | — |
| node-conditioned, no traversal | 40.4 | 30.79 | |
| global train-frequency, size-matched | 40.2 | 29.60 | |
| flat catalog (all 155 classes) | 155 | 27.07 | |
| random (floor) | 40.2 | 8.36 |
Table 8: Graph-necessity ablation (E9/E9b). Only the evaluation-time option space changes; training, features, readout recipe and budget are identical across arms (5 seeds). and CIs are paired against declared with a trace-clustered bootstrap (2,000 resamples, seed-0 model); “—” is the reference arm.
| decision rule | acc. (%) | in-space (%) | cands | note | |
|---|---|---|---|---|---|
| majority arm | 26.8 | – | 1 | 3,875 | relevance floor |
| feature MLP, CE † | 32.8 | 35.6 | 1 | 3,875 | strongest prior readout |
| typed encoder, frozen | 27.7 | 30.1 | 1 | 3,875 | parity arm |
| typed encoder, fine-tuned | 32.2 | 34.9 | 1 | 3,875 | 3 epochs, seed 0 |
| typed encoder, fine-tuned | 31.0 | – | 1 | 200 | same sample as the rows below |
| jpt-4b, all options | 9.0 | 9.9 | 46 | 200 | zero-shot |
Table 9: What the typed interface can take over, on the held-out test fold. The two zero-shot language-model rows were drawn on a seeded 200-decision sample of the same fold and are accompanied by the typed encoder measured on that same sample, so that comparison is same-denominator; the LoRA-tuned jpt-4b row is scored on the full fold instead. “cands” is the number of options offered to the deciding model. The last row is the containment ceiling of the shortlist, not the accuracy of any model.
| state given to the typed encoder | adds | acc. (%) | in-space (%) | top-20 (%) |
|---|---|---|---|---|
| last eight actions | published renderer | 32.00 | 34.75 | 89.74 |
| last eight actions goal | task intent | 33.08 | 35.93 | 89.24 |
| goal full own history | length, not intent | 31.69 | 34.42 | 89.18 |
Table 10: State conditioning. The three arms share one training recipe, one seed and one batch size and differ only in the state string handed to the typed encoder; the published renderer is the “last eight actions” arm. The goal field is present in the released records and absent from the published renderer. “top-20” is masked recall over each state’s declared options.
| key | state content | covered | agree. (%) | agree., (%) |
|---|---|---|---|---|
| L1 | current station | 3,857 | 25.6 | 25.6 |
| L2 | one preceding action | 3,400 | 26.5 | 27.6 |
| L3 | three preceding actions | 1,694 | 38.6 | 43.4 |
| L4 | the full action prefix | 1,234 | 42.8 | 44.9 |
| L5 | L3 plus observable features | 1,301 | 41.2 | 44.7 |
Table 11: The information ceiling of the released state. “covered” counts held-out decisions whose key also occurs in another trace; “agree” is the leave-one-out modal agreement among those other traces; the last column requires at least two such other-trace observations. The key is made more specific from top to bottom. L4 and L5 differ in kind: L5 adds engineered features instead of history and does not improve on L4.
| accepted coverage (%) | 5 | 10 | 15 | 20 | 30 | 50 | 80 | 100 |
|---|---|---|---|---|---|---|---|---|
| accepted accuracy (%) | 96.9 | 93.6 | 91.9 | 82.2 | 67.0 | 50.7 | 37.7 | 32.2 |
Table 12: The accept-or-escalate frontier of the fine-tuned typed encoder on the held-out fold, sorted by top-1 probability. Keeping the most confident 15% of decisions holds accepted accuracy at 91.9%; the curve has no 90% point beyond 15% coverage. Temperature was fitted on the validation fold (fitted value 1.0). AURC is 0.426.
| decision rule | params | acc. (%) | per decision | note |
|---|---|---|---|---|
| jpt-4b, all options | 4B | 9.9 | 0.22 s | zero-shot, decoder pass |
| jpt-4b, LoRA-tuned | 4B | 31.9 | 0.24 s | fine-tuned, full fold |
| jpt-4b, 20 candidates | 4B | 12.1 | 0.14 s | zero-shot, shortlist |
| jpt-4b, 10 candidates | 4B | 14.4 | 0.11 s | zero-shot, shortlist |
| jpt-4b, 5 candidates | 4B | 19.9 | 0.10 s | zero-shot, shortlist |
| Laya zero-shot typed choice | 421M | 8.6 | 10 ms | domain encoder, no tuning |
Table 13: Same-denominator comparison on the identical held-out question: choose the realised next hop among the declared options. Every jpt-4b row but the last is zero-shot; the last is LoRA-tuned on the training fold, which makes it the like-for-like opponent for the typed encoder. The typed encoder is fine-tuned. Latency is one decoder pass for the language model and one forward pass for the typed encoder.
Figure 8: Accept-or-escalate frontier of the fine-tuned typed encoder on the held-out fold, under three confidence rules. Accepted accuracy stays above 90% up to 15% of coverage, then falls towards the accuracy of the whole fold as coverage approaches one. The three rules trace nearly the same curve, so the frontier reflects the state rather than the choice of score.
| acquisition arm | admitted | edges | acc. (%) | in-space (%) | 90% cov. | @20% cov. |
| label-free gate | 2,283 | 1,257 | 28.80 | 31.28 | 0.068 | 71.6 |
| random, same cap (400/round) | 3,200 | 1,543 | 29.29 | 31.81 | 0.070 | 73.3 |
| random, volume-matched (280/round) | 2,240 | 1,395 | 27.56 | 29.93 | 0.070 | 72.8 |
| no typed model (frequency backoff) | 4,800 | 1,741 | 25.91 | 28.14 | 0.001 | 63.5 |
| seed buffer only, no rounds | 0 | 1,015 | 28.18 | 30.61 | 0.153 | 74.8 |
| full supervised train fold | 12,613 | — | 32.15 | 34.92 | 0.150 | — |
Table 14: The gated self-training loop over eight rounds, held-out test fold. “admitted” counts the pool rows the arm added across all rounds, so the arms differ in both rule and volume; the third row is the volume-matched random control for the first. “edges” is the number of distinct forward transitions in the induced graph at round 8. “90% cov.” is the largest fraction of decisions that can be accepted while holding accepted accuracy above 90%. The last two rows are not loop arms: they bracket it from below and above. The four loop rows are seed 0; three-seed means and the seed spread are given in the text.
Figure 9: Held-out top-1 accuracy of the typed encoder over eight rounds of gated self-training, for four acquisition rules. The three typed arms plateau between 29% and 31% and end below their own peak; the model-free arm climbs steadily from a lower floor and never reaches them. The dashed line is full supervised training on the same fold, at 32.15%, which no arm reaches. Round 0 is each arm’s own anchor set, so the starting points differ slightly (seed 0; the round-0 spread over seven runs is 1.9 points).
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Class-space margin over the edge-ID readout versus catalog budget (top- train-frequency actions; masked accuracy, 3 of the 5 protocol seeds: both arms are refit at every budget point under pub_sweep.py ’s seeds-0/1/2 convention; the main-text reading and exception accounting are in § 6 and App. B). The 7,318-name ToolBench catalog ( Qin et al., 2023 ) provides the only multi-budget axis: the class vocabulary’s margin is non-monotone but ends wider (never crossing zero in the plotted 3-seed means; per-arm seed standard deviations stay under 0.7 points; net figures in text)—the opposite of the edge-pays-at-scale prediction. Squares mark the five process logs at their 11–27-name training vocabularies (noise-level ties); the single diamond is the evaluation subset at its 275-name training catalog, the only plotted contrary slice (2.2 points to edge IDs over 92 of its 161 held-out hops, scored in-catalog—2.4 points on the fixed-row paired rerun). API-Bank is a second small-catalog reversal (1.7 points to edge IDs under the paired convention of the fixed-row rerun—one whose 335-row training fold sits below that sweep’s own floor and whose paired gap is one-seed-driven, per-seed differences 0.0/0.0/5.2 points—and 3.5 points on unrounded means in the CE-arm pair of the main API-Bank runs, § 6 ).
More in this topic
LLM Agent Routing
LLM agent routing selects models, tools, skills, or collaboration workflows for each task or step to balance capability, cost, latency, and reliability.
See all 96 papers