Local or Escalate? Post-Training Expert Routing for Real-Time Duplex Speech Models
Authors: Ruoyu He, Jisen Li, Anthony Ji, Changyi Yang, Zetian Li, Runduo Ma, Zhen Zhang, Zhongzhu Zhou
Organizations: University of California, Los Angeles · University of Illinois at Urbana-Champaign · University of British Columbia · Carnegie Mellon University · Stanford University · University of Wisconsin–Madison · University of California, Santa Barbara · University of Sydney
A full-duplex model should answer easy questions immediately, but hard questions often need stronger reasoning, which may require waiting. The challenge is to choose between these paths at the moment the model begins to speak. Recent systems learn related mode decisions through special-token generation inside the dialogue model, coupling routing to turn-taking and temporally annotated full-duplex training. We study a post-training alternative. We freeze MiniCPM-o 4.5, using only its speech input and output, and fit a logistic failure readout on hidden states already available at its native listen-to-speak transition. The resulting gate requires no additional model generation and exposes thresholds that can be retuned without changing the duplex policy. A probe fitted on text-input states also reads the states of spoken questions, including unseen human recordings. On the internal benchmark, selective handoff improves answer accuracy by 27.1 points. Compared with always using the expert, it reduces mean server time to first answer audio by 31.5 percent. It also outperforms matched random routing across external speech question-answering pools, as well as a zero-shot GPT-5.4 mini router that rates the transcribed question. The training recipe also transfers to a second duplex model family. These results show that an existing duplex model can be equipped to answer easy turns immediately and wait for stronger reasoning or fresher knowledge on the turns that need it, without further duplex training.
Figures & tables
Figure 1: Post-training routing at the native speak boundary. (1) Speech streams in 1 s chunks to a frozen full-duplex local model. (2) When the model commits to speak, an L22 linear readout scores local failure risk in tens of milliseconds. (3) Low-risk turns continue immediately, while high-risk turns are sent to a stronger expert and the talker holds the floor. (4) The expert answer is voiced by the local talker. On all 240 internal queries, the aggressive policy raises accuracy from 42.1% to 69.2% at 50.0% escalation.
Figure 2: Text-input transfer depends on layer and position. Held-out-pool (LOPO) math AUC of per-layer failure probes, read at the last token (left) or mean-pooled (right). The dotted line marks the deployed L22 readout, and § 5.2 discusses the late-layer decline.
Internal
TriviaQA
WebQ
Llama Q.
SD-QA
avg
Δ
MiniCPM-o 4.5
always-local
42.1
54.0
54.8
79.6
49.0
59.4
—
+ gate, conservative
48.8
57.6
58.0
78.8
52.0
61.6
+2.3
+ gate, balanced
60.0
68.8
66.0
82.0
67.0
71.0
+11.6
+ gate, aggressive
69.2
87.2
75.2
87.2
83.0
83.2
+23.8
matched random, aggressive
60.8
77.2
69.6
82.7
74.6
76.0
+16.7
Table 1: Answer-content accuracy (%), time, and expert use. Both models use 240 Internal IDs and four external QA pools. The avg column is the external mean and Δ its gain over local. Random rows match each aggressive call rate. Expert USD/query is estimated by replaying escalated uplink transcripts through the identical expert configuration at official rates (Appendix D.2 ).
Figure 3: Accuracy–cost trade-offs across routing policies. Top row: accuracy versus expert call rate in the main run. Bottom row: accuracy versus mean server TTFA in separate deployment runs on the same query sets; two anomalous samples are excluded. L/C/B/A/W denote local, conservative, balanced, aggressive, and always-expert (GPT-5.5) policies. Dashed orange: the zero-shot GPT-5.4 mini router applied to the same sessions at the gate’s C, B and A call rates. M: the same router run live as the escalation decision at the aggressive budget. Each row uses its own judged answers, so W accuracy can differ between rows. Accuracy includes failed attempts; TTFA requires an answer timestamp. Run sources, denominators, and a 60 s exclusion sensitivity analysis are in Appendix E.4 .
Figure 4: When the escalation decision is made (medians, aggressive run). (a) The readout decides when the model commits to speak. (b) A text router needs the transcript and an LLM call, so it decides after the commit even with the call started as soon as the transcript is ready. Live, it adds 0.64 s on turns both policies answer locally (Appendix E.4 ).
Figure 5: Text-trained failure probes read speech states. Left: text-trained probes scored on TTS audio of the same 600 questions. Right: MiniCPM-o 4.5 on 200 unseen SD-QA human recordings, with audio-trained and text-input references in gray. Dots mark the deployed L22.
Figure 6: Conversation ratings and preferences. (a) Means and 95% bootstrap intervals (10,000 participant resamples, 16 conversations per policy). (b) Aggressive wins/ties/losses.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
tier
Internal (TTFA = expert + system)
External macro
local-only
1.08 = 0.00 + 1.08
1.74 = 0.00 + 1.74
conservative
2.15 = 0.85 + 1.31
1.75 = 0.27 + 1.48
balanced
4.67 = 2.96 + 1.71
2.45 = 0.87 + 1.59
aggressive
5.95 = 3.92 + 2.03
4.07 = 2.01 + 2.06
always
8.68 = 6.28 + 2.39
5.38 = 3.54 + 1.85
Appendix
Table 2: Mean server TTFA in the main run decomposed as expert response time plus system overhead (gate wait, transcription, relay synthesis, scheduling), over all completed sessions (nonnegative-wait convention, with zero expert time for unfired sessions). On the always arm, where every session escalates, the system overhead beyond the expert call is 2.39 s mean / 1.96 s median on Internal (external pools pooled: 1.85/1.67).
Stratum
n
Local
A
Always
A call rate
Chat
60
61.7
70.0
80.0
30.0
Factual QA
40
42.5
92.5
95.0
60.0
Knowledge
60
13.3
45.0
63.3
71.7
Math
60
65.0
66.7
80.0
25.0
SimpleQA traps
20
0.0
100.0
95.0
100.0
Appendix
Table 3: Descriptive strata of the full 240-query Internal test set, with unit query weights. Local, aggressive (A), and always report answer-content accuracy (%), and call rate is the aggressive arm’s realized rate. All five categories and all original queries are retained.
model
best mid-layer
final layer
depth profile
Qwen3-8B
0.964 (L33/36)
0.958
plateau
MiniCPM-o 4.5
0.931 (L22/36)
0.366
late decline
Qwen2.5-7B
0.893 (L21/28)
0.809
mild dip
Qwen2.5-Omni
0.794 (L16/28)
0.746
diffuse decline
MiniCPM-o 2.6
0.822 (L21/28)
0.540
late decline
Appendix
Table 4: Text-input LOPO math AUC by depth for matched model families.
β
modules
L33
L34
L35
turns ended
0
–
0.498
0.347
0.364
99/100
0.25
attention only
0.492
0.401
0.423
97/100
0.25
MLP only
0.591
0.477
0.382
100/100
0.25
all
0.591
0.515
0.434
87/100
0.5
all
0.660
0.634
0.435
0/100
Appendix
Table 5: Weight interpolation of L28–35 toward Qwen3-8B: text-input LOPO math AUC of last-token probes and turns ended in 100 native duplex queries.
expert budget
15%
30%
50%
probe-guided accuracy
55.0
65.0
75.7
matched-random accuracy
50.7
57.7
67.0
Appendix
Table 6: VoiceChat-11B cached native-frame replay, equally averaged over four external English speech-QA pools. Random uses the same call rate.
Pool
n
Conservative
Balanced
Aggressive
Always
Internal
240
8.3
28.3
50.0
100.0
Speech TriviaQA
250
3.6
19.2
55.2
100.0
Speech WebQ
250
6.4
23.6
56.8
100.0
Speech Llama Q.
250
0.0
2.8
21.2
100.0
SD-QA
200
7.5
24.5
62.5
100.0
Appendix
Table 7: Realized escalation rates (%) for the MiniCPM results in Table 1 . Internal retains all 240 queries, and every pool uses unit query weights. Rates can differ from nominal calibration and VoiceChat replay budgets.
tier
calls
accuracy
random
local-only
0.0
42.1
—
conservative
8.3
48.8
45.2
balanced
28.3
60.0
52.7
aggressive
50.0
69.2
60.8
always
100.0
79.6
—
Appendix
Table 8: Internal results on all 240 original test queries (%), with unit query weights. Random mixes the measured local/always endpoints at the corresponding call rate.
tier
mean
P50
P95
P99
local-only
1.08
0.78
2.61
3.66
conservative
2.15
0.92
7.23
22.74
balanced
4.67
1.60
17.63
39.40
aggressive
5.95
3.52
19.19
40.48
always
8.68
6.75
22.90
34.11
Appendix
Table 9: Internal server TTFA (seconds), measured on all 240 test IDs: 240 attempts per arm, with 238/238/239/238/239 completed and 2/2/1/2/1 failed in row order. TTFA is the nonnegative wait from scheduled input end to first answer PCM. All completed sessions count, and the 19/16/14/9/0 early responses have zero wait. Failed attempts have no TTFA value. Timing ends at the first local PCM-bearing chunk return or relay PCM yield, excluding candidate/stall audio and client playback.
Aggressive tier
Live
Fixed outcomes
Internal
+8.3
+8.1
S. TriviaQA
+10.0
+8.8
S. WebQ
+5.6
+5.1
S. Llama Q.
+4.5
+4.7
SD-QA
+8.4
+8.9
External macro
+7.1
+6.9
Appendix
Table 10: Gate accuracy minus matched-random accuracy (points). Live: independently sampled Table 1 arms, paired at the query level. Fixed outcomes: the local-only and always-expert answers held fixed per query, escalation chosen by the readout of the local-only session at the deployed thresholds.
Score
OOF
En-4 AUC
Agg. gain (5-pool mean)
TF–IDF, gold text
0.764
0.532
+3.1
GPT-5.4 mini zero-shot, gold text
0.767
0.723
+4.5
GPT-5.4 mini zero-shot, ASR text
0.767
0.722
+4.6
Hidden-state readout (deployed)
0.848
0.783
+7.2
Appendix
Table 11: Failure-prediction AUC and routing gain over matched random at the aggressive call rate (points, mean over the five pools), same training rows, folds, and labels for all learned rows. On the external pools every row ranks the main run’s queries and is judged on its local-only and always-expert answers. The ASR row scores the live router’s transcript, and calibration rows have only gold text. The GPT-5.4 mini rows are unfitted, so their first column is plain AUC on the gold calibration text.
Question family (source)
n
Readout
LLM
Readout − LLM
Long-tail entities (PopQA)
741
0.863
0.701
+0.162
TruthfulQA
441
0.693
0.698
− 0.005
All English rows
4828
0.848
0.771
+0.077
All Chinese rows
400
0.782
0.578
+0.204
Appendix
Table 12: Failure-prediction AUC by question family on the calibration rows (gold question text, label = native failure of the local model). Readout = deployed out-of-fold score, LLM = zero-shot gpt-5.4-mini rating. Of the 17 families, PopQA and TruthfulQA (the only one where the LLM scores higher) are shown. All 17 are in revision_data/probe_llm_pools.json , and one further family with a single outcome class is omitted.
Head
OOF
Internal
English mean
Logistic, raw features (deployed)
0.848
0.876
0.771
Logistic, standardized
0.818
0.838
0.704
Elastic net, standardized
0.843
0.884
0.764
MLP, 256 ReLU units, standardized
0.815
0.819
0.702
PCA-256 and gradient-boosted trees
0.825
0.845
0.715
Appendix
Table 13: Head form on the deployed features and calibration rows. OOF is the five-fold out-of-fold AUC on the 5,228 calibration rows. Internal and the English mean are AUCs on a separate native feature-dump run of the evaluation queries.
Escalation decision
Call p50 (s)
Mean TTFA (s), Δ in parentheses
after the commit
at the transcript
Hidden-state readout (deployed)
none
4.44
GPT-5.4 mini, serving region, sequential
0.70
5.34 ( +0.90 )
4.96 ( +0.52 )
GPT-5.4 mini, serving region, 8 concurrent
0.77
5.41 ( +0.97 )
5.00 ( +0.56 )
GPT-5.4 mini, degraded network (assumed)
1.00
5.64 ( +1.20 )
5.14 ( +0.70 )
Appendix
Table 14: Mean TTFA when the escalation decision is made by the readout or by the zero-shot gpt-5.4-mini router on the ASR transcript, replayed on the main aggressive run (1,181 completed sessions, five pools, non-negative TTFA). This is a pooled session mean, not the per-pool macro average of the main table. The router’s ASR starts at the scheduled input end like the gate arm’s. The LLM call either waits for the commit or starts as soon as the transcript is ready, with the decision waiting for both. LLM latencies are measured medians, except the assumed 1 s row.
Accuracy n /arm
TTFA n by arm
Pool
L
C
B
A
W
M
Internal
240
239
240
238
238
239
237
Speech TriviaQA
250
250
250
250
248
250
250
SD-QA
200
200
200
200
200
200
200
Appendix
Table 15: Bottom-row deployment cohorts in Figure 3 . Accuracy uses all attempts in each arm; TTFA counts every completed session with an answer timestamp, except the two long waits.
Pool
n
Call rate (%)
Accuracy (%)
Extra wait (s), both local
gate
router
gate
router ( Δ )
router − gate
Internal
237
51.9
54.4
71.7
68.4 ( − 3.4)
+ 0.79 ( n=78 )
Speech TriviaQA
250
56.0
46.8
89.2
84.0 ( − 5.2)
+ 0.55 ( n=84 )
SD-QA
200
65.5
63.0
81.5
79.5 ( − 2.0)
+ 0.56 ( n=40 )
Overall
687
80.8
77.3 ( − 3.5)
+ 0.64 ( n=202 )
Appendix
Table 16: Live comparison at the aggressive budget: the gate versus zero-shot GPT-5.4 mini (small_fails ≥ 0.12, ASR from the input end, call at the transcript) as the escalation decision, paired by query. Accuracy is the main judge on the delivered answer, and TTFA is the non-negative wait after the end of the input. Deltas are router minus gate. The latency column is the extra wait on turns neither arm escalated, with their count in parentheses. The overall row reports macro accuracy across the three pools and pooled counts and latency. Both arms read the routed session.
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.