SEAR: Schema-Based Evaluation and Routing for LLM Gateways

Zecheng Zhang; Han Zheng; Yue Xu

cs.DBMar 20, 2026

SEAR: Schema-Based Evaluation and Routing for LLM Gateways

Authors: Zecheng Zhang, Han Zheng, Yue Xu

Organizations: Strukto.AI, USA · Infron.AI, USA

Abstract

Evaluating production LLM responses and routing requests across providers in LLM gateways requires fine-grained quality signals and operationally grounded decisions. To address this gap, we present SEAR, a schema-based evaluation and routing system for multi-model, multi-provider LLM gateways. SEAR defines an extensible relational schema covering both LLM evaluation signals (context, intent, response characteristics, issue attribution, and quality scores) and gateway operational metrics (latency, cost, throughput), with cross-table consistency links across around one hundred typed, SQL-queryable columns. To populate the evaluation signals reliably, SEAR proposes self-contained signal instructions, in-schema reasoning, and multi-stage generation that produces database-ready structured outputs. Because signals are derived through LLM reasoning rather than shallow classifiers, SEAR captures complex request semantics, enables human-interpretable routing explanations, and unifies evaluation and routing in a single query layer. Across thousands of production sessions, SEAR achieves strong signal accuracy on human-labeled data and supports practical routing decisions, including large cost reductions with comparable quality.

Figures & tables

Figure 1. SEAR system architecture and database schema. A central gateway routes requests to LLM providers, samples traffic to the SEAR judge for evaluation, and logs operational metrics for all requests. Solid arrows denote mandatory foreign keys and dashed arrows denote optional foreign keys from the gateway metrics table.

Table	Bool	Cat	Ord	Text	Static	Total
context_info	17	4	2	2	20	45
llm_response_info	15	1	2	1	0	19
issue_attribution	1	19	0	0	0	20
evaluation	2	1	28	0	0	31
Total	35	25	32	3	20	115

Table 1. Signal composition per semantic evaluation table. Columns show the number of signals by type; Static denotes columns extracted directly from raw session data (e.g., message counts, token counts per role) without the LLM judge.

Figure 2. Each table’s structured output call receives the conversation context and all upstream structured table outputs. Dashed arrows indicate input dependencies.

Capability	Ours	Text	Single	Rubric	Templ.
Root-cause attribution	✓	$\sim$	✗	$\sim$	$\sim$
Quality scores	✓	$\sim$	$\sim$	✓	✓
Session aggregation	✓	$\sim$	✓	✓	$\sim$
Per-signal slice analysis	✓	$\sim$	✗	$\sim$	$\sim$
Joint eval. + oper. metrics	✓	$\sim$	$\sim$	$\sim$	$\sim$
Consistency checks	✓	✗	✗	✗	✗

Table 2. Qualitative comparison of evaluation approaches. ✓ = natively supported,

\sim

= partial, ✗ = not supported.

Capability	Ours	Signal	Learned
Per-signal explanation	✓	$\sim$	$\sim$
Deep semantic extraction	✓	✗	$\sim$
Preference optimization	✓	✗	✓
Quality–cost–latency tuning	✓	$\sim$	$\sim$
Oper.-metric-aware routing	✓	✓	$\sim$
Joint evaluation and routing	✓	✗	✗

Table 3. Qualitative comparison of routing approaches. ✓ = natively supported,

\sim

= partial, ✗ = not supported.

	GPT-5-mini				GPT-5.2
Metric	Low (w/o)	Low (w/)	High (w/o)	High (w/)	Low (w/)	High (w/)
Error rate $\downarrow$	0.1044	0.0989	0.0926	0.0916	0.0965	0.0851
Hamming loss $\downarrow$	0.1003	0.0958	0.0878	0.0869	0.0914	0.0794
Boolean accuracy $\uparrow$	0.9170	0.9184	0.9368	0.9376	0.9354	0.9549
Boolean micro F1 $\uparrow$	0.8243	0.8292	0.8646	0.8680	0.8510	0.8990
Categorical accuracy $\uparrow$	0.9223	0.9295	0.9237	0.9258	0.9287	0.9337

Table 4. Overall judge metrics by model, reasoning effort (low/high), and in-schema reasoning (w/o = without, w/ = with). GPT-5.2 is reported only with in-schema reasoning.

Model	Quality	$/M in	$/M out
gemini-2.5-flash-lite	17.57	0.10	0.40
claude-haiku-4-5 (deployed)	17.00	1.00	5.00
grok-4-1-fast	16.86	0.20	0.50
qwen3-80b	15.66	0.15	1.20

Table 5. Model ranking on the simple-complexity slice by composite quality (sum of six ordinal signals, max = 18).

Figure 3. Per-table accuracy by signal type (GPT-5-mini, high effort, with in-schema reasoning).

Metric	GPT-5-nano
Error rate $\downarrow$	0.2094
Hamming loss $\downarrow$	0.2094
Boolean accuracy $\uparrow$	0.8263
Boolean micro F1 $\uparrow$	0.6299
Categorical accuracy $\uparrow$	0.7233
Ordinal MAE $\downarrow$	0.6433

Table 6. Real-time context classification metrics for GPT-5-nano with minimal reasoning effort and in-schema reasoning, evaluated on the context_info signals (23 columns).

Configuration	Incon.	Filter%	Imp. Err $\uparrow$	Imp. HL $\uparrow$
GPT-5-mini low (w/o)	34	11.33%	$+$ 7.47%	$+$ 6.78%
GPT-5-mini low (w/)	9	3.00%	$+$ 2.22%	$+$ 2.09%
GPT-5-mini high (w/o)	11	3.67%	$+$ 0.86%	$+$ 0.46%
GPT-5-mini high (w/)	2	0.67%	$-$ 0.33%	$-$ 0.35%
GPT-5.2 low (w/)	17	5.67%	$+$ 4.56%	$+$ 4.38%
GPT-5.2 high (w/)	6	2.00%	$+$ 0.82%	$+$ 0.76%

Table 7. Cross-table consistency filtering. Incon. is the number of inconsistent records. Imp. columns show relative change after filtering (positive = improvement).

GPT-5-mini		GPT-5.2
Low	High	Low	High
7.3% (22 / 300)	3.7% (11 / 300)	2.0% (6 / 300)	0.0% (0 / 300)

Table 8. Meta-task confusion rate for schema-conforming judge runs (all with in-schema reasoning), by model and reasoning effort.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4. Full SEAR database schema including the four semantic evaluation tables and the gateway metrics table. Each table lists all columns with their types.

Signal	Type	Org A	Org B	Org C	All
safety_sensitive_content	bool	1%	0%	0%	0%
noisy	bool	16%	0%	0%	5%
persona_or_role_instruction	bool	100%	100%	100%	100%
reference_material	bool	90%	100%	100%	97%
tool_call	bool	1%	0%	0%	0%
code_task	bool	0%	0%	0%	0%

Appendix

Table 9. Ground-truth distribution of context_info signals (300 sessions, 100 per organization). Boolean signals show True%; categorical/ordinal signals show the dominant value and its percentage.

Signal	Type	Org A	Org B	Org C	All
tool_call	bool	0%	0%	0%	0%
code_task	bool	1%	0%	2%	1%
math_task	bool	0%	0%	0%	0%
stylistic_transformation_task	bool	22%	7%	90%	40%
information_extraction_task	bool	13%	0%	1%	5%
multistep_reasoning	bool	6%	14%	0%	7%

Appendix

Table 10. Ground-truth distribution of llm_response_info signals. Format follows Table 9 .

Signal	Type	Org A	Org B	Org C	All
hallucination	bool	0%	0%	0%	0%
tool_call	ord	n/a (99%)	n/a (100%)	n/a (100%)	n/a (100%)
code_task	ord	n/a (100%)	n/a (100%)	n/a (100%)	n/a (100%)
math_task	ord	n/a (100%)	n/a (100%)	n/a (100%)	n/a (100%)
stylistic_transformation_task	ord	n/a (74%)	n/a (93%)	none (100%)	n/a (56%)
information_extraction_task	ord	n/a (82%)	n/a (95%)	n/a (100%)	n/a (92%)

Appendix

Table 11. Ground-truth distribution of issue_attribution signals. Format follows Table 9 .

Signal	Type	Org A	Org B	Org C	All
appropriate	bool	100%	100%	100%	100%
verbose	bool	7%	17%	0%	8%
tool_call	ord	n/a (99%)	n/a (100%)	n/a (100%)	n/a (100%)
code_task	ord	n/a (100%)	n/a (100%)	n/a (100%)	n/a (100%)
math_task	ord	n/a (100%)	n/a (100%)	n/a (100%)	n/a (100%)
stylistic_transformation_task	cat	n/a (71%)	n/a (93%)	none (100%)	n/a (55%)

Appendix

Table 12. Ground-truth distribution of evaluation signals. Format follows Table 9 .

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 349 papers

RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing
Jun 17, 2026Guannan Lai, Haoran Hu, Han-Jia YeLLM Evaluation Pairwise Preference Evaluation
School of Artificial Intelligence, Nanjing University, China · 2National Key Laboratory for Novel Software Technology, Nanjing University, 210023, China · 3SinapisAI
SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing
Sep 28, 2026Vasilis Perifanis, Nikolaos Pavlidis, Symeon SymeonidisLLM Routing
Indigma Innovations · Democritus University of Thrace · Athena Research Center
LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
Aug 7, 2026Tao Feng, Fangxu Yu, Haozhen Zhang +9LLM Evaluation LLM Routing
University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2

Jun 17, 2026J/K move · Enter open · S save

RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing

Guannan Lai, Haoran Hu, Han-Jia Ye

School of Artificial Intelligence, Nanjing University, China · 2National Key Laboratory for Novel Software Technology, Nanjing University, 210023, China · 3SinapisAI

LLM Evaluation Pairwise Preference Evaluation Cost-Aware Inference LLM Routing

Open paper AskPDF arXiv

We present RouteJudge, an online pairwise preference evaluation framework for LLM routing systems, with a public platform available at https://routejudge.cn. Different from model-level response evaluation, RouteJudge focuses on router-level decision quality. For each user query, multiple routing strategies independently recommend candidate models under the same model pool and budget constraints. The selected model responses are then presented to users through anonymous pairwise comparisons, and the resulting user preferences are attributed back to the routing strategies behind the compared responses. Each evaluation record stores the query, routing decisions, model responses, preference labels, cost, latency, and task metadata, enabling preference-aware, cost-aware, and task-conditioned analysis of LLM routers. To support the continuous expansion of routing methods in RouteJudge, we further release ORBIT (Optimal Routing and Budgeted Inference Toolbox), a modular and extensible toolbox that standardizes the end-to-end workflow of LLM routing. ORBIT provides unified interfaces for benchmark loading, query representation, router implementation, budget-aware evaluation, and method comparison, allowing researchers to develop and evaluate routing algorithms under consistent protocols. It also serves as the submission and integration layer for RouteJudge: researchers can implement routing methods within ORBIT, validate them on existing routing benchmarks, and submit compatible routers for online preference-based evaluation. The code of ORBIT is available at https://github.com/LAMDA-Model-Reuse/ORBIT.

Abstract

LLM Routing

Explore similar work

RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing

SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers