EvalRx: Agentic Diagnosis and Verified Repair of Foundation Models

Jiaqi Liu; Ruinan Jin; Yitong Tang; Beidi Zhao; Gexin Huang; Qiong Zhang; Xiaoxiao Li

cs.AIOct 19, 2026

EvalRx: Agentic Diagnosis and Verified Repair of Foundation Models

Authors: Jiaqi Liu, Ruinan Jin, Yitong Tang, Beidi Zhao, Gexin Huang, Qiong Zhang, Xiaoxiao Li

Organizations: The University of British Columbia · Vector Institute · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)

Abstract

Model evaluation reveals where models fail, but rarely why or whether a proposed fix genuinely works. We introduce EvalRx, an open-source agentic framework for statistically grounded model diagnosis and repair. EvalRx discovers systematic failure modes, explores candidate mechanisms, formulates falsifiable hypotheses, and verifies them by executing frozen, pre-specified tests before using supported diagnoses to guide tiered repair. Candidate repairs are validated against the unchanged model and promoted only when they improve the target behavior without unacceptable regression. Across language, vision-language, and audio-language settings spanning Gemini, Qwen, Gemma, and Nemotron, EvalRx improves performance in all 12 evaluated model family-modality combinations, with gains ranging from +1.4 to +10.9 percentage points (+5.8 points on average). These results show that model evaluation can serve not only to measure failures, but to support an auditable process for diagnosing and repairing them.

Figures & tables

Figure 1: Verification-gated model evolution. A candidate repair is promoted to the next model version only after its improvement is confirmed on held-out data. Otherwise, the current model remains unchanged and the diagnosis-repair cycle continues.

System family & examples	Starts with	What it delivers	What remains external
Auditing & slice discovery EvalTree, ErrorMap	Evaluation failures	Failure patterns or error slices	Held-out validation, repair, and post-repair audit
Propose-and-verify D3, VibeCheck, HypoGeniC	Candidate hypotheses	Tested hypotheses	Repair and version-level audit
Prompting, editing & retraining ProTeGi, TextGrad, GEPA; Claude Code ∗	A supplied target, reward, or task	Candidate repair	Failure diagnosis and a common verification rule
EvalRx	Labeled evaluation set	Verified repair, or an explicit inconclusive result	Nothing: re-audits the promoted version and recurses

Table 1: Where existing approaches stop, and where EvalRx continues. EvalRx connects failure discovery, statistical hypothesis testing, repair selection, and version re-audit in one evidence-gated loop. Examples are representative, not exhaustive.

Figure 2: Overview of EvalRx . EvalRx organizes model diagnosis and repair into five stages: M1 (Probe) discovers failures and gathers diagnostic evidence. M2 (Explore) analyzes failure structure and identifies candidate explanatory signals. M3 (Diagnose) formulates and freezes falsifiable hypotheses. M4 (Verify) tests these hypotheses with statistical evidence, and M5 (Repair) uses supported mechanisms to guide targeted repairs and verifies their improvement against the unchanged model.

Figure 3: M1 turns raw model outcomes into an auditable diagnostic signal table. A normalized case packet is routed to compatible analyzer families, assembled into a typed per-case signal matrix, and audited for coverage and missingness before M2.

Access and modality	Per-case diagnostic signals
Black-box, all modalities	Calibration, consistency under prompt or input contrasts, uncertainty, termination, response-format violations, and counterfactual outcome changes.
White-box language models	Token entropy, logits, attention, representation similarity, and logit-lens-style readouts.
Vision-language and audio-language models	Grounding, hallucination, attention or attribution over non-text tokens, modality ablations, local masking, and temporal or spatial perturbations.
Agentic trajectories	Loop detection, ignored-observation analysis, first-error localization, tool-use traces, and counterfactual trajectory tests.

Table 2: M1 selects analyzers that match the target model and task protocol. These representative analyzer families differ in availability across modalities and model interfaces.

Figure 4: M2-M3 turns observed failure structure into a falsifiable, frozen test contract. M2 records descriptive candidates; M3 challenges competing explanations and freezes the prediction, control, threshold, falsifier, and test design before M4 using them.

Figure 5: M4 admits a mechanism claim through statistical adjudication. Frozen test specifications and case identifiers enter a locked runner; statistical and protocol-consistency gates jointly route each hypothesis to supported, refuted, or inconclusive.

Tier	Evolution path	Space	Candidate treatment
L1	Non-parametric	Input	Prompt rewrites, explicit constraints, output formats, and evidence-use instructions.
L2	Non-parametric	Scaffold	Deterministic multi-call workflows, preprocessing, external tools, conditional execution, and output aggregation around the unchanged model.
L3	Non-parametric	Internals	Read-only logits or attention, or audited inference-time interventions such as attention reweighting or activation steering, without persistent weight updates.
L4	Parametric	Parameters	Dataset construction and fine-tuning or adapter updates. These create a new checkpoint and require explicit human execution or approval.

Table 3: The repair ladder makes intervention cost and parameter change explicit. L1-L3 are non-parametric evolution because the checkpoint weights remain frozen; L4 is parametric evolution. The tier constrains the intervention space, not repair quality.

Figure 6: M5 promotes only repairs that beat the unchanged baseline under a paired test. A supported hypothesis first receives a causal probe; repair search then starts at the least invasive permitted tier, freezes a candidate and its success rule, and compares it with the unchanged baseline on matched cases. The decision is to ship, stop, try the next tier, or return filtered data to M1.

Panel of Figure 10 — Algorithm 1: A guarded iterative version loop. A version can seed the next cycle only after frozen-contract diagnosis, held-out paired repair evaluation, and the approved intervention ceiling all pass.

Task	Models	Role
LLM	Gemini-2.5-Flash-Lite, Qwen3.5 2B, Gemma E2B, Nemotron-3-Nano-4B	Four text datasets, cross-scale and cross-family comparison.
VLM	Gemini-2.5-Flash-Lite, Qwen3.5 2B, Gemma E2B; Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8	Three visual datasets, cross-scale and cross-family comparison.
ALM	Gemini-2.5-Flash-Lite, Qwen3-Omni-30B-A3B-Instruct, Gemma E2B, Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8	Two audio datasets, cross-family comparison.

Table 4: A three-modality evaluation roster. The table lists the models in benchmark and the role of each LLM, VLM, and ALM track in evaluating EvalRx .

Figure 7: Verified repairs across the LLM benchmark track. Each panel compares baseline and post-repair performance; labels report the panel-specific change in percentage points.

Figure 8: Verified repairs across the VLM benchmark track. The figure compares baseline and post-repair performance using Claude Opus-5 as the base agent.

Figure 9: Verified repairs across the ALM benchmark track. The figure compares baseline and post-repair performance using Claude Opus-5 as the base agent.

Figure 10: LLM repair outcomes with a second agent backbone. The figure compares baseline and post-repair performance using Antigravity Gemini-3.1-Pro as the base agent.

Model	BBH (word sorting)	gsm8k	MMAU	AudioCaps-Hallucination	ChartQA	Spatial457
Gemma-4-E2B	$56.5\%$ $\Delta\,{+}3.2\%$	$87.6\%$ $\Delta\,{+}2.0\%$	$36.3\%$ $\Delta\,{+}20.3\%$	$56.6\%$ $\Delta\,{+}0.0\%$	$48.4\%$ $\Delta\,{+}21.9\%$	$18.4\%$ $\Delta\,{+}0.0\%$
Gemma-4-E4B	$62.0\%$ $\Delta\,{+}21.6\%$	$91.4\%$ $\Delta\,{+}0.0\%$	$65.3\%$ $\Delta\,{+}12.0\%$	$61.7\%$ $\Delta\,{+}16.7\%$	$58.3\%$ $\Delta\,{+}26.0\%$	$16.7\%$ $\Delta\,{+}22.7\%$
Gemma-4-12B	$82.4\%$ $\Delta\,{+}11.2\%$	$93.4\%$ $\Delta\,{+}1.2\%$	$63.0\%$ $\Delta\,{+}16.7\%$	$54.0\%$ $\Delta\,{+}18.7\%$	$73.7\%$ $\Delta\,{+}17.3\%$	$16.3\%$ $\Delta\,{+}19.3\%$

Table 5: Repair gains across the Gemma-4 scale range. The table reports baseline accuracy and post-repair improvement for each Gemma-4 model-task setting.

Figure 11: A complete, mechanism-verified repair loop on ChartQA. For Gemma-4-E2B, the L2 scaffold is promoted only after its hypotheses are supported under frozen tests and its paired outcome improves from

49.2\%

79.7\%

Figure 12: A mechanism-verified L1 repair on BBH (word sorting). For Nemotron-3-Nano-4B-FP8, the supported strict-copy intervention improves held-out accuracy from

12.0\%

25.6\%

Figure 13: An empirical repair that remains mechanistically unresolved on MMAU. For Gemma-4-E2B, the L2 scaffold improves held-out accuracy from

43.0\%

60.2\%

, but no mechanism reaches the supported verdict; the result is therefore not presented as a confirmed mechanism repair.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,425 papers

MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights
Jun 5, 2026Yilun Liu, Miao Zhang, Shimin Tao +9Multilingual Language Model Evaluation LLM Evaluation
Huawei, China
A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation
Sep 1, 2026Hodong Lee, Sanghee Park, Dohoon Ryu +4ML Reproducibility Multimodal Foundation Models
NAVER Cloud AI · Korea University · KAIST AI +1
Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training
Sep 30, 2026Kunlun Zhu, Xuyan Ye, Yibo Li +3Agent Failure Analysis LLM Agent Training
Apodex

Jun 5, 2026J/K move · Enter open · S save

MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights

Yilun Liu, Miao Zhang, Shimin Tao, Minggui He+8

Huawei, China

Multilingual Language Model Evaluation LLM Evaluation Cross-Cultural Language Model Evaluation LLM Agent Evaluation

Open paper AskPDF arXiv

Multilingual and multicultural benchmarks now cover dozens of languages and model families, but the resulting score landscapes remain metric-rich and insight-poor, necessitating fine-grained multilingual post-evaluation diagnosis. However, single LLMs and open-ended agents are easily swamped by the long, noisy diagnostic input, and no reusable taxonomy exists for it. To address this, we propose MADE, a Multilingual Agentic Diagnosing Engine that decomposes post-evaluation analysis into planning, aggregate analysis, instance-level case inspection, multilingual and cultural reflection, and grounded report synthesis. MADE is paired with an expert-led 54-query and 15-language diagnostic set, evaluated on top of a large-scale multilingual evaluation substrate (33 model families, 11 benchmarks, 26 languages, 34 cultures, 8.66M evaluation records). Experiments show that MADE outperforms the strongest shared baseline by 47% in diagnosis report quality and is preferred by human multilingual experts in 87.9% of pairwise comparisons. Applied with multilingual experts, MADE further surfaces four actionable findings on deployment, iteration, and cross-cultural pitfalls, turning benchmark score tables into model-selection and remediation guidance.

Abstract

LLM Evaluation

Explore similar work

MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights

A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation

Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training