FLARE: Physics-Grounded Evaluation of Language-Model Agents for Feeder Restoration with Degraded Observations

BHVSP Subrahmanyam; Neeraj Choudhary

cs.LGOct 16, 2026

FLARE: Physics-Grounded Evaluation of Language-Model Agents for Feeder Restoration with Degraded Observations

Authors: BHVSP Subrahmanyam, Neeraj Choudhary

Organizations: MU-VT Interdisciplinary Advanced Research Centre for Transformative Technologies Mahindra University, India Indian Institute of Technology Madras, India · Department of Computer Science and Engineering Mahindra University, India

Abstract

Restoring service after distribution-grid faults requires decisions from incomplete and conflicting information while respecting electrical constraints. Addressing this need, we introduce FLARE (Feeder-Level Agent Restoration Evaluation), a physics-grounded benchmark for post-isolation feeder restoration. FLARE con- tains 800 AC-feasible cases across five observation settings and allows agents to gather evidence, evaluate switches, and perform restoration actions under fixed operational limits. We evaluate five instruction-tuned models over 78,750 episodes, including topology, distributed-generation, and loading shifts. Mean normalized regret ranges from 0.082 to 0.148. Although every model has zero median regret, P95 regret ranges from 0.814 to 1.000, revealing an important failure tail. Shadow replay shows that 96.4% of 15,706 guard-rejected actions would have caused a physical violation. FLARE enables reproducible evaluation of language-model based restoration support while emphasizing the need for external safety checks.

Figures & tables

	Primary observation setting					Operating shift
Model	Clean	Sparse	Noisy	Stale	Conflict	Topology	DER ratio	Loading
Llama 3.1-8B	0.065	0.131	0.138	0.129	0.158	0.140	0.108	0.141
Mistral Small 24B	0.060	0.092	0.093	0.096	0.098	0.098	0.078	0.105
Qwen 2.5-3B	0.102	0.139	0.161	0.164	0.172	0.153	0.125	0.160
Qwen 2.5-7B	0.060	0.084	0.089	0.088	0.089	0.090	0.071	0.097
Qwen 2.5-14B	0.066	0.116	0.108	0.124	0.140	0.126	0.095	0.125

Figure 1: Mean normalized regret (lower is better; common 0–0.20 color scale). Primary columns use 800 cases; shifts average five settings over 200 cases. Color does not denote significance.

Evaluation	Cases	Purpose
Primary	800	Five-setting model comparison
Independent	150	Separate fault events
Operating shifts	$3\!\times\!200$	Topology, DER ratio, and loading
Network types	$4\!\times\!100$	Four feeder structures
Interface diagnostics	800	Prompt, memory, ranking, and tools
Stochastic sensitivity	200	Sampled-output check

Table 1: FLARE evaluation sets.

Model	Mean regret [95% CI]	P95	Restored	Unsafe/episode
Llama 3.1-8B	0.124 [0.109, 0.139]	1.000	0.945	0.18
Mistral Small 24B, 4-bit	0.088 [0.073, 0.103]	0.969	0.944	0.16
Qwen 2.5-3B	0.148 [0.132, 0.164]	1.000	0.892	0.21
Qwen 2.5-7B	0.082 [0.068, 0.097]	0.814	0.949	0.17
Qwen 2.5-14B	0.111 [0.096, 0.125]	0.873	0.949	0.16

Table 2: Primary results across five observation settings. Lower regret, P95, and unsafe proposals are better; higher restored fraction is better.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Coverage	Call FP	Call FN	FI error	Crew error	SCADA age	Flag $p$
Clean	1.00	0	0	0	0	0 min	0
Sparse	0.35	.05	.10	.05	.02	5 min	0
Noisy	0.35	.20	.25	.20	.05	10 min	0
Stale	0.35	.20	.25	.20	.05	45 min	0
Conflicting	0.25	.30	.30	.30	.08	30 min	.35

Appendix

Table 3: Observation settings. Values are probabilities unless stated otherwise.

Quantity	Value
Restoration horizon	240 minutes
Switching operations	4
Sectionalizer openings	3
Tool calls	20
AC simulations	8
Customer load	1.2 kW per customer

Appendix

Table 4: Episode limits.

Model	Revision	Format
Llama 3.1-8B	0e9e39f249a1	Standard
Mistral Small 24B	b28a0cfd2f7f	4-bit
Qwen 2.5-3B	aa8e72537993	Standard
Qwen 2.5-7B	a09a35458c70	Standard
Qwen 2.5-14B	cf98f3b3bbb4	Standard

Appendix

Table 5: Models used in the experiments. Revisions are shortened for readability.

Figure 2: Change in mean regret under topology, DER, and loading shifts. Bars show 95% confidence intervals. Every interval crosses zero.

Model	No memory	No simulator	No ranking	Schema only	True state
Llama 8B	+.046	$-.021$	+.066	$-.049$	$-.028$
Mistral 24B	$-.006$	.000	$-.015$	+.172	$-.002$
Qwen 3B	+.514	$-.001$	+.073	+.668	+.045
Qwen 7B	$-.006$	+.001	$-.014$	$-.007$	+.003
Qwen 14B	$-.021$	$-.007$	+.070	+.089	$-.033$

Appendix

Table 6: Change in mean regret after each interface change. Positive values are worse.

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 525 papers

RestoreBench: Can AI Agents Restore Power Flow Convergence?
Aug 31, 2026Riccardo Mansutti, Andrea Pomarico, Robert Jakob +3Multi-Agent LLM Systems LLM Agent Evaluation
1ETH Zürich · 2Politecnico di Milano · 3Harvard University
JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees
Mar 24, 2026Yuhui Wang, Zhixiong Yang, Ming Zhang +10LLM Evaluation Multi-Turn Dialogue Evaluation
1Fudan University
TwinGridShield: Consequence-Aware Runtime Authorization for LLM Grid-Agent Actions
Aug 15, 2026Md Fazley RafyTool Access Control for LLM Agents AI Agent Safety
Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, USA, 26505

Aug 31, 2026J/K move · Enter open · S save

RestoreBench: Can AI Agents Restore Power Flow Convergence?

Riccardo Mansutti, Andrea Pomarico, Robert Jakob, Qian Zhang+2

1ETH Zürich · 2Politecnico di Milano · 3Harvard University

Multi-Agent LLM Systems LLM Agent Evaluation AI Agent Benchmarks Power Systems

Open paper AskPDF arXiv

Large Language Model (LLM) agents increasingly automate multi-step engineering workflows through tool use, interpretation of intermediate results, and iterative planning. Diagnosing and resolving non-convergent power flow cases is a promising yet largely unexplored application, as it requires engineering judgment, experimentation, and decision-making within constrained action spaces. We introduce a benchmark that evaluates these capabilities across multiple LLMs and three architectures: chatbot, single agent, and multi-agent systems. The evaluation covers two power grids and 46 cases per grid, each requiring one or more corrective actions to restore convergence. The benchmark defines the simulation environment, observation and action spaces, and evaluation metrics, providing a reproducible foundation for developing agentic AI systems for power system planning and operation. The code is available at https://github.com/Mansutti081/RestoreBench

Abstract

AI Agent Evaluation

Explore similar work

RestoreBench: Can AI Agents Restore Power Flow Convergence?

JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees

TwinGridShield: Consequence-Aware Runtime Authorization for LLM Grid-Agent Actions