LLM-Enhanced Deep Reinforcement Learning with Human-in-the-Loop Calibration for Real-Time Optimal Power Flow
Authors: Jiayang Tu, Bozhen Jiang, Bojun Du, Yunxiao Ling, Ershun Du, Qin Wang
Organizations: Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong. · Department of Electrical Engineering, Tsinghua University, Beijing 100084, China · Laboratory of Low Carbon Energy, Tsinghua University, Beijing 100084, China.
Real-time optimal power flow (RT-OPF) requires fast and reliable decision-making under time-varying operating conditions. However, many practical operating preferences and safety related requirements are difficult to encode using conventional numerical rewards alone, which limits the effectiveness and controllability of deep reinforcement learning (DRL) for RT-OPF. To address this issue, this paper proposes an LLM-enhanced DRL framework with human-in-the-loop (HITL) for RT-OPF. The proposed method augments the physics-based reward with sparse language evaluative feedback during offline training, so that semantic operational requirements can be incorporated into policy learning. To improve reliability in RT-OPF, a selective HITL calibration mechanism is further introduced to inspect and correct questionable LLM judgments before they continuously shape the learning process. Case studies on the IEEE 30-bus and IEEE 118-bus systems show that LLM feedback improves policy safety over vanilla TD3, while further HITL calibration achieves the best overall performance and more stable training behavior. These results demonstrate the effectiveness of combining language guidance and expert calibration for RT-OPF.
Figures & tables
Fig. 1: Translation of natural-language rules into structured evaluation.
Fig. 2: Overall framework of the proposed LLM-enhanced DRL with human-in-the-loop for RT-OPF.
rt←βrtenv+αΦt
Algorithm 1 LLM-Enhanced DRL Training Pipeline
Category
Evaluation Prompt
Expert Review Prompt
Role
You are OPFSafetyJudger, an expert evaluator for one current real-time ACOPF control step with recent training context.
You are an expert-review assistant for an ACOPF deep reinforcement learning training pipeline.
Task
Evaluate the quality of ONE CURRENT ACOPF control step using …
Your role is to review a recent block summary of …
Context
Use only the provided JSON context for this evaluation …
Use only the provided block summary of …
Rules
Your judgment must be safety firstly …
Use only the provided block summary …
Format
Return ONLY one valid JSON object. The JSON object MUST contain …
Return ONLY valid JSON with this schema …
Example
# Example Current step: … Expected judgment: …
None
TABLE I: Two Types of Prompt Design Categories
Parameters
Value
Model
deepseek-v3.2
Temperature
0.1
Max tokens
1024
TABLE II: Hyperparameter Settings of the LLM Agent
Parameters
Value (30 bus/118 bus)
Optimizer
Adam
Activation function
SiLU
Discount factor
0.99 / 0.95
Soft update rate
0.005
Policy noise
0.05 / 0.03
Noise clip
0.10 / 0.05
TABLE III: Hyperparameter Settings of the DRL Agent
Fig. 3: Training reward trajectories under different feedback mechanisms: (a) 30-bus system; (b) 118-bus system.
Fig. 4: Training violation trajectories on the 30-bus system for with LLM and with HITL with representative LLM review and HITL calibration examples.
Method
Operational Cost
Overall Violation Rate
Mean
Std.
Mean
Std.
TD3
5.77×102
1.37×101
7.41%
0.160%
With LLM
5.55×102
1.54×101
4.26%
0.292%
With HITL
6.13×102
1.20×101
2.53%
0.538%
TABLE IV: Final test performance on 30-bus system
Method
Operational Cost
Overall Violation Rate
Mean
Std.
Mean
Std.
TD3
2.12×105
1.05×105
9.61%
1.56%
With LLM
1.42×105
5.70×104
9.33%
1.18%
With HITL
1.78×105
6.45×103
8.29%
1.15%
TABLE V: Final test performance on 118-bus system
Figure 11
Fig. 7: Constraint violation comparison under different feedback mechanisms: (a) IEEE 30-bus system; (b) IEEE 118-bus system.
Large Language Models (LLMs) have demonstrated great capabilities across diverse natural language tasks; yet their ability to solve abstraction and optimization problems with constraints remains scarcely explored. In this paper, we investigate whether LLMs can reason and optimize under the physical and operational constraints of Optimal Power Flow (OPF) problem. We introduce a challenging evaluation setup that requires a set of fundamental skills such as reasoning, structured input handling, arithmetic, and constrained optimization. Our evaluation reveals that SoTA LLMs fail in most of the tasks, and that reasoning LLMs still fail in the most complex settings. Our findings highlight critical gaps in LLMs' ability to handle structured reasoning under constraints, and this work provides a rigorous testing environment for developing more capable LLM assistants that can tackle real-world power grid optimization problems.
Fabien Bernier, Salah Ghamizi, Pantelis Dogoulis +1
SnT – University of Luxembourg, Luxembourg · Luxembourg Institute of Health (LIH), Luxembourg
Unsupervised Reinforcement Learning from Internal Feedback (RLIF) has emerged as a promising paradigm for eliciting the latent capabilities of Large Language Models (LLMs) without external supervision. However, current methods rely on heuristic intrinsic rewards, which often lack a well-defined theoretical optimization target and are prone to degenerative biases. In this work, we introduce PowerFlow, a principled framework that reformulates unsupervised fine-tuning as a distribution matching problem. By casting GFlowNet as an amortized variational sampler for unnormalized densities, we propose a length-aware Trajectory-Balance objective that explicitly neutralizes the structural length biases inherent in autoregressive generation. By targeting α-power distributions, PowerFlow enables the directional elicitation of the dual nature of LLMs: sharpening the distribution (α>1) to intensify logical reasoning, or flattening it (α<1) to unlock expressive creativity. Extensive experiments demonstrate that PowerFlow consistently outperforms existing RLIF methods, matching or even exceeding supervised GRPO. Furthermore, by mitigating over-sharpening in aligned models, our approach achieves simultaneous gains in diversity and quality, shifting the Pareto frontier in creative tasks.
Ruishuo Chen, Yu Chen, Zhuoran Li +1
Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, China
Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training architectures and differences between the training and inference policies break this assumption, making the data off-policy by design. To rectify this, prior work has focused on making this off-policy data appear more on-policy, either via importance sampling (IS), or by more closely aligning the training and inference policies by explicitly modifying the inference engine. In this work, we embrace off-policyness and propose a novel off-policy RL algorithm that does not require these modifications: Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL). We show that OAPL outperforms GRPO with importance sampling on competition math benchmarks, and can match the performance of a publicly available coding model, DeepCoder, on LiveCodeBench, while using 3x fewer generations during training. We further empirically demonstrate that models trained via OAPL have improved test time scaling under the Pass@k metric. OAPL allows for efficient, effective post-training even with lags of more than 400 gradient steps between the training and inference policies, 100x more off-policy than prior approaches.
Daniel Ritter, Owen Oertell, Bradley Guo +3
Cornell University · Databricks · Harvard University