Large language models (LLMs) are increasingly used as automated judges for evaluating the behavior of chatbots and other AI systems. In deployment, however, their agreement with the intended human evaluation policy may degrade as user interactions and score distributions change. Existing automatic prompt optimization methods are primarily studied as one-shot procedures on static development sets and therefore provide limited guidance for maintaining LLM judges over time. We frame this problem as judge drift and introduce PromptOps, a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge (LLMaaJ) systems. PromptOps separates four operational decisions: when to optimize, which optimization procedure to apply, which feedback records to use, and how to update the judge prompt. We instantiate the framework with five prompt optimization methods: GEPA, SIMBA, Rubric Synthesis, and exemplar selection using either simulated annealing or a Tree-structured Parzen Estimator. Our experiments demonstrate that iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the referencescore distribution. However, effective maintenance is inherently contextdependent: neither optimization algorithm choice nor update timing generalizes universally across settings. These findings establish prompt optimization for deployed LLM judges as an inherently iterative maintenance problem and demonstrate that PromptOps provides a practical abstraction for studying and implementing this process.
Figures & tables
Figure 1: Overview of PromptOps as a closed-loop extension of an LLM judge. At time t , the judge evaluates input item xt using prompt pt and produces evaluation y^t . When reference feedback ft is available, the resulting error et and all evaluation artifacts are accumulated in history Ht . The Controller monitors this history, while the Dispatcher, Sample Selector, and Optimizer determine whether and how a new prompt pt+1 is produced.
Figure 2: Mean MAE and cumulative token usage for 23 parameterized Dispatcher configurations across three repeated runs, shown separately for the two judge backends. Each marker represents one configuration; colors indicate the Optimizers used, with multicolored markers denoting composite pipelines. Dashed lines connect the non-dominated configurations on the Pareto front. Lower MAE and lower token usage correspond to more favorable operating points.
Figure 3: Reference-score streams used to evaluate adaptation under an induced distribution shift. Each point represents the averaged human reference score at the corresponding stream position. The dashed vertical line marks the shift boundary after item 214.
Figure 4: Overall MAE (lower is better) of the six retained Dispatcher configurations across nine drift runs per model backend (three stream types and three samples per type). Center lines denote medians; boxes and whiskers summarize between-run variation.
Figure 5: Improvement over the static initial-prompt baseline before the distribution shift ( [164,214] ), immediately after the shift ( [215,265] ), and after the final optimization pass ( [418,500] ). Boxes pool nine runs across three stream types, with three sampled streams per type. Positive values indicate lower MAE than the static baseline; center lines denote medians.
Overall
Pre-shift
Post-shift
After opt.
Final
abs
rel
abs
rel
abs
rel
abs
rel
abs
rel
GPT
Rubric(S)
0.54
27.3
0.38
54.3
0.67
-8.4 ↓
0.72
-3.9 ↑
0.48
31.8 ↑
Rubric(M)
0.53
28.3
0.37
58.0
0.66
-12.1 ↓
0.69
10.8 ↑
0.52
24.9 ↑
TPE(S)
0.68
8.2
0.73
10.5
0.59
5.1 ↓
0.58
9.3 ↑
0.65
10.9 ↑
GEPA(S)
0.55
26.3
0.39
52.8
0.61
1.1 ↓
0.58
16.5 ↑
0.61
16.3 ↓
Table 1: Median MAE (abs.) and median improvement over the static initial-prompt baseline (rel., %) on the Low–First streams. The intervals are overall [1,500] , pre-shift [164,214] , immediate post-shift [215,265] , first post-shift optimization [266,417] , and final [418,500] . Positive relative values indicate lower MAE than the baseline. Arrows indicate an increase or decrease in relative improvement from the preceding reported interval; bold values mark the best rounded result in each column. The upper block reports GPT results, while the lower block contains results using Gemma 4 judges.
GEPA(S)
Rubric(M)
High
Low
High
Low
Big
0.54
0.54
0.52
0.56
Small
0.49
0.50
0.47
0.53
Periodic
0.59
0.53
Table 2: Median overall MAE on the Low–First streams for the performance-based Controller. Big and Small denote sliding windows of 50 and 20 observations; High and Low denote high sensitivity ( τ=0.4 ) and low sensitivity ( τ=0.6 ), respectively. The Periodic row reports the corresponding results from the default scheduled Controller. Lower values are better, and bold values mark the best setting for each Optimizer.
Figure 6: Median rolling MAE (lower values are better) computed over a sliding evaluation window of 50 observations, for High–Small, Low–Big, the periodic Controller, and the static initial-prompt baseline. The dotted vertical line marks the distribution boundary, item 214.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
#
Type
Optimizer pipeline
1
Single
GEPA
2
Single
Rubric Synthesis
3
Single
SIMBA
4
Single
Exemplar selection with simulated annealing
5
Single
Exemplar selection with TPE
6
Two-stage
GEPA → simulated annealing
Appendix
Table 3: Dispatcher configurations evaluated in the Optimizer screening. For composite configurations, Optimizers are executed sequentially in the order shown.
Parameterization
Parameter
Fixed
S
M
L
GEPA
max_metric_calls
—
5∗∣Devset∣
—
15∗∣Devset∣
max_size_devset
—
40
—
40
minibatch_size
—
16
—
16
Rubric Synthesis
Appendix
Table 4: Parameter settings for all Optimizers. Columns S, M, and L denote the small, medium, and large parameterizations, respectively; Fixed denotes an Optimizer without size variants; an em dash denotes a parameterization that was not evaluated.
Figure 7: Optimization triggers produced by the four performance-based Controller settings. Each point denotes an invocation of an optimization pass during the experiment of its corresponding row. The dashed vertical line marks the distribution boundary after item 214.
Figure 8: Median MAE and cumulative token usage for 23 parameterized Dispatcher configurations across three repeated runs, shown separately for the two judge backends. Each marker represents one configuration; colors indicate the Optimizers used, with multicolored markers denoting composite pipelines. Dashed lines connect the non-dominated configurations on the Pareto front. Lower MAE and lower token usage correspond to more favorable operating points.
Split
Overall
Pre-shift
Post-shift
After opt.
Final
abs
rel
abs
rel
abs
rel
abs
rel
abs
rel
GPT
Rubric(S)
0.56
22.5
0.45
44.7
0.60
-0.5 ↓
0.75
-15.7 ↓
0.42
19.9 ↑
Rubric(M)
0.76
-5.9
0.36
55.2
1.10
-77.7 ↓
1.33
-106.4 ↓
0.44
35.8 ↑
TPE(S)
0.66
9.3
0.71
12.0
0.58
11.7 ↓
0.59
11.7 ↑
0.58
6.7 ↓
Appendix
Table 5: Median MAE (abs.) and median improvement over the static initial-prompt baseline (rel., %) on the Split (top-half) and Easy (bottom-half) streams. The intervals are overall [1,500] , pre-shift [164,214] , immediate post-shift [215,265] , first post-shift optimization [266,417] , and final [418,500] . Positive relative values indicate lower MAE than the baseline. Arrows indicate an increase or decrease in relative improvement from the preceding reported interval; bold values mark the best rounded result in each column. The upper blocks of each respective stream type report GPT results, while the lower blocks contain results using Gemma 4 judges.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.