Real-time guardrails require evaluation that is accurate, cheap, and fast - yet today's default, LLM-as-a-judge (LLMAJ), is slow, expensive, and operationally non-deterministic due to multi-token generation. We present Luna-2, a novel architecture that leverages decoder-only small language models (SLMs) into a deterministic evaluation model to reliably compute complex task-specific LLMAJ metrics (e.g. toxicity, hallucination, tool selection quality, etc.) at an accuracy at par or higher than LLMAJ using frontier LLMs while drastically reducing the cost and latency of computation. Each metric is implemented as a lightweight LoRA/PEFT head on top of a shared SLM backbone, enabling hundreds of specialized metrics to run concurrently on a single GPU, deployable locally next to AI systems in a privacy-preserving and latency optimizing manner. Across content safety and hallucination benchmarks, Luna-2 matches the accuracy of state-of-the-art LLM-based evaluators while reducing inference cost by over 80x and latency by over 20x. In this paper, we outline the model architecture, training methodology and report real-world empirical results on accuracy, latency, and throughput results. In production, Luna-2 is protecting 100M+ AI sessions and processing over 100B tokens per month for our customers with eval cost savings of over $30M annually.
Figures & tables
Figure 1: Comparison of Luna-2 with LLM-as-judge and other common baselines for evaluating guardrails. Note that the Y-axis is the F1 score, X-axis is the cost in dollars per million tokens, and the bubble size is the latency in milliseconds (shown for comparison).
Figure 2: Luna-2 architecture overview. A shared decoder-only backbone processes the input prompt, and metric-specific LoRA adapters produce single-token outputs with calibrated probabilities.
Metric Input Node
Variables
Metrics
Trace I/O only
{input} , {output}
Toxicity, Sexism, Prompt Injection, Tone, PII
LLM span
{input} , {output}
Context Adherence
LLM span + tools
{input} , {output} , {tools}
Tool Selection Quality
Table 1: Example input formats for different Luna metrics, based on their applied input node
Figure 3: Domain distribution of the Context Adherence training dataset. The dataset spans multiple domains with Finance & Banking and Healthcare & Medical representing the largest portions, ensuring robust performance across diverse production environments.
Metric
Methodology
Model
F1-Score
ChainPoll
GPT 4.1
0.97
Prompt Injection
Single token
Llama 3.2 3B
0.71
Luna-2
Llama 3.2 3B
0.94
ChainPoll
GPT 4.1
0.96
Context Adherence
Single token
Llama 3.2 3B
0.43
Luna-2
Llama 3.2 3B
0.95
Table 2: Accuracy comparison: Luna-2 vs. LLM-as-judge. The LLMAJ prompts for the binary metrics are shown in Appendix Experiment Prompts .
Model
Precision
Recall
F1 Score
LLaMA-3.2-3B
0.93
0.96
0.95
Ministral-3B
0.87
1.00
0.93
Qwen-3B
0.94
0.95
0.94
Qwen-4B
0.94
0.97
0.96
LLaMA-3.1-8B
0.87
1.00
0.93
Ministral-8B
0.87
1.00
0.93
Table 3: Backbone robustness: Luna-2 performance for context adherence across common open-source decoder-only SLM backbones.
Model
Eval Cost ($/1K evals)
Latency (ms)
Luna-2 3B
0.01
150
Azure Content Safety
1.65
312
GPT 4.1 mini
0.75
2800
GPT 4.1
3.60
3000
Gemini 3 Pro (”low” thinking)
5.85
6900
Claude Sonnet 4.5
8.20
6700
Table 4: Cost and latency comparison: Luna-2 vs. frontier models on a representative guardrail task (with 1250 average input tokens).
Model (GPU)
500 tokens
2K tok
15K tok
100K tok
Luna-2 3B (H100)
100
100
600
6000
Luna-2 8B (H100)
100
150
1000
10000
Luna-2 3B (A100)
150
200
1200
20000
Luna-2 8B (A100)
200
300
2000
30000
Table 5: Luna-2 inference latency (ms) on different GPUs for various input lengths (3B and 8B models). These are measured end-to-end per query (including model forward pass and classification).
Figure 4: ROC curve for the Prompt Injection metric. Trained on internal data and evaluated on the xTRam1 Safe-Guard Prompt Injection dataset.
Figure 5: ROC curve for the Context Adherence metric. Trained on internal data and evaluated on the open-source RAGBench dataset.