LLM Guardrails

LLM guardrails assess prompts, generated responses, and multi-step agent actions for harmful content, policy violations, prompt injection, and unsafe tool use. Common approaches include specialized classifiers and LLM judges, hidden-state or logit probes, streaming monitors, and runtime controls such as policy-as-code, constrained decoding, and formal verification. Evaluations span moderation and jailbreak benchmarks such as HarmBench, agent-task settings such as AgentDojo and τ²-Bench, and domain-specific, multilingual, and long-context tests. The field’s central tension is balancing missed risks against over-refusal, latency, and cost, while determining whether strong benchmark scores translate into safer end-to-end systems.