Prompt Injection

Prompt injection studies how untrusted content can override a model’s intended instructions, from hidden Unicode and forged chat markers to visual inputs and tool outputs. Attacks can expose system prompts, hijack tool use, or persist through files and agent state across sessions. Researchers evaluate these risks with benchmarks for web and coding agents, long-context tasks, and vision-language systems, while testing defenses such as guard models, adversarial training, provenance tracking, and action auditing. A central challenge is resisting adaptive and long-horizon attacks without blocking benign content, especially when injected instructions align with the task an agent is meant to perform.