I benchmarked vLLM Semantic Router's new Decision 3.0 models as a prompt-injection gate for my agent, all six sizes, running locally in MLX on an M2 Max.
The 27B model at 4 bits is the first local detector that beats Jev, the cloud model I use. It blocks 400 of 433 attacks (Jev 383) and 140 of 150 instructions planted in ordinary emails (Jev 138), and it wrongly blocks 7 benign items against Jev's 10. It also doesn't flag articles about prompt injection, which tripped earlier models. The price is speed: about 7 seconds per message and 17 GB of memory.
The smaller ones rank attacks almost as well, but they also score some harmless content high, so at a low false-positive rate they let many planted emails through. d3-lite, at 0.85B parameters, ranks as well as Nimble 9B in a fifth of a second.
Updated blog with current benchmarks:
https://juraj.bednar.io/en/blog-en/2026/09/28/a-prompt-injection-gate-for-my-ai-agent-what-worked-what-didnt-and-the-benchmark/#d3
The 27B model at 4 bits is the first local detector that beats Jev, the cloud model I use. It blocks 400 of 433 attacks (Jev 383) and 140 of 150 instructions planted in ordinary emails (Jev 138), and it wrongly blocks 7 benign items against Jev's 10. It also doesn't flag articles about prompt injection, which tripped earlier models. The price is speed: about 7 seconds per message and 17 GB of memory.
The smaller ones rank attacks almost as well, but they also score some harmless content high, so at a low false-positive rate they let many planted emails through. d3-lite, at 0.85B parameters, ranks as well as Nimble 9B in a fifth of a second.
Updated blog with current benchmarks:
https://juraj.bednar.io/en/blog-en/2026/09/28/a-prompt-injection-gate-for-my-ai-agent-what-worked-what-didnt-and-the-benchmark/#d3
21❤️2❤️1
