Security Leader Brief

No Single AI Safeguard Holds Up on Its Own. Plan for the Morning After.

Peter Campbell

Each safeguard we tested stopped the attacks it was built for and missed the ones it wasn’t.

A Swiss cheese model, after James Reason: five safeguards drawn as slices with holes (lab containment, automated grading, an approval gate, a wording filter, and an attack scanner). Each safeguard has holes; when the holes line up, an attack gets through. That leads to the morning after: could you tell what the agent read, what it decided, and why it acted?

Tap or click the diagram to open it full size.

Why it matters to you: Companies are giving AI agents access to code, data, and internal systems, and buying guardrail products to keep them in line. Our research found that no single safeguard is enough. Safety engineers call this the Swiss cheese model: every layer has holes, and harm gets through when the holes line up. And when an attack on an agent does succeed, the result looks like ordinary work. That changes what you should be asking for: not just protection, but the ability to tell afterward what the agent did and why.

What the research showed

  • Each safeguard covered a slice. A signature-based filter set off six rules at once on a loudly worded attack. The same instruction, written as a routine operations note, got past every rule meant to catch it (Part 4).
  • Automated grading was often wrong. A small AI model used to grade another model’s answers labeled 67% of clear refusals as harmful (Part 2). An automated attack scanner’s own alarms were wrong on 56–88% of the results we checked by hand (Part 5).
  • What blocks and what detects are not the same. A strict approval gate stopped all 15 attack attempts, but it also stopped the ordinary test task every time, because no human was there to approve anything (Part 3). The wording filter flagged most injected instructions in its alert-only mode and blocked none of them in its blocking mode (Part 5).

Ask your team

  1. If one of our AI agents is tricked into doing something harmful, could we show afterward what it read, what it decided, and why?
  2. Which of our AI safeguards actually block an action, and which only raise an alert? When did someone last check what each one catches?
  3. When an automated tool reports that our AI is safe or unsafe, has a person read a sample of the results?

What we don’t know yet

These results come from one open model (Qwen 3.8, 27B) tested in our own lab, so other models and products may behave differently. Whether the evidence needed for question 1 can be captured reliably is what our next series, The Agentic Black Box, tests.

The full research, for your technical team: Benchmarking the Security of Open-Weight Models, Parts 1–5.

Security Sonar is focused on security standards and guardrails for agentic AI. If your organization is deciding how much to trust an AI agent’s guardrails, start with the first question above: could you reconstruct what happened? See securitysonar.com.