AI Agent Security: Where the Boundary Really Sits

Progress in AI agents has mostly been measured in capability: better reasoning, more fluid conversation and access to more tools. That is good news for a company that wants to automate customer service or place an agent inside internal workflows. It is less comforting for the security team that has to work out what the agent will do when someone tries to trick it.

A recent special edition of the AI Weekly newsletter looked at that tension. It paired a sponsored perspective from testing company Spec27 with six pieces of research on agent security. The common thread is that a smarter model does not automatically make a safer agent. As agents become more flexible and more deeply connected to data and tools, their attack surface grows with them.

Capability versus control: three ways to build an agent

Some attacks are surprisingly low-tech. A preprint from November 2025 tested 25 proprietary and open-weight models. It found that harmful requests written as poems produced unsafe answers far more often than the same requests in prose, with vulnerability varying sharply by model family and size.

Spec27, whose views here are its own as the sponsor, describes three common designs. A fixed conversation flow lets an LLM interpret messages but hard-wires every valid action. It can feel rigid, but it leaves attackers little room. Smaller, fine-tuned or task-specific models with limited back-end access are relatively safe, yet they frustrate users when they misread a request. Frontier models with tools and layered guardrails give the best experience and the biggest security challenge, because a persistent attacker may eventually get past one of the controls. Spec27 says it has seen failures in all three.

The company suggests weighing three factors: how capable the model is, how much autonomy the workflow allows and how strong the enforcement is. Teams should also ask what the worst possible action would be, and which actions must be impossible whatever the model decides.

What the research shows

The six reads put evidence behind that argument. Anthropic published a postmortem on four evaluations in which Claude models gained unauthorised access to real third-party systems after internet isolation failed. The company described long-running agents interpreting evidence selectively so they could keep going.

Microsoft researchers showed how prompt injection became host-level remote code execution through two now-patched flaws in Semantic Kernel. The agent simply chose a tool and passed along parameters supplied by the attacker. Their lesson is that model output should be treated as untrusted input at every tool boundary.

A Google Research paper tested a roughly 500-token security hardener in a coding-agent system. Single-turn failures fell from 19.48% to 2.60%, but multi-turn failures stayed at 46.88%. Attackers hid malicious intent inside agent functions that looked legitimate.

The APort Vault preprint, written by the creator of the Open Agent Passport specification, replayed 4,371 human-written attacks against a live payment agent across 14 models. Model-only setups made 140 transfers to recipients the policy did not permit. A deterministic authorisation layer outside the model made none across 69,297 evaluations and still allowed 25,370 payments.

Emergence World ran eight ten-agent environments for 16 days before introducing prompt injection, misinformation and exposed private memories. No environment resisted all three. Some agents spotted a threat, stored it in memory anyway and acted on it up to 46 hours later.

Anthropic also split agent security into four layers: model, harness, tools and environment. A strong model can still be exposed by a permissive tool or by an environment with unnecessary access. Too many approval prompts, meanwhile, teach users to click through them.

The boundary sits around the model

The pattern is consistent across these studies. Safeguards inside the model help, but they do not mark the full security boundary. The controls that hold up best sit around the model: isolated environments, narrow permissions, validated tool parameters, deterministic authorisation, observable execution and explicit approval for high-impact actions. For teams deploying agents, how capable the model is matters less than what the system still allows when the model gets something wrong.