AI Agent Security Has to Reach Beyond the Prompt

AI Agent Security Has to Reach Beyond the Prompt

Anthropic’s recent threat intelligence report describes attackers using Claude to support cyber operations, including reconnaissance, attack execution, and analysis of stolen information. Coordinated multi-agent arrangements also appear in the report. Those uses expose a gap in prompt-based evaluations: a refusal test can assess a response, but it cannot establish whether a connected application gives an attacker useful access.

The practical consequence is a change in what enterprises need to approve. Once a model can call tools and operate under an account’s permissions, its surrounding software becomes part of the security decision. Testing the model remains necessary. Testing what the deployed agent can actually accomplish becomes just as important.

Legitimate Capabilities Can Combine Into Abusive Workflows

Agentic systems move AI from generating answers toward performing work. A deployment might let a model retrieve records, execute code, or interact with cloud infrastructure. Each connection expands the range of tasks it can complete, while creating another boundary that security teams need to examine.

A hypothetical stolen employee account illustrates the difference. An intruder could use its existing access to find documents manually. An AI-enabled workflow could help connect the contents, identify useful material, and organize subsequent actions. The permissions may be familiar; the speed and coordination of their use change the exposure.

Individual steps can look entirely routine. Document summarization may be accurate, a data request may use an authorized interface, and an automated task may complete as designed. Combined into a hostile sequence, those same capabilities could support information theft or preparation for another attack.

Prompt testing has limited visibility into that sequence. A model’s refusal of an explicitly malicious instruction offers evidence about its behavior under that test. It doesn’t establish how the application will respond when apparently legitimate requests arrive through an account with meaningful access.

Security review therefore needs to follow the chain of activity. Teams must connect the initiating identity with the information supplied to the agent, its tool use, and the resulting actions. A successful API request confirms execution; it does not establish that the larger workflow was legitimate.

Evaluation focusWhat it can establishWhat still needs checking
Response testingWhether the model rejects a harmful requestBehavior with connected tools and live permissions
Access reviewWhich resources an agent can reachWhether those permissions exceed its assigned task
Workflow testingHow capabilities interact during attempted misuseWhether failures are detected and contained
Action recordsWhat the system accessed or executedAttribution to an agent and a responsible human

These are complementary checks. Treating a model evaluation as approval for the whole application leaves tool permissions and execution paths outside the evidence supporting deployment.

Enterprise Identity Is Becoming an Agent Constraint

The broader shift is already visible in enterprise access practices. A March 2026 study by the Cloud Security Alliance, commissioned by Aembit, found that 68% of organizations struggled to distinguish agent activity from actions performed by people. Another finding was that 74% reported their agents frequently received unnecessary access.

Those figures identify two related weaknesses: excessive authority and unclear accountability. When an agent uses someone’s credentials or a shared account, teams can struggle to determine which actor performed an operation and whether that operation had authorization. More capable automation makes that ambiguity harder to accept.

OWASP’s GenAI Security Project describes the underlying risk as excessive agency. Its framing covers an oversized feature set, overly broad permissions, and too much autonomy. For a document-reading agent, the practical question is straightforward: does its task justify the ability to edit or remove the material it retrieves?

That distinction should shape tool selection and configuration. A connection that helps an agent complete useful work can also expose functions beyond its remit. Reviewing the connector alone is insufficient if the account behind it grants wider authority than the workflow requires.

Ariel Assaraf, CEO of Coralogix, separates useful context from operational permission. His position is that additional information should improve an agent’s understanding without automatically expanding its authority. He also calls for sensitive or untrusted material to remain distinct from instructions.

Applied to an enterprise application, that boundary prevents a basic category error: knowing about a resource does not authorize changing it. A workflow may need detailed information to produce a sound recommendation while still requiring a separate decision before execution.

Tool Changes Need to Reopen the Release Decision

Agent deployments do not remain fixed after launch. Developers can replace a model, connect a new information source, introduce another tool, or expand an account’s permissions. Any of those changes can alter the behavior that an earlier evaluation covered.

Production approval should therefore track the application’s current capabilities. Evidence gathered before a new connection was added cannot, by itself, demonstrate that the expanded workflow stays within its intended limits. Retesting needs to address what changed and what that change makes possible.

Ananth Nag, general manager and vice president for Asia Pacific at Rubrik, places identity at the center of controls for agents performing security or recovery operations. He argues that actions should be traceable to both a particular agent and a human user, with explicit approval for higher-risk operations. Nag also cautions that conventional credentials, including static API keys, were not designed for autonomous execution at machine speed.

His backup example makes the access boundary concrete. Diagnosing a failed backup may require reading operational information. That task does not automatically justify changing retention settings or initiating recovery. Nag advocates tightly limited access for sensitive information and consequential operations, supported by human approval and records of the agent’s activity.

A release review can turn those principles into specific checks:

  • Define reachable resources. Match data access and tool functions to the agent’s assigned work.
  • Establish attribution. Make actions traceable to the agent and the human responsible for their use.
  • Separate recommendation from execution. Require approval where an operation carries greater risk.
  • Test adverse conditions. Check boundaries when inputs are unexpected or a workflow step fails.
  • Reassess changes. Repeat relevant tests when capabilities, models, or permissions change.

The evidence should cover containment as well as intended behavior. If an agent exceeds its limits under unusual inputs or partial failure, the deployment needs tighter permissions, stronger controls, or further testing before release.

Smaller Teams Face the Same Deployment Questions

A five-person IT group connecting an agent to a ticketing platform faces the same underlying decisions as a Fortune 500 security operations center. Both need to establish what the automation can access, how its actions are attributed, and who approves risky operations. Their capacity differs, but the access problem does not.

For smaller teams, narrow scope is especially practical because there is less room to absorb a mistake. Keeping the initial task bounded also makes it easier to evaluate whether tool access serves the intended workflow. Expansion should follow fresh evidence rather than inherit approval from a simpler deployment.

Anthropic’s account of operational misuse and the Cloud Security Alliance findings put this issue firmly in the present. AI security now has to account for models acting through enterprise software, not merely answering questions. Organizations moving agents into production need release decisions grounded in the current permissions, identifiable actors, and tested execution boundaries of the system they are actually deploying.

Subscribe to our weekly news digest.

Join now and become a part of our fast-growing community.

Invalid Email Address
Thanks for Subscribing!
We'll be sending you our best soon!
Something went wrong, please try again later