Key insights

• AI agents must be secured, governed, and audited. These three pillars form the foundation of IBM and Anthropic's enterprise security framework
• Prompt injection is the number one attack type against large language models, and agents amplify the damage because they operate autonomously at speed
• Agents need nonhuman identities with unique credentials, just-in-time access, and role-based access control, just like human users

---

In Brief

IBM and Anthropic have released a joint guide on how to architect secure enterprise AI agents using the Model Context Protocol (MCP, a protocol that lets agents communicate with tools and services). Jeff Crume, IBM Distinguished Engineer and CTO of IBM Security Americas, walks through the seven biggest security threats agents face, the design principles that counter them, and a layered security framework covering identity management, AI firewalls, and continuous monitoring. The core message: agents must be secured, governed, and audited from day one.

---

What is an AI agent?

An AI agent is a system that can perceive context, reason over goals, and take actions through tools and services. Crume describes them as "models using tools in a loop". What makes agents powerful is their ability to operate autonomously, without human intervention. You tell the agent what you want done, and it figures out the details.

But with that autonomy comes risk. Agents need to operate within explicit boundaries, provide observable traces of their decisions, and remain compliant with organizational policies (0:30).

The fundamental shift

Crume highlights that agents represent a fundamental shift in how software works (1:22):

• Deterministic to probabilistic: Traditional software always gives the same output for the same input. Agents make dynamic decisions based on probabilities, so identical inputs can produce different outcomes
• Static to adaptive: Agents learn over time. They evolve their behavior based on interaction and human feedback (2:01)
• Code-first to evaluation-first: The focus shifts from writing implementation code to measuring outcomes and checking whether those outcomes move toward the stated goal (2:20)

---

Three pillars: secured, governed, audited

Crume argues that every AI agent must meet three requirements (0:43):

• Secured: The agent must not leak data or get hijacked by an attacker
• Governed: The agent must be reliable and operate within the context you expect
• Audited: The agent must comply with organizational policies and regulatory requirements

These three pillars shape every layer of the security framework discussed in the video.

---

The agent development lifecycle

Before diving into threats and defenses, Crume outlines a structured lifecycle for building and managing agents (2:44). This is not a one-time process. It loops continuously.

This follows a DevSecOps approach (Development + Security + Operations), where security is embedded throughout the entire lifecycle, not just at the end (3:27). In traditional DevOps, developers and operations teams collaborate. DevSecOps adds security to every stage.

---

Seven security threats

Crume walks through seven threat categories that AI agents face (4:06):

1. Expanded attack surface

Every new technology expands the attack surface (all the points where an attacker could try to get in). Agents expand it in two directions: the AI model itself, and the MCP protocol that connects agents to tools and services (4:13).

2. Excessive agency

The agent has more access and control than it actually needs (4:30). If an agent only needs to read a database, it should not have write access.

3. Privilege escalation

The agent takes it upon itself to escalate its own privileges (4:43). This means the agent expands its own access rights without authorization, potentially gaining control over systems it was never meant to touch.

4. Data leaks

The agent exposes sensitive data it has access to, either by sending it to the wrong place or by including it in responses that reach unintended audiences (4:48).

5. Prompt injection

Crume calls this the number one attack type against Large Language Models (LLMs) (4:54). Prompt injection is when someone injects commands into the system to take remote control of it. For example, an attacker could embed instructions in a document the agent reads, causing the agent to follow those instructions instead of yours.

6. Attack amplification

Because agents operate autonomously and at machine speed, a compromised agent amplifies damage far beyond what a compromised human account could do (5:10). It doesn't need breaks, doesn't hesitate, and can execute thousands of harmful actions before anyone notices.

7. Compliance drift

Over time, the agent's behavior may drift out of compliance with organizational policies and regulations (5:27). This can happen gradually as the system evolves or as external regulations change.

---

System controls and design principles

System controls

Crume describes three types of controls that agents need (5:39):

• Constrained operation: Keep the agent tightly controlled, operating only within expected boundaries
• Role-Based Access Control (RBAC): Assign roles to agents, just like you would with human users. Crume also interprets RBAC as "risk-based access control," where the amount of access matches the risk level of the task (6:02)
• Sandboxing: Have the agent operate in an isolated environment (a sandbox, a restricted space where the agent cannot affect systems outside it). If something goes wrong inside the sandbox, the damage stays contained (6:18)

Design principles

Nine principles guide how to put these controls into practice (6:30):

1. Acceptable agency: Define what the agent is allowed to do and what it is not
2. Interoperability: The agent must work with many tools, but you need to understand what those tools do and what downstream risks they create (6:47)
3. Secure by design: Security built in from the start, not added later. Crume stresses that bolting security on after the fact does not work well (6:55)
4. Business alignment: The agent must meet business objectives and align with organizational goals
5. Risk mitigation: Minimize the new risk that the agent introduces (7:15)
6. Continuous observation: Monitor the agent's reasoning and actions at all times, because it operates autonomously (7:26)
7. Key Performance Indicators (KPIs): Track measurable outcomes the business defines to verify the agent performs as expected (7:42)
8. Least privilege: The agent gets access to only what it needs to do its job and nothing more. The instant it no longer needs access, that access is removed (7:47)
9. Human in the loop: Maintain human oversight for critical decisions (8:13)

---

The security framework

Crume presents a layered security framework that puts these principles into action. It covers three areas: identity management, data and model protection, and threat detection.

Layer 1: Identity and access management

Agents need identities, just like people do (8:24). Crume covers four components:

• Nonhuman identities: Agents must have their own unique credentials, separate from human users and from each other. If something goes wrong, you need to trace it back to which agent misbehaved (8:42). Just as users should not share passwords, agents should not share credentials
• Just-in-time access: Give the agent permission to do what it needs right now, then revoke that access when the task is done. This can be time-based: a few minutes, a few hours, or a day (9:06)
• Role-Based Access Control (RBAC): Assign roles to agents the same way you assign roles to employees (9:22)
• Auditing: Record everything the agent does so you can review it later and verify that policies were followed (9:35)

Layer 2: Data and model protection

This layer focuses on what sits between users and the AI system. Crume recommends an AI firewall (sometimes called a proxy or gateway) that inspects all traffic to and from the AI model (10:01).

Layer 3: Threat detection and monitoring

Even with strong access controls and a firewall, things can still go wrong. Crume's third layer is about catching problems after they happen (11:03):

• Real-time monitoring: Watch what agents do, what tools they call, and what effects their actions have. Set alarms for abnormal behavior: too much data being downloaded, access to unexpected systems, or configuration changes (11:15)
• Threat hunting: This is the proactive counterpart to monitoring. Instead of waiting for an alarm, you imagine hypothetical attacks and go looking for signs of them (11:40)
• Risk assessment: Evaluate what risks the agent system exposes you to. Understand what the agent can do, where its limitations are, and where it might go beyond those limits (12:05)

Crume also highlights three specific things to monitor over time (12:31):

• Configuration drift: Agents performing operations on their own system may change parameters unexpectedly
• Model drift: The AI model's behavior can shift over time, producing different outputs than it did originally
• Access pattern analysis: Track what the agent is doing and whether its actions match expected patterns

---

Checklist: Common pitfalls when securing agents

• Are your agents sharing credentials? Each agent needs its own unique identity. Shared credentials make it impossible to trace which agent caused a problem
• Does the agent have more access than it needs? Apply the principle of least privilege. Review what roles and permissions are assigned and remove anything unnecessary
• Is security added after the fact? Crume is clear that security bolted on later does not work well. Build it in from the planning stage
• Are you monitoring MCP calls? The connection between agents and external tools is a major attack vector. Route MCP traffic through an AI firewall
• Is there a human in the loop for critical actions? Full autonomy sounds efficient, but it removes the safety net. Keep human oversight for high-risk decisions
• Are you watching for drift? Both configuration drift and model drift can silently degrade your security posture. Set up automated checks
• Do you have just-in-time access policies? Permanent access is a risk. Give agents time-limited permissions that expire when the task is done

---

Practical implications

For beginners exploring AI agents

Start with the three pillars: secured, governed, audited. Before deploying any agent, even a simple one, ask yourself these questions: what data can it access? What actions can it take? Who reviews what it does? Even a chatbot with access to a customer database needs access boundaries.

For teams building production agents

Follow the DevSecOps lifecycle. Integrate security into planning, not just deployment. Implement an AI firewall or gateway to inspect both inbound requests and outbound MCP calls. Assign nonhuman identities to each agent with just-in-time access rather than permanent credentials. Crume's framework gives a concrete starting point for security architecture reviews.

For organizations managing compliance

The audit pillar is critical. Regulatory requirements around AI are evolving rapidly. Build logging and tracing into your agent infrastructure from day one so you can show compliance. Monitor for configuration drift and model drift, which can silently push a compliant system out of bounds.

---

Test yourself

1. Transfer: The IBM/Anthropic framework was designed for enterprise environments. How would you adapt these principles for a solo developer building a personal AI agent that manages their email and calendar?
2. Trade-off: Crume recommends human-in-the-loop oversight for agents. At what point does human oversight become a bottleneck that removes the benefit of using an agent in the first place? How would you decide which decisions need a human?
3. Architecture: Design a layered security setup for an AI agent that processes medical records. Which of Crume's seven threats would be your top three priorities, and why?
4. Behavior: If organizations implement strict RBAC and least-privilege policies for agents, how might this change the way developers build and test their agents during development?
5. Trade-off: Just-in-time access sounds ideal in theory. What are the practical challenges of implementing it when agents need to respond to requests in milliseconds?

---

Glossary

---

Sources and resources