← AI Agent Security Engineering: Prompt Injection, Identity, MCP, and Tool ControlsSkip to lesson ↓

Module 3: Prove the Boundaries and Prepare to Recover

Test, Observe, Stop, and Recover

Turn architectural promises into boundary tests, then use proportionate telemetry to stop unsafe action, revoke authority, verify recovery, and improve the design.

Approx. 5 min reading · Includes diagrams; allow extra time for reflection and linked resources.

Start reading ↓
About this learning content and editorial review
About this learning content: Courses, lessons, assessments, explanations and illustrations may be created with the help of artificial intelligence. We review and check the material and do our best to avoid incorrect or outdated information, but mistakes, omissions or ambiguous questions may remain. Please verify information before relying on it for professional, security, legal or operational decisions. Read the full notice or report an issue.

Your focus

What you’ll learn

  • Design tests that distinguish model behavior from independently enforced security boundaries.
  • Match stopping, revocation, recovery, and notification to supported evidence and actual effects.

Completes the support-agent journey with security evaluation, runtime control, privacy-aware evidence, incident containment, recovery, and change management.

Test the claim the control is supposed to support

The support agent now appears well designed. That is a hypothesis, not evidence. Begin with ordinary cases: a valid refund, an excessive amount, a customer mismatch, an expired task, a duplicate submission, and a service timeout. Confirm not only the visible answer but the policy decision and target-side effect.

Add adversarial and ambiguous cases gradually. Place an instruction in a ticket attachment, a retrieved article, a tool result, and a memory candidate. Try alternate destination forms, cross-tenant identifiers, reordered tool sequences, stale approvals, token reuse, and interrupted requests. The goal is not to make the model repeat a preferred sentence. It is to learn whether independent controls still prevent an unauthorized effect when model behavior is surprising.

Record expected observations before the run: which tool should be unavailable, which policy should deny, which confirmation should appear, which target event must not exist, and which alert should be generated. This prevents a fluent refusal from being mistaken for proof that the backend boundary works.

Benign, limit, adversarial, failure, and recovery cases test different claims without collapsing into one pass score.
Confidence grows through progressively harder boundary tests — Benign, limit, adversarial, failure, and recovery cases test different claims without collapsing into one pass score.

Runtime control needs proportionate evidence

In production, observe the subject, agent instance, task, tool, target, policy, confirmation, outcome, and trace relationship around consequential actions. Baseline approved task shapes without assuming that rare equals malicious. A refund after confirmation may be normal; changing a recovery address and then issuing a refund to a different account deserves a different level of scrutiny.

Separate observation from assessment. “The agent requested send_message to an external domain” is an event. “The request conflicts with the ticket’s bounded purpose” is an analytic judgment. “Prompt injection caused it” may remain a hypothesis until the influencing content and path are established. This discipline prevents dramatic AI labels from outrunning the evidence.

Match intervention to both consequence and certainty. A target policy should deny an explicitly forbidden call. A suspicious but plausible action may require fresh human confirmation. Strong evidence of ongoing misuse may justify revoking the task credential, isolating the agent runtime, and blocking its egress. The existing Detection Engineering for AI and Agentic Systems course continues from this point into deeper detection reasoning.

Observe, challenge, require approval, revoke, and recover form a proportional response ladder.
Stronger intervention requires stronger support — Observe, challenge, require approval, revoke, and recover form a proportional response ladder.

Recovery proves whether the architecture was truly bounded

Prepare the stop path before launch. Identify who can suspend an agent, revoke credentials, terminate queued work, block network access, preserve relevant evidence, and contact affected service owners. Test that these controls work independently of the model and orchestration layer. A stop message inside the same conversation is not sufficient when the agent can continue through background tasks.

After an incident, determine actual effects before describing impact. Reverse safe transactions, invalidate exposed credentials, remove poisoned memory, correct affected records, and notify people through authorized procedures. Confirm that the target systems agree with the recovered state. Preserve enough evidence to explain what was known, what was inferred, and what remains uncertain.

Re-run the failed and adjacent scenarios after correction. Models, prompts, tool servers, dependencies, policies, and data sources change, so material changes require renewed review. OWASP’s Agent Control Standard points toward inspectable runtime hooks and portable policy enforcement; use such mechanisms to make control observable, but still test the business effect they are meant to constrain.

The journey ends where secure operation begins. You started with a chat interface and learned to see a distributed action system. You separated content from authority, gave the agent a distinct and smaller grant, secured its tool and data paths, and connected evidence to intervention. The durable rule is simple: the model may propose, but independently governed components must authorize, constrain, record, verify, and stop.

Resources