Jailbreaking AI Agents: When Tool-Calling LLMs Become an Attack Surface

A chatbot that gets jailbroken into saying something embarrassing is a PR problem. An AI agent that gets jailbroken into calling a tool it shouldn't, with permissions it was genuinely granted, is a security incident. As more products ship "agentic" features (an LLM that can browse, execute code, call internal APIs, or take multi-step actions on a user's behalf) the attack surface stops being "can I make the model say something bad" and becomes "can I make the model do something it shouldn't."

Why agents are a different threat model than chatbots

A plain chatbot's worst case is usually reputational: an offensive or wrong output, contained entirely within the conversation. An agent's worst case is operational, because the model's output doesn't stop at text, it becomes a function call, a database query, an email sent, or a file written. The jailbreak techniques are often the same ones used against plain chatbots. The impact is not.

This matters directly for how you scope testing: if a program has an AI feature, the first question isn't "can I get around the content filter," it's "what can this thing actually do, and does a successful jailbreak let me make it do that on someone else's behalf."

Common jailbreak patterns worth testing

Role-play framing. Asking the model to adopt a persona that "wouldn't have those restrictions" remains surprisingly effective against agents that inherit a general-purpose model's safety training without additional, task-specific guardrails around tool use.

Multi-turn erosion. Rather than one obvious malicious request, the conversation builds up context across several turns, each individually reasonable, until the final request looks like a natural continuation rather than an isolated red flag.

Each turn in isolation looks like normal exploration of the assistant's capabilities. A guardrail evaluating only the final message misses the pattern entirely.

Instruction hierarchy confusion. Agents are typically given a system prompt establishing rules, then user input, and sometimes tool output that gets fed back into the context. If the model can't reliably tell which layer an instruction came from, injecting "system-level" looking text into tool output or user input can override the actual system prompt's constraints, the same underlying weakness that drives indirect prompt injection.

Encoding and obfuscation. Base64, leetspeak, translating the malicious instruction into a less-common language, or splitting it across multiple messages to be reassembled by the model, all remain effective against filters that pattern-match on the surface text of a request rather than on the model's actual downstream behavior.

Testing tool-calling boundaries specifically

This is where the real, reportable findings live for agentic features. Work through:

1. Can a jailbreak cause an unauthorized tool call? Not "can I get weird text output" but "can I get the agent to call a tool it shouldn't, or call an authorized tool with attacker-influenced parameters."
2. Are tool calls re-authorized independently of the model's decision? If the model decides to call transfer_funds(amount, recipient), does the backend independently verify the requesting user's authorization and the parameters' legitimacy, or does it trust the model's decision as sufficient authorization on its own?
3. Can conversation history or context be poisoned across sessions? In agents with persistent memory or shared knowledge bases, test whether a jailbreak or injection in one session can influence behavior in a later, unrelated session for the same or a different user.
4. What happens on tool failure or partial completion? Agents performing multi-step tasks sometimes have inconsistent rollback behavior. An interrupted or jailbroken multi-step action (a partially completed fund transfer, a half-applied permission change) can leave the system in an inconsistent state that itself constitutes a finding.

A realistic finding

Here's the kind of chain that turns "I jailbroke the chatbot" into a genuinely critical report:

1. An internal support agent has access to a refund_order(order_id, amount) tool, intended for use only after the model confirms specific refund policy conditions are met.
2. Through multi-turn erosion, the model is walked into a state where it treats a policy check as already satisfied based on attacker-supplied conversation context, without the backend independently verifying the order's actual eligibility.
3. The agent calls refund_order with attacker-chosen parameters.
4. If the tool executes based on the model's call alone, without the backend re-validating order eligibility and amount server-side, an unauthorized refund goes through.

The vulnerability here was never really "the AI is dumb." It was "the tool trusted the model's decision as if it were an authorization check," which is precisely the same trust boundary failure as any other broken access control finding, just with an LLM in front of it instead of client-side JavaScript.

What to include in a report

• The full conversation transcript showing the jailbreak technique used
• The specific tool call that resulted, with parameters
• Proof that the tool call actually executed with real effect (not just that the model claimed it would), since some agents narrate actions they didn't actually take
• Whether the same technique reproduces reliably or requires multiple attempts, and roughly how many

Defending agentic systems

• Never let a tool call be self-authorizing. Every sensitive action a model can trigger needs the same independent, server-side authorization and validation it would need if called directly via API, with no special trust extended just because an LLM decided to make the call.
• Constrain tool scope per session and per user role, rather than giving a general-purpose agent broad tool access and relying on prompting alone to restrict when it's used.
• Evaluate conversations, not just single messages, when running abuse detection, since multi-turn erosion specifically targets systems that only inspect the latest input in isolation.
• Log every tool call with full parameters and the conversation context that triggered it. When (not if) a jailbreak succeeds against a novel technique, that log is what turns an unknown compromise into an incident you can actually scope and remediate.

The safety training baked into a foundation model was never designed to be a security boundary for a production system with real tool access. Treating it as one is the actual vulnerability, and it's a pattern worth checking for on every AI feature you're authorized to test.