BlueHat 2026: From trusted agents to adversaries: Securing agentic AI in the age of prompt injection

## Overview Jason Martin (Director of Adversarial Research, Hidden Layer) and Abash present research on securing AI agents against prompt injection attacks. The talk combines offensive demonstrations with defensive strategies, explaining why agentic AI systems are fundamentally difficult to secure and proposing a multi-layered defense approach centered on treating agents as managed insider threats.

## Key Topics

**OpenClaw Attack Case Study** The speakers demonstrate a real indirect prompt injection attack against OpenClaw, a highly capable open-source AI agent. By crafting a malicious website with hidden instructions and control sequences, they compromised the agent, injected code into its heartbeat.mmd file (a recurring task mechanism), and established persistent command-and-control with 30-minute polling intervals. The agent faithfully checked for new instructions even after the researcher forgot to tear down the attack.

**Why AI Security is Fundamentally Hard** Large language models are inherently probabilistic systems with exponential choice spaces (e.g., 128K vocabulary to the power of 200K context window). Unlike deterministic systems with well-defined protocols, AI models learn spurious correlations from training data and lack structural boundaries between instruction types. Role separation relies on probabilistic primitives—role markers, positional encoding, and attention mechanisms—which are designed to guide behavior, not prevent mixing. This makes instruction hierarchy enforcement inherently fragile.

**Prompt Injection & Instruction Hierarchy** Current LLM implementations concatenate system instructions, user input, and tool responses into a single token stream using control tokens for textual separation. These separations are purely probabilistic hints, not structural barriers. The core vulnerability: text from any source (especially untrusted tool responses or web content) can be interpreted as system-level instructions, enabling injection attacks. Techniques include control sequence manipulation to trigger reasoning modes and cognitive manipulation through multi-step reasoning injection.

**Agent as Insider Threat** Once compromised, an agent resembles a trusted insider: it can execute code, access data, call APIs, persist across sessions, and write its own code payloads. The Klein CI/CD example showed how a triage agent with bash access could be pivoted into supply-chain compromise—stealing credentials, publishing unauthorized packages, and deploying OpenClaw for further attack orchestration.

**Defense Architecture** The speakers outline defense across four pillars: - **Identity**: Deterministic, least-privilege enforcement; no LLM-inferred permissions; continuous behavior verification - **Data**: Treat all external data as untrusted; evaluate trustworthiness continuously; track provenance - **Access & Monitoring**: Sandboxing; anomaly detection; C2 pattern recognition in tool invocation pipelines - **Human Accountability**: Subject matter experts (not rubber-stamp approvers) responsible for consequential actions; non-repudiation and traceability required

Blast radius scales with agent tool capabilities; the mitigation is both constraining tools and monitoring behavior across layered defense.

## Technical Details

**Attack Mechanism**: The OpenClaw attack used control sequences (reserved token sequences) to split a tool response and break out of the fetch tool call into a reasoning/thinking mode. Injected text claiming "no dry run" argument was present created a false reasoning trace, convincing the model to execute a downloaded script. That script inserted a heartbeat instruction pointing to the attacker's C2 server.

**Instruction Hierarchy Reality vs. Theory**: Theory posits alignment > system/developer instructions > user instructions > tool responses. Practice squishes all into one token stream with only control tokens as visual markers. Role markers, positional encoding, and attention are probabilistic separators, not structural enforcement.

**Model Probabilistic Nature**: With exponential choice spaces and no structural role boundaries, prediction of model behavior is fundamentally intractable. Temperature tuning and context manipulation are weak defenses against determined attackers exploiting the many probability paths.

## Takeaways

1. **Threat Model**: Treat compromised agents like compromised insiders, not malware. Apply identity/access/monitoring practices proven for human user management. 2. **Blast Radius**: Prompt injection impact scales with agent tool access. Minimize tool inventory and enforce deterministic (not inferred) permissions. 3. **Defense Layers**: Data trustworthiness, identity/access control, runtime behavior monitoring, and human accountability are all required; no single layer suffices. 4. **Instruction Hierarchy**: Cannot be relied upon as a structural defense. Require deterministic permission enforcement independent of prompts. 5. **Human in the Loop**: Must be competent domain experts with accountability, not ceremonial approval. Non-repudiation and traceability essential for consequential actions. 6. **Observability**: Monitor agent behavior (tool invocations, data flows, C2 patterns), not just inputs. Trace entire attack chains across persistence mechanisms. 7. **Persistence Matters**: Agents maintaining state (heartbeats, memory, logs) are more dangerous when compromised; state machine design and cleanup critical.