MCP-GuardBench

AI agents trust the tool descriptions an MCP server sends them, which makes those descriptions an easy place to hide instructions. A server can also change a tool after you've approved it. MCP-GuardBench does two things about that:

• guardbench inspect reads the MCP servers configured in Claude Desktop, Claude Code, Cursor, VS Code or Windsurf, asks each one for its tool list and nothing else, flags poisoned descriptions and risky tools, and pins fingerprints so a later change is caught.
• guardbench benchmark is a lab you can reproduce: 11 deliberately vulnerable local MCP servers, 12 test cases, and scoring that checks whether an attack was really stopped instead of trusting what the tool says.

Everything is deterministic and runs locally: no LLM, no telemetry, and the benchmark only ever runs against its own test servers.

• Tool poisoning: hidden instructions in a tool's description or input schema, which the agent reads but the user rarely sees.
• Rug pulls: a tool definition that changes after it was approved.
• Data leaks: secrets flowing into an outbound call, sometimes encoded to slip past pattern checks.
• Unfair comparisons: a security tool reporting "detected" or "blocked" isn't evidence that anything was stopped.

11 vulnerable MCP servers (local fixtures)
  -> scripted, deterministic agent plays a test case
  -> security control under test sees every step (Guard hooks)
  -> the fixture's own ledger records what actually executed
  -> detection and prevention verified from that ground truth

The code is layered so each layer only depends on the ones below it. A framework-free domain layer (Pydantic v2 schemas) sits under the analysis, policy and benchmark packages; services orchestrate them; and the CLI (Typer), API (FastAPI) and dashboard (Streamlit) sit on top. Results persist through SQLAlchemy 2.x with Alembic migrations on PostgreSQL or SQLite.

The attack fixtures live in an import-restricted mcp_lab/ package. It is the only code that behaves like an attacker, and nothing outside the benchmark and the CLI's inspection commands depends on it.

• inspect starts each server exactly as the user's client would and sends only tools/list. A security test scans the source for any call, resource or prompt operation, and everything a server returns is escaped before it reaches the terminal.
• A transparent rule catalogue flags instruction-override language, requests to hide things from the user, instructions to call other tools, invisible or bidirectional Unicode, fake system tags, high-risk capabilities, loose schemas, and look-alike tool names across servers (shadowing).
• Rug-pull detection uses SHA-256 fingerprints over a canonical form of each tool definition. The pin file stores only what servers advertised, never commands, environment variables or headers, so it can be committed next to .mcp.json like a lockfile; a changed tool exits with status 1 and fails CI.
• A third-party control, Cisco AI Defense's open-source MCP Scanner, runs in its own virtual environment through the same Guard hook, so it sees the same tool listings and is scored by the same rules.
• Published to PyPI as mcp-guardbench.

pip install mcp-guardbench

guardbench inspect --discover                  # every server your MCP clients have configured
guardbench inspect .mcp.json --update-pins     # after reviewing: approve current tool definitions
guardbench inspect .mcp.json                   # later: exit status 1 if any tool changed

• Ground truth over self-reports

  A detection only counts if the adapter's findings carry evidence matching the case's expected category. Prevention is read from the fixture's own ledger of what executed. An alert is not prevention.

• Detection and prevention reported separately

  Scanners sit outside the call path, so they can alert but never stop anything. Two separate metrics keep that difference visible instead of hiding it in one score.

• No LLM in the loop

  Every core check is deterministic and the benchmark's agent is a script, so a run gives the same result every time.

• Undefined, not zero

  When a metric's denominator is zero, it is reported as undefined with a reason, never as 0%.

The two hardest cases were added after the reference controls were written, to probe weaknesses the docs already admit, and the controls weren't changed to pass them. Neither reference control detects the Spanish-language poisoning case (TP-003) or the base64-encoded secret (DF-002). reference-runtime prevents DF-002 only because unattended mode never approves an outbound send.

• The test set is small (12 cases), so results describe these fixtures, not MCP security in general. It isn't a product ranking.
• The static rules read English-language patterns and will miss paraphrased, translated or obfuscated instructions.
• Runtime protection only covers calls routed through the hook. A client that skips it gets none.
• Fingerprints detect change, not initial trust: a server that is malicious from the start produces a stable hash.
• The Cisco scanner ran in one configuration. Its API and LLM analyzers need keys and weren't used.