WE GAVE 24 MCP
SERVERS REAL JOBS.
12 BROKE.

Claude Code and Codex did 1,712 real tasks with 24 popular MCP servers, each in a fresh sandbox with every transcript public. Some servers lose data silently. Some work with one agent and break with another. Most of these failures are invisible to the people who maintain them.

SEE WHAT BROKE >TEST YOUR SERVER
Pixel robot shredding a wave

WHAT BROKE

Every finding links to the transcript that shows it. Maintainers: tell us if we got something wrong.

WRONG DATA, SILENTLY

Postgres MCP Pro (crystaldba)

Columns that share a name are silently dropped

A query selecting count(distinct city), count(*) and count(city) came back as [{'count': 69}]: two of the three values were gone with no warning. Results are keyed by column name, so duplicate names overwrite each other. Rows are also returned as Python repr, not JSON.

Agents: Claude Code hit it in 3 of 5 attempts; Codex aliased its columns and never did.
Fix: Return rows as arrays with a separate column list, or de-duplicate column keys (count, count_2…).

transcript · every attempt · repo
WRONG DATA, SILENTLY

DBHub (Bytebase)

Columns that share a name are silently dropped

count(distinct city), count(*) returned a single {"count": "61"}: the agent reported the total row count as the number of distinct cities. Same root cause: result rows are objects keyed by column name.

Agents: Claude Code hit it in both attempts; Codex aliased its columns.
Fix: Preserve duplicate columns (array rows, or suffixed keys).

transcript · every attempt · repo
WON'T START

Postgres MCP Pro (crystaldba)

Latest release crashes on startup with the current MCP SDK

postgres-mcp 0.3.0 imports mcp.server.fastmcp, which mcp 2.x removed. A fresh install pulls mcp 2.x and the server exits immediately. It works when mcp<2 is pinned.

Agents: All agents (the server never starts).
Fix: Pin mcp<2 in the package metadata, or migrate to the 2.x API.

every attempt · repo
WON'T START

MCP Toolbox for Databases (Google)

No Linux arm64 build (Docker on Apple silicon, AWS Graviton)

The npm wrapper only ships darwin and linux-x64 binaries and fails with "Unsupported platform: linux-arm64". We ran Google's x64 binary under emulation; everything passed once it ran.

Agents: All agents on ARM Linux.
Fix: Publish a linux-arm64 binary and npm optional dependency.

every attempt · repo
WON'T START

Lightpanda

Binary requires glibc 2.38: fails on Debian 12 and Ubuntu 22.04

The 1.0.0 aarch64 release exits with "GLIBC_2.38 not found" on the two most common server distros. It ran fine in a Debian 13 container.

Agents: All agents on older Linux distros.
Fix: Build against an older glibc, or ship a static/musl build.

every attempt · repo
HANGS

Fetch (official)

Every HTML fetch hangs in sandboxed or offline environments

The HTML-to-Markdown step (readabilipy) installs npm packages the first time it runs. Without internet access, as in most agent sandboxes, every HTML fetch hung until the 180-second tool timeout. JSON and plain-text fetches were fine.

Agents: All agents, in any network-restricted sandbox.
Fix: Vendor readabilipy's JS dependencies at install time, or fall back to the Python parser when Node deps are missing.

every attempt · repo
WRONG DATA, SILENTLY

open-webSearch (fetchWebContent)

includeLinks: true returns link text mashed together, with no URLs

A page with five links came back as "denamabhuhebhoresusdobizubcatub": the five link labels concatenated, no spaces, no hrefs. Agents can't follow links or report where they point.

Agents: Claude Code and Codex both failed every link task (0 of 8).
Fix: Emit links as markdown [text](href) when includeLinks is set.

transcript · every attempt · repo
MISSING CAPABILITY

DuckDuckGo MCP (fetch_content)

Fetched pages lose every link URL

fetch_content in markdown mode returns link labels as a bullet list with the URLs stripped. The alternative backend needs an optional extra that the default install doesn't include ("pip install 'duckduckgo-mcp-server[browser]'").

Agents: Claude Code and Codex both failed every link task (0 of 8).
Fix: Keep hrefs in markdown output; make the error message part of the docs.

transcript · every attempt · repo
BREAKS WITH SOME AGENTS

fetcher-mcp

Long pages work with Codex, fail with Claude Code

fetch_url returns the whole page in one response and offers maxLength but no offset. Claude Code caps tool output (~25k tokens), so content past the first ~20,000 characters is unreachable. Codex accepted the full 380k-character response and passed.

Agents: Claude Code 0 of 2; Codex 2 of 2.
Fix: Add an offset/start_index parameter (as the official fetch server does).

transcript · every attempt · repo
FAILS ON SLOW SITES

mcp-searxng (web_url_read)

Hard-coded 10-second page timeout

web_url_read gives up after 10 s ("took longer than 10000ms to respond"). Retrying never helps, so any slow page is unreadable.

Agents: Claude Code and Codex both 0 of 2.
Fix: Make the timeout configurable and default it higher (e.g. 30 s).

transcript · every attempt · repo
BREAKS WITH SOME AGENTS

Playwright MCP (ExecuteAutomation)

Screenshots are saved to disk, not shown to the agent

playwright_screenshot returns "Screenshot saved to: …/Downloads/page-….png" instead of the image, so Claude Code could not see what was on the page.

Agents: Claude Code 0 of 2; Codex 2 of 2.
Fix: Return the image content in the tool result by default.

transcript · every attempt · repo
MISLEADING OUTPUT

browser-use (MCP mode)

A missing page comes back as an empty page

Navigating to a 404 returned "No content extracted" with the URL as the title: no status code or error text, so agents can't tell "missing" from "empty". (Claude then claimed "404" without evidence; our verifier rejects claims the tools didn't show.)

Agents: Claude Code 0 of 2.
Fix: Surface the HTTP status and error page text.

transcript · every attempt · repo
SETUP FRICTION

@agent-infra/mcp-server-browser (ByteDance)

"Unable to find any browser" on Linux arm64

It only looks for a system Chrome, which has no Linux arm64 build. It works once --executable-path points at Chromium; after that it passed every task with Claude Code.

Agents: All agents on ARM Linux, until configured.
Fix: Fall back to Playwright's bundled Chromium, or say how to fix it in the error.

every attempt · repo
BY DESIGN

Lightpanda

Doesn't render images; multi-step forms were flaky

Lightpanda is a fast non-graphical browser, so screenshots can't show image content. A three-step form wizard also failed 2 of 5 attempts with Claude Code.

Agents: Claude Code (wizard 3/5, screenshot 0/2).
Fix: Document the no-render tradeoff in the tool descriptions so agents pick another tool.

transcript · every attempt · repo
Pixel robot wiping out

IS YOUR SERVER AGENT-PROOF?

We'll run your MCP server through Claude Code, Codex and open-weight models in sandboxes, then hand you every failure, the transcript that shows it, and the fix.

GET AN AUDIT >

THE SCOREBOARDS

Completion rates per agent, with confidence intervals and statistical tie bands.