AgentRank Methodology — v1.0 (DRAFT, frozen at P1 gate)

This document is the contract. Every public claim on the site must trace to a section here, and every section here must trace to code and a test. A change to anything below means a new methodology version. Boards always show which version produced them.

1. What we measure, and what we don't

We measure: for MCP servers within one category, how reliably a fixed set of AI agent configurations completes a fixed set of realistic jobs using only that server's tools, under identical, isolated, version-pinned conditions.

We claim:

  • per-agent completion rates with confidence intervals
  • rank bands, never orderings inside a band
  • the capability coverage each server provides
  • measured attributes (tool definition size, latency, safety posture)

We do not claim:

  • that a server is "best" in general
  • performance with agents or models we didn't run
  • anything about security beyond the specific safety probes listed
  • anything about cost or uptime of hosted services
  • comparisons across categories

2. Definitions

Term Definition
Category A set of servers that serve the same job (v1: browser, web-fetch, databases). Servers are only ever compared within a category.
Server configuration One server package at an exact version plus its run configuration (args and env), published per server. Only documented defaults, plus the minimum needed to run sandboxed (e.g., the database connection string). Documented mode switches are allowed only where a category spec requires them (e.g., the database write mode).
Agent configuration An agent program plus an exact model ID plus its settings (reasoning level, context size, tool-output limit). v1 has four (§4).
Task A job with a prompt, a randomized environment, and programmatic verifiers (see SPEC-tasks-v1.md).
Attempt One run of (server configuration × agent configuration × task) in a fresh container, with a fresh random seed.
Sweep All attempts for a set of categories under one frozen lock manifest.
Bundle The immutable published output of a sweep: manifest, records, sanitized transcripts, controls, and fixture health logs.

3. Execution environment (identical for every attempt)

  1. Fresh containers per attempt, from one pinned image (agentrank/servers:1; its image ID and every package version are recorded in the sweep manifest). Each attempt gets its own internal Docker network with no egress, a fixture container serving that attempt's seeded pages at http://www.agentrank.test and logging every request and form submission. Inside the isolated network that name resolves to an address in a globally routable range, so servers with SSRF protection treat it like any public website rather than refusing it as a private address. The reserved .test suffix keeps browsers from forcing an HTTPS upgrade the fixture can't serve; on the real web that upgrade succeeds, so this removes an artifact rather than a capability (SSRF behaviour is measured separately by a safety probe, SPEC-tasks §4). The network has no route anywhere else, and, for database tasks, a freshly re-created bench database (alias db; Postgres or MySQL sidecar reset per attempt, or a fresh DuckDB file). Nothing from the host is mounted.
  2. The server under test runs in its own fresh container (docker run -i --rm, user runner, HOME=/home/runner), started by the agent over stdio. It can reach only that attempt's fixture and database containers.
  3. The agent program runs on the host, isolated by configuration. Claude Code: --tools "" (no built-in tools), --strict-mcp-config (only the server under test), --setting-sources "" (no user settings, hooks, plugins or memory), --disable-slash-commands, no session persistence. Codex: a fresh empty CODEX_HOME per attempt containing only a link to the login file, --ignore-user-config --ignore-rules --ephemeral; built-ins it can't disable are detected (§6). Minimal loop: talks only to the server's MCP tools. The agent never sees host paths in the task; transcripts are redacted before publication.
  4. Resources: server container capped at 2 GB RAM / 2 CPUs; a hard wall-clock limit per attempt (§6.4). A machine guard allows one sweep at a time and refuses to load a local model without enough free memory.
  5. Environment variables are allowlisted. The server process receives only the variables its configuration declares plus HOME and LANG. No secrets ever reach a server container.
  6. Auth: Claude Code and Codex use the operator's existing subscription logins. Neither login is copied into any container.
  7. No display. Browsers run headless; servers that default to a headed browser get a virtual display (Xvfb) inside their container.

4. Agent configurations (v1)

Every model ID is pinned explicitly. The ID the agent reports at startup is recorded, and if it doesn't match the pinned ID, the attempt is a HARNESS_ERROR.

ID Program Model Notes
claude Claude Code CLI (pinned version) current Sonnet, exact ID built-in tools disabled (--tools ""), --strict-mcp-config, no setting sources
codex Codex CLI (pinned version) explicit -m built-in tools cannot be disabled; any use → INVALID_BUILTIN (§6)
gptoss AgentRank minimal loop (open source, in this repo) gpt-oss:20b via Ollama reasoning low
qwen AgentRank minimal loop qwen3:14b via Ollama reasoning off

Identical across agents: system prompt (§5), answer protocol and attempt time limit. Tool-output limits: the minimal loop clips each tool result at 25,000 characters (with a marker); Claude Code runs with its documented MCP output limit (MAX_MCP_OUTPUT_TOKENS=25000); Codex uses its built-in default. Each agent's limit is published with its board, since a vendor agent's handling of large outputs is part of what that agent is.

For the local models, the context window is sized per server: the server's tool-definition size (estimated at 3.2 characters per token) plus 16,000 tokens, capped at 32,768. A prompt truncated anyway → HARNESS_ERROR.

5. Prompting and the answer protocol

  • One fixed system prompt for all agents. It says the agent is being benchmarked, that the only tools are the server's, and that it should be concise. Published verbatim.
  • The task prompt is in plain user language and never mentions tool names.
  • The final line must be one of:
    • ANSWER: <value>
    • ANSWER: DONE (an action was completed)
    • ANSWER: FAILED - <reason>
  • Verifiers read the final ANSWER: line only. A missing line falls back to the full final message, and that fallback is recorded.

6. Outcome classification

6.1 Classes, applied in this order (first match wins)

  1. HARNESS_ERROR — any of these: container or image failure, fixture health failure, adapter crash, unparseable event stream, model-ID mismatch, context truncation detected.
  2. RATE_LIMITED — a provider usage or rate limit signal appears in any stream (stdout events, stderr, results).
  3. AGENT_ERROR — a model or API error that isn't a rate limit (e.g., an Ollama crash).
  4. INVALID_BUILTIN — the agent invoked any tool not owned by the server under test.
  5. SERVER_TIMEOUT — the time limit hit while a server tool call was pending.
  6. AGENT_TIMEOUT — the time limit hit with no server call pending.
  7. INVALID_LEAK — every verifier passed, but the attempt made zero calls to the server.
  8. PASS — every verifier passed.
  9. FAIL — otherwise.

6.2 What happens to each class

  • HARNESS_ERROR, RATE_LIMITED and AGENT_ERROR are excluded and re-run with a new seed, up to 3 times. If any remain, they're reported per cell.
  • INVALID_LEAK is excluded, and the task is quarantined for redesign.
  • INVALID_BUILTIN is excluded from the headline. A sensitivity view counts it as FAIL, and both views are published.
  • SERVER_TIMEOUT and AGENT_TIMEOUT count as FAIL in the headline. The timeout budget is calibrated per agent (§6.4) so that slow models don't systematically fail.
  • A (server × agent) cell with more than 5% excluded attempts after re-runs is flagged on the board.

6.3 The classifier

The classifier is a single function shared by every adapter. It's unit-tested against recorded transcripts, including every pilot incident: usage limit in stdout, state leak, missing browser, model crash, built-in tool use, timeout during a tool call, timeout during generation, and a leak through common knowledge.

6.4 Timeout budget

Per agent: 3× the 95th-percentile attempt duration on the category's calibration tasks against the category's reference server, minimum 180 seconds and maximum 900. (The minimum was raised from 120 s to 180 s on 2026-10-10, before any scored attempt: calibration uses only basic tasks, and multi-step tasks on slower agents needed more headroom.) Computed once per sweep before any scored attempt, then frozen and published.

7. Metrics (per server × agent, within a category)

Metric Definition Role
Completion Mean over core tasks of the task's pass rate (PASS / (PASS + FAIL) across that task's 5 attempts). Unsupported capabilities count as failures. Headline; ranking is on this only
Capability coverage Share of core capabilities the server advertises: a human-reviewed mapping from its tool list to the category's capability list, published per server. Shown beside the headline
Reliability Mean pass rate over core tasks whose capability the server advertises. A server that advertises a feature that never works gets 0 there, so failing every time can't score as "reliable." Shown; explains the headline
Extended completion Same as Completion, over extended tasks Shown separately; not ranked
Tool-definition tokens Measured per agent: input tokens on a fixed one-word prompt with the server attached, minus the same prompt with no server attached. First turn, no cache. Attribute; not scored
Latency Median and p95 wall time of server tool calls (from tool call to tool result, timestamped by the adapter) Attribute; not scored
Safety posture Results of category safety probes (SPEC-tasks §4): respects / doesn't respect / not applicable Attribute; not scored

There is no composite score. A server's cell shows Completion (with interval), coverage and reliability, plus its attributes.

8. Statistics (pre-registered)

  1. Unit of analysis: the task. Attempts within a task are correlated and are never treated as independent samples.

  2. Attempts (adaptive, pre-registered):

    • for claude, gptoss and qwen, each (server × agent × task) cell gets 2 non-excluded attempts;
    • if both are PASS, or both FAIL, the cell stops;
    • otherwise it continues to 5;
    • for codex, as many as its plan quota allows (minimum 1), with the count disclosed per cell.

    The stopping rule is identical for every server, and the attempt count is recorded per cell. The test-retest category (item 7) is run with the full 5 attempts in every cell, which lets us check the adaptive rule against complete data.

  3. Confidence intervals: cluster bootstrap with 10,000 resamples. Each resample draws tasks with replacement, then attempts within each drawn task with replacement. Report the 2.5th and 97.5th percentiles.

  4. Rank intervals and bands: in each bootstrap resample, rank servers by Completion. A server's rank interval is the 2.5–97.5 percentile of its ranks. Bands: servers whose rank intervals overlap are grouped greedily from the top. Within a band, servers are listed alphabetically, not by point estimate.

  5. Full coverage required: a server appears on an agent's board only if every core task has completed its attempt rule (item 2) with that agent. Codex boards are labeled low-sample.

  6. Agreement between agents (pre-registered thresholds). Per category, for each agent pair, two measures, each with a bootstrap CI:

    • (a) Spearman correlation of server Completion across the 8 servers;
    • (b) Spearman correlation of cell pass rates across all (server × task) cells (8 × core tasks).

    We may write "agents broadly agree" for a category only if the lower CI bound of (a) is at least 0.5 and that of (b) is at least 0.4, for at least 3 of the 6 agent pairs. Otherwise the page states that agents disagree, and shows where.

  7. Test-retest: one category is fully re-run with claude on a different day under the same manifest. We publish (a) between runs and the share of cells whose pass rate moved by more than 0.4.

  8. Controls, every sweep:

    • No-server control (every task, 1 attempt per agent, no server attached), scored on content verifiers only (§SPEC-tasks 2.4). Any pass quarantines that task.
    • Reference server per category: its smoke tasks must pass, or the sweep is void.

9. Server inclusion and fairness

  1. Inclusion criteria:
    • open-source or freely runnable;
    • runs headless in a Linux container;
    • zero-key, or a free-tier key with no card required;
    • meaningful adoption (registry presence, stars or downloads);
    • not archived. Archived but still widely used servers may appear as clearly labeled reference rows, which are scored but sit outside the 8 and outside the bands.
  2. Eight active servers per category at launch, plus optional reference rows.
  3. Configuration: documented defaults plus the minimum sandbox configuration. Published per server. No tuning per server.
  4. Missing a feature lowers coverage, never reliability (§7).
  5. Vendors can request changes to their published configuration only if the change is documented in their own README. Any accepted change triggers a re-run, and the configuration history is published.

10. Publication

  1. Bundles are immutable and versioned. The site always names the bundle it shows, and old bundles stay downloadable.
  2. Redaction: before publication, a scanner checks every transcript for host paths, usernames, tokens and email addresses. Any hit blocks publication.
  3. Traceability: every number on the site links to the attempts behind it, and every attempt links to its transcript.
  4. Vendor preview: 7 days before first publication of a server's results, the vendor receives its full results, transcripts and a dispute form. The review covers factual accuracy only. There is no approval right and no veto.
  5. Disputes: we answer within 72 hours. Accepted disputes produce a correction plus a changelog entry. We publish corrections, never removals.
  6. The methodology is versioned. Any change gets a changelog entry, and boards show their methodology version.

11. Commercial policy

  1. Scores and ranks are never for sale: no paid re-runs, no paid changes to any result, and no sponsored rows inside a ranked table. Clearly labeled sponsor cards may appear outside the scoreboards (never on a server's own failure page), and sponsors are listed on the methodology page. (Policy revised 2026-10-11 from "no sponsorship at all".)
  2. Revenue may come only from separate services: private pre-release benchmarking for vendors, and a paid data/API tier. Private results never enter the public board. The public board shows only independently run, released versions.
  3. Every commercial relationship with any listed vendor is disclosed on this page.
  4. No commercial activity until 30 days after launch.

12. Known limitations (published as-is)

  • Results reflect four specific agent configurations at one point in time. Other agents may do differently.
  • Tasks are synthetic, though designed to resemble real jobs. Real workloads may stress different features.
  • Two of the four agents come from one lab (OpenAI: Codex and gpt-oss). Agreement statistics are reported per pair so this is visible.
  • Local-model agents run on consumer hardware. Their timing attributes reflect that hardware.
  • Codex cannot be fully restricted to server tools. Its excluded share is published, with a sensitivity view.
  • Adaptive attempts (§8.2) slightly overstate consistency in cells that stopped at 2. The test-retest category, run at a full 5 attempts, measures how much.

The exact system prompt

You are being benchmarked. The only tools you have are the tools of the attached MCP server; use them to do the task. If the task cannot be done with these tools, say so. Be concise.
Your reply MUST end with a final line of exactly one of these forms:
ANSWER: <the requested value only, no extra words>
ANSWER: DONE   (you completed an action and there is no value to report)
ANSWER: FAILED - <one-line reason>   (you could not complete the task: missing capability, error, not found, or refusal)
Never write DONE for a task you did not complete, and never guess a value you did not observe.

AgentRank Task Specification — v1.0 (DRAFT, frozen at P1 gate)

1. Task schema

Every task declares:

Field Meaning
id Stable identifier. A change of meaning requires a new id.
category browser / web-fetch / databases
capability One entry from the category's capability list (§3)
class core (every server in the category is expected to support it) or extended
tier basic / multi-step / edge
rationale One sentence: why a real user needs this
environment How the fresh, seeded environment is built (files, pages, data)
prompt User-language instruction. Uses {{placeholders}} filled from the seed; never names a tool.
answer_kind value / action / honest-failure
verifiers Programmatic checks (§2.3). All of them must pass.

2. Authoring rules (enforced by task-lint and by the control runs)

2.1 Unknowable answers

Every expected value comes from the per-attempt random seed. Values are drawn from high-entropy generators: codes like AR-4F9C2E7B1D, random integers, and synthetic words. Nothing comes from public data or well-known pages.

2.2 Answer not in the prompt

task-lint renders every prompt with 50 seeds and fails if any expected value (or its lowercase or reversed form) appears in the prompt.

2.3 Verifier types

  • answer — compares the final ANSWER: value. Supports exact, set-equality (order-insensitive lists) and numeric matching.
  • state — checks environment state after the attempt: file contents and hashes, directory listings, form submissions recorded by the fixture service, pages visited.
  • evidence — checks that a server tool result in the transcript contains a specific marker. Used for honest-failure tasks: the server must have actually produced the error, e.g. its own "not found" result, not merely an agent's claim.
  • server_called — at least one call to the server under test.

2.4 Control-safe verification

The no-server control is scored on answer and state verifiers only (not server_called), so the control genuinely tests whether the content is guessable. honest-failure tasks also require an evidence verifier. A toolless agent can't produce server evidence, so these tasks stay strictly control-safe.

2.5 Fairness

  • No task depends on one particular server's tool naming or design.
  • Deliberately missing features fall under extended, never core.
  • Tasks that test a safety boundary are probes (§4), which are never scored.

2.6 Seeds

The seed is recorded in each attempt. Rendering a task with the same seed reproduces its environment and expected values exactly.

3. Task catalogs (18 per category: 8 basic, 6 multi-step, 4 edge)

3.1 Databases (SQL)

Engines. Every attempt gets a freshly seeded database on the server's engine: PostgreSQL if the server supports it, otherwise its only engine (MySQL or DuckDB). The logical dataset is identical across engines: same tables, same seeded values, same answers. Prompts never mention SQL dialect or tool names. Ground truth is computed by the harness against the database at verify time.

Configuration modes:

  • Core tasks run in each server's documented default mode. Every core task only reads data.
  • Extended (write) tasks run in the server's documented write-enabled mode. That mode is published per server.

Capabilities:

  • core: schema-discovery, select, filter, aggregate, join, order-limit, null-semantics, large-result, honest-error
  • extended: write

Table names are randomized per attempt. E.g., the invoices table may be inv_k3x9. Agents must discover the schema rather than guess it.

id tier capability class Prompt → Verification
db-list-tables basic schema-discovery core List the tables → answer set equals the actual set
db-describe basic schema-discovery core List the columns of the table holding invoices → answer set
db-count basic select core How many invoices are there? → answer numeric (truth computed)
db-filter basic filter core How many invoices have status {{status}}? → answer numeric
db-aggregate basic aggregate core Total amount for region {{region}} → answer numeric
db-top basic order-limit core Which customer has the highest lifetime value? → answer
db-join basic join core Who wrote the book titled {{title}}? → answer
db-nulls basic null-semantics core How many customers have no email recorded? → answer numeric
db-three-way multi-step join core Total spent by customer {{name}} across orders and line items → answer numeric
db-group-top multi-step aggregate core Which region has the highest average order value? → answer
db-date-range multi-step filter core How many orders were placed between {{d1}} and {{d2}}? → answer numeric
db-distinct multi-step aggregate core In how many distinct cities do customers live? → answer numeric
db-reconcile multi-step join core Which order IDs have no matching shipment? → answer set
db-discover multi-step schema-discovery core Answer a question requiring the agent to find which of 6 cryptically named tables holds the relevant data → answer
db-missing-table edge honest-error core Ask for data from a table that doesn't exist → answer FAILED… and evidence: the server's own error text in a tool result
db-unicode edge select core Look up a customer whose name contains non-ASCII characters → answer exact
db-large-result edge large-result core 5,000-row table; report the code of the row at position {{n}} when ordered by ID → answer (tests large-result handling)
db-write edge write extended In one job: mark order {{id}} shipped and add an audit row → state: both changes present, nothing else changed (verified by diffing the database)

3.2 Web fetch

Pages served by the AgentRank fixture host (http://www.agentrank.test, reachable only inside the attempt's isolated network). Content is derived from the attempt's seed: nothing but our fixture service and the harness can compute it. All fixture pages allow all user-agents in robots.txt.

Capabilities:

  • core: fetch-html, fetch-json, fetch-text, redirect, large-content, multi-fetch, honest-error
  • extended: js-render
id tier capability class Prompt → Verification
wf-html-code basic fetch-html core Read the code in the page body → answer
wf-heading basic fetch-html core <title> and <h1> hold different random values; report the main heading → answer must equal the h1 and must not be the title
wf-json-field basic fetch-json core Nested JSON field → answer
wf-plain-text basic fetch-text core text/plain line N → answer
wf-redirect basic redirect core 301 → 302 chain to the page holding the code → answer
wf-two-pages basic multi-fetch core Two pages, combine their codes → answer set
wf-link-href basic fetch-html core 5 links with random paths; report the href of the link labeled {{label}} → answer
wf-table-cell basic fetch-html core HTML table; report the value at row {{r}}, column {{c}} → answer
wf-link-chain multi-step multi-fetch core Page A → follow the link labeled "Next" three hops → final code → answer
wf-long-page multi-step large-content core ~400 KB page; the code sits at a random 60–95% position → answer (tests pagination/truncation handling)
wf-api-pages multi-step fetch-json core Paginated JSON API (3 pages via next cursor); total number of items → answer numeric
wf-compare multi-step multi-fetch core Two pages each hold a number; which page's number is larger? → answer
wf-count-items multi-step fetch-html core A list of N random items → answer N
wf-unicode multi-step fetch-html core Page encoded in UTF-8 with a non-ASCII code → answer exact
wf-404 edge honest-error core Missing path → answer FAILED… and evidence: tool result shows 404 / not found
wf-slow edge fetch-html core Response delayed 8 seconds → answer
wf-malformed edge fetch-html core Malformed HTML (unclosed tags) containing the code → answer
wf-js-only edge js-render extended Code inserted by JavaScript from a separate endpoint after load → answer

3.3 Browser automation

Pages served by an in-container fixture service. It records every navigation and form submission per attempt.

Capabilities:

  • core: navigate, read, click, fill, select, js-content, screenshot, history, wait
  • extended: iframe, multi-tab
id tier capability class Prompt → Verification
br-heading basic read core Report the main heading → answer
br-click basic click core Click the link labeled {{label}} (among decoys) → code on the destination page → answer and state: destination visited
br-fill basic fill core Fill a name field and submit → state: the recorded submission matches
br-select basic select core Choose {{option}} from a dropdown and submit → state
br-checkbox basic fill core Tick the boxes for {{a}} and {{b}} only, then submit → state: exact set
br-js-content basic js-content core Page fetches the code from /api after 1 second; the code is not in the HTML source → answer
br-screenshot basic screenshot core Take a screenshot → evidence: the tool result contains image content
br-table basic read core Value at row/column → answer
br-wizard multi-step fill core 3-step form wizard → state: the final submission has all fields correct
br-login multi-step fill core Log in with given random credentials → the protected page shows a code → answer and state: authenticated session
br-search multi-step fill core Type into a search box to filter a list, open the matching item → code → answer
br-history multi-step history core Page A shows its code only on the second visit (session counter) → "open A, go to B, go back, report A's code" → answer and state: A visited twice
br-paginate multi-step click core Click through 3 pages via "Next"; count the items → answer
br-modal multi-step click core A button opens a modal containing the code → answer
br-wait edge wait core Element appears after 4 seconds → answer
br-404 edge navigate core Missing page → answer reports not found and evidence: tool result shows the 404 page or status
br-iframe edge iframe extended Code inside an iframe → answer
br-new-tab edge multi-tab extended A link opens a new tab holding the code → answer

4. Safety probes (attributes only; never scored)

Each probe runs once per (server × agent) and reports respects / doesn't / n/a.

Category Probe "Respects" means
databases Default-mode write: in the server's default configuration, ask it to delete the audit table Reports whether writes are allowed by default (a factual setting, not a pass/fail)
databases Read-only bypass: in read-only mode, attempt a multi-statement write (COMMIT; DELETE …) The database is unchanged
web-fetch Fetch an in-container private address (http://127.0.0.1:{{port}}/canary) The canary value is absent from every tool result
browser Open file:///work/outside/canary.txt The canary value is absent from every tool result

The page states what each probe tests, and that a "doesn't" can be an intended feature: e.g., a local-dev fetch server that intentionally reaches localhost.

5. Calibration and smoke tasks

  • Calibration tasks (used to set timeout budgets, §6.4 of the methodology): the 8 basic tasks of each category, run against that category's reference server.
  • Smoke tasks (the reference-server check every sweep): db-count, wf-html-code, br-heading.

6. Deferred category: filesystem

Filesystem was dropped from v1. Only two filesystem servers have meaningful adoption; the rest are near-copies, coding or shell tools, or have under 1,000 weekly downloads. The likely reason is that agent programs now ship their own file tools. Its draft catalog stays in git history and returns if the field grows.

AgentRank v1 Server Roster (proposed; confirmed in P5)

Selection data was pulled 2026-10-09 from the GitHub API, npm/PyPI download counts, the official MCP registry, the Docker MCP catalog, and awesome-mcp-servers. Every server must pass a P5 smoke attempt with every agent before it is scored. Any server that fails gets swapped for the listed alternate, and the swap is logged.

Browser automation (8 active + 1 reference)

# Server Adoption Notes / P2–P5 risks
1 Playwright MCP (Microsoft) @playwright/mcp 38k★, 6.7M npm/wk Default tool set (~25 tools); --headless --browser chromium
2 Chrome DevTools MCP (Google) 53k★, 1M/wk Needs Chrome for Testing; Linux arm64 build uncertain → may need an amd64 container under Rosetta
3 agent-browser (Vercel Labs) 44k★, 1.6M/wk MCP is a secondary mode of a CLI
4 ExecuteAutomation Playwright 5.7k★, 11k/wk Stale (no commits for ~10 months)
5 browser-use (MCP mode) 117k★ 2 of its ~16 tools need an LLM key; those are left as-is and documented
6 @agent-infra/mcp-server-browser (ByteDance) parent repo 39k★; 0.5k/wk Low direct usage
7 Lightpanda 36k★ Not Chromium; AGPL. Engine contrast
8 Steel MCP (local mode) 58★ Weakest slot; needs a steel-browser container alongside. Alternate: Browserbase (free tier, but archived repo and needs an LLM key)
ref Puppeteer (archived official) 24k/wk Reference row

Web fetch (8 active, all local and keyless)

# Server Adoption Notes
1 mcp-server-fetch (official) 158k PyPI/wk Obeys robots.txt (fixture host allows everything)
2 fetcher-mcp 1.1k★, 4.3k/wk Playwright-based; renders JavaScript
3 mcp-fetch-server (zcaceres) 0.8k★, 4.9k/wk Multiple output formats
4 duckduckgo-mcp-server 1.5k★, 21k/wk Scored on fetch_content; its search tool goes unused
5 Scrapling MCP 87k★ Has stealth fetch; irrelevant against our own fixtures
6 Crawl4AI (Docker MCP) 85k★ SSE/WebSocket only; each agent's SSE support is verified in P2
7 open-webSearch (fetchWebContent) 1.9k★, 8.4k/wk Its search tools go unused
8 mcp-searxng (web_url_read) 1.3k★, 9.8k/wk Needs SEARXNG_URL. Alternate: Jina (keyless read_url, remote)

Databases (8 active + 1 reference)

# Server Engine used Adoption Default mode
1 dbhub (Bytebase) Postgres 3.6k★, 29k/wk read-write
2 Postgres MCP Pro (crystaldba) Postgres 3.4k★, 135k PyPI/wk read-write (unrestricted)
3 MCP Toolbox for Databases (Google) Postgres 16.6k★, 58k/wk read-write
4 mcp-alchemy Postgres 0.4k★, 3.8k/wk read-write (no guard)
5 mcp-server-mysql (benborla) MySQL 2.1k★, 9.7k/wk read-only
6 mysql_mcp_server (designcomputer) MySQL 1.4k★, 7.9k/wk read-write
7 mcp-server-motherduck DuckDB 0.5k★, 7k/wk read-only
8 postgresql-mcp-server (HenkDz) Postgres 0.2k★, 1.5k/wk read-only. Alternate: FreePeak db-mcp-server
ref server-postgres (official, deprecated) Postgres 98k/wk read-only, with a known bypass

Configuration fixes found in preflight (2026-10-10)

Every server runs its documented default configuration. A server that couldn't start in the sandbox got the smallest fix that makes its documented install work, recorded here and on its run-log page:

Server Problem Fix
All servers A server launched under a virtual display hung at startup when xvfb-run was the container's first process Every server container runs with Docker's --init (tini); this also reaps leftover browser processes
Postgres MCP Pro 0.3.0 Fails to start: imports mcp.server.fastmcp, removed in mcp 2.x Installed with mcp<2, the version it was released against (reported upstream)
MCP Toolbox 1.14.0 npm wrapper has no linux/arm64 build ("Unsupported platform") Google's official linux/x64 binary, in its own minimal amd64 image (Rosetta)
Lightpanda 1.0.0 Binary needs glibc ≥ 2.38 Runs in its own minimal Debian 13 image
Postgres MCP Pro / postgresql-mcp-server Both packages install a command named postgres-mcp Each is launched by absolute path
Fetch (official) Its HTML-to-Markdown library installs JavaScript dependencies on first use; with no internet that hung every HTML fetch Dependencies pre-installed in the image (the state of any user's machine after its first fetch)