mcpmark

Name: mcpmark
Author: eval-sys

449stars40forks

How to install mcpmark?

eval-sys/mcpmark

$git clone https://github.com/eval-sys/mcpmark

Installs into the current project.

Summary

</div>

Files · 1

View on GitHub

README.md

1	<div align="center">
2
3	# MCPMark: Stress-Testing Comprehensive MCP Use
4
5	[![Website](https://img.shields.io/badge/Website-mcpmark.ai-4285F4?style=for-the-badge&logo=google-chrome&logoColor=white)](https://mcpmark.ai)
6	[![arXiv](https://img.shields.io/badge/arXiv-2509.24002-b31b1b?style=for-the-badge&logo=arxiv&logoColor=white)](https://arxiv.org/abs/2509.24002)
7	[![Discord](https://img.shields.io/badge/Join_our_discord-5865F2?style=for-the-badge&logo=discord&logoColor=white)](https://discord.gg/HrKkJAxDnA)
8	[![Docs](https://img.shields.io/badge/Docs-000000?style=for-the-badge&logo=mdbook&color=105864)](https://mcpmark.ai/docs)
9	[![Hugging Face](https://img.shields.io/badge/Trajectory_Logs-FFD21E?style=for-the-badge&logo=huggingface&logoColor=black)](https://huggingface.co/datasets/Jakumetsu/mcpmark-trajectory-log)
10
11	</div>
12
13	An evaluation suite for agentic models in real MCP tool environments (Notion / GitHub / Filesystem / Postgres / Playwright).
14
15	MCPMark provides a reproducible, extensible benchmark for researchers and engineers: one-command tasks, isolated sandboxes, auto-resume for failures, unified metrics, and aggregated reports.
16
17	> 🚀 MCPMark Verified is now the default. The standard tasks in this repository are the Verified set — every environment version-pinned and every verifier stabilized. Results from earlier task versions are deprecated and not directly comparable, so please report new numbers as MCPMark Verified. On the Verified set, `gpt-5.5` (xhigh) leads at 92.9% and `kimi-k2.7` reaches 81.1%. See [#264](https://github.com/eval-sys/mcpmark/pull/264).
18
19	[![MCPMark](https://github.com/user-attachments/assets/dfc06a41-e387-45e3-bc98-db7097ffa3dc)](https://mcpmark.ai)
20
21	## News
22
23	- 🚀 12 Jun — MCPMark Verified is now the default task set: every environment version-pinned and every verifier stabilized. Earlier task versions are deprecated; please report scores as MCPMark Verified. On the Verified set, `gpt-5.5` (xhigh) leads at 92.9% and `kimi-k2.7` reaches 81.1%. See [#264](https://github.com/eval-sys/mcpmark/pull/264).
24	- 📣 27 May — The previous Notion Source Hub page is deprecated; please use the new link: [MCPMark Source Hub](https://gossamer-sawfish-47c.notion.site/MCPMark-Source-Hub-dc32b7e8cebd82b8959b81ae322df87a).
25	- 📌 21 Jan — Pinned MCP server versions for reproducible benchmarks: GitHub MCP Server `v0.15.0` (switched to Docker for version control), Notion MCP Server `@1.9.1` (Notion released 2.0 but it has many bugs, not recommended). See [#246](https://github.com/eval-sys/mcpmark/pull/246).
26	- 🔥 13 Dec — Added auto-compaction support (`--compaction-token`) to summarize long conversations and avoid context overflow during evaluation ([#236](https://github.com/eval-sys/mcpmark/pull/236])).
27	- 🏅 02 Dec — Evaluated `gemini-3-pro-preview` (thinking: low): Pass@1 50.6% ± 2.3% — so close to `gpt-5-high` (51.6%)! Also `deepseek-v3.2-thinking` 36.8% and `deepseek-v3.2-chat` 29.7%
28	- 🔥 02 Dec — Obfuscate GitHub @mentions to prevent notification spam during evaluation ([#229](https://github.com/eval-sys/mcpmark/pull/229))
29	- 🏅 01 Dec — DeepSeek v3.2 uses MCPMark! Kudos on securing the best open-source model. [X Post](https://x.com/deepseek_ai/status/1995452650557763728) \| [Technical Report](https://huggingface.co/deepseek-ai/DeepSeek-V3.2/resolve/main/assets/paper.pdf)
30	- 🔥 17 Nov — Added 50 easy tasks (10 per MCP server) for smaller open-source models ([#225](https://github.com/eval-sys/mcpmark/pull/225))
31	- 🤝 31 Oct — Community PR from insforge: better MCP servers achieve better results with fewer tokens! ([#214](https://github.com/eval-sys/mcpmark/pull/214))
32	- 🔥 13 Oct — Added ReAct agent support. PRs for new agent scaffolds welcome! ([#209](https://github.com/eval-sys/mcpmark/pull/209))
33	- 🏅 10 Sep — `qwen-3-coder-plus` is the best open-source model! Kudos to Qwen team. [X Post](https://x.com/Alibaba_Qwen/status/1965457023438651532)
34
35	---
36
37	#

People who liked this also viewed

6 picks

Type

fastmcp🚀 The fast, Pythonic way to build MCP servers and clients.MCP ServersJul 202686M27k
playwright-mcpPlaywright MCP serverMCP ServersJul 202626M35k
ext-appsOfficial repo for spec & SDK of MCP Apps protocol - standard for UIs embedded AI chatbots, served by MCP serversMCP ServersJul 20268.5M2.6k
langchain-mcp-adaptersLangChain 🔌 MCPMCP ServersJul 20267.0M3.6k
mcporterCall MCPs via TypeScript, masquerading as simple TypeScript API. Or package them as cli.MCP ServersJul 20262.5M4.8k
mcp-proxyA bridge between Streamable HTTP and stdio MCP transportsMCP ServersJul 20262.2M2.7k