MCP Server Usability Audit: One-Third Received D or F Grades
Comprehensive testing reveals critical documentation gaps in popular MCP implementations.
1Critical Flaws in MCP Servers
Compliance with Model Context Protocol specifications doesn't guarantee agent usability. Analysis shows many servers fail due to poor documentation and schema design, causing AI agents to select wrong tools or generate invalid arguments. The protocol layer works fine, but the human-readable elements are failing.
2Documentation Crisis
Undocumented parameters emerged as the most widespread issue. Major companies' servers like MongoDB and Notion contained dozens of documentation errors. Automated schema generators stripping descriptions create systemic problems, leaving models without crucial context for tool selection and argument formation.
3The Large Catalog Challenge
While smaller, well-documented tool catalogs generally perform better, exceptions like shrimp-task-manager prove large catalogs can succeed with discipline. However, maintaining consistent documentation quality becomes exponentially harder as catalog size increases, creating reliability tradeoffs.
4Compliance ≠ Usability
The study reveals a stark gap between technical compliance and practical usability. Archived reference implementations outperformed actively maintained commercial servers, proving agent reliability depends more on writing quality than engineering effort. Current compliance checkers miss these human factors entirely.
5Live Model Test Results
Dynamic testing with real models confirmed static analysis findings. Poorly documented servers showed 84% tool selection accuracy and 50% incorrect tool usage rates for out-of-scope tasks. This 'false positive' behavior where agents act when they should abstain poses significant production risks.
6Hallmarks of Quality MCP Servers
Top performers share clear patterns: tool descriptions answering what/when/return questions, parameter docs with formats and examples, consistent verb_object naming, and explicit required fields. These human-centric design choices dramatically improve agent reliability without protocol changes.