Analyzing MCP Server Tool Documentation Integrity in Isolated Environments
A technical audit of 67 MCP servers reveals discrepancies between README claims and implementation.
1Isolated Testing Methodology
The study analyzed 67 MCP servers by running them in a restricted environment with no internet access. By forcing all network requests to the loopback address, the researchers ensured no external dependencies influenced the results. This controlled setup allowed for a pure assessment of how servers document their internal tool capabilities.
2README Documentation Discrepancies
The audit compared the tools offered by each server against their respective README files. Out of 1,143 total tools offered, 24 servers provided tools that were not explicitly named in their documentation. This highlights a significant gap between the actual functional implementation of these servers and the information provided to end users.
3Measurement Constraints and Caveats
The methodology relied strictly on the README files shipped within each package. Servers that documented tools in external reference files were flagged as having unnamed tools, potentially skewing results for packages with small README files under three kilobytes. The findings reflect the strictness of the automated rule as much as the package quality.
4Server Performance and Reliability
All 67 measured packages successfully started in the isolated environment, confirming basic operational stability for the tested set. No row in the dataset indicates a failure to initialize, providing a baseline for tool availability. The study explicitly avoids ranking servers by usage, focusing instead on the technical accuracy of their provided documentation.
5Verification of Tool Claims
The calibration process utilized ten curated measurements to verify that the instrument correctly identifies planted claims about tool functionality. While the primary analysis reads README files via automated rules, this calibration ensures the detection mechanism is robust. It confirms that the tool lists are accurately captured without needing manual human intervention.
6Transparency and Evidence Access
Full evidence for every row, including tool lists, README digests, and complete execution traces, is available in the level1/evidence/ directory. Each package is represented by a single JSON file alongside the source code used for the measurement. This transparency allows developers to verify the findings and understand the specific discrepancies identified.
7Responsible Disclosure Protocol
The project follows a strict disclosure timeline, with 29 findings currently unpublished. These pages are released only after maintainers have been notified and a three-day waiting period has elapsed. This approach ensures that developers have an opportunity to address documentation gaps before their specific package findings are made public to the community.
8Implications for MCP Ecosystem
The study underscores the necessity for standardized documentation practices within the MCP ecosystem. As servers become more complex, the reliance on accurate README files for tool discovery grows. Developers should prioritize aligning their code-level tool definitions with their public documentation to prevent integration errors and improve the overall reliability of server implementations.