# Analyzing MCP Server Tool Documentation Integrity in Isolated Environments A technical audit of 67 MCP servers reveals discrepancies between README claims and implementation. - **Source:** [Hacker News: Show HN](https://news.ycombinator.com) - **Published:** 2026-11-01 - **Topic:** [Software & Engineering](https://brifing.si/c/engineering) - **Original article:** - **On Brifing:** https://brifing.si/s/2ac9124f-ca19-46b6-8987-c6f6424bb093 ## Isolated Testing Methodology The study analyzed 67 MCP servers by running them in a restricted environment with no internet access. By forcing all network requests to the loopback address, the researchers ensured no external dependencies influenced the results. This controlled setup allowed for a pure assessment of how servers document their internal tool capabilities. ## README Documentation Discrepancies The audit compared the tools offered by each server against their respective README files. Out of 1,143 total tools offered, 24 servers provided tools that were not explicitly named in their documentation. This highlights a significant gap between the actual functional implementation of these servers and the information provided to end users. ## Measurement Constraints and Caveats The methodology relied strictly on the README files shipped within each package. Servers that documented tools in external reference files were flagged as having unnamed tools, potentially skewing results for packages with small README files under three kilobytes. The findings reflect the strictness of the automated rule as much as the package quality. ## Server Performance and Reliability All 67 measured packages successfully started in the isolated environment, confirming basic operational stability for the tested set. No row in the dataset indicates a failure to initialize, providing a baseline for tool availability. The study explicitly avoids ranking servers by usage, focusing instead on the technical accuracy of their provided documentation. ## Verification of Tool Claims The calibration process utilized ten curated measurements to verify that the instrument correctly identifies planted claims about tool functionality. While the primary analysis reads README files via automated rules, this calibration ensures the detection mechanism is robust. It confirms that the tool lists are accurately captured without needing manual human intervention. ## Transparency and Evidence Access Full evidence for every row, including tool lists, README digests, and complete execution traces, is available in the level1/evidence/ directory. Each package is represented by a single JSON file alongside the source code used for the measurement. This transparency allows developers to verify the findings and understand the specific discrepancies identified. ## Responsible Disclosure Protocol The project follows a strict disclosure timeline, with 29 findings currently unpublished. These pages are released only after maintainers have been notified and a three-day waiting period has elapsed. This approach ensures that developers have an opportunity to address documentation gaps before their specific package findings are made public to the community. ## Implications for MCP Ecosystem The study underscores the necessity for standardized documentation practices within the MCP ecosystem. As servers become more complex, the reliance on accurate README files for tool discovery grows. Developers should prioritize aligning their code-level tool definitions with their public documentation to prevent integration errors and improve the overall reliability of server implementations. ## Sources covering this story - https://mcp.brydge-ai.com — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._