Cloudflare and OpenAI Partner to Enhance AI Web Discovery
A joint research pilot improves how AI systems index and surface web content.
1Joint Research Pilot Launch
In July 2026, Cloudflare and OpenAI initiated a research pilot to determine if network signals could improve content discovery for ChatGPT users. By leveraging data from Cloudflare’s infrastructure, the project aims to help AI systems identify and index high-quality, relevant web pages more efficiently than traditional crawling methods currently allow.
2Expanding the Search Index
The pilot involved approximately 1% of Cloudflare’s total network, resulting in the successful discovery and indexing of hundreds of millions of previously unknown web pages. This expansion provides ChatGPT with a broader, more diverse range of sources, ensuring users receive timely and accurate information when querying the AI for specific tasks.
3Optimizing Crawler Efficiency
Cloudflare provides specific signals that help OpenAI’s crawlers distinguish between valuable content and redundant or non-existent pages. By filtering out broken links, unnecessary redirects, and duplicate content, the system reduces wasted computational resources. This optimization ensures that crawler activity remains focused on pages that offer distinct and useful information for users.
4Enhancing User Safety Protocols
Beyond discovery, the research includes testing signals to identify potentially harmful pages, such as those containing fake login forms. By flagging specific compromised pages rather than penalizing entire domains, the system maintains user safety. This granular approach allows for more reliable discovery while protecting visitors from malicious content found on otherwise legitimate sites.
5Balancing Crawl Load Demands
Frequent crawling of unchanged pages creates unnecessary server load for website owners without providing any tangible benefit. The pilot utilizes change signals to determine when a page truly requires a revisit. This balance helps maintain up-to-date information while minimizing the technical burden on infrastructure, creating a more sustainable model for web indexing.
6Feedback Loops and Quality
Once Cloudflare identifies potential pages, OpenAI independently evaluates and ranks them based on relevance and helpfulness. This collaborative feedback loop improves the overall quality of information surfaced to ChatGPT users. By linking back to original sources, the system also drives traffic to publishers, helping users explore the content behind the answers.
7Control and Domain Ownership
The research emphasizes that metadata signals do not grant automatic permission to access site content. Website owners retain control through tools like Cloudflare’s AI Crawl Control, which manages how AI systems interact with their domains. This ensures that participation remains a choice for publishers while maintaining clear boundaries regarding data usage and access.
8Future Phases of Development
The next phase of the partnership will focus on refining page selection and testing faster, more reliable change signals. Cloudflare and OpenAI are currently measuring the business outcomes of improved discovery. Future efforts will continue to explore how better indexing translates into real-world value for both content creators and AI system users.
Original article: Cloudflare Blog — Cloudflare and OpenAI Partner to Enhance AI Web Discovery