# AI Bots Are Overwhelming Linux Development Infrastructure Web scrapers targeting AI training data are causing significant resource drain on kernel development systems. - **Source:** [Hacker News](https://news.ycombinator.com) - **Published:** 2026-08-29 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/8e1aabb8-dab0-4b74-a021-f887ff464470 ## Impact of Scrapers on System HTML rendering requests from AI bots consume more CPU resources than all legitimate git access combined on Linux kernel development infrastructure. Across 5 geographically distributed nodes, 14 CPU cores are constantly occupied just converting commits to HTML. This creates a persistent 'background radiation' load on the system, tying up significant capacity for single-purpose data extraction. ## The Open Source Dilemma Linux development is inherently transparent - git repositories and discussion archives are publicly accessible. This creates a goldmine of training data for large language models, especially valuable as pure pre-AI human-generated content. However, this openness also makes it vulnerable to exploitation, with systems originally designed for human collaboration now being mined by automated scrapers at massive scale. ## Inefficient Data Collection Methods Rather than using efficient methods like git clone, bots choose the most resource-intensive approach: rendering each commit individually as HTML then parsing it. With linux.git containing 1.48 million commits and 922 forks, this generates billions of unnecessary requests. Although forks mostly contain the same objects, bots systematically scrape every possible URL variation regardless. ## Evolution of Bot Detection Initially, bots identifying themselves via user-agent strings could be blocked with fail2ban. Then bots began masquerading as regular browsers. IP-based blocking followed, but bots spread across entire subnets. ASN (Autonomous System Number) blocking proved partially effective but also impacted legitimate automation tools, creating collateral damage in the fight against scrapers. ## Modern Bot Behavior Today's bots originate from millions of residential and mobile IPs, each making just 4-5 requests before disappearing. Blocking is ineffective since IPs are never reused. This 'proxy SDK monetization' has become a major industry, with many IoT devices (including smart TVs) participating in these distributed scraping networks unbeknownst to their owners. ## The Anubis Solution The Anubis challenge was implemented to stop bots: requiring them to find a string that, combined with their IP and a secret, produces a SHA256 hash with 4 leading zeros. Initially very effective, bots adapted within months to solve difficulty level 4. The challenge was increased to level 5, which takes seconds on mobile devices and causes noticeable heating, creating user friction. ## Current Situation and Impacts Today git.kernel.org receives about 6 million daily requests. While 66% are blocked by Anubis, 33% now solve the mathematical challenge to access the site. This demonstrates the perceived value of the training data, as AI companies are willing to expend significant computational resources to obtain kernel development artifacts, despite the increasing costs imposed by defensive measures. ## Future Concerns This situation raises serious questions about the sustainability of open source infrastructure. As AI companies profit from consuming public resources, communities bear the costs. The escalating arms race between scrapers and defenders threatens the fundamental openness that made these resources valuable in the first place, potentially requiring architectural changes to preserve the collaborative development model. ## Sources covering this story - https://people.kernel.org/monsieuricon/creepy-crawlies — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._