Cloudflare is requiring AI companies to separate their web crawlers from search bots by September 15 or face automatic blocking on publisher sites. The move aims to ensure content creators are compensated for material used in AI training.
Cloudflare has issued a deadline for artificial intelligence companies to distinguish between web crawlers used for search indexing and those deployed for AI model training. Companies failing to comply by September 15 will be blocked by default across many publisher websites using Cloudflare's services.
The policy addresses growing tensions between AI developers and content creators over unauthorized use of published material. Publishers argue that AI companies scrape their content without permission or compensation to train large language models and other AI systems.
Under Cloudflare's framework, AI crawlers would need separate identification from search engine bots. This allows publishers to permit search indexing—which drives traffic—while blocking AI training operations. Publishers can then choose to allow AI access only through paid licensing agreements.
Cloudflare's move targets the fundamental business model dispute in AI development. Companies like OpenAI, Google, and others rely heavily on web-scraped content to train their systems. Meanwhile, publishers including news outlets and content creators argue they deserve compensation or control over how their work is used.
The deadline gives AI companies three months to restructure their crawling infrastructure. Cloudflare will implement automated blocking of non-compliant crawlers across its network, which protects millions of websites globally.
This policy represents one of the first major platform-level interventions in the AI-content creator dispute. Rather than taking sides, Cloudflare is creating technical infrastructure for publishers to enforce their own preferences.
The move could pressure AI companies to negotiate licensing deals with publishers or develop alternative training methods. Some AI firms may already use separate crawlers for different purposes and could easily comply. Others may need significant technical changes.
Cloudflare's approach offers a middle ground between blanket AI bans and unrestricted scraping, but its effectiveness depends on widespread adoption and enforcement across the internet.
Z.ai released GLM-5.3's weights on Hugging Face under a new license that requires large companies to undergo security review before hosting the model. The change marks a departure from the standard MIT license.
Anthropic has introduced the Model Hardware Standard (MHS), a unified interface enabling AI agents to operate robotic arms, lab instruments, and other physical devices. Early testing shows integration time has dropped from weeks to hours.
Open-weight AI companies—those releasing freely available models—are attracting major acquisition interest from tech giants. The trend reflects growing capital investment in the business model of distributing AI models at no cost.
Google Deepmind has upgraded its Co-Scientist AI system to autonomously plan experiments, operate lab equipment, and publish scientific papers. The Gemini-based multi-agent platform demonstrated experimentally validated results across materials science, chemistry, and medical AI development.