Summary
Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/ Fou…
What this means for you
This story is about a proposed website crawling standard backed by Google, Microsoft, Anthropic and OpenAI, not about visas, immigration, jobs, tuition or consumer prices. For Chinese readers in the US who operate websites or publish content online, the article says the standard aims to let them block all scrapers that collect training data for generative AI models with a single robots.txt entry, instead of adding a separate entry for every AI provider. Microsoft says it will provide a crawling setting called AI Grounding so website operators can prevent its services from sending requests to access a webpage to answer a user question, and OpenAI plans similar controls. The article states that Microsoft and OpenAI intend to add support in the coming months and that Google will follow suit sometime next, but it does not describe how this would affect individual users, prices or access.
Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC today announced plans to support an upcoming website crawling standard.
The technology will enable website operators to block artificial intelligence developers from using their content to train their models.
Many websites contain a text file titled “robots.txt” that informs crawlers what content they may access. A publisher can use the file to block scraping across the board or only make specific webpages inaccessible. It’s also possible to include certain other instructions. For example, a company can ask crawlers not to visit its website too often.
Website operators historically used robots.txt files to manage how search engines interact with their properties. Nowadays, the technology also helps regulate the activity of the scrapers with which AI providers collect training data.
A company that wishes to disallow training-related scraping must make multiple edits to its website’s robots.txt file. In particular, it needs to add a separate entry for every single AI provider. If an AI provider uses multiple scrapers to collect data, each scraper requires its own robots.txt entry.
The crawling standard Google, Microsoft, Anthropic and OpenAI are backing aims to simplify the process. It would allow website operators to block all scrapers that collect training data for generative AI models with a single robots.txt entry. The setting wouldn’t apply to models that “perform exclusively non-generative tasks.”
The structure of robots.txt files follows a standardized format called the Robots Exclusion Protocol. Google, Microsoft, Anthropic and OpenAI plan to incorporate their crawling standard into the format. They have already submitted a draft version of their proposal to the Internet Engineering Task Force, the nonprofit that maintains the Robots Exclusion Protocol.
Microsoft stated in a blog post that it will also provide a crawling setting called AI Grounding. It’s designed for situations where an AI model seeks to access a webpage in order to answer a user question rather than to collect training data.
Microsoft’s setting will enable website operators to prevent its services from sending such requests. The feature will also extend to third-party AI tools that use Microsoft technologies to process prompts.
OpenAI plans to offer similar controls. Both the ChatGPT developer and Microsoft intend to add support for the crawling standard in the coming months, while Google will follow suit sometime next year. Meta Platforms Inc. expressed support for the idea behind the technology but hasn’t yet committed to implementing it.
A message from John Furrier, co-founder of SiliconANGLE:
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.
Are you an AWS customer? Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.
Source: SiliconANGLE
Join the discussion
If you run or publish a website, would you use a robots.txt setting to block AI training scrapers, or would you allow scraping? What has shaped that choice?
