How far should a file built for robots be relied upon to adhere to a human’s personal wishes?
It may be a strange question to ask, but new European Union (EU) proposals affecting one area of web scraping make it a very live one.
This summer, the European Data Protection Board (EDPB), an EU independent body charged with providing guidance on the application of General Data Protection Regulation (GDPR) across member states, published “Guidelines 03/2026 on web scraping in the context of generative AI”, which addressed this very topic.
What’s going on?
The guidelines function as a highly targeted, practical instruction manual for how private entities should behave under the GDPR when scraping the open web for data to train or fine-tune generative AI models.
Right now, it is a proposal, out for consultation until October 30, 2026.
Although EDPB guidance is not law, in general , its guidance becomes the official European baseline for data protection which is enforced by European states’ data protection authorities. So it is critical that this draft guidance on AI data-scraping compliance is properly scrutinised.
What’s on the table?
The draft is careful, detailed and narrow in scope: it addresses scraping by private entities for the purpose of training generative AI models, and it does not set out to regulate web scraping overall, which is to be welcomed.
Machine protocols in the spotlight
But, within that scope, it would do something that deserves closer attention. It would give real legal weight to on-site technical measures in deciding whether the collection of personal data is lawful. That means standards like robots.txt, the nascent ai.txt format, "robots authentication” and CAPTCHA.
Related: The copyright question
Files like robots.txt are already being considered as a valid mechanism with which website owners can express legally-binding preferences to opt out from being scraped for the entirely unrelated purpose of copyright protection.
That EU AI act’s code of practice says: “Crawling systems must possess the capability to recognise and honor machine-readable rights reservations, including robots.txt files, consistent with RFC 9309 standards. … .” A number of general purpose AI developers have voluntarily signed up to this code. Some regional European courts have made similar determinations but those cases were heavily contextual and fact-specific.
In the case of copyright for AI training the position is far from settled. By contrast, the EPDB’s new proposals regarding personal data for AI training under GDPR could result in a very fixed position on this principle in Europe in the near term.
What could change?
The EPDB’s draft addresses these technical measures in two places:
1. ‘Data minimisation’ principle
Because GDPR already urges a principle of minimal data collection, section 37 of the EPDB guidelines says AI scrapers must "exclude from the collection websites which clearly oppose the scraping of their content, (as indicated) through the use of technical measures”.
Which measures? Those “such as the use of robots authentication to view content, robots.txt or ai.txt files, or CAPTCHA, which … aim to prohibit access to pages by robots”.
2. Legitimate interest
Having a “legitimate interest” is already one of six “lawful bases” under which a data controller can collect someone’s personal data.
The EPDB’s draft guidelines lay out a three-condition assessment, including a “balancing test”, to determine whether an entity scraping for AI has such a legitimate interest. And it uses two examples to bring this to life.
Example 4: Measures absent - scraping for AI allowed
“A person uploads their data to a content sharing platform that is freely accessible and does not contain any prohibitions against web scraping, and the platform communicates the possibility of the content being scraped to its users. In this instance, data subjects can reasonably expect third parties to scrape that data to develop AI models.”
Example 5: Measures present - scraping for AI discouraged
“If a person uploads their data to a content sharing platform that prohibits scraping through the use of robots.txt files and the implementation of CAPTCHA, and expressly states on their site that they do not allow the use of their users' data for the development of AI models, they cannot reasonably expect third parties to scrape that data for that purpose.”
Adding it up
Read together, the effect is to make a technical artefact close to decisive. Its presence tends toward unlawfulness; its absence toward lawfulness.
But this is where the EDPB logic arguably is strained.
Files are faulty signals
Two problems sit at the heart of the EDPB’s approach here.
1. robots.txt is unsuitable for expressing human preferences
First, standards like robots.txt were never designed to express the personal preference of an individual whose data appears online. They are not even able to identify what is “personal data” in the first place.
- Rather, the “robots exclusion protocol” is a measure intended to manage automated traffic and to fend off bot attacks. robots.txt is a voluntary measure addressing how automated crawlers behave on a site; in its most developed form it speaks to crawler behaviour, not subsequent AI use, and it binds no one.
- Likewise, CAPTCHA is a measure intended to avoid bot attacks and similar automated abuse.
Neither measure was conceived as a statement of a data subject's preferences, and neither distinguishes personal data from the ordinary, non-personal content that makes up most of the web.
2. Individuals don’t control robots.txt
Second, the EPDB has misunderstood a critical aspect about who actually controls and publishes these technical files and measures.
Most of the time these days, it is not ordinary individuals (ie. owners of “personal data”) who set these signals (few of us really run a website); rather, it is platform operators which play host to individual users. So what the guidance is proposing is that platforms set privacy signals on people’s behalf.
If you treat a platform’s technical setting as a proxy for a person’s intent, you attribute a choice to individuals which they never made, and over which they have no control.
This risks misrepresenting an individual’s wish entirely. For example, someone may post content to a social network precisely because they wanted it to be public, but this wish may be smothered if the platform switches to a “no-AI” policy.
There is a neat internal tension here that the draft could resolve in its own favour. Paragraph 45 of the EDPB draft guidelines rightly states that the absence of a robots.txt file does not amount to an individual’s consent to have personal data scraped for AI.
- That is correct: a voluntary technical convention does not carry the legal weight of a considered expression of will.
- But the logic cuts both ways. If absence is not consent, presence cannot be prohibition.
The honest position is that neither should be decisive - a signal set by a platform is, at most, one factor in a holistic assessment, not a switch that can turn lawfulness on or off in the same way that cookie banner options can. Given there is no interaction between the human whose personal data is impacted, and the setting of a robots.txt file, this cannot form the expression of a human preference.
Negative consequences
The EDPB’s guidelines, in their current form, would introduce several market risks.
Unintended anti-competitive dynamics
The draft measures would create a situation in which the largest platforms, which hold the most personal data, are effectively able to define the boundaries of a statutory legal basis by editing a text file.
The result is asymmetric:
- A dominant platform deploying these kinds of measures - robots.txt, CAPTCHA, or different treatment for scraper traffic - would find its own lawfulness upweighted against that of rivals.
- Meanwhile, though, it could continue to exploit that very same first-party data for its own AI, or even license it selectively to chosen partners, as per terms and conditions its users sign.
Smaller competitors, new entrants, academic researchers and public-interest users are shut out; incumbents are entrenched.
Such an outcome sits awkwardly beside the EU's own data-access and open-data objectives. Both of the Data Act and the Digital Markets Act seek to reduce the data advantages of large gatekeepers, not reinforce them.
The European Data Protection Board is a data protection authority, not a competition regulator, and I am not suggesting it should become one. The point is narrower: the integrity of the balancing test itself is undermined if a unilateral measure controlled by a dominant operator is allowed to operate as a de facto legal veto over lawful processing by everyone else.
The open internet is at stake
It would be easy to dismiss this as a niche debate about AI training. It is not.
Publicly accessible information underpins a wide range of lawful, valuable activities - comparison and price-transparency services that benefit consumers; academic and scientific research; investigative journalism; accessibility and archiving; fraud and brand-abuse monitoring; and ordinary competitive market intelligence. Much of it depends on the ability to access public data at scale.
Widespread over-reading
Guidance framed for generative-AI training needs to stay in its lane. EDPB guidelines are intended as a non-binding form of “soft law”, designed for adoption by national data protection authorities. But its reasoning is routinely cited by supervisory authorities, by courts, and by private parties - well beyond its stated scope.
A principle that elevates any website measure into a bar on collection risks solidifying into a general proposition that "robots.txt equals law". This would chill far more than AI development.
The better tool is already in the drawer
None of this is an argument against protecting people. It is an argument about how.
Rather than resting so much weight on internet signals that were never meant to bear it, I would hope the final guidance ties responsible web data collection and use - and the legitimate collection and use of personal data - to the privacy-by design methods and technologies that actually do the work: anonymisation, obfuscation and pseudonymisation among them.
The EDPB does not need to look far. Its own draft “Guidelines 02/2026 on Anonymisation” (adopted the very same day and open for consultation on the very same deadline) set out a modern, context-sensitive framework for when data ceases to be personal at all.
It says effective anonymisation can help a data controller satisfy the legitimate-interest balancing test: “The immediate anonymisation of the given data, and immediate deletion of the original data, can limit the overall impact.”
Anchoring the scraping guidance to that framework - focusing on what a scraper actually does to protect individuals in the data, rather than on who happened to configure a file on a server - would be a recognisably human-centred and GDPR-principled approach.
Protecting individuals and protecting an open, contestable web are not competing goals. A proportionate, evidence-based treatment of technical signals serves both.
EDPB’s consultation is open until October 30, 2026. It is worth engaging.
