I was recently invited to speak at the IAPP Europe Data Protection Congress in Brussels about web scraping and GDPR. The panel also included Claire François of Hunton Andrews Kurth and Peter Brown from the Information Commissioner’s Office (ICO). For more information, you can check out my blog about this topic GDPR Compliance for Web Scrapers: The Step-by-Step Guide.

Key takeaways from the event:

1: Scraping Personal Data - Legitimate Interest

There are only two legal bases for scraping personal data (1) consent or (2) legitimate interest. While consent is rare in web scraping cases, it’s the cleaner of the two options, so much of the panel discussion at the IAPP Congress was spent on legitimate interest. In reality, legitimate interest will typically be the only legal basis at your disposal when scraping personal data, so is there a compliant way to use legitimate interest as a legal basis when web scraping?? Maybe . . . sometimes . . . if you’re really careful.

2: Legitimate Interest Explained

Where no other legal basis is available, many companies are turning to a legitimate interest. Legitimate interest can be used where the use case for the personal data is a use that the data subject would reasonably expect and have a minimal privacy impact. When determining if this is the case, this three-factor test can be utilized:

1. Identify the legitimate interest (for example, Recital 47 of the GDPR states that “...the processing of personal data for direct marketing purposes may be regarded as carried out for a legitimate interest.”);
2. Show that processing is necessary to achieve that legitimate interest; and
3. Balance the legitimate interest against the individual’s rights.

Following on from the Recital 47 example above, in order to complete the final two steps, you would need to (1) show that your scraping of the personal data is required in order to achieve your legitimate interest of direct marketing (meaning couldn’t get the data through some other legal basis, like consent), and (2) ensure that your legitimate interest to the data is not outweighed by the individual’s right to privacy. When weighing the interests, think about the privacy impact your use of the data might have on the individuals and whether the people whose data you scrape would be surprised or likely to object to your use of their data.

Always ensure that you document how you assessed legitimate interest, and if you need additional guidance the ICO has published a legitimate interest assessment form on their site. If you are able to successfully pass the three-factor test and assessment, you may be able to use legitimate interest as your legal basis for scraping personal data.

3: Protecting Data Subject’s Rights?

Well, that’s where things get trickier. If for example you’re using Recital 47 and make a determination that your processing of personal information for direct marketing purposes qualifies as a legitimate interest, how do you inform the data subjects that you have their information or provide them with their right to access data, correct errors, object to processing, and request erasure?

Some ideas considered during our panel discussion:

1. Conducting a Data Processing Impact Assessment (DPIA)
2. Review the use case for the data to determine if it aligns with the data subject’s original purpose for sharing the data
3. Territorial scope -- consider where the scraping is taking place and the location of the company that is conducting the scraping. Remember, GDPR only applies if:
   

   (a) you are established in the EU and you are scraping data in the context of the activities of your EU establishment; or

   (b) you are not established in the EU and you scrape personal data of individuals in the EU.

4. If the privacy policy of the website scraped lists categories of third parties that may access the personal data and you fall within those categories
5. Obtaining consent after scraping the data.

There are potential pitfalls with all of these options that would require legal guidance, but it was great to get this conversation going in an environment full of data protection experts.

4: ICO Recommendations

It was great to hear the ICO’s recommendation, given that they are the ones enforcing GDPR. The ICO was clear that they don’t have any specific recommendations on web scraping, but you can look to their recommendations on “Invisible Processing” to get some guidance. Invisible Processing is the “processing of personal data that has not been obtained directly from the data subject in circumstances where the controller considers that compliance with Article 14 would prove impossible or involve a disproportionate effort.” The ICO considers Invisible Processing “high risk” and thus requires a DPIA to be conducted prior to such processing.

A DPIA is an assessment that helps you analyze, identify, and minimize the data protection risks of a project, to ensure compliance with GDPR. The ICO provides a step-by-step list for conducting a DPIA, which includes:

1. Identify the need for a DPIA
2. Describe the processing
3. Consider outside consultation
4. Assess necessity and proportionality
5. Identify and assess risks
6. Identify measures to mitigate the risks
7. Sign off and record outcomes
8. Integrate outcomes into a project plan
9. Keep your DPIA under review

There are also various data protection software packages on the market, which walk you through a step-by-step DPIA process. At Zyte , if we were to utilize the DPIA approach, it would be our preference to conduct it within the data protection software we use, so that we’re conducting the most robust and thorough analysis possible.

Conclusions

Attending and speaking at the IAPP Congress helped to get web scraping on the minds of some of the leading data protection experts in the world, and we’re hopeful that this will turn into direct guidance from organizations like the ICO about web scraping. In the meantime, Zyte will continue to advocate for fair scraping of public data and will continue to guide our customers to help them lawfully scrape personal data.

Disclaimer: I am a lawyer, but I am not your lawyer and the recommendations in this post do not constitute legal advice. The commentary and recommendations outlined are based on Zyte’s experience helping our clients (startups to Fortune 100’s) maintain GDPR compliance whilst scraping 7 billion web pages per month. If you want legal advice regarding your specific situation then you should consult a lawyer.