How data compliance companies are turning to web crawlers to take advantage of the GDPR business opportunity

Over the last couple weeks, GDPR has brought data protection center stage. What was once a fringe concern for most businesses overnight became a burning problem that needed to be solved immediately.

With the sweeping changes that GDPR has introduced, it has proven itself to be a huge headache for companies big and small.

However, GDPR has been a goldmine for some savvy companies who positioned themselves to take full advantage of the surge in demand for data compliance solutions.

In this article, I’m going to share with you some of the ways compliance and marketing companies have been using data extraction technologies to take full advantage of the GDPR business opportunity, and how you can too.

GDPR Business Opportunity

Over the last number of years, the personal data of technology company customers has become “the new oil”. With the emergence of more powerful data processing systems and artificial intelligence, companies have been able to develop ever more sophisticated business intelligence systems to precisely predict the needs and wants of consumers.

However, recent scandals such as Cambridge Analytica and Facebook showed the world that our personal data can often be used for unintended and sometimes malicious purposes.

Despite being in the works for years, the introduction of GDPR has sync’d up precisely with the time consumers needed reassurance about their personal data the most.

GDPR introduced a raft of new regulations in terms of how the personal data of EU citizens can be collected, stored and used for business purposes. Along with some pretty hefty fines if you are found to be in breach of these regulations.

This has forced companies big and small to rethink how they interact with their customers and how they manage and use their data.

However, a recent survey from data analytics firm SAS found only 53 percent of EU companies and 30 percent of US companies will meet the deadline to comply.

"I think a lot of companies are kind of sitting on their hands and seeing, well, how does this play out?" said David Smith, head of GDPR technologies at SAS UK & Ireland.

With GDPR now officially in effect, compliance and data management companies are taking full advantage of the GDPR business opportunity.

From consultancy services to data mapping software and personal information management solutions there has been an explosion in demand from companies for GDPR related services.

As a result of Zyte being one of the leading providers of data extraction and web crawling solutions we’ve been able to witness first-hand the range of technologies that companies have been developing to take advantage of this opportunity.  

Due to confidentiality constraints, we can’t discuss who we’ve worked with, however, we can give you examples of the technologies we’ve been asked to build for our clients in the data protection space.

GDPR Auditing

With the transition to GDPR, one of the biggest needs for companies has been the need to audit their current practices and how they manage the personal data of customers and online visitors.

For smaller companies, this can be a relatively easy task to accomplish manually, however, as the size of a company’s website and databases increases it becomes virtually impossible for this auditing process to be completed manually.

With this realization in mind, Zyte has seen a surge in interest from data compliance consultants, software providers and even companies themselves looking to develop automated GDPR auditing solutions.

These auditing solutions can crawl websites and databases to identify and analyze forms, cookies, and personal information for GDPR compliance.

Using tools like these companies and data compliance consultants are able to automate the time-consuming task of identifying all the marketing scripts (first and third party) and cookies that are being fired on the site. Segment them into high and low-risk compliance issues, and identify the corrective actions needed to ensure GDPR compliance.

These solutions analyze websites for the “fingerprints” of data collection, tracking cookies and marketing scripts that are now covered by GDPR. Some of the most common fingerprints our solutions have been designed for:

1. Social Media Retargeting Pixels (Facebook, etc.)
2. Email Sign up Forms
3. Traffic Analytics Cookies – Google Analytics

Using these automated auditing solutions data compliance consultants have been able to provide comprehensive analyses of their client’s websites at scale, enabling them to take on more clients without compromising the quality of their work.

GDPR Compliance Scanners As Lead Generation Tools

While a lot of data compliance companies have developed GDPR auditing tools to make their in-house systems more productive, quite a number of companies are now using GDPR compliance scanners to directly grow their businesses.

Organisations such as “Everywhere Network”, have released free GDPR compliance scanners and tools that act as lead generators for their own consultancy firms.

Interested companies can have their website audited for GDPR compliance in exchange for giving them their email address. Once the company sees the results of their free scan they can be upsold the consultancy services or data compliance software solutions that the company provides.

These free GDPR compliance scanners have given companies a constant stream of highly qualified leads who are in need of data compliance advice.

Some other companies have gone so far as to use these GDPR scanners as outreach tools.

Instead of conducting business development on unqualified prospects, companies are now pre-scanning the websites of prospects for GDPR compliance issues and only contacting those who are in non-compliance with GDPR.

Personal Data Finders

Web scraping has traditionally had a bad name when it comes to personal data, however, with the onset of GDPR a number of companies have started to use web scraping solutions to help their clients ensure they are compliant with GDPR.

One such example is a tool Zyte developed for a client that crawled their client’s websites to identify and record any personal data that they might unknowingly be displaying.

For smaller companies with relatively small websites, this mightn’t be much of an issue, however, if you manage a large old site which has traditionally had multiple contributors, keeping track of the personal information being displayed on the website can be very difficult. This is particularly a problem for large organizations such as universities, government agencies, multinational corporations who’ve built large highly distributed sites with numerous contributors.

Using these personal data finders, site administrators can identify all the personal information being displayed on their website and seek the permission of that person for their data to remain or to be removed from the website. Thereby, ensuring GDPR compliance.

Finally, a number of companies have gone one step further and are empowering people to take back control of their personal data.

Companies like LePrivacy.org are giving the average person an end-to-end solution to finding and removing all unauthorized mentions of their personal data online.

These companies have built custom search engines that search the websites of the major data-brokers and people-search websites on a daily basis to identify any mentions of their client’s personal information.

Clients simply give the company their first name, last name, occupation, and address, and these custom-built search engines will identify any websites infringing on their personal data.

If identified, these tools can then automatically navigate to the perpetrating websites opt-out/removal page and request access to all the data the company holds on their client and if necessary request for this data to be deleted.

Opportunity to Grow Your Business

With GDPR still in its infancy and it is widely toted that other countries are going to follow the EU’s lead and tighten regulations around consumers personal data it still isn’t too late for companies to take advantage of the huge business opportunity GDPR presents.

If you have any interest in the web crawling tools described above or if you have your own ideas, then don’t hesitate to contact our team of crawl engineers who can assist you in developing your own GDPR and data compliance tools.

At Zyte we always love to hear what our readers think of our content and any questions you might. So please leave a comment below with what you thought of the article and what you are working on.