Joanne O’Flynn meets with Pablo Hoffman and Shane Evans to find out what inspired them to set up web crawling company Zyte.

Zyte may be a very young company but it already has a great story to tell. Shane Evans from Ireland and Pablo Hoffman from Uruguay came together in a meeting of great web crawling minds to form the business after working together on the same project but for different companies.

In 2007, Shane was leading software development for MyDeco, a London-based startup. Shane’s team needed data to develop the MyDeco systems and set about trying to find an appropriate company that they could trust to deliver the data. Frustrated with the lack of high quality software and services available, Shane took the matter into his own hands and he decided to create a framework with his team to build web crawlers to the highest standard.

This turned out well and it didn’t take long to write plugins for the most important websites that they wanted to obtain data from. Continued support for more websites and maintenance of the framework was required so Shane looked to find someone to help.

Meanwhile, after studying computer science and electrical engineering, Pablo graduated in 2007. Soon after graduating, he set up Insophia, a Python development outsourcing company in Montevideo, Uruguay. One of Pablo’s main clients recommended Insophia to MyDeco and it wasn’t long before he was running the MyDeco web scraping team and helping to develop the data processing architecture.

Pablo could see massive potential in the code and asked Shane six months later if they could open source this web crawler. Cleverly combining the words ‘Scrape’ and ‘Python’, the Scrapy project was born. After Pablo spent many months refining it and releasing updates, Scrapy became quite popular and word reached his ears that several high-profile companies, including a social media giant, were using their technology!

In 2010, after developing a fantastic working relationship, Shane and Pablo could see there was an opportunity to start a company that could really make a difference by providing web crawling services, while continuing to advance open source projects. The two men decided to go into business together with one goal: To make it easier to get structured data from the internet.

With a tight and knowledgeable core group of developers and a huge drive to provide the most functional and efficient web crawlers, Shane and Pablo formed Zyte. They started off as just a handful of hard-working programmers; by 2011 there were 10 employees, by 2012 there were 20 and staff numbers continued to double each year, up to the present where there are now almost 100 employees - or Zytebers as they affectionately call themselves - globally dedicated to developing the best web crawling and data processing solutions.

In 2014 alone, the company scraped and stored data from more than 10 billion pages (more than five times the amount the company did in 2013) and an extra 5 billion have passed through Crawlera. 2015 is already off to a great start with the release of two new open source projects: ScrapyRT and Skinfer, a tool for inferring JSON schemas. The icing on the cake is Zyte’s February announcement of its participation in the DARPA project Memex. It’s testament to the knowledge and experience of a very dedicated team working together all around the world. From small beginnings come great things and it’s clear that for Zyte, a very bright future awaits.