PINGDOM_CHECK
ArticleInsight and analysisDeveloper interest

Looking Back at 2018: A Year of Growth and Challenges

Looking Back at 2018 - Join us in reminiscing about the highlights and achievements of the year 2018.

Ian Kerins

4 min read ·

Looking Back at 2018: A Year of Growth and Challenges

Looking back at 2018

What a year 2018 has been for Zyte !!

It’s hard to know where to start…

This year has seen tremendous growth at Zyte , setting us up to have a great 2019.

Here are some of the highlights of 2018…

Data, data and more data

Data continues to be an important element of our customers business strategy and to support this we saw continued growth in the amount of data we help them extract from over 108.6 billion pages in 2018, an average of over 9 billion pages per month.

What is more, the number of items we extracted from those pages rose by 102% from 103 billion in 2017 to 209 billion in 2018. Signifying the growing appetite for every more new data types amongst our customers.

To deliver this level and scale of service for our customers we continued to invest heavily in our core technologies ensuring continued reliability and robustness of Scrapy Cloud and Smart Proxy Manager .

Scrapy Cloud - to deploy and run your web crawlers

Building on the improvements we made in 2017, we again focused on improving the technical capabilities of the platform to ensure the 1M Developers who rely on our tools can get the data they need.

Putting this into numbers we saw another year of growth in the platform with nearly 120,000 developers signing up to use Scrapy Cloud.

What’s more behind the scenes we’ve been working on an overhaul of the platforms user interface which will be released in early 2019. We’re super excited!!!

Smart Proxy Manager

Smart Proxy Manager has gone from strength to strength in 2018. This year Smart Proxy Manager processed over 90 billion successful requests, up from 60.5 billion in 2017.

Zyte making waves

This year, Zytans (formerly Shubbers) were active as ever in the open source, remote working and developer communities.

Zyte participated in and were sponsors of the Running Remote conference in Bali and attended the Grow Remote conference in Ireland, along with attending numerous other remote working conferences and meetups.

Sanaea Daruwalla, Zyte's Head of Legal, was invited to speak at the annual IAPP Europe Data Protection Congress in Brussels, to share her legal insights on how to safely scrape the web in the era of data protection and GDPR.

And always, our developers were active participants and contributors to the open source community along with attending countless developer conferences and meetups throughout the year.

So many Zytans (formerly Shubbers)!!!

It seemed like not a week went by without us welcoming a new member to the team on Slack. Which is actually not an exaggeration, as over the course of 2018 the number of Shubbers has grown 47%.

An increase of 69 people, which is more than one per week. So it was no wonder it felt like we were welcoming new people to the team every day!

Meaning that we are positioned to support our ever growing customers base in 2019 and continue to advance the technologies for web scrapers around the world.

But doesn’t mean we’re planning on stopping here...

We are still looking to hire world-class people in all areas of our business, from developers and data scientists, to salespeople, marketers and support functions. So if you’re interested in working with one of the best remote working companies around then check out the open positions we’re hiring for at the moment.

Culture & team spirit

With such a fast growing fully distributed team, we continued our efforts to ensure our culture and company values were alive and well across the organisation.

We re-launched our values after much collaboration and input from fellow Shubbers, expanded our Shubber praise and recognition programme, created cross functional teams to support global collaboration within a remote environment, set up the Shubber Voice to allow people to voice their ideas for the company, and WaterCooler channels to enable Shubbers to talk and gossip whilst being thousands of kilometres apart.

But no year is complete without our signature Shubber GetTogethers…

{% video_player "embed_player" overrideable=False, type='scriptV4', hide_playlist=True, viral_sharing=False, embed_button=False, width='1920', height='1080', player_id='6848504722', style='' %}

This November 120 Shubbers travelled from all over the world to Lisbon to meet up, make new friends and discuss the company direction.

We held a two-day Town Hall where every Scrapinghub team got a chance to share the progress they’ve made over the last year and their plans for the coming year.

And of course, it wasn’t all work and no play. There was plenty of time for workshops and team bonding to strengthen the Scrapinghub team.

Senior leadership

2018 was the year we completed our senior leadership team, putting experienced leaders in charge of each aspect of Scrapinghub’s key areas. Our leadership team is:
And we welcomed...

  • Oonagh Killeen, as Head of Finance
  • Sanaea Daruwalla, as Head of Legal
  • Suzanne Hassett, as Head of Professional Services
  • Marie Moynihan, as Head of Marketing

Ensuring we have a 50/50 gender split amongst our Senior Leadership team. All in all, having our SLT in place gives Zyte a solid foundation to grow and expand in the years to come.

Looking ahead to 2019

With the ever-increasing use of big data and artificial intelligence by the world’s leading companies there is a huge appetite for reliable sources of high-quality web data, so 2019 is shaping up to be a great year for Zyte .

Not only that, we also have numerous new products and product enhancements in the works that will increase our capabilities and ensure we can better serve our customers in 2019.

If you’d like to be part of this journey then we’re Hiring, so see if any of our open roles takes your fancy. Until next year...Happy Scraping!

Try Zyte API

Build your first scraper in minutes

Free trial, no credit card. From a single request to production in an afternoon.

Get started

Continue reading

The harness matters more than the model - Podcast EP07
AI-assisted data extraction

The harness matters more than the model - Podcast EP07

"The model is the engine — but the harness is everything else." In Episode 7, we dig into why the infrastructure layer around your AI model matters more than the model itself, rank the best models available right now, and ask whether the open-weighted revolution is about to make frontier subscriptions obsolete.

John Rooney11 min read
How to build your first Scrapy extension
Developer interest

How to build your first Scrapy extension

My Scrapy project now plays a triumphant fanfare when a crawl finishes clean and a sad trombone when it doesn't, which is also how I finally learned what Scrapy's extension points are actually for.

Ayan Pahwa11 min read
Now what exactly is loop engineering? And where do Anthropic's Fable 5 model and web scraping fit in?
Developer interest

Now what exactly is loop engineering? And where do Anthropic's Fable 5 model and web scraping fit in?

A viral clip from Claude Code's creator put a name to something a lot of us have been circling: loop engineering. Here's why web scraping may be its best-fit domain — and what that means in practice.

Ayan Pahwa12 min read
Automate deployment of your web scraper on VPS with Ubuntu 24.04 cloud-init
Developer interest

Automate deployment of your web scraper on VPS with Ubuntu 24.04 cloud-init

Your VPS is ready, but now you need to work through the same sequence you have run a dozen times before: apt update, apt install python3-pip, pip install scrapy, playwright install chromium, the Chromium dependency list that never installs cleanly on the first try, Redis, possibly Postgres, whatever else this particular project needs.

Ayan Pahwa9 min read
NotAnInterview: “I Have Superpowers Now"
Developer interest

NotAnInterview: “I Have Superpowers Now"

The problem was a project with 12,000 websites to crawl, and there’s no world where you write custom spiders for 12,000 websites, not with a human team and certainly not sustainably. So Javier built a workflow: a set of AI prompts that could analyze a website, figure out its structure, and generate a crawl configuration that a generic spider could then use.

Neha Setia Nagpal9 min read
Flatcar Linux for web scrapers: deploy immutable containers with just one config file
Developer interest

Flatcar Linux for web scrapers: deploy immutable containers with just one config file

The next time you spin up a VPS to give it a persistent home, you spend the better part of an afternoon rebuilding from memory. Here's a tool to help using Flatcar Linux

Ayan Pahwa11 min read

The Community · Newsletter

The best of Zyte and the data web, in your inbox.

One curated edition — new articles, product updates, and the stories shaping the data web. No noise.