PINGDOM_CHECK
ArticleInsight and analysisDeveloper interest

Extract Summit 2021 - Come Join The Web Data Revolution

We are delighted to announce that Extract Summit 2021 will be a virtual event again. We will create an online experience that will allow the community to interact.

Marie Moynihan

2 min read ·

Extract Summit 2021 - Come Join The Web Data Revolution

Extract Summit 2021 - come join the web data revolution

*For information on 2022 Extract Summit visit this link*

We are delighted to announce that Extract Summit 2021 will be a virtual event again in 2021 brought to you by Zyte . We had to make a quick decision last year and turn the event into a virtual one - the good news was it allowed so many of our community to join us from all over the world. We had 3,000 web data extraction experts sign up. 

We loved that so many more of you could join us so this year we want to create an online experience that will allow the community to interact and network more.

Web data extraction needs are changing and organizations have never been more reliant on data as the differentiator that fuels innovation and business growth. As the web gets bigger and more complex, it’s a challenge for developers to keep pace with changed pages that break spiders, or legitimate extraction efforts that get banned, blocked, or blacklisted by overzealous bots.

We hope to address some of these challenges at Extract Summit and get data enthusiasts all over the world to network and learn from each other.

If you have a great topic you would like covered, our call for proposal is until the end of June.

If you missed the event in 2020 you can still take a look at the recordings - I’ve highlighted some of my favorite sessions from the day

Panel discussion on Legal Compliance in the world of web scraping hosted by Sanaea Daruwalla - Head of Legal at Zyte and Panelists: Sarah McKenna - CEO of SequentuM, Marc Zwillinger - Founder of ZwillGen PLLC and Paul Griffin - CEO of First Compliance. Great discussion and guidance on web scraping compliance and updates in the legal landscape.

Amanda Towler also gave us an update on the work on the TellFinder Alliance: Tackling Online Exploitation with Data, fascinating to see the use of data and the impact it has on solving these crimes. 

Another great panel discussion on Cutting Edge Ways To Tackle Antibot Challenges: Antibots introduce a big and important challenge to solve for anyone who wants to scrape the web at scale. If you don’t have a reliable way to solve these challenges created by antibots, you will not be able to access any data. In this session, the panel of antibot experts at Zyte discussed how to dissect this problem and look at possible solutions.

Mikhail Krobov took us on a whirlwind journey on top tips and tricks for super users of Scrapy. 

We are really excited by some of the speakers on board this year and expect a stellar line-up. Don’t miss out, submit a talk now or register and join thousands of data lovers as we explore all things web scraping and data extraction.

Oh, and did we mention it's completely free!  Come join the data revolution.

Try Zyte API

Build your first scraper in minutes

Free trial, no credit card. From a single request to production in an afternoon.

Get started

Continue reading

The harness matters more than the model - Podcast EP07
AI-assisted data extraction

The harness matters more than the model - Podcast EP07

"The model is the engine — but the harness is everything else." In Episode 7, we dig into why the infrastructure layer around your AI model matters more than the model itself, rank the best models available right now, and ask whether the open-weighted revolution is about to make frontier subscriptions obsolete.

John Rooney11 min read
How to build your first Scrapy extension
Developer interest

How to build your first Scrapy extension

My Scrapy project now plays a triumphant fanfare when a crawl finishes clean and a sad trombone when it doesn't, which is also how I finally learned what Scrapy's extension points are actually for.

Ayan Pahwa11 min read
Now what exactly is loop engineering? And where do Anthropic's Fable 5 model and web scraping fit in?
Developer interest

Now what exactly is loop engineering? And where do Anthropic's Fable 5 model and web scraping fit in?

A viral clip from Claude Code's creator put a name to something a lot of us have been circling: loop engineering. Here's why web scraping may be its best-fit domain — and what that means in practice.

Ayan Pahwa12 min read
Automate deployment of your web scraper on VPS with Ubuntu 24.04 cloud-init
Developer interest

Automate deployment of your web scraper on VPS with Ubuntu 24.04 cloud-init

Your VPS is ready, but now you need to work through the same sequence you have run a dozen times before: apt update, apt install python3-pip, pip install scrapy, playwright install chromium, the Chromium dependency list that never installs cleanly on the first try, Redis, possibly Postgres, whatever else this particular project needs.

Ayan Pahwa9 min read
NotAnInterview: “I Have Superpowers Now"
Developer interest

NotAnInterview: “I Have Superpowers Now"

The problem was a project with 12,000 websites to crawl, and there’s no world where you write custom spiders for 12,000 websites, not with a human team and certainly not sustainably. So Javier built a workflow: a set of AI prompts that could analyze a website, figure out its structure, and generate a crawl configuration that a generic spider could then use.

Neha Setia Nagpal9 min read
Flatcar Linux for web scrapers: deploy immutable containers with just one config file
Developer interest

Flatcar Linux for web scrapers: deploy immutable containers with just one config file

The next time you spin up a VPS to give it a persistent home, you spend the better part of an afternoon rebuilding from memory. Here's a tool to help using Flatcar Linux

Ayan Pahwa11 min read

The Community · Newsletter

The best of Zyte and the data web, in your inbox.

One curated edition — new articles, product updates, and the stories shaping the data web. No noise.