How Zyte API takes care of the fundamental needs of your web scraping project!

In this Newsletter, I want to talk to you all and share my excitement about the launch of Zyte API -it might just change the way you scrape the web forever. I hope after reading this you are inspired to create an account and give it a try.

Zyte API is a game-changer for web scraping, it eliminates the most time-consuming and difficult challenges of scraping, making designing and developing scalable web data projects incredibly simple - so, now you can focus on solving real problems with less code, with quality data at hand.

I am so proud of the talented developers at Zyte who created an API that I can't stop bragging about. So let me actually tell you why I am in awe of Zyte API!

In this issue, you will learn:

1. How Zyte API simplifies the fundamentals of your web scraping project! 2. How to try Zyte API yourself.
3. How to integrate Zyte API with Scrapy and Python.
4. How to Migrate to Zyte from Smart Proxy Manager.

How Zyte API takes care of the fundamental needs of your web scraping project!

When you plan the tech stack for a web scraping project, there are six pieces of the puzzle that require your attention and set the foundation of the project namely -

1. A base technology/ framework, for example, Scrapy.

2. A rotating proxy solution like Smart Proxy Manager.

3. An advanced anti-ban solution like Smart Browser.

4. A browser automation tool to process Javascript and extract dynamic

   elements, e.g. headless browser libraries like Playwright, Puppeteer, or

   Selenium.

5. A software to deploy spiders/scrapers to run for days/weeks, like Scrapy

   Cloud.

6. A maintenance and monitoring tool, like Spidermon.

P.S. The examples given in the steps above, are the tech stack that developers use at Zyte.

The graph flows like this:
1—>2—>3—>4—>5—>6
Scrapy Smart Proxy Manager —> Advanced Anti-ban Solution —> Browser Automation —> Scrapy Cloud —> Spidermon.

This list grows even further if you don’t use the Scrapy framework and use other languages like Python, Java, Node.js, or C#.

When putting these puzzle pieces together, the biggest challenge is integration. Six levels of integration take a lot of time, resources, and management. Especially when it comes to scaling it up.

The good news is that Zyte API is powerful enough to take care of the rotating proxy solution, anti-bans, browser automation and a lot more. So basically, Zyte API drastically simplifies the tech stack for you.

1 —> [2 + 3 + 4] —> 5 —> 6 :: Scrapy Zyte API Scrapy Cloud Spidermon. The entire puzzle is now reduced from 6 steps to 4.

Don’t you believe me? Try Zyte API yourself.

I have curated a list of links that you would need to try out yourself.

1. Sign up and Get the API key.
2. Zyte API’s Web Dashboard.
3. General Usage.
4. Browserhtml.
5. Screenshots.
6. Browser actions.
7. Full end-to-end tutorial.

Migrating from Smart Proxy Manager to Zyte API

This guide draws detailed parallels between a rotating proxy solution - Smart Proxy Manager and all in one solution-Zyte API, which will make sense of multiple topics- residential proxies, cookie handling, session management, smart geolocation, browser HTML, screenshots, browser actions, user throttling, etc. Consider this guide as a curated list of these topics in one place.

Zyte API client software- Integration with Scrapy and Python

As I mentioned earlier, when choosing the products for any large-scale project, it is important to select the products that integrate seamlessly with each other.

One thing I absolutely love about Zyte is that they actually work on those pain points. Since a majority of the developers use either Python or Scrapy framework for web scraping, Zyte came up with the following solutions that make using Zyte API easier.

• python-zyte-api is a Python package that offers both a command-line client and an asyncio API.

• scrapy-zyte-api is a Scrapy plugin.

A Big Thank you 🍀 for being with us on this journey, and here's to many more editions of the Extract Data Newsletter! Share your feedback, suggestions, and topics you'd love to see covered in our upcoming newsletters.