From products to SERPs: AI scraping now does it all

Zyte API’s out-of-the-box AI scraping just took a big leap.

Now, developers can use Zyte to unblock and extract product data with zero setup, making it quicker and easier than ever to collect even more data from the public web.

That same plug-and-play experience is now available for three more core use cases: articles, job listings, and SERPs.

On top of that, the API has been upgraded with a composite AI architecture, combining probabilistic LLMs with deterministic ML models—so you can balance control, cost, and flexibility in your pipelines without managing multiple tools.

Let’s explore what’s new and what it unlocks for your team.

New data types now supported by AI-powered spider templates

AI Scraping was launched first for product data. Why? Because product data represents the vast majority of scraped content across the web. E-commerce use cases were the most immediate need, and solving them first laid the groundwork for more structured data types.

We’re expanding that functionality to support three more high-demand data types. Zyte API's AI-powered spider templates now come with built-in support for:

• Products: Extract structured product data from any e-commerce site, including product lists and category navigation → Documentation

• Articles: Scrape full-text articles, lists, pagination, and metadata from news and content websites → Documentation

• Jobs: Extract job postings from detail pages, company listings, and category navigation → Documentation

• SERPs: Extract search engine result pages for brand monitoring, SEO insights, or competitor tracking, including Google's SERP update in January 2025. → Documentation

Each template is pre-built to support AI-powered crawling and parsing, including pagination and navigation. But they’re not one-size-fits-all. These spiders allow full customization through schema extension and override logic. That’s where composite AI comes in.

Smarter customization with composite AI

At the core of Zyte API’s upgrade is the idea that not all scraping problems are the same and don’t deserve the same solution. In fact, we believe using a one-size-fits-all approach is harmful to your business, squandering valuable resources.

That’s why Zyte’s spiders now run on a composite AI engine, blending three approaches:

1. Custom code: built on Scrapy, ideal for crawling logic and structure-specific rules

2. Supervised ML: fast and accurate for structured fields in known schemas (like product titles, prices, and article headlines)

3. Generative AI (LLMs):  great for extraction that requires reasoning, such as parsing unstructured text, or extracting custom tags.

Developers can mix these approaches inside any spider template to maximise efficiency.

For example, use ML for common product fields and add either selectors or custom LLM prompts to extract anything outside the standard schema. This layered configuration gives you flexibility in parsing technique without sacrificing the ability to scale or control your code.

Why relying only on LLMs isn’t cost-efficient

It’s tempting to plug an LLM into your scraping pipeline and let it handle everything, but that approach rarely scales well, especially if you’re feeding the whole HTML code into the context window.

Most web data extraction tasks are deterministic, meaning they involve structured HTML, recurring patterns, and consistent field locations. That’s precisely where supervised ML and code-based spiders shine: they’re more accurate for this type of work and are so cheap in comparison that you can use them at run-time.

Running every request through a generative model adds unnecessary compute costs when most fields don’t need it. This is the key difference in Zyte’s approach. Composite AI lets you run LLMs only where needed, so you don’t pay premium prices to extract a product name that a more cost-efficient ML could handle.

Know what you’re paying for

With Zyte API, each request is priced based on what actually happens under the hood:

1. Unblocking: Zyte automatically uses the leanest strategy to access the site—proxies, headless browsers, or session emulation.

2. Structured extraction: Our supervised ML models power the parsing of standard schemas.

3. Custom fields: LLMs are used only for the fields you extend or request using generative prompts.

This separation means your costs stay predictable. You’re not locking yourself into a full LLM system for every task. You’re scaling smarter by aligning the right AI with the right part of the page.

A faster way to launch new scrapers—with AI that adapts

With these new templates and the composite AI core, you can launch spiders in minutes, even on domains you’ve never scraped. No more writing Xpaths or guessing anti-bot strategies. The API handles unblocking and adapts to layout changes using AI trained for web data extraction.

And when you need to go beyond default schemas, the UI lets you extend with LLMs instantly—without leaving your browser or managing new infrastructure.

AI scraping at scale, without complexity

With full support for products, articles, jobs, and SERPs—plus customizable, AI-powered spiders you can deploy in minutes—Zyte API is now the fastest way to get structured web data into your workflows.

This is how you combine performance and flexibility without the overhead of custom tooling.

Start building smarter; get started with a free trial.