Global Retailer Enlists Zyte for Data-Driven, AI-Powered Pricing Intelligence

Gathering product data in hard-to-reach places

In 2022, we embarked on an ongoing managed data extraction project for a large global retailer (that, sadly, we cannot share – we take privacy very seriously). This project has since evolved into a collaborative and innovative partnership, exploring the full potential of web data today.

The Challenge

The client required weekly pricing intelligence data from hundreds of websites to help set unique product prices across stores in 30+ countries. To achieve this accurately, they needed more than just standard pricing data. Unusually, they required hard-to-access product attributes buried inside unstructured text (such as description text), making standard extraction practices inadequate.

These requirements and the extensive number of global target sites necessitated the creation of a unique framework, hundreds of scripts, and custom technology leveraging Zyte’s API infrastructure.

Project Requirements

• Gathering product data from hundreds of different retail websites

• Extracting data buried in unstructured text for each product

• Transforming data to the client’s preferred format

• Ensuring data is ethically and legally obtained

• Deliver Weekly Data Feeds

Zyte’s data delivery team managed the entire process, simplifying it for the client. They could simply provide the list of target websites and SKUs and wait for the data to flow.

For each product page, we used our own powerful AI to extract all product details matching our standard product schema. We then used a custom-built LLM to extract non-standard items from the page.

Key Achievements

• Gathering Product Data: The machine learning model powering Zyte API’s AI Scraping accelerated spider implementation and successfully extracted most of the client’s data needs. This milestone was achieved three times faster using our AI-powered automatic product data extraction technology, leaving only a handful of cases for manual creation by our delivery team.

• Extracting Unstructured Text: Zyte’s data delivery team developed and used an LLM-based solution to extract relevant data from descriptions and unstructured text. The unstructured description text was first extracted using our AI Scraping and then processed through the LLM’s special prompts created by the team.

• Data Transformation: After extracting the unstructured data via the LLM, it was transformed into the client’s standard format using a Python library. The final transformed data was then incorporated into the final data schema for delivery.

• Ethical and Legal Compliance: Zyte’s world-class legal team is involved in every Zyte Data project. They provide legal guidance before any work begins and evaluate the client’s data requirements to ensure it is ethically and legally obtained.

• Weekly Data Feeds: Zyte’s data delivery team created a seamless integration point to the client’s cloud storage provider, allowing for automatic delivery of weekly data feeds.

The Result

Vast amounts of complex and hard-to-collect data were successfully and consistently delivered to our client in record time using some of the most advanced technology available today. They needed the best data available to price their products globally. Zyte delivered on time and at scale. We continue to explore new ways to enhance their data capabilities, leveraging the latest technology confidently and reliably at scale.

If you are looking for help with your web scraping project, we are here to help. Feel free to reach out, and we'll have a quick discovery call. No pressure.

Summary

With the help of Zyte Data, the client successfully acquired the product data needed for their price intelligence efforts. Zyte API’s AI scraping was important to reduce development time and easily gather the structured data for the client. LLMs strategically complimented Zyte API’s AI Scraping to gather the hard-to-get data accurately and quickly.