Once upon a time, companies needing web data faced limited options: build in-house data collection capabilities, subscribe to a handful of traditional, high-end providers, or simply make do without.
Fast forward to today, and we're witnessing a flourishing ecosystem of specialized data suppliers.
That shift represents one of the most significant but understated changes in the business landscape of the past decade.
Worldwide data output is forecast to reach 181 zettabytes in 2025 - more than throughout the whole of 2010 and 2018 - and this is forecast to increase by 300% in the next five years, according to Edge Delta.
As web data becomes more plentiful, it becomes more valuable.
The exponential growth has produced an entirely new ecosystem of data suppliers to help businesses harness, process, and derive value from this information deluge.
The new category of Data as a Service (DaaS) is forecast by Mordor Intelligence to grow at a compound annual growth rate of 20%, reaching nearly $62 billion by 2030.
Several converging forces have catalyzed the rapid expansion of the data vendor landscape:
Digital Transformation: As companies across all sectors undergo digital transformation, they generate and require more data than ever before. Industries from manufacturing to healthcare are increasingly relying on external data to fuel decision-making and operational improvements.
Data Alchemy: The ability to harness data from the web doesn't just transform existing businesses, it is creating whole new companies. Whether it be through aggregation, comparison or intelligence, web data brought together, interpreted and rearticulated is causing entrepreneurs to seek the sources they need.
The AI Revolution: The rise of machine learning and artificial intelligence has created unprecedented demand for large, diverse datasets. As OpenAI's GPT models and similar AI systems demonstrate, the quality and quantity of training data directly impact performance. This has spawned a quest for AI-ready datasets.
Regulations and Roadblocks: Regulations like GDPR and CCPA have fueled a keenness to outsource data collection to those with compliance expertise, while the increasing sophistication of web pages themselves has made direct data collection more complex. Companies are increasingly turning to specialized vendors who can navigate both kinds of challenge.
Responding to those demands, the data vendor landscape has evolved dramatically over the past decade, with several distinct phases of development:
The traditional data provider ecosystem was dominated by established, large players in specific verticals.
Institutional companies in the markets information space have long provided market data, pricing information, and financial analytics. Others aggregated consumer credit information, while market research agencies collected and sold consumer behavior data, and another class of provider offered information about companies.
These vendors typically operated with proprietary rather than public data, rigid delivery models, and high price points that limited access to large enterprises.
Around 2015, a new generation of data vendors emerged, leveraging APIs to make data more accessible and actionable.
Where data owners themselves did not provide API access, a new class of vendors built their entire business models around providing third-party data through developer-friendly APIs in areas like profile enrichment. Meanwhile, "alternative data providers" began aggregating non-traditional data sources for investment insights, available through structured feeds.
This phase marked a significant shift toward more flexible, programmatic access to data. But, without the ability to pick target web data, it still left the majority of sources on the table.
To choose from a full menu of web data, it has always been necessary to scrape from the web. Indeed, many companies have long employed teams to develop scraping pipelines using available frameworks.
Others who do not want to take on the task in-house have frequently enlisted all-purpose developer shops to custom-build scrapers for one-off data collection services.
Outsourced developers can quickly and affordably deliver solutions tailored to immediate business needs. But many customers have discovered inconsistent data quality, compliance risks and short-term support, adding up to fragile supply.
To solve that problem, the industry spawned marketplaces for pre-scraped data.
Marketplaces aim to meet data buyers' need for immediacy and quality, the equivalent of a bazaar or emporium, offering a range of ready-made datasets.
While the ability to browse and buy those datasets can be appealing to some with rapid needs for common datasets, companies with exacting target data needs frequently find marketplace offerings too narrow, with limited choice, customization and varying underlying quality.
That is why we have recently seen the emergence of a novel category of data vendor, the managed extraction service.
Managed extraction vendors aim to provide a combined, integrated offering:
Targeting: Scraping from a buyers' bespoke list of target sites, including niche or long-tail sites, complex on-page content types and crawls for page discovery.
Done-for-you service: Setup and creation of scrapers for data acquisition.
Ongoing supply: Feeds of data from target sites, updated on customers' preferred cadence.
Human help: Skills from expert scrapers and legal experts to create and manage data pipelines, advise on compliance issues and monitor for data quality.
Managed data extraction services tend to be found within multi-purpose scraping vendors - that is, those which offer a combination of software and services.
In fact, their existence can be thought of as largely thanks to the growth in all these distinct categories to the point where an integrated offering, drawing on each under one roof, has been made possible.
The data extraction software market is forecast by Dimension Market Research to grow from $1.5 billion in 2024 to $4.9 billion in 2033, and managed data extraction will be one part of that future.
Indeed, the category is already evolving as new technologies and customer demands present themselves.
Alongside its Zyte API software, Zyte has long offered a managed data extraction service from its in-house experts, under the Zyte Data banner.
Over the last year, AI advances have been accelerating that service.
AI Extraction can automatically parse content types including product pages, news articles, job listings and search results, without needing to predefine complex rules for each site.
LLM capabilities now present in Zyte API now allow users to extract complex, unstructured content from pages by issuing natural-language instructions.
Those enhancements don't just make life easier for hands-on Zyte API customers doing scraping for themselves. They have also allowed the Zyte Data team to eliminate setup costs for managed extraction services in a great number of cases.
And, because scrapers infused by AI Extraction and LLM capabilities can dynamically adapt to changing page mark-up, this is also growing the reliability of data feeds for managed data extraction customers.
In this way, many in the industry expect the managed data extraction sphere to follow a trend seen in software at large - the hybridisation of software-as-a-service and done-for-you service.
As expert, done-for-you extraction services are increasingly able to make gains from AI acceleration, they will likely begin to look more like software services themselves - a front-end of data feeds and controls, underpinned by expert help.
That combination - falling prices plus increased hands-on control, backed by expert oversight - is likely to prompt a reassessment of the classic enterprise buyers' dilemma.
For many, the changing economics of web data acquisition will make the decision to outsource more appealing than previously - and that could raise the overall quality of business data supply across the board.
Today, companies across industries recognize that systematic, reliable access to data from outside their organization is a strategic necessity.
This recognition is driving continued innovation and investment in the data supply chain.
What began as a luxury for large enterprises has evolved into a strategic business function supported by a thriving ecosystem of specialized vendors.
Industry analysts observe that we're seeing the industrialization of data acquisition and processing - following the same pattern we've seen with other technologies, from in-house development to specialized service providers to essential business infrastructure.
The growth in this space is in many ways, the story of how web data itself has transformed from a nice-to-have resource into one of the most valuable commodities in the modern economy.