Anyone involved in web data probably knows how much AI has changed the work in 2026.
Agents are writing scrapers, investigating failures and helping engineers get data feeds working faster, while AI companies are buying that data and asking tougher questions about its quality and origins.
That was the common thread reported by speakers at Extract Summit Austin on October 8, 2026, when web data engineers, researchers and business leaders gathered to share what is working, what still breaks and how they are tackling it.
Next up, they said, comes making those gains hold up in production.
1. Faster development demands better checks
Shane Evans, CEO and co-founder of Zyte, reported that data extraction and quality assurance effort to produce a first data feed fell from about 42 hours in May to just 20 hours in September.
That’s a huge boost for businesses whose roadmap, market expansion or conversion optimisation depend on speed.
But data at that new speed also demands checking.
Pravin Khandke, staff software engineer at Insight Global, described the financial version of the same risk. A transposed amount can pass a schema check because it is still a valid number.
The answer is independent verification: calculations, reconciliation, representative tests and human review where ambiguity remains.
Zyte puts that verification into practice with source-page comparisons in its internal QA tool, checks that catch contradictions such as a discounted price exceeding the regular price, and Spidermon alerts when field coverage or record counts fall.
2. Buyers want proof of what a dataset contains
Zyte chief operating officer Suzanne Hassett kicked off the day by saying the key moment is all about data provenance.
Andrew Harris, data acquisition consultant at AB Harris Methods, contrasted two buying questions: “How many records can you get me?” and “What did you miss, and how would you know?”
He said AI buyers need coverage, freshness, provenance and reproducibility. Missing sources can bias a corpus, while old values can look current in a freshly delivered file and a parser change can alter a benchmark without anyone realizing the data changed.
“Coverage is a denominator, not a success rate,” Harris’ said. In other words, a fetch dashboard cannot reveal sources the system never attempted.
For teams commissioning managed web data extraction, here’s the implication: specify the evidence alongside the fields and delivery schedule. Which sources were excluded? When was each value last confirmed? Can it be traced to a source snapshot?
Zyte’s Evans summarized why these questions matter. “Code fails loudly, web data fails quietly,” he said.
3. Self-repairing systems need a way to stop
Konstantin Lopukhin, head of R&D at Zyte, presented an experimental spider that pauses at a layout mismatch, asks a coding agent for a repair, checks it and resumes the crawl. The goal and checking gate remain outside the agent’s control.
That separation matters. “Give an agent a check, and it will try to pass it,” Lopukhin’s presentation warned. But a weak check can reward code that fills a field without doing the intended job.
Joao Drummond, founder and CEO of Crawly, showed agents coordinating requirements, implementation and review, with a person approving the merge. One of Crawly’s rules: “Split the doer from the judge.”
On the same theme, Evans’ keynote demonstrated a related monitoring-and-repair workflow being tested at Zyte.
Across the presentations, useful autonomy meant checked repairs, bounded attempts and escalation - not endless retries. When Lopukhin’s test store hid prices behind sign-in, the prototype stopped rather than pretending it could recover them.
That is the maintenance opportunity behind the autonomous data pipeline: less waiting for someone to investigate, without letting the repair system redefine success.
4. Access is an ongoing optimization problem
Joseph Dye, co-founder and software engineer at Byteful, showed how TCP/IP behavior can reveal a mismatch between a client’s claimed operating system and the traffic it sends. Access engineering reaches below browser headers into the networking stack.
Logan Harless, CTO at String, examined how to choose configurations whose success rates are uncertain and changing. “Every fetch trades cost, latency and quality,” his presentation stated. Quality means receiving the real, usable page - not merely an HTTP response.
Always using the most expensive setup wastes money. Choosing a cheap configuration once and never revisiting it leaves teams exposed when conditions change. Harless explored learning from request outcomes, using simulations to illustrate the trade-offs.
Zyte’s Shane Evans supplied the wider context on this: AI helps engineers handle access problems, but it does not stop new ones arriving. As he put it: “The scraper is nearly free. The web isn’t.”
Automated access handling in Zyte API addresses that continuing work. Our State of Web Access explains why different sources demand very different approaches.
5. Evaluate the whole operation, including its suppliers
Cheaper code changes only part of the scraping bill. In three company examples shared by Evans, people accounted for 63% to 73% of costs, while building new spiders represented 3% to 28%.
These were examples, not an industry benchmark, but they showed why faster development does not produce equivalent overall savings.
Downtime, incorrect data and delayed delivery matter, too. So does the infrastructure carrying the requests.
Jason Grad, CEO and co-founder of Massive, put the sourcing question plainly: “Every residential IP is someone’s home device. Its owner should know it’s being used.”
Grad described clear opt-in, immediate ways to pause participation, clean uninstallation, partner oversight and customer vetting. Those are procurement questions alongside price and performance - not details to assume a provider has handled.
They also belong beside the wider compliance and provenance obligations of collecting and using web data.
The useful measure of progress is whether the business gets dependable data sooner, with less effort and fewer surprises. Faster development gives teams room to improve the checks, maintenance and infrastructure that make that possible.
