PINGDOM_CHECK

#ExtractSummit2026 The world's largest web scraping conference returns. Austin Oct 7–8 · Dublin Nov 10–11

Register now
Data Services
Login
Try Zyte APIContact Sales
  • Unblocking and Extraction

    Zyte API

    The ultimate API for web scraping. Avoid website bans and access a headless browser or AI Parsing

    Ban Handling

    Headless Browser

    AI Extraction

    SERP

    Enterprise

    DocumentationSupport

    Hosting and Deployment

    Scrapy Cloud

    Run, monitor, and control your Scrapy spiders however you want to.

    Coding Agent Add-Ons

    Agentic Web Data

    Plugins that give coding agents the context to build production Scrapy projects. Starts with Claude Code.

  • Data Services
  • Zyte API

    Zyte Data

    Scrapy Cloud

  • Browse

    • BlogArticles, podcasts, videos
    • Case studiesCustomer outcomes
    • White papersIn-depth reports
    • DocumentationGuides & API reference
    • EventsConferences, webinars, recordings

    Subscribe

    • NewsletterSwiftly delivered
    • Join our community2,000+ web scraping engineers
  • Product and E-commerce

    From e-commerce and online marketplaces

    Data for AI

    Collect and structure web data to feed AI

    Job Posting

    From job boards and recruitment websites

    Real Estate

    From Listings portals and specialist websites

    News and Article

    From online publishers and news websites

    Search

    Search engine results page data (SERP)

    Social Media

    From social media platforms online

  • Meet Zyte

    Our story, people and values

    Contact us

    Get in touch

    Support

    Knowledge base and raise support tickets

    Terms and Policies

    Accept our terms and policies

    Open Source

    Our open source projects and contributions

    Web Data Compliance

    Guidelines and resources for compliant web data collection

    Affiliate Program

    Join Zyte’s affiliate program and start earning commissions today

    Join the team building the future of web data
    We're Hiring
    Trust Center
    Security, compliance & certifications
Login
Try Zyte APIContact Sales
All articles
AI75, 75 articles
Data quality15, 15 articles
Developer interest60, 60 articles
Integration3, 3 articles
Open-source50, 50 articles
Proxies35, 35 articles
Scraping practice35, 35 articles
Scraping strategy48, 48 articles
Search results4, 4 articles
Web data75, 75 articles
Web scraping APIs49, 49 articles
Scrapy47, 47 articles
Scrapy Cloud26, 26 articles
Web Scraping Copilot11, 11 articles
Zyte API71, 71 articles
AI & Machine Learning3, 3 articles
Automotive3, 3 articles
E-commerce & retail35, 35 articles
Entertainment & Streaming2, 2 articles
Financial Services8, 8 articles
Government2, 2 articles
Market Research & Intelligence7, 7 articles
Media & publishing11, 11 articles
Real Estate2, 2 articles
Recruitment & HR3, 3 articles
Transportation & Logistics2, 2 articles
Travel & hospitality3, 3 articles
iPaaS2, 2 articles
Large language model29, 29 articles
MCP3, 3 articles
Python110, 110 articles
Scraping at Scale7, 7 articles
Scraping Fundamentals11, 11 articles
Web Scraping Industry Report20, 20 articles

Appearance

Discord Community
BlogAIAgentic browser automation with Zyte CDP
ArticleTutorial / How-toAIUse case

Agentic browser automation with Zyte CDP

What if your AI agent could use a real browser to search, click, compare products, and collect data, without you writing a selector for every step? By connecting it to a remote Zyte CDP browser, you can turn plain-language instructions into practical browser automation.

John Rooney · Developer Engagement Manager

September 11, 2026

Agentic browser automation with Zyte CDP

Most browser automation examples start with selectors. Find the button, identify the input, write the script, and hope the page does not change before the next run.

That approach works, but it becomes awkward when a site requires real browser interaction. Some information only appears after a click. Some pages run browser checks. And if the agent is running on a remote server, even launching and maintaining a local browser can become a project of its own.

In this walkthrough, I connected an agent to a Zyte CDP browser through Playwright CLI and gave it instructions in plain language. The interesting part was not simply connecting to a remote browser. It was seeing what the agent could do once it had access to a real, interactive browser session.

Video timestamps are included throughout this post.

follow along

The setup: a short instruction file and Playwright CLI [00:18]

I started with a small Markdown file that explains how the agent should use the Zyte CDP browser with Playwright CLI. It tells the agent to make sure Playwright CLI is installed and meets the required version, then gives it the basic connection instructions.

This creates a simple bridge between the agent and the browser. Once the browser is connected, the agent can use the operations available through Playwright CLI: opening pages, finding fields, clicking buttons, entering text, and reading the resulting page.

The difference is that I do not have to write a separate Playwright script for every interaction. I can describe the task and let the agent work out how to perform it on the page.

A deliberately interactive demo site [00:39]

The demo website is intentionally simple. It contains categories, a search box, product controls, and several buttons. It also includes a browser check, which appears briefly when the page loads.

The site has a useful interaction flow:

  • Search for products.
  • Select products.
  • Click a compare button.
  • Read the comparison table that appears.
  • Open an individual product page.
  • Click a button to check availability.

Some of this information might be available through Ajax requests or an underlying API. In a real project, it can be worth investigating those options. But there are also cases where the most reliable path is to use the website as a user would and click through the interface.

That is where a browser-connected agent becomes useful.

Replacing selector work with instructions [01:13]

The traditional Playwright workflow requires you to inspect the page and choose selectors. You need to work out which element is the search field, which button triggers comparison, and which part of the page contains the result.

With the agent, the instruction can be much closer to the actual task:

Go to the website, search for "brake", and return the products you find.

The agent can inspect the page, locate the relevant field, enter the search term, and read the results. It is still using selectors under the hood, but you do not have to discover and maintain those selectors yourself for each request.

This is especially useful when the task changes from run to run. The instruction can describe the outcome rather than encoding every click in a fixed script.

Connecting the agent securely [01:28]

For the demonstration, I used OpenCode with DeepSeek Flash through OpenRouter. This task does not require a large amount of reasoning. The agent mainly needs to follow the browser instructions and carry out a sequence of actions, so a relatively inexpensive model is enough.

The connection also needs authentication. The API key is read from an environment variable and used to create a short-lived configuration file in a temporary location. That file is passed to Playwright CLI when it connects to the browser.

The important practice here is to keep the key out of the instruction file and out of the agent's visible working material. The connection configuration should be created securely and cleaned up according to the needs of the environment.

After that, the agent can use the browser through Playwright CLI.

Searching without writing selectors [03:02]

The first task was simple: open the site and search for "brake".

The agent found the search bar, entered the term, and returned the matching products. During the process, Playwright CLI reported the elements it found and the actions it took. The model was verbose, but the important result was that no custom selector script was needed.

The same approach works when the page is more complicated than the demo. You can describe the field or action in terms a user would understand, and the agent can inspect the current page before deciding what to interact with.

Using the site's compare feature [04:07]

Next, I asked the agent to take the top three products, use the website's compare feature, and return the comparison table.

This matters because the table does not appear until the products have been selected and the compare action has been triggered. It is not simply a block of data sitting in the initial HTML.

The agent had to find the product controls, select the relevant items, click the compare button, and read the table that appeared below the results. Again, the instruction described the goal rather than the selectors:

For the top three products, use the website's compare feature and return the table it shows.

The example is deliberately basic, and the same information might have been available elsewhere on this particular site. The point is to demonstrate the interaction pattern. Once the agent has a browser, it can perform actions that depend on the page's state and then collect the result.

Checking availability on each product page [05:13]

I then asked the agent to visit each product page and retrieve the availability information.

The product pages include a "check availability" button. The agent can follow the product information it already collected, open each page, find that button, click it, and return the availability result.

This is the kind of task that can become a collection of small scripts when written manually. The agent can handle the sequence from a single higher-level instruction, provided the task is clear and the browser session remains available.

Why use a remote browser? [05:48]

If Playwright can launch a browser locally, why connect to a remote one?

The two main reasons are access and maintenance.

Access to sites that expect a real browser

Some sites do not respond well to a basic automation setup. They may perform browser checks or apply other forms of traffic protection. In the demo, the site was configured so that a standard Playwright launch would not get access, while the remote browser could connect successfully.

There are open-source stealth browser options, and some of them work well. But they need to be maintained. Browser behavior changes, sites change their checks, and the tools need to keep up.

Less local infrastructure to maintain

Running a browser locally can also create deployment work. An agent may be running on a server without a display, which means you may need a virtual display such as Xvfb. Then there is the browser installation, version management, operating system configuration, and the rest of the environment around it.

None of these problems is impossible to solve. They are just extra moving parts, and every extra moving part is another thing that can fail in an agentic workflow.

A remote browser moves much of that browser infrastructure outside the agent's environment. The agent only needs to connect, perform the work, collect the result, and disconnect.

Session limits and closing the connection [02:28, 07:34]

The browser session in the demonstration lasted five minutes. That limit is important for two reasons: the agent needs enough time to finish its work, and the connection should be closed as soon as the work is complete.

When I spent too long talking through the demonstration, the session expired. The agent reconnected and continued with the task, but this is a useful reminder to design the workflow around short, purposeful sessions.

In a production-style flow, the agent might receive one instruction, connect to the Zyte CDP browser, complete the interaction, return the data, and close the browser. Closing the connection ends the session and avoids paying for unnecessary additional browser time.

This cleanup step should be part of the instructions or application logic, not something left to chance.

The larger idea

The real benefit here is not that an agent can click a few buttons on a demo site. It is that browser automation can be expressed in terms of the work you want done:

  • Search for a product.
  • Compare the first three results.
  • Visit each product page.
  • Check availability.
  • Return the results.

The agent still needs a browser automation tool, authentication, sensible session limits, and clear instructions. But you do not have to begin by turning every task into a collection of selectors and hard-coded steps.

That makes a remote CDP browser a useful option for agentic web interaction, especially when a site requires clicks, state changes, browser checks, or other behavior that a simple scraping request cannot reproduce.

The instructional Markdown file and the browser CDP resources used in the demonstration are linked below the video. If you are experimenting with browser-connected agents, this is a straightforward pattern to try: give the agent a small set of browser instructions, let it act on the page, and make sure it closes the connection when the job is done.

Zyte CDP browser documentation:
https://docs.zyte.com/zyte-api/usage/cdp.html

Zyte headless browser:
https://www.zyte.com/zyte-api/headless-browser/

Instruction Markdown file:
https://raw.githubusercontent.com/zytelabs/zyte-cdp-examples/refs/heads/main/agent-onboard.md

Try Zyte API

Build your first scraper in minutes

Free trial, no credit card. From a single request to production in an afternoon.

Get started
AIUse case

John Rooney

Developer Engagement Manager

John is the Developer Engagement Manager at Zyte, working closely with the community, creating content and helping developers learn web scraping, Zyte products an much more. He has spoken at Extract Summit's and also creates the workshop's for the events.

  • X (Twitter)
  • LinkedIn
More from this author

In this article

  • The setup: a short instruction file and Playwright CLI [00:18]
  • A deliberately interactive demo site [00:39]
  • Replacing selector work with instructions [01:13]
  • Connecting the agent securely [01:28]
  • Searching without writing selectors [03:02]
  • Using the site's compare feature [04:07]
  • Checking availability on each product page [05:13]
  • Why use a remote browser? [05:48]
  • Access to sites that expect a real browser
  • Less local infrastructure to maintain
  • Session limits and closing the connection [02:28, 07:34]
  • The larger idea

Follow

Get the latest

Zyte and the data web in your inbox — or wherever you already are.

Subscribe

Or follow elsewhere

Continue reading

Scraping Swiss Army Knife: My personal fix for web setup fatigue using Docker, Scrapy and Zyte
Use case

Scraping Swiss Army Knife: My personal fix for web setup fatigue using Docker, Scrapy and Zyte

Tired of repeating web scraping setup? Learn how a multi-arch Docker container with Scrapy, Zyte, Requests, and Pandas speeds up exploration and debugging.

Ayan Pahwa·10 min·February 5, 2026
How I trade gold using e-ink, live data and an old Raspberry Pi
Use case

How I trade gold using e-ink, live data and an old Raspberry Pi

Track real-world gold and silver retail prices automatically using Zyte API, Python, and a Raspberry Pi with an e-ink display. Learn how to scrape rendered HTML, parse prices, and build an always-on trading dashboard.

Ayan Pahwa·10 min·February 2, 2026
How price extraction is fuelling insights for modern retailers
Use case

How price extraction is fuelling insights for modern retailers

Retail pricing has long combined data, experience, and instinct – but today’s market volatility demands a faster, smarter approach.

Theresia Tanzil·7 mins·July 23, 2025
Beyond Hello World: The Operational Gaps in LLM-Powered Scraping Tools
Use case

Beyond Hello World: The Operational Gaps in LLM-Powered Scraping Tools

The difference between writing a scraper and running a scraping operation

Theresia Tanzil·10 Mins·February 7, 2025
Analyze web data quickly with Jupyter Notebooks and Zyte API
How To

Analyze web data quickly with Jupyter Notebooks and Zyte API

With AI Scraping in Zyte API, you can pull data from any e-commerce website straight into your Jupyter notebooks.

Neha Setia Nagpal·2 mins·December 13, 2024
Leveraging Web Scraping and Big Data: The New Frontier in Optimized Delivery Solutions
How To

Leveraging Web Scraping and Big Data: The New Frontier in Optimized Delivery Solutions

Big Data Delivery isn’t just about moving information around—it’s about making it work for you, helping businesses spot trends, predict what’s next, and stay ahead in a cutthroat market.

Karlo Jedud·10 mins·January 2, 2025

The Community · Newsletter

The best of Zyte and the data web, in your inbox.

One curated edition — new articles, product updates, and the stories shaping the data web. No noise.

Services

Zyte Data

Fully managed web data extraction, delivered to your spec.

Explore Zyte Data

Web Scraping API

Zyte API

Scrape any website at scale with automatic proxy rotation and ban handling.

Sign Up

Developers

Zyte Developers

Docs, tools, and a community to help you build and scale scrapers.

Join Us
    • Zyte API
    • Ban Handling
    • AI Extraction
    • SERP
    • Enterprise
    • Scrapy Cloud
    • Agentic Web Data
    • Pricing
    • Product & E-commerce
    • Data for AI
    • Job Posting
    • Real Estate
    • News & Articles
    • Search
    • Social Media
    • Blog
    • Learn
    • Case Studies
    • Webinars
    • White Papers
    • Join our community
    • Join our Affiliate Program
    • Documentation
    • Meet Zyte
    • Contact us
    • Jobs
    • Support
    • Terms and Policies
    • Trust Center
    • Do not sell
    • Cookie settings
    • Web Data Compliance
    • Open Source
    • What is Web Scraping
    • Web Scraping in Python: Ultimate Guide
    • Stop getting blocked, start scraping
  • Logo EWDCILogo Most Loved WorkplaceLogo Job TogetherISO 27001 SealMedal Leader Europe Winter 2025Fastest Implementation Winter 2025Logo Leader Winter 2025Grid Leader Spring 2025Grid Leader Summer 2025Leader Fall 2025Leader Winter 2026
    XFacebookInstagramYouTubeLinkedInDiscord

    © Zyte Group Limited 2026