How to use browser DevTools for web scraping

The scrapers that take the least time to write are the ones where most of the work happened in the browser before any code, working out where the data lives, how the page is put together and whether JavaScript builds any of it. Every modern browser ships with developer tools that answer those questions in a few minutes, and once you know five or six of their features you stop guessing at selectors and wondering why your script returns nothing.

This guide walks through those features in Chrome, with notes for Firefox where it differs, using books.toscrape.com and quotes.toscrape.com, two practice sites Zyte runs for people learning to scrape.

Opening DevTools

Press F12, or Ctrl+Shift+I, which is Cmd+Option+I on a Mac, or right-click anything on the page and choose Inspect. The last option is the one to get used to, because it opens DevTools with the element you clicked already selected.

For scraping, you will spend nearly all your time in three panels, Elements for the page structure, Console for testing selectors and Network for seeing what the page downloads.

1. Inspect the element you want

Right-click a book title on books.toscrape.com and choose Inspect. The Elements panel opens with the title's <a> tag highlighted:

1<h3><a href="catalogue/a-light-in-the-attic_1000/index.html"
2       title="A Light in the Attic">A Light in the ...</a></h3>

Now look around the highlighted line rather than only at it. As you move your mouse up the tree each element lights up on the page, and you will find that the title sits inside an <article class="product_pod"> that also holds the price, the rating and the stock status. That <article> is the container, the element that repeats once per book, and finding it is the first thing to do on any new page, because a good scraper selects each container and then reads the fields inside it.

While you are here, notice what is and is not visible text. The link text is cut short to "A Light in the ...", but the full title is in the title attribute, and the star rating has no text at all, since the site stores it as a class name, star-rating Three. You would never spot either detail by looking at the page, only by looking at the HTML.

2. Search the page with a selector

With the Elements panel focused, press Ctrl+F, or Cmd+F on a Mac. The search box accepts plain text, CSS selectors and XPath expressions, and it tells you how many elements match:

1article.product_pod          → 1 of 20
2//p[contains(@class, "star-rating Five")]   → 1 of 4

Press Enter to step through the matches, and DevTools highlights each one on the page. It is the quickest way to check that a selector matches the things you want and nothing else.

3. Test selectors in the Console

The Console runs selectors and shows the results as a list you can expand:

1$$("article.product_pod").length
2// 20
3
4$$("p.price_color").map(p => p.textContent)
5// ['£51.77', '£53.74', '£50.10', ...]
6
7$x("//h3/a/@title").map(a => a.value)
8// ['A Light in the Attic', 'Tipping the Velvet', ...]

$$() runs a CSS selector and $x() runs XPath. Both are console shortcuts that Chrome and Firefox provide, not standard JavaScript, so they will not work in your scraper's code, although the selectors inside them will.

4. Check what the server sent, not what the browser built

This is where most beginners lose time. The Elements panel shows the live DOM, the page after JavaScript has run, while a script using requests or fetch only gets the HTML the server sent, and the two are often different.

There are two quick ways to see the original HTML:

• View Page Source. Right-click and choose View Page Source, or press Ctrl+U, to see the raw HTML, then press Ctrl+F and search for a piece of text you want to scrape. If it is not there, JavaScript added it.
• Disable JavaScript and reload. In Chrome, open the Command Menu with Ctrl+Shift+P, type "Disable JavaScript", press Enter and reload the page. In Firefox the setting is javascript.enabled in about:config. What remains is roughly what your scraper will see.

Try both on quotes.toscrape.com/js. With JavaScript on you see ten quotes, in the page source you see no quote elements at all, only a <script> containing var data = [...] with the quotes inside it as JavaScript, and with JavaScript disabled the page is empty.

That tells you a selector for div.quote will fail in Python even though it works in the Elements panel, and that the data is there in a different form.

Related: How to find a website's hidden API for web scraping

5. Watch the Network tab

The Network tab lists every request the page makes, and three settings make it much easier to use for scraping:

• Fetch/XHR filter. It hides everything except the requests JavaScript makes for data, which is where hidden APIs show up.
• Preserve log. DevTools keeps the request list when the page navigates, so you can follow what happens across a redirect.
• Disable cache. You see every request, including ones the browser would otherwise serve from its cache.

Open quotes.toscrape.com/scroll, filter to Fetch/XHR and scroll down. Requests to api/quotes?page=2, page=3 and so on appear, each returning JSON with the next ten quotes, and when you click one, the Headers, Payload, Preview and Response tabs show what the page asked for and what came back.

Two other features in this panel save a lot of hunting:

• Search across all responses. Press Ctrl+F with the Network panel open and search for a value you can see on the page, and DevTools shows which request delivered it.
• Copy as cURL. Right-click a request and choose Copy → Copy as cURL to get the exact request with its headers and cookies, ready to paste into a terminal or into a converter for Python.

6. Look at cookies and storage

The Application tab in Chrome, called Storage in Firefox, shows the cookies, local storage and session storage for the site. You need it when a page only shows the data after you have accepted a consent banner or chosen a region, because the state that controls it usually lives in a cookie, and your scraper will need to send the same cookie or go through the same steps to get one.

Where DevTools can mislead you

• "Copy selector" gives you a fragile selector. Right-clicking an element and choosing Copy → Copy selector produces a long positional path such as #default > div > div > div > div > section > div:nth-child(2) > ol > li:nth-child(1) > article > h3 > a, which matches one element today and breaks the moment the layout shifts. Use it to find your way, then write a short selector based on classes or attributes.
• The Elements panel is not the page source. Browsers repair broken HTML and add elements, for example inserting a <tbody> into every table. The product information table on any books.toscrape.com detail page shows this, since DevTools displays table > tbody > tr but the source has no <tbody> at all, so that selector works in the console and returns nothing in Python, while table tr works in both.
• Your browser and your script look different to the site. The page in your browser loaded with your cookies, your browser's fingerprint and your IP address, and a script requesting the same URL can get a different response, a consent wall, another region's prices or a block page. Always check what your script received, for example by saving response.text to a file and opening it.

When that last difference is the problem, and a page that loads fine in your browser returns a 403 or a challenge page to your script, you are dealing with anti-bot protection rather than a bug in your scraper. Zyte API handles that part and returns the page as a normal browser would see it, so the selectors you worked out in DevTools still apply.

A five-minute checklist for any new page

1. Inspect the value you want and find its repeating container.
2. Search the Elements panel with a short CSS selector and check the match count.
3. Open View Page Source and search for the value, to see whether it is in the original HTML.
4. If it is not, check the Network tab with the Fetch/XHR filter and look for a JSON response containing it.
5. Note any cookies or steps the page needs before the data appears.

By the end of that list you know which tool the job needs, requests and a parser, a direct API call or a browser, and you already have selectors you have tested.

Questions developers ask about DevTools for scraping

Why does a selector work in DevTools but return nothing in my scraper?

The Elements panel shows the page after JavaScript has run and after the browser has repaired the HTML, for example by adding <tbody> to tables, while your script only gets the HTML the server sent, so check View Page Source to see what your scraper will actually receive.

How can I tell whether a page uses JavaScript to load its data?

Open View Page Source with Ctrl+U and search for a piece of text you want to scrape, and if it is not there, JavaScript added it, which you can confirm by disabling JavaScript and reloading the page.

Should I use "Copy selector" from DevTools in my scraper?

No, it produces a long positional path that matches one element today and breaks when the layout shifts, so use it to find where you are and then write a short selector based on classes or attributes.

Can I use $$() and $x() in my scraper code?

No, they are console shortcuts that Chrome and Firefox provide rather than standard JavaScript, but the CSS and XPath selectors you test inside them will work in your scraper.