A question comes up often enough in the web scraping community that it has a familiar shape to it: I like working in scrapy-playwright, but I want to run camoufox, or another Python browser library, instead of the browser it ships with, so how do I actually wire that in? For a long time the honest answer was some version of "you can, but it isn't pretty." People raised it in GitHub issues and talked around it on forums, and the workarounds that existed all asked you to figure out a way to run that browser separately and use CDP.
That has changed with a small addition to scrapy-playwright: PLAYWRIGHT_BROWSER_PROVIDER, that hands the browser lifecycle to whatever Playwright-compatible package you want to use. It’s a small Python class that integrates into Scrapy and scrapy-playwright. It is the kind of change that reads like a one-line entry in a changelog and quietly removes a frustration a lot of people had learned to live with.
Why the stock browsers get caught
To see why this matters, it helps to remember what people are reaching for a different browser to do. Playwright is excellent at driving a browser, and scrapy-playwright wires that capability into Scrapy at the download handler so a rendered page arrives through the normal request and response cycle, asynchronously and quickly. The catch is the binary. The Chromium and Firefox builds that Playwright ships are the standard, well-known ones, which means even fairly basic anti-bot systems recognize them almost immediately, and the sites you most want to reach are rarely running basic anti-bot systems. These browsers were designed for automated testing, and no scraping specifically.
Getting around that recognition used to be a matter of patching the browser at runtime, but that has stopped being enough. The more effective projects now change the browser at the source and compile their modifications into the binary itself, instead of patching a stock build once it is running. Runtime patches tend to leave small inconsistencies behind, and modern fingerprinting is built to notice exactly those, in the same way that the network fingerprint has become a central battleground between scrapers and site operators. Each mismatch lowers the trust a site assigns to your session and makes a challenge more likely, so the goal is to present a browser with nothing out of place. Building the changes at compile time is harder to detect, and because Chromium and Firefox expose different things, what needs changing is different in each. This is one important part of how modern stealth browsers work, not the whole story, but it is enough to explain why swapping the browser is worth the effort, especially when anti-bot vendors now ship changes at a pace that punishes any setup relying on manual tuning.
The middle ground that was missing
scrapy-playwright already had one good answer for using a different browser: connect to one that is running somewhere else. Through PLAYWRIGHT_CDP_URL or Playwright's own connect endpoint, you point the plugin at an already-running browser over the network, and that route is genuinely useful when the browser lives on a paid service or on infrastructure you run yourself. I have written before about standing up a self-hosted browser service and connecting Playwright to it over a WebSocket, and it works well, but it is worth being clear about what it asks of you. You have to run the browser in debug mode, keep that instance alive, watch it for memory leaks, and restart it when it dies, which is a reasonable amount of operational surface for what started as "I just want to use camoufox locally."
That was the gap. Between the stock browser that gets caught and a full separate browser service that you connect to remotely, there was a large group of people who simply wanted a stealthier browser running inline with their spider, on their own machine, without building a service around it. The workarounds they were left with were the ugly ones. One approach was to swap the binary by hand: download the original, remove it, and drop the replacement in where Playwright expected to find it, which is as fragile as it sounds and breaks the moment anything updates. Another leaned on the fact that some of these browsers shipped an experimental built-in server you could expose and connect to, which was fine until you needed one of the browsers that had no such server, at which point you were back to no clean route at all. That last point bites hardest for the Firefox-based options, because the connect-to-a-running-browser story was never as clean for them as it was for Chromium.
How browser providers actually work
The new setting works by naming a provider, which is just a class that owns starting and stopping the browser. The default provider wraps vanilla Playwright and gives you everything the README already documented, and if you want a different backend you point PLAYWRIGHT_BROWSER_PROVIDER at a class of your own. The provider implements a small asynchronous lifecycle, start, launch_browser, launch_persistent_context, and close, and because packages like camoufox, patchright, and invisible_playwright keep the standard Browser, BrowserContext, and Page objects, everything downstream of the launch, your pages, contexts, and routing code, keeps working unchanged. Only how the browser starts and stops is different.
Here is the camoufox provider from the documentation, which is the shape almost any provider will take:
1from contextlib import AsyncExitStack
2
3from scrapy_playwright.handler import Config, PERSISTENT_CONTEXT_PATH_KEY
4
5class CamoufoxBrowserProvider:
6 def __init__(self, config: Config) -> None:
7 self.config = config
8 self.stack = AsyncExitStack()
9
10 async def start(self) -> None:
11 pass
12
13 async def launch_browser(self):
14 from camoufox.async_api import AsyncCamoufox
15
16 return await self.stack.enter_async_context(
17 AsyncCamoufox(**self.config.launch_options)
18 )
19
20 async def launch_persistent_context(self, context_kwargs: dict):
21 from camoufox.async_api import AsyncCamoufox
22
23 return await self.stack.enter_async_context(
24 AsyncCamoufox(
25 persistent_context=True,
26 user_data_dir=context_kwargs[PERSISTENT_CONTEXT_PATH_KEY],
27 **self.config.launch_options,
28 )
29 )
30
31 async def close(self) -> None:
32 await self.stack.aclose()Point the settings at it, and because camoufox is Firefox-based, set the browser type to match:
1# settings.py
2PLAYWRIGHT_BROWSER_PROVIDER = "myproject.providers.CamoufoxBrowserProvider"
3PLAYWRIGHT_BROWSER_TYPE = "firefox"I ran this exact example from the documentation, and it works. That is roughly all there is to it: one class, two settings, and the stealth browser is running inline with your spider. Importing the third-party library lazily inside the methods that need it, as the example does, is a nice detail, because it means the setting can point at a provider whose backend is only installed in some of your environments without breaking the others.
Why this fits the way Scrapy is built
None of this is an accident of design. Scrapy is lean on purpose. It is a full framework, but the core deliberately avoids telling you which browser to use, which extraction approach to take, or how your infrastructure should look, and it pushes those decisions out to extensions instead, some official and some from the community. Browser providers extend that same principle right down to the choice of browser binary, which is a very Scrapy way to solve the problem: rather than blessing one stealth browser, the plugin adds a seam and lets you bring your own.
It is worth saying where scrapy-playwright sits in that picture, because it shapes why this addition feels significant rather than incidental. The plugin lives in the scrapy-plugins organization and has been driven largely by a colleague at Zyte, which puts it in an interesting spot, not a headline product with a marketing page, but not a random third-party bolt-on either. Seeing a genuinely useful capability land in a project like that, one that many people had quietly filed under "mature, does what it does," is a good reminder that the tools we lean on are still being improved by people who hit the same friction we do.
What it changes, and what it doesn't
I would put this on the "relatively big deal" side of the ledger rather than the nice-to-have side, precisely because of the group it serves. If you were already connecting to a remote browser service, your setup does not change, and if you only ever needed the stock browser, this will pass you by. But for the sizable middle, the people who wanted a stealthier browser running locally with scrapy-playwright and were stuck with fragile workarounds, this turns a hack into a configuration choice, and lowering the cost of that experiment is the real value here.
A couple of honest caveats belong alongside that. The provider examples in the documentation are offered as guidelines, not supported parts of scrapy-playwright, and the plugin is not affiliated with camoufox, patchright, or any of the other projects it references, so you should read each project's own documentation and evaluate any dependency yourself before you rely on it. Swapping in a stealth browser is also not a finish line. It removes one common way of being spotted, but it sits inside a larger system of proxies, behavior, and session management, and treating it as a single switch that solves blocking will lead you astray. When you would rather not run and maintain any of this yourself, Zyte API's headless browser handles the binary, the fingerprinting, and the proxy rotation behind a single request, which is the right answer for a different set of problems than the one browser providers solve. What providers give you is a clean way to make the choice yourself, and for a lot of people that is exactly what was missing.
