How to rotate proxies in Python

Rotating a proxy means sending consecutive requests through different addresses instead of one. You do it because a site counts requests per address, and one address is one thing to count and one thing to block.

A proxy is an intermediary server that hides your IP, so you can navigate through web traffic anonymously and securely. Proxies have very interesting use-cases, the most prominent of them being web scraping for pricing intelligence, SEO monitoring, data collection for market research, etc. And the correct use of rotating proxies is a key ingredient of this.

If you want to know more about proxies for web scraping and how proxies work, feel free to skim through our companion post on what a proxy server is.

For the internet, your IP address is your identity. One can only make limited requests to a website with one IP. Think of websites as some sort of regulator. Websites get suspicious of requests coming from the same IP over and over again. This is 'IP Rate Limitation'. IP rate limitations applied by websites can cause blocking, throttling, or CAPTCHAs. One way to overcome this is to rotate proxies.

The code is short. In Requests you pass a different value to the proxies argument each time. In Scrapy you set a different proxy meta value per request, usually from a downloader middleware. What takes the work is everything that happens after an address stops working. Most of this page covers that.

By the end you will have:

• A single proxy configured in Requests, and the reason it sometimes gets ignored.
• A rotating pool in Requests that detects failure, retires an address and retries elsewhere.
• The same in Scrapy, both with an existing package and with your own middleware.
• A way to tell a rotation problem from a problem rotation will never solve.

So let's get started!

What do you need installed before you start?

Requests. python -m pip install requests

Scrapy, for the second half. python -m pip install scrapy

A rotation middleware for Scrapy, if you would rather not write one. python -m pip install scrapy-rotating-proxies

A proxy list. The examples use the addresses from the Requests documentation, such as 10.10.1.10:3128, and reserved example names such as proxy1.example.com:8000. Substitute your own. If your product/service relies on web scraped data, a free proxy solution will probably not be enough for your needs. Nothing on this page needs a paid proxy to read, and a free public proxy list is not a substitute for one when you run this for real, because you have no idea who operates the node your traffic is passing through.

Which of the methods below should you use?

Now let's get to the "how" part.

The step that changes the outcome is from row three to row four. A pool with no memory keeps handing work to an address that has already been blocked.

Let's discuss them one by one:

How do you set a single proxy in Requests?

Pass a dictionary mapping scheme to proxy URL.

1import requests
2
3proxies = {
4    "http": "http://10.10.1.10:3128",
5    "https": "http://10.10.1.10:1080",
6}
7
8response = requests.get("http://example.org", proxies=proxies, timeout=(3.05, 27))

The documentation states that you "can configure individual requests with the proxies argument to any request method", that you can set it once for a whole Session, and that Basic authentication uses the http://user:password@host/ syntax in any of those entries (Requests advanced usage).

You can configure proxies for individual URLs even if the schema is the same. This comes in handy when you want to use different proxies for different websites you wish to scrape.

You can also give a proxy for one scheme and host, rather than for a whole scheme, by using the scheme://hostname form as the key. The documentation's example is proxies = {'http://10.20.1.128': 'http://10.10.1.10:5323'}, and it notes that "proxy URLs must include the scheme" (Requests advanced usage).

The timeout above is not decoration. Requests states that "by default, requests do not time out unless a timeout value is set explicitly. Without a timeout, your code may hang for minutes or more", and the tuple sets the connect and read timeouts separately (Requests advanced usage). A rotator without a timeout stalls on the first dead proxy in the list and never reaches the rest of it.

For SOCKS, install the extra with python -m pip install 'requests[socks]' and use a socks5:// or socks5h:// URL. The scheme decides where DNS resolution happens: "using the scheme socks5 causes the DNS resolution to happen on the client, rather than on the proxy server", and socks5h resolves on the proxy (Requests advanced usage).

You set a proxy on the session and Requests ignored it. What happened?

An environment variable overrode it.

requests.get essentially uses the requests.Session under the hood.

This is documented:

"Setting session.proxies may behave differently than expected. Values provided will be overwritten by environmental proxies (those returned by urllib.request.getproxies). To ensure the use of proxies in the presence of environmental proxies, explicitly specify the proxies argument on all individual requests as initially explained above."

That is from the Requests documentation, which also lists the variables it reads when no per-request proxy is given: http_proxy, https_proxy, no_proxy and all_proxy, plus their uppercase forms (Requests advanced usage).

This has two consequences for a rotator. Setting session.proxies once and rotating by mutating it is unreliable, so pass proxies per request instead. And a machine with HTTPS_PROXY set in its shell profile is sending everything through that address whether your code mentions a proxy or not. That explains a script that rotates correctly on one machine and appears to use one address on another.

How do you rotate proxies with Requests?

Pass a different proxy per call, and handle the failure.

1import random
2
3import requests
4
5PROXIES = [
6    "http://10.10.1.10:3128",
7    "http://10.10.1.11:3128",
8    "http://10.10.1.12:3128",
9]
10
11
12def fetch(url, attempts=3):
13    candidates = random.sample(PROXIES, min(attempts, len(PROXIES)))
14    for proxy in candidates:
15        try:
16            return requests.get(
17                url,
18                proxies={"http": proxy, "https": proxy},
19                timeout=(3.05, 27),
20            )
21        except requests.exceptions.RequestException:
22            continue
23    raise RuntimeError("no proxy returned a response for %s" % url)

random.sample rather than random.choice inside a loop, so a retry never goes back through the address that just failed.

This is still not a pool. It has no memory: the address that failed on this URL will be picked again on the next one, and again after that.

Why is picking one at random not enough?

Because a pool has to remember what happened.

Four behaviours separate a pool from a list, and random selection has none of them.

Failure detection is the hard one, because a block, a rate limit, a rendering failure and a real 404 look similar from outside and need different handling. A connection error is unambiguous. A 403 or a 429 can mean the address. A 200 carrying an empty results page means the site answered and gave you nothing, and no status-code rule will catch it.

Here is the same fetch with all four behaviours.

In the code shown below, first, we create a proxy pool dictionary. Then, randomly pick a proxy to use for our request. If the proxy works properly we can access the given site. If there's a connection error we may have to delete this proxy from the list and retry the same URL with another proxy.

1import random
2import time
3
4import requests
5
6COOLDOWN = 300  # seconds a failed address sits out
7
8
9class ProxyPool:
10    def __init__(self, proxies):
11        self.available_at = {p: 0.0 for p in proxies}
12
13    def _free(self):
14        now = time.monotonic()
15        return [p for p, when in self.available_at.items() if when <= now]
16
17    def _retire(self, proxy):
18        self.available_at[proxy] = time.monotonic() + COOLDOWN
19
20    def fetch(self, url, attempts=3):
21        tried = set()
22        for _ in range(attempts):
23            candidates = [p for p in self._free() if p not in tried]
24            if not candidates:
25                raise RuntimeError("no address available for %s" % url)
26            proxy = random.choice(candidates)
27            tried.add(proxy)
28            try:
29                response = requests.get(
30                    url,
31                    proxies={"http": proxy, "https": proxy},
32                    timeout=(3.05, 27),
33                )
34            except requests.exceptions.RequestException:
35                self._retire(proxy)
36                continue
37            if response.status_code in (403, 429) or response.status_code >= 500:
38                self._retire(proxy)
39                continue
40            return response
41        raise RuntimeError("no proxy returned a usable response for %s" % url)

The 300-second cooldown is not arbitrary. It is the default backoff in the scrapy-rotating-proxies package, whose ROTATING_PROXY_BACKOFF_BASE is "base backoff time, in seconds. Default is 300 (i.e. 5 min)" and whose cap is "3600 (i.e. 60 min)" (scrapy-rotating-proxies documentation).

Sometimes the proxy that you are trying to use is just simply banned. In this case, there's not much you can do about it other than remove it from the pool and retry using another proxy. But other times if it isn't banned you just have to wait a little bit before using the same proxy again.

Add your own success test to the fetch method for the empty-page case. A length check on the response body, or a check that the selector you care about matched, catches what a status code cannot.

How do you rotate proxies in Scrapy?

Install scrapy-rotating-proxies and add two middlewares.

Scrapy comes with a middleware that makes rotating proxies a breeze, once you have a list of working proxies. Its last release, 0.6.2, dates from May 2019 (scrapy-rotating-proxies on PyPI), so test it against your Scrapy version before you depend on it.

1# settings.py
2ROTATING_PROXY_LIST = [
3    "proxy1.example.com:8000",
4    "proxy2.example.com:8031",
5]
6
7DOWNLOADER_MIDDLEWARES = {
8    "rotating_proxies.middlewares.RotatingProxyMiddleware": 610,
9    "rotating_proxies.middlewares.BanDetectionMiddleware": 620,
10}

That's it! Now all your requests will automatically be routed randomly between the proxies.

If you want more external control over the IPs, you can even load it from a file. Use ROTATING_PROXY_LIST_PATH instead once the list is long enough that you want to change it without a deployment. The package's documentation notes that "ROTATING_PROXY_LIST_PATH takes precedence over ROTATING_PROXY_LIST if both options are present", so setting both and editing the wrong one is a real way to lose an afternoon (scrapy-rotating-proxies documentation).

The four behaviours above map onto the package's settings, and its defaults are the useful part of its documentation:

All four are from the scrapy-rotating-proxies documentation, which also states the default rule for deciding an address is dead: a proxy is treated as banned if "response status code is not 200, response body is empty or if there was an exception".

Configure that rule rather than accepting it. Two failure modes it gets wrong. A site that answers a blocked request with a 200 and a populated but empty results page passes the default test, so the address is never retired. And a legitimate 301, 304 or 404 fails the default test, so a healthy address is retired for returning a correct response. The package lets you supply your own ban detection, and a rule written against the site you are collecting from fits it better than a general one.

Read the pool statistics it logs every 30 seconds. A steady fall in the alive count over a run is the pool burning, and it shows there before it shows in your output.

How do you write your own rotation middleware instead?

Set request.meta["proxy"] and give the middleware an order below 750.

1# middlewares.py
2import random
3
4
5class RandomProxyMiddleware:
6    def __init__(self, proxies):
7        self.proxies = proxies
8
9    @classmethod
10    def from_crawler(cls, crawler):
11        return cls(crawler.settings.getlist("MY_PROXY_LIST"))
12
13    def process_request(self, request, spider):
14        request.meta["proxy"] = random.choice(self.proxies)

1# settings.py
2MY_PROXY_LIST = [
3    "http://proxy1.example.com:8000",
4    "http://proxy2.example.com:8031",
5]
6
7DOWNLOADER_MIDDLEWARES = {
8    "myproject.middlewares.RandomProxyMiddleware": 543,
9}

The number is the ordering, and 750 is the number to stay below. Scrapy's built-in HttpProxyMiddleware "sets the HTTP proxy to use for requests, by setting the proxy meta value for Request objects" (Scrapy downloader middleware documentation), and it sits at 750 in DOWNLOADER_MIDDLEWARES_BASE, where "low orders are closer to the engine, high orders are closer to the downloader" (Scrapy settings reference). Your middleware has to write the meta key before the built-in one reads it, so any order below 750 works. The documentation's own example for a custom downloader middleware uses 543.

Three Scrapy defaults that change what this code does, all from the Scrapy downloader middleware documentation:

• HTTPPROXY_ENABLED defaults to True, and the middleware obeys the http_proxy, https_proxy and no_proxy environment variables. Same trap as in Requests.
• HTTPPROXY_AUTH_ENCODING defaults to "latin-1", which matters if your proxy password is not ASCII.
• The proxy meta value accepts http://username:password@some_proxy_server:port, and SOCKS URLs where the download handler supports them.

For the retry half, Scrapy already has RetryMiddleware, "a middleware to retry failed requests that are potentially caused by temporary problems such as a connection timeout or HTTP 500 error" (Scrapy downloader middleware documentation). It sits at order 550, with RETRY_TIMES defaulting to 2 (Scrapy settings reference). A retried request is scheduled again and passes through your process_request again, so it gets a freshly chosen address without extra code. With random.choice that can be the same address, which is what the pool logic above exists to prevent.

You want to use a managed proxy with Scrapy. What is the integration?

scrapy-zyte-smartproxy is the downloader middleware for it. Its own documentation describes it as "a Scrapy downloader middleware to use one of Zyte's proxy services: either the proxy mode of Zyte API or Zyte Smart Proxy Manager (formerly Crawlera)" (scrapy-zyte-smartproxy repository).

Install it, enable the middleware, and set your API key from the environment. Never write a key into a spider file, and never into one you publish.

1# settings.py
2import os
3
4DOWNLOADER_MIDDLEWARES = {
5    "scrapy_zyte_smartproxy.ZyteSmartProxyMiddleware": 610,
6}
7
8ZYTE_SMARTPROXY_ENABLED = True
9ZYTE_SMARTPROXY_APIKEY = os.environ["ZYTE_SMARTPROXY_APIKEY"]

Read the repository for the current settings and headers before you copy this. The package supports two different services and the configuration differs between them.

Every address fails in the same way. Is that a proxy problem?

No. If every address fails immediately, at the same point, with the same response, the site is not measuring your address.

Rotation defeats counting per address. It does nothing against a site reading your request headers, your TLS handshake characteristics, your header order or your browser signature, because none of those change when the address does. Adding proxies to that situation makes the run slower and no more successful.

That is about a third of sites, and growing. Across the HTTP Archive corpus, 33% of desktop sites and 32% of mobile sites were running a bot protection service in 2024, up from 29% and 26% in 2022 (Web Almanac 2024, Security chapter). Those services read considerably more than the address.

The diagnosis table below separates the two cases.

The site is answering 429. Should you rotate or slow down?

Slow down first.

A 429 is the site telling you its rate limit. The status "indicates that the user has sent too many requests in a given amount of time ('rate limiting')", and the response "MAY include a Retry-After header indicating how long to wait before making a new request" (RFC 6585, April 2012).

Rotating in response to a 429 answers the symptom and spends your pool doing it. Reading Retry-After and waiting the stated time answers the cause, costs no addresses, and is what a collector that intends to keep working does. In Scrapy that means DOWNLOAD_DELAY and AUTOTHROTTLE_ENABLED; in plain Python it means a sleep between requests to the same host.

Rotate when the limit is per address and you need more throughput than one address is allowed. Slow down when the limit is per site and no number of addresses will change it.

How many proxies do you need?

There is no fixed number.

It follows from three things: how many requests per minute you send to one host, how tolerant that host is, and how well you retire failing addresses. The third is the one under your control. A pool without retirement and cooldown keeps spending requests on addresses that are already blocked, however large it is, and has no way to know.

Measure it instead. Log the success rate per address per run. If the alive count in your pool is falling across a run, the pool is too small for the rate, or your detection is retiring healthy addresses. Both are visible in the same log line.

When does running the pool yourself stop making sense?

At three points.

When the list needs replacing faster than you can maintain it. When the sites you want are reading signals other than the address, so a bigger pool changes nothing. And when you are making this decision per site rather than once.

The above-discussed ways to rotate proxies work well for building demos and minimum viable products. But things can get tricky as soon as you decide to scale your data extraction project. Infrastructure management of proxy pools is quite challenging, time-consuming, and resource extensive. You will soon find yourself refurbishing proxies to keep the pool healthy, managing bans and sessions, rotating user agents, etc. Proxy infrastructure also needs to be configured to work with headless browsers to crawl javascript-heavy websites. Phew! It's not shocking how quickly your data extraction project gets converted into a proxy management project.

Zyte API handles address selection, browser rendering and unblocking behind one endpoint, so failure detection, retirement, retry and cooldown stop being code you own. Zyte sells that. The code above is free, and for a small pool against forgiving targets it is the right answer.

FAQ

Q: How do I rotate proxies in Python?
A: Pass a different proxy to each request. In Requests that is a different value for the proxies argument per call. In Scrapy it is a different proxy meta value per request, set from a downloader middleware. Both are a few lines. The work is in what happens when an address stops working.

Q: Why is picking a proxy at random not enough?
A: Because it has no memory. A failing address stays in the pool and keeps being chosen, so the success rate falls steadily with no visible cause. A pool needs failure detection, retirement of failing addresses, retry through a different one, and a rule for when a retired address comes back.

Q: Why is Requests ignoring the proxy I set on the session?
A: Because an environment variable is overriding it. The documentation states that values set on session.proxies are overwritten by environmental proxies, and recommends passing the proxies argument on individual requests instead. Check HTTP_PROXY, HTTPS_PROXY and ALL_PROXY in your shell.

Q: How do I rotate proxies in Scrapy?
A: Either install a rotation middleware and give it a list, or write your own middleware that sets request.meta["proxy"]. Give your own middleware an order below 750, which is where Scrapy's built-in proxy middleware sits, so the meta key is written before the built-in one reads it.

Q: What order should my proxy middleware have?
A: Anything below 750. Lower orders run closer to the engine, and the built-in HttpProxyMiddleware is at 750 in Scrapy's default middleware list, so yours has to be lower. The documentation's own example for a custom middleware uses 543.

Q: How many proxies do I need?
A: It depends on your request rate to one host, that host's tolerance, and how well you retire failing addresses. There is no fixed number. Measure the success rate per address before buying more, because a pool that does not retire failing addresses wastes requests at any size.

Q: Why do all my proxies get blocked at once?
A: If every address fails immediately and in the same way, the block is not on the addresses. The site is reading your headers, your TLS characteristics or your browser signature, and none of those change when the address does.

Q: Should I rotate or slow down when I get a 429?
A: Slow down first. A 429 is a rate limit, and the response may carry a Retry-After header saying how long to wait. Rotating answers the symptom and spends your pool doing it. Rotate when the limit is per address, slow down when it is per site.

Q: Should I use free proxies?
A: No, for anything real. Beyond the unreliability, your traffic passes through a node operated by somebody you know nothing about, along with your request headers and anything else in the request.

Q: Do I need residential proxies to rotate?
A: No. Rotation is logic and works with any pool. Residential addresses are a separate decision, about how your traffic looks to the site.

Q: What is the difference between a proxy list and a proxy API?
A: A list gives you addresses and leaves selection, failure detection, retries and rendering to you. An API takes a request and returns a response, choosing the address itself, so the four behaviours above become somebody else's code.