Pass a dictionary to the proxies argument with a key for each scheme.
Configuring and using proxies is not easy, especially when sending HTTP requests. Third-party tools like Requests come in handy when it comes to proxies.
In the web scraping world, there are many obstacles we need to overcome and choosing the right proxy tool that fits your needs is just one piece of the puzzle. One huge challenge is when your scraper gets blocked. To solve it, you need to know some strategies to avoid bans. In this article, I'm going to show you how to use proxies to avoid your scraper getting banned.
In this part, we're going to cover how to configure proxies in Requests. To get started we need a working proxy and a URL we want to send the request to.
1import requests
2
3proxies = {
4 "http": "http://10.10.1.10:3128",
5 "https": "http://10.10.1.10:1080",
6}
7
8response = requests.get("https://httpbin.org/ip", proxies=proxies)
9print(response.text)httpbin.org/ip returns the address the request appeared to come from. If the proxy is working you see the proxy's address rather than your own.
The proxies dictionary must follow this scheme. It is not enough to define only the proxy address and port. You also need to specify the protocol. You can use the same proxy for multiple protocols. The Requests documentation states the scheme requirement directly, and it states as well that the dictionary key selects which of your requests the proxy applies to.
Where should you set the proxy: per request, per session, or in the environment?
Four places, and they interact in a way that catches people.
| Where | What it looks like | Use it when |
|---|---|---|
| Per request | requests.get(url, proxies=proxies) |
One-off, or the proxy changes per call. Also the only form that always wins |
| On a session | session.proxies.update(proxies) |
Several requests through the same proxy, keeping cookies and connection reuse |
| Environment variables | HTTP_PROXY, HTTPS_PROXY, NO_PROXY, ALL_PROXY |
You want no proxy code in the script at all |
| Per scheme and host | {"http://10.20.1.128": "http://10.10.1.10:5323"} |
One proxy for one host, direct for everything else |
In the above example, you can define proxies for each individual request. If you don't need this kind of customization you can just set these environment variables:
1export HTTP_PROXY="http://10.10.1.10:3128"
2export HTTPS_PROXY="http://10.10.1.10:1080"This way you don't need to define any proxies in your code. Just make the request and it will work.
Sometimes you need to create a session and use a proxy at the same time. In this case, you first have to create a new session object and add proxies, then finally send the request through the session object:
1import requests
2
3session = requests.Session()
4session.proxies.update(proxies)
5
6response = session.get("https://httpbin.org/ip")The interaction that catches people. Requests honours the environment variables automatically, and they take precedence over a proxy set on a session. The documentation says so plainly: "Setting session.proxies may behave differently than expected. Values provided will be overwritten by environmental proxies", and it gives the remedy, which is to "explicitly specify the proxies argument on all individual requests" (Requests, proxies). So if a script is routing somewhere you did not configure, check the environment before you read the code. Lowercase http_proxy and https_proxy work as well as the uppercase forms.
How do you use a proxy that needs a username and password?
If you need authentication use this syntax for your proxy:
1http://user:password@host:portPut the credentials in the proxy URL in whichever of the four places above you are using:
1import os
2
3proxy = f"http://{os.environ['PROXY_USER']}:{os.environ['PROXY_PASS']}@10.10.1.10:3128"
4proxies = {"http": proxy, "https": proxy}Do not write the credentials as a literal, because that puts them in your version control. The example reads them from environment variables to keep them out of the file, and the Requests documentation warns that even that is risky: "Storing sensitive username and password information in an environment variable or a version-controlled file is a security risk and is highly discouraged" (Requests, proxies). Read them from a secret store where you have one, keep the URL construction in a single place, and treat the proxy URL as a credential everywhere it is logged.
If the proxy rejects you, the server returns 407 Proxy Authentication Required, which is a different failure from the target refusing you (RFC 9110, section 15.5.8).
Can you use a SOCKS proxy with Requests?
Yes, with an extra dependency. SOCKS support has been available since Requests 2.10.0 and needs the extra installed:
1python -m pip install "requests[socks]"1proxies = {
2 "http": "socks5://user:pass@host:port",
3 "https": "socks5://user:pass@host:port",
4}One detail decides where DNS resolution happens. The socks5 scheme resolves the hostname on your machine. The socks5h scheme resolves it on the proxy server. If you are collecting content that varies by location, use socks5h: a name resolved on your machine can point to a different server than the one the proxy's region would reach.
Why is my proxy not working?
Work down this table.
| What you see | What it means | What to do |
|---|---|---|
Your own address from httpbin.org/ip |
The proxy is not being used | Check the scheme key. An https:// URL uses the https key, and a dict with only http sends it direct |
| A proxy you did not configure | An environment variable is set | Check HTTP_PROXY, HTTPS_PROXY and ALL_PROXY and their lowercase forms. They override a session's proxies |
ProxyError or a connection error |
The proxy is unreachable or refusing you | Check the address, the port, and whether it needs credentials |
407 |
The proxy needs authentication | Add credentials to the proxy URL |
An SSLError mentioning certificate verification |
Something is intercepting TLS, such as a corporate proxy | Add the proxy's root certificate to your trust store, or point REQUESTS_CA_BUNDLE at a bundle containing it |
403 from the target |
The proxy works and the target refused you | Not a proxy configuration problem. See the last section |
429 from the target |
You are sending too many requests in a given period | Slow down. Read Retry-After if the server sent one. Rotating faster is still faster |
| It hangs | No timeout is set | Set timeout= on every proxied request |
On the certificate row. Do not reach for verify=False. The documentation is explicit about what that does: Requests "will accept any TLS certificate presented by the server, and will ignore hostname mismatches and/or expired certificates, which will make your application vulnerable to man-in-the-middle (MitM) attacks" (Requests, SSL verification).
On timeouts. Requests does not time out unless you tell it to. From the documentation: "Nearly all production code should use this parameter in nearly all requests. Failure to do so can cause your program to hang indefinitely" (Requests, timeouts). A proxy adds a second place the connection can stall, so pass a tuple and set the connect and read timeouts separately:
1requests.get(url, proxies=proxies, timeout=(3.05, 27))Does Requests retry a failed request?
No. "By default, Requests does not retry failed connections" (Requests, transport adapters). You get retries by mounting an adapter with a urllib3 retry policy on a session:
1from requests import Session
2from requests.adapters import HTTPAdapter
3from urllib3.util import Retry
4
5session = Session()
6retries = Retry(total=3, backoff_factor=0.5, status_forcelist=[502, 503, 504])
7session.mount("https://", HTTPAdapter(max_retries=retries))That covers transient network and server failures. It does not cover a proxy that has been blocked, because a block can arrive as a valid 403 rather than as a connection error, and 403 is not in the list above for a reason: retrying the same request through the same address will get the same answer.
How do you rotate proxies in Requests?
Hold a pool and choose one per request.
As discussed earlier, a common problem that we encounter while extracting data from the web is that our scraper gets blocked. It's frustrating because if we can't even reach the website we won't be able to scrape it either. The solution for this is to use some kind of proxy or rather multiple rotating proxies. This lets you get around the IP ban.
To be able to rotate IPs, we first need to have a pool of IP addresses. Be aware, that if your product/service relies on scraped data a free proxy solution will probably not be enough for your needs.
So let's say we have a list of proxies. Something like this:
1pool = ["http://10.10.1.10:3128", "http://10.10.1.11:3128"]Then, we can randomly pick a proxy request. If it works properly we can access the given site. If there's a connection error we might want to delete this proxy from the list and retry the same URL with another proxy.
1import random
2import requests
3
4pool = ["http://10.10.1.10:3128", "http://10.10.1.11:3128"]
5
6def fetch(url):
7 proxy = random.choice(pool)
8 return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=10)The sample above picks an address and does nothing else with the result. Nothing ever leaves the pool, so it keeps using addresses that have stopped working.
There are multiple ways you can handle connection errors. Because sometimes the proxy that you are trying to use is just simply banned. In this case, there's not much you can do about it other than removing it from the pool and retrying using another proxy. But other times if it isn't banned you just have to wait a little bit before using the same proxy again.
A one-hour block and a permanent block look identical at the moment of failure, so the choice between retiring an address and waiting for it has to be made without knowing which of the two you have.
Implementing your own smart proxy solution free of error is a very hard task. Next time you have proxy rotation issues don't go for the first proxy tool you find.
How to rotate proxies in Python covers the full version, and what a rotating proxy is covers the buy-or-build decision underneath it.
The proxy works and the site still refuses me. Now what?
Then the address is not what is being read. A site deciding whether to serve you looks at several signals at once, and the address is one: your header set and their order, your TLS characteristics, your request rate against that host, your browser fingerprint if you are rendering, and whether a session cookie is arriving from several addresses. Buying more addresses changes none of those.
Two checks separate the cases. If a fresh address fails on its first request, it is not an address problem. If addresses work for a while and then fail, it is, and you need more of them or faster turnover. How anti-bot systems work covers what else is being read, and how to get past IP bans covers the applied version.
Managing a pool properly means ban detection, retirement, retry policy and session handling working together rather than separately. As a closing note, I want to show you how to solve issues with python requests proxies using Zyte API. It makes that selection per target site as part of handling access, so it stops being a decision you make per site, and unblocking is the detail. Zyte sells that. Everything above this paragraph applies if you build it yourself, and which proxy provider to use covers buying addresses on their own.
FAQ
Q: How do I use a proxy with Python Requests?
A: Pass a dictionary to the proxies argument with http and https keys, both holding a full proxy URL including the scheme. Test it against an endpoint that echoes the requesting address and you should see the proxy's address rather than your own.
Q: Why is my proxy not working in Requests?
A: Two causes to check first: a dictionary with only an http key used against an https URL, in which case the request goes direct while looking configured, and an environment variable overriding a proxy set on a session. Check the scheme keys, then the environment, then whether the proxy is reachable at all.
Q: Does Requests use the HTTP_PROXY environment variable?
A: Yes, automatically, and it also reads HTTPS_PROXY, NO_PROXY and ALL_PROXY in upper or lower case. The documentation warns that these override proxies set on a session, and that passing proxies on each individual request is the way to be sure yours are used.
Q: How do I use an authenticated proxy?
A: Put the credentials in the proxy URL as http://user:password@host:port. Keep them out of the source, because a literal puts them in version control, and use a secret store where you have one: the Requests documentation discourages keeping them in environment variables too. A 407 response means the proxy rejected the credentials.
Q: Can I use a SOCKS proxy with Requests?
A: Yes, since version 2.10.0, after installing the extra with pip install "requests[socks]". Use the socks5 scheme to resolve hostnames on your own machine and socks5h to resolve them on the proxy, which matters when you are collecting content that varies by location.
Q: Should I set verify=False to fix a proxy certificate error?
A: No. That error can mean a proxy is intercepting TLS, and turning verification off makes the client accept any certificate, including expired ones and ones for the wrong hostname. Add the proxy's root certificate to your trust store, or point REQUESTS_CA_BUNDLE at a bundle that contains it.
Q: Does Requests retry automatically?
A: No. Requests does not retry failed connections by default. Mount an HTTPAdapter with a urllib3 Retry policy on a session to get retries with backoff, and note that a 403 from a blocked address is a valid response rather than a failure, so retrying it changes nothing.
Q: Why am I still getting blocked when the proxy works?
A: Because the address is one signal among several. Headers, header order, TLS characteristics, request rate and browser fingerprint are all read as well. If a fresh address fails on its first request, more addresses will not help.
