Introducing Zyte API Proxy Mode

Zyte API proxy mode lets software that already sends its requests through an HTTP proxy use Zyte API instead. You point the proxy setting at api.zyte.com:8011 and give your Zyte API key as the username, with an empty password. Zyte's documentation puts it this way: "To use Zyte API as a proxy, use the api.zyte.com:8011 endpoint, with your Zyte API key and proxy headers" (Zyte API proxy mode).

Zyte API is the web scraping API that Zyte sells. We built it as an HTTP API. This was a conscious design decision: an API to support all your web scraping needs would not work well with the limitations of a proxy API.

Through the HTTP API, you send the options in a request body and Zyte API can return browser-rendered HTML and screenshots, run browser actions, capture network traffic, and extract structured e-commerce, article and job posting data (Zyte API browser automation, Zyte API automatic extraction).

However, we have always known that, for people already using a service with a proxy API, like Zyte Smart Proxy Manager, migrating to Zyte API's HTTP API may not be straightforward. We knew we had to bridge the gap between the two different types of APIs so people can still access proxies and other ban-handling tech via Zyte API.

We have made things easier for Scrapy users through scrapy-zyte-api integration, and while we cannot provide all Zyte API features behind a proxy API, we have implemented a proxy mode for Zyte API that provides some of those features.

Smart Proxy Manager is now end-of-life. Zyte's documentation says "SPM is end-of-life" and describes moving the remaining Smart Proxy Manager traffic onto proxy mode through a translation layer, with the automatic migration planned for 9 December 2025 (SPM to Zyte API via proxy mode, customer FAQ). What that means for existing accounts, hostnames and keys is covered in Zyte API is the successor to Smart Proxy Manager.

How do you use Zyte API proxy mode?

Set your HTTP client's proxy to api.zyte.com:8011, with your Zyte API key as the username and an empty password.

To use Zyte API's proxy mode, set http://YOUR_API_KEY:@api.zyte.com:8011 as your proxy, replacing YOUR_API_KEY with your actual Zyte API key (but keeping the : after the key!).

If you do not have a key yet, signing up gives you $5 of free credit for your first billing month (Zyte API pricing).

The way you define your proxy varies across different technologies. You can see different examples in our docs. This is how to implement it in curl:

1curl \
2    --proxy api.zyte.com:8011 \
3    --proxy-user YOUR_ZYTE_API_KEY: \
4    --compressed \
5    https://toscrape.com

And… that's it!

The command is the one in Zyte's documentation (Zyte API proxy mode). In the curl manual, --proxy sets the proxy, and an address with no scheme in front of it is treated as an HTTP proxy. --proxy-user takes the username and password for proxy authentication in the form user:password, so YOUR_ZYTE_API_KEY: is your key followed by an empty password. --compressed asks for a compressed response and decompresses it.

The target is a web scraping sandbox. Its books site describes itself as "A fictional bookstore that desperately wants to be scraped" (toscrape.com).

The same endpoint takes HTTP and HTTPS addresses. In the documentation's words: "The main endpoint works both for HTTP and HTTPS URLs, you do not need an HTTPS proxy interface to access HTTPS URLs." A stack that needs an HTTPS proxy interface can use api.zyte.com:8014 instead, with the Zyte CA certificate installed (Zyte API proxy mode).

If every request fails on certificate verification, install the Zyte CA certificate. The documentation gives the sign: "You can tell that is your case when every attempt to use a given Zyte service results in an error about SSL certificate verification." (Installing the Zyte CA certificate).

How do you send requests through proxy mode from Python?

Give Requests the same proxy URL for both schemes.

In Requests, "If you need to use a proxy, you can configure individual requests with the proxies argument to any request method" (Requests documentation, advanced usage). This script reads your key from an environment variable called ZYTE_API_KEY, sends one request through proxy mode, and prints the status code, the request ID Zyte API returns, and the start of the page:

1import os
2
3import requests
4
5API_KEY = os.environ["ZYTE_API_KEY"]
6PROXY = f"http://{API_KEY}:@api.zyte.com:8011"
7
8response = requests.get(
9    "https://toscrape.com",
10    proxies={"http": PROXY, "https": PROXY},
11    timeout=120,
12)
13print(response.status_code)
14print(response.headers.get("Zyte-Request-ID"))
15print(response.text[:300])

Zyte's documentation adds a note for this library: "You need to install and configure our CA certificate for the requests library." Its CA page says how: "To use requests, build a CA bundle and point the REQUESTS_CA_BUNDLE environment variable to it." (Zyte API proxy mode, Installing the Zyte CA certificate). Requests reads that variable itself (Requests documentation, advanced usage).

To keep one IP address across several requests, with a session header, see backconnect proxies explained. For proxies in Requests in general, see Python Requests proxies.

Your Scrapy project already uses a proxy middleware. What changes?

Point scrapy-zyte-smartproxy at the proxy mode endpoint and give it your Zyte API key.

scrapy-zyte-smartproxy is a Scrapy downloader middleware for "either the proxy mode of Zyte API or Zyte Smart Proxy Manager (formerly Crawlera)" (scrapy-zyte-smartproxy documentation). Zyte's documentation gives the two settings that send it to proxy mode: ZYTE_SMARTPROXY_URL, set to http://api.zyte.com:8011, and ZYTE_SMARTPROXY_APIKEY, set to your Zyte API key (Zyte API proxy mode). In your project's settings.py:

1import os
2
3DOWNLOADER_MIDDLEWARES = {
4    "scrapy_zyte_smartproxy.ZyteSmartProxyMiddleware": 610,
5}
6ZYTE_SMARTPROXY_ENABLED = True
7ZYTE_SMARTPROXY_URL = "http://api.zyte.com:8011"
8ZYTE_SMARTPROXY_APIKEY = os.environ["ZYTE_API_KEY"]

The middleware entry at 610 and ZYTE_SMARTPROXY_ENABLED come from the scrapy-zyte-smartproxy documentation. Use the Zyte API key, not a Scrapy Cloud API key; Zyte's documentation calls that out as important (Zyte API proxy mode).

Proxy mode headers go on a request like any other header: the middleware's documentation says you can set them "as regular Scrapy headers, e.g. using the headers parameter of Request or using the DEFAULT_REQUEST_HEADERS setting" (scrapy-zyte-smartproxy documentation).

A single-file spider with these settings, which you can run without a project, is in Zyte API is the successor to Smart Proxy Manager. Setting a proxy in Scrapy without any Zyte product is covered in how to set up a proxy in Scrapy.

Which headers change what proxy mode does?

Request headers whose names start with Zyte-. Zyte's documentation describes them as headers that change how a request is sent through Zyte API in proxy mode.

Proxy mode also supports a few additional features through special headers, such as setting a custom geolocation or device emulation.

Source: Zyte API proxy mode, request headers, and for the device values the Zyte API reference.

Two of these change what a request costs or needs. Setting an extended geolocation explicitly, instead of letting Zyte API choose, affects the cost, and setting the IP type to residential requires business verification and affects the cost (Zyte API shared features).

Zyte-Browser-Html returns the page as a browser renders it. It does not work together with Zyte-Disable-Follow-Redirect (Zyte API proxy mode). The documented curl command:

1curl \
2    --proxy api.zyte.com:8011 \
3    --proxy-user YOUR_ZYTE_API_KEY: \
4    --compressed \
5    -H "Zyte-Browser-Html: true" \
6    https://toscrape.com

Zyte-Override-Headers exists because Zyte API sends some headers of its own: "Zyte API automatically sends some request headers for ban avoidance." Your headers replace most of them, but not Accept, Accept-Encoding or User-Agent unless you name them in Zyte-Override-Headers, and the documentation warns: "Overriding headers can break Zyte API ban avoidance." (Zyte API proxy mode).

How the country of a request changes what a site shows is covered in scrape from anywhere with geolocation settings, and running sessions in advanced use cases for session management.

What can proxy mode not do?

Screenshots, browser actions, network capture, turning off JavaScript, automatic extraction, server-managed sessions and echo data, which all need the HTTP API.

However, as I hinted earlier, proxy mode does have some limitations when compared to the HTTP API. Mainly, you can not take advantage of browser automation and automatic extraction features. I strongly suggest you consider using the HTTP API to unblock all the potential of Zyte API.

Browser-rendered HTML is the one browser feature proxy mode has, through the Zyte-Browser-Html header shown above.

Source: Zyte API proxy mode, key differences.

On overhead, the documentation says the HTTP API adds some, mainly from base64-encoding the response body, while "with proxy mode the only overhead you get is some additional response headers." On cookies: "With proxy mode, you can only set cookies for the domain of the target URL, you cannot manually set cookies for additional domains that may be reached through redirection." (Zyte API proxy mode).

If your scraper drives a browser through a proxy, the documentation says: "The proxy mode is not optimized for use in combination with browser automation tools." It points to Zyte API's own browser features instead, which web scraping dynamic websites with Zyte API walks through.

A proxy mode request failed. What does the response tell you?

What went wrong, in the body and in three headers.

"Note that the response body of unsuccessful responses is always the actual JSON response from the HTTP API that provides error details." Two response headers summarise it: Zyte-Error-Type, a URI reference that identifies the problem type, and Zyte-Error-Title, a short summary of it. A third, Zyte-Request-ID, is "A unique identifier of the request", and Zyte asks for its value when you report a problem with a request (Zyte API proxy mode).

A 400 can come from your own headers. Proxy mode refuses a request that carries any of 16 listed headers, among them X-Forwarded-For, Forwarded-For, True-Client-IP and Via: in the documentation's words, "any request with one or more of them will result in an HTTP 400 response" (Zyte API proxy mode).

On cost, Zyte API's pricing page says "We charge only for successful responses and provide volume discounts." (Zyte API pricing).

FAQ

Q: What is the Zyte API proxy mode endpoint?
A: api.zyte.com:8011. It takes both HTTP and HTTPS addresses, so you do not need an HTTPS proxy to fetch HTTPS pages. A stack that needs an HTTPS proxy interface can use api.zyte.com:8014, with the Zyte CA certificate installed.

Q: Why does the proxy username end with a colon?
A: Because your Zyte API key is the username and the password is empty. The colon separates the two. In curl it is --proxy-user YOUR_ZYTE_API_KEY:, and in a proxy URL it is http://YOUR_ZYTE_API_KEY:@api.zyte.com:8011.

Q: Can Zyte API proxy mode return browser-rendered HTML?
A: Yes. Send the header Zyte-Browser-Html: true. It cannot be combined with Zyte-Disable-Follow-Redirect. Screenshots and browser actions still need the HTTP API.

Q: How do I set the country of a proxy mode request?
A: Send the Zyte-Geolocation header with a country code, for example AU, which is the code Zyte's own example uses. Without it, Zyte API uses the geolocation it judges most fitting for the target website. Setting an extended geolocation explicitly affects the cost.

Q: Can I override the User-Agent header in proxy mode?
A: Yes, by naming it in Zyte-Override-Headers, for example Zyte-Override-Headers: User-Agent. Your own headers replace most of the headers Zyte API sends, but not Accept, Accept-Encoding or User-Agent unless they are named there. Zyte's documentation warns that overriding headers can break its ban avoidance.

Q: Why does a failed proxy mode request return JSON?
A: Because the body of an unsuccessful response is the HTTP API's own JSON error, with the details. The Zyte-Error-Type and Zyte-Error-Title response headers summarise the same error.

Q: Why did proxy mode answer 400 to my request?
A: Check your headers first. Proxy mode refuses any request that carries one of a listed set of forwarding headers, such as X-Forwarded-For or Via. Remove them and send the request again.

Q: What should I include when I report a failed proxy mode request?
A: The value of the Zyte-Request-ID response header. It identifies the request, and Zyte's documentation asks for it in any report about a request's outcome.