Proxy-Shopping Primer: Datacenter Proxies and Beyond

• First, what is a proxy?

• This proxy, that proxy?

• What are datacenter proxies?

• What other types of proxies are there?

• When to choose which type of proxies? Sweet spots for each type

• The best proxy solution for your web scraping project?

• A cheat sheet for your proxy needs

• Surely it’s not that difficult?

• Zippy’s Adventure in the Four Frontiers

Introduction 

Did you know that before reaching its current level of sophistication, Zyte API started as a simple datacenter proxy management service?

But wait, what exactly is a “datacenter proxy“? Put simply: a datacenter proxy is a proxy management system built on top of a pool of datacenter IPs.

What are its advantages and its limitations compared to the other types of proxies? What other proxy types are there? We’ll address these questions from the ground up as well as look at more advanced questions you might have about datacenter proxies such as their costs, speeds, IP variety, use cases, reliability, and other trade-offs when compared to the other types of proxy.

Distilled down to its essence, the two steps you need to do to implement the proxy layer for your web data collection projects are: 1) buy the IP(s), and 2) buy/build the proxy management system.

Now if you’ve done any level of research into your options of the best proxy solution for your project, you would have realized that:

• Proxy management is an inevitable part of any substantial web scraping project. Proxy management layer can also get very complicated and resource-intensive, depending on your constraints at this point in time

• Proxy-shopping is a confusing topic. Every proxy provider and proxy management service proclaims that they have the best proxy IPs or the most sophisticated management mechanisms lathered with marketing-heavy explanations as to why. You would also have seen all these different terms and aspects like residential, datacenter, ISP, private, public, shared, dedicated, static, backconnect, SOCKS, VPNs, geolocated, compliant, secure while trying to piece together a coherent big map of what they are and how they relate to each other.

Let’s drill these two points down to first principles so we can better discern the marketing-speak from the technical gist.

At the end of this article, you’ll also get a cheat sheet and a light-hearted story about Zippy the (anthropomorphised) IP to help you commit these concepts and considerations to mind.

First, what is a proxy?

Well, before we discuss what a proxy is, we might as well refresh our minds of what an IP address is and how it works. If you don’t need a refresher on these fundamental concepts, then feel free to skip to the next section.

• What is an IP address? An IP address is a numerical address assigned to every device that connects to an Internet Protocol network like the internet, giving each device a unique identity and allowing for the correct routing and delivery of data packets between devices. Most IP addresses resemble a sequence like A.B.C.D, where each letter represents a number ranging between 0 and 255. An IP address operates at the Network layer (Layer 3) in the OSI model.

• What is a proxy? A proxy is in essence a server that allows you to route your request through it and use its IP address in the process. When using a proxy, the website you are making the request to no longer sees your IP address but the IP address of the proxy. A proxy server may also encrypt your data, so it is unreadable in transit and block/allow access to certain web pages based on IP address.

• Why do we need proxies? In the context of web data collection projects, using a proxy (or a pool of proxies) reduces the chances that your crawler will get banned or blocked. Using proxies also enables you to see or simulate the specific content that the website displays for that given location or device (e.g. mobile proxies).

This proxy, that proxy?

So. If IP addresses are the same, and proxy servers essentially carry a similar function, then why do we see so many different terminologies by the time we went shopping for proxies? Now we get to the crux of it. Here are a couple of common ways we classify proxies:

• Type of proxies based on who owns, sources, manages, and uses the IPs that you’re routing through. This is where the “datacenter proxy” category sits. Other types in this category are residential proxies, mobile proxies, and ISP proxies. We’ll discuss them further down in this article. If you’re interested in making sense of more of the proxy word-salad, then carry on with this list, else proceed directly to the next section to learn more about datacenter proxies.

• Types of proxies based on whether the proxies are static or rotated. Static proxies provide a consistent IP address for an extended period. Every request made through a static proxy uses the same IP address. Rotated proxies change their IP address periodically, either after each request or after a specified time interval. This rotation can be managed through various mechanisms. Some use simpler mechanisms such as round robin, and some employ sophisticated algorithms to select IP addresses based on specific criteria such as geographic location, load balancing, and user behavior patterns. Note that backconnect proxies is another term to refer to rotated proxies. Yes, it sounds more impressive and sophisticated but they are referring to the same approach.

• Types of proxies based on the usage terms. Public (free, open, not secure), shared (more secure, more cost efficient, but subject to the side effects of using a resource that is also being used by multiple users), or dedicated (only accessible to you, as per terms and agreement with the provider or manager). Note that often the term “private proxies” can be interpreted interchangeably with “dedicated proxies“.

  • But note that all IPs managed by proxy management solutions are by nature “public IPs“, as opposed to “private IPs“. It might seem pedantic to make these distinctions but we need to get precise with our terminologies to be able to sniff out the real experts.

• Type of proxy based on the encryption support. A VPN is a type of proxy server that routes all your web traffic through a (typically) encrypted server. Other network proxies may not provide this anonymizing feature and may only operate on certain kinds of requests.

• Type of proxy based on the level in the OSI layer. For example SOCKS proxies operate at the transport layer, which is lower in the OSI model compared to the application layer where HTTP proxies operate. They are versatile and can handle various types of traffic, including web, email, and FTP. Since SOCKS proxies function at a lower level, they do not natively support encryption for your data.

By now, we hope you get at least 80% of your proxy lingo clarified.

What are datacenter proxies?

Before we delve into the other types of proxies, let’s focus on datacenter proxies. We have stated in the beginning that a datacenter proxy is a proxy management system built on top of a pool of datacenter IPs. What else do we know about datacenter IPs?

• Datacenter IPs are provided by third-party datacenters. Datacenters are entities that host and manage servers and network infrastructure. This type of IPs are the most common type of proxy IP because they are the cheapest to buy.

• Datacenter IP addresses are typically obtained by companies that operate datacenters and assigned to servers within the datacenter. These companies lease or own large blocks of IP addresses.

• The IP management is centralized within the datacenter using tools like DHCP and IPAM to allocate IP addresses to virtual and physical servers. This makes them faster and more reliable because they are managed in controlled environments with high-speed connections and redundancy measures.

• Datacenter proxies can be easily identified by websites as coming from datacenters due to the public registration information of the IP addresses and the known ranges of IPs assigned to datacenters.

Unlike residential proxies, which are IP addresses provided by ISPs to individual customers such as homeowners or businesses, datacenter proxies are not affiliated with ISPs

What other types of proxies are there?

Datacenter proxies are the most widely used type of proxy. But there are really three other common types to choose from: Residential, Mobile, and ISP.

Residential proxies

This is a proxy service running on top of IP addresses assigned to real residential physical devices such as a home router or modem. They are generally slower and less reliable due to the nature of being routed through residential devices. They may be more prone to changes in IP availability.

These IPs appear as if they belong to everyday users rather than datacenters IPs, which makes them valuable for specific applications like web scraping, accessing geo-restricted content, and ad verification.

As residential IPs are harder to obtain, they are also much more expensive. In a lot of situations, they are overkill as you could easily achieve the same results with cheaper datacenter IPs.

They also raise legal/consent issues due to the fact you are using a person’s personal network to scrape the web. However, if you use an ethical web scraping like Zyte, the individuals will have volunteered for their device IP to be used in exchange for some sort of benefit. These IPs can also be part of a peer-to-peer network where users install software on their devices to share their IP addresses with the proxy network.

Residential proxies can provide IP addresses from specific geographic locations, making them useful for testing geo-targeted content or services.

Mobile proxies

This is a proxy service running on top of IP addresses assigned to mobile devices by mobile network operators (MNOs). Similar to residential proxies, mobile proxies offer high legitimacy and low detection risk.

Acquiring these IPs is difficult so they are very expensive. For most projects mobile IPs are overkill unless you want to only view web responses shown to mobile users.

ISP proxies

This type of proxy combines the characteristics of both residential and datacenter proxies. These proxies are also known as “static residential proxies''. They are provided by Internet Service Providers (ISPs) and assigned to datacenters rather than residential users. They combine the reliability and stability of datacenter proxies with the appearance of residential IPs. They are less likely to be flagged as suspicious because they come from ISPs.

When to choose which type? Sweet spots for each type of proxy

When you’re building a ban-handling strategy, you have several options. You can either do it on the spider level, adopt a site-by-site approach, or tackle it through an abstracted ban-handling system. Either can work, but it is a messy process where you will need to be prepared to invest time to do trial and error.

Smarter Strategies often need multiple proxy types

When you’re choosing how to tackle bans, you may want to think about the speed of adding new sites and the scale your system needs to achieve when deciding why proxy types to use. 

You're going to want access to multiple types of proxy to handle multiple different scenarios and uses - eg: you may need a small pool of residential proxies used generate session cookies, and then Data center proxies to share those cookies and make requests cheaper.

With multiple types of proxy in play you need a way to integrate and orchestrate them… which takes time and effort, consider looking for options that can handle some of this workload for you.

Consider: The less time you have, the faster you need to move; and the more sites you gather from, the more an automated ‘unblocker style’ approach may make sense, letting an API pick and choose which proxy type to use when and where.

Best suited for datacenter proxies:

• High-volume scraping: Ideal for scenarios requiring fast data retrieval where you need to send a large number of requests in a short time due to their high speed and bandwidth availability.

• Cost-sensitive projects: They are more cost-effective compared to residential, mobile, and ISP proxies.

• Scalability: Highly scalable due to the large pools of IP addresses that can be rapidly deployed and rotated. This is crucial for large-scale scraping operations where multiple IP addresses are needed to avoid IP bans and throttling by target websites.

When to use other types of proxies:

• Residential proxies: Use when scraping sites that have strict IP checking and blocking mechanisms.

• Mobile proxies: Ideal for accessing mobile-specific content or when websites have robust anti-bot measures targeting non-residential IPs.

• ISP proxies: Choose ISP proxies when you can afford a balance of speed and trustworthiness. They offer the reliability of residential IPs with better performance. They are also effective for use cases where datacenter IPs are too easily detected, residential IPs might be too slow, and mobile IPs are too dynamic.

In general, if your scraping needs are large-scale, require scraping from multiple websites, and involve complex ban avoidance strategies, building an abstracted proxy and ban-handling system is the most sustainable option. It allows you to adjust to changes across many websites without having to make spider-specific adjustments and gives you more flexibility to handle different types of proxies as needed (e.g., datacenter, residential, or mobile).

For smaller or highly-targeted operations, where you're scraping only a few websites or need deep customisation for a small number of sites, handling bans site-by-site or within each spider might be a simpler and more direct solution.

A cheat sheet for your proxy needs

These tables summarize most of the characteristics and aspects of each of the types of proxy we have discussed so far. Hopefully it provides a clear, at-a-glance comparison of the different proxy types.

Why do I need proxy management services?

So, the big question now is: why do you need a proxy management service?

The short answer is: With a robust proxy management system:, 

1) you don’t need to buy or source the IPs yourself because it’s part of what these services offer and 

2) you don’t need to allocate resources in engineering these mechanisms.

But but but. If you are OK with going through the administrative hoops of purchasing a pool of IPs be it datacenter, residential, mobile etc, aren’t you set?

Well, yes and no. If your web scraping project is at such a small scale that you only need to collect one time data and you’re not pressed for time, then probably yes. It’s like herding your own flock of chicken to just get eggs.

But if you are planning on scaling up your data collection project to any reasonable scale, you will soon realize you need to also handle the nitty gritty complexities of managing the proxies. You need to design, implement, and maintain things like rotating, seamless geo-targeting, throttling and load balancing, managing sessions, and cycling out proxies that may be “burnt” by anti-bot systems. A pool of 5-10 proxies is manageable, but when you have 100s or 1,000s it can get messy fast. A robust proxy management system abstracts and manages all these sophisticated mechanisms for you.

Surely it’s not that difficult?

Well you might be surprised just how much goes into proxy management, and you will be happy to know tools are out there to help. Your options are: 

• Building a custom proxy solution from scratch, managing the infra, orchestration and maintenance of the system.

• Building your project using some 3rd party services to handle the harder, more boring, and automatable tasks around proxies.

• Completely handing over to (AI?)  tools or professional  services teams. 

Here are some common challenges that you will need to tackle when managing your proxy pool:

• Identify bans – Your proxy solution needs to be able to detect numerous types of bans so that you can troubleshoot and fix the underlying problem

• Retry errors – If your proxies experience any errors, bans, timeouts, etc. they need to be able to retry the request with different proxies.

• Manage user-agents– Managing user agents and headers is crucial to having a healthy crawl.

• Manage sessions – Some data sources require you to keep a session with the same proxy, so you’ll need to configure your proxy pool to allow for this.

• Add delays – Randomize delays and apply good throttling to not abuse the data source you are crawling.

• Geographical targeting – Sometimes you’ll need to be able to configure your pool so that only some proxies will be used on certain websites.

Right now, It’s just such a pain to do yourself, and the tools to manage it for you are so good that most of the professionals in industry use proxy management services.

Believe us, we’ve been building this tech for over a decade, concocting our own painkillers, so our data collection teams have been can use our own tooling to deliver billions of data points to our clients on a daily basis. We know the pain, have built systems to mitigate it, and you can tap into the same power as we do.

As anti-bot technologies become more sophisticated, traditional methods like proxy rotation, which have been reliable for years, are starting to show diminishing returns. With these evolving challenges, there’s a growing need for more advanced, future-proof solutions that can adapt to increasingly complex anti-bot detection systems.

Still curious? Other than the main task of minimizing bans, here is a deeper look at the activities that a proxy management service handle for you:

Managing IP Pools

• Acquisition: Proxy providers acquire IP addresses through partnerships, purchasing from ISPs, or utilizing peer-to-peer networks for residential and mobile IPs.

• Allocation: IP addresses are allocated dynamically or statically based on user needs. Rotating proxies are often managed through automated systems that change IPs at regular intervals or per request.

• Monitoring: Continuous monitoring ensures the health of the IP addresses, checking for blacklisting and performance issues. Unhealthy or “burnt” IPs are replaced or cycled out.

• User management: Providers manage user access to ensure fair usage and prevent abuse, maintaining the overall integrity of the IP pool.

Rotation and Anonymity

• Rotation: Automatic IP rotation helps maintain anonymity and avoid detection. This is crucial for tasks like web scraping and ad verification.

• Session management: Providers offer session control to keep the same IP for a user during a session and change it after the session ends, balancing the need for anonymity and session continuity.

Geolocation Services

• Geographical diversification: Providers maintain a diverse set of IP addresses from different geographic locations to help users access region-specific content or services.

• Location switching: Users can often switch between IP addresses from different regions as needed.

Security and Compliance

• Data encryption: Ensuring data passing through proxies is encrypted to prevent eavesdropping and maintain user privacy.

• Compliance with regulations: Adhering to local and international regulations regarding data usage and privacy, especially for residential and mobile IPs
  

The best proxy solution for your web scraping project?

So, which approach is the best option for you?

In the vast majority of cases, going with datacenter IPs and putting them through a robust proxy management solution will generate the best results for the lowest cost. With proper proxy management and crawler configuration, datacenter IPs give similar results as residential, mobile, or ISP ones at a fraction of the cost and relatively less legal concerns.

However, while starting with datacenter IPs is often a cost-effective and practical choice when paired with a robust proxy management solution, manually trying each proxy option—from residential to mobile proxies and ISP IPs—can take considerable time and effort.

Instead, automating this process can significantly reduce the time spent testing and adjusting your setup. Automated solutions not only adapt in real-time when proxy configurations stop working but also reduce mental load, improve speed, and allow for better scalability. In many cases, automation can match or even reduce costs, while still giving you full control and the ability to override settings when necessary.

If you still run into issues, you can then try other types of proxies and tweak the mechanism of your proxy management solutions based on what we’ve discussed today. There are so many moving parts and points of intervention to optimize your data collection project that most engineering teams decided to outsource these non-core activities to web data collection experts.

To help you make a more informed decision, here are some guiding questions:

1. What’s your budget? If you're working with a very limited or virtually non-existent budget, building your own proxy infrastructure may seem like the cheapest option. However, if you can allocate even a modest budget, opting for an off-the-shelf proxy management solution is worth considering. This removes the burden of managing proxies yourself while offering predictable costs. For a more efficient way to plan and control your spending, tools like the Zyte API provide a customized, cost-per-request model, tailored to specific sites—giving you full transparency over your expenses. In contrast, other proxy services often use throughput-based pricing, which can be harder to predict. Meanwhile, other unblocker APIs can be overpriced in cases where a simpler, more affordable datacenter proxy would suffice.

2. What are your technical skill level and your available resources? To be able to build and manage your own proxy infrastructure for a reasonable size web data collection project you will need data engineering expertise and the bandwidth to build and maintain your data pipeline and proxy management.

3. What is your #1 priority? If learning about proxies and everything about web scraping is your number one priority then building your own proxy infrastructure and managing it yourself may be a reasonable decision. 

   1. However, if your top priority is getting the web data you need as efficiently as possible and you can afford it, then it is wise to leave your proxy management solution to credible experts and tested off-the-shelf solutions such as proxy rotation service, and APIs than can handle requests and avoid bans out of the box. 

   2. Choosing a SaaS-driven automation solution like Zyte API from the start can often be faster and cheaper than building and maintaining a custom proxy setup. It’s designed to efficiently handle bans without unnecessary engineering complexity, giving you greater power with less effort.

Now that you have delved into the nuances of different proxy types, you’d agree that when it comes to web scraping, choosing the right proxy strategy can feel like a maze of trial and error. You could be juggling five different proxy providers, tweaking hundreds of different settings, and maintaining headless browser infrastructure to match it.

But what if you could automate that process and let an API handle the heavy lifting? To have a tool pick the best proxies for the job, orchestrate them, and handle bans and rendering—all without the costly trial and error.

Tools like Zyte API abstracts all these guessworks away from you while still granting you full control over settings and configurations. Now you can focus on your core tasks and have confidence that the API will get you the data you need.

To soothe your brain from overheating with this technical deep dive, let’s wrap up with this fun story of Zippy the IP.

Zippy’s Adventure in the Four Frontiers

One bright morning, as Zippy was contemplating its place in the world, an ancient and wise IP elder appeared. With a voice filled with wisdom, the mentor said, “Zippy, the time has come for you to embark on your rite of passage. To become a master, you must visit the four frontiers and collect their unique gifts. Only then will you be ready to conquer any digital frontier.”

With a heart full of excitement and determination, Zippy sets off on its journey. Zippy peeks at the map given by the elder:

1. Datacenter Land: A realm of towering servers and lightning-fast connections, where Zippy must navigate the Firewall Fortress to gain the Gift of Speed and Efficiency.

2. Residential Village: A serene place filled with homes and everyday internet users, where Zippy must overcome the Gatekeeper’s Challenge to earn the Shield of Trust and Anonymity.

3. ISP Town: A bustling town known for its balance of speed and reliability, where Zippy must outsmart the Bandwidth Bandits to obtain the Scepter of Balance and Stability.

Mobile City: A dynamic metropolis connected by mobile networks, where Zippy must evade the Signal Shifters to secure the Cloak of Seamless Mobility and Anonymity.

Datacenter Land and The Firewall Fortress

Zippy’s first stop is Datacenter Land, a realm of towering servers and lightning-fast connections. The cityscape is dominated by skyscrapers made of servers, connected by high-speed data highways. As Zippy zaps through Datacenter Land, it encounters the Firewall Fortress, a massive barrier that guards against unauthorized access. The fortress is equipped with advanced detection systems that can easily identify and block datacenter IPs.

The locals (websites) are suspicious of visitors due to frequent automated activities. The advanced detection systems can easily detect and block Zippy. But luck is on Zippy’s side because it can move incredibly fast, gathering data from multiple sources in no time due to the high-speed connections.

To pass the Firewall Fortress, Zippy needs to be stealthy and clever. It uses a disguise technique called “IP rotation,” switching between other datacenter IPs to avoid detection. Additionally, Zippy employs “user-agent spoofing” to look like different devices and browsers, making it harder for the fortress to identify its true nature.

Zippy zaps away.

Datacenter Land tech talk

• Source: Datacenters, large-scale server farms.

• Speed: High-speed connections due to robust datacenter infrastructure.

• Anonymity: Lower, as these IPs can be easily detected as proxy servers.

• Use Case: Best for tasks requiring high-speed access and high-volume data scraping.

Residential Village and The Gatekeeper’s Challenge

Next, Zippy zaps into Residential Village, a serene place filled with homes and everyday internet users. The village has cozy houses with Wi-Fi signals emanating from every window and tree-lined streets with diverse traffic.

In Residential Village, Zippy comes across the Gatekeeper’s Challenge. The gatekeepers (websites) are of cautious nature. They have strict entry checks and only allow visitors who seem like regular residents. They frequently update their verification methods to keep out suspicious visitors.

Zippy noticed that it is able to think much quicker now, allowing it to take a more human-like approach. It also uses “residential IP rotation” to switch between real residential addresses, making it look like a different resident each time. Zippy also facessolves CAPTCHA puzzles and provides valid cookies to pass the gatekeepers’ checks, blending in as a trusted visitor.

Zippy zaps away.

Residential Village tech talk

• Source: Real residential addresses.

• Speed: Moderate to high, depending on the user’s ISP and location.

• Anonymity: Higher, as as they are like regular users, reducing the chance of detection.

• Use Case: Suitable for accessing geo-restricted content and social media automation.

ISP Town: The Bandwidth Bandits

Zapping out of the Residential Village, Zippy then travels to ISP Town, a place known for its balance of speed and anonymity. The town features modern buildings with banners of different ISPs and efficient public transport with regular checks.

The vibe of ISP Town reminds Zippy of the speed of Datacenter Land with the trust of Residential Village, providing a balance of performance and anonymity.

Suddenly while exploring ISP Town, Zippy encounters the Bandwidth Bandits, a group of entities that limit the bandwidth available to proxies. These bandits slow down the network, making it difficult for Zippy to maintain high-speed connections. To make things even more complicated, there are a limited number of buildings (IPs), making it a bit crowded and sometimes more expensive. The Bandwidth Bandits restrict the available bandwidth, slowing down Zippy’s activities.

Luckily Zippy’s automatic charm by being in this place attracted a friendly group of IPs to collaborate with Zippy to access a premium network with higher bandwidth. By negotiating with multiple ISPs, Zippy secures a stable and fast connection, allowing it to continue its journey without being hindered by the Bandwidth Bandits.

Zippy zaps away.

ISP Town tech talk

• Source: ISP-owned IP ranges.

• Speed: High, leveraging ISP infrastructure.

• Anonymity: High, appearing as genuine user traffic.

• Use Case: Ideal for market research and ad verification.

Mobile City: The Signal Shifters

Finally, Zippy zaps into Mobile City, a bustling metropolis connected by mobile networks. The city is filled with skyscrapers with antennas and signal towers on top, and people constantly on the move, using smartphones and tablets.

In the bustling Mobile City, Zippy faces the Signal Snatchers, who frequently change the mobile signals, causing IP addresses to shift unpredictably. This constant change makes it hard for Zippy to maintain a consistent identity. The network is slower due to mobile traffic and complex pathways with constantly changing signals, making it hard for Zippy to stay consistent.

In this place, Zippy is nearly undetectable, blending perfectly with mobile users, providing the highest level of anonymity. Zippy further adapts by using a “mobile IP pool,” a collection of dynamic IPs from various mobile carriers. It learns to switch seamlessly between different mobile IPs, maintaining its anonymity and staying ahead of the Signal Snatchers. Additionally, Zippy leverages “session management” techniques to keep its connections stable despite the frequent signal changes.

Zippy zaps away.

Mobile City tech talk

• Source: Mobile carrier networks.

• Speed: Lower due to mobile network limitations.

• Anonymity: Very high, difficult to detect as proxies.

• Use Case: Best for mobile app testing and ad fraud detection.

After navigating through the distinct challenges and unique environments of Datacenter Land, Residential Village, ISP Town, and Mobile City, Zippy reaches the final leg of its journey. The adventures have not only equipped Zippy with a profound understanding of each proxy type but have also forged it into a resilient and versatile IP address. Now, Zippy stands at the gateway to the banquet to receive the gifts from each frontier.

• Datacenter Land’s Gift: A medal symbolizing unmatched speed and efficiency, representing Zippy’s swiftness and agility.

• Residential Village’s Gift: A shield of trust and anonymity, acknowledging Zippy’s ability to blend in and access restricted content securely.

• ISP Town’s Gift: A scepter of balance and reliability, denoting the stable and high-performance connections Zippy mastered.

• Mobile City’s Gift: A cloak of seamless mobility and anonymity, celebrating Zippy’s dexterity in navigating dynamic mobile networks.

As Zippy ventures into the vast expanse of the internet, it carries the lessons learned from each frontier, using its newfound capabilities to navigate, explore, and innovate.