Web Scraping API's are cheaper than Proxies?

Web scraping APIs are expensive, right? Well, no. I don't think they are. And in fact, I think they offer a much better value proposition for anyone trying to scrape data, especially compared to just using proxies by yourself. And in this video, I'm going to convince you why, or at least that's my plan. If we look at this playground I've got open here, this is our API. I'm going to make a request to this blog post. Straightforward HTML page. I just want the HTML data back. We can see that I get that here. And this was the cost for the request. This is fairly standard for this sort of request with us. 0.065 of a cent of a dollar sorry per request. Now this is a tier three site which basically you know we categorize our sites into tiers. So you get a better idea of how much it's going to cost how much how much compute it requires for us how much we have work we have to do to get the response. This is fairly typical for a very very simple page. Right? In this instance if I was to say this cost 0.0065 0065 uh per request. And I copy this out and we come over to the calculator. Let's put this in here. 0.00.65. And let's say we'll work on a a working of 100,000 requests. So for 100,000 requests, it's going to cost you $65. Um now to compare this to proxies, I'm going to compare the flat cost initially and then we'll look at why the products are not the same and and we'll talk a bit more about it there. So to compare that because proxies proxies are charged per gigabyte, we need to know how heavy the page is, right? So this is the page right here and if I was to go to view page source, this is the raw HTML. That's pretty much would have been exactly what we got back. And if I save this out to my code editor, it's going to look like this. It's truncated it all or whatever. Now, if I open this file in Finder, you can see that this is 381 kilobytes. If I was to open this page in my browser, not a lot of things are going to load up because, you know, we're not making any of the JavaScript running or anything like that. You can see we have the text here. And this is pretty much standard, I think, of what you can expect. Now, this was 381 kilobytes. So, if I was to do 381 kilobytes in gigabytes, that gives us 0.0381. And then I do this times by 100,000. We're going to get 38.1 GB. Now the typically gig per gig for residential proxies you're looking at about $4ish. Now obviously this depends on supplier provider. It depends on the volume. But if we worked on the worst case scenario of let's call it four uh $4. So let's do times by four. And this overall 100,000 of these pages would cost you about $150. Now, that's quite significantly more. But even let's say we had a better deal with our proxies and it was, you know, 38 GB and we were working on the $2 per gigabyte, that's still $76, which is more than our web scraping API in this instance. Now, there's a few reasons for this. The first one being that when we do our request like this, I'm running a few different things through here. I'll explain that in a minute. When we do this, we work out what the cheapest IP, the cheapest proxy that we can use to make work with that website and then we pass those cost savings on to you. So, I'm comparing us getting the data versus you wanting to use residential proxies because they have the best success rate. Now, if you wanted to do that yourself, you're already looking at extra time that's going to take you to manage those proxies. So you're going to need to manage uh what ones work for what sites that you're scraping, right? So you need to have access to multiple proxy accounts or you know multiple proxy strings and you need to manage those going into your program into your code. With us it's just one request. Just say that you want the HTML response body and our API will handle it all for you. The second point is that when you start to get blocked or banned, our API will automatically ramp up what's required to get that data for you. Whereas your proxy is not going to do that. It's just going to return you whatever it was there, whatever came back, you know, Cloudflare captures or, you know, anything like that. But with a proxy, that still counts as data. It's still returning information through your proxy. So, you're going to have to pay for that. So, anytime something fails, you're still paying for that data, for that traffic. But through our API, you don't pay for that. You only pay for successful requests. Now, obviously, this is a very difficult comparison to make. But generally speaking, and I talked about this at the start of the video, it's the value proposition that you get. So, not only do you get access to, you know, just a one request API that does everything for you, which is also extremely configurable and has a load more options over proxies. It takes away any of the headaches of having to manage that all as well. Managing proxies is not straightforward. It's difficult. It requires upkeep. At the very le at the very least it requires some maintenance which is going to cost you your time. Whereas with the Zite API we do it all and you don't have to worry. Now I talked about these not being the same product. For example, if you were to start using a browser that you needed to scrape data from a site or you needed to run actions that's going to take much more gigabytes of data off of your proxies. You can do you can do certain things like you can block images, you can block certain things from running. But then this is all extra overhead that you're going to have to manage yourself. Let alone running multiple instances of browsers to do it all for you. It's it's difficult and it becomes quite challenging. Now obviously you can do this through Z API. You can run it as browser HTML is equal to true. You would just put that in here and that would actually then allow you to run a browser in the background on our servers that will do it all for you. Now this obviously comes with a bit more of an extra cost because we're running more infrastructure for you. We're doing it for you. but it's still taking away that pain from you. Now, I like to look at this a couple of different ways. Not only is our web scraping API going to be cheaper than proxies proxies in some cases, it's also going to give you that peace of mind that it's always going to work and you are charged by request, not per the amount of data that you use. And if anything changes, you can just add it straight into your request and away you go rather than having to, I don't know, rewrite part of your proxies or manage your write a whole proxy management program for example to do. It's going to save you a load of time there. I just want to touch on one other thing before I let you off is that not only do we give you access to these sites, but we can also return you structured data. And that's what I was doing in there. I was returning structured uh JSON data that our ML models have passed out of the page. Now obviously this is going to cost you a little bit more money but if I just show you here run this against the article type. Now we have like quite a few data types for the common sort of things that when this comes back you're going to get a article returned to you in JSON data which means you don't have to do any HTML passing and this only put the cost up by an extra you know 0.004. Now, if I look at this cost uh in our calculator, I think that would come out at about $100 per 100,000 requests, right? So, this way, maybe it's costing you slightly more than the proxies, but you're getting JSON data back. You don't have to do any HTML passing, and that just saves you so much more time. And that's why I think using a web scraping API over proxies is a much better value proposition and you know a much better way to maintain consistency and accuracy of getting your data from the web.