Comparing Popular Web Scraping & Proxy APIs
Our report benchmarks nearly a dozen popular unblockers and web scraping APIs. These remote scrapers simplify the process of web data collection by overcoming CAPTCHAs, JavaScript virtual machines, and other roadblocks erected by anti-bot systems.
Though growing in popularity every year, web scraping APIs have become especially relevant with the rise of AI models, closing down of online platforms, and commercialization of bot protection approaches.
Our main goal is to see how well the APIs are able to unblock protected websites in late 2024 (for earlier reports refer to: comparison of web scraping APIs (2023), comparison of proxy APIs (2023)). We also take a look at their features and pricing strategies to get a well-rounded view of the market.
Summary
- Our list of participants included 11 API providers, which we tested on 10 protected websites at a rate of 10 requests per second.
- Five APIs managed to open all targets consistently, while the others failed to unblock between one and five websites. Oxylabs, Zyte, and Bright Data had the highest average success rate of around 98%.
- Zyte’s API worked extremely fast, managing to unblock all targets without headless browsers. Bright Data’s Web Unlocker, though second to last in speed, was the only API that didn’t fail a single run.
- G2 (Cloudflare) proved to be the hardest target by success rate, while the largest number of participants – five – failed to unblock Allegro (DataDome).
- Compared to proxy APIs (also called web unblockers), web scraping APIs have more features: asynchronous delivery, data parsing capabilities, and specialized endpoints. Some also include a proxy mode, making the distinction arbitrary.
- We’re seeing more providers release specialized endpoints for popular targets. In addition, several AI-based parsing approaches have appeared, ranging from models trained on page types to AI-generated parsing schemas.
- Credit-based pricing models are usually chosen by providers that service smaller customers; while extremely cheap for basic websites, they impose huge multipliers when accessing challenging targets.
- Out of the business-oriented providers, Zyte is very hard to beat on price for unblocking tasks. For targets that rely on JavaScript or need special functionality (e. g. localized Google or Amazon queries), Smartproxy and Oxylabs offer a compelling balance between performance and cost.
Participants
Our research includes 11 major providers of web scraping and proxy APIs (often called web unblockers). The tools technically form different product categories, but we decided not to separate them. Both tend to use the same tech, and web scraping APIs sometimes have a proxy mode as one of the integration formats, which further blurs the distinction.
Most participants are well known in the industry, though not necessarily for their scraping infrastructure. Here’s the full list:
Methodology
We gave all participants the methodology doc in advance. Some actively monitored our progress, making adjustments to their scrapers on the fly. This is fine, as web scraping is a dynamic process, and it should be approached as such. Hopefully, we also helped to improve the success rate for actual customers.
We chose 10 targets based on their popularity and bot protection system. Our goal was to try the scrapers with all major anti-bot vendors.
There are some caveats to consider:
- Anti-bot systems may have different levels of protection based on the website (or even categories of the same website).
- Some bot protection vendors focus on securing sensitive endpoints (such as internal APIs or login pages), so they may not show up in full force against simple collection of public content.
We ran at least three tests for each target throughout several weeks. We fetched ~6,000 unique URLs, navigating directly to the page. The rate was 10 requests per second, with a timeout of 600 seconds. This is enough to trigger bot protection systems – and, as we’ll see, seriously tax some of the scrapers.
We used a custom Python script – its function was to simply send the request to the scraper and receive the response, measuring the time it took to reach us. Our server was located in the US.
Participants were free to suggest the optimal parameters for the targets, and some did. Otherwise, we used our own discretion, starting with the simplest configuration and enabling optional features (such as premium proxies and headless browsers) if we couldn’t unblock or load the valuable content.
We verified a request’s success by checking the response code and HTML size. The latter was necessary, as some websites (such as Safeway) tend to return 200-coded responses without data.
Some providers imposed concurrency limits that could potentially affect our scraping rate:
- ScraperAPI’s largest public plan has a limit of 100 concurrent threads, which isn’t enough for 10 requests/second, especially when headless browsers are involved.
- Infatica and Scrapingdog have the same restriction.
- Zyte’s default rate limit is 500 requests per minute (~8.3/second).
ScraperAPI, Scrapingdog, and Zyte lifted their restrictions for us. Infatica wasn’t able to, forcing us to scrape most websites at 1 req/s. We also encountered SOAX’s internal limits and decided to stick with ~5 req/s for more complex targets.
Benchmark Results
The results show the best run each API had with each target. We’ll provide comments to give you more context where necessary.
Overall Performance
| Oxylabs | Zyte | Bright Data | Smartproxy | Nimble | NetNut |
|---|---|---|---|---|---|
| 98.50% | 98.38% | 97.90% | 96.29% | 95.48% | 80.82% |
| SOAX | ScraperAPI | Scrapingdog | Infatica | Rayobyte | |
| 69.30% | 67.72% | 43.84% | 38.40% | 37.65% |
| Zyte | NetNut | Smartproxy | Scrapingdog | Nimble | SOAX |
|---|---|---|---|---|---|
| 6.61 s | 9.71 s | 10.91 s | 10.92 s | 13.01 s | 13.41 s |
| Oxylabs | ScraperAPI | Infatica | Bright Data | Rayobyte | |
| 13.45 s | 15.39 s | 17.15 s | 22.08 s | 26.24 s |
Five providers managed to open all targets more or less consistently, which is an excellent result. We can distinguish Oxylabs and Zyte for their overall success rate; and though it doesn’t reflect here, Bright Data was amazingly dependable, never failing a single test.
The rest had at least one target that gave them trouble. But you shouldn’t discount these APIs just by looking at one aggregate number. For example, aside from Lowe’s and Safeway, NetNut performed flawlessly with most websites, and so on.
In terms of response time, Zyte’s API was a speed demon, beating others by up to four times. The provider made some adjustments during the process, and by the end, it could somehow open all targets without requiring JavaScript rendering.
Bright Data obviously prioritized unblocking success, and so it performed slower than expected. We believe that a better way to scale its APIs would be through more parallel requests, which our testing parameters didn’t fully exploit.
Hardest Targets to Unblock
We omitted Infatica’s and SOAX’s results from the success rate column, as they were tested under lower rate limits (one and five requests per second, respectively).
G2 (Cloudflare) proved the hardest to unblock looking at the average success rate of all providers. However, it was actually Allegro that the most participants failed to open consistently. Its average success rate is boosted by the rest.
On the other hand, most APIs managed to open Google and Amazon nearly perfectly. As major web scraping targets, they’re the baseline for any commercial data collection service.
Breakdown by Individual Target
As one participant exclaimed, Allegro is very hard to unblock. The website uses DataDome, and it’s even featured as a success case in the anti-bot vendor’s website.
In reality, we saw one of two extremes: an API either opened Allegro perfectly, or it failed completely. The same story repeated consistently across our tests. All in all, Oxylabs and Smartproxy performed particularly well here.
Amazon is the website for web scraping, so unblocking it is a must for any self-respecting service. As such, Amazon proved to be the least problematic target.
We saw consistent results with little deviation across runs. Though the difference was minimal, ScraperAPI had the best showing.
The Canada Goose store includes only several hundred products, but it’s visited by hundreds of thousands of people every month. The website uses Kasada, which was a hard nut to crack. The participants either had a bypass method, or they failed to unblock this target.
Like in The Web Scraping Club’s benchmark, NetNut’s unblocker had the best success rate, though it wasn’t the fastest. Some APIs had significant variance between runs: Nimble, ScraperAPI, and Smartproxy failed several tests and then fixed their scrapers for others.
G2, a major company review website, is protected by Cloudflare. We found it to be the most challenging target, giving even the most solid APIs a run for their money.
Again, NetNut showed the best performance, both in success rate and response time. As with Canada Goose, the results weren’t always consistent between runs for more than one participant.
Google is a must for any web scraping API. The search engine is protected by the infamous reCAPTCHA, which quickly rate limits suspicious visitors. But it proved no challenge to all but one API.
We’re particularly impressed with Zyte’s performance. Zyte API not only achieved a perfect success rate, but it returned requests in under one second – much faster than the rest.
Contrary to our expectations, Indeed wasn’t a hard target for the scrapers. The website employs Shape, a notoriously hard anti-bot system, but we either failed to trigger it or Indeed is using a lenient configuration.
In any case, at least five providers had amazing results, making it hard to distinguish one outlier. Similar outcomes repeated throughout all runs, with the exception of ScraperAPI.
Instagram is another major source of web data, though TikTok has probably started challenging it in popularity by now. The social media network uses its own bot protection system that redirects suspicious users to a login page. In our tests, however, Instagram didn’t cause big issues for most participants.
Overall, Nimble’s results look the best on paper. It’s also interesting that Zyte adjusted its scraper in the process, and our last test ran successfully without JavaScript rendering enabled. As a result, Zyte’s response time is mighty impressive.
Lowe’s is a decently popular target that employs Akamai’s bot protection system. It brought down a third of the participants, including NetNut, which came out strong with other anti-bots.
On the other hand, six APIs succeeded over 99% of the time, which is a great result.
Safeway, the U.S. supermarket chain, is protected by Imperva and imposes aggressive geo-restrictions outside North America. The website isn’t a very popular target, so most participants found it tricky, requiring several runs to adjust.
All in all, Zyte’s performance looks amazing on paper, but it was Bright Data that ensured consistent results throughout all tests.
Though probably eclipsed by Amazon, Walmart is a major e-commerce data source. It tends to juggle anti-bot systems but is generally associated with PerimeterX.
Most participants didn’t find Walmart problematic. However, we did see Nimble’s and Scraping Robot’s success rates crash after PerimeterX’s update late August.
Other Observations
- When accessing protected targets, commercial APIs can be brittle. Less popular websites may need to be unblocked, even if the provider has a general bypass for that bot system. Popular targets like Walmart or G2 may too temporarily break after major updates.
- Providers use different approaches for unblocking the same websites. Nimble relies on what it calls browserless drivers – they render JavaScript without invoking traditional headless browsers. We saw a big reliance on these drivers. On the other hand, by the end of our tests Zyte was able to access all targets without requiring browser-rendered HTML at all.
- There’s a big difference between running tests at one request per second and ten or more. First, we wouldn’t have discovered that providers had scaling issues. Second, some websites don’t start seriously blocking until five requests per second or more.
Feature Overview
Let’s take a quick look at what you can do with the scraper and proxy APIs.
Proxy vs API Integration
The question of integration method is often already decided before purchase: if your codebase takes the proxy format, you’ll naturally gravitate towards it. But is there a real difference between the features API and proxy integration methods offer? In a way, yes.
Proxy APIs:
| Integration | Geolocation | Sessions | Custom headers | JS rendering | Specialized endpoints | Data parsing | |
|---|---|---|---|---|---|---|---|
| Bright Data | Proxy, async API | 150+ countries with city & ASN targeting | ✅ | ✅ | Automated, toggle | Search engines | Specialized endpoints |
| NetNut | Proxy | 150+ countries | ✅ | ✅ | Toggle | ❌ | ❌ |
Web scraping APIs:
| Integration | Geolocation | Sessions | Custom headers | JS rendering | Specialized endpoints | Data parsing | |
|---|---|---|---|---|---|---|---|
| Infatica | Real-time, async API | 150+ countries | ✅ | ✅ | Toggle | Search, e-commerce | Specialized endpoints |
| Nimble | Real-time, async API (with batching, cloud storage) | 150+ countries with state & city targeting | ✅ | ✅ | Toggle, instructions | Search, e-commerce, social media | Manual, autoparser, special endpoints |
| Oxylabs | Real-time, async API (with batching, cloud storage), proxy | 150+ countries with ZIP for Amazon, city & coordinates for Google | ✅ | ✅ | Toggle, instructions | Search, e-commerce | Manual, special endpoints, parser builder |
| Rayobyte | Real-time, async API (with batching) | 150+ countries | ✅ | ✅ | Toggle, instructions | Search, e-commerce | Manual, special endpoints |
| ScraperAPI | Real-time, async API (with batching), proxy | 12 countries with 50+ upon request, ZIP code for Amazon | ✅ | ✅ | Toggle, instructions | Search, e-commerce | Manual, special endpoints |
| Scraping Dog | Real-time, async API, proxy | 15 countries | ✅ | ✅ | Toggle, instructions | Search, e-commerce, social media, more | Special endpoints |
| Smartproxy | Real-time, async API (with batching), proxy | 150+ countries with ZIP for Amazon, city & coordinates for Google | ✅ | ✅ | Toggle, instructions | Search, e-commerce, social media | Manual, special endpoints |
| SOAX | Real-time | 150+ countries | ❌ | Cookies | Toggle | Search, e-commerce, social media | Special endpoints |
| Zyte | Real-time API, proxy | 150+ countries | ✅ | ✅ | Toggle, instructions, scripting | ❌ | Manual, category based |
Proxy APIs are often meant to be a direct upsell to proxy servers with a drop-in replacement process. At the same time, because you’re effectively outsourcing the page opening stage, proxy APIs need to go beyond regular proxy network features like geo-location to cover request manipulation and even JavaScript rendering. So they do.
Despite their wealth of features, proxy APIs can still be limited. For example, they rarely offer specialized endpoints, on-demand access to scraped output, or data structuring features. Another big drawback for complex scenarios is incompatibility with headless browser libraries, with no browser instructions on their own. This is where web scraping APIs offer more flexibility.
Exceptions exist. Bright Data’s SERP API integrates as a proxy, but in reality it’s a highly specialized scraper with data parsing and custom parameters. Funnily enough, some providers that sell web unblockers also offer web scraping APIs with a fully-featured proxy mode. In these scenarios, the difference hinges on the pricing method and, likely, marketing strategy.
How do you work with proxy and web scraping APIs? Obviously, the main requirement is sending an HTTP request to the provider’s server. However, the way you configure that request can differ. It’s usually either a GET request with parameters in the URL, headers, or a POST request with a JSON payload.
Exploring Individual Features
All modern proxy and web scraping APIs can render JavaScript. With pages becoming increasingly interactive, a question that arises more often every year is: what else can I do on the page? Proxy APIs tend to ignore it; the solution of scraping API developers is to expose browser controls through special parameters.
Bright Data, Infatica, NetNut – only basic rendering functionality available.
It’s possible to combine the instructions. For example, you can select a field, enter text, click on it, and wait for the response. Providers impose execution time limits, which often range between 60 and 120 seconds.
Zyte takes this a step further. Its clients get access to a cloud-hosted VS Code environment, where they can write their own interaction scripts.
The latter functionality isn’t common: instead, we’re seeing new categories emerge with the aim to increase web scraping success while providing standard compatibility with headless browser libraries. Some examples would be Undetect, Bright Data’s Scraping Browser, and major anti-detect browsers like Multilogin and Gologin.
Specialized endpoints are tailor made for websites or their properties (such as Amazon product pages). They often have custom parameters and data parsing capabilities. For instance, a Google SERP endpoint may be able to fetch local results (city-wide or in particular coordinates), which would otherwise be unavailable when targeting a general-purpose API.
NetNut, Zyte – no specialized endpoints available for the tested products.
Compared to the year before, we’re seeing an interesting trend: scraper vendors have been introducing more specialized endpoints to their products. One example is ScraperAPI, which now offers scrapers for Amazon, Google, and Walmart. Another is Nimbleway – the provider has set out to build what it calls online pipelines for targets in various verticals.
The direction is interesting, considering that LLMs have reduced the barrier to entry particularly when it comes to parsing, and that they tempt to consolidate toward one all-encompassing tool. Maybe a single purpose reassures that the scraper will be fit for the task?
Data parsing is an area where some of the most exciting developments are taking place. Of course, this is thanks to machine learning and large language models. But we’re also seeing changes in less sophisticated approaches: since our previous report, Oxylabs, ScraperAPI, and Smartproxy have all implemented selector support for building parsers by hand.
NetNut – no data parsing available for the tested products.
Let’s explore several different approaches to AI-based parsing that lie in the modest Other column.
#1. Custom machine learning models trained on specific page types.
Zyte has been playing around with machine learning for years now. Instead of parsing individual targets, Zyte trained multiple in-house models for whole page categories: products, news, directories, etc. The caveat was that they relied on AI vision, which required browsers. Still, during its conference roughly a year ago, Zyte bragged about being dozens of times cheaper and more accurate than ChatGPT.
Since then, Zyte has adapted the models to non-rendered requests, significantly cutting down the cost. It’s also experimenting with supplementary LLM features. They can make the schema more flexible by adding custom data points, and they can also transform data: translate, normalize, summarize, etc.
#2. A universal AI parser.
Similarly to Zyte, Nimble uses HTML-trained AI agents to extract data from various page types. Unlike Zyte, the provider automatically chooses the relevant agent depending on the page, keeping the decision process in the backend.
In a way, this makes the customer’s job easier. But it’s also way less predictable (Willl this target work? What’s the schema?). During our tests, we found the functionality to be more miss than hit: it parsed Lowe’s but failed to structure Canadagoose or G2. We’re sure it’s bound to improve fast.
To make the agents more robust, Nimble is preparing to release the ability to generate custom schemas. This feature will accept simple, likely natural language instructions and translate them into parsers. According to Nimble’s documentation, these parsers will get reusable IDs and heal automatically after identifying a failure.
For now, Nimble’s stopgap solution combines the dynamic parser with manual selectors to build a parser for the page.
#3. LLM-assisted parsers generated upon request.
This is the approach Oxylabs announced during its recent web scraping conference. Basically, you send a URL with natural language instructions to an LLM, then it generates a schema and selectors for scraping the data points. You get a preview of the output and the ability to adjust the schema to your needs. Once you’re happy, the selectors get added to the API request code.
Oxylabs’ approach is highly pragmatic, as the language model is invoked only once and not with every page access. However, it has limitations, namely that once a parser breaks, you have to manually repeat the generation process.
Pricing Approaches
We’ll overview the pricing models of the participants and how much our benchmarks would’ve cost.
Request, Credit-Based and Black Box Models
There are multiple ways to price proxy and web scraping APIs. The former use traffic or requests as the main metric. The latter charge for successful requests, either by keeping their model simplistic (one page = one request) or creating increasingly elaborate schemes based on credits.
Zyte’s model is closer to credits, for it includes variables that affect the final rate. But it’s also unique because the cost may change with time, depending on how hard Zyte finds the target to scrape. According to the provider, these revisions take place once per quarter and affect around 0.1% of websites. Still, such a pricing scheme works as a kind of black box.
The table provides some interesting data points:
- Scraper vendors prefer trials over paying as you go. Only Scraping Robot has PAYG as its sole pricing model, and Zyte starts requiring commitment after $100. Furthermore, some trials extend to free plans that refresh monthly.
- Credit-based pricing usually targets customers with smaller needs. This is evident from looking at the price ranges of public plans.
Price Modifiers
To understand how exactly request-based and credit-based pricing models compare, we’ll have to explore the base price and available modifiers.
The CPM at $100 column shows how much 1,000 requests would cost when spending $100 with each participant. It may a little be biased against enterprise-minded providers, as they start to scale well at $1,000 and up.
NetNut – no public pricing available.
Credit-based pricing models can have huge multipliers reaching tens or even hundreds of times. These variables interact with one another: for example, you can toggle both JavaScript rendering and better quality proxies. From the user standpoint, having these options exposed can feel burdensome, as you need to experiment with parameters and mind the credit cost.
Having said that, they’re really efficient for basic websites that require neither residential proxies nor JavaScript rendering. The low baseline price also makes these scrapers look really good in marketing materials. However, for hard targets like G2, you’ll be likely to overpay.
Request-based models have the opposite problem: they’re really expensive if the target doesn’t bite. But given that these providers are often enterprise-oriented, different considerations kick in, such as scalability and unblocking success.
The Cost to Run Our Benchmarks
So, how much did we pay to complete the full 180,000 requests (10 * 6,000 * 3)? The graph shows aggregate costs, taking the rate of the closest suitable plan.
Three participants failed to unblock some targets consistently (at least 20% of the time) and use credit-based pricing. We didn’t want to speculate on the configuration, so we excluded the following from the graph:
- Infatica: Canadagoose, G2, Lowe’s, Safeway, Walmart.
- Scrapingdog: Allegro, Canadagoose, G2.
- ScraperAPI: Allegro, G2.
Hover on a label to highlight, click to filter.
For general unblocking without JavaScript rendering or data parsing, Zyte delivered incredible value considering its performance results. The provider’s price was closer to entry-level APIs like Infatica and Scrapingdog than the premium competitors.
Smartproxy and Oxylabs also look compelling, more so if you need headless browsers or the bundled parsing features. And while this API may not be the most efficient choice in general, ScraperAPI’s prices for Amazon and Walmart in particular are definitely worthy of attention.
Conclusion
This concludes the report. Assuming that very few readers will reach this part, we moved the summary to the beginning. But since you’re here – thank you for getting to the end! If you have any questions, feel free to contact us through info at proxyway dot com or our Discord server.
