Research - Proxyway

Proxy Market Research 2026

Adam Dubois — Wed, 08 Apr 2026 08:16:59 +0000

Research

Welcome to our report on the proxy server industry. It overviews the market as a whole, then benchmarks and compares major proxy networks. Proxy Market Research has been running each year since 2019.

You’ll find this report useful whether you’re a customer looking for a proxy provider, an analyst investigating the market, or a vendor evaluating where your service stands compared to established names.

This year’s report:

Features 13 companies that cover all segments of the market.
Overviews three proxy server types: residential, mobile, and shared datacenter.
Describes the prevailing market trends informed by a questionnaire we sent to all participants.
Provides large-scale performance benchmarks enriched with years of historical data.

The majority of data were collected in March and April of 2026.

Participants

Here are this year’s participants with the proxy networks we tested for the report. If you’re unfamiliar with the types of proxy servers, refer to our guide explaining their differences.

We tried to segment the participants by their target audience. Our taxonomy relied on brand messaging, pricing schemes, and enterprise-oriented features:

Entry-level providers work best for individual or small-scale projects.
Mid-market providers cater to a wide range of scraping, account management, and similar tasks.
Enterprise providers focus on an integrated experience that accommodates large-scale use by teams.

	Target audience	Residential	Mobile (pool)	Datacenter (shared)
NetNut	Enterprise	✅	✅	✅
Oxylabs	Enterprise	✅	✅	✅
Infatica	Enterprise leaning	✅	✅	✅
Byteful	Mid-market	✅	✅	❌
DataImpulse	Mid-market	✅	✅	✅
Decodo	Mid-market	✅	✅	✅
IPRoyal	Mid-market	✅	Not tested	❌
SOAX	Mid-market	✅	✅	❌
Proxyrack	Mid-market	✅	Not tested	✅
Rayobyte	Entry/mid-market	✅	✅	✅
Evomi	Entry-level	✅	✅	✅
ProxyEmpire	Entry-level	✅	✅	Not tested
Webshare	Entry-level	✅	❌	✅

Provider	Key Information	Description
	Country: UK Founded: 2018 Employees (LinkedIn): Up to 10 Type: Independent Segment: Mid-market Proxy networks: residential, ISP, mobile Other services: –	A former sneaker proxy vendor turned general purpose, Byteful focuses on tech and platform building. It was known as Ping Proxies until recently. To our knowledge, Byteful acquires ISP proxies in-house and residential IPs through an SDK, among other sources.
	Country: US Founded: 2022 Employees (LinkedIn): 10-25 Segment: Entry-level Type: Company group Proxy networks: datacenter, residential, mobile Other services: –	A product of the Ukrainian dev house Softoria, DataImpulse runs on affordable plans with non-expiring traffic. Its sister brand is a popular search API DataForSeo. The provider sources proxies through an SDK, as well as a bandwidth-sharing app.
	Country: Lithuania Founded: 2018 Employees (LinkedIn): 100-150 Segment: Mid-market Type: Company group Proxy networks: datacenter, residential, ISP, mobile Other services: scraping APIs, datasets	Known as Smartproxy until April 2025, Decodo is a top five provider and a long-standing value choice. The company is associated with the Lithuanian Tesonet group.
	Country: Switzerland Founded: 2024 Employees (LinkedIn): Up to 10 Segment: Entry-level Type: Independent Proxy networks: datacenter, residential, ISP, mobile Other services: scraping APIs, browser	The successor of a brand called InfiniteProxies, Evomi emphasizes its Swiss roots and reliability. The provider uses an extremely flexible pricing model for its residential IPs, which are sourced through Earn.fm, an SDK, and maybe other proxy vendors.
	Country: Singapore Founded: 2019 Employees (LinkedIn): 25-50 Segment: Enterprise leaning Type: Independent Proxy networks: datacenter, residential, ISP, mobile Other services: scraping APIs, datasets	A company with at least three offices worldwide, Infatica can be considered a consistenly growing mid-sized provider. It openly sources IPs through an SDK, actively participates in conferences, and has been going upmarket for the last few years.
	Country: Lithuania Founded: 2021 Employees (LinkedIn): 100-150 Segment: Mid-market Type: Independent Proxy networks: datacenter, residential, ISP, mobile Other services: scraping API	A quickly-growing value choice with enterprise aspirations. The company’s claim to fame has been scaling pay-as-you-go residential traffic. IPRoyal maintains its own residential proxy pool through an app called Pawns.
	Country: Israel Founded: 2018 Employees (LinkedIn): 100-150 Segment: Enterprise Type: Independent Proxy networks: datacenter, residential, ISP, mobile Other services: scraping APIs, datasets	The only publicly traded company here and a top five proxy provider by size. NetNut focuses on serving enterprise customers that operate at scale.
	Country: Lithuania Founded: 2015 Employees (LinkedIn): 400-500 Segment: Enterprise Type: Company group Proxy networks: datacenter, residential, ISP, mobile Other services: scraping APIs, browser, datasets	Perhaps the market leader, competing with Bright Data in the enterprise segment. Oxylabs sources IPs through partner apps Honeygain, JumpTask, and an SDK.
	Country: Bulgaria Founded: 2018 Employees (LinkedIn): Up to 10 Type: Reseller Segment: Entry-level Proxy networks: datacenter, residential, ISP, mobile Other services: –	A well-known brand, ProxyEmpire offers all major types of proxy servers at affordable entry points. Its major selling point is traffic that carries over subscription cycles.
	Country: Hong Kong Founded: 2014 Employees (LinkedIn): Up to 10 Type: Independent Segment: Mid-market Proxy networks: datacenter, residential, ISP, mobile Other services: Scraping APIs	A veteran provider known for its unlimited traffic residential proxy plans. Proxyrack keeps a low profile when it comes to its team; but unlike other Hong Kong providers, it imposes strict usage terms. The company sources IPs through an SDK and proxyware.
	Country: United States Founded: 2015 Employees (LinkedIn): 25-50 Segment: Mid-market Type: Independent Proxy networks: datacenter, residential, ISP, mobile Other services: scraping API, browser	A former datacenter proxy powerhouse, Rayobyte now offers a broad range of products. The provider talks about enterprise but tailors its service and communication in favor of entry-level users.
	Country: UK Founded: 2020 Employees (LinkedIn): 50-100 Segment: Mid-market Type: Independent Proxy networks: datacenter, residential, mobile Other services: scraping APIs, datasets	A mid-sized technologist with an international team. SOAX offers a wealth of features and a unified platform for all its products.
	Country: US Founded: 2018 Employees (LinkedIn): 25-50 Segment: Entry-level Type: Company group Proxy networks: datacenter, residential, ISP Other services: –	A hugely popular destination for server-based proxies thanks to its flexible pricing and powerful self-service features. The company was acquired by Oxylabs in 2022, which likely supplies Webshare with residential IPs.

🔎 Market Overview

This section provides an overview of the proxy server market. It’s based on our own observations and enriched with responses from a questionnaire we sent to all participants.

🗞️ 2025 Report: AI Continues Propelling the Market Hard

In 2025, AI-related startups received more venture capital than all other companies, creating a US-based data infrastructure industry.
Proxy providers also reaped the benefits, some growing 50% of more year over year.
The plans for 2026 involved maintaining and growing capacity, as well as continuing to scale vertically.

Full version

We Tested Smartproxy.org’s IP Pool: Here’s What We Found

David Barnes — Tue, 31 Mar 2026 13:32:16 +0000

Research

In late January 2026, Google disrupted IPIDEA – one of the largest residential proxy networks in the world, linked to botnets, cybercrime, and deceptive device recruitment. Weeks later, we tested a provider, also known as the original Smartproxy (now Decodo) impersonator, Smartproxy.org. We purchased a weekly plan, ran roughly seven million requests, and compared the exit IPs against a verified IPIDEA database.

Key Findings

The test surfaced 2,023,029 unique IP addresses (2,019,488 IPv4 and 3,541 IPv6).
We compared these IPs against a verified IPIDEA dataset of 16,192,293 unique IPs covering January 2026.
773,087 IPs (38.21%) from Smartproxy.org’s pool were also present in the IPIDEA dataset.
The overlap strongly suggests that Smartproxy.org either resells IPIDEA’s infrastructure directly or uses it as a significant IP source.

Background: What Happened to IPIDEA

On January 28, 2026, Google announced coordinated action against IPIDEA, a Chinese company operating one of the world’s largest residential proxy networks. According to The Wall Street Journal, the measures were expected to knock over nine million devices off the network.

IPIDEA controls or is related to more than a dozen Hong Kong-incorporated brands selling proxy servers, including 360Proxy, 922Proxy, ABC Proxy, Cherry Proxy, IP2World, LunaProxy, PIA S5 Proxy, PyProxy, and TabProxy.

Google Threat Intelligence Group’s action had three main components:

Domain takedowns. Legal action to seize both the public-facing domains of IPIDEA’s brands and the command-and-control domains used to manage the proxy network.
Intelligence sharing. Technical intelligence on IPIDEA’s SDKs and proxy software was shared with platform providers, law enforcement, and research firms. SDKs are components installed into apps like free VPNs to silently recruit devices into the proxy network.
Google Play enforcement. Google Play Protect now automatically warns users and removes applications known to incorporate IPIDEA SDKs, while also blocking future install attempts.

Why It Matters?

Google’s research found that IPIDEA’s network had facilitated several large botnets, including BADBOX2.0, Kimwolf, and Aisuru. Over 550 threat groups used IPIDEA’s proxies to mask malicious activity. External research by Synthient surfaced vulnerabilities in IPIDEA’s infrastructure that allowed botnet operators to access and recruit other devices on local networks.

This wasn’t a minor compliance issue. IPIDEA had become a significant vector for cybercrime, and its reach extended well beyond the brands that shared its name.

IPIDEA’s Impact

Between 2022 and 2024, IPIDEA-related brands had a profound effect on the proxy market. Over 20 new entrants, most incorporated in Hong Kong, flooded the space with cheap residential proxies. They undercut established competitors and, more importantly, popularized unlimited plans that offered unmetered bandwidth and requests for a fixed fee. This was a departure from the industry standard of per-gigabyte or per-request pricing.

In the WSJ article, IPIDEA’s spokeswoman acknowledged that the company and its partners had engaged in aggressive market expansion and promotional activities in inappropriate venues, including hacker forums. While we’ve since seen measures to appear legitimate, like professing ethical IP sourcing and customer screening (KYC), we found them to lack grounding and actual enforcement.

Why We Investigated Smartproxy.org?

We’d had suspicions about Smartproxy.org’s ties to IPIDEA for a while. The provider shares several characteristics with the cluster of IPIDEA-affiliated brands mentioned earlier.

Unlimited Pricing

IPIDEA-related brands made the unlimited residential proxy model their signature. Smartproxy.org offers the same structure – unmetered bandwidth and requests for a fixed hourly, daily, or monthly fee.

The economics of offering genuinely sourced residential proxies at these price points have always been questionable. Running a large-scale residential proxy network requires paying for bandwidth, maintaining infrastructure, and compensating the owners of the devices that serve as exit nodes.

Unlimited plans at low price points only make sense if the cost of IP sourcing is close to zero, which, in IPIDEA’s case, it effectively was, since devices were recruited through SDKs embedded in apps without proper disclosure.

Limited Transparency

Many IPIDEA-related brands shared a pattern of providing little verifiable information about their corporate structure, IP sourcing methods, or compliance practices. Smartproxy.org fits this description. The website offers minimal detail about the company behind the service, and there is no publicly available information about how its residential IPs are sourced or what consent mechanisms are in place.

Market Position

Like other IPIDEA-adjacent providers, Smartproxy.org positions itself as a budget alternative to established proxy services while claiming large residential IP pools. This mirrors the playbook that IPIDEA-linked brands used to gain market share between 2022 and 2024.

How We Tested It?

The methodology was deliberately simple. We wanted something anyone with a subscription could replicate using basic tooling.

Step #1: Purchase a Plan

We bought a weekly unlimited residential proxy plan from Smartproxy.org. No special configuration, no enterprise-tier access, just a standard subscription available to any customer.

Step #2: Generate Traffic

Over the course of one week in February 2026, we routed approximately 7 million requests through Smartproxy.org’s proxy infrastructure. Each request was directed to an IP-checking endpoint, which recorded the exit IP address. In principle, any IP checker would work for this purpose. You’re simply logging which IP address each request originates from.

Step #3: Collect and Deduplicate IPs

Out of nearly 7 million total requests, the test returned 2,023,029 unique IP addresses. Here’s the breakdown:

Metric	Value
Total requests	~6,963,000
Successful requests	~6,285,000
Success rate	90.25%
Unique IPs (total)	2,023,029
Unique IPv4	2,019,488
Unique IPv6	3,541

The pool was overwhelmingly IPv4. The 90.25% success rate means roughly one in ten requests failed or timed out, a separate performance conversation, but for this analysis, the pool composition mattered more than speed or reliability.

Step #4: Compare against the IPIDEA Reference Dataset

For the comparison, we used a publicly available IPIDEA IP dataset published by Antoine Vastel, a cybersecurity researcher who runs the proxy detection database. On January 29, 2026, the day after Google’s announcement, Vastel released two downloadable CSVs containing verified exit IPs whose last observed provider was part of the IPIDEA network.

The dataset covered a 30-day window ending January 29, 2026, and contained 16,192,293 unique IP addresses. Critically, every IP in the dataset was verified, meaning Vastel actually routed traffic through it and confirmed it worked as an active proxy endpoint. This is not a speculative or self-reported list; it’s operational data.

The IPs were broken down by the last observed provider as follows:

IPIDEA-linked provider	Unique IPs
PyProxy	13,429,821
PIA S5 Proxy	2,213,174
Luna Proxy	549,300
Total	16,192,295

All three providers, PyProxy, PIA S5 Proxy, and Luna Proxy, are brands Google explicitly tied to the IPIDEA network.

Time Gap Between Tests

Our test ran in early February 2026, a few weeks after Vastel’s dataset coverage ended on January 29. This gap matters in both directions. Some IPs active on IPIDEA’s network in January may have rotated out of Smartproxy.org’s pool by the time we tested. Conversely, new IPs may have entered both pools in the interim. If anything, the gap likely understates the true overlap, since residential IP churn means fewer matches over time, not more.

Results

The comparison between Smartproxy.org’s IP pool and Vastel’s IPIDEA dataset produced the following:

Metric	Value
Smartproxy.org unique IPs	2,023,029
IPIDEA dataset unique IPs	16,192,293
IPs present in both pools	773,087
Overlap as a share of Smartproxy.org	38.21%
Overlap as a share of IPIDEA	4.77%

In plain terms, more than one in three IP addresses we encountered through Smartproxy.org also appeared in the IPIDEA dataset.

The 4.77% figure from the other direction is less dramatic, but that’s expected. IPIDEA’s dataset is roughly eight times larger and covers a longer observation period (30 days vs. our one week). A smaller pool will always represent a modest fraction of a much bigger one, even if the smaller pool draws heavily from it.

Interpreting the Overlap

A 38% overlap between two residential proxy pools is substantial, but IP overlap data requires careful interpretation. Residential IPs are dynamic – they rotate between users, get reassigned by ISPs, and can appear across multiple proxy networks over time. Some degree of overlap between any two large residential pools is natural. The question is how much is too much to be coincidental.

What the data does establish is that IPIDEA is very likely either a direct upstream supplier or a significant source within Smartproxy.org’s proxy network. The overlap is too large and the timing too tight for the explanation to be a coincidence.

IP Retention as a Baseline

To understand the significance of 38%, it helps to understand how quickly residential IP pools change. According to IPinfo, monthly IPv4 retention in residential proxy pools sits at around 40%. That means roughly 40% of the IPs you see in a given pool this month will still be there next month, while the rest rotate out and get replaced. Residential pools are fluid but not entirely unpredictable. They maintain a recognizable core of addresses over short time windows.

If two pools were completely independent, sourcing IPs from different SDKs, different apps, different device populations across different geographies, you’d expect the overlap to reflect the general background rate of IP reuse across the internet.

For large pools of this size, that background rate would be significantly lower than 38%. The internet has over 4 billion routable IPv4 addresses – even accounting for the fact that residential proxy IPs cluster in certain ASNs and regions, a coincidental overlap of 38% between two genuinely independent pools would be extremely unlikely.

Why the Timing Matters

Our test ran only a few weeks after the IPIDEA database’s coverage window closed. Given the natural churn in residential pools, that’s a short interval. If we had tested six months later, a high overlap could more plausibly be attributed to ordinary IP recycling across the internet – the same addresses eventually showing up on different networks simply because ISPs reassign them. At a few weeks’ distance, the connection is much harder to explain away. The IPs were active on IPIDEA’s network in January and active on Smartproxy.org’s network in early February.

What the Pool Size Ratio Tells Us

Smartproxy.org’s pool (two million unique IPs over one week) is a fraction of IPIDEA’s (sixteen million over one month). If Smartproxy.org were one of many customers reselling IPIDEA’s infrastructure or sourcing a portion of its IPs from the same upstream network, this is roughly the ratio you’d expect. A downstream reseller wouldn’t need access to the entire IPIDEA pool – it would draw from a subset.

The 38% overlap from Smartproxy.org’s perspective, combined with the 4.77% from IPIDEA’s, fits the pattern of a smaller provider accessing a portion of a large wholesale network. From IPIDEA’s side, Smartproxy.org would be one customer among many; from Smartproxy.org’s side, IPIDEA would be a major (possibly the primary) source.

The Broader Pattern: IP Fingerprinting as a Detection Method

Smartproxy.org is unlikely to be the only provider sourcing from IPIDEA’s infrastructure. Google’s action disrupted the most visible IPIDEA brands – LunaProxy, ABCProxy, and others had their websites taken down, and their proxy networks were degraded. But the underlying infrastructure doesn’t necessarily disappear when a domain goes offline.

Proxy networks are built on IP sourcing arrangements, SDKs embedded in applications, and device-level software. Changing a brand name or registering a new domain does nothing to alter the underlying pool of devices that serve as exit nodes. The apps that recruited those devices are still installed, the SDKs are still running, and the IPs continue to route traffic.

This is exactly what makes IP overlap analysis useful as a detection method. Brand names can be changed, corporate structures reshuffled, and websites relaunched overnight. But the actual IPs that exist from a network function as a fingerprint. If two providers consistently share a large portion of their exit addresses within a narrow time window, the simplest explanation is shared infrastructure – regardless of what the marketing materials say.

The approach has limitations. Some natural overlap between large residential pools is expected, and the method doesn’t reveal the nature of the commercial relationship (reselling, sub-licensing, or direct infrastructure control). But for a first-pass investigation into whether a provider draws from a known network, IP comparison is fast, replicable, and hard to fake.

What This Means for Smartproxy.org Users?

If Smartproxy.org sources a significant share of its IPs from IPIDEA-related infrastructure, it raises several practical concerns for its users.

Association with Compromised Infrastructure

IPIDEA’s network wasn’t just large – it was actively implicated in botnet operations and cybercrime. Traffic routed through these IPs could be sharing exit nodes with malicious actors, which increases the risk of IP reputation issues, blacklisting, and guilt by association. For use cases like ad verification, SEO monitoring, or market research, a poor IP reputation directly degrades results. Anti-bot systems maintained by major websites track and score IP addresses – if a proxy IP is flagged for malicious activity in one context, requests from the same IP in a different context are more likely to be challenged or blocked.

The Device Sourcing Question

A core part of Google’s case against IPIDEA was how it recruited devices into its network. SDKs embedded in free VPNs and utility apps turned phones and computers into proxy exit nodes – often without meaningful consent from their owners. Google explicitly noted that many of the apps involved did not clearly disclose that installing them would enroll the user’s device in a proxy network.

If Smartproxy.org’s IPs come from the same device pool, the same ethical concerns around consent and transparency apply to its users. In practical terms, you may be routing your traffic through devices whose owners never agreed to serve as proxies.

Service Durability

Google’s disruption of IPIDEA is ongoing, not a one-time event. The domain takedowns, intelligence sharing, and Play Store enforcement are designed to degrade the network over time progressively. If Smartproxy.org depends on IPIDEA’s infrastructure upstream, further enforcement actions could shrink its IP pool without warning. Users relying on the network for business-critical scraping or data collection tasks would face an operational risk that they might not anticipate.

Name Confusion with Decodo (formerly Smartproxy)

Smartproxy.org has no relation to Smartproxy, the well-known Lithuanian proxy provider that rebranded to Decodo in 2025. Despite the nearly identical name, these are two completely separate companies.

Decodo (formerly Smartproxy) operates out of Vilnius, has been in the market since 2018, and is one of the most established residential proxy providers in the industry. Smartproxy.org is a much more obscure operation. The name similarity can and likely does cause confusion among buyers, which raises its own set of questions about the provider’s intent. But for this investigation, we’re focused solely on Smartproxy.org’s IP pool and its relationship to IPIDEA. Decodo has also shared their experience fighting lookalike brands, including Smartproxy.org, and advocated for ethicality and transparency in the industry.

Google’s disruption of IPIDEA in January gave us a concrete reason to test these suspicions. If Smartproxy.org were drawing from the same IP pool, the overlap should be visible in the data.

Bottom Line

We set out to test a long-standing suspicion about Smartproxy.org’s IP sourcing, and the data support it. 38% of Smartproxy.org’s residential IPs overlap with a verified IPIDEA dataset – well above what independent networks would share, and measured within a narrow time window that makes coincidental IP recycling an unlikely explanation.

The most straightforward interpretation – Smartproxy.org either resells IPIDEA’s infrastructure directly or sources a substantial portion of its IPs from the same network Google took action against in January 2026.

Smartproxy.org continues to operate at the time of writing. We’ll monitor the situation and update this report as new information becomes available.

Adam Dubois

Proxy geek and developer.

March 31, 2026

The post We Tested Smartproxy.org’s IP Pool: Here’s What We Found appeared first on Proxyway.

Search APIs in 2026: Overview & Benchmarks

Adam Dubois — Tue, 24 Feb 2026 08:11:26 +0000

Research

The rising tide of AI has lifted all web scraping boats, but retrieval-augmented generation and AI agents have made SERP APIs (lately called Search APIs) in particularly high demand. Well-financed and ambitious companies are reshaping what was an established category of web scraping tools.

Our report aims to capture this shift: we explore the status quo in early 2026, compare the three types of search APIs, and benchmark 15 providers to see how they perform.

Key Findings

The market has segmented into full search APIs used for human-oriented purposes, indices aimed at serving AI, and fast APIs that bridge the first two categories.
The main difference between fast APIs and indices is that the former return real-time data but rarely extract the pages found through search.
Google has been working hard to stop web scraping ever since January 2025, constantly changing access mechanisms and even taking scraper companies to court.
In our speed test, the fastest indices like Exa and Perplexity fetched results in under 0.4 seconds with little variance. Oxylabs and Serper were the fastest real-time APIs, reaching a median response time of 0.6-0.7 seconds.
Real-time APIs cost less and scale better, but they require other tools to carry out search and page extraction. Indices converge at around $5 per 1,000 requests.

SERP Scraping Before 2024

Before LLMs, search engine scraping had relatively defined problem areas. It serviced SEO agencies and tools, ad verifiers, brand protection companies, and other businesses that relied on search data.

The aim of SERP APIs was to reliably fetch as much of the search engine page as they could. At first they included only organic results, ads, people-also-ask boxes, and related queries. With time, their reach expanded to knowledge graphs, snack maps, local results, shopping boxes, search verticals (like Hotels), and all the other doodads that grew to populate the search page.

We always found it curious that it was only SERP APIs which formed their own subset of web scraping, with brands dedicated exclusively to search engines. Their main target was, of course, Google, followed by Bing, Yandex, and sometimes the search pages of e-commerce websites like Amazon.

In any case, consumers of SERP data were looking to understand and manipulate how search engines interacted with humans, within the boundaries of the platform they worked with.

Post-AI Search APIs

By 2023, ChatGPT had been available for a year. It became clear that training datasets weren’t enough: to become truly relevant, chatbots needed a way to access the world’s knowledge as it was generated. Retrieval-augmented generation (RAG) took off to free language models from the constraint of knowledge cut-off dates.

This gave a huge new impetus for accessing search engines as both the source of knowledge and the guide for reaching it. On the promise of making search more accessible for AI, LinkUp raised $13M, Tavily got $25M, Exa received $85M, Parallel secured $100M, while You pivoted from a consumer search engine with a similarly impressive $100M investment round.

Web search for AI has very different requirements compared to human consumers:

For one, data retrieval needs to be fast – way faster than the 4+ seconds of a typical SERP API response. This has prompted companies to create their own pre-scraped indices: very expensive but made possible by venture capital funding.
Because an index isn’t real-time and thus unfaithful to Google, APIs have to re-rank results based on their own criteria. For example, Tavily assigns each page it surfaces with a relevance score.
Furthermore, language models aren’t content with just the search results page – they also need to know the contents of the results. The response can include the full output, highlights that fit context windows, or straight up LLM-written answers to the query.
And finally, AI doesn’t need the full SERP. Ads, widgets, and mobile layouts serve little use in a knowledge lookup, except for certain edge case scenarios.

In 2025, the Silicon Valley hivemind converged around a new use case – AI agents. It shifted SERP APIs even further away from their conceptual roots. Their main purpose became to rebuild web search for a new consumer, which was no longer human but rather AI. In other words, companies stopped wanting to work within a search engine – instead, they hoped to replace existing search interfaces with their own (not without consequences, but more on that later).

While search APIs were making headlines, another product category quietly sprung up, the so-called fast or light SERP APIs. They sacrifice most unnecessary page elements in exchange for pure speed, often reaching less than a second per response. The original and most successful fast SERP API has been Serper; established providers like Oxylabs, Bright Data, and SerpApi have since introduced fast versions of their own.

Google’s Response

Google has always been one of the main web scraping targets, but the search giant seemingly didn’t pay it much mind. Sure, datacenter proxy servers stopped working well at some point, and badly configured spiders would receive their share of CAPTCHAs. But before 2025, the platform never posed a serious challenge like LinkedIn or some e-commerce stores.

How come? Blocking scrapers while being the largest web scraper doesn’t lead to great optics. But in a more practical sense, prior business models never seriously threatened Google’s ad empire. On the other hand, AI agents don’t care about viewing, and they certainly don’t care about buying ads. By the time Bright Data brought together a bunch of search APIs to talk about upending Google in November 2025, the straw had already broken the camel’s back.

In mid-January 2025, Google disrupted every single web scraper (and a good deal of accessibility tools) by implementing SearchGuard – a JavaScript-dependent bot protection system. Though presenting few changes to the regular user, this move increased the scraping latency and costs by an order of magnitude. Several days later, things more or less stabilized, but a die had been cast.

Google’s second hit came in September when it removed the &num=100 parameter. The omission aimed explicitly at web scrapers, primarily AI companies. Among those affected the most were various SEO tools like Ahrefs, which relied on the extended SERP for visibility into rankings and keywords. Half a year in, their functionality remains more limited than before the change.

Alongside these seismic events, Google relentlessly works to make scraping the search engine harder. It hired dedicated anti-bot engineers to persistently break unblocking strategies, page parsing schemas, and stitched-together &num=100 substitutes.

Discontent with engineering efforts alone, Google brought the matter to court, suing SerpApi in December 2025. The case hinged on two arguments: circumvention of Google’s SearchGuard technology, incurring a deadweight loss; and unauthorized scraping of licensed content like the Knowledge Panel, which allegedly infringed on the Digital Millennium Copyright Act. The claimant seeks $200-$2,500 for each scraped page, which everyone knows is impossible to satisfy.

There are two important keywords in the lawsuit: deadweight loss, which really means ignored our ads, and the scale of scraping: based on Google’s data, SerpApi increased the number of requests by 25,000% over two years. Bright Data’s revenue data, which shows over 700% increase for its SERP API, corroborates this pattern.

While SerpApi and Google are fighting it out in the court, the legal teams of other providers watch closely, while their engineers shovel coal into the web scraping engine as fast as they can. Such is the price of opportunity.

Three Kinds of Google Scrapers

We’ve established that the market has splintered into three kinds of search engine scrapers: full, fast, and index-based. How do they compare? We’ve made a table to showcase their main features and strengths.

In brief, each chooses different trade-offs in the triangle of accuracy, speed, and completeness. If we compare two directly competing options, fast APIs and indices:

Fast APIs give complete and fresh access to the SERP but not the pages it contains, requiring additional web scraping tools.
Indices surface relevant pages and retrieve their contents very quickly, but the corpus of data risks being stale or incomplete.

	Full APIs	Fast APIs	Indices
Output	All SERP elements, local search, ads	Crucial SERP elements (organic search, news)	Listed + extracted relevant pages
Retrieval	Real-time or very recent	Real-time or very recent	Index first, real-time as a fallback
Representation of the SERP	Faithful	Faithful	Unfaithful
Primary user	Human	AI, then human	AI
Use cases	Marketing agencies, SEO tools	Real-time data needs, RAG, deep research	RAG, deep research, AI agents
Strengths	Full SERP coverage, fresh output	Very fast, inexpensive, fresh output	Fastest, pre-processed output for AI
Weaknesses	Slow, limited suitability for AI	Limited SERP coverage, needs other tools for search & extract functionality	Limited SERP coverage, potentially stale data, expensive

Landscape of Web Search APIs

Demand for search engine APIs is booming, and the market reflects that in the number of competitors. While we couldn’t list every one of them (most notably, the list misses Firecrawl and Parallel), we bought and benchmarked 15 services covering all three scraper types. Many were founded within the last few years.

	Full API	Fast API	Index
Brave	❌	❌	✅
Bright Data	✅	✅	❌
DataForSEO	✅	❌	❌
Exa	❌	❌	✅ (+live retrieval)
Jina	❌	❌	✅ (+live retrieval)
LinkUp	❌	❌	✅
Oxylabs	✅	✅	❌
Perplexity	❌	❌	✅
ScrapingBee	✅	✅	❌
Scrapingdog	✅	✅	❌
SearchApi	✅	✅	❌
SerpApi	✅	✅	❌
Serper	❌	✅	❌
Tavily	❌	❌	✅ (+live retrieval)
You	❌	❌	✅ (+live retrieval)

Search API Speed Test

AI use cases put a big premium on speed, especially when they function in latency-sensitive formats like audio conversations. Another important criterion is data relevance; but, as we’ll be comparing both direct and indirect representations of the search page, we have no good way to evaluate it.

To benchmark the search APIs for speed, we ran three tests on different days in mid-January (with the exception of Exa – Instant, launched in mid-February), each involving 2,000 requests at the rate of one per second.

We made randomly-generated queries, such as how tall are dogs. With indices, we chose their fastest mode where possible, which may impact result quality in production scenarios. Our server was located in the East Coast US.

At this scale, success rate was generally not a concern, so we omitted this metric. A word of caution for those with larger needs: scraping Google in real time is way less stable than it used to be.

As expected, most index-type APIs were extremely fast. Perplexity, Exa, and Tavily all had effectively identical P50 response times, while You and Jina returned data around twice slower. Another distinguishing feature of indices was the small variance between the median and P95th result.

According to Exa, any search API that wraps Google has a minimum 700 ms P50. At our scale, we found this to be untrue: Oxylabs returned the SERP page nearly 20% faster, while Serper hovered around the 700 ms mark. Compared to indices, live APIs had a much bigger spread in latency. In four cases, the P95 response time exceeded five seconds.

Calculating the Costs

The differences between search API types make them surprisingly hard to compare. A closer investigation reveals several tendencies:

Indices tend to rely on usage-based models: the low entry price makes them easy to pick up, but it also makes their public rates inflexible. The prices converge at around $5 per 1,000 results.
The main price modifier is the processing needed to return a response.
Real-time APIs rely on subscription plans much more than indices, which means that they require more initial commitment but scale better.
However, they also need to factor in website extraction costs. Harder targets can charge multiples of the base request price.

	Model	Starting CPM	$1,000 CPM	Modifiers	Trial results
Brave	PAYG	$3	$3	Usage conditions	2,000/mo
Bright Data	PAYG, sub	$1.5 (+$1.5 for extraction)	$1.1 (+$1.1 for extraction)	Premium domains (x2)	1,000
DataForSEO	PAYG	$0.6	$0.6	Faster queue, live (x2, x3.3)	1,000
Exa	PAYG	$5	$5	Search depth (x3)	670-2,000
LinkUp	PAYG	$5	$5	Search depth (x10)	100-1,000
Oxylabs (Regular API)	Sub	$1 (+$1.15 for extraction)	$0.6 (+$0.75 for extraction)	JS rendering (x1.35)	2,000
Perplexity	PAYG	$5	$5	❌	¯\_(ツ)_/¯
ScrapingBee	Sub	$1.96 (+$0.2 for extraction)	$0.71 (+$0.07 for extraction)	API type (x2), extraction difficulty (up to x75)	100
Scrapingdog	Sub	$1 (+$0.2 for extraction)	$0.25 (+$0.05 for extraction)	API type (x2), extraction difficulty (up to x25)	100-200
SearchApi	Sub	$4	$1.8	Response speed (x2)	100
SerpApi	Sub	$25	$7.25	Response speed (x2, x4)	250/mo
Serper	PAYG	$1 (+$1 for extraction)	$0.75 (+$0.75 for extraction)	❌	2,500
Tavily	PAYG, sub	$7.5	Custom	Search depth (x2)	62-125/mo
You	PAYG	$6.25	$6.25	# of results (x1.28)	12.5-16k

Let’s try graphing the price per 1,000 search & extract requests. This will better reflect the scaling – but be aware that we’re not taking into account any modifiers.

Real-time APIs nearly always come out cheaper. However, they require more work to achieve the same results as an index.

Conclusion

Thanks for reading our report! If you have any questions or comments, feel free to reach out through info at proxyway dot com or our Discord server.

Adam Dubois

Proxy geek and developer.

March 4, 2026

The post Search APIs in 2026: Overview & Benchmarks appeared first on Proxyway.

Internet Sharing SDKs: a Closer Look at the Emerging App Monetization Method

David Barnes — Wed, 17 Dec 2025 14:28:35 +0000

Research

Once, sharing your internet meant handing over the Wi-Fi password. These days, it might mean turning a slice of unused bandwidth into passive revenue. The idea sounds promising, but it also raises fair questions. What’s really being shared? Who’s profiting? And most importantly, is it safe? That’s why we decided to chime in and look at how internet sharing SDKs are reshaping app monetization.

What Is an Internet Sharing SDK

An internet sharing SDK (Software Development Kit) is a lightweight component integrated into mobile, desktop, TV apps, or even browser extensions, letting users voluntarily share small portions of their unused internet connection with approved third parties. In return, developers earn passive income – without showing more ads, selling upgrades, or begging for subscriptions.

Instead of routing traffic through traditional data center servers – which can be flagged or blocked due to their non-residential nature – these SDKs rely on real user devices. The connection is opt-in, and the user’s IP address becomes part of a larger distributed network used by businesses for legitimate use cases.

Here’s how it typically works:

The SDK is embedded directly into the app, just like any other monetization or analytics component.
Users opt in voluntarily, usually during onboarding or when unlocking features like premium access, rewards, or ad-free experiences. In GDPR-compliant regions, this consent must be explicit and informed.
A certain amount of approved traffic is routed through the user’s device, borrowing their IP address. Effectively, the user turns into a proxy.
All traffic is anonymized, encrypted, and used by businesses for tasks like SEO monitoring, ad fraud detection, competitive analysis, and market research.
Developers earn passive income based on geography and usage volume, often calculated using CPM-style payouts. Top rates range from ~$0.001–$0.03 per user per day or up to $30 per 1,000 Daily Active Users, depending on region and provider.*

*These figures reflect published rates as of 2024-2025; actual developer earnings will vary by app usage and user base demographics.

Where Are the Internet Sharing SDKs Used?

Utilities like Free Snipping Tool offer users premium features in exchange for opting into SDK. The integration is transparent and opt-out is available. Media and Streaming Apps, such as Megacubo, bundle SDK to unlock extras like screen recording – again, only after user opt-in.

Crowdfunding Platforms like CareBuzz leverage SDK to convert shared bandwidth into charitable donations, turning the model into a non-cash incentive. Smart TV and Puzzle Games using SDK show how passive monetization can boost ARPDAU (average revenue per daily active user) in long-session, low-engagement formats.

Why Developers Are Considering Internet Sharing SDKs?

Internet sharing SDKs don’t sell clicks, show banners, or unlock loot boxes. So why are they showing up in monetization stacks? The short answer: they shift the model. Instead of charging for attention or conversion, they monetize presence.

It’s no surprise the model first gained traction in VPNs, utility tools, and desktop software – categories where users are already familiar with trading idle resources for perks. But there’s more to it:

Monetizing passive users. Clicking on ads or making a purchase is not what most users do, but they still use app resources. Internet sharing SDKs provide a way to generate income from that passive usage. Rather than increasing engagement, they offer value in the background.
Low implementation overhead. Integration time is short, typically 1–2 days, and there’s little need for ongoing optimization. Unlike ads, which require testing placements, formats, and timing, or IAPs that demand balancing user incentives, SDK monetization tends to be plug-and-observe.
Good earning potential. Some SDK providers offer revenue based on CPM-like models – meaning developers are paid a fixed amount per 1,000 active users or sessions, similar to how ad impressions are monetized. Rates can range from $1.50 to $6.00 per 1,000 daily active users, or $10–$70 per 1,000 unique IPs per month in high-demand regions. These are top-end figures – real-world earnings depend heavily on geography, user engagement, and session quality.
Could be part of a hybrid monetization model. Unlike ads or subscriptions, internet sharing SDKs don’t cannibalize existing revenue channels. They operate in parallel, making them a common choice for hybrid monetization strategies.
Works across device categories. These SDKs can be integrated into Android, iOS, Mac, Windows software, Smart TV apps – categories where other monetization models might underperform.

That doesn’t make this model superior, or right for everyone. But it helps explain why, over the past few years, more developers have begun quietly testing it – especially in apps with large user bases and limited direct revenue per user.

Back to the Roots: From Proxies to Monetization

To understand the need for bandwidth sharing SDKs, you may want to learn where this monetization method really comes from. It all started with good old proxies. Back in the 2000s, if you wanted to dodge a geo-block or test a website from another country, using proxy servers was often the best way.

By the early 2010s, companies increasingly realized proxies weren’t just for anonymous browsing – they could be used at scale to enable business use cases. By routing traffic through actual residential connections, businesses could test how ads are rendered across different countries, audit local search results, or simply overcome toughening website restrictions. This shift gave rise to residential proxies as a commercial tool.

Running these networks at a global scale was expensive. You needed thousands of residential devices, which couldn’t just be bought in bulk like datacenter-based IPs. Around 2013 Hola VPN appeared with what looked like a clever shortcut: it created a peer-to-peer network where users’ devices acted as access points for each other. In exchange, they got free VPN access – unknowingly powering the network themselves.

All was smooth until 2015, when it was discovered that Hola was also reselling its users’ bandwidth through a commercial branch, Luminati Networks, without user knowledge. The fallout prompted one of the first wake-up calls around user transparency. At the same time, new privacy laws were reshaping how the data was being used.

The EU’s General Data Protection Regulation (GDPR) took effect in 2018, requiring organisations to obtain explicit user consent and improve transparency. In the US, California’s Consumer Privacy Act (CCPA) followed soon after, pushing companies toward the same standards nationwide. The message was clear: earn user trust or lose access.

That shift cleared the path for user-first platforms like Honeygain or Pawns.app, where people opted in to share some of their unused bandwidth and got paid for it. It was the same concept that once caused outrage – this time rebuilt on transparency and consent. Internet sharing had finally gone legit.

Not long after, the model branched out, turning its focus to developers and app publishers. Instead of building entire apps around bandwidth sharing, some companies started packaging the functionality into modular SDKs. The underlying principle remains the same, but SDK creators now give much more attention to ensuring more transparency and safer use. While bandwidth-sharing SDKs remain low key – and sometimes carrying a stigma from the earlier days of misuse – they’re becoming a legitimate alternative to dominant monetization methods.

So before evaluating whether that stack holds up, it’s worth revisiting how the current monetization pillars (ads, IAPs, subscriptions) actually perform today.

How Internet Sharing SDKs Compare to Prevalent Monetization Methods

Most app developers lean on the same three monetization methods: ads, in-app purchases (IAPs), and subscriptions. They’re easy to implement, scalable across geographies and long time. However, they’re also fraught with issues that negatively affect user experience on the app.

Advertising is the classic default: show a banner, interstitial, or rewarded video, and you get paid when users see or click. The payout is usually measured by eCPM (effective cost per thousand impressions). However, since Apple’s App Tracking Transparency (ATT) policy rolled out in 2021, iOS ad performance has taken a serious hit. By some estimates, eCPMs dropped nearly 50% as advertisers lost access to user-level identifiers, such as device IDs and behavioral profiles, that enabled personalized targeting.

Ads still work, especially in large-scale apps and Tier-1 markets. But between shrinking margins, seasonality, and shifts in targeting accuracy, ad revenue has become increasingly volatile – making it harder to forecast or rely on consistently. Relying on this model alone is like balancing on a one-legged stool – possible but bound to topple.

In-app purchases (IAPs) offer direct monetization from engaged users. For example, in games, this means buying extra lives, virtual currency, or cosmetic upgrades. The revenue can be massive – according to Sensor Tower, IAPs generated $150 billion in 2024. That makes it the highest-earning app monetization method by revenue, even if it relies on a small pool of spenders.

On the other hand, it has been found that only around 5% of users spend money, and conversion rates in emerging markets are even lower. That means most of your audience contributes nothing financially, even if they open the app daily.

Meanwhile, internet sharing SDKs entered the game by making it possible to earn from the 95% who never hit “buy” – not by converting them, but by letting their devices do the contributing. The opt-in model preserves user choice but broadens the revenue base beyond power users.

Last but not least, subscriptions offer recurring access in exchange for monthly or yearly fees, a model popularized by media, productivity, and fitness apps. The data from Statista reveals that subscription-based apps made approximately $45.6 billion in 2023, a sizable share of the market.

But retention remains the Achilles’ heel. According to the RevenueCat report, nearly 30% of annual subscriptions are canceled within the first month, and over 40% of users say they’re tired of managing multiple app subscriptions. The fatigue and hesitance to pay for casual use has made long-term retention increasingly difficult.

As we mentioned previously, internet sharing SDKs offer a less intrusive option. Users can opt in once, change their minds later, and still contribute in the meantime.

Feature	Internet Sharing SDKs	In-App Purchases (IAPs)	Subscriptions	Ads
Revenue model	Bandwidth/active users	Purchases by users	Recurring payments	Impressions / clicks
Monethisation method	Opt-in once	Active spending	Long-term app usage	Engagement with the app
Ongoing management	Low (monitoring opt-in only)	High (pricing and item balancing)	Medium (churn, renewal flows)	High (A/B testing, placements)
Compliance requirements	High (GDPR, opt-in, traffic purpose clarity)	Financial + privacy compliance	Platform billing and opt-ins	Moderate (GDPR, IDFA use)
User experience impact	Low (runs in background)	Low to moderate (Impacts limited to gating features)	High (subscription prompts)	Moderate to High (banners, pop-ups)
Potential audience monetization	Up to 100% of consenting users monetized	~2–5% of users	~70% retention after 1st month	~95–99% of users exposed (region-dependent)
Devices	Mobile, Desktop, Smart TV, Browser Ext.	Mobile, possible on Desktop, TV	Mobile, possible on Desktop, TV	Mobile, Web

Possible Risks and Considerations of Internet Sharing SDKs

No monetization model is immune to scrutiny – and Internet sharing SDKs are no exception. While the model offers a compelling alternative to ads and IAPs, it hinges on the assumption that users understand and accept what’s happening in the background. That assumption doesn’t always hold.

There are critical factors worth evaluating before jumping in. Some are technical, some reputational, and some simply structural. Here’s what any team considering integration should consider.

User consent? Bandwidth sharing must be clearly disclosed, and users must explicitly agree before any traffic is routed. Consent can be collected at install, onboarding, or later, but nothing should activate until it’s given. Apple, Google, GDPR, and CCPA all emphasize transparency and informed choice, though implementation standards vary slightly. If the opt-in is buried, vague, or misleading, both compliance and user trust are at risk.
Isn’t this basically a botnet? At a glance, routing third-party traffic through thousands of end-user devices looks suspiciously like a botnet. The difference? Consent and transparency. Botnets operate without user knowledge or control, while internet sharing SDKs require clear opt-in, restrict traffic to non-invasive use cases, and comply with platform and legal standards. But the optics still matter, and even legitimate networks can raise alarms if poorly explained.
Does it collect personal data? These SDKs aren’t designed to harvest contact lists, GPS, or identifiers. The primary asset shared is bandwidth and an IP address, and traffic is typically anonymized. Still, devs should verify that no unnecessary permissions are being requested, any data claims should match actual behavior.
What’s the risk of slowing down the device? Most SDKs are designed to be lightweight, routing traffic only during idle periods and throttling usage to minimize disruption. But “lightweight” doesn’t mean invisible. Under real-world conditions, like low bandwidth, roaming, or battery-saving modes, even minor background activity can affect responsiveness or battery life. Though testing across edge cases is essential.
Lack of transparency around buyers. SDK providers often list “brand protection,” “SEO monitoring,” or “market insights” as traffic destinations. These are categories, not clients. Developers rarely know who is actually routing traffic through their users’ devices, which leaves room for misuse or reputational blowback. What would establish transparency is asking for clarity on buyer vetting processes, resale restrictions, and whether third-party auditing is in place. If these answers aren’t forthcoming, it’s worth reconsidering the integration.
Reputational fallout. Even if everything works as promised, perception matters. Users are sensitive to anything involving “background data” or “sharing internet.” If this isn’t communicated clearly (or if your app appears in headlines for the wrong reasons), expect uninstalls – and questions from partners. Developers might want to consider build opt-in flows that clearly state the value exchange, avoid burying terms in obscure clauses, and consider adding an always-available opt-out.
Lack of auditing into SDK apps. According to security reviews by organizations like Guardsquare and Synopsys, many app developers do not perform deep audits of third-party SDKs, even those that handle network traffic or permissions. One should take into consideration the risk of data exposure or misuse.

So, who’s actually behind these SDKs, and what do they offer? Let’s take a look at the major players shaping the space right now.

Key Providers

Honeygain SDK

Honeygain is one of the most recognizable players in the internet sharing field. Launched in 2019, it began as a standalone passive income app, paying users small rewards for sharing idle bandwidth. It later rolled out an SDK, allowing developers to embed the same monetization layer into their own apps.

Honeygain emphasizes transparent consent and says the SDK has minimal impact on performance. According to the company, traffic is only sold to business clients for use cases like “price comparison, brand protection, web monitoring”. It complies with GDPR and CCPA regulations and maintains transparency through clearly documented consent flows and integration metrics such as opt-in rates, bandwidth contributed, session duration, and estimated earnings per user

Honeygain offers a self-service SDK with accessible documentation and integration code available after registration. It provides more transparency than closed or invitation-only platforms by allowing developers to review and evaluate the integration before entering a formal partnership. Payouts are based on geography – with monthly estimates ranging from $100 to $400 per 1,000 daily active users (DAU) in top regions.

Notably, in 2022 the company signed an exclusive partnership with Oxylabs, a major company in the proxy server industry, providing user-sourced IPs to power one of the largest residential proxy pools globally.

Platforms supported: Android, Windows, macOS, Apple iOS, FireOS, LG WebOS, Flutter, Samsung Tizen, Unity,

Bright SDK (Bright Data)

Bright SDK comes from Bright Data (formerly Luminati), the same company that once drew criticism for reselling Hola VPN user bandwidth. Today, it positions itself as a compliance-first infrastructure provider, emphasizing KYC checks, transparent opt-in flows, and a focus on public web data access.

Unlike traffic-based models often used by bandwidth marketplaces – platforms where developers get paid based on the amount of user data transferred (usually per GB) – Bright SDK pays a flat daily rate per opted-in user.Bright SDK pays a flat monthly rate per opted-in user. Monthly payouts range from $100 to $350 per 1,000 IPs, depending on device and region

The SDK code isn’t publicly available: integration requires a partnership or enterprise agreement. It requires a formal integration process, which may limit the level of self-service analytics or visibility compared to open platforms like Honeygain. Reporting and metrics may instead be handled through direct partner communication.

In exchange, developers gain access to an established infrastructure with consistent demand from enterprise clients in various fields. That could translate into more stable and predictable earnings compared to smaller or less mature networks.

Bright SDK has partnered with some established companies such as Wizard Games, Daemon Tools, and Aeria Canada. While its legacy still raises eyebrows in some circles, it remains one of the most established players in the SDK monetization space.

Platforms supported: Android, Windows, macOS, Apple iOS, LG WebOS, Samsung Tizen, Huawei.

Infatica SDK

Infatica keeps a lower profile and focuses on enterprise partnerships rather than broad developer adoption. Infatica’s SDK isn’t openly distributed, developers must apply for access via direct contact rather than downloading it. The company pitches “peer-to-business” bandwidth resale but publishes little detail about developer dashboards or integration workflow.

In 2021, the company drew serious transparency and privacy critiques, when reports emerged of it flirting with browser extension developers to include the SDK with limited disclosure. The company claims to vet clients and uses threat detection partners like Bitdefender to scan and filter outbound traffic. Estimated payouts are usage-based and vary by volume and region.

Infatica targets apps with high daily active user (DAU) thresholds and persistent background activity – VPNs, desktop utilities, and smart TV platforms – where network sharing can run continuously and monetization benefits scale with retention.

Platforms supported: Android, macOS, Windows.

Conclusion

Internet sharing SDKs aren’t magic, but they’re not malware in disguise either. One can think of it like adding a basement rental to your app’s monetization house: it’s out of sight, earns on the side, and only works if you lay proper plumbing.

For developers, the pitch is simple: most users won’t pay, but their devices can still contribute. This model opens a new revenue stream that doesn’t compete with ads or IAPs – it complements them. And yes, the numbers can be meaningful if you have scale and the right geography.

These SDKs handle live network traffic and affect how your app behaves in the background – which means Apple, Google, and your users will care how they’re implemented. The best providers build in safeguards: encrypted traffic, public data only, verified clients. Still, it’s on developers to audit what they ship, explain what it does, and respect what users agree to.

Used responsibly, internet sharing can play a solid role in a hybrid monetization stack. Whether it belongs in your stack depends on how it fits your product, and how well you can explain it to your users.

Adam Dubois

Proxy geek and developer.

December 16, 2025

The post Internet Sharing SDKs: a Closer Look at the Emerging App Monetization Method appeared first on Proxyway.

Web Scraping API Report 2025

Adam Dubois — Wed, 03 Dec 2025 09:00:57 +0000

Research

Welcome to our annual report on web scraping infrastructure. This edition includes two parts. The first is technical: it benchmarks nearly a dozen popular web scraping APIs, aiming to determine their effectiveness and cost of unblocking protected websites. The second part is exploratory: there, we try to analyze how the AI boom has affected our industry.

All in all, it’s been a transformative (you could even say: hectic) year that’s introduced many new players, made the incumbents reconsider their position, and placed web data access at the center of a trillion dollar gold rush.

Summary

Our unblocking test included 11 scraping APIs and 15 protected websites.
- Four providers (Zyte, Decodo, Oxylabs, and ScrapingBee) managed to open them over 80% of the time, with Zyte leading the way.
- Zyte’s API was also among the fastest to return results; ScraperAPI, Decodo, and Oxylabs followed.
- Shein, G2, and Hyatt gave our participants the most trouble, with nearly a half failing to scrape them.
- Providers with variable pricing models (Zyte, Zenrows, ScrapingBee & ScraperAPI) had the cheapest base rates, but their top configurations charged up to 100x more!
- With our methodology, which involved JavaScript & premium unblocking mechanisms, Decodo cost the least.
By August, AI companies had received nearly as much funding in 2025 as non-AI startups. This has created a new generation of US-based web scraping infrastructure providers like Firecrawl and Browserbase.
Their novel approach to AI-based parsing, cloud browser management, open source tooling, and crawling strategies has forced entrenched companies to play catch-up.
Despite the narrative shifting to AI agents, LLM training continues to generate the bulk of web traffic, now focusing on multimodal data.
Growing attention to web scraping is bringing negative consequences, with Cloudflare and Google intensifying their determent efforts and the anti-bot industry experiencing fast growth.

State of Unblocking

Participants

Our research includes 11 major providers of web scraping APIs. Most gave us access after approaching them directly. We bought Firecrawl and ZenRows on our own.

-                                        Participant
+                                        Target audience
-                                        Apify
+                                        Small to medium customers
-                                        Crawlbase
+                                        Medium to large customers
-                                        Decodo
+                                        Small to medium customers
-                                        Firecrawl
+                                        Small to medium customers
-                                        NetNut
+                                        Enterprise
-                                        Nimble
+                                        Enterprise
-                                        Oxylabs
+                                        Enterprise
-                                        ScraperAPI
+                                        Small to large customers
-                                        ScrapingBee
+                                        Small to medium customers
-                                        ZenRows
+                                        Small to medium customers
-                                        Zyte
+                                        Medium to large customers

Methodology

We informed the participants about our general methodology but didn’t disclose the list of websites in advance. The intent was to avoid preemptive optimization.

We ran the bulk of our tests in October 2025.

We chose 15 targets for our benchmark. The selection criteria for these websites were 1) popularity and 2) being protected by major anti-bot vendors.

Target	Property	Bot protection
Allegro	Product	DataDome
Amazon	Product	In-house
ChatGPT	Logged out answer	Cloudflare / login wall
G2	Company reviews	DataDome
Google	SERP	In-house
Hyatt	Search result	Kasada
Immobilienscout24	Listing	Incapsula
Instagram	Profile	In-house
Leboncoin	Listing	DataDome
Lowe's	Product	Akamai
Nordstrom	Category	Shape
Shein	Product	In-house
Walmart	Product	PerimeterX + extra
YouTube	Transcript	In-house
Zillow	Listing	PerimeterX

The selection skews toward e-commerce websites, though it really represents quite a few verticals:

We collected ~6,000 unique URLs for each target website.
We then wrote a basic Python script that would call the APIs and retrieve the output. If an API had special endpoints for websites, we used those.
Our server was located in the US, and in most cases, we also set the geolocation of the APIs to the US.
We figured out a working configuration with each target, starting from base parameters. For example, some targets required JavaScript or enabling premium proxies to return valuable content.
We ran our script to fetch the 6,000 URLs twice: at 2 req/s and 10 req/s, with a timeout of 600s. This was enough to trigger bot protection systems – and sometimes hit the concurrency limits of the APIs.
We validated the results by looking at the response code, page size, and page title. In some cases (such as with ChatGPT), we also fetched a CSS selector.

Our ZenRows and Firecrawl plans had concurrency limits which caused some requests to fail at 10 req/s. This was mostly a problem with Zenrows, so the results may not represent its full capabilities.
Anti-bot systems may have different levels of protection based on the website (or even categories of the same website), so they may not show up in full force against scraping of public web data.
We tried opening the target websites at a similar time but couldn’t always guarantee perfect parity. While we maintain that an API should be able to access targets consistently, there’s always a chance to catch a scraper within its maintenance window between website updates.
In the same vein, our benchmarks occurred in a limited timeframe, so they merely give a snapshot of API performance.

Benchmark Results

We excluded runs that succeeded under 5% of the time. And if an API couldn’t unblock a target at 2 req/s reliably, we didn’t try running 10 req/s.

Just to give you some context, two requests per second translate to just over five million per month, while 10 requests per second add up to nearly 26M monthly requests.

Aggregated results
(Hover on a label to highlight, click to hide)

Just like last year, Zyte did an amazing job at unblocking tough websites. Decodo, Oxylabs, and ScrapingBee found themselves close behind.

ZenRows and ScraperAPI also proved to be strong all-rounders, while the rest could be considered specialist providers in our context. Firecrawl placed last, but this API is best suited for crawling the long-tail rather than individual protected targets.

Target breakdown – 2 requests/second
(Hover on a label to highlight, click to hide.)

The graph is somewhat hard to parse (you can hide websites by clicking on their labels), but it shows which targets tripped the APIs. For example, Shein was a problem for most, while G2 proved too much to handle for Oxylabs, Decodo, and ScrapingBee – but not ScraperAPI.

Target breakdown – 10 requests/second
(Hover on a label to highlight, click to hide.)

Increasing the scraping speed five-fold had an impact, but it was smaller than we expected. ZenRows suffered the most, likely due to hitting concurrency limits.

We noticed that the response times were similar whether the script ran at 2 or 10 req/s – and sometimes even lower at the faster rate! Therefore, we decided to only show the results of the fastest successful run.

The response times between the APIs varied by a lot. Firecrawl took an interesting strategy of failing fast rather than repeating requests internally. It does work, but it offloads the burden of retrying to the user.

All things considered, Zyte was probably the fastest, followed by ScraperAPI, Decodo, and Oxylabs. Notice that all these providers also did well at unblocking the targets.

At a median response time of 5.05s (we’re using median due to one significant outlier), none of the APIs were adequate for agentic Google access. Granted, we didn’t use the so-called light or fast endpoints but rather the generic option.

Number of consecutive results per hour
(Hover on a label to highlight, click to hide)

Let’s try a different approach. How many successful results could we get if we sent requests consecutively, keeping one open connection? Compared to the previous graph, this accounts for the time consumed by failed requests.

The leaders remained similar. ZenRows and Nimble benefited thanks to YouTube, NetNut fell by several ranks, and Firecrawl flipped from the first position to being last.

Apify’s system is distinct enough to warrant its own section.

First of all, it’s a marketplace of scrapers, many of which were made and are maintained by third parties.
Second, the scrapers create a Docker container for each run; you’re free to assign them as much memory and CPU power as you need.
Then, they take URLs in batches instead of looping through individual pages.
And finally, you can only suggest concurrency – the platform often ignores this setting and adjusts it dynamically.

Apify – successful runs

	Actor used	Results	Runtime	Comment
Allegro	Generic	7,442	19m 19s	~6.42 req/s
Amazon	Specialized	5,946	3h 30min	~0.47 req/s
G2	Specialized	5,926	14m 56s	~6.6 req/s
Google	Specialized	6,000	28m 33s	~3.5 req/s
Instagram	Specialized	5,956	22m 42s	~4.37 req/s
Walmart	Specialized	4993	14h 3m	~0.01 req/s
YouTube	Specialized	6,001	6h 12m	~0.27 req/s
Zillow	Specialized	5,998	12m 26s	~8 req/s

Some of Apify’s scrapers performed very well – G2 in particular. However, we had little control over their concurrency, resulting in wildly different runtimes.

For comparison, at 2 req/s, a run takes around 50 minutes, and at 10 req/s, it should end in roughly ten minutes.

Apify – failed runs

	Actor used	Results	Runtime	Comment
ChatGPT	Generic (custom Camoufox)	4584	21h 10m	Stopped early
Hyatt	Generic (custom Camoufox)	0	12m 4s	All requests failed
Immobilienscout	Generic (Playwright)	4,145	16h 6m	Stopped early
Leboncoin	Specialized	620	17h 44m	Stopped early
Lowe's	Specialized	89	1h 27m	Stopped after 4 retries
Nordstrom	Specialized	2,078	17h 32m	Stopped early
Shein	Generic	541	11m	Finished without retrying

Even when Apify had dedicated actors, they failed to work with around half of the targets. For example, the Shein scraper just looped through all URLs, got blocked, and didn’t even try to retry requests. Leboncoin, on the other hand, took way too long to complete, so we had to abort the run.

Hardest Targets to Unblock

	Avg. success rate at 2 req/s	APIs with over 80% failure
Shein	21.88%	5
G2	36.63%	5
Hyatt	43.75%	5
Lowes	52.57%	4
Instagram	59.54%	3
Nordstrom	61.97%	3
Leboncoin	63.83%	3
Allegro	66.98%	2
ChatGPT	71.04%	1
Immobilienscout	71.68%	1
YouTube	93.05%	0
Walmart	93.05%	0
Amazon	93.30%	0
Google	94.78%	0
Zillow	97.85%	0

At least in 2025, Shein is where web scraping dreams die. Very few APIs were able to open it at all, let alone reliably. In addition to having huge page sizes when downloading the HTML, this target is very well protected.

G2, which now uses DataDome, proved to be another headache. Guarded by Kasada, Hyatt was the third.

The most popular web scraping targets, Google and Amazon, painted another story. The main concern here wasn’t if the APIs will open these targets, but rather how fast. Had we tested data parsing capabilities, this would have been another point of relevance.

Cost to Run Our Tests

Assuming we made 14,000 requests per target (6,000 x 2 + 2,000 for leeway), how much did it cost us?

Hover on a label to highlight, click to filter.

Surprisingly, the providers that performed the best were also the most financially viable options. Decodo and Oxylabs benefitted from their relatively flat pricing structures which have seen at least one price cut since last year.

Zyte was among the cheapest providers with many of the targets (our 14,000 YouTube requests cost just $1!), but it also scaled by orders of magnitude when tougher protection mechanisms or headless browsing got involved. G2 and Hyatt alone consumed over a half of our budget.

While affordable in regular scenarios, providers with credit-based models like ScraperAPI and ScrapingBee are evidently not ideal for opening protected websites. Zenrows, however, has lower caps and thus compares better.

Apify is once again the odd duck. Its individual scrapers may have different pricing models (traffic/compute/requests), add-ons, and even subscription fees for access. For example, the Walmart scraper charges $30/mo just for getting to use it. As such, some of the scrapers – like G2 or Instagram – were very affordable, while Leboncoin ate over $60 for returning one tenth of the results.

Base and Most Expensive Rates

Our calculations are unlikely to represent real usage scenarios, as few users target exclusively protected websites. Here are the cheapest and costliest rates per 1,000 requests (CPM) at the arbitrary budget of $500.

Hover on a label to highlight, click to filter.

The graph reveals a huge gap with some of the APIs. For example, Zyte costs peanuts for basic targets but charges a premium for the toughest websites. So do ScraperAPI and ScrapingBee.

The latter two have relatively defined credit schemes, but Zyte makes it challenging to understand your expenses upfront. While there is a handy calculator, the provider segments websites into five buckets; the rates further react to volume discounts, rendering, and parsing options.

Compared to 2024, we’re seeing more segmentation in pricing models. Oxylabs has introduced separate rates for rendered requests, Amazon, and Google. Crawlbase has added Zyte-like difficulty tiers and even calculates volume discounts separately for each.

Narrative of the Year: AI

Artificial intelligence – more specifically, large language models – has been sucking all air out of the room for what, two years now? The web scraping industry experiences this especially acutely, as data (either datasets or real-time web content) underlies AI training and interactions with the web. Basically, we’re selling shovels during a gold rush.

Given how important the relatively obscure niche of web data collection has become, the word of the year is attention – whether we’re talking about the business, product, or even legal side of things. Let’s dissect some of the ways this state of affairs has transformed our industry.

Unprecedented Influx of Venture Capital

How do you quantify hype? It’s simple: just follow the flow of venture capital. In this case, the numbers are more than obvious: in 2024, AI startups received twice as much funding than the year before; by August 2025, they had already raised more than in 2024 as a whole. Out of total VC investments this year, AI had raked nearly a half of the pie, compared to just one third the year before.

Much of this money went to several largest companies in the field, such as the $40 billion investment into OpenAI. While miniscule in comparison, over $150 million were allocated to startups in the area of web data access. The only other big VC investment event in this market has seen (or at least, we’ve seen) involved EMK Capital’s $100M+ acquisition of Bright Data – back in 2017!

Company	Founded	Headquarters	Funding (2024-2025)
Exa.ai	2021	California, US	$85M
Browserbase	2024	California, US	$67.5M
Tavily	2024	New York, US	$25M
OnKernel	2025	Delaware, US	$22M
Browser-Use	2024	California, US	$17M
Firecrawl	2024	Delaware, US	$14.5M

The investees are all based in the United States, with an average age of under two years. American companies have always participated in this industry, though never really as dominant players. But now, venture capital has effectively spurred a new generation of well-equipped web scraping infrastructure providers.

New Generation of AI Web Scraping Companies

Let’s talk about this new generation. Does it have any defining characteristics to distinguish it from the old guard, or are we getting more of the same? In brief, these companies are different, both in features and mindset.

The focus areas of gen-AI web scraping companies

LLM-friendly scraping	Web search (Google, in-house, mixed index)	Browser agents
Firecrawl Crawl4AI ScrapeGraphAI	Exa Tavily Perplexity	Browserbase Browser-Use OnKernel

First, they are AI-native businesses, which is obvious looking at the marketing. Firecrawl offers to “turn websites into LLM-friendly data”, Browserbase promises “a web browser for your AI”, and Tavily entices to “connect your {AI} agent to the web”. There’s no question who the desired user is, or how this user can benefit from the service. Quoting Tavily’s CEO, these companies first and foremost identify themselves as AI enablers.

Gen-AI’s approach also differs when it comes to the product. For one, they love to simplify input by either directly accepting natural language or building abstractions that do. As an example, Browserbase’s Stagehand library gives users the choice when to use code or plain language instructions to automate a browser. The later-released Director goes a step further and expects only plain language commands.

Furthermore, gen-AI pays particular attention to website crawling, which web scraping infrastructure providers traditionally avoided. Now, it’s easier than ever to fetch the URL structure or complete contents of a website; all you need to do is call an endpoint and tweak a few knobs.

In the same way, you can request to scrape not only the list of Google’s search results but also their content. The addition of Markdown formatting makes wholesale crawling directly useful for language models. Exa and Tavily go a step further: instead of simply scraping Google, they cache, interpret, and repackage results to replace the search engine for AI agents.

Gen-AI providers themselves make heavy use of LLMs in data processing. Machine-generated static parsers for humans are relatively rigid; human-specified parsing rules for AI, on the other hand, work decently well – and, more importantly, transfer between domains without breaking too much. This unlocks what Zyte calls the long-tail of the web and marks a big step in scalability over handcrafted parsing rules.

The final, though probably least quantifiable, characteristic is the mindset. While they foremost offer infrastructure, the new crop seem to have a builder’s mentality. They show no qualms over layering research assistants, lead enrichment agents, and other functionality upon the core service. This showcases the platform’s capabilities, expands the problem space, and creates potential opportunities for a pivot.

In contrast, incumbents have clearly-defined domains and focus on solving engineering challenges within them. As such, their approach is often iterative and oriented toward maintaining rather than creating. This results in less hype but more solid backbones. And, of course, without impatient venture capital firms breathing down their necks, there’s less external pressure to gun for moonshots.

Emergence of Powerful Open-Source Tools

An interesting corollary of the builder mindset is the attention shown to open source. Apify and Zyte have created amazing web scraping frameworks (Crawlee and Scrapy, respectively). But these two examples aside, most of the open tooling has come from individual developers giving rather than businesses giving back to the community.

Things changed in 2024. The gen-AI scraping companies made multiple GitHub repositories that, in the least hyperbolical sense of the word, have exploded in popularity:

Repository	Purpose	GitHub stars (Sep ’25)	License
Browser-Use	Manipulation of headless browsers	70K	MIT
Firecrawl	Crawling, scraping & data parsing	60K	AGPL-3
Crawl4AI	Crawling, scraping & data parsing	54K	Apache-2.0
GTP Researcher (Tavily)	Creating deep research agents	23.5K	Apache-2.0
Stagehand (Browserbase)	Manipulation of headless browsers	17K	MIT

There are two things to highlight about this phenomenon. The first is that the community didn’t just get a bone with useful but ultimately minor functionality, such as simulating mouse movements. Crawl4AI is a complete and fully open-sourced web scraper. Firecrawl and Browser-Use, in turn, run an open-core business model which monetizes cloud hosting, anti-bot avoidance, and quality-of-life features.

The second striking observation is that all but one of the listed repositories enforce highly permissive licensing models. In essence, only Firecrawl limits redistribution, requiring adopters to share any changes they make to the code.

Where’s the value for the company? Well, open-core can be a viable business model in its own right, as exemplified by RedHat. But really, it’s an amazing approach for building mindshare, and for gradually moving businesses to the managed service once they encounter the hairy areas of web scraping.

Adaptation of the Incumbents

How has the AI boom affected the so-called establishment of web data providers? Some embraced it like their only newborn; others, conversely, have made few changes that cut beyond skin deep. In any case, most have acknowledged the importance of large language models in their businesses.

The first and most obvious area to embrace AI was marketing. Early birds like Apify and Bright Data started courting startups by the end of 2024. First, the efforts were modest: a section in the navigation bar, a landing page, or a banner. By March 2025, Bright Data had completely pivoted to serving AI use cases, reducing other clients to everything else and then simply BI in its headline.

The gambit seems to have paid off: in late 2025, Bright Data announced that AI enabled it to hit $300M per year (up from $100M in 2021), and that it expected to reach $400M ARR by mid-2026.

Others followed suit throughout 2025. Apify now provides real-time data for your AI, NetNut powers AI models (in bold) & complex pipelines, Oxylabs’ products empower the AI industry and beyond, while SOAX offers real-time public web data for AI teams. Not all had converted when we wrote this, though: ScraperAPI, Zenrows, and even Zyte remained modest about wooing AI companies.

Web scraping APIs are a natural match for large language models. But the peculiar needs they bring, such as limited context sizes or their hunger for Google search data, also required implementing new features. The list includes Markdown, plain text, or Toon output formats, lightweight versions of Google parsers, and lately MCP servers for agentic browsing.

In addition, the AI craze gave impetus for launching new products to wrestle with emerging competitors. For example, Bright Data made a crawling API and spun off its cloud browsers into a new, AI-first brand browser.ai. In its own right, Oxylabs announced an unblocker browser service, and it runs a separate suite of Firecrawl-like scrapers called AI Studio.

Comparison of AI-oriented features

	Discovery endpoints	Search endpoints	Browser endpoints	Structured data	Output formats	MCP server
Apify	Crawler, mapper, target-specific	SERP APIs	❌	Target-specific	HTML, JSON, CSV, Markdown,	✅
Crawlbase	Target-specific	SERP APIs	❌	Target-specific	HTML, JSON, screenshot	✅
Decodo	Target-specific	SERP APIs	❌	AI selector generator, target-specific	HTML, JSON, Markdown, screenshot	✅
NetNut	❌	SERP APIs	❌	Target-specific	HTML, JSON	❌
Nimble	Target-specific	SERP APIs	❌	Target-specific, AI parser	HTML, JSON, screenshot	✅
Oxylabs	Crawler, mapper, target-specific	SERP APIs, fast search, search + extract	✅	Target-specific, AI selector generator, AI parser	HTML, JSON, Markdown, screenshot, Toon	✅
Scrapeless	Crawler, target-specific	SERP APIs	✅	Target-specific	HTML, JSON, text, Markdown, page content	✅
ScraperAPI	Crawler (beta), target-specific	SERP APIs	❌	Target-specific	HTML, JSON, text, Markdown, screenshot	✅
ScrapingBee	Target-specific	SERP APIs	❌	Target-specific, AI parser	HTML, JSON, text, Markdown, screenshot	✅
ZenRows	Target-specific	SERP APIs	✅	Target-specific, output filters (links emails, etc.)	HTML, JSON, Markdown, text	❌
Zyte	Products, articles, jobs	SERP API	Through hosted VS Code	Page types (products, articles, jobs, SERP)	HTML, JSON, page content	❌

The same models that rely on web scraping to improve have found themselves in the list of scraper API features, creating an interesting synergy. Their main customer-facing role lies in data transformation: either by generating parsing rules from a prompt, or the other way around – extracting data directly with the rules given to them. This slide from OxyCon shows the winding path established providers take in adopting AI for parsing:

Even then, few of the entrenched providers have introduced agentic web scraping that could extract information end-to-end using a prompt. This, and the more modest task of LLM parsing in general, are hardly enterprise-ready in reliability and cost, so they compete with other methods for structuring the long-tail of the web.

The most active in this domain is probably Zyte, whose machine learning models are able to extract various page types (product, news, etc.) in predictable schemas. Some others have implemented it as well, such as Scrapfly. But this method has received limited adoption in general.

AI-oriented and regular web scrapers can be distinct enough to warrant putting them into separate taxonomies. But this is where we, as reviewers, encounter head-scratching problems. Is an AI web scraper something that helps AI use cases or implements AI to function better? Maybe the answer lies somewhere in-between.

Multimodal Data (and Training Data in General) Remain in Big Demand

With the hype cycle around LLMs shifting to RAG and then agents, you’d think that the web as a source of training data has been exhausted. Indeed, some have turned to synthetic, user-generated, and licensed data to improve inputs. But according to Cloudflare, that’s not the case: three quarters of all AI traffic in mid-2025 were generated for training purposes.

Of course, it would be unwise to treat this matter as either-or. As always, the answer requires more nuance. Textual web data for training purposes, while foundational, has indeed become less relevant and extracted many times over. It may also be starting to suffer from recursion, where an LLM ingests content generated by another AI model. Multimodal data like videos, however, remains the frontier for further training. We wrote about it back in spring, and we still believe it to be the case.

To reinforce our point, we’ll once again whip out the GitHub trendline of yt-dlp, a popular library for downloading videos. What’s different today is that the market now also offers a choice of commercial scrapers and pre-collected datasets. Oxylabs and Wyndlabs are just some examples of providers that are capitalizing on this opportunity.

If we take Cloudflare’s graph as ground truth, agentic scraping has generated more talk than action so far. However, with the way things – and funds – are going, its share is bound to grow.

Intensifying Efforts to Curb Data Collection for AI

More attention to web scraping has brought its fair share of negative consequences. Companies have never been keen on giving unfettered access to their data, whether to keep a competitive edge or to avoid unnecessary hosting expenses. But now, in addition to reaching an unprecedented scale, web scraping has started threatening established business models.

Should we believe an insights company called Mordor Intelligence, the market for Web Application Firewalls (WAFs) is at $11 billion. Akamai alone makes nearly $4 billion per year, and we’ve seen some high-profile mergers and acquisitions, such as the ten-figure acquisition of Shape Security by F5, or the merger between PerimeterX and Human.

The general sentiment in major web scraping conferences like OxyCon and Extract Summit is that anti-bot efforts have toughened significantly. One participant half-jokingly shared that two days of unblocking efforts used to give two weeks of access to websites; now, it’s become the other way around. Zyte went as far as to proclaim the death of proxies, arguing that there’s no longer sense or skill for many to build a web scraper in-house.

Bot detection services can still cost a lot. But they’re becoming both more advanced and more accessible for businesses. Many smaller websites find Cloudflare’s ubiquitous “are you human?” checkbox good enough, and this feature comes for free.

Even the web giants which were always challenging but reasonable to scrape are bristling up. The case in point is Google: in January, it sent the industry into chaos by starting to require JavaScript in Search. In September, it stopped returning 100 search results in one request, breaking many SEO tracking tools. YouTube has also been affected: it may require solving a CAPTCHA before playing a video or straight out refuse to do so until the suspicious user logs in.

Cloudflare and its various AI-focused initiatives warrant a separate mention. In 2025, Cloudflare’s users received the option to trap AI crawlers in a labyrinth, then to block them altogether. The company didn’t stop here: it undertook to control access by first launching a bot authentication protocol and then an option to pay for crawling through the use of 402, a forgotten HTTP response code. These audacious moves both promote cryptocurrency (which is used to pay for access) and try to redefine the web’s incentive system.

Cloudflare’s efforts have received mixed reception. The new crop of web scrapers like Browserbase and Browser-Use actually jumped at the opportunity, integrating web bot auth and pay-per-crawl into their systems. Entrenched providers like Bright Data saw this as an affront against the open web and an attempt to become a gatekeeper. We tend to lean towards the latter position.

Conclusion

This concludes the report – thanks for reading! Since the summary is at the front, we’ll use this opportunity to invite you to ask questions or give feedback. Feel free to reach out through info at proxyway dot com or our Discord server.

Adam Dubois

Proxy geek and developer.

December 4, 2025

The post Web Scraping API Report 2025 appeared first on Proxyway.

Proxy Service Awards 2025

Adam Dubois — Tue, 03 Jun 2025 07:23:02 +0000

Research

Halfway through 2025, we find ourselves in a vibrant and thriving proxy server market. The landscape is shaped by AI companies with their insatiable appetite for data, commoditization of proxies as a product, and intense competition, both direct and through white labels.

We recently launched our annual proxy market report, where we observed and documented these trends. And now, it’s time to recognize the providers whose tireless efforts have made proxy servers more performant, accessible, and otherwise better – for us and the market at large.

These are the nominees for 2025’s awards:

Best Enterprise Provider: Oxylabs

Best Enterprise Provider: Oxylabs

The past year brought the proxy server market to a simmering point, with 250 companies fighting for your business. All the while, the lucrative enterprise segment remained relatively untouched. It’s no wonder, then, that providers like Infatica, SOAX, DataImpulse, and even Rayobyte, which was all about the everyday Joe a moment ago, have been moving up.

However, that’s not exactly easy to achieve. As one of the companies exclaimed, enterprise competitors are well-funded, have strong brand recognition, and long-standing relationships. They make it easy to ensure compliance and work on projects as a team. And, of course, they are able to support the scale needed for large operations.

In our eyes, Oxylabs is the prime example of an enterprise-minded proxy provider. Not only is its proxy network huge and market-leading in performance, but you also get team access, detailed observability and budgeting tools, various assurances, and great human care. Some aspects of the user experience could be improved, but they do little to detract from the overwhelming positives.

And so, with Bright Data dropping out this year, and NetNut’s platform once again failing to support its impressive proxy infrastructure, we unashamedly nominate Oxylabs as the Best Enterprise Provider of 2025. Congratulations!

Contender of the Year:
SOAX

Contender of the Year: SOAX

For a while, it seemed like SOAX had fallen into a lull. It fumbled our 2024 benchmarks and invested a lot into building a no-code AI scraper, whose public interface could hardly even be called a proof of concept and is no longer available. Meanwhile, its edge in features vanished while competitors like NetNut raced full speed ahead. We’re painting a pretty grim picture, aren’t we?

Well, the SOAX of today seems to be in a much better – or at least livelier – place than where we left it. The provider has once again claimed its role as a leading technologist, being one of the first to introduce features like UDP and QUIC support. During our tests, it launched more gateway servers, which markedly improved network performance. And, of course, in recent months, SOAX joined the ranks of the few providers that offer unified platform-based pricing.

If that wasn’t enough, SOAX recently announced the acquisition of not one but two proxy server vendors to further strengthen its infrastructure. All things considered, we can’t imagine any other provider more deserving of our 2025 Contender of the Year award. Well done!

Value Choice:
Decodo

Value Choice: Decodo

Giving our Best Value award each year is like reliving Groundhog day – the nominee always ends up being Smartproxy (now Decodo). But unlike the hero in Groundhog day, who resets the clock every time he does something wrong, Decodo continues doing things right. And frankly, we’re fine with that.

What exactly about Decodo generates so much value? First, as a mid-market provider, it simply punches above its weight. Decodo’s proxy networks rival and often beat even more expensive alternatives in key metrics. For example, its residential proxy pool was top three both in size and infrastructure performance. Then, there’s the matter of price. Decodo remains highly competitive, especially with its shared server-based proxies, whose rates are downright unfair.

But this wouldn’t work so well if it weren’t for Decodo’s user experience. The provider relentlessly iterates to make its dashboard and interactions friction-free. The documentation is well-written and maintained. Decodo’s excellent customer service ties it all together, with competent live chat help at any hour of the week.

Of course, the competition doesn’t sleep. IPRoyal is breathing down Decodo’s neck with its still oh-so-compelling forever packages. And there are the less scrupulous but very efficient providers from the East. But do they have enough to break the spell? Our answer is clear for now.

Newcomers of the Year:
Evomi & Massive

Newcomers of the Year: Evomi & Massive

Last year, we came to an impasse. There were two providers – both very different but absolutely deserving of our Newcomer of the Year award. After long deliberation, we chose one from a market segment that was represented less.

This year, we found ourselves in the same boat. Evomi and Massive, just one year in the making, started off strong and managed to hold their own against the very best. What went around came around, so we decided to take a different approach and recognize both newcomers for their efforts.

What made them stand out? For Evomi, there were several factors. For one, the provider has already built a solid base to stand on, with competitive proxy pools and infrastructure performance. It also invests in branding, highlighting trust and Swiss origins. That said, Evomi’s differentiating quality comes in the way it packages the residential proxy network, with amazingly cheap base rates and unique filters like proxy latency and fraud score.

Massive has fewer fancy features to flaunt (other than device filtering, which is really neat), but that’s not what this provider is about. Its core functionality is simply good, to say the least, no matter how you cut it. While not the largest, Massive’s proxy network is home-sourced, fast, and effective. It topped our benchmarks multiple times and handled everything we threw at it. The platform and variety lag behind; but with only one year in, there’s plenty of time to build them out.

Looking beyond their strong first impressions, do Evomi and Massive have enough to entrench themselves among the greats? Evomi’s challenge will be to ensure the pool’s quality at the price it charges – we already saw it stumble in our real-world tests. Massive needs to get its name out there – and, being an American VC-backed company – follow through with the path it’s taking.

But hey – let’s carpe diem and leave future worries for the future. Congratulations to both providers – you deserve it!

Greatest Progress:
DataImpulse

Greatest Progress: DataImpulse

DataImpulse is our previous Newcomer of the Year, chosen after a long deliberation and with high hopes for the future. A year later, it looks like the provider didn’t disappoint.

DataImpulse retains the qualities that made it so appealing. Its proxy networks are still extremely affordable, outfitted with advanced features and non-expiring plans. You’re still greeted by a human, and this is even touted as a proud selling point. The platform is also surprisingly robust, with useful capabilities like visibility into online IPs.

All the while, DataImpulse’s proxy networks have grown faster and been better maintained. The provider has launched an SDK alongside its bandwidth sharing app to increase the pool size. There’s even a Premium plan now that cherry-picks the best performing proxies and promises particular attention to custom needs.

It’s hard to call DataImpulse’s changes a revolution. Rather, the provider has been steadily building upon its strengths. And frankly, this is all that’s needed, at least to receive our 2025 Greatest Progress award. Keep it up!

Adam Dubois

Proxy geek and developer.

June 2, 2025

The post Proxy Service Awards 2025 appeared first on Proxyway.

Proxy Market Research 2025

Adam Dubois — Tue, 29 Apr 2025 10:11:52 +0000

Research

Proxy Market Research is a report on the proxy server industry, published annually since 2019. It covers the market trends and thoroughly compares major proxy server providers.

You’ll find Proxy Market Research useful whether you’re shopping around for proxies, investigating the market, or evaluating where your service stands compared to household names.

This year’s report:

Features 11 (plus one) companies that cover all segments of the market.
Overviews four proxy server types: residential, mobile, shared datacenter, and dedicated ISP proxies.
Relies on a questionnaire sent to all participants, as well as our own know-how.
Provides large-scale performance benchmarks, enriched with years of historical data.

The majority of data were collected in February and March of 2025, with performance benchmarks reaching into April.

Participants

Below are this year’s participants with the proxy server types we tested for this report. Bright Data also took part but decided to withdraw at the last moment. Knowing how important this provider is, we’ll add it as a shadow participant, using 2024 performance data for some of the tests – purely for reference.

We categorize the providers based on the market segment they align with the most:

Entry-level providers work best for individual or small-scale projects.
Mid-market providers cater to a wide range of scraping, account management, and similar tasks.
Enterprise providers focus on an integrated experience that accommodates large-scale use by teams.

Segment	Provider	Residential	Mobile (peer to peer)	Datacenter (rotating)	ISP (dedicated)
Enterprise	Bright Data	✅	✅	✅	❌
	NetNut	✅	✅	✅	✅
	Oxylabs	✅	✅	✅	✅
Mid-market	Decodo (ex Smartproxy)	✅	✅	✅	✅
	Infatica	✅	✅	❌	✅
	IPRoyal	✅	❌	❌	✅
	Massive	✅	❌	❌	✅
	SOAX	✅	✅	✅	✅
Entry/mid-market	DataImpulse	✅	✅	✅	❌
	Evomi	✅	✅	✅	❌
	Rayobyte	✅	❌	✅	✅
	Webshare	✅	❌	✅	✅

Such classification is somewhat rigid: SOAX and Infatica are trying to position as enterprise providers, and DataImpulse has launched an enterprise-oriented residential proxy pool. But we’re focusing on the current state of affairs rather than intent.

Bright Data is one of the biggest players in the proxy server and web data collection industries. Owned by a venture capital fund, the provider is known for its robust platform and focus on ethical matters. Bright Data’s sources IPs through various methods, including an SDK, proxyware app, and more.

Country: Israel
Founded: 2014
Employees (LinkedIn): 400+
Segment: Enterprise
Proxy networks: datacenter, residential, ISP, mobile
Other services: scraping APIs, scraping browser, IDE, datasets, e-commerce insights

DataImpulse is run by the Ukrainian dev house Softoria, which also owns a popular search engine scraping API. The provider sources proxies in-house and offers them in non-expiring packages, the rates of which are hard to beat.

Country: US
Founded: 2022
Employees (LinkedIn): Up to 20
Segment: Entry-level
Proxy networks: datacenter, residential, mobile
Other services: –

A long-running value choice and top five proxy provider which rebranded from Smartproxy in April 2025. Decodo positions itself as an easier and more affordable alternative to enterprise competitors – so far, to great success. The company acquires proxies through “reputable providers and a peer-to-peer network”.

Country: Lithuania
Founded: 2018
Employees (LinkedIn): 100-150
Segment: Mid-market
Proxy networks: datacenter, residential, ISP, mobile
Other services: scraping APIs, antidetect browser

Though Evomi is a new brand, the people behind it have been in the market for years. The provider is still minor and relatively unknown, but it has own IP sources, competent branding, and Swiss origins, which the founders advertise as a seal of quality.

Country: Switzerland
Founded: 2024
Employees (LinkedIn): Up to 20
Segment: Entry/mid-market
Proxy networks: datacenter, residential, ISP, mobile
Other services: –

A Singapore (and now UK) based company with Eastern European roots, Infatica can be considered a mid-sized proxy provider. It has expanded to sell all proxy types, scraping infrastructure, and even datasets, lately with an eye to enterprise clients. Infatica sources IPs through an openly-promoted SDK.

Country: Singapore
Founded: 2019
Employees (LinkedIn): 25-50
Segment: Mid-market
Proxy networks: datacenter, residential, ISP, mobile
Other services: scraping APIs, datasets

Since its founding in 2021, IPRoyal has grown into a force among value-oriented providers. Its main value propositions are scaling pay-as-you-go traffic and approachable plans for small users. The provider still focuses on proxy servers, though we expect vertical expansion soon. IPRoyal maintains its own residential proxy pool through an app called Pawns.

Country: Lithuania
Founded: 2021
Employees (LinkedIn): 50-100
Segment: Mid-market
Proxy networks: datacenter, residential, ISP, mobile
Other services: –

Originally a venture capital-backed resource sharing platform, Massive seriously entered the proxy server market only in 2024. As such, it has the manpower and expertise to threaten the entrenched providers, but there’s still a lot of groundwork to be laid. Massive openly sources IPs through its SDK.

Country: US
Founded: 2018
Employees (LinkedIn): 25-50
Segment: Mid-market
Proxy networks: residential, ISP
Other services: –

The only publicly traded company among our participants, NetNut probably makes it into top five proxy providers by size. It was one of the original suppliers of ISP proxies, now fueled by residential proxy servers like most. The provider gets IPs through its “proprietary reflection technology”, then sells them to companies with large data collection needs and whitelabels.

Country: Israel
Founded: 2018
Employees (LinkedIn): 100-150
Segment: Enterprise
Proxy networks: datacenter, residential, ISP, mobile
Other services: scraping APIs, datasets

One of, and by now maybe the market leader for proxy servers in the enterprise segment. Oxylabs controls a large infrastructure of IPs sourced through its partner app Honeygain and an SDK. We often award this provider for strong performance.

Country: Lithuania
Founded: 2015
Employees (LinkedIn): 300-400
Segment: Enterprise
Proxy networks: datacenter, residential, ISP, mobile
Other services: scraping APIs, scraping browser, datasets

A self-proclaimed largest US datacenter proxy vendor, Rayobyte is now embracing residential IPs. The company’s brand has pivoted several times throughout the years, looking for its ideal customer. Overall, Rayobyte emphasizes approachability and ethics as its main talking points.

Country: United States
Founded: 2015
Employees (LinkedIn): 25-50
Segment: Entry/mid-market
Proxy networks: datacenter, residential, ISP, mobile
Other services: scraping API

An established mid-sized provider with Eastern European roots and international presence. SOAX is known for its wealth of features. Lately, the company has been shifting attention to web scraping tools. There’s no public information about how this provider gets its IPs.

Country: UK
Founded: 2020
Employees (LinkedIn): 50-100
Segment: Mid-market
Proxy networks: datacenter, residential, ISP, mobile
Other services: scraping APIs, AI scraper

Based in the US, Webshare has been a popular destination for small-time customers in need of proxies. Its killer duo is the customizable and developer-friendly platform, paired with highly affordable shared proxies. The company was acquired by Oxylabs in 2022, which likely supplies it with residential IPs.

Country: US
Founded: 2018
Employees (LinkedIn): 25-50
Segment: Entry/mid-market
Proxy networks: datacenter, residential, ISP
Other services: –

Methodology

This section describes the technical methodology used for this report.

All participants were introduced to it prior to testing. As a result, this data may sometimes represent their best rather than default configuration. That said, we used one configuration for all benchmarks, which hopefully prevents overoptimization.

This is our main benchmark. It serves a dual purpose. First, it aims to check how many unique IPs a provider’s proxy network contains. At the same time, it measures the speed and success rate when connecting to a small page that doesn’t block requests.

Parameters

Scraper: custom Python script
Target: endpoint of a global CDN. The page weighs ~6KB and accesses a data center nearest to the proxy server.
Geolocation data: latest IP2Location & MaxMind databases.
Residential/mobile IP percentage: IP2Location’s Usage type data point, ISP and/or MOB IPs.

Residential proxies

Proxy location	Days	Total requests	Testing server
Global	21	1.2 million	DE
EU (DE, ES, FR, IT, NL)	14	1.2 million	DE
US	14	560,000	US
UK	14	560,000	DE
India	14	560,000	SG
Brazil	14	560,000	US
Australia	7	140,000	SG

Mobile proxies

Proxy location	Days	Total requests	Testing server
Random	14	280,000	DE
EU (DE, ES, FR, IT, NL)	14	280,000	DE
US	14	280,000	US
UK	14	280,000	DE
India	14	280,000	SG
Brazil	14	280,000	US
Australia	7	140,000	SG

Datacenter & ISP proxies

Proxy location	Days	Total requests	Testing server
US	7	70,000	US

This benchmark aims to evaluate the proxy networks in more realistic conditions. It has too many variables (such as the scraper’s quality and changes in bot protection) to be reliable on its own. However, the test has value when comparing providers under the same conditions.

Parameters

Scraper: custom HTTPX script for Amazon, Puppeteer script for Instagram, and Splash script for Google.
Rate: 1 request per second.
Request validation: Response code (200), HTML size, page title

All proxy types

Targets	Proxy location	Testing server	Requests
Amazon Google Instagram	US	US	5,200 (1,800 * 3 tests)

We also run a collection of smaller benchmarks to test various aspects of the proxy networks. The brackets indicate which product the benchmark applies to.

UDP protocol support (all proxy types)

This test uses a third-party tool to check if the proxy network is able to route traffic through UDP.

Download speed (datacenter & ISP proxies)

Tool	Testing server	IPs checked
Hetzner’s 100 MB & 1 GB speed tests	US	At least 10

IP quality (residential proxies)

This benchmark uses a popular third-party tool to measure how many IPs it’s flagged as potentially malicious.

Tool	Checks (US)	Checks (Global)
IPQualityScore	~10,000	~20,000

Infrastructure performance with larger pages (residential proxies)

This test targets an endpoint of a global CDN. It downloads a 2 MB package from a data center nearest to the proxy server.

Proxy location	Requests	Testing server
Global	15,000	DE
US	5,000	US

Session persistence (residential proxies)

This benchmark measures how long proxy peers remain online. We establish a sticky session and send requests every 20 seconds until 30 minutes pass or the IP changes.

Target	Proxy location	Session count	Testing server
http://icanhazip.com	US	250	US

Proxy operating system (residential proxies)

This test tries to determine which types of devices (Windows, Android, iOS, etc.) comprise a provider’s proxy pool. It’s based on zardaxt’s passive fingerprinting tool.

Target	Proxy location	Proxies checked
Our own domain	Global	500,000

Market Trends

This section provides an overview of the proxy server market. It’s based on our own observations, as well as responses from a survey sent to all participants.

2024 Report: A Year of Growth Fueled by AI

Providers described 2024 as a strong year, with multiple reports of double digit growth.
Their results were strongly impacted by AI, which received record venture capital funding.
The emphasis in 2025 is on continuity – if the global environment will allow it.

Full version

Comparing Popular Web Scraping & Proxy APIs

Adam Dubois — Fri, 27 Sep 2024 08:41:58 +0000

Research

Our report benchmarks nearly a dozen popular unblockers and web scraping APIs. These remote scrapers simplify the process of web data collection by overcoming CAPTCHAs, JavaScript virtual machines, and other roadblocks erected by anti-bot systems.

Though growing in popularity every year, web scraping APIs have become especially relevant with the rise of AI models, closing down of online platforms, and commercialization of bot protection approaches.

Our main goal is to see how well the APIs are able to unblock protected websites in late 2024 (for earlier reports refer to: comparison of web scraping APIs (2023), comparison of proxy APIs (2023)). We also take a look at their features and pricing strategies to get a well-rounded view of the market.

Summary

Our list of participants included 11 API providers, which we tested on 10 protected websites at a rate of 10 requests per second.
Five APIs managed to open all targets consistently, while the others failed to unblock between one and five websites. Oxylabs, Zyte, and Bright Data had the highest average success rate of around 98%.
Zyte’s API worked extremely fast, managing to unblock all targets without headless browsers. Bright Data’s Web Unlocker, though second to last in speed, was the only API that didn’t fail a single run.
G2 (Cloudflare) proved to be the hardest target by success rate, while the largest number of participants – five – failed to unblock Allegro (DataDome).
Compared to proxy APIs (also called web unblockers), web scraping APIs have more features: asynchronous delivery, data parsing capabilities, and specialized endpoints. Some also include a proxy mode, making the distinction arbitrary.
We’re seeing more providers release specialized endpoints for popular targets. In addition, several AI-based parsing approaches have appeared, ranging from models trained on page types to AI-generated parsing schemas.
Credit-based pricing models are usually chosen by providers that service smaller customers; while extremely cheap for basic websites, they impose huge multipliers when accessing challenging targets.
Out of the business-oriented providers, Zyte is very hard to beat on price for unblocking tasks. For targets that rely on JavaScript or need special functionality (e. g. localized Google or Amazon queries), Smartproxy and Oxylabs offer a compelling balance between performance and cost.

Participants

Our research includes 11 major providers of web scraping and proxy APIs (often called web unblockers). The tools technically form different product categories, but we decided not to separate them. Both tend to use the same tech, and web scraping APIs sometimes have a proxy mode as one of the integration formats, which further blurs the distinction.

Most participants are well known in the industry, though not necessarily for their scraping infrastructure. Here’s the full list:

Participant	Tested products	Target audience
Bright Data	Web Unlocker, SERP API	Companies & enterprise
Infatica	Web, SERP, E-Commerce APIs	Individuals & small businesses
NetNut	Web Unblocker	Companies & enterprise
Nimble	Web, SERP, E-Commerce APIs	Companies & enterprise
Oxylabs	Web Scraper API	Companies & enterprise
Rayobyte	Scraping Robot	Individuals & small businesses
ScraperAPI	Scraping API	Individuals & small businesses
Scrapingdog	Web Scraping API	Individuals & small businesses
Smartproxy	Web, SERP, E-Comm, Social APIs	Small to medium businesses
SOAX	Web Unblocker	Companies & enterprise
Zyte	Zyte API	Individuals to enterprise

Methodology

We gave all participants the methodology doc in advance. Some actively monitored our progress, making adjustments to their scrapers on the fly. This is fine, as web scraping is a dynamic process, and it should be approached as such. Hopefully, we also helped to improve the success rate for actual customers.

We chose 10 targets based on their popularity and bot protection system. Our goal was to try the scrapers with all major anti-bot vendors.

Target	Bot protection
Allegro (products)	DataDome
Amazon (products)	In-house
Canadagoose (products)	Kasada
G2 (product reviews)	Cloudflare
Google (SERPs)	In-house
Indeed (location directories)	Shape
Instagram (HTML profiles)	In-house
Lowe’s (products)	Akamai
Safeway (products)	Imperva
Walmart (products)	PerimeterX

There are some caveats to consider:

Anti-bot systems may have different levels of protection based on the website (or even categories of the same website).
Some bot protection vendors focus on securing sensitive endpoints (such as internal APIs or login pages), so they may not show up in full force against simple collection of public content.

We ran at least three tests for each target throughout several weeks. We fetched ~6,000 unique URLs, navigating directly to the page. The rate was 10 requests per second, with a timeout of 600 seconds. This is enough to trigger bot protection systems – and, as we’ll see, seriously tax some of the scrapers.

We used a custom Python script – its function was to simply send the request to the scraper and receive the response, measuring the time it took to reach us. Our server was located in the US.

Participants were free to suggest the optimal parameters for the targets, and some did. Otherwise, we used our own discretion, starting with the simplest configuration and enabling optional features (such as premium proxies and headless browsers) if we couldn’t unblock or load the valuable content.

We verified a request’s success by checking the response code and HTML size. The latter was necessary, as some websites (such as Safeway) tend to return 200-coded responses without data.

Some providers imposed concurrency limits that could potentially affect our scraping rate:

ScraperAPI’s largest public plan has a limit of 100 concurrent threads, which isn’t enough for 10 requests/second, especially when headless browsers are involved.
Infatica and Scrapingdog have the same restriction.
Zyte’s default rate limit is 500 requests per minute (~8.3/second).

ScraperAPI, Scrapingdog, and Zyte lifted their restrictions for us. Infatica wasn’t able to, forcing us to scrape most websites at 1 req/s. We also encountered SOAX’s internal limits and decided to stick with ~5 req/s for more complex targets.

Benchmark Results

The results show the best run each API had with each target. We’ll provide comments to give you more context where necessary.

Overall Performance

Oxylabs	Zyte	Bright Data	Smartproxy	Nimble	NetNut
98.50%	98.38%	97.90%	96.29%	95.48%	80.82%
SOAX	ScraperAPI	Scrapingdog	Infatica	Rayobyte
69.30%	67.72%	43.84%	38.40%	37.65%

Zyte	NetNut	Smartproxy	Scrapingdog	Nimble	SOAX
6.61 s	9.71 s	10.91 s	10.92 s	13.01 s	13.41 s
Oxylabs	ScraperAPI	Infatica	Bright Data	Rayobyte
13.45 s	15.39 s	17.15 s	22.08 s	26.24 s

Five providers managed to open all targets more or less consistently, which is an excellent result. We can distinguish Oxylabs and Zyte for their overall success rate; and though it doesn’t reflect here, Bright Data was amazingly dependable, never failing a single test.

The rest had at least one target that gave them trouble. But you shouldn’t discount these APIs just by looking at one aggregate number. For example, aside from Lowe’s and Safeway, NetNut performed flawlessly with most websites, and so on.

In terms of response time, Zyte’s API was a speed demon, beating others by up to four times. The provider made some adjustments during the process, and by the end, it could somehow open all targets without requiring JavaScript rendering.

Bright Data obviously prioritized unblocking success, and so it performed slower than expected. We believe that a better way to scale its APIs would be through more parallel requests, which our testing parameters didn’t fully exploit.

Hardest Targets to Unblock

We omitted Infatica’s and SOAX’s results from the success rate column, as they were tested under lower rate limits (one and five requests per second, respectively).

	Avg. success rate	Participants with over 80% failure rate
G2	60.39%	4
Lowe’s	67.17%	3
Allegro	68.32%	5
Safeway	68.93%	4
Canadagoose	75.73%	4
Indeed	81.40%	1
Instagram	89.82%	0
Google	93.77%	0
Walmart	94.54%	1
Amazon	96.12%	0

G2 (Cloudflare) proved the hardest to unblock looking at the average success rate of all providers. However, it was actually Allegro that the most participants failed to open consistently. Its average success rate is boosted by the rest.

On the other hand, most APIs managed to open Google and Amazon nearly perfectly. As major web scraping targets, they’re the baseline for any commercial data collection service.

Breakdown by Individual Target

Provider	Success rate	Response time
Oxylabs	100%	1.96 s
Smartproxy	100%	2.38 s
Bright Data	99.90%	3.68 s
Nimble	99.80%	12.06 s
NetNut	99.62%	6.18 s
Zyte	99.13%	4.80 s
Scraping Dog	7.88%	10.85 s
ScraperAPI	7.01%	26.68 s
SOAX	2.12%	6.30 s
Rayobyte	1.54%	7.65 s
Infatica	Failed to unblock

As one participant exclaimed, Allegro is very hard to unblock. The website uses DataDome, and it’s even featured as a success case in the anti-bot vendor’s website.

In reality, we saw one of two extremes: an API either opened Allegro perfectly, or it failed completely. The same story repeated consistently across our tests. All in all, Oxylabs and Smartproxy performed particularly well here.

Provider	Success rate	Response time
ScraperAPI	100%	3.79 s
Oxylabs	100%	5.08 s
Bright Data	99.85%	5.88 s
Smartproxy	99.83%	5.05 s
Nimble	99.82%	6.39 s
Zyte	99.80%	3.26 s
NetNut	99.73%	6.21 s
SOAX	99.67%	12.11 s
Infatica	94.66%	8.85 s
Rayobyte	87.86%	12.93 s
Scraping Dog	78.23%	13.97 s

Amazon is the website for web scraping, so unblocking it is a must for any self-respecting service. As such, Amazon proved to be the least problematic target.

We saw consistent results with little deviation across runs. Though the difference was minimal, ScraperAPI had the best showing.

Provider	Success rate	Response time
NetNut	99.90%	7.01 s
Zyte	99.88%	15.26 s
ScraperAPI	99.79%	3.58 s
Bright Data	99.60%	4.45 s
Oxylabs	98.87%	4.09 s
Nimble	90.73%	11.85 s
Smartproxy	79.88%	6.22 s
Rayobyte	12.95%	56.83 s
SOAX	Failed to unblock
Infatica	Failed to unblock
Scraping Dog	Failed to unblock

The Canada Goose store includes only several hundred products, but it’s visited by hundreds of thousands of people every month. The website uses Kasada, which was a hard nut to crack. The participants either had a bypass method, or they failed to unblock this target.

Like in The Web Scraping Club’s benchmark, NetNut’s unblocker had the best success rate, though it wasn’t the fastest. Some APIs had significant variance between runs: Nimble, ScraperAPI, and Smartproxy failed several tests and then fixed their scrapers for others.

Provider	Success rate	Response time
NetNut	99.80%	4.79 s
SOAX	99.38%	13.75 s
Bright Data	91.74%	26.80 s
Zyte	90.12%	6.71 s
Oxylabs	87.35%	27.45 s
Smartproxy	83.95%	6.92 s
Nimble	69.11%	39.23 s
Scraping Dog	19.80%	3.33 s
ScraperAPI	1.36%	22.29 s
Rayobyte	0.32%	94.20 s
Infatica	Failed to unblock

G2, a major company review website, is protected by Cloudflare. We found it to be the most challenging target, giving even the most solid APIs a run for their money.

Again, NetNut showed the best performance, both in success rate and response time. As with Canada Goose, the results weren’t always consistent between runs for more than one participant.

Provider	Success rate	Response time
Zyte	100%	0.81 s
NetNut	100%	2.10 s
Nimble	100%	3.24 s
Smartproxy	100%	5.37 s
Oxylabs	99.98%	4.79 s
Scraping Dog	99.97%	2.93 s
Bright Data	99.86%	10.12 s
Infatica	95.07%	2.44 s
SOAX	94.13%	8.70 s
Rayobyte	92.20%	4.49 s
ScraperAPI	51.93%	5.83 s

Google is a must for any web scraping API. The search engine is protected by the infamous reCAPTCHA, which quickly rate limits suspicious visitors. But it proved no challenge to all but one API.

We’re particularly impressed with Zyte’s performance. Zyte API not only achieved a perfect success rate, but it returned requests in under one second – much faster than the rest.

Provider	Success rate	Response time
NetNut	100%	2.52 s
Smartproxy	100%	3.38 s
Bright Data	100%	4.67 s
Oxylabs	99.88%	3.69 s
Infatica	99.84%	3.12 s
Nimble	99.76%	10.80 s
Zyte	99.53%	10.85 s
SOAX	98.92%	12.84 s
ScraperAPI	98.80%	5.02 s
Scrapingdog	25.46%	20.03 s
Rayobyte	9.19%	21.51 s

Contrary to our expectations, Indeed wasn’t a hard target for the scrapers. The website employs Shape, a notoriously hard anti-bot system, but we either failed to trigger it or Indeed is using a lenient configuration.

In any case, at least five providers had amazing results, making it hard to distinguish one outlier. Similar outcomes repeated throughout all runs, with the exception of ScraperAPI.

Provider	Success rate	Response time
Nimble	99.97%	7.01 s
SOAX	99.73%	8.96 s
Oxylabs	99.55%	27.46 s
Smartproxy	99.48%	23.46 s
Zyte	99.13%	2.63 s
Bright Data	96.61%	55.04 s
NetNut	96.21%	25.31 s
Infatica	93.04%	20.40 s
ScraperAPI	79.33%	21.90 s
Scraping Dog	75.36%	8.83 s
Rayobyte	62.75%	13.63 s

Instagram is another major source of web data, though TikTok has probably started challenging it in popularity by now. The social media network uses its own bot protection system that redirects suspicious users to a login page. In our tests, however, Instagram didn’t cause big issues for most participants.

Overall, Nimble’s results look the best on paper. It’s also interesting that Zyte adjusted its scraper in the process, and our last test ran successfully without JavaScript rendering enabled. As a result, Zyte’s response time is mighty impressive.

Provider	Success rate	Response time
Zyte	100%	17.78 s
Smartproxy	99.98%	24.20 s
SOAX	99.83%	14.16 s
Nimble	99.81%	18.56 s
Oxylabs	99.75%	29.58 s
Bright Data	99.14%	75.61 s
ScraperAPI	63.13%	34.45 s
NetNut	27.00%	16.09 s
Scraping Dog	9.90%	23.31 s
Rayobyte	5.79%	39.36 s
Infatica	1.40%	50.93 s

Lowe’s is a decently popular target that employs Akamai’s bot protection system. It brought down a third of the participants, including NetNut, which came out strong with other anti-bots.

On the other hand, six APIs succeeded over 99% of the time, which is a great result.

Provider	Success rate	Response time
Zyte	100%	1.65 s
Smartproxy	99.81%	28.36 s
Oxylabs	99.69%	27.57 s
Nimble	95.95%	9.82 s
Bright Data	92.33%	29.33 s
ScraperAPI	75.84%	25.32 s
Scraping Dog	50.07%	2.61 s
Rayobyte	6.61%	2.09 s
NetNut	0.05%	11.55 s
SOAX	0.04%	27.31 s
Infatica	Failed to unblock

Safeway, the U.S. supermarket chain, is protected by Imperva and imposes aggressive geo-restrictions outside North America. The website isn’t a very popular target, so most participants found it tricky, requiring several runs to adjust.

All in all, Zyte’s performance looks amazing on paper, but it was Bright Data that ensured consistent results throughout all tests.

Provider	Success rate	Response time
Smartproxy	99.98%	3.80 s
ScraperAPI	99.98%	5.04 s
Bright Data	99.98%	5.20 s
Oxylabs	99.88%	2.84 s
Nimble	99.88%	11.12 s
SOAX	99.25%	16.58 s
Rayobyte	97.32%	9.68 s
Zyte	96.22%	2.31 s
NetNut	85.91%	15.68 s
Scraping Dog	71.70%	12.46 s
Infatica	Failed to unblock

Though probably eclipsed by Amazon, Walmart is a major e-commerce data source. It tends to juggle anti-bot systems but is generally associated with PerimeterX.

Most participants didn’t find Walmart problematic. However, we did see Nimble’s and Scraping Robot’s success rates crash after PerimeterX’s update late August.

Other Observations

When accessing protected targets, commercial APIs can be brittle. Less popular websites may need to be unblocked, even if the provider has a general bypass for that bot system. Popular targets like Walmart or G2 may too temporarily break after major updates.
Providers use different approaches for unblocking the same websites. Nimble relies on what it calls browserless drivers – they render JavaScript without invoking traditional headless browsers. We saw a big reliance on these drivers. On the other hand, by the end of our tests Zyte was able to access all targets without requiring browser-rendered HTML at all.
There’s a big difference between running tests at one request per second and ten or more. First, we wouldn’t have discovered that providers had scaling issues. Second, some websites don’t start seriously blocking until five requests per second or more.

Feature Overview

Let’s take a quick look at what you can do with the scraper and proxy APIs.

Proxy vs API Integration

The question of integration method is often already decided before purchase: if your codebase takes the proxy format, you’ll naturally gravitate towards it. But is there a real difference between the features API and proxy integration methods offer? In a way, yes.

	Proxy APIs (unblockers)	Web scraping APIs
Data delivery	Real-time	Real-time or on-demand, sometimes with batching & cloud storage
Geo-location selection	Often country-wide, sometimes up to city & ASN	Usually at the country level
Sessions	✅	✅
Custom headers & cookies	✅	✅
JavaScript rendering	A toggle	A toggle with optional instructions for scrolling, waiting, and more
Specialized endpoints	Usually unavailable	For popular websites with tailored parameters (e. g. ASIN entry, ZIP selection for Amazon)
Data parsing	Usually unavailable	Through specialized endpoints, manual selectors, or lately LLMs
Output formats	HTML	HTML, JSON, sometimes CSV

Proxy APIs:

	Integration	Geolocation	Sessions	Custom headers	JS rendering	Specialized endpoints	Data parsing
Bright Data	Proxy, async API	150+ countries with city & ASN targeting	✅	✅	Automated, toggle	Search engines	Specialized endpoints
NetNut	Proxy	150+ countries	✅	✅	Toggle	❌	❌

Web scraping APIs:

	Integration	Geolocation	Sessions	Custom headers	JS rendering	Specialized endpoints	Data parsing
Infatica	Real-time, async API	150+ countries	✅	✅	Toggle	Search, e-commerce	Specialized endpoints
Nimble	Real-time, async API (with batching, cloud storage)	150+ countries with state & city targeting	✅	✅	Toggle, instructions	Search, e-commerce, social media	Manual, autoparser, special endpoints
Oxylabs	Real-time, async API (with batching, cloud storage), proxy	150+ countries with ZIP for Amazon, city & coordinates for Google	✅	✅	Toggle, instructions	Search, e-commerce	Manual, special endpoints, parser builder
Rayobyte	Real-time, async API (with batching)	150+ countries	✅	✅	Toggle, instructions	Search, e-commerce	Manual, special endpoints
ScraperAPI	Real-time, async API (with batching), proxy	12 countries with 50+ upon request, ZIP code for Amazon	✅	✅	Toggle, instructions	Search, e-commerce	Manual, special endpoints
Scraping Dog	Real-time, async API, proxy	15 countries	✅	✅	Toggle, instructions	Search, e-commerce, social media, more	Special endpoints
Smartproxy	Real-time, async API (with batching), proxy	150+ countries with ZIP for Amazon, city & coordinates for Google	✅	✅	Toggle, instructions	Search, e-commerce, social media	Manual, special endpoints
SOAX	Real-time	150+ countries	❌	Cookies	Toggle	Search, e-commerce, social media	Special endpoints
Zyte	Real-time API, proxy	150+ countries	✅	✅	Toggle, instructions, scripting	❌	Manual, category based

Proxy APIs are often meant to be a direct upsell to proxy servers with a drop-in replacement process. At the same time, because you’re effectively outsourcing the page opening stage, proxy APIs need to go beyond regular proxy network features like geo-location to cover request manipulation and even JavaScript rendering. So they do.

Despite their wealth of features, proxy APIs can still be limited. For example, they rarely offer specialized endpoints, on-demand access to scraped output, or data structuring features. Another big drawback for complex scenarios is incompatibility with headless browser libraries, with no browser instructions on their own. This is where web scraping APIs offer more flexibility.

Exceptions exist. Bright Data’s SERP API integrates as a proxy, but in reality it’s a highly specialized scraper with data parsing and custom parameters. Funnily enough, some providers that sell web unblockers also offer web scraping APIs with a fully-featured proxy mode. In these scenarios, the difference hinges on the pricing method and, likely, marketing strategy.

How do you work with proxy and web scraping APIs? Obviously, the main requirement is sending an HTTP request to the provider’s server. However, the way you configure that request can differ. It’s usually either a GET request with parameters in the URL, headers, or a POST request with a JSON payload.

Exploring Individual Features

All modern proxy and web scraping APIs can render JavaScript. With pages becoming increasingly interactive, a question that arises more often every year is: what else can I do on the page? Proxy APIs tend to ignore it; the solution of scraping API developers is to expose browser controls through special parameters.

	Screenshot	Click	Input	Scroll	Wait
Nimble	✅	✅	✅	✅	✅
Oxylabs	✅	✅	✅	✅	✅
Rayobyte	✅	✅	✅	✅	✅
ScraperAPI	❌	✅	✅	✅	✅
Scraping Dog	✅	❌	❌	❌	✅
Smartproxy	✅	✅	✅	✅	✅
SOAX	✅	❌	❌	❌	✅
Zyte	✅	✅	✅	✅	✅

Bright Data, Infatica, NetNut – only basic rendering functionality available.

It’s possible to combine the instructions. For example, you can select a field, enter text, click on it, and wait for the response. Providers impose execution time limits, which often range between 60 and 120 seconds.

Zyte takes this a step further. Its clients get access to a cloud-hosted VS Code environment, where they can write their own interaction scripts.

The latter functionality isn’t common: instead, we’re seeing new categories emerge with the aim to increase web scraping success while providing standard compatibility with headless browser libraries. Some examples would be Undetect, Bright Data’s Scraping Browser, and major anti-detect browsers like Multilogin and Gologin.

Specialized endpoints are tailor made for websites or their properties (such as Amazon product pages). They often have custom parameters and data parsing capabilities. For instance, a Google SERP endpoint may be able to fetch local results (city-wide or in particular coordinates), which would otherwise be unavailable when targeting a general-purpose API.

	Google	Amazon	Others
Bright Data	SERP, ads, search types, local search	❌ (available in other products)	Bing, Yandex, DDG
Infatica	SERP, ads	Search, product	Booking
Nimble	SERP, ad optimization, local search	Search, product (incl. ZIP code)	Bing, Yandex, adding more fast
Oxylabs	SERP, ads, search types, hyperlocal search	Product, search, sellers, reviews, more (incl. ZIP code)	Walmart, Bing, Etsy, BestBuy, Target
Rayobyte	SERP	Product	❌
ScraperAPI	SERP, several search types	Product, search, offers, reviews (incl. ZIP)	Walmart
Scraping Dog	SERP, search types	Product, search (incl. ZIP)	LinkedIn, Twitter, Yelp, Indeed
Smartproxy	SERP, search types, hyperlocal search	Search, product, sellers, reviews, more (incl. ZIP)	❌
SOAX	SERP, search types	Search, product, reviews, questions	Walmart, all major search engines & social media platforms

NetNut, Zyte – no specialized endpoints available for the tested products.

Compared to the year before, we’re seeing an interesting trend: scraper vendors have been introducing more specialized endpoints to their products. One example is ScraperAPI, which now offers scrapers for Amazon, Google, and Walmart. Another is Nimbleway – the provider has set out to build what it calls online pipelines for targets in various verticals.

The direction is interesting, considering that LLMs have reduced the barrier to entry particularly when it comes to parsing, and that they tempt to consolidate toward one all-encompassing tool. Maybe a single purpose reassures that the scraper will be fit for the task?

Data parsing is an area where some of the most exciting developments are taking place. Of course, this is thanks to machine learning and large language models. But we’re also seeing changes in less sophisticated approaches: since our previous report, Oxylabs, ScraperAPI, and Smartproxy have all implemented selector support for building parsers by hand.

	Manual parsing	Pre-made templates	Other
Bright Data	❌	Specialized endpoints
Infatica	❌	Specialized endpoints
Nimble	Selectors	Specialized endpoints	Autoparsing, AI parser schemas
Oxylabs	Selectors	Specialized endpoints	AI parser schemas
Rayobyte	Selectors	Specialized endpoints
ScraperAPI	Selectors	Specialized endpoints
Scrapingdog	❌	Specialized endpoints
Smartproxy	Selectors	Specialized endpoints
SOAX	❌	Specialized endpoints
Zyte	Selectors		Models trained on page types

NetNut – no data parsing available for the tested products.

Let’s explore several different approaches to AI-based parsing that lie in the modest Other column.

#1. Custom machine learning models trained on specific page types.

Zyte has been playing around with machine learning for years now. Instead of parsing individual targets, Zyte trained multiple in-house models for whole page categories: products, news, directories, etc. The caveat was that they relied on AI vision, which required browsers. Still, during its conference roughly a year ago, Zyte bragged about being dozens of times cheaper and more accurate than ChatGPT.

Since then, Zyte has adapted the models to non-rendered requests, significantly cutting down the cost. It’s also experimenting with supplementary LLM features. They can make the schema more flexible by adding custom data points, and they can also transform data: translate, normalize, summarize, etc.

#2. A universal AI parser.

Similarly to Zyte, Nimble uses HTML-trained AI agents to extract data from various page types. Unlike Zyte, the provider automatically chooses the relevant agent depending on the page, keeping the decision process in the backend.

In a way, this makes the customer’s job easier. But it’s also way less predictable (Willl this target work? What’s the schema?). During our tests, we found the functionality to be more miss than hit: it parsed Lowe’s but failed to structure Canadagoose or G2. We’re sure it’s bound to improve fast.

To make the agents more robust, Nimble is preparing to release the ability to generate custom schemas. This feature will accept simple, likely natural language instructions and translate them into parsers. According to Nimble’s documentation, these parsers will get reusable IDs and heal automatically after identifying a failure.

For now, Nimble’s stopgap solution combines the dynamic parser with manual selectors to build a parser for the page.

#3. LLM-assisted parsers generated upon request.

This is the approach Oxylabs announced during its recent web scraping conference. Basically, you send a URL with natural language instructions to an LLM, then it generates a schema and selectors for scraping the data points. You get a preview of the output and the ability to adjust the schema to your needs. Once you’re happy, the selectors get added to the API request code.

Oxylabs’ approach is highly pragmatic, as the language model is invoked only once and not with every page access. However, it has limitations, namely that once a parser breaks, you have to manually repeat the generation process.

Pricing Approaches

We’ll overview the pricing models of the participants and how much our benchmarks would’ve cost.

Request, Credit-Based and Black Box Models

There are multiple ways to price proxy and web scraping APIs. The former use traffic or requests as the main metric. The latter charge for successful requests, either by keeping their model simplistic (one page = one request) or creating increasingly elaborate schemes based on credits.

Zyte’s model is closer to credits, for it includes variables that affect the final rate. But it’s also unique because the cost may change with time, depending on how hard Zyte finds the target to scrape. According to the provider, these revisions take place once per quarter and affect around 0.1% of websites. Still, such a pricing scheme works as a kind of black box.

	Model	Structure	Price range	Trial
Bright Data	Requests	PAYG, subscription	$1-$2,000	7 days for companies
Infatica	Credits	Subscription	$25-$240	5k req, 7 days
NetNut	Requests	Subscription	Not public	7 days for companies
Nimble	Requests	PAYG, subscription	$3-$3,000	Available
Oxylabs	Requests	Subscription	$49-$2,000	5k req, 7 days
Rayobyte	Requests	PAYG	$1.8	5k free req / month
ScraperAPI	Credits	Subscription	$49-$299	1k free credits / month, 7-day trial
Scraping Dog	Credits	Subscription	$40-$200	1k credits, 30 days
Smartproxy	Requests	Subscription	$30-$500	1k req, 7 days
SOAX	Requests	Subscription	$2.5-$2,200	Available
Zyte	Dynamic	PAYG, subscription	$1-not specified	$5 credits for 30 days

The table provides some interesting data points:

Scraper vendors prefer trials over paying as you go. Only Scraping Robot has PAYG as its sole pricing model, and Zyte starts requiring commitment after $100. Furthermore, some trials extend to free plans that refresh monthly.
Credit-based pricing usually targets customers with smaller needs. This is evident from looking at the price ranges of public plans.

Price Modifiers

To understand how exactly request-based and credit-based pricing models compare, we’ll have to explore the base price and available modifiers.

The CPM at $100 column shows how much 1,000 requests would cost when spending $100 with each participant. It may a little be biased against enterprise-minded providers, as they start to scale well at $1,000 and up.

	Base CPM at $100	Price modifiers
Bright Data	$3	A list of premium websites (2x)
Infatica	$0.09	JS rendering (10x), E-comm & SERP (10x), JS + E-comm/SERP (20x), LinkedIn (130x)
Oxylabs	$1.80	–
Nimble	$3	–
Rayobyte	$1.80	–
ScraperAPI	$0.49	Amazon (5x), SERP (30x), Social (30x), JS rendering (10x), premium IPs (10x), premium IPs + JS (30x), ultra premium IPs (30x), ultra premium + JS (75x)
Scraping Dog	$0.09	Google (5x), JS rendering (5x), premium IPs (10x), premium + JS (25x), LinkedIn (200x)
Smartproxy	$1	–
SOAX	$2.50	–
Zyte	From $0.10	Target (up to 10x), parsing (up to 3x), JS rendering (up to 15x), JS + parsing (up to 25x), screenshot (up to 25x)

NetNut – no public pricing available.

Credit-based pricing models can have huge multipliers reaching tens or even hundreds of times. These variables interact with one another: for example, you can toggle both JavaScript rendering and better quality proxies. From the user standpoint, having these options exposed can feel burdensome, as you need to experiment with parameters and mind the credit cost.

Having said that, they’re really efficient for basic websites that require neither residential proxies nor JavaScript rendering. The low baseline price also makes these scrapers look really good in marketing materials. However, for hard targets like G2, you’ll be likely to overpay.

Request-based models have the opposite problem: they’re really expensive if the target doesn’t bite. But given that these providers are often enterprise-oriented, different considerations kick in, such as scalability and unblocking success.

The Cost to Run Our Benchmarks

So, how much did we pay to complete the full 180,000 requests (10 * 6,000 * 3)? The graph shows aggregate costs, taking the rate of the closest suitable plan.

Three participants failed to unblock some targets consistently (at least 20% of the time) and use credit-based pricing. We didn’t want to speculate on the configuration, so we excluded the following from the graph:

Infatica: Canadagoose, G2, Lowe’s, Safeway, Walmart.
Scrapingdog: Allegro, Canadagoose, G2.
ScraperAPI: Allegro, G2.

Hover on a label to highlight, click to filter.

For general unblocking without JavaScript rendering or data parsing, Zyte delivered incredible value considering its performance results. The provider’s price was closer to entry-level APIs like Infatica and Scrapingdog than the premium competitors.

Smartproxy and Oxylabs also look compelling, more so if you need headless browsers or the bundled parsing features. And while this API may not be the most efficient choice in general, ScraperAPI’s prices for Amazon and Walmart in particular are definitely worthy of attention.

Conclusion

This concludes the report. Assuming that very few readers will reach this part, we moved the summary to the beginning. But since you’re here – thank you for getting to the end! If you have any questions, feel free to contact us through info at proxyway dot com or our Discord server.

Adam Dubois

Proxy geek and developer.

September 17, 2024

The post Comparing Popular Web Scraping & Proxy APIs appeared first on Proxyway.

Proxy Service Awards 2024

Adam Dubois — Mon, 20 May 2024 07:33:41 +0000

Research

Another year has passed in the proxy server industry. The midst of 2024 brings us to a thriving ecosystem: more accessible, diverse, and competitive than it’s been in a long time. Prices are dropping, new providers are entering the market, while the old ones are turning up the heat. Sounds intense, right? It sure is, and we’re all the better for it.

At Proxyway, we tried our best to capture the status quo in the annual proxy market research. And now, it’s time to take a moment and celebrate the proxy providers that stood out among all the rest.

These are 2024’s proxy service awards:

Best Proxies for Enterprise
Best Platform for Proxies
Best Value Provider
Contender of the Year
Best Entry Choice
Newcomer of the Year

Best Proxies for Enterprise – Oxylabs

Enterprise companies are a tough crowd, with scale and policies that put a burden even on the most established providers. These are the customers for whom you write SLAs, toll on certifications, and make sure that your proxy networks are spick and span – day in, day out. What’s more, the current focus on operational efficiency has made the selection procedure even more cut-throat than usual.

When it comes to enterprise-grade proxies, we’d be hard-pressed to find a stronger choice than Oxylabs. Throughout weeks of testing, its proxy networks barely ever skipped a beat and were faster than all competitors, sometimes several times so. We also received access to proxy pools that were among the largest in the market. In short, we documented a story that repeats itself year after year.

The best part is that Oxylabs doesn’t rest on its laurels. Compared to 2023, you’ll get more features (such as coordinate-level targeting), significantly lower rates, and even better performance. The last part is particularly impressive, considering how high the baseline already was. In fact, a better part of our tested providers are still catching up to the Oxylabs of yesteryear. Let that sink in.

Of course, the improvements weren’t for nothing. There are proxy vendors, both entrenched and emerging, that are breathing down Oxylabs’ neck, and competition in general is fiercer than ever. Today, at least when it comes to proxies, Oxylabs remains the premium provider to beat.

Best Platform for Proxies – Bright Data

It would probably surprise no one if we said that Bright Data always had great proxy servers. Large, robust, and feature-rich pools made the provider into the company it is. Despite taking a less central role nowadays, they continue to propel Bright Data’s broader platform forward.

And if there’s one thing that defines Bright Data in an industry where all gaps are closing, it’s the platform. We’ve criticized it for complexity and opaqueness; but after all these years, we have to admit that Bright Data’s tooling remains a north star for many providers aspiring to serve the most demanding clients.

It would be hard to pinpoint one thing about it. But maybe that’s exactly what we shouldn’t do. Unlike many others, Bright Data has built a coherent experience that really makes you feel like you’re using a platform rather than a collection of individual products bunched under one dashboard. Account-based pricing, one API for all proxy networks, and tools like Proxy Manager are but a few examples to illustrate our point.

And even if from this point onwards Bright Data decided to stop making improvements to its tooling, it would still take all but a select few competitors years to catch up. Acknowledging this, we’re happy to hand Bright Data our Best Platform for Proxies award.

Best Value Provider – Smartproxy

The concepts of cheap and expensive, despite being relative like most things, are straightforward to understand. For example, an expensive service carries the expectation of extra functionality, a more pleasant experience, and hopefully better performance. But will all this translate to better value for the price you’re paying? This is where a delicate and highly subjective balancing act begins.

Still, should we look for a combination of cost and benefit that covers a wide range of use cases, Smartproxy would probably be our best bet. Same as last year, or the year before it. Are we biased? Or maybe just lazy? That may be so, but give us a chance to explain ourselves.

After all these years, we still can’t find a provider that combines mid-market rates, premium-grade performance, and newbie-friendly user experience as well as Smartproxy. You can get proxies that work effectively as well or better than enterprise providers, without investing thousands of dollars to reach their bulk rates. The plans start at single digits, but even then you can make use of Smartproxy’s excellent customer service.

Are there truly no worthy competitors? Not exactly. After its pricing overhaul in August, SOAX looked mighty strong, but it unfortunately tanked our benchmarks. IPRoyal is steadily improving every year, with a non-expiring pricing model that’s hard to pass up. Even so, we can’t push ourselves to hand this award to anyone else yet because Smartproxy remains such a well-balanced choice for most customers.

Contender of the Year – NetNut

When NetNut last won the Contender of the Year award in 2021, it was in a very different position. The provider had a unique selling point (the static residential proxy pool), but its marketing was opaque and performance frankly uninspiring. Even worse, NetNut was forced to compete with the parent company’s other product lines that were bleeding money and attention away from the project.

Still, the provider had just launched two new proxy networks, and it seemed like it had the potential to compete with premium peers.

After three years, it looks like NetNut’s efforts are starting to come to fruition. Its owner, Alarum, shed all other initiatives to focus on the proxy server business. With more resources at hand (NetNut’s spending on IPs doubled from 2022 to 2023), the provider has grown, embellished, and solidified the proxy networks, to the point where they can stand up to the best competitors. Even the marketing has improved, which historically was NetNut’s weakest point.

The efforts seem to have paid off: as a public company, NetNut has been demonstrating an impressive growth trajectory. It made over $20M in revenue in 2023, over $8M in Q1 of 2024 alone, and has two new lines of products to scale. All things considered, NetNut of 2024 fits our definition of contender to a tee. Congratulations!

Best Entry Choice – Webshare

We’ve mostly covered enterprise and mid-market so far; but what about customers who are just starting with these proxy server shenanigans? Whose needs are still small, intermittent, or who’d prefer to avoid expensive commitments until they’re more accustomed to the tool? This is the domain of entry-level providers, and Webshare stands at the forefront.

Two things make Webshare stand out. First, the flexibility. We know very few alternatives that allow customizing so many aspects of the service, all on your own. You can get a thousand static proxies from thirty locations, or just one that changes every five minutes. The second is price. Once you dress down the proxies bare, it’s possible to still buy perfectly functional servers for less than $0.1 apiece (even ISP proxies).

Alongside proxies, Webshare bundles useful features for both less technically experienced and developers. The former can choose to enable optional rotation without setting it up by themselves. The latter get surprisingly detailed statistics and programmatic access through an API that punches above its weight. And perhaps most crucially for this type of service, you can try it out without any commitment or time constraints by claiming the 10 free proxies offered to all users.

There is one caveat: Webshare’s magic extends to datacenter and ISP proxy servers. The residential proxies, though strong on their own, currently lack the oomph compared to Smartproxy, IPRoyal, and even Dataimpulse at $50 and up. However, taken as a whole, we consider Webshare to be a top entry choice, well deserving of this award. Well done!

Newcomer of the Year – Dataimpulse

The decision of whom to give the Newcomer of the Year award this year was tough. There were three strong nominees from different market segments: Dataimpulse, NodeMaven, and Nimbleway. In the end, we chose the former, as entry-level providers are the least represented this year, with very little doubt that we’ll be seeing the latter two in years to come.

Why Dataimpulse? In many ways, it reminds us of PacketStream: a self-grown pool of proxies, small barriers to entry, and prices that wipe away all but the biggest objections. But since 2020, when we awarded PacketStream for the very same reasons, its potential never materialized. The provider seemed content with the status quo, choosing to onboard new resellers rather than palpably improve the service. A pity.

Where PacketStream languished, Dataimpulse looks intent to push forward. Between August 2023 and March 2024, the provider has notably improved in all metrics. Not only that – it’s also launched two new proxy networks, both exerting the same price pressure on the competition.

In this context, two questions arise. Can Dataimpulse keep up the pace? And can it retain its defining qualities while doing so? This remains to be seen. But from what we’ve witnessed already, we believe the Dataimpulse is worthy of this award. Here’s to potential.

Adam Dubois

Proxy geek and developer.

May 17, 2024

The post Proxy Service Awards 2024 appeared first on Proxyway.

Proxy Market Research 2024

Adam Dubois — Mon, 06 May 2024 13:19:00 +0000

Research

Proxy Market Research is the most comprehensive public report on the proxy server industry, released annually since 2019. It covers the market trends, provides large-scale performance benchmarks, and compares leading proxy server providers.

You’ll find this report useful whether you’re a proxy vendor looking to improve your product, a customer shopping around for proxies, or simply someone interested to see where the industry stands and how it’s changed throughout the years.

This year’s report:

Includes 13 proxy providers, from familiar names like Bright Data and Oxylabs to newcomers like Dataimpulse and Nimbleway.
Investigates three proxy server types: residential, mobile, and rotating datacenter proxies.
Draws insights from a questionnaire sent to all participants, as well as our own know-how.
Provides performance data based on millions of connection requests, made even more accurate thanks to our improved infrastructure.

Changes and improvements

Testing Major ISP Proxy Services

Chris Becker — Fri, 13 Oct 2023 14:27:43 +0000

Research

ISP proxies, the curious combination of datacenter and residential proxy servers, have always fascinated us. However, we never really had the time to sit down and take a proper look at them on a larger scale. That’s what this report is about.

We grabbed thousands of ISP proxies from major providers and dissected them the best we could. How big and diverse can these networks be? How do they perform compared to datacenter and residential addresses? Are there any gotchas you should be aware of? (Hint: there are!) Let’s find out!

Key Findings

ISP proxies are usually made by renting IPs from brokers and paying residential internet service providers to announce the IPs on their networks (ASNs).
Major proxy providers prefer selling ISP proxies in pool based formats rather than lists.
Their pools in the US can exceed 100,000 IPs. But compared to peer-to-peer residential proxies, they’re hundreds of times less diverse.
Providers manage to contract with top American ASNs, mostly Comcast and Sprint. However, the IP’s owner often doesn’t change, which can cause issues with databases like IP2Location.
Connections to ISP proxies succeed 99.8% of the time, and they’re around 35% faster than residential proxy servers.
It may be challenging to find locations outside the US or major European countries, and even harder to get IPs in specific cities.
ISP proxies cost 25% more per gigabyte than residential networks. That said, it is possible to get them cheaply – if you’re okay with the restrictions.

A Primer on ISP Proxies

ISP proxies are IP addresses you can use to hide your own. They’re hosted in data centers and inherit the fast and stable connection of datacenter proxy servers. At the same time, they’re associated with consumer internet service providers and not cloud hosting companies. This gives ISP proxies a solid reputation in the eyes of websites, preventing premature IP-based restrictions or outright blocks.

ISP proxies initially found the most success in the sneaker scalping community, which valued connection speed, unlimited bandwidth, and faced well-protected websites. Predictable uptime and more transparent procurement are the other characteristics that make this proxy type a desirable choice, even in formats that meter traffic use.

How Are ISP Proxies Sourced?

The process usually involves these steps:

Rent IP addresses from an IP broker (or in some cases, bring your own).
Find an ISP that’s willing to share its IP space.
Pay a hefty amount and sign a letter of authorization to announce your IPs on the ISP’s network. This results in the IPs acquiring the ASN (identifying number) of that internet service provider.
Get major IP databases to update their records with the new information.

Some proxy vendors manage to source IPs directly from internet service providers. For example, NetNut pays them for sharing bandwidth through DiviNetworks (similarly to how Honeygain pays end users). This is hard to achieve with major ISPs – but as we’ll see, not impossible.

Participants & Methodology

We approached multiple major providers that offer ISP proxy services, disclosing the methodology and targets in advance. In the end, our line-up included eight companies. Here are the ISP proxy formats they offer:

	Proxy pool	Shared list	Dedicated list
Bright Data	✅ (10,000 IPs)	✅ (~80,000 available at the time of testing)	✅ (~50,000 available at the time of testing)
GeoSurf	✅ (~10,000 IPs)	❌	✅ (not public)
NetNut	✅ (<1M IPs)	❌	❌
Oxylabs	✅	❌	✅
Rayobyte	✅	✅	✅
Smartproxy	✅ (~20,000 IPs)	❌	❌
SOAX	✅ (100,000 IPs)	❌	❌
Webshare	✅ (Up to 20,000 IPs/month)	✅ (>25,000 available at the time of testing)	✅ (>1,000 available at the time of testing)

Benchmarked Products

Of course, it’d be too much to cover all the formats. So we had to make compromises.

Location	Format	Exceptions
US	Proxy pool	Webshare, Bright Data (5,000 shared IPs)

The initial idea was to benchmark multiple locations. But looking at what the providers had to offer, we decided to stick to the US.

We also wanted to get our hands on as many IPs as we could. This ruled out dedicated IPs, as providers would be unwilling to give us exclusive access for a week. Instead, we opted for proxy pools where possible, given that some advertised as many as one million addresses!

Though Bright Data and Webshare offer pool-based access, we chose their shared IP lists instead. Bright Data’s pool implementation randomly assigns 10,000 IPs from dozens of countries, which resulted in only 256 US IPs. Webshare, on the other hand, gives up to 20,000 IPs per month. But this translates to only five ports rotating every five minutes, which wasn’t enough for our purposes.

Luckily, both providers have systems that automatically translate IP lists to rotating endpoints. We chose an arbitrary number of 5,000 shared IPs in the US – in hindsight, we probably could’ve tested even more.

Testing Methodology

Web scraper: custom Python script, requests sent from Germany
Target: Nearest data center of a global CDN (chosen by the CDN based on IP location)
Duration: 7 days
Requests: 100,000/day (~780,000 total)
IP databases: MaxMind (IP location, ASN name), IP2Location (ISP, ISP usage type), IPinfo (ASN type)

Diving into ISP Proxy Pools

Let’s have a look at how big the ISP proxy networks are and what they’re composed of.

ISP Proxy Networks Can Get Really Big, But There’s Not Much Variety

Some of our tested providers returned over 100,000 IPs in the US. Considering that ISP proxies are supposed to always be online (as opposed to residential IPs that come and go), you get really large proxy pools to work with.

It’s little surprise that NetNut had the most proxies. The company is known as one of the original and largest vendors of static residential proxies.

What did surprise us was Oxylabs. The provider is very vague in advertising its ISP proxies, so we expected less. SOAX subverted our expectations as well, but not for the better. The provider advertises 100,000 IPs, and that’s what we hoped to get. Instead, the first 700,000 requests returned only ~4,300 IPs, boosting the IP count with the last 80,000 connections.

	ASNs	C-class subnets	IPs/subnet
NetNut	3	567	254
Oxylabs	19	515	236
Smartproxy	2	64	256
SOAX	8	53	200
GeoSurf	3	35	253
Rayobyte	4	17	180

Note the number of subnets. It reveals how uniform ISP proxy networks are. And also, how little variety they offer. For comparison’s sake, we also tested a peer-to-peer residential proxy network in parallel. We found 82,000 unique IPs that were scattered throughout 17,000 different subnets – a usual number for this proxy type.

In some cases, dozens (or even hundreds) of subnets can come from just one ASN. Naturally, they’ll be structured into neat blocks. With ISP proxies, ensuring diversity is a big challenge.

	Unique IPs	ASNs	C-class subnets	IPs/subnet
Bright Data	5,148	11	234	22
Webshare	5,258	4	56	94

When you don’t exhaust a provider’s IP stock, the uniformity is less striking. Bright Data’s system did a great job at distributing our proxies throughout over 200 networks, providing around 22 IPs per subnet. The ratio would’ve likely regressed to the mean on a larger scale.

Most ISP Proxies Really Do Come from Residential ASNs

We checked the ASN type data point in the IPinfo database. It shows which ASNs function as ISPs, business or hosting networks.

The majority of IPs were under residential ASNs. That said, some providers (such as Oxylabs and SOAX), should keep their pools cleaner. Maybe some lease agreements have expired and the IPs reverted to their original ASNs? In any case, that’s not ideal.

For example, SOAX had a subnet from Charles River Corporation, a hosting company. Looking at WHOIS data, the subnet was likely assigned to Cogent before.

In addition, while the IPs themselves were predominantly located in the US, some ASNs weren’t US based. Oxylabs’ pool included proxies under Bite Lietuva, a Baltic ISP, and Bright Data had IPs with a Bulgarian AS number. We’re unsure if this has much significance if the other aspects are in order, so we decided not to pay much attention to it this time.

Providers Are Able to Source Surprisingly Many IPs from Top ASNs

We looked for IPs under the 10 largest American ASNs: Comcast, Charter, AT&T, Verizon, Cox, Altice, Lumen (CenturyLink), Frontier, Mediacom, and Astound (RCN).

Conceptually, an ISP proxy should do its job as long as it’s associated with a network that databases treat as residential. But as with all things in life, here you also want to get the best ISPs possible. This may be a vanity metric for a big chunk of jobs, but it begins to matter once you start accessing hardened websites.

Some providers make it a priority to fill their pools with major ASNs. This is especially evident with Webshare, Rayobyte, and SOAX. Others, like Smartproxy and Bright Data, aim to strike a balance, while Oxylabs and NetNut focus on IP quantity to ensure quality through a cleaner usage history.

For the fun of it, let’s see which top ASNs are the most popular.

The ASN and Owner of These IPs Often Don’t Match – Which May Be a Problem

The sourcing practices of ISP proxies create a frequent situation where the ISP (owner of an IP address) and the ASN (the network it’s on) don’t match. It’s not unusual for databases to identify the ASN as Verizon and the ISP as IPXO, an IP broker.

Here’s what IP2Location had to say about our tested ISP proxy pools. We mapped the relationship between ASNs and ISP manually, and some cases were less than obvious, to say the least. So, you should treat the matching ASN & ISP data point as an approximation:

	IPs with matching ASN & ISP
NetNut	59%
Oxylabs	35%
SOAX	21.5%
Bright Data	2.5%
GeoSurf, Rayobyte, Smartproxy, Webshare	0%

Across the board, few IPs had matching ASNs and ISP organizations. NetNut led the way, which is expected considering its bandwidth sharing model. Top 10 ASNs were the least likely to match. Having said that, SOAX somehow managed to source over 2,000 IPs directly from Comcast – an impressive feat but also an exception to the status quo.

When is the ASN-ISP match relevant? We’ve learned that anti-bot systems may monitor inconsistent IPs more closely. Of course, things aren’t as simple: factors like IP usage history and the web scraper’s fingerprint matter greatly.

The choice of an IP database also makes a difference. We found that Maxmind is naive with organizations, often matching them with ASNs. IP2Location and IPinfo, on the other hand, are quick to catch any inconsistencies.

IP2Location has a field called usage type that classifies an organization based on its business function. Many of these ISPs are IP brokers and hosting companies, so they get identified as data centers. And once that happens, the database tends to also mark their IPs as datacenter proxies. Scamalytics, an IP abuse service, does something similar: it relies on the ISP’s organization to evaluate fraud risk.

	ISP usage type – landline or mobile
Oxylabs	99%
Bright Data	76.85%
NetNut	59% (the rest – educational)
Rayobyte	58.65%
SOAX	28.59%
Webshare	15.45%
GeoSurf, Smartproxy	0%

As you can see, having a residential ASN isn’t enough to fool IP identification services. Fortunately, not every database tracks this, and not every user considers it important to take action.

Infrastructure Performance

Knowing that ISP proxies are hosted in data centers, we can expect them to have datacenter-level performance. We tried to verify this by measuring their success rate, response time, and download speed.

ISP Proxy Infrastructure Works Nearly Without Fail

The heading says it – at their best, the ISP proxy infrastructures reached an almost perfect success rate. Even the median result was 99.81%, promising very few connection errors. For some reason, NetNut had unusually many timeouts, which brought down its results.

	ISP (8 providers, 700k requests each)	Residential (11 providers, 500k requests each)	Datacenter (6 providers, 50k requests each)
Median success rate	99.81%	98.29%	99.94%

ISP proxies look well compared to other proxy types. Though residential proxy networks have improved massively over the years, they’re still less reliable. With datacenter proxies, this depends on the server they’re hosted on and is hard to generalize. That said, our benchmarks produced similar results.

These Proxies Outpace Residential and Even Datacenter Proxy Servers

The ISP proxies had a median response time of 0.64 seconds, which is great. If you’ve ever used hyper-optimized IPs for sneakers, this may not seem that fast to you – but remember, we tested thousands of IPs, and our hardware was halfway across the world.

The three best performing proxy pools showed nearly identical results and were over twice faster than the slowest providers. Note that our benchmark was biased towards providers that had infrastructure in Europe. Rayobyte’s load balancing server was located in the US, so it took around 150 ms longer to reach from Germany.

	ISP (8 providers, 700k requests each)	Residential (11 providers, 500k requests each)	Datacenter (6 providers, 50k requests each)
Median response time	0.64 s	1 s	0.87 s

Compared to other proxy types, ISP proxies had the lowest response time in the US: they were 35% faster than residential proxy servers and 25% quicker than rotating datacenter proxies.

Throughput Isn’t an Issue – Most of the Time

Another benefit of ISP proxies is the throughput you get. We tested 10 random IPs from each proxy network, and their median download speed reached 10.57 MB/s. Of course, the results will largely depend on your own connection quality. But the point is – ISP proxies are fast.

They’re also highly variable. The fastest provider, Oxylabs, reached over 30 MB/s on average and actually beat our download speed without proxies. On the other hand, even the fastest proxy from Rayobyte’s 10 IPs ran at only 3.5 MB/s. This raises a three-pronged question: are some ISP proxies extremely overloaded, do providers limit throughput, or were we just unlucky?

Feature Overview

Let’s have a look at the functionality of ISP proxy services – more specifically, their location variety, protocol support, and rotation options.

ISP Proxies Are Closer to Datacenter Than Residential IPs in Location Variety

Being hosted on servers and not consumer hardware, ISP proxies face the same challenges as datacenter addresses: 1) it’s hard to get them in many countries, 2) it’s even harder to offer different cities within a country. This is evident in the table below:

* Dedicated proxy list only
	Available countries	Country selection	State selection	City selection
Bright Data	~50	✅	❌	✅
NetNut	30+	✅	✅	✅
Oxylabs	10+	✅	❌	❌
GeoSurf	US, UK, AU, GB, DE, FR, SG	✅	❌	❌
Rayobyte	US, UK, Germany, Canada*	✅	❌	✅*
Smartproxy	US, Hong Kong, Canada, France	✅	❌	❌
Webshare	US, Italy	✅	❌	❌
SOAX	US	❌	❌	❌

Still, some providers make it possible, Bright Data and NetNut being prime examples. Otherwise, the main priority is to procure IPs in the US, then a major Western European location, and branch out to other continents afterwards.

City selection often works only in the US. Everywhere else, you’ll get the capital and up to three other cities, if you’re lucky (or the country is big). Peer-to-peer residential proxies are much more flexible in this regard.

Some ISP Proxy Services Have to Rotate – But Why?

ISP proxies can be static, and it’s one of the main qualities that distinguishes them from residential proxy servers. But most participants also have rotating formats for web scraping tasks.

	Formats that rotate	Frequency
Bright Data	All	Every request, customizable w/ Proxy Manager
Webshare	All	Every request, 5 or 20 mins (pool only)
GeoSurf	Proxy pool	30 minutes
NetNut	Proxy pool	Every request
Oxylabs	Proxy pool	Every request, 5 hours (forced)
Rayobyte	Proxy pool	Every request
Smartproxy	Proxy pool	Every request
SOAX	Proxy pool	Every request, 24 hours (forced)

What we don’t understand is why some of the ISP proxy pools force rotation. There may be good reasons, but without knowing them, the decision looks arbitrary and limiting on purpose.

Bright Data and Webshare have really flexible systems where you can buy an IP list and get it to rotate without writing the logic yourself. Bright Data’s system puts all products behind an endpoint by default, and you can specify a particular IP by adding a parameter.

Webshare does it the other way around: it provides a list and offers the ability to create an endpoint for any product on its dashboard. Alternatively, you can get new IPs by automatically refreshing your list at set intervals (for example, 15 minutes). The scale and frequency depends on your budget.

Pricing Approaches

Let’s take a look at how ISP proxy providers structure their pricing and how much you’ll have to shell out for access.

You Likely Won’t Be Able to Pay as You Go

Only Bright Data offers the pay-as-you-go payment model. This contrasts sharply with residential proxy services where a no-commitment option has become an industry standard.

Pool Based Formats Have a Steep Entry Cost

	Proxy pools		Proxy lists
	Entry price	Traffic	Entry price	IPs
NetNut	$350	20 GB	–	–
Oxylabs	$340	20 GB	Not public	Not public
SOAX	$99	15 GB	–	–
Webshare	$94	250 GB	$6	20 IP
Smartproxy	$28	2 GB	–	–
Bright Data	$15	1 GB	$15.5	1 IP
Rayobyte	Not public	Not public	$12.5	5 IPs
Average	$154		$9.7

Without an ability to pay without committing, you’ll have to rely on subscription plans. The bad news is that their entry fee is pretty steep – especially with business-minded providers like NetNut and Oxylabs, which likely treat ISP proxies as a premium product.

The situation is significantly better with ISP proxy lists. However, their price per unit is still high enough that you won’t get much. Webshare’s value proposition is the best without looking at the details (which we’ll do soon enough).

It’s Possible to Get ISP Proxies for as Little as $0.2/GB – But Residential Proxies Still Cost Less on Average

At 100 GB of data, the average gigabyte price for ISP proxy pools is $9.2. Compared to the pricing of residential proxy services by these same providers, the rates are 25% more expensive. That said, if you’re okay with restrictions, ISP proxy traffic can also get pretty cheap.

	5 GB	20 GB	100 GB	250 GB	500 GB	1,000 GB
Bright Data	$15	$12.75	$11.25	$10.50	Custom	Custom
GeoSurf	–	$10	$8	$7	Custom	Custom
NetNut	–	$15	$10	$8	$6.5	$5
Oxylabs	–	$14	$11	–	–	$6
Smartproxy	$14	$11	$9.5	$8.5	$7.5	$6.5
SOAX	–	$6.17	$5.68	$4	$3.2	$3
Webshare (20,000 IPs/mo)	–	–	–	$0.52	–	$0.2
Webshare (5,000 IPs)	–	–	–	$6.6	–	$2.31

How? Webshare. The 20,000 IP/month plan translates to $0.52/GB at 250 GB and $0.2 at 1 TB of data. But 60 IPs per hour (five ports that rotate every five minutes) might not be enough for large scale tasks.

Our list configuration increases the price tenfold, as you’ll probably want more than the base 500 threads (+37.5% to the price). It’s possible to further choose network priority (+25% on top), a monthly refresh (+20%), and even IPs that have been verified to work with Google (x10). The takeaway is that you should be mindful of the add-ons – and that comparing Webshare to more straightforward pricing models is hard.

Otherwise, SOAX handily beats most alternatives, Smartproxy tries to position itself the second value choice, and NetNut only gets started at 1 TB.

Chris Becker

Proxy reviewer and tester.

November 7, 2023

The post Testing Major ISP Proxy Services appeared first on Proxyway.

In-Depth Look into Popular Proxy APIs (Web Unblockers)

Chris Becker — Mon, 18 Sep 2023 08:39:05 +0000

Research

This investigation takes a look at five popular proxy APIs like Bright Data’s Web Unlocker and Oxylabs’ Web Unblocker. We compare their features, pricing models, as well as ability to unblock websites behind major bot protection systems, such as DataDome or Shape.

Key Findings

Proxy APIs integrate using the hostname:port format and remove the burden of managing proxy servers, running headless browsers, or dealing with blocks.
All five participants managed to open protected websites over 90% of the time. Shape’s antibot gave them the most trouble, followed by the photo-focused social media platform.
Proxy APIs allow modifying the request to an extent, such as sending custom headers. Several APIs offer structured data functionality, and proxy vendors like Bright Data and Oxylabs support precise filtering options. Their main drawback is limited interaction with dynamic pages.
It makes more sense to pay for requests than traffic, unless you’re scraping small pages or API endpoints. Zyte’s dynamic pricing is very efficient for pages that don’t need JavaScript rendering.

A Primer on Proxy APIs

With websites getting harder to scrape, proxy providers have developed a new type of service called proxy API. It integrates like a regular proxy server. But in the backend, the API combines multiple IP types, an intelligent proxy management layer, and website unblocking mechanisms. If a request fails (encounters a block or CAPTCHA), the API adjusts its configuration and tries again until it succeeds. Some tools even have smart error handling to identify non-obvious failures like an empty 200 response.

So in addition to buying a bunch of IPs, you get a tool that takes care of CAPTCHAs, bot protection systems, and often even renders JavaScript on its own. This reduces web scraping’s elaborate cat-and-mouse game to sending requests and receiving the page’s HTML. Most vendors promise a nearly perfect success rate – even with something as simple as a cURL command.

That said, you shouldn’t treat a proxy API as just a CAPTCHA solver. Though they do remove the need to engage with streetlight marking and button holding, this is usually achieved by avoiding the challenge and not brute-forcing through it. Pages that have configured CAPTCHAs to pop-up every time may still need your attention.

How do you use a proxy API? The process differs little from any other rotating proxy server: there’s a hostname and port with authentication details. Then, you can append various parameters – such as location preferences – to the credentials or send them as a custom header. The proxy API intercepts the request and fetches the page based on your configuration.

In our previous research on web scraping APIs, we noticed that some had the ability to integrate as a proxy. But lately proxy APIs have been emerging as a separate category – whether for marketing purposes or based on real technical considerations. You’ll often find them called unblockers or smart proxy managers.

Participants & Methodology

We approached multiple companies with a proposal to participate in the research. Five providers agreed. For transparency, we disclosed which websites we’d test in advance, so it was possible to prepare accordingly.

These are the participants:

Bright Data – one of the biggest companies in the field of data collection. We tested its Web Unlocker, together with SERP API for Google. While these are two separate tools, Bright Data has a unified system which lets customers use any product without getting a new subscription. Bright Data’s been offering Web Unlocker for at least several years, and it was the company’s main growth driver in 2022.
Crawlbase – a well established provider of web scraping tools with over 45,000 paying customers. We tested Crawling API – primarily a web scraping API with a proxy mode. We chose it over Crawlbase’s designated proxy API, Smart Proxy, because Crawling API allows paying as you go and doesn’t lock features behind pricing tiers. Otherwise, there’s little difference between the two – they even share the same parameters.
Oxylabs – Bright Data’s main competitor in the proxy server market with excellent infrastructure. We tried Web Unblocker – a not-exactly-new proxy API that looks new because it rebranded from Next-Gen Residential Proxies in early 2023.
Smartproxy – another major proxy provider looking to branch out into web scraping tools. We tested Site Unblocker – a very recent proxy API launched in July 2023.
Zyte API – a long-standing data collection company that focuses on e-commerce and maintains several popular open source tools like Scrapy. We wanted to use Smart Proxy Manager, which is one of the oldest proxy APIs. But it turns out Zyte is sunsetting the tool. So we ended up testing Zyte’s API after an unintentional bait-and-switch. As a silver lining, Zyte plans to add a proxy mode to Zyte API soon.

Testing Methodology

We used a custom asynchronous Python script to send requests. We ran multiple tests with the US versions of target websites, making 1 request per second. Our computer was located in Germany. We made around 1,800 requests for each target using one proxy API. To verify that a request was successful, we looked at its response code, size, and the page’s title. Some APIs returned their own status codes in addition to website responses; we didn’t use them.

Unblocking Popular Websites & Anti-Bot Systems

We wanted to see how good the APIs are at their main job – unblocking challenging websites. We selected seven targets protected by various anti-bot systems:

	Anti-bot	Rendering required?
Amazon	In-house CAPTCHA, returns empty 200-coded responses	❌
Google	reCAPTCHA	❌
Photo-focused social media network	In-house protection, asks for login if triggered	✅ (We accessed the web interface and not the GraphQL endpoint)
Kohls	Akamai	✅
Nordstrom	F5/Shape	✅
Petco	DataDome, Cloudflare	❌
Walmart	Akamai, FingerprintJS, PerimeterX, ThreatMetrix	❌

Which Target Was the Hardest to Unblock?

That’d be Nordstrom, which is behind Shape. In addition, it took the longest to return results, even though Kohls also required JavaScript and its pages weighed 3 MB – five times more. The social media network was another tough cookie for some, redirecting the APIs to its login page.

	Avg. success rate	Avg. response time
Kohls	99.01%	28.97 s
Amazon	98.65%	5.73 s
Walmart	96.38%	12.68 s
Google	96.00%	5.41 s
Petco	95.48%	11.20 s
Photo-focused social media network	92.19%	19.74 s
Nordstrom	80.16%	33.1 s

How Did the APIs Do?

All participants managed to succeed at least nine times out of ten, so the answer is – pretty well:

Zyte had a particularly strong showing, both in terms of success rate and response time. However, the provider did have to fix its social media scraper during our tests to perform as well as it did.
Bright Data was amazingly fast with targets that required JavaScript, but it also slowed down to a crawl with Petco, which others had no issues with.
Oxylabs and Smartproxy prioritized success rate and would’ve aced the tests if not for Nordstrom. However, their headless implementations were among the slowest.
Crawlbase was relatively fast, but it failed more requests across the board, even with non-problematic targets like Google.

	Oxylabs	Bright Data	Smartproxy	Zyte	Crawlbase
Avg. success rate	93.37%	94.65%	93.79%	97.82%	90.00%

	Oxylabs	Bright Data	Smartproxy	Zyte	Crawlbase
Avg. response time	20.43 s	15.05 s	20.05 s	11.70 s	16.21 s

Breakdown by Individual Target

Amazon is one of the biggest web scraping targets. However, it’s not that hard to unblock – companies are still able to scrape the platform at scale with datacenter addresses. This shows looking at the results – everyone except Zyte had nearly perfect success rates.

	Avg. success rate	Avg. response time
Crawlbase	100%	11.05 s
Bright Data	99.54%	5.59 s
Oxylabs	99.17%	4.14 s
Smartproxy	99.08%	3.99 s
Zyte	95.64%	3.90 s

Google is another major target that’s inspired a whole subset of web scrapers called SERP APIs. While more reliant on IP quality, its protection is also not that challenging to bypass. Only Crawlbase had some issues that manifested in 520 error codes.

	Avg. success rate	Avg. response time
Zyte	100%	2.16 s
Bright Data	100%	4.63 s
Smartproxy	99.94%	6.91 s
Oxylabs	99.91%	6.66 s
Crawlbase	78.61%	6.71 s

There are two ways to access the photo-focused social media platform: via a hidden API endpoint or the front-end. The former is actually preferred, as it doesn’t require JavaScript rendering. But we wanted to load the website fully.

This social media platform was among the hardest nuts to crack. Bright Data was extremely fast, but 15% of the requests were redirected to login. Oxylabs and Smartproxy loaded the pages nearly perfectly at the expense of speed. Crawlbase accomplished neither very well, and Zyte aced both.

	Avg. success rate	Avg. response time
Zyte	99.61%	19.78 s
Smartproxy	99.79%	27.92 s
Oxylabs	99.72%	28.39 s
Bright Data	85.84%	5.16 s
Crawlbase	76.01%	17.45 s

Kohls relies on JavaScript to load, which reflected in the response time. Other than that, Akamai failed to pose a challenge for any proxy API. Bright Data’s Web Unlocker used some voodoo magic (or caching) to outpace all competitors by five to eight times.

	Avg. success rate	Avg. response time
Bright Data	99.78%	4.98 s
Smartproxy	99.11%	41.89 s
Zyte	99.10%	29.44 s
Crawlbase	98.38%	25.44 s
Oxylabs	98.66%	43.08 s

Nordstrom is protected by Shape and requires JavaScript – a scary combination given that Shape’s one of the hardest anti-bot systems to overcome. Crawlbase allowed us to run only one request per minute, making the scraping process significantly more drawn out than we intended it to be.

Oxylabs and Smartproxy found Nordstrom especially challenging; we saw a large number of 520 codes and internal errors signaling too many retry attempts. Overall, Zyte did the best job in both performance metrics.

	Avg. success rate	Avg. response time
Zyte	99.38%	20.42 s
Crawlbase	89.16%	40.58 s
Bright Data	88.98%	39.97 s
Smartproxy	62.07%	31.34 s
Oxylabs	61.20%	33.19 s

DataDome was a surprisingly easy obstacle to overcome. Add in the fact that Petco doesn’t need JavaScript, and you can scrape it quickly and efficiently. At least that’s the case with Smartproxy and Oxylabs. For some reason, Bright Data struggled, especially looking at the response time. Maybe it had to spin up headless instances to unblock the website?

	Avg. success rate	Avg. response time
Smartproxy	99.88%	2.79 s
Oxylabs	99.56%	2.91 s
Zyte	94.68%	3.49 s
Crawlbase	91.75%	5.96 s
Bright Data	91.55%	40.82 s

Walmart actually uses four different anti-bot systems, which is insane. The company evidently doesn’t want to be scraped. That said, even these extreme measures failed to stop the proxy APIs. Some took time to retrieve responses, while others like Zyte were extremely fast. But in general, all tools sustained a success rate of over 95%.

	Avg. success rate	Avg. response time
Bright Data	96.83%	4.23 s
Smartproxy	96.64%	25.50 s
Zyte	96.53%	2.69 s
Crawlbase	96.50%	6.31 s
Oxylabs	95.38%	24.66 s

Features and Limitations of Proxy APIs

Proxy APIs are designed to be a drop-in replacement for proxy servers. Accordingly, their main expected features are location targeting and ability to establish sessions.

As we can see below, all participants support this. Oxylabs, Smartproxy, and Bright Data are foremost proxy server providers, so they can afford offering granular location settings that reach co-ordinate and ISP level.

	Oxylabs	Bright Data	Smartproxy	Zyte	Crawlbase
Localization	Countries (all), states, cities, co-ordinates	Countries (all), states, cities, ASNs	Countries (all), states, cities, co-ordinates	Countries (50)	Countries (26)
Sessions	✅	✅	✅	✅	✅
Proxy selection	Automated	Automated	Automated	Automated	Automated, with an option to route via TOR

But proxy APIs aren’t just regular proxies. They mediate all communication with the target, which means you’re inevitably giving away some level of control. Let’s see how you can interact with the APIs and what’s out of their reach. We’ll omit Zyte for now, as it’s unclear which features will carry over to the proxy format.

	Oxylabs	Bright Data	Smartproxy	Crawlbase
JavaScript	✅ (options to render JS, return page screenshot)	✅ (handled automatically)	✅ (options to render JS, return page screenshot)	✅ (different API token, option to return page screenshot)
Request modifiers	Custom headers, cookies	Custom parameters for search engines	Custom headers, cookies	Custom headers, cookies
Page interactions	POST requests	❌	POST requests	POST requests, wait for load, scroll
Other		Asynchronous requests, parsers for search engines		CSS selectors, parsers for select websites

Bright Data’s implementation is the purest – it doesn’t expose anything beyond expected proxy functionality, with the exception of an asynchronous API endpoint (no longer a proxy!). The tool even decides when to render JavaScript on its own, which makes it very simple to use.

Bright Data’s specialized SERP API is a different beast – it offers custom parameters for building the request, such as search query, pagination, and location. It can also parse various properties of Google and other search engines for structured data. Most of this is achieved by appending parameters to the URL.

Search engines aside, the proxy APIs of Oxylabs and Smartproxy are more versatile: they accept custom cookies and request headers. There’s also an option to send POST requests with form or other data. And, you can opt to receive a screenshot instead of the HTML source.

Crawlbase’s Smart Proxy inherits the parameters from the provider’s other APIs. So in addition to achieving everything mentioned above, you can also interact with the page by waiting for it to load or scrolling down. Like Bright Data, Crawlbase offers parsers for several search engines, social media, and e-commerce websites, with an option to extract particular CSS elements from any website.

Main Limitation of Proxy APIs

The main drawback of proxy APIs lies in their inflexibility with JavaScript-dependent content. Most can render JavaScript (which is great), but they don’t expose the parameters needed to interact with the page. So you may not be able to properly scrape some websites that need extra time to load or depend on scroll events to reveal all content.

You’d think this could be solved by simply plugging the proxy API into Puppeteer or another headless library. But with a few exceptions, the APIs are incompatible by design.

Navigating the Different Pricing Models

Though some providers categorize proxy APIs as proxy servers (and not web scrapers), it doesn’t mean they follow the same conventions. Let’s have a look at the table below:

	Oxylabs	Bright Data	Smartproxy	Crawlbase	Zyte
Format	Traffic	Successful requests	Traffic	Successful requests	Successful requests
Modifiers	❌	Premium domains, city & ASN targeting	❌	JS rendering	Dinamically adjusted by target difficulty, JS rendering
Max price difference	x1	x2 + $4/CPM	x1	x2	x30+

There are several things to take away. First, that there’s no one pricing format. Three providers charge for successful requests, which is the standard for web scraping tools. Oxylabs and Smartproxy, on the other hand, try to continue with the proxy paradigm where residential and mobile proxy networks usually monitor traffic use.

The second aspect to look at is price modifiers. The most frequent multiplier is JavaScript rendering, as a full browser is computationally much more taxing than an HTTP library. Once you enable it, Crawlbase doubles the rate, and Zyte becomes an order of magnitude more expensive.

Zyte’s pricing is interesting in general. The provider adjusts request cost dynamically, based on the website’s difficulty and optional parameters. This means your rates may become more or less expensive over time, even when accessing the same website. There’s a calculator on the dashboard where you can check how much a request will cost.

Bright Data also segments pricing based on targets, but the approach here is simpler – you can enable a list of premium domains like Zillow or Nordstrom that cost an additional $4 per 1,000 requests.

Which Approach Makes More Sense?

It depends on the provider’s rates. But given that proxy APIs are supposed to be an upsell to residential proxies, they tend to be pretty expensive when metered by traffic. In this case, we’re partial to request-based pricing, especially if the website has large page sizes.

Among the five participants, Zyte’s dynamic pricing proved to be the least expensive in scenarios that didn’t require JavaScript. Otherwise, the other request-based providers gave it a tough competition. The traffic-based tools were less cost-efficient, but we could see them shine with small pages or when scraping API endpoints.

Chris Becker

Proxy reviewer and tester.

September 22, 2023

The post In-Depth Look into Popular Proxy APIs (Web Unblockers) appeared first on Proxyway.