Oxylabs Unifies Scraper Line-Up, Introduces AI Copilot
Inspired by a certain dev platform, the feature can structure any website using natural language instructions.
- Published:
Oxylabs, the Lithuanian provider of web scraping infrastructure and services, has revised its line-up of web scraping APIs, fitting them with more AI functionality and reducing the price.
Formerly segmented by website category into SERP, e-commerce, and general-purpose APIs, the scrapers now fall under one Web Scraper API. This is great news for customers with diverse web scraping needs, as they’ll no longer need to buy separate subscriptions.
But the biggest highlight is the newly-added OxyCopilot feature on Oxylabs’ dashboard. First announced during this year’s OxyCon, it takes in natural language instructions, sends them to a large language model, and puts out API request code, which you can then paste into a Python, Node.js, or other script.
OxyCopilot not only simplifies onboarding, but it also includes a powerful ability to generate parser code for any target. For example, it’s possible to feed it a product URL from an e-commerce website and specify the data points to retrieve, such as: “Parse the item’s name, rating, and price.”
The playground scrapes the page, creates a parser schema, and shows both the structured output and underlying selectors it generates. You can play around with the schema and generate new parsing code or start scraping if you’re happy with the result.
The AI parser sets out to address a large need: according to a recent survey made by the provider, 50% of developers identify parsing as one of the biggest web scraping challenges, and they spend between 10 and 40 hours every week on parsing processes.
By choosing to generate selectors rather than feed every page to AI, Oxylabs avoids the current cost pitfalls of large language models. It also reduces reliance on a middleman by only invoking AI in the preparatory stage. On the downside, once the parser code breaks, it won’t self-adjust without manual intervention.
OxyCopilot comes at no additional cost. To further sweeten the deal, Oxylabs has significantly decreased the rates of all public plans:
* Rate for 1,000 requests
The feature is still in beta. Despite some quirks and inefficiencies that inevitably pop up at this stage, it shows Oxylabs’ commitment to innovation in the web data extraction space. You can read more about the research process and story behind OxyCopilot here.
Even without this new AI functionality, the whole package is now more appealing than it’s ever been.
Get proxy news and updates directly to your inbox.
