By 2023, ChatGPT had been available for a year. It became clear that training datasets weren’t enough: to become truly relevant, chatbots needed a way to access the world’s knowledge as it was generated. Retrieval-augmented generation (RAG) took off to free language models from the constraint of knowledge cut-off dates. 

This gave a huge new impetus for accessing search engines as both the source of knowledge and the guide for reaching it. On the promise of making search more accessible for AI, LinkUp raised $13M, Tavily got $25M, Exa received $85M, Parallel secured $100M, while You pivoted from a consumer search engine with a similarly impressive $100M investment round. 

Web search for AI has very different requirements compared to human consumers:

• For one, data retrieval needs to be fast – way faster than the 4+ seconds of a typical SERP API response. This has prompted companies to create their own pre-scraped indices: very expensive but made possible by venture capital funding. 
• Because an index isn’t real-time and thus unfaithful to Google, APIs have to re-rank results based on their own criteria. For example, Tavily assigns each page it surfaces with a relevance score. 
• Furthermore, language models aren’t content with just the search results page – they also need to know the contents of the results. The response can include the full output, highlights that fit context windows, or straight up LLM-written answers to the query.
• And finally, AI doesn’t need the full SERP. Ads, widgets, and mobile layouts serve little use in a knowledge lookup, except for certain edge case scenarios.

Demand for search engine APIs is booming, and the market reflects that in the number of competitors. While we couldn’t list every one of them (most notably, the list misses Firecrawl and Parallel), we bought and benchmarked 15 services covering all three scraper types. Many were founded within the last few years.