How to Bypass CAPTCHAs When Web Scraping
No more pictures of traffic lights, please.
Unless you’re scraping tiny websites in the middle of Internet-nowhere, you’ve probably encountered a CAPTCHA. It’s one of the main ways domains try to protect themselves, popular for its effectiveness and simple implementation. CAPTCHAs make your spider go, “huh?” and clog up your data collection pipeline worse than a holiday turd. But it doesn’t mean there’s nothing you can do about them.
This article will teach you how to bypass CAPTCHAs or mitigate them using multiple methods. It includes general information about CAPTCHAs that you might find useful, such as what triggers a CAPTCHA challenge or what challenges you can expect. If that’s not relevant to you, feel free to skip to the parts that are.
What Is CAPTCHA?
CAPTCHA stands for Completely Automated Public Turing test to tell Computers and Humans Apart. If you don’t know what Turing test means, well – the acronym explains that too. It’s a test to determine whether the entity you’re interacting with is a computer or human. In other words, if that girl you’re trying to hook up with on Tinder is really a person, or just an elaborate chatbot that’ll try to shill an expensive webcam site.
What Is the Purpose of CAPTCHA?
The main purpose of CAPTCHA tests is to filter human traffic from bots (yes, web scrapers are bots). They do so by presenting various challenges to website visitors. The challenges are designed to be easily solvable by humans but very hard to crack for computers. CAPTCHAs allows website administrators to curb unwelcome automated activities, such as spam, DDoS attacks, and sometimes web scraping.
CAPTCHAs also have secondary purposes. Originally, they helped to digitize badly-scanned text passages that optical content recognition (OCR) technologies couldn’t crack. Nowadays, we provide free labor for Google’s machine learning algorithms by labeling objects in images. Talk about a noble cause.
How Do CAPTCHAs Work?
CAPTCHAs function as a final test to determine if a website’s visitor is human or bot. They appear when a website detects unusual traffic; then they present the visitor with a challenge.
The exact configuration of a CAPTCHA depends on the webmaster: it can protect the whole website or specific pages. Sometimes, a page will always throw up a CAPTCHA, especially if it’s a registration, comment form, or checkout page. But more often, it needs some kind of trigger to appear.
What Triggers a CAPTCHA Challenge?
The main factors that cause a CAPTCHA are:
- Simple CAPTCHA triggers. These include unusual traffic, high number of connections from a single IP address, or the use of low quality datacenter IPs. For example, VPN users see more CAPTCHAs than regular website visitors because VPNs get their IPs from a data center. The same is with corporate networks that share an IP address between many employees.
- Passive fingerprinting. A collection of parameters that evaluate your network and device. The most important are HTTP headers, user agent, TLS and TCP/IP data.
- Active fingerprinting. An even more elaborate technique that sniffs out advanced information about your hardware and software through JavaScript. It looks into WebGL parameters, fonts, plugins, and more.
These triggers don’t have to involve CAPTCHAs – they can simply block a visitor from browsing the website altogether. They’re combined whenever fingerprinting or another protection method fails to conclusively prove that a visitor is non-human. Here are the combinations you can expect and their frequency:
| Combination | Frequency |
| Simple trigger + CAPTCHA | Most common |
| Passive fingerprinting + CAPTCHA | Common |
| Active fingerprinting + CAPTCHA | Relatively rare |
| Simple trigger + passive + active fingerprinting + CAPTCHA | Rare |
As you can see, many websites won’t bother implementing elaborate fingerprint checks. That’s because doing so requires a lot of resources, and it can also harm user experience. For example, Cloudflare uses active fingerprinting to trigger CAPTCHAs, and I’m sure many people aren’t thrilled to be constantly interrupted by its “Checking your browser” screen.
Once a CAPTCHA is triggered, it presents the visitor with a challenge.
The Main Types of CAPTCHA Challenges
There are many different types of CAPTCHA challenges, and it’d be hard to list all of them here. Instead, let’s lump them into several big categories you’re most likely to encounter:
Text entry CAPTCHAs
This type presents a string of distorted letters and numbers. To pass the challenge, you have to retype them into a text field.
Image CAPTCHAs
A typical example of an image challenge would be reCAPTCHA’s grid of images, where you have to select squares that contain some object. If you succeed, you’re allowed to go past; otherwise, you get another grid or fail the test.
Image CAPTCHAs are very popular, and you’re likely to encounter them the most often. There are multiple variations of image-based challenges, such as defining an object’s boundaries or labeling what you see by category.
Audio CAPTCHAs
These challenges give an audio excerpt and then ask to type in the letters, words, or numbers you’ve heard.
Audio CAPTCHAs rarely come standalone. Instead, they function as a fallback option to other types of challenges for limited accessibility users. To make it harder for speech recognition software, audio tests sometimes add distortions to the sound.
Puzzle CAPTCHAs
This type of CAPTCHA includes math problems (addition, subtraction, and other operations), word puzzles, spatial tasks, and similar tests.
For example, a popular CAPTCHA system called fun CAPTCHA often asks website visitors to roll a ball with 3D models inside. Another CAPTCHA system, Geetest, requires you to move a piece to complete a puzzle. Puzzle CAPTCHAs rely on motion and similar mechanics to avoid recognition tasks which machine learning models have become very good at solving.
Button CAPTCHAs
This type is also called noCAPTCHA because it asks the visitor to click on a checkbox instead of presenting them with a challenge. So, if everything goes okay, a regular user won’t have to do anything more to pass. If the verification fails, a regular challenge (usually an image) will appear.
Button CAPTCHAs are widely used by Google’s reCAPTCHA and hCAPTCHA – two of the most popular systems on the internet. It reduces the friction of solving challenges and is quite effective at deterring bots. The system uses behavioral cues to monitor how visitors tick the checkbox; we briefly describe them below, under reCAPTCHA v2.
Invisible CAPTCHAs
An invisible CAPTCHA doesn’t even give you a checkbox to tick – in fact, a regular person shouldn’t see it at all. It works completely in the background, where the system monitors visitors and decides whether to present them with a challenge.
Invisible CAPTCHAs are the most recent advancement in technology, championed by Google. Their aim is to become even less inconvenient for people. However, this kind of CAPTCHAs has been criticized for using intrusive and privacy-violating technologies to filter bot traffic.
Social Media Sign Ins
A social media sign in asks new users to register using their social media account before they can see content or use a service.
While not exactly a CAPTCHA in the strict sense, a social media sign is also used to filter human traffic from bots. They’re quite effective because it’s not enough to fill in a form with false information – you also need to have a fake social media account. So, you suddenly have to deal with two websites instead of one, and social platforms aren’t an easy nut to crack for automation tools.
The Most Popular CAPTCHA Systems
Here are some of the more prevalent CAPTCHA systems on the internet:
reCAPTCHA v2
Google’s own CAPTCHA solution. It was released in 2013 and has since replaced the company’s text-based reCAPTCHA v1. Even though there’s a third version already, version 2 remains a very popular tool for webmasters.
reCAPTCHA v2 is famous for being a “no CAPTCHA reCAPTCHA”. That’s because it gives a simple “are you a robot?” checkmark box instead of a challenge. Sneaker scalpers call the box a “one-click CAPTCHA”. It reduces the friction of solving a CAPTCHA every time.
What makes a verification fail or succeed? Google considers a user’s cookie history, mouse movements, and other behavioral data. If these parameters are missing or mismatched, they can trigger a CAPTCHA challenge. Due to its reliance on cookies, v2 has become infamous for serving more challenges to non-Chrome users.
In 2017, Google introduced invisible reCAPTCHA. It works like the regular v2, but instead of clicking the special box, you can bind it to trigger on any button click.
reCAPTCHA v3
hCAPTCHA
Amazon CAPTCHA
How to Bypass CAPTCHA
If your web scraper is encountering CAPTCHAs, your first recourse should be to rotate your IP address. This helps surprisingly often, especially if you’re using a quality proxy network.
Otherwise, there are two main approaches to bypassing CAPTCHAs: you can either try to solve the challenge or avoid it altogether.
Solving the Challenge
- Text-based challenges can be overcome with machine learning. You can download the images that contain the text, segment them, and train a neural network to recognize the letters. It takes time but deals with the problem altogether.
- Image-based challenges have several solutions. You can find or train a convolutional neural network to recognise images. Alternatively, almost all image-based captchas have an accessibility mode for disabled people. Instead of solving images, it allows downloading an audio file and processing it with any online (free) speech-to-text API. Just note that Google has strengthened its active fingerprinting algorithms, so you might not always get the option for the audio.
Avoiding the Challenge
Of course, the ideal outcome would be to avoid CAPTCHAs altogether. This is harder to achieve than simply brute forcing through them but usually more rewarding. We suggest trying the following:
- Use quality IP addresses. IP recognition is often the first line of defence that websites use. With a good and “clean” residential IP address, you’ll be less likely to encounter a CAPTCHA.
- Limit the number of requests you make. You shouldn’t barrage the website with a high number of requests from the same IP. Change the duration between your requests, make organic timeouts. Don’t scrape too fast or all day round without pauses.
- Improve your web scraper’s fingerprint. Try to be as organic as possible when you scrape: match TLS parameters, HTTP headers, have a database of real user agents, discard cookies when they’re no longer needed.
- Mimic human behavior. If the website is particularly well protected, you can try navigating through its pages, clicking on links, and imitating mouse movements. You might want to use a headless browser, as well. Headless libraries like Puppeteer fully render JavaScript and include other useful functionality to help disguise your web scraper.
Conclusion
Now you know what CAPTCHAs are, their main types, and some ways to bypass them. Note that not every CAPTCHA is made equal – their triggers and difficulty depend both on the website’s security and your actions. Keep that in mind and good luck with your web scraping!
Frequently Asked Questions About Bypassing CAPTCHAs
When they were first introduced, CAPTCHAs were very effective at stopping bots. Nowadays, they are less effective thanks to advances in machine learning, but a CAPTCHA still stops a good deal of unsophisticated bot traffic.
Yes. reCAPTCHA trains Google’s AI models to better understand images and text. Other systems like hCAPTCHA label data for companies that pay for their services.
Their technical underpinnings are very similar. The main difference involves their principles. reCAPTCHA comes from Google, so it doesn’t always respect our privacy and uses the data it gets to improve Google’s algorithms. hCAPTCHA is maintained by a different company that’s not involved in the advertising business and claims to care about user privacy.
