How to Follow Redirects With cURL
With cURL, you can manage data transfers – both uploading and downloading – via command line. It is an essential tool in the web scraper’s toolbox. However, no tool set is complete without a way to handle redirects – for example, when a website sends you to its www2 version for load-balancing purposes. This is how you follow redirects with cURL.
Following Redirects With cURL
To follow a redirect, add -L (--location) to the cURL request. This will instruct the software to follow all HTTP redirects until the final URL is reached. It will follow a maximum of 30 redirects to avoid creating loops.
Here’s how it looks:
curl -L "https://proxyway.com/glossary/curl"
Note: cURL can’t follow HTML tag or JavaScript-based redirects.
Limiting the Number of Redirects With cURL
If 30 redirects is too many for your purposes, add --max-redirs [number] to your cURL command:
curl -L --max-redirs 5 "https://proxyway.com/glossary/curl"
Following Redirects Without Displaying cURL Progress Tracker
To follow redirects without displaying the progress tracker use -Ls instead of -L:
curl -Ls "https://proxyway.com/glossary/curl"
Following Redirects With cURL Displaying More Data
To get more detail by showing HTTP redirects – useful for debugging – add -I to your command:
curl -L -I "https://proxyway.com/glossary/curl"
If you need to know more about using cURL (such as how to use it with a proxy), feel free to follow our cURL guide.
Frequently Asked Questions About Anti-Detect Browsers
A redirect occurs when a web user or a search engine is sent to a different URL than the one they entered, usually via HTTP forwarding functionality (marked by status codes 3XX, like 301 and 302). Redirects are often used to temporarily route users when the original page is under maintenance or overloaded, or more permanently to maintain links when the original content is moved.
cURL is a command-line tool used to transfer files over the internet, often used in webscraping.
cURL, as a tool, works on a very basic level unless ordered to do differently. Therefore, following redirects is one of those options that have to be manually toggled by the user.
It is generally safe to follow redirects. While malicious redirects – such as open redirect scams – exist, web scraping is a task that is not exactly vulnerable to them.
Neither cURL nor Wget can follow non-HTTP redirects. If you need to follow HTML or JavaScript redirects, you will need to use another tool.
If you haven’t done so, add -L to the cURL command to make it follow redirects. If it still doesn’t follow, also add -v to the command to get a full report on your request for troubleshooting. Note that cURL doesn’t follow HTML or JavaScript redirects.
