How to Use cURL with a Proxy Server

Learn how to efficiently test your proxies via the command line.

curl with proxy thumbnail

cURL is a handy command line tool for testing proxies and even doing some light web scraping. This guide will give you the basics you need to effectively use cURL with a proxy server. Let’s get started!

Contents

  1. Checking Your IP Address
  2. Checking Your Proxy Server
  3. Using cURL with HTTPS Proxies
  4. Using cURL with SOCKS Proxies
  5. Using Proxies with Authentication
  6. Seeing What’s Happening in the Background
  7. Savings Results to a File

cURL Basics

What Is cURL?

But first, what exactly is cURL? The name stands for client URL, but that might not tell you much. So, let’s try again.

cURL is a text-based tool that runs in the terminal and allows transferring data over the internet. It’s over 20 years old, supports most internet protocols, and runs out of the box in all modern operating systems.

If you have time, you can read this great e-book to learn all about cURL. This article focuses on delivering practical knowledge that you can apply right away.

What Is cURL Used for?

cURL is often used with APIs: to get acquainted them, play around with endpoints, or do straight up serious work. In the proxy industry, cURL has become a popular tool for testing backconnect proxy servers (so, anything that involves the terms rotating, residential, or mobile).

Why cURL?

For one, cURL is long standing, widely available, and uses a simple text interface that works the same on every system. That aside, cURL is also pretty powerful. The tool can be used to write elaborate scripts that involve authentication, SSL connections, proxy tunnelling, cookies, and more. If that’s unnecessary, the basic syntax is easy to grasp, so you can quickly start doing useful work with it.

How cURL Works

At its most basic, cURL uses the following syntax:

curl [option] [url]

[option] refers to commands that tell cURL what to do. For example, -x tells cURL that the connection will go through a proxy server. Many options have alternative names: instead of -x, you can also write ‐‐proxy. Some options may go after the URL, for example, -v that asks cURL to display the connection information. Options aren’t obligatory to the syntax, but you’ll be using them most of the time.

[url] refers to the domain you want to interact with.

Which Protocols Does cURL Support?

cURL supports over 20 protocols, which should be enough for all proxy server use cases. The list includes HTTP, HTTPS, SOCKS, POP3, SMTP, IMAP, and many others you probably won’t use. If you don’t specify the protocol in the URL, cURL will assume you want to use the HTTP protocol.

Do I Have cURL?

If you use a modern operating system, probably yes. By modern, I refer to Windows 10 or any supported version of macOS. Linux-based distributions may or may not have cURL installed – there’s simply too much variety to say for certain.

How to check? Simple: open the Terminal and write curl ‐‐man. The manual page for cURL should appear.

If it doesn’t, here’s how to install cURL:

  • Older Windows versions: downloading it from here and simply unzip the package.
  • Debian-based Linux distributions: run sudo apt install curl.
  • Other Linux distributions: follow the instructions here.

How to Use cURL with Proxy Servers

Alright, now that you know what cURL is and have it installed on your computer, let’s see what we can do with it.

Checking Your IP Address

If you’ve set up a proxy server on your OS, you can quickly check your IP address and location by running the following command:

curl ipinfo.io

Example response:

{
"ip": "45.152.180.180",
"city": "New York City",
"region": "New York",
"country": "US",
"loc": "40.7143,-74.0060",
"org": "AS9009 M247 Ltd",
"postal": "10004",
"timezone": "America/New_York",
"readme": "https://ipinfo.io/missingauth"
}

Checking Your Proxy Server

Of course, it’s not often that you’d want to set up proxies on the operating system level. So, you can test if your proxies are working using another way. cURL has the -x or ‐‐proxy option to specify the proxy that’s going to be used in the request:

curl -x 127.0.0.1:80 ipinfo.io

or

curl --proxy 127.0.0.1:80 ipinfo.io

In this example, 127.0.0.1 is the proxy server’s IP address and 80 is the port number.

Instead of an IP address, you can also use a domain name:

curl -x http://us.proxyendpoint.com:20000 ipinfo.io

Using cURL with HTTPS Proxies

Most providers that offer HTTPS proxies let you connect to the proxy server via HTTP and then establish an HTTPS connection with the website. So, your cURL command should look the same whether you’re connecting to an HTTP or HTTPS proxy server.

Using cURL with SOCKS Proxies

If you want to use cURL with the SOCKS protocol, there are several ways to do so. You can specify the protocol together with the proxy IP. Alternatively, you can add socks5:// or use the ‐‐socks5 option instead of -x:

curl -x socks5://127.0.0.1:80 ipinfo.io
curl --socks5 127.0.0.1:80 ipinfo.io

Using Proxies with Authentication

Paid providers often require proxy authentication before you can use them. If you’ve whitelisted your IP address, that’s fine. But if you’re using a username and password, you’ll need to include another option called -U or ‐‐proxy-user:

curl -U username:password -x us.proxyendpoint.com:20000 ipinfo.io

It’s also possible to pass the username and password together with the proxy itself:

curl -x http://username:[email protected]:80 ipinfo.io
curl -x http://username:[email protected]:20000 ipinfo.io

Seeing What’s Happening in the Background

You might want more information about what happens when you send a request. For example, you might be interested in the request or response headers, response code, or the user agent you send. Adding the -v or ‐‐verbose option will print out what’s going on behind the scenes. Knowing how to use this option can be very useful for debugging.

curl -x http://username:[email protected]:20000 ipinfo.io -v

* Rebuilt URL to: ipinfo.io/
*   Trying 51.161.68.240...
* TCP_NODELAY set
* Connected to us.proxyendpoint.com (127.0.0.1) port 20000 (#0) < We see that the proxy server and port are reachable
* Proxy auth using Basic with user 'username'
> GET http://ipinfo.io/ HTTP/1.1
> Host: ipinfo.io
> Proxy-Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ= < Header containing your username and password encoded in Base64
> User-Agent: curl/7.55.1 < This is how the web server sees the requester. Any server would know that cURL is used to access it unless the User-Agent is changed
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK < Response code
< content-length: 325
< access-control-allow-origin: *
< content-type: application/json; charset=utf-8
< date: Thu, 29 Jul 2021 09:22:32 GMT
< referrer-policy: strict-origin-when-cross-origin
< via: 1.1 google
< x-content-type-options: nosniff
< x-envoy-upstream-service-time: 2
< x-frame-options: DENY
< x-xss-protection: 1; mode=block

Response codes: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status

Saving Results to a File

If you want to save the logs (to analyze them or berate your proxy provider for things that don’t work), you can use the -o or ‐‐output option to write the result to a file.

For example, the command below creates a new file called output.txt in your active directory and prints the response directly to it:

curl -x http://username:[email protected]:20000 ipinfo.io -v -o "output.txt"

Advanced cURL Settings

Setting Headers

If you don’t want a website to treat you as a bot, you’ll need to send appropriate headers. cURL allows doing just that.

To see what headers you normally send to the target website, click the right mouse button, select Inspect, and navigate to the Network tab. Refresh the site, and you’ll see all of the requests being made while it’s loading.

Pro tip: You can right click on the request and copy it as a cURL command to take a better look at it.

So, how do you actually set those headers in cURL? With the -H or ‐‐header option. For example, we can send an Accept header to the target site:

curl -x http://username:[email protected]:20000 ipinfo.io -v -H "Accept: text/html"

More on headers:

https://developer.mozilla.org/en-US/docs/Glossary/Request_header

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers

Adding a User-Agent

By default, cURL announces itself to websites as… well, cURL. There are times when we don’t want that – once again, to prevent getting treated as a bot and then blocked. cURL allows us to spoof our user-agent using the option -A or ‐‐user-agent:

curl -x http://username:[email protected]:20000 ipinfo.io -v -A "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"

The verbose output lets us see that the user-agent has changed:

> User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0

It’s also possible to send the User-Agent in the header using the -H option:

curl -x http://username:[email protected]:20000 ipinfo.io -v -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"

More on user-agents: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent

Adding Cookies

Cookies help websites remember information about you, and sometimes they’re necessary for the site to work. If you need them, you use the -b or ‐‐cookie option to specify cookies or the file they should be read from.

For example, if there’s an active session and you need to send the ‘sess_id’ cookie of its ID with the request, you can use:

curl -x< http://username:[email protected]:20000 example.com -v -b ‘sess_id=qwerty123’

It’s also possible to receive cookies from the target server and store them for later use. cURL has the -c or ‐‐cookie-jar option that allows specifying the output file:

curl example.com -c “cookies.txt”

You can then read those cookies from the file and use them for subsequent requests with the -b command:

curl example.com -b “cookies.txt”

Following Redirects

Sometimes, you may encounter an issue when the output just says Found or Found. Redirecting to https://somewhere.com. In such cases, you’ll see a HTTP code 302 in the verbose output.

The issue is easy to solve: just add the -L or ‐‐location option to the request:

curl -x http://username:[email protected]:20000 ipinfo.io -v -L

Ignoring SSL errors

There may be times when the target websites will use a self-signed or an expired certificate. You’ll receive the SSL certificate: invalid certificate chain error when trying to access it.

cURL is able to ignore certificate errors by adding the -k or ‐‐insecure option:

curl -x http://username:[email protected]:20000 ipinfo.io -v -k

Sending Data with POST Requests

cURL allows sending POST requests to the web server instead of just fetching data. This way, you can fill in forms, log in, and otherwise interact with the website. To do so, use the -X or ‐‐request option:

curl -X POST https://example.com

To specify the data you want to send, you’ll also need the -d or ‐‐data option.

Sending values to specific variables:

curl -X POST https://example.com -d "name=John%last_name=Doe"

Sending JSON-formatted data:

curl -x POST https://example.com -d '{"name" : "John", "last_name" : "Doe"}'

Note: Sometimes, you may need to escape specific characters (more often on Windows systems) using the backslash character (\). An escaped JSON would look like this:

curl -x POST https://example.com -d '{\"name\" : \"John\", \"last_name\" : \"Doe\"}'

Conclusion

This marks the end of my brief introduction to using cURL with proxy servers. I hope you’ve found it instructive. Now go curl some proxies!

Frequently Asked Questions About Using cURL with Proxy Servers

Does cURL Use TCP or UDP?

By default cURL uses TCP.

Is cURL Free?

Yes, you can download and use cURL free of charge.

Can I Use cURL with Username and Password Authentication?

Yes, you can. cURL has the -U option for username and password authentication.

Can I Use cURL without SSL?

Sure. cURL supports plain old HTTP, and you can configure it to ignore SSL errors.

How Can I Check My IP with cURL?

Simply enter curl ipinfo.io into the terminal, and it will show your current IP address.

Submit a comment

Your email address will not be published. Required fields are marked *