How to Use Wget with a Proxy: A Tutorial
Wget is a great tool for quickly downloading web content. It also offers the flexibility to route your requests through a proxy server. Here you’ll learn how to use Wget with a proxy.
There are many command-line tools for downloading web content, such as cURL. However, if you want to handle recursive downloads and resuming tasks when the connection is unstable, Wget is your best option.
What Is Wget?
Wget is a GNU Project command-line utility built to download HTTP(S) and FTP(S) files. It’s a non-interactive tool, making it especially useful for downloading content in the background while completing other tasks.
Wget was specifically designed to handle content downloads on unstable networks, so it will automatically try to resume the job once the connection is restored. Typically used on Unix-like operating systems such as Linux and macOS, the command-line tool is available on Windows, too.
Key Wget Features
Even though the tool was first introduced in the 90s, it’s still widely used today due to its simplicity and reliability to steadily download data from the web. Here are some key features of Wget:
- Resuming interrupted downloads. If a download is interrupted because of connectivity issues or system shutdown, Wget will automatically retry the task once the connection is restored – no manual input is needed.
- Automated file download. Wget can batch process downloads or schedule them for repetitive tasks.
- Recursive download support. You can create a local copy of a website with Wget to view it offline or archive the website’s snapshot for a future reference.
- High control over downloads. You can script Wget to limit bandwidth, adjust headers and user agents, as well as adjust retries for downloads.
- Proxy support. Wget supports HTTP and HTTPS proxies if you need to download geo-restricted or otherwise protected content.
However, while Wget is versatile and very easy to use, there are some risks you should be aware of when using it.
If you don’t customize Wget correctly, it might unintentionally download large amounts of data, like entire websites. Make sure you give precise commands to avoid consuming too much bandwidth and storage.
Additionally, it’s vital to be aware of the quality of the URL you’re downloading content from. Downloading from untrustworthy or malicious URLs can pose security risks, such as exposing your system to malware.
Wget vs. cURL: the Differences
Both Wget and cURL are command-line tools used for data transferring. However, their functionality and niches slightly differ.
Wget is primarily used to download content from the web. On the other hand, cURL is used for data transfer (upload and download). Therefore, cURL is slightly more complex than Wget as it can move the content to and fro, rather than only receive it.
How to Install Wget?
Wget’s installation process is quite straightforward but it slightly varies based on the operating system you are using. Let’s see how to install Wget on Windows and MacOS.
Being a command-line utility, Wget is used with Terminal and CMD – command-line interfaces used to interact with an operating system. Terminal is a Unix-based shell interface (used in macOS, Linux operating systems), while CMD (Command Prompt) is Windows’ default command-line interface. Thus, to install and use Wget, you’ll have to use Terminal or CMD.
- For Windows users, you’ll need to download and install the Wget package first. Once it’s installed, copy and paste the wget.exe file to the system32 folder. Then, run wget in Command Prompt (CMD) to check if it works.
- For those using MacOS, you’ll firstly need to install package manager Homebrew by running xcode-select –install in your Terminal. Once a pop-up window appears, select install, and complete the installation. Then, run , enter Admin password when prompted, and press Enter to continue. You’ll see “Installation successful!” message once it’s done. Then, you can finally install Wget by running wget -v in Terminal.
Changing the User-Agent with Wget
If you’re planning to use Wget for downloads often, you should modify your user-agent string to avoid bot detection and access geo-restricted content. You can change your user-agent for all future uses by editing the .wgetrc file, or write a command for one-time use.
Modifying the User-Agent for a Single Download
Whether you’re a Windows or a macOS user who wants to change your user-agent for a single download, simply run wget –user-agent=”CustomUserAgent” https://example.com in Command Prompt (CMD) or the Terminal.
For example, it can look something like this:
wget --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" https://example.com
Modifying the User-Agent Permanently
If you’d like to consistently use a different user-agent, you can change the Wget configuration in the .wgetrc file. The custom user-agent string you’ll put there will be used for all future jobs until you change it.
Simply locate the .wgetrc file and add user_agent = “CustomUserAgent”
It should look something like this:
user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
How to Use Wget?
Due to the tool’s nature, you’ll be sending download requests through the command-line. Let’s take a look at how to download files and retrieve links from webpages using Wget.
Downloading a Single File with Wget
Retrieving a single file using Wget is simple – simply open your command-line interface, and run wget with an attached link of the file you want to retrieve. It should look like this:
wget https://example.com/new-file.txt
Downloading Multiple Files with Wget
There are a couple of ways to download multiple files with Wget. The first method is to send all URLs separated by a space. Here’s how it should look like if you were to download three files at the same time:
~$ wget https://example.com/file1.txt https://example.com/file2.txt https://example.com/file3.txt
Using this method is ideal when you have a limited number of URLs. However, if you have dozens of files you want to download, it becomes much more complex. Therefore, if you have multiple files you want to retrieve, using a different method is more reliable.
The second method relies on writing down all URLs in a .txt file, and using the -i or –input-file option. In this case, Wget will read the URLs from the file and download them.
Let’s say you named the file myurls.txt. You can use an –input-file argument:
~$ wget --input-file=myurls.txt
Or the -i argument to download files listed in the file:
~$ wget -i myurls.txt
Getting Links from a Webpage with Wget
You can also use Wget to extract links directly from a webpage. Wget will either list these links for you or you can choose to download them.
If you want Wget to crawl a page, find all the links and list them without downloading, you can run this command:
wget --spider --force-html -r -l1 https://example.com 2>&1 | grep -oE 'http[s]?://[^ ]+'
If you’d like Wget to find the URLs and download them for you, simply remove the –spider and –force.html commands that crawl and parse the HTML pages. Instead, your command should look something like this:
wget -r -l1 https://example.com
How to Use Wget with Proxy?
If you want to use Wget with a proxy, there are a couple of ways you can go about this. You can either set up proxy settings in the configuration file or write down proxy settings in a command for one-time use. However, Wget only supports HTTP and HTTPS proxies, so make sure you have those.
How to Set Up Proxy for Wget
In order to set up a proxy for Wget, firstly you’ll have to get the proxy details. Then, set the proxy variables for HTTP and HTTPS in the .wgetrc file that holds the configuration content for Wget.
Checking Current IP Address
First thing’s first, you need to figure out your machine’s IP address. You can do that by sending a request to the HTTPBin IP endpoint with Wget:
wget -qO- https://httpbin.io/ip
You should receive an output similar to the one below:
{
“origin”: “123.45.67.89:000”
}
Note: this is not a real IP address, rather an example to familiarize you with the format.
Here, the value of the origin key is your current IP address. We can now use a proxy and see if the output (IP address) will change.
If you don’t have a proxy provider or aren’t sure how to choose one, we composed an article that can help you make the decision. We also highly advise against using free proxies as they’re often compromised and can cause data leaks, and are often unethically sourced.
Discover top proxy service providers – thoroughly tested and ranked to help you choose.
Modifying Proxy Settings
If proxy settings are not yet defined in the .wgetrc file, you can add them manually.
If you use Windows OS, run notepad C:\Users\YourUsername\.wgetrc in CMD. MacOS users should use run -e ~/.wgetrc in Terminal. If the file doesn’t exist in your system, this command will automatically create and open it.
Once the file is open, add proxy settings to the file:
use_proxy = on
http_proxy = http://proxyserver:port
https_proxy = https://proxyserver:port
Note: use actual proxy server address and a correct port number when editing the file. These will be given to you by your proxy service provider.
Once you write down proxy settings, you can send a request to HTTPBin to check if the IP address has changed.
Wget Proxy Authentication
Most reputable proxy server providers will require authentication before using a proxy with Wget. Typically, you’ll need to specify your username and password when connecting to a proxy server.
You can do that by adding a couple of extra lines to the .wgetrc file.
proxy_user = YOUR_USERNAME
proxy_password = YOUR_PASSWORD
So, the entire addition to the file should look like this:
use_proxy = on
http_proxy = http://proxyserver:port
https_proxy = https://proxyserver:port
proxy_user = YOUR_USERNAME
proxy_password = YOUR_PASSWORD
You can again send a request to HTTPBin to check if the proxy works as intended.
Using Wget with a Proxy for a Single Download
For using proxies with multiple downloads, we recommend setting a proxy configuration in the .wgetrc file instead. However, you can also specify proxy settings for Wget if you’re planning to use Wget with a proxy once. Instead of modifying the .wgetrc file, you can run a command directly in Terminal or CMD.
It should look like this:
wget -e use_proxy=yes -e http_proxy=https://username:password@proxyserver:port https://example.com/file.zip
Note: the example uses http_proxy, but Wget supports HTTPS proxies too, so you can use https_proxy for your proxy settings.
