How to Set Up a Proxy with Selenium using Python

This is a step-by-step guide on how to set up and authenticate a proxy with Selenium using Python.

Selenium is a tool primarily used for web testing and browser automation. It lets you control headless browsers programmatically: open websites, take screenshots, and otherwise interact with the page. And with the growing popularity of JavaScript, web scrapers found its strength in dealing with dynamic websites.

However, you won’t be able to do much web scraping or testing without a proxy server because websites are rigorous towards heavy automation. That’s why you need a proxy server – a middleman computer between you and the internet.

This Python tutorial will show you how to set up a proxy server with Selenium and how to handle proxy authentication.

How to Set Up Proxies using Selenium Wire

If you’ve subscribed to a proxy service, you’ll most likely need to authenticate your proxies before you can start using them. Otherwise, you won’t be able to run it in Selenium. Depending on the scale of your project, you can use Selenium with a regular browser or a headless one. In this example, we’ll show you how to set up and authenticate a proxy using a headless browser.

First, you’ll need to install Selenium Wire to extend Selenium’s Python bindings. When you need to authenticate your proxies, the default Selenium module makes the process too complicated.

Step 1. Install Selenium Wire and import WebDriver.

				
					pip install selenium-wire
from seleniumwire import webdriver

Note: You can refer to GitHub documentation on how to install Selenium Wire.

Step 2. We recommend installing WebDriver-manager, so you won’t need to download Selenium after each update.

				
					pip install webdriver-manager

Step 3. Add your proxies to the options argument that you pass to the WebDriver.

				
					options = {
    'proxy': {
        'http': http://USER:PASSWORD@ENDPOINT',
        'https': 'https://USER:PASSWORD@ENDPOINT',
    }
}

driver = webdriver.Chrome(seleniumwire_options=options)

How to Set Up a HTTP(S) Proxy with Authentication

Step 1. To authenticate your proxies, you’ll need to specify your username and password in the URL.

				
					options = {
    'proxy': {
        'https': 'https://USER:PASSWORD@ENDPOINT',
    }
}

Step 2. You can also set your proxy through environment variables. If you do so, you won’t need to define your proxies in the code.

				
					$ export HTTP_PROXY="http://ENDPOINT"

$ export HTTPS_PROXY="https://ENDPOINT"

$ export NO_PROXY="LOCALHOST,ENDPOINT"

How to Set Up SOCKS5 Proxy with Authentication

With SOCKS5 proxies, the authentication process is similar to HTTP(S) proxy – you only need to set the scheme to socks5. If your proxy doesn’t require authentication, exclude ‘user’ and ‘password’.

Step 1. Authenticate your SOCKS5 proxies by specifying your username and password in the URL.

				
					options = {
    'proxy': {
        'http': 'socks5://USER:PASSWORD@ENDPOINT',
        'https': 'socks5://USER:PASSWORD@ENDPOINT',
        'no_proxy': 'localhost,ENDPOINT'
    }
}
driver = webdriver.Chrome(seleniumwire_options=options)