How to Set Up Proxies with Puppeteer

This is a step-by-step guide on how to set up and authenticate a proxy with Puppeteer using headless Chromium.

Puppeteer is a NodeJS library for controlling a headless Chromium browser. It’s a popular choice for web scraping JavaScript-heavy websites – Puppeteer can fully automate most browser interactions like moving mouse movements or filling out forms, and it’s backed by Chromium developers.

Web scraping and proxies go hand in hand. So, if you want to get the most out of Puppeteer – avoid IP address ban and other web scraping obstacles – you’ll need quality proxies. But things aren’t that simple when it comes to setting up a Puppeteer proxy.

In this guide, you’ll learn how to set up a proxy in Puppeteer with a headless Chrome browser. Also, you’ll find out the most common issues you might encounter setting up proxies with Puppeteer.

How to Set Up a Proxy with Puppeteer

When you use a regular browser like Chrome, proxy authentication is simple – you’ll be required to add your credentials (username and password) into a pop-up bar. However, the problem is that headless browsers (in this case, Chromium) don’t have a user interface. That means there are no visual elements for you to type that information in.

Getting Started with Puppeteer Installation

Before you begin, you’ll need the following tools:

Node.js – it includes the package manager npm for installing Puppeteer.
Code editor – you can use any code editor you prefer.

Then, create a new folder where you can store JavaScript files. Step 1. To set up a puppeteer proxy, run the initialization command in the folder you’ve just created.

				
					npm init -y

Step 2. Install the Puppeteer package.

				
					npm install puppeteer

How to Set Up a Proxy with Authentication in Puppeteer

We’ve prepared a real-life example which opens a browser with Puppeteer, goes to ipinfo.io website, and scrapes your IP address. This method supports proxy authentication and is the most popular approach for setting up Puppeteer proxies. It uses Puppeteer’s page.authenticate() string to pass your proxy credentials to the browser.

If you don’t need to authenticate your proxy, remove or comment ‘proxyUser’, ‘proxyPass’ and ‘await page.authenticate()’ lines.

Step 1. First, import the Puppeteer package.

				
					const puppeteer = require('puppeteer');

Step 2. Let’s define the proxy server, username, and password. Then, launch the browser.

				
					(async () => {
	const proxyServer = 'proxyserver.com:PORT';
	const proxyUser = 'USERNAME';
	const proxyPass = 'PASSWORD'
	const browser = await puppeteer.launch({
    	args: [`--proxy-server=${proxyServer}`],
	});

Step 3. After you launch the browser, initiate a new page.

				
						const page = await browser.newPage();

Step 4. Then, authenticate your proxy.

				
						await page.authenticate({proxyUser, proxyPass});

Step 5. Now, let’s open the browser and start scraping.

				
						await page.goto('https://ipinfo.io');

1) Create a CSS selector for the element where the IP is listed on ipinfo.io website.

				
						const ipSelector = '#ip-string .text-green-05';

2) Tell the browser to wait until that element appears. This will allow the page to fully load.

				
						await page.waitForSelector(ipSelector);

3) Find the element using the CSS selector above and assign it to a variable.

				
						let ipElem = await page.$(ipSelector);

4) Get the text from that element.

				
						let ip = await page.evaluate(el => el.innerText, ipElem)

Step 6. Print out your results (in this case – the IP address).

				
						console.log(ip);

Step 7. Close the browser.

				
						await browser.close();
})();

Here’s the full script:

				
					const puppeteer = require('puppeteer');

(async () => {
    const proxyServer = 'proxyserver.com:PORT';
    const proxyUser = 'USERNAME';
    const proxyPass = 'PASSWORD'
    const browser = await puppeteer.launch({
        args: [`--proxy-server=${proxyServer}`],
    });

    const page = await browser.newPage();

    await page.authenticate({proxyUser, proxyPass});
    await page.goto('https://ipinfo.io');

    const ipSelector = '#ip-string .text-green-05';
    await page.waitForSelector(ipSelector);

    let ipElem = await page.$(ipSelector);
    let ip = await page.evaluate(el => el.innerText, ipElem)

    console.log(ip);

    await browser.close();
})();

Most Common Errors

Page Isn’t Loading

If your internet connection is unstable, Puppeteer might fail to load your target page. In this case, you should set a higher timeout value in the Puppeteeroptions object. This way, your connection won’t have that huge of an impact on your scraper.

				
					const options = {
timeout: 20000,
}

Other Issues

Not all proxies are compatible with Puppeteer, so you might need to experiment with different types to find one that works. Additionally, your IP address might be blocked by your target website. If this is the case, change your IP to a different one.