How to Use Inspect Element for Web Scraping
Modern, interactive websites are intuitive and user-friendly, but have you ever wondered how much valuable data they hold? If you know how to use Inspect Element, you can reveal the underlying HTML and CSS structure that is crucial for web scraping.
Developers and designers spend a lot of time making websites visually appealing and easy to use. However, underneath all that user experience stands the source code – the backbone structure that makes it all work. Thanks to Inspect Element, we, as users, can peek inside and see how websites are made.
Using Inspect Element has several benefits, especially if you’re a web designer, developer, a marketer, or you’re building a custom web scraper. The tool allows you to look into how elements function, check how they appear on different screen resolutions, get data from the source code for scraping purposes, and more. So, let’s dig deeper into how to use Inspect Element.
What Is Inspect Element?
Inspect Element is a web browser feature that lets you peek into the HTML structure of the website. It’s built into most modern desktop browsers, such as Chrome, Firefox, or Safari, and is a part of Developer Tools – a set of tools that allow developers to test, troubleshoot, and experiment with websites.
By using Inspect Element, you can view and temporarily edit HTML, CSS, and JavaScript code, monitor website’s performance, as well as find layout information about specific elements (i.e., the dimensions and margins of a button).
Inspect Element can also be useful for web scraping. When building a scraper, you’ll likely need it to extract only the relevant data points rather than the entire HTML structure. The feature allows you to find and collect specific CSS selectors, metadata, or hidden elements that you want to scrape.
Why Use Inspect Element?
The Inspect Element feature makes web designer’s and developer’s lives easier by allowing them to temporarily modify element features in real time. Basically, it allows them to test bug fixes and experiment with new features before making actual changes to the website yet.
The feature can also be useful for web scraping.
- It can help you collect specific CSS selectors or other data points for your scraper.
- Inspect Element can also help to see if your scraper’s request header won’t be blocked by the website. The headers used by scrapers often look different compared to regular users. But the feature will let you see it, decide if it looks natural enough, and make changes accordingly.
Inspect Element is handy for marketers and SEO specialists.
- By reviewing the HTML and CSS code, marketing specialists can check if the website’s content is up to par. You can see if the colors and fonts follow the brand guidelines, check link information (a follow or a nofollow link), review if the website’s content looks good on desktop, responsive, and mobile views, and more.
Website users can also find ways to benefit from it.
- If a website makes it hard to download certain media, you can use Inspect Element to find the source URL, and download it from there.
- You can also have some fun – try using Inspect Element to rewrite this entire article by changing the HTML element values.
There are other use cases for Inspect Element, too. Sometimes, you can bypass a paywall by modifying the source code to gain full access to the site. However, we discourage you from using Inspect Element for unethical reasons – not only can you violate the website’s terms of use, but face legal implications, too.
How to Use Inspect Element: A Detailed Explanation
Using Inspect Element is quite simple. There are several different tabs in the tool that you can explore, but you’ll find the most use cases for:
- Elements panel allows inspecting and modifying HTML and CSS code.
- Console panel allows inspecting and debugging JavaScript.
- Sources panel allows inspecting and modifying CSS or JavaScript source files (pop-ups, analytical files).
- Network panel allows monitoring HTTP requests.
- Performance panel allows analyzing website’s performance – load, and download speeds.
- Application panel allows managing cookies, local storage, and other session data.
Keep in mind that these panels have different names in each browser. For example, the “Elements panel” in Chrome and Safari is called “Elements,” while Firefox calls it “Inspector.” However, the content there remains the same.
If it’s your first time using Inspect Element, you might want to use it on the same website with different browsers to familiarize yourself with the tool’s terms.
The easiest way to access the HTML code is to right-click any element on the website and select “Inspect” or “Inspect element.” However, it varies slightly across different browsers and operating systems. Let’s break them down.
Google Chrome & Mozilla Firefox
There are three ways to open Inspect Element on Google Chrome and Mozilla Firefox: 1) right-clicking to inspect, 2) using a shortcut, or 3) opening Developer Tools through the browser’s menu bar.
- Using right-click: use your mouse or a trackpad, right-click any element on the website. Then, select “Inspect.”
- Using a shortcut: you can use a keyboard shortcut to open Inspect Element. On MacOS, click
command+option+i. On Windows, clickCtrl+Shift+i. Additionally, you can use F12 to open Developer Tools on both Windows and MacOS (sometimes you might need to press the Fn key as well). - Using the browser’s menu: locate the kebab menu (three stacked dots) on Chrome or hamburger menu (three stacked lines) on Firefox. Typically, it’s located on the right side of the URL bar. Find “More tools,” and then select “Developer Tools.”
Safari
If you’re a Windows user, you can’t use the latest versions of Safari, so opening Inspect Element on this browser is only possible on MacOS. The process is also significantly different than on Chrome or Firefox.
Firstly, you need to enable Developer Tools in Safari settings. To do so, you can either press command and , (comma button), or in the Menu bar on top of your device’s screen (the bar with an Apple logo), click on Safari, and Settings… In the settings panel, navigate to the “Advanced” bar. There, check the “Show features for web developers” box. Now you’re able to open Inspect Element.
- Using right-click: right-click any element on the website, and press “Inspect Element”
- Using a shortcut: press
command+option+ion your keyboard to open Inspect element. Safari doesn’t use the F12 key shortcut by default, but you can manually set it up through keyboard settings. - Using a menu bar on your device: once you have a page on Safari open, you can access various settings on the menu bar on the top of your screen (make sure Safari screen is active). There, you can press on “Develop” and click on “Show Web Inspector”.
Mobile devices (iOS and Android)
There’s no easy way to use Inspect Element on iOS or Android as most of the time it still requires a desktop. However, there are a few methods you can try.
Remote Debugging
You can try remote debugging using Chrome on Android or Safari on iOS. Remote debugging allows you to use your phone’s environment on your computer. This method requires a computer, and it’s the most functional.
On an Android device:
- Enable developer options. In Settings, click About phone, and tap Build number seven times.
- Then, connect your mobile device to your computer via USB.
- Open Chrome on your computer, and go to chrome://inspect. There, you’ll see your connected device.
- Now you can go to your target website, and open Inspect Element. This way, you can inspect the mobile view on your desktop.
On an iOS device, the process is similar:
- On your mobile device, go to Settings, and find Safari.
- When Safari settings open, find Advanced, and toggle the Web Inspector there.
- Then, connect your iOS device with your Mac, and open Safari on your computer. Go to Develop in the Menu bar, and select your iOS device.
- Now, go to your target website, and open Inspect Element.
Using Browser Developer Tools
Some mobile browsers also have an option to “Request Desktop Site”, which gives a layout similar to a desktop one, so you can do some basic inspection. The feature is helpful if you don’t have access to a computer but need to check different layouts of a website (mobile & desktop). That said, you still won’t be able to open Inspect Element.
You can also see how websites adapt to screen resolutions on a desktop. Many browsers have a “Responsive Design Mode” that can help to see if a website’s content looks good on different screen parameters. You can open Inspect Element while using the feature, but it requires a computer.
Using Third-Party Apps
There are some third-party applications for element inspection – you can download them for your iOS and Android devices. However, they can be pricey, and their functionality might be limited compared to using Developer Tools on a desktop.
Inspect Element Playground
Now that you know what Inspect Element is and how to use it, you can try putting your knowledge into practice. While some changes are very easy (i.e., changing the text), some might require basic HTML and CSS understanding. If you don’t know how to use them, there are various free HTML, CSS, and JavaScript tutorials online.
Inspect Element Playground for Web Scraping
If you’re looking to use Inspect Element for scraping purposes, you’ll likely need to copy some selectors from the target website – it helps the scraper extract only what’s relevant for you. Seasoned developers will know how to find these selectors and write the scraping logic, but if you’re new to this, ChatGPT can help to scrape, too.
In the playground below, you’ll find a table with terms and their descriptions. Use Inspect Element to find CSS selectors for them (hovering over code with your mouse will highlight the area that piece of code defines).
Once you do, you can write a simple Python scraper to scrape all terms and descriptions from the table below, and save them in a CSV format.
| Term | Description |
|---|---|
| Proxy server | A proxy server is a middleman between the client and server (website). It masks the user’s original IP address, thus increasing security, privacy, and allows bypassing geo-restrictions. |
| Web browser | A web browser is a software that allows users access and view websites on the internet. |
| IP address | An IP address is a unique string of numbers given to any device connected to the internet. IPs help identify devices on the internet. |
| Web scraper | Web scraper is a software that automatically extracts data from websites. You can build a web scraper yourself using languages like Python or choose a pre-made one. |
Other Easy Ideas to Test Inspect Element
Below you’ll find several typical HTML elements that can be manipulated with Inspect Element – headings, tables, images. Here are some ideas for you to try out.
Idea No.1: Modify the size of the headings using the Inspect Element tool.
- Locate the
<h2>heading in the Inspect Element section. - The code should look like this:
<h2> ... </h2>. - Double-click on the
<h2>tag and change the number (2) to any number between 1 and 6. - Observe how the website updates as the heading size changes.
Idea No.2: Change the content inside paragraphs.
- Locate the
<p>tag that contains the text: “This is a paragraph.” - Double-click on the paragraph text.
- Replace the text with your own sentence or phrase.
Idea No.3: Add more rows and columns to the table.
- Locate the table in the source code. Table rows are defined by
<tr>tags, and columns by<td>tags. - Add a new
<tr>tag to create a new row. - Inside the new row, use multiple
<td>tags to add columns. - Experiment with changing the width of each
<td>by adjusting its attributes with CSS (e.g.,'width="100px"').
Idea No.4: Customize the table’s background color using CSS.
- Find the
<table>tag in the source code. - Inside the “
style” attribute, which already contains properties like “width” and “border-collapse”, add a new rule: ‘background-color:Tomato;’. - Make sure all properties remain inside the quotation marks, like this: ‘
"style="border-collapse: collapse; width: 100%; background-color:Tomato;"’. - Check how the background color changes.
Idea No.5: Replace an image using a different URL.
- Locate the
<img>tag in the source code. - Find the “
src” attribute, which contains the current image source (URL). - Double-click the URL, delete it, and replace it with the link to any other image.
- The new image should now appear in place of the original one.
THIS IS AN H2 HEADING
This Is an H3 Heading
This is a paragraph.
| Cell 1 | Cell 2 |
| Content | Content |
| More content | More content |
Conclusion
The Inspect Element feature is an excellent tool for web developers, designers, marketers, and regular web users alike. It’s also a beneficial feature for people who build their own web scrapers – Inspect Element allows exploring, finding, and collecting necessary data points for scrapers from the raw HTML structure of the website.
Using Inspect Element is easiest with a desktop, but some basic inspection is also available on mobile devices. While the process slightly varies on different browsers and operating systems, right-clicking and selecting “Inspect” is the simplest way to see the HTML structure of sites and find relevant data pieces for scrapers.
Getting acquainted with Inspect Element and Developer Tools, in general, can provide insights into a website’s performance, help test and troubleshoot web scrapers, fix bugs, and improve SEO performance, as well as provide information about how websites behave.
