Selenium

Selenium is a tool for browser automation. You can programmatically control what a web browser does.

It is often used for testing, but is also useful for other applications.

Installation

Install Selenium For Python:

pip install selenium

You also need a browser driver for the web browser you want to automate.

Install Web Browser drivers explains different ways to get drivers.

Selenium Manager - automates everything
WebDriver Manager for Python code to automatically update your WebDriver, but requires you write extra Python code
Download the driver yourself and put it on your PATH. (This is the original way.)

These instructions use the 3rd approach.

Download a browser driver.
- Firefox: https://github.com/mozilla/geckodriver/releases
- Safari: already installed as /usr/bin/safaridriver. But visit the link for examples of how to use it.
- Chrome & Chromium: https://chromedriver.chromium.org/downloads
- Other browser: https://www.selenium.dev/documentation/webdriver/getting_started/install_drivers/#quick-reference
Install it in a directory on your shell PATH.
- For most users, $HOME/bin works.
- Preferrably use a directory without spaces in the path.

Getting Started

Presentation: Selenium Exercise

Here are the steps start a browser and display a web page using Firefox. For this to work, Firefox needs to someone on your search PATH. You can use another browser instead of Firefox, provided you have a webdriver.

The Goal

What are the Top 10 Web Pages for a search of “Kasetsart University”?

We will write code to tell a browser to visit https://duckduckgo.com and search for “Kasetsart University”. Then we will look for hyperlinks in the search results.

from selenium import webdriver
from selenium.webdriver.common.by import By

# URL to fetch and display
url = "https://duckduckgo.com"

# create a browser instance. You can use Chrome or Safari instead of Firefox.
driver = webdriver.Firefox()
driver.get( url )

Or do the same thing using Chrome. The parameter (/path/to/chromedriver) is needed only if it is not on your shell search PATH.

browser = webdriver.Chrome('/path/to/chromedriver')
browser.get( url )

The Duckduckgo home page contains a search box (input field) with ID searchbox_input.

You can discover the element ID yourself by right-clicking on the search box and choosing Inspect in Firefox or Chrome.

Tell the browser to select the element using its ‘id’ attribute:

input_field = browser.find_element(By.ID, 'searchbox_input')
# did it work?
assert input_field != None

Enter some text in the field and press RETURN to search:

from selenium.webdriver.common.keys import Keys

# text appear in the search field when you call send_keys()
input_field.send_keys("Kasetsart University")
input_field.send_keys(Keys.RETURN)

Searching Data on a Page

The next step is to get the search results, and to get the URL for each search result.

WebDriver has find_element(by,value) and find_elements(by,value) methods to find things on a page. The by parameter can be:

By.CLASS_NAME (css class)
By.ID
By.LINK_TEXT or By.PARTIAL_LINK_TEXT
By.NAME
By.TAG_NAME

A simple technique is to look for all “a” tags.

elements = browser.find_elements(By.TAG_NAME, "a")
len(elements)
125

Too many results!

Instead, search for hyperlinks containing the text “Kasetsart”:

elements = browser.find_elements(By.PARTIAL_LINK_TEXT, "Kasetsart")
len(elements)
13

The values returned by find_elements are WebElement objects for parts of the web page’s Document Object Model (DOM). WebElements may contain:

text
attributes
other WebElements

Print the value of href for the first result:

>>> elements[0].get_attribute('href')
'https://duckduckgo.com/?q=Kasetsart%20University&t=h_'

this link refers to something on Duckduckgo itself, not what we want.

Print the top 10 links:

>>> for k in range(0,10):
>>>     print(k, elements[w].get_attribute('href'))

Results will vary. Something like:

0 https://duckducgo.com/y.js?blah_blah_blah (tracking url)
1 https://www.ku.ac.th/
2 https://en.wikipedia.org/wiki/Kasetsart_University
3 https://www.topuniversities.com/universities/kasetsart-university
4 https://www.ku.ac.th/en/campus-information
...

Simulate clicking on a hyperlink (pick one you want to visit). Let’s visit Wikipedia:

elements[2].click()

it should show the Wikipedia page for KU.

Go back:

browser.back()

Question About `WebElement` Methods

Why does WebElement have both find_element and find_elements methods?

Isn’t that redundant?

Use Type Hints to Improve Coding with Selenium

Python type hints enable the IDE to offer better command completion, syntax help, and automatic type checking.

In Selenium, the classes you interact with most often are WebElement and WebDriver, plus the Python list class (for results).

from typing import List
from selenium.webdriver.remote.webelement import WebElement
from selenium.webdriver.remote.webdriver import WebDriver

# add type hints to improve code completion in IDE

browser: WebDriver = webdriver.Firefox()

# you get a page
browser.get("https://www.google.com/search?q=Kasetsart+University")

# when you search for elements, you get a List of WebElements
elements: List[WebElement] = browser.find_element(By.TAG_NAME, "a")

Writing Unit Tests with Selenium

A good tutorial to get started is Test Automation using Selenium a good 8-part tutorial about automatic testing. Has explanations of the code.

The tutorial uses pytest, but you can do the same thing using unittest.

The Safari WebDriver page also has examples of unit testing with Selenium.

To run your unit tests on a CI server, run the browser in headless mode, which means no browser window is shown. Headless mode is much faster, too.

Resources

Test Automation using Selenium a good 8-part tutorial about automatic testing. Has explanations of the code.

Getting Started in selenium-python.readthedocs.io how to install and start using Selenium.

Locating Elements in the Selenium docs explains use of “By” with examples.

Web Automation with Python and Selenium automate a browser to play music on bandcamp.com, article on RealPython.com.

Selenium Headless Browser Testing has a good explanation of how to configure different browsers for headless mode. Examples use Java.

Chrome Driver Getting Started.

Selenium API Docs

Official API Docs, best for API reference
ReadTheDocs Version also has other info about installing and using Selenium

The two most important classes are WebDriver and WebElement:

selenium.webdriver.remote.webdriver.WebDriver
- this is the class that the browser implements
- provides get(url), back(), find_element_by_*(value), quit(), save_screenshot(), and more
selenium.webdriver.remote.webelement.WebElement
- represents an element on a web page
- this is how you interact with a web page
- WebElement is returned when you invoke find_element_by_*(value) or (in list) find_elements_by_*(value)

Web Browser in Headless Mode

A headless browser is one without a visible window as user interface. It runs entirely without any UI. Running a browser without the GUI interface makes it faster. This is useful for testing, esp. on a CI server where you must run Selenium tests in headless mode.

To run Firefox in headless mode with Selenium, use:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

my_options = Options()
my_options.headless = True     # deprecated: options.set_headless()
# test that it worked
assert my_options.headless
# this should NOT show a browser window
browser = webdriver.Firefox(options=my_options)

# gets page, but nothing shown
browser.get("https://duckduckgo.com")

This also works for Chrome (change the import).

On Linux, if options.headless = True doesn’t work, you can also set it as a command-line flag:

my_options = Options()
my_options.add_argument("--headless")
...

Running Chromedriver as a Service

If you use selenium in unit tests, the tests usually start a browser before each test and stop it after each test, which uses a lot of time. You can run Chrome as a service to reduce start-up time.