Selenium

Selenium is a tool for browser automation. You can programmatically control what a web browser does.

It is often used for testing, but it has many other uses, like page scraping.

Installation

Install Selenium For Python:

pip install selenium

You also need a browser driver (“web driver”) for the web browser you want to automate.

Install a Web Driver

There are 3 ways to get a web driver (also called browser driver).

  1. Download the driver yourself and put it on your PATH. (This is the original way.)
  2. Install Selenium Manager - it automates everything
  3. Install WebDriver Manager for Python code to automatically update your WebDriver, but requires you write extra Python code

Install Web Browser drivers from the Selenium docs explains how to get drivers.

These instructions use the 1st approach.

  1. Download a browser driver.
  2. Install it in a directory on your shell PATH.
    • For most users, $HOME/bin works.
    • Preferrably use a directory without spaces in the path.
  3. Test it. This code should open a new browser window: 
    from selenium import webdriver

# This should open a new browser window.
# You can use Chrome or Safari instead of Firefox.
driver = webdriver.Firefox()

    If it opens a window, try fetching a page (any URL is ok).

    driver.get( "https://www.cpe.ku.ac.th" )

Getting Started

Presentation: Selenium Exercise

Here are the steps start a browser and display a web page using Firefox. For this to work, Firefox needs to someone on your search PATH. You can use another browser instead of Firefox, provided you have a webdriver.

The Goal

What are the Top 10 Web Pages for a search of “Kasetsart University”?

We will write code to tell a browser to visit https://duckduckgo.com and search for “Kasetsart University”. Then we will look for hyperlinks in the search results.

from selenium import webdriver
from selenium.webdriver.common.by import By

# URL to fetch and display
url = "https://duckduckgo.com"

# create a browser instance. You can use Chrome or Safari instead of Firefox.
driver = webdriver.Firefox()
driver.get( url )

Or do the same thing using Chrome. The parameter (/path/to/chromedriver) is needed only if it is not on your shell search PATH.

browser = webdriver.Chrome('/path/to/chromedriver')
browser.get( url )

The Duckduckgo home page contains a search box (input field) with ID searchbox_input.

You can discover the element ID yourself by right-clicking on the search box and choosing Inspect in Firefox or Chrome.

Tell the browser to select the element using its ‘id’ attribute:

input_field = browser.find_element(By.ID, 'searchbox_input')
# did it work?
assert input_field != None

Enter some text in the field and press RETURN to search:

from selenium.webdriver.common.keys import Keys

# text appear in the search field when you call send_keys()
input_field.send_keys("Kasetsart University")
input_field.send_keys(Keys.RETURN)

Searching Data on a Page

The next step is to get the search results, and to get the URL for each search result.

WebDriver has find_element(by,value) and find_elements(by,value) methods to find things on a page. The by parameter can be:

  • By.CLASS_NAME (css class)
  • By.ID
  • By.LINK_TEXT or By.PARTIAL_LINK_TEXT
  • By.NAME
  • By.TAG_NAME

A simple technique is to look for all “a” tags.

elements = browser.find_elements(By.TAG_NAME, "a")
len(elements)
125

Too many results!

Instead, search for hyperlinks containing the text “Kasetsart”:

elements = browser.find_elements(By.PARTIAL_LINK_TEXT, "Kasetsart")
len(elements)
13

The values returned by find_elements are WebElement objects for parts of the web page’s Document Object Model (DOM). WebElements may contain:

  • text
  • attributes
  • other WebElements

Print the value of href for the first result:

>>> elements[0].get_attribute('href')
'https://duckduckgo.com/?q=Kasetsart%20University&t=h_'

this link refers to something on Duckduckgo itself, not what we want.

Print the top 10 links:

>>> for k in range(0,10):
>>>     print(k, elements[w].get_attribute('href'))

Results will vary. Something like:

0 https://duckducgo.com/y.js?blah_blah_blah (tracking url)
1 https://www.ku.ac.th/
2 https://en.wikipedia.org/wiki/Kasetsart_University
3 https://www.topuniversities.com/universities/kasetsart-university
4 https://www.ku.ac.th/en/campus-information
...

Simulate clicking on a hyperlink (pick one you want to visit). Let’s visit Wikipedia:

elements[2].click()

it should show the Wikipedia page for KU.

Go back:

browser.back()

Question About WebElement Methods

Why does WebElement have both find_element and find_elements methods?

Isn’t that redundant?

Use Type Hints to Improve Coding with Selenium

Python type hints enable the IDE to offer better command completion, syntax help, and automatic type checking.

In Selenium, the classes you interact with most often are WebElement and WebDriver, plus the Python list class (for results).

from typing import List
from selenium.webdriver.remote.webelement import WebElement
from selenium.webdriver.remote.webdriver import WebDriver

# add type hints to improve code completion in IDE

browser: WebDriver = webdriver.Firefox()

# you get a page
browser.get("https://www.google.com/search?q=Kasetsart+University")

# when you search for elements, you get a List of WebElements
elements: List[WebElement] = browser.find_elements(By.TAG_NAME, "a")

Wait for Pages to Load

When you use Selenium is a Python script, the command to “get” a page may return before the page finishes loading. Then the next command might not find the thing that you want on the page (which is not completely loaded yet).

Selenium has several commands to set “wait” times for results.

One of them is implicit_wait, which applies to all page loads after it is set:

# Wait up to 10 seconds for pages to load

browser.implicit_wait(10)

Writing Unit Tests with Selenium

A good tutorial to get started is Test Automation using Selenium a good 8-part tutorial about automatic testing. Has explanations of the code.

The tutorial uses pytest, but you can do the same thing using unittest.

The Safari WebDriver page also has examples of unit testing with Selenium.

To run your unit tests on a CI server, run the browser in headless mode, which means no browser window is shown. Headless mode is much faster, too.

Resources

Test Automation using Selenium a good 8-part tutorial about automatic testing. Has explanations of the code.

Getting Started in selenium-python.readthedocs.io how to install and start using Selenium.

Locating Elements in the Selenium docs explains use of “By” with examples.

Web Automation with Python and Selenium automate a browser to play music on bandcamp.com, article on RealPython.com.

Selenium Headless Browser Testing has a good explanation of how to configure different browsers for headless mode. Examples use Java.

Chrome Driver Getting Started.

Selenium API Docs

The two most important classes are WebDriver and WebElement:

Web Browser in Headless Mode

A headless browser is one without a visible window as user interface. It runs entirely without any UI. Running a browser without the GUI interface makes it faster. This is useful for testing, esp. on a CI server where you must run Selenium tests in headless mode.

To run Firefox in headless mode with Selenium, use:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

my_options = Options()
my_options.headless = True     # deprecated: options.set_headless()
# test that it worked
assert my_options.headless
# this should NOT show a browser window
browser = webdriver.Firefox(options=my_options)

# gets page, but nothing shown
browser.get("https://duckduckgo.com")

This also works for Chrome (change the import).

On Linux, if options.headless = True doesn’t work, you can also set it as a command-line flag:

my_options = Options()
my_options.add_argument("--headless")
...

Running Chromedriver as a Service

If you use selenium in unit tests, the tests usually start a browser before each test and stop it after each test, which uses a lot of time. You can run Chrome as a service to reduce start-up time.