Testing Brython apps using PyTest and Selenium

2016-09-09 00:00:00 Python, Brython, Selenium, PyTest, Travis

During my vacation this year I spent some time writing a web-based latex editor. I chose Python for the serverside and Javascript on the client but soon got fed up with Javascript. I eventually stumbled upon Brython and tried to integrate it with the Angular JS library. After hitting a wall several times and spending unproductive hours trying to figure out what is wrong I decided that it might be easier to write an angular clone in Python rather than trying to shoehorn the Javascript library into working well with Brython. Thus was born the Circular library. This blog post explains how I go about testing the library.

I chose to use the PyTest testing framework, mainly because of the minimal boilerplate code needed to write tests. Since the Circular library is not a normal python library but a library which is supposed to be used in the browser, testing it is not as straightforward as a regular python library. Some parts of it (e.g. the expression parser) can be tested as normal, however, the parts dealing with the DOM (e.g. the templates) require some preparation. The simplest approach (and the approach I took first) would be to just write a dummy implementation of the Brython browser module which would simulate the browser environment and use that in the tests. This works quite well for basic tests, but it can only go so far. After a comment by Glyph on the Brython issues list, I decided to try an alternative approach testing the library in a real Browser environment.

Basic setup

The first problem I needed to overcome with this approach is to avoid having to manually fire up the browser to run the tests. Since I wanted the tests to run on Travis CI, this was a no-go. Fortunately, this problem has long been solved. The selenium browser automator even has a nice PyTest plugin. This plugin needs a driver (essentially a plugin for the specific browser used to run the tests). I chose to use the PhantomJS driver since it is headless and thus suitable for use on Travis CI.

Getting all the dependencies is a matter of a few straightforward pip install and npm install commands:

/tmp/blog $ pip install pytest
/tmp/blog $ pip install pytest-selenium
/tmp/blog $ npm install phantomjs
/tmp/blog $ export PATH=$PATH:/tmp/blog/node_modules/.bin/

The npm command installs the phantomjs binary into node_modules/.bin and the last line puts it in the search path so that that pytest-selenium can find it.

Writing a test using pytest-selenium is now very easy:

def test_google(selenium):
    selenium.get("http://www.google.com")
    logo = selenium.find_element_by_id("hplogo")
    assert logo.get_attribute('title') == "Google"

Lets save the above file as tests/test_google.py and run the test:

/tmp/blog $ mkdir tests/results
/tmp/blog $ python -m pytest -rw --driver PhantomJS --html tests/results/results.html tests
python -m pytest -rw --driver PhantomJS --html tests/results/results.html tests
======================================== test session starts ========================================
platform linux -- Python 3.4.3, pytest-3.0.2, py-1.4.31, pluggy-0.3.1
sensitiveurl: .*
driver: PhantomJS
rootdir: /tmp/blog, inifile:
plugins: base-url-1.1.0, html-1.10.0, variables-1.4, selenium-1.3.1
collected 1 items

tests/test_google.py .

--------------------- generated html file: /tmp/blog/tests/results/results.html ---------------------
===================================== 1 passed in 3.21 seconds ======================================

The --html option will tell pytest to create a nice summary of the tests in tests/tesults/results.html.

Writing a real test

O.K., so the basic setup is ready and working. Now we will write a real test. Say we would like to test that the interpolation features of circular work, i.e. if we have a template

<span> Hello {{ name }} </span>

and bind it to a context with a suitable name attribute, we would like to test that the result will be, e.g.,

<span> Hello Peter </span>

In reality the full page would look something like this:

<!DOCTYPE html>
<html>
<head>

<!--
    This tells Brython to look in the lib subdirectory of
    the web root for the circular library
-->
<link rel="pythonpath" href="lib" hreflang="py" />

<!--
    This loads the Brython library (assuming it is installed
    in the lib subdirectory of the web root)
-->
<script src="/lib/brython/www/src/brython_dist.js"></script>

<!--
    The onLoadHandler calls the brython function to compile and
    run all python scripts on the page
-->
<script>
var onLoadHandler = function() {brython();};
</script>

<!--
    This contains a python script which sets up the template
    and data binding
-->
<script type="text/python">

// Python code setting up the context, template and binding

</script>
</head>
<body onload="onLoadHandler()">
<div id='template'>
    <span id='greeting'> Hello {{ name }} </span>
</div>
</body>
</html>

where the script tag with the test/python type would contain:

from browser import document as doc, html
from circular.template import Template, Context, set_prefix

# Compile the template whose root element is
# the div element with id 'template'
tpl = Template(doc['template'])

# Create a new context
ctx = Context()

# Bind the template to the context
tpl.bind_ctx(ctx)

# Set the name attribute to an appropriate value
ctx.name = 'Peter'

# This should update the template with the new value
# (alternatively, we could just wait a few miliseconds
#  before the template updates automatically)
tpl.update()

Lets save this to the file test_interpolation.html. If we opene the file in a browser, it should greet Peter. Now we need to write a test for it. For this we will start a web-server to serve the page and the libraries, say on localhost:7654. This is easily done with the http module in Python 3 or SimpleHTTPServer in Python 2:

/tmp/blog $ python -m http.server 7654

The test itself is pretty straightforward:

def test_interpolation(selenium):
    selenium.get("http://localhost:7654/test_interpolation.html")
    greeting = selenium.find_element_by_id("greeting")
    assert greeting.text == "Hello Peter"

However, this test would probably fail. The reason is that the assertion would be hit before the circular library had the time to do its job. We would like to start testing only after the setup completed. This is a common problem and selenium solves it using Waits. The idea is to wait for a certain element to appear on the page before continuing with the tests. We can then add this element at the end of our script:

from browser import document as doc, html
from circular.template import Template, Context, set_prefix

tpl = Template(doc['template'])
ctx = Context()
tpl.bind_ctx(ctx)
ctx.name = 'Peter'
tpl.update()

# we add a div signalling that we
# are ready to test
doc <= html.DIV(id='finished')

In our test we use the WebDriverWait function:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def test_interpolation(selenium):
    selenium.get("http://localhost:7654/test_interpolation.html")
    WebDriverWait(selenium, 2).until(
        EC.presence_of_element_located((By.ID, "finished"))
    )
    greeting = selenium.find_element_by_id("greeting")
    assert greeting.text == "Hello Peter"

The parameter 2 tells selenium to wait at most 2 seconds and, if the element is not present after the two seconds, raise an exception (leading to a failed test).

Automating stuff

Now the test should work and we could call it a day. However though it works, it is a bit inconvenient. To write a test we need to create a html page with a lot of boilerplate code and a separate python test script with more boilerplate code. That is a lot of boring and unnecessary work. If writing tests is hard, no one will write them. Additionally, since the test is split into two files there is always the danger that when we do some changes, we update only one of them. It would be great if we could put everyting in a single file. Ideally, we would like to write the above test for example like this:

def pagescript_interpolation():
    """
        <span id='greeting'>Hello {{ name }}!</span>
    """
    from browser import document as doc, html
    from circular.template import Template, Context, set_prefix

    set_prefix('tpl-')
    tpl = Template(doc['greeting'])
    ctx = Context()
    tpl.bind_ctx(ctx)
    ctx.name = 'Peter'
    tpl.update()


def test_interpolation(selenium):
    wait_for_script(selenium)
    greeting = selenium.find_element_by_id("greeting")
    assert greeting.text == "Hello Peter!"

It turns out that, with a little bit of work, it is rather easy to achieve this. The idea is to run a setup function before each test which creates the html file for us putting the code of the pagescript_interpolation function in the <head> tag and putting the html code (which is specified in the docstring) inside the <body> tag. With a clever use of the inspect module and Jinja templates it is just a few lines of code (saved in utils.py):

from jinja2 import Template

tpl = Template(open('template.tpl', 'r').read())

def selenium_setup_helper(func):
    # Get the module from which the helper is called
    mod = inspect.getmodule(func)

    # The name of the script function is constructed
    # by replacing the "test_" prefix with the "pagescript_"
    # prefix
    script_name = 'pagescript_'+func.__name__[5:]

    # Get the actual script function from the module
    script = getattr(mod, script_name)

    # Get the code of the script function and add a command
    # to run it
    script_src = inspect.getsource(script)+"\n"+script_name+"()\n"

    # Get the content of the body tag from the docstring
    html = inspect.getdoc(script)

    # The file will be saved in the webroot subdirectory of the
    # directory where the setup_helper function is defined
    # (in practice, this should probably be somewhere more sensible)
    test_dir = os.path.dirname(inspect.getfile(selenium_setup_helper))+'/webroot/'

    # Construct the filename of the html file so that it includes
    # the name of the module and the name of the test
    out_file_name = mod.__name__+'-'+func.__name__[5:]

    # Open the file for writing
    out = open('%s/%s.html' % (test_dir, out_file_name), 'w')

    # Use the Jinja2 template to construct the html file
    out.write(tpl.render({
        'script': script_src,
        'content': html
    }))
    out.close()

def wait_for_script(selenium):
    # Find out the caller so that we can get the name
    # of the html page to open
    frame = inspect.stack()[1]
    mod = inspect.getmodule(frame[0])
    fname = frame[3]
    test_html_file = mod.__name__ + '-' + fname[5:]

    # Open the test page
    selenium.get('http://localhost:7000/tests/%s.html' % test_html_file)

    # Wait for the finished element to appear on the page
    WebDriverWait(selenium, 2).until(
        EC.presence_of_element_located((By.ID, "finished"))
    )

The template.tpl contains:

<!DOCTYPE html>
<html>
<head>
<link rel="pythonpath" href="lib" hreflang="py" />
<script src="../lib/brython/www/src/brython_dist.js"></script>
<script>
var onLoadHandler = function() {brython();};
</script>
<script type="text/python">
from browser import document as doc, html

{{ script }}

doc <= html.DIV(id='finished')
</script>
</head>
<body onload="onLoadHandler()">
{{ content }}
</body>
</html>

Now the test file looks almost exactly as we wanted:

from utils import wait_for_script, selenium_setup_helper

def setup_function(func):
    selenium_setup_helper(func)

def pagescript_interpolation():
    """
        <span id='greeting'>Hello {{ name }}!</span>
    """
    from browser import document as doc, html
    from circular.template import Template, Context, set_prefix

    set_prefix('tpl-')
    tpl = Template(doc['greeting'])
    ctx = Context()
    tpl.bind_ctx(ctx)
    ctx.name = 'Peter'
    tpl.update()


def test_interpolation(selenium):
    wait_for_script(selenium)
    greeting = selenium.find_element_by_id("greeting")
    assert greeting.text == "Hello Peter!"

Running the tests on Travis

Running the test on Travis CI is now a simple matter of writing the right script. The .travis.yml file for the circular library contains:

language: python

python:
  - "3.4"

cache: pip

# command to install dependencies
install:
  - pip install -r requirements.txt
  - npm install phantomjs

# Tests
script: ./management/test/run_tests.sh

and the script run_tests.sh contains (approximately)

# The html files for the tests are located in the
# ./tests/selenium/webroot folder. We start a
# server to serve them on the localhost on port 7000
cd ./tests/selenium/webroot
python -m http.server 7000 &
cd ../../../

python -m pytest -rw --driver PhantomJS --ignore=./tests/selenium/webroot/ tests

Exercise

Write a simple Bottle app which would serve the test files and also allow testing AJAX requests, file uploads &c.