Writing automated tests with Django&Selenium

on Apr 25, 2013 by Ramona


     Getting our Selenium tests to run faster can be complicated at times, because there are a number of factors to be taken into consideration. Debugging these tests is time consuming and the end result is almost always the same one - a large part of the tests needs to be refactored.

      Luckily, for projects implemented with Django, we have the possibility to explore a wide range of testing solutions, backed up by an active internet community.
Instead of always trying to use Selenium as a standalone solution, we could analyze the possibility of having those tests integrated in our Django project, and hence, be able to run with a simple command, all tests. By all tests, I mean unit tests, functional tests (please see related articles on this blog), and Selenium tests (which theoretically should focus more on the JS/CSS/HTML side of the application, rather than on its entire functionality).
 
     OK, sounds good in theory, but where to from this point?
Although it might seem complicated at first, all you need to do is clone your Django project and start adding your tests in your app’s folder (several apps on be maintained and tested for the same Django project).
 
      For a practical example, please have a look at this GitHub repo. This is just an example of how Selenium tests can be integrated with Django, with the use of LiveServerTestCase class.
Make sure to run pip install -r requirements.txt and you’re good to go. It is recommended that you use a virtualenv, so that you avoid any conflicts with software already installed on your computer.
 
     The application used for testing is django-registration, which provides enough support for developing functional tests. We inserted a js library (password_strength_plugin.js) to better illustrate the need of Selenium (JS/CSS/HTML focused) automated tests.
 
     The advantages of this approach are numerous - you are able to test the build as well, not only the deploy. Tests are faster, and, if you decide to write your tests using a page object pattern method, then debugging will become easier, as you’ll be able to faster track failures and their causes. 


Tags: testing django selenium | Category: testing , python , django Back To Top

Testing your django app with webtest

on Apr 25, 2013 by Danu


     I’ve been watching both presentations that Carl Meyer held at Pycon 2012/13 and I highly recommend them if you want a deep dive into writing tests with django. They outline some very good principles for writing effective and maintainable tests. They also highlight a suite of test utilities and frameworks which help you in writing better tests. Among the others, Webtest caught my attention via django-webtest for writing integration/functional tests.

     Why you should choose django-webtest instead of django client ?
     Well, first of all, it can better capture the user experience mainly because you can submit forms and follow links. You are not only testing the views but also the template html. Secondly, it will allow you to write more simple and readable tests, an important fact when it comes to integration or functional tests.
 
  1. def testLoginProcess(self):
  2.     login = self.app.get(reverse('auth_login'))
  3.     login.form['username'] = 'danu'
  4.     login.form['password'] = 'test123'
  5.     response = login.form.submit('Log in').follow()
  6.     assert_equals('200 OK', response.status)
  7.     assert_contains(response, 'Welcome danu :]', count=1, status_code=200)
  8.  
  9. def testLoginWithInvalidCredentials(self):
  10.     login = self.app.get(reverse('auth_login'))
  11.     login.form['username'] = 'foo'
  12.     login.form['password'] = 'bar'
  13.     response = login.form.submit('Log in')
  14.     assert_contains(
  15.         response,
  16.         'Please enter a correct username and password. '
  17.         'Note that both fields are case-sensitive.',
  18.         count=1,
  19.         status_code=200
  20.     )
 
     It interacts with django via the WSGI interface so ajax, js will not be tested. For that purpose you normally use selenium (see Writing automated tests with Django&Selenium).
 
     I’ve set up a project on github to illustrate some automated tests with django-webtest. It uses nose test suite runner since i don’t want to to import all my tests into tests/__init__.py and for rest of the goodness that nose offers.

 


Tags: django functional webtest integration | Category: testing , python , django Back To Top

Running python tests with GhostDriver in Travis CI

on Feb 18, 2013 by Ramona


Selenium is great by itself, but its API grows every day - it’s not obvious anymore whether to use “type” or “typeKeys”. 

So Google has come up with a new API - WebDriver - which is integrated within Selenium 2.0. Rather than being a JavaScript application running within the browser, WebDriver uses whichever mechanism is most appropriate to control the browser - for Firefox, it’s an extension; for IE, WebDriver makes use of IE's Automation controls and so on. By changing the way the browser is controlled, WebDriver manages to overcome some of the restrictions put in place by the JavaScript security model.

Selenium WebDriver
Selenium 2.0 has integrated the WebDriver API, thus combining the advantages of both technologies - Selenium has support for many common situations you might want to test, and WebDriver drives the browser directly (unlike Selenium RC, which simply injects JavaScript into the browser), using the browser's built-in support for automation.
 
Hence, there is more to Selenium than it might seem at a first glance, and things have evolved over the years. The addition of WebDriver, iPhoneDriver, ChromeDriver, etc., has made it easier for test engineers to view Selenium as the tool of choice for automating testing on their products.
 
And the best is yet to come...
We all know that Selenium is a great tool, but we also know that sometimes it takes much more time than desired to actually see all tests have completely run.
So, how about a headless browser?
 
Running Selenium scripts headless is now easier than before, if using PhantomJS & GhostDriver. Of course, you can still use xvfb (or even better PyVirtualDisplay wrapper) for GUI headless tests.
 
GhostDriver is a pure JavaScript implementation of the WebDriver Wire Protocol for PhantomJS. It's a Remote WebDriver that uses PhantomJS as back-end. Ok, so what’s PhantomJS?
PhantomJS is a headless WebKit with JavaScript API. As you know, WebKit is the layout engine that Chrome, Safari and other browsers use. So, PhantomJS is a browser, but a headless one. This means that the rendered web pages are never actually displayed.
Let’s give it a try and see how can we set up the environment for GhostDriver & PhantomJS.
 
  1. PhantomJS
  • On Windows:
    • Download PhantomJS from here and extract it.
    • Add the folder to the system path and run phantomjs --version in a new command prompt. You should see the actual version (just to verify there are no errors so far).
  • On Linux (Ubuntu) - command line:
cd /usr/local/share
wget http://phantomjs.googlecode.com/files/phantomjs-1.8.1-linux-x86_64.tar.bz2
tar xjf phantomjs-1.8.1-linux-x86_64.tar.bz2
sudo ln -s /usr/local/share/phantomjs-1.8.1-linux-x86_64/bin/phantomjs /usr/local/share/phantomjs
sudo ln -s /usr/local/share/phantomjs-1.8.1-linux-x86_64/bin/phantomjs /usr/local/bin/phantomjs
sudo ln -s /usr/local/share/phantomjs-1.8.1-linux-x86_64/bin/phantomjs /usr/bin/phantomjs

It is important that you have installed version 1.8*, because GhostDriver is not integrated into PhantomJS’s older versions.

  1. Start GhostDriver by setting phantomjs --webdriver=PORT, where PORT can be any available port. You should see a successful message, indicating that GhostDriver is running on port PORT.
  2. Configure your test environment to use PhantomJS:
  • For example, create a folder structure like:
    • headless_testing, containing a subfolder test, where I will add my selenium scripts written with Python
    • Add the following two files inside test folder:
      • base_test.py (but delete the first two lines - there's no need for these as we will place all py files in the same folder):
import sys
sys.path.insert(0, "utils")
  • Modify the base_test.py file as follows:
    • replace port 8080 with the port where GhostDriver is running for you (line 59)
    • replace open("../config.ini") with open("test/config.ini"(line 40)
  • Add your own selenium python script to the same test folder. Make sure your methods are named as test_*, so that you can run them using unittest module.
  • Add a config.ini file in the test folder, containing driver=phantomjs. In this way, you’re telling Selenium it should use the headless PhantomJS browser. If you don't want your config.ini file to be mixed up with the python scripts, you could add it to a separate folder, and modify the base_test.py file accordingly.
  • pip install discover ( the test discovery mechanism and load_tests protocol for unittest backported from Python 2.7 to work with Python 2.4 or more recent (including Python 3).)
Give it a try: while in headless_testing folder, open a command prompt window and run:
python -m unittest discover -s test --pattern=test*.py
 
GhostDriver in Travis CI
And since automation tests are most valuable in a CI server, here’s an example of how you could set up selenium scripts with PhantomJS on Travis CI.
  • Log in to Travis with GitHub and enable service hooks for the repositories you want added to Travis. More details here.
  • On your local machine, drill inside headless_testing folder and add a file called .travis.yml - this is the config file that will tell Travis how to run your selenium python scripts:

 

language: python

python:
- "2.7"

install: "pip install -r requirements.txt --use-mirrors"

script:
- "python -m unittest discover -s test --pattern=test*.py"

before_script:
- "python test/start_ghostdriver.py"
  • Add this to the requirements.txt file (separate lines): selenium==2.28.0 and discover==0.4.0
  • The start_ghostdriver.py script is a script I created in order to be able to start GhostDriver and get the control back (so that you are able to run other commands from the command line as well). The content of this script is:

 

import os
os.system('phantomjs --webdriver=PORT 2>&1 > log.txt &')
#os.system('phantomjs --webdriver=PORT 2>&1 > /dev/null &')
  • Travis comes with PhantomJS configured, so no other action is needed in this direction.​​
  • Make sure to add the start_ghostdriver.py file inside headless_testing/test folder. What it does is that it starts a process in the background and it captures the output in the log.txt file. Adding the & ensures the fact that you will gain control back in the command line. Only two lines will be written inside the log file, so you shouldn’t worry about it too much.
  • Push the content of the headless_testing folder to GitHub (you might want to use a .gitignore file as well).
 
Travis will detect the new push and as soon as an worker is available, it will start building your code. You will be able to see the output in real time and enjoy the benefits of headless testing with PhantomJS&GhostDriver.
 
Please have a look as well at the GitHub repository we've created for PhantomJS&GhostDriver.
 


Tags: travis testing python selenium automated ci | Category: testing , python Back To Top

Functional tests with lettuce and django test client

on Feb 01, 2013 by Danu


     When running a functional test, you fire up “browser” and do the "same" actions as a real user (or API client). There are different “browsers” for testing your applications, some of them are real, like selenium and some of them are less real, like django test client. Depending on the context, each of them has its pros and cons.

- django test client is very fast, since you don’t need a browser engine, real http listening and so on, but JavaScript and/or Ajax views are not tested

- selenium is a browser-driving library, opens a real browser and tests rendered HTML alongside with behavior of Web pages. Write selenium tests for Ajax, other JS/server interactions.

A complete test suite should contain both test types.

 

Functional tests and Django

Write functional/integration/system (has more names than it needs) tests for views. Unit testing the view is hard because views have many dependencies (templates, db, middleware, url routing etc). 

You should definitely have django functional tests but chances are you should have fewer than you have now.  See the software testing pyramid by Alister Scott which provides solid approach to automated testing and shows the mix of testing a team should aim for.

 

Functional tests:

- test that the whole integrated system works, catch regressions

- they tend to be slow

- will catch bugs that unit tests will not, but it's harder to debug

- write fewer (more unit tests)

 

So what can you test with django-test client ?

- the correct view is executed for a given url

- simulate post, get, head, put etc. requests

- the returned content has the expected values (you can use beautiful soup, lxml or html5lib for parsing the content)

 

Example of functional test with lettuce and django test client

Feature: Register
    In order to get access to app
    A user should be able to register
 
 
Scenario: User registers
    Given I go to the "/register/" URL
    When I fill register form with:
        | username | email       | password1 | password2 |
        | danul    | dan@dan.com | test123   | test123   |
    And I submit the data
    Then I should see "Check your email"
    And I should receive an email at "dan@dan.com" with the subject "Activate your djangoproject.com account - you have 7 days!"
    And I activate the account
    Then I should see "Congratulations!"
 
 
Scenario: Users login
    Given following users exist
      | username | password |
      | danu     | test123  |
      | lulu     | test123  |
    When I go to the "/login/" URL
    And I login as user "danu"
    Then I should see "Welcome, danu. Thanks for logging in."

 

Steps

import re

from bs4 import BeautifulSoup

from lettuce import *
from django.core import mail
from nose.tools import assert_equals

from django.contrib.auth.models import User


@step(u'I go to the "(.*)" URL')
def i_go_to_the_url(step, url):
    response = world.browser.get(url)
    assert_equals(response.status_code, 200)
    world.html = BeautifulSoup(response.content)

@step(u'When I fill register form with:')
def when_i_fill_in_user_data_with(step):
    for data in step.hashes:
        world.data = data
    assert_equals(len(world.html.select('form')), 1)
    assert_equals(len(world.html.find_all('input', 'required')), 4)

@@step(u'And I submit the data')
def and_i_submit_the_data(step):
    world.response = world.browser.post('/register/', world.data, follow=True)
    assert_equals(
        User.objects.filter(username=world.data['username']).exists(), True
    )
    assert_equals(world.response.status_code, 200)

@step(u'I should see "(.*)"')
def i_should_see(step, expected_response):
    html = BeautifulSoup(world.response.content)
    expected_text = html.find('h1').get_text()
    assert_equals(expected_text, expected_response)

@step(u'And I should receive an email at "([^"]*)" with the subject "([^"]*)"')
def i_should_receive_email_with_subject(step, address, subject):
    assert_equals(mail.outbox[0].to[0], address)
    assert_equals(mail.outbox[0].subject, subject)

@step(u'And I activate the account')
def and_i_activate_the_account(step):
    activation_url = re.findall(
        r'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+',
        mail.outbox[0].body
    )
    world.response = world.browser.get(activation_url[0], follow=True)
    assert_equals(world.response.status_code, 200)

@step(u'Given following users exist')
def given_following_users_exist(step):
    for user_hash in step.hashes:
        user_data = user_hash.copy()
        password = user_data.pop("password", None)
        user, created = User.objects.get_or_create(**user_data)
        user.set_password(password)
        user.save()

@step(u'And I login as user "([^"]*)"')
def and_i_login_as_user_group1(step, username):
    if not User.objects.filter(username=username).exists():
        User.objects.create_user(
            username=username,
            password='test123',
        )
    assert_equals(len(world.html.select('form')), 1)
    assert_equals(len(world.html.find_all('input')), 3)
    world.response = world.browser.post(
        '/login/',
        {'username': username, 'password': 'test123'},
        follow=True
    )
    assert_equals(world.response.status_code, 200)

 

terrain

import logging

from django.conf import settings
from django.core.management import call_command
from django.test import Client
from django.db import connection
#from django.test.utils import setup_test_enviroment, teardown_test_environment

from lettuce import *


@before.all
def initial_setup():
    logging.info("Loading django's test client ...\n")
    world.browser = Client()

    logging.info("Setting up a test database ...\n")

    try:
        from south.management.commands import patch_for_test_db_setup
        patch_for_test_db_setup()
    except ImportError:
        pass

    # Setup test environment / called by harvest.py
    #setup_test_enviroment()

    world.testdb = connection.creation.create_test_db(
        verbosity=2,
        autoclobber=False,
    )


@after.all
def teardown_browser(total):
    logging.info("Destroy test database ...\n")
    connection.creation.destroy_test_db(world.testdb, verbosity=2)

    # Tear Down the test environment / called by harvest.py
    #teardown_test_enviroment()

 

What's next ?

Well ... WebTest  :) (http://pyvideo.org/video/699/testing-and-django)

Be a good person and write functional tests. Functional testing is something that every app needs, no testing strategy is complete without high-level tests to ensure the entire programming system works together.

 

Source code for this article:

https://github.com/danclaudiupop/django-lettuce-labtests


Tags: test lettuce django client bs4 selenium bdd | Category: testing , python Back To Top

Testing registration scenario against django with lettuce and selenium webdriver

on Jan 14, 2013 by Danu


Installation

The best way to “understand” is by following a good example, so we prepared a sample project which uses django-registration library in order to have a simple, generic user-registration application for our tests. To play with it, just create a new virtualenv and checkout the sample project on Github. The only dependencies are django, selenium, nose and django-registration, so a simple pip install -r requirements.txt inside your virtualenv will be enough.
 

Settings

webdriver

Because the default webdriver API is rather low-level and lacks things necessary for creating suites of automated web tests another option is to use splinter, which provides a much nicer Python interface to Selenium. We'll stick to selenium-webdriver because we don't need another layer of abstraction for this simple demo app.

We’ll use Firefox by default since it has webdriver support built-in. If you want to use Chrome you’ll need to download the chrome driver separately. 

 

lettuce_settings

First let's extend django default settings with lettuce specific configurations. This way we avoid running tests against dev db and declare another db to run tests against. This is actually bad, the lack of test database integration with django and the fact that lettuce server is used over LiveServerTestCase may cause reliability issues in the future (see django test database). Add lettuce.django to INSTALLED_APPS. You can also specify the app names in LETTUCE_APPS to run tests against, so that you won’t need to type the app name as command-line parameters. 

from settings import *


DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': 'lettuce.db',                   # Or path to database file if using sqlite3.
        'USER': '',                             # Not used with sqlite3.
        'PASSWORD': '',                         # Not used with sqlite3.
        'HOST': '',                             # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '',                             # Set to empty string for default. Not used with sqlite3.
    }
}

INSTALLED_APPS += (
    'lettuce.django',
)

LETTUCE_APPS = (
    'registerap',
)

 

Test structure

Lettuce will look for a features folder inside every installed app. For each app you want to create lettuce tests you have to create a sub folder called features. Our features are stored under registerapp folder. The "feature" file contains scenarios, while ".py" file contains python code executed for steps used in scenarios.

register.feature

Feature: Register
    In order to get access to app 
    A user should be able to register


Scenario: User registers
    Given I go to the "/register" URL
    When I fill in "username" with "danul"
    And I fill in "email" with "danclaudiupop@gmail.com"
    And I fill in "password1" with "test123"
    And I fill in "password2" with "test123"
    And I press "submit"
    Then I should see "Check your email"
    And I should receive an email at "danclaudiupop@gmail.com" with the subject "Activate your djangoproject.com account - you have 7 days!"
    And I activate the account
    Then I should see "Congratulations!"

Scenario: User logs in successfully
    Given I go to the "/login" URL
    When I fill in "username" with "danul"
    And I fill in "password" with "test123"
    And I press "submit"
    Then I should see "Welcome, danul. Thanks for logging in."

 

register.py

We are importing django_url utility function so we can easily access web pages within steps using relative urls. We also importing the email backend that sends mails to a multiprocessing queue. Because Django server runs in a different process than lettuce you are not able to use the EMAIL_BACKEND specified in settings file, not to mention that harvest.py is using django.test.utils.setup_test_environment that overrides email backend to locmem

import re

from lettuce import step, world
from lettuce.django import django_url
from lettuce.django import mail
from nose.tools import assert_equals


@step(u'I go to the "(.*)" URL')
def i_go_to_the_url(step, url):
    world.response = world.browser.get(django_url(url))

@step(u'I fill in "(.*)" with "(.*)"')
def i_fill_in(step, field, value):
    world.browser.find_element_by_name(field).send_keys(value)

@step(u'I press "(.*)"')
def i_press(step, button_id):
    button = world.browser.find_element_by_id(button_id)
    button.click()

@step(u'I should see "(.*)"')
def i_should_see(step, expected_response):
    h1 = world.browser.find_element_by_tag_name('h1')
    assert_equals(h1.text, expected_response)

@step(u'And I should receive an email at "([^"]*)" with the subject "([^"]*)"')
def i_should_receive_email_with_subject(step, address, subject):
    message = mail.queue.get(True, timeout=5)
    world.email_body = message.body
    assert_equals(message.subject, subject)
    assert_equals(message.recipients(), [address])

@step(u'And I activate the account')
def and_i_activate_the_account(step):
    activation_url = re.findall(
        r'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+',
        world.email_body
    )
    world.response = world.browser.get(activation_url)

 

terrain.py 

Its about setup and teardown. We setup the browser instance using webdriver. We prepare the test environment by running the django syncdb command which creates the database tables as well loading any fixtures named initial_data. Teardown test environment by destroying the test database. Different hooks and commands can be used, for e.g: before each scenario load specific fixtures using loaddata.

from django.core.management import call_command
from django.db import connection
from django.conf import settings

from lettuce import before, after, world
from selenium import webdriver


@before.all
def initial_setup():
    call_command('syncdb', interactive=False, verbosity=1)
    world.browser = webdriver.Firefox()

@after.all
def teardown_browser(total):
    connection.creation.destroy_test_db(settings.DATABASES['default']['NAME'])
    world.browser.quit()

 

Running lettuce tests

All that’s left is running the tests:

python manage.py harvest tests/lettuce_selenium_tests --settings=conf.settings_lettuce


Source code for this project can be found on github:

https://github.com/danclaudiupop/django-lettuce-labtests

See also:

Web development fun with Lettuce and Django


Tags: webdriver django selenium lettuce bdd | Category: testing , python Back To Top