1. Getting Started

  • All project tests can be run from the command line with pytest. By default, this will:

    • load test db schema from a sql file (to speed up db creation). This step will be skipped if the test db is already created.

    • ensure that all migrations are applied.

    • load initial data into the db if it does not exists.

    • run all tests that are:

      • under testpaths defined in pytest.ini and,

      • in python modules whose names start with test_ and,

      • defined as method of a TestCase subclass. The method name must start with test.

  • Why pytest as a test runner?

    • clearer test output

    • provides useful options specifically for tests and has a healthy ecosystem of plugins

    • ability to configure running tests with sensible defaults using pytest.ini

1.1. About Running Tests

  • By default, tests run with PEARL_PROJECT=config.api.settings that is needed for running the PEARL api tests. To run green door tests, export_green_door && pytest pearlcertification/green_door/tests

  • To force db re-creation, use pytest --create-db

  • By default only one test db is created which corresponds to the ‘default’ db. This speeds up test setup phase. However, if the second db is needed for the tests, set the environment variable MULTI_DB_TESTS=True and a second test db will be setup automatically when running pytest.

  • use pytest -x to stop after first failing test and pytest --max-fail=n to stop after n failing tests.

  • Different ways to run tests:

    • all tests: pytest

    • all tests under a directory: pytest path/to/directory/

    • all tests in a file: pytest path/to/test_file.py

    • all tests in a class: pytest path/to/test_file.py::TestClass

    • just one test: pytest path/to/test_file.py::TestClass::test_method

  • Drop to a python debugger on first failure: pytest -x --pdb

More information about running tests can be found here

1.2. About Writing Tests

Before writing any tests, make sure to check the following resources to get a better understanding of how to write tests:

  • Mozilla has a well-written testing tutorial. It’s a must-read before writing tests.

  • Since we are writing unittest style tests (test classes inherit from TestCase), this django testing tutorial is also a good place to check out the testing tools django provides.

To be added later (but already present in references in this page)

  • Approach to testing (why pytest and sticking with Django TestCase)

  • Process for Test Discovery

  • notes on methods run when a test is run (setUpTestData/setUp/tearDown)

  • On writing tests

  • Note about TransactionTestCase

  • Good options to keep in mind while running pytest

1.3. Note about scoring tests in case of any change in question options

In Case of add, edit or delete of a question option in order for the tests database to take account of the changes the following steps should be applied:

  • On a fresh recent production database migrate the changes you desire.

  • Run the management command dump_dbdata of data_management app this will update the json db files under dbdata folder.

  • Run the management command load_dbdata of data_management app on any database you need.

You should now be able to run your tests.

1.4. Update DB dump to speed up test db creation

Over time, the number of migrations grows and with it the time needed to create the test database. To avoid having to wait a long time during test db creation, we have a ready-to-use db dump dev/files/pearl.sql that we keep up-to-date with latest migrations to avoid having to run migrations from scratch. Whenever, it starts taking more time to run migrations after loading the db dump, it means it’s time to update the db with the latest migrations.

To do that, follow these step:

  • create a new db from the command line inside the vagrant VM echo "create database test_db;" | mysql -u root -p

  • update the DATABASE_URL env variable to DATABASE_URL=mysql://root:root@localhost:3306/test_db

  • run migrations for each site in the project:

    • export_cert && migrate

    • export_api && migrate

    • export_pqs && migrate

    • export_green_door && migrate

  • Finally, update pearl.sql using mysqldump -u root -proot test_db > /vagrant/dev/files/pearl.sql