wptrunner Design

The design of wptrunner is intended to meet the following requirements:

• Possible to run tests from W3C web-platform-tests.

• Tests should be run as fast as possible. In particular it should not be necessary to restart the browser between tests, or similar.

• As far as possible, the tests should run in a “normal” browser and browsing context. In particular many tests assume that they are running in a top-level browsing context, so we must avoid the use of an iframe test container.

• It must be possible to deal with all kinds of behaviour of the browser under test, for example, crashing, hanging, etc.

• It should be possible to add support for new platforms and browsers with minimal code changes.

• It must be possible to run tests in parallel to further improve performance.

• Test output must be in a machine readable form.

Architecture

In order to meet the above requirements, wptrunner is designed to push as much of the test scheduling as possible into the harness. This allows the harness to monitor the state of the browser and perform appropriate action if it gets into an unwanted state e.g. kill the browser if it appears to be hung.

The harness will typically communicate with the browser via some remote control protocol such as WebDriver. However for browsers where no such protocol is supported, other implementation strategies are possible, typically at the expense of speed.

The overall architecture of wptrunner is shown in the diagram below:

The main entry point to the code is run_tests() in wptrunner.py. This is responsible for setting up the test environment, loading the list of tests to be executed, and invoking the remainder of the code to actually execute some tests.

The test environment is encapsulated in the TestEnvironment class. This defers to code in web-platform-tests which actually starts the required servers to run the tests.

The set of tests to run is defined by the TestLoader. This is constructed with a TestFilter (not shown), which takes any filter arguments from the command line to restrict the set of tests that will be run. The TestLoader reads both the web-platform-tests JSON manifest and the expectation data stored in ini files and produces a multiprocessing.Queue of tests to run, and their expected results.

Actually running the tests happens through the ManagerGroup object. This takes the Queue of tests to be run and starts a TestRunnerManager for each instance of the browser under test that will be started. These TestRunnerManager instances are each started in their own thread.

A TestRunnerManager coordinates starting the product under test, and outputting results from the test. In the case that the test has timed out or the browser has crashed, it has to restart the browser to ensure the test run can continue. The functionality for initialising the browser under test, and probing its state (e.g. whether the process is still alive) is implemented through a Browser object. An implementation of this class must be provided for each product that is supported.

The functionality for actually running the tests is provided by a TestRunner object. TestRunner instances are run in their own child process created with the multiprocessing module. This allows them to run concurrently and to be killed and restarted as required. Communication between the TestRunnerManager and the TestRunner is provided by a pair of queues, one for sending messages in each direction. In particular test results are sent from the TestRunner to the TestRunnerManager using one of these queues.

The TestRunner object is generic in that the same TestRunner is used regardless of the product under test. However the details of how to run the test may vary greatly with the product since different products support different remote control protocols (or none at all). These protocol-specific parts are placed in the TestExecutor object. There is typically a different TestExecutor class for each combination of control protocol and test type. The TestRunner is responsible for pulling each test off the multiprocessing.Queue of tests and passing it down to the TestExecutor.

The executor often requires access to details of the particular browser instance that it is testing so that it knows e.g. which port to connect to to send commands to the browser. These details are encapsulated in the ExecutorBrowser class.