The design of wptrunner is intended to meet the following requirements:
Possible to run tests from W3C web-platform-tests.
Tests should be run as fast as possible. In particular it should not be necessary to restart the browser between tests, or similar.
As far as possible, the tests should run in a “normal” browser and browsing context. In particular many tests assume that they are running in a top-level browsing context, so we must avoid the use of an
It must be possible to deal with all kinds of behaviour of the browser under test, for example, crashing, hanging, etc.
It should be possible to add support for new platforms and browsers with minimal code changes.
It must be possible to run tests in parallel to further improve performance.
Test output must be in a machine readable form.
In order to meet the above requirements, wptrunner is designed to push as much of the test scheduling as possible into the harness. This allows the harness to monitor the state of the browser and perform appropriate action if it gets into an unwanted state e.g. kill the browser if it appears to be hung.
The harness will typically communicate with the browser via some remote control protocol such as WebDriver. However for browsers where no such protocol is supported, other implementation strategies are possible, typically at the expense of speed.
The overall architecture of wptrunner is shown in the diagram below:
The main entry point to the code is
wptrunner.py. This is responsible for setting up the test
environment, loading the list of tests to be executed, and invoking
the remainder of the code to actually execute some tests.
The test environment is encapsulated in the
TestEnvironment class. This defers to code in
web-platform-tests which actually starts the required servers to
run the tests.
The set of tests to run is defined by the
TestLoader. This is constructed with a
TestFilter (not shown), which takes any filter arguments
from the command line to restrict the set of tests that will be
TestLoader reads both the
JSON manifest and the expectation data stored in ini files and
multiprocessing.Queue of tests to run, and
their expected results.
Actually running the tests happens through the
ManagerGroup object. This takes the
tests to be run and starts a
TestRunnerManager for each
instance of the browser under test that will be started. These
TestRunnerManager instances are each started in their own
TestRunnerManager coordinates starting the product under
test, and outputting results from the test. In the case that the test
has timed out or the browser has crashed, it has to restart the
browser to ensure the test run can continue. The functionality for
initialising the browser under test, and probing its state
(e.g. whether the process is still alive) is implemented through a
Browser object. An implementation of this class must be
provided for each product that is supported.
The functionality for actually running the tests is provided by a
TestRunner instances are
run in their own child process created with the
multiprocessing module. This allows them to run concurrently
and to be killed and restarted as required. Communication between the
TestRunnerManager and the
provided by a pair of queues, one for sending messages in each
direction. In particular test results are sent from the
TestRunner to the
TestRunnerManager using one
of these queues.
TestRunner object is generic in that the same
TestRunner is used regardless of the product under
test. However the details of how to run the test may vary greatly with
the product since different products support different remote control
protocols (or none at all). These protocol-specific parts are placed
TestExecutor object. There is typically a different
TestExecutor class for each combination of control protocol
and test type. The
TestRunner is responsible for pulling
each test off the
multiprocessing.Queue of tests and passing it down to
The executor often requires access to details of the particular
browser instance that it is testing so that it knows e.g. which port
to connect to to send commands to the browser. These details are
encapsulated in the