wptrunner Internals

class wptrunner.browsers.base.Browser(logger: StructuredLogger)

Abstract class serving as the basis for Browser implementations.

The Browser is used in the TestRunnerManager to start and stop the browser process, and to check the state of that process.

Parameters:

logger – Structured logger to use for output.

check_crash(process: int, test: str) → bool

Check if a crash occured and output any useful information to the log. Returns a boolean indicating whether a crash occured.

cleanup() → None

Browser-specific cleanup that is run after the testrun is finished

executor_browser() → Tuple[Type[ExecutorBrowser], Mapping[str, Any]]

Returns the ExecutorBrowser subclass for this Browser subclass and the keyword arguments with which it should be instantiated

abstract is_alive() → bool

Boolean indicating whether the browser process is still running

abstract property pid: int | None

pid of the browser process or None if there is no pid

settings(test: Test) → Mapping[str, Any]

Dictionary of metadata that is constant for a specific launch of a browser.

This is used to determine when the browser instance configuration changes, requiring a relaunch of the browser. The test runner calls this method for each test, and if the returned value differs from that for the previous test, the browser is relaunched.

setup() → None

Used for browser-specific setup that happens at the start of a test run

abstract start(group_metadata: Mapping[str, Any], **kwargs: Any) → None

Launch the browser object and get it into a state where is is ready to run tests

abstract stop(force: bool = False) → bool

Stop the running browser process.

Return True iff the browser was successfully stopped.

exception wptrunner.browsers.base.BrowserError

class wptrunner.browsers.base.ExecutorBrowser(**kwargs: Any)

View of the Browser used by the Executor object. This is needed because the Executor runs in a child process and we can’t ship Browser instances between processes on Windows.

Typically this will have a few product-specific properties set, but in some cases it may have more elaborate methods for setting up the browser from the runner process.

class wptrunner.browsers.base.NullBrowser(logger: StructuredLogger, **kwargs: Any)

is_alive() → bool

Boolean indicating whether the browser process is still running

property pid: int | None

pid of the browser process or None if there is no pid

start(group_metadata: Mapping[str, Any], **kwargs: Any) → None

No-op browser to use in scenarios where the TestRunnerManager shouldn’t actually own the browser process (e.g. Servo where we start one browser per test)

stop(force: bool = False) → bool

Stop the running browser process.

Return True iff the browser was successfully stopped.

class wptrunner.browsers.base.OutputHandler(logger: StructuredLogger, command: List[str], **kwargs: Any)

Class for handling output from a browser process.

This class is responsible for consuming the logging from a browser process and passing it into the relevant logger. A class instance is designed to be passed as the processOutputLine argument to mozprocess.ProcessHandler.

The setup of this class is complex for various reasons:

• We need to create an instance of the class before starting the process

• We want access to data about the running process e.g. the pid

• We want to launch the process and later setup additional log handling which is restrospectively applied to any existing output (this supports prelaunching browsers for performance, but having log output depend on the tests that are run e.g. for leak suppression).

Therefore the lifecycle is as follows:

output_handler = OutputHandler(logger, command, **output_handler_kwargs)
proc = ProcessHandler(command, ..., processOutputLine=output_handler)
output_handler.after_process_start(proc.pid)
[...]
# All logging to this point was buffered in-memory, but after start()
# it's actually sent to the logger.
output_handler.start(**output_logger_start_kwargs)
[...]
proc.wait()
output_handler.after_process_stop()

Since the process lifetime and the output handler lifetime are coupled (it doesn’t work to reuse an output handler for multiple processes), it might make sense to have a single class that owns the process and the output processing for the process. This is complicated by the fact that we don’t always run the process directly, but sometimes use a wrapper e.g. mozrunner.

class wptrunner.browsers.base.OutputHandlerState(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

class wptrunner.browsers.base.WebDriverBrowser(logger: StructuredLogger, binary: str | None = None, webdriver_binary: str | None = None, webdriver_args: List[str] | None = None, host: str = '127.0.0.1', port: int | None = None, base_path: str = '/', env: Mapping[str, str] | None = None, supports_pac: bool = True, **kwargs: Any)

cleanup() → None

Browser-specific cleanup that is run after the testrun is finished

create_output_handler(cmd: List[str]) → OutputHandler

Return an instance of the class used to handle application output.

This can be overridden by subclasses which have particular requirements for parsing, or otherwise using, the output.

executor_browser() → Tuple[Type[ExecutorBrowser], Mapping[str, Any]]

Returns the ExecutorBrowser subclass for this Browser subclass and the keyword arguments with which it should be instantiated

is_alive() → bool

Boolean indicating whether the browser process is still running

make_command() → List[str]

Returns the full command for starting the server process as a list.

property pid: int | None

pid of the browser process or None if there is no pid

settings(test: Test) → Mapping[str, Any]

Dictionary of metadata that is constant for a specific launch of a browser.

This is used to determine when the browser instance configuration changes, requiring a relaunch of the browser. The test runner calls this method for each test, and if the returned value differs from that for the previous test, the browser is relaunched.

start(group_metadata: Mapping[str, Any], **kwargs: Any) → None

Launch the browser object and get it into a state where is is ready to run tests

stop(force: bool = False) → bool

Stop the running browser process.

Return True iff the browser was successfully stopped.

wptrunner.browsers.base.certificate_domain_list(list_of_domains: List[str], certificate_file: str) → List[Mapping[str, Any]]

Build a list of domains where certificate_file should be used

wptrunner.browsers.base.get_free_port() → int

Get a random unbound port

class wptrunner.environment.ProxyLoggingContext(logger)

Context manager object that handles setup and teardown of a log queue for handling logging messages from wptserve.

class wptrunner.environment.TestEnvironment(test_paths, testharness_timeout_multipler, pause_after_test, debug_test, debug_info, options, ssl_config, env_extras, enable_webtransport=False, mojojs_path=None, inject_script=None, suppress_handler_traceback=None)

Context manager that owns the test environment i.e. the http and websockets servers

exception wptrunner.environment.TestEnvironmentError

class wptrunner.environment.TestdriverLoader

A special static handler for serving /resources/testdriver.js.

This handler lazily reads testdriver{,-extra}.js so that wptrunner doesn’t need to pass the entire file contents to child wptserve processes, which can slow wptserve startup by several seconds (crbug.com/1479850).

wptrunner.environment.wait_for_service(logger: StructuredLogger, host: str, port: int, timeout: float = 60, server_process: ProcessHandler | None = None) → bool

Waits until network service given as a tuple of (host, port) becomes available, timeout duration is reached, or the server_process exits at which point socket.error is raised.

class wptrunner.executors.base.CallbackHandler(logger, protocol, test_window)

Handle callbacks from testdriver-using tests.

The default implementation here makes sense for things that are roughly like WebDriver. Things that are more different to WebDriver may need to create a fully custom implementation.

class wptrunner.executors.base.CrashtestExecutor(logger, browser, server_config, timeout_multiplier=1, debug_info=None, subsuite=None, **kwargs)

exception wptrunner.executors.base.ExecutorException(status, message)

class wptrunner.executors.base.PrintRefTestExecutor(logger, browser, server_config, timeout_multiplier=1, debug_info=None, subsuite=None, **kwargs)

class wptrunner.executors.base.RefTestExecutor(logger, browser, server_config, timeout_multiplier=1, screenshot_cache=None, debug_info=None, reftest_screenshot='unexpected', **kwargs)

class wptrunner.executors.base.TestExecutor(logger, browser, server_config, timeout_multiplier=1, debug_info=None, subsuite=None, **kwargs)

Abstract Base class for object that actually executes the tests in a specific browser. Typically there will be a different TestExecutor subclass for each test type and method of executing tests.

Parameters:

• browser – ExecutorBrowser instance providing properties of the browser that will be tested.

• server_config – Dictionary of wptserve server configuration of the form stored in TestEnvironment.config

• timeout_multiplier – Multiplier relative to base timeout to use when setting test timeout.

abstract do_test(test)

Test-type and protocol specific implementation of running a specific test.

Parameters:

test – The test to run.

reset()

Re-initialize internal state to facilitate repeated test execution as implemented by the –rerun command-line argument.

run_test(test)

Run a particular test.

Parameters:

test – The test to run

setup(runner)

Run steps needed before tests can be started e.g. connecting to browser instance

Parameters:

runner – TestRunner instance that is going to run the tests

teardown()

Run cleanup steps after tests have finished

class wptrunner.executors.base.TestharnessExecutor(logger, browser, server_config, timeout_multiplier=1, debug_info=None, subsuite=None, **kwargs)

class wptrunner.executors.base.WdspecExecutor(logger, browser, server_config, webdriver_binary, webdriver_args, timeout_multiplier=1, capabilities=None, debug_info=None, binary=None, binary_args=None, **kwargs)

do_test(test)

Test-type and protocol specific implementation of running a specific test.

Parameters:

test – The test to run.

protocol_cls

alias of WdspecProtocol

setup(runner)

Run steps needed before tests can be started e.g. connecting to browser instance

Parameters:

runner – TestRunner instance that is going to run the tests

wptrunner.executors.base.get_pages(ranges_value, total_pages)

Get a set of page numbers to include in a print reftest.

Parameters:

• ranges_value – Parsed page ranges as a list e.g. [[1,2], [4], [6,None]]

• total_pages – Integer total number of pages in the paginated output.

Retval:

Set containing integer page numbers to include in the comparison e.g. for the example ranges value and 10 total pages this would be {1,2,4,6,7,8,9,10}

wptrunner.executors.base.hash_screenshots(screenshots)

Computes the sha1 checksum of a list of base64-encoded screenshots.

wptrunner.executors.base.strip_server(url)

Remove the scheme and netloc from a url, leaving only the path and any query or fragment.

url - the url to strip

e.g. http://example.org:8000/tests?id=1#2 becomes /tests?id=1#2

class wptrunner.wptrunner.TestStatus

Class that stores information on the results of test runs for later reference

wptrunner.wptrunner.evaluate_runs(test_status, **kwargs)

Evaluates the test counts after the given number of repeat runs has finished

wptrunner.wptrunner.logger = None

Runner for web-platform-tests

The runner has several design goals:

• Tests should run with no modification from upstream.

• Tests should be regarded as “untrusted” so that errors, timeouts and even crashes in the tests can be handled without failing the entire test run.

• For performance tests can be run in multiple browsers in parallel.

The upstream repository has the facility for creating a test manifest in JSON format. This manifest is used directly to determine which tests exist. Local metadata files are used to store the expected test results.

wptrunner.wptrunner.main()

Main entry point when calling from the command line

wptrunner.wptrunner.run_test_iteration(test_status, test_loader, test_queue_builder, recording, test_environment, product, kwargs)

Runs the entire test suite. This is called for each repeat run requested.

wptrunner.wptrunner.run_tests(config, product, test_paths, **kwargs)

Set up the test environment, load the list of tests to be executed, and invoke the remainder of the code to execute tests

class wptrunner.testloader.DirectoryHashChunker(total_chunks: int, chunk_number: int, **kwargs: Any)

Like HashChunker except the directory is hashed.

This ensures that all tests in the same directory end up in the same chunk.

class wptrunner.testloader.HashChunker(total_chunks: int, chunk_number: int, **kwargs: Any)

class wptrunner.testloader.IDHashChunker(total_chunks: int, chunk_number: int, **kwargs: Any)

class wptrunner.testloader.PathHashChunker(total_chunks: int, chunk_number: int, **kwargs: Any)

class wptrunner.testloader.TestChunker(total_chunks: int, chunk_number: int, **kwargs: Any)

class wptrunner.testloader.TestFilter(test_manifests, include=None, exclude=None, manifest_path=None, explicit=False)

Callable that restricts the set of tests in a given manifest according to initial criteria

class wptrunner.testloader.TestGroup(group, subsuite, test_type, metadata)

group

Alias for field number 0

metadata

Alias for field number 3

subsuite

Alias for field number 1

test_type

Alias for field number 2

class wptrunner.testloader.TestLoader(test_manifests, test_types, base_run_info, subsuites=None, manifest_filters=None, test_filters=None, chunk_type='none', total_chunks=1, chunk_number=1, include_https=True, include_h2=True, include_webtransport_h3=False, skip_timeout=False, skip_crash=False, skip_implementation_status=None, chunker_kwargs=None)

Loads tests according to a WPT manifest and any associated expectation files

class wptrunner.testloader.Unchunked(*args, **kwargs)

class wptrunner.testrunner.ManagerGroup(suite_name, test_queue_builder, test_implementations, retry_index=0, rerun=1, pause_after_test=False, pause_on_unexpected=False, restart_on_unexpected=True, debug_info=None, capture_stdio=True, restart_on_new_group=True, recording=None, max_restarts=5)

Main thread object that owns all the TestRunnerManager threads.

run(tests)

Start all managers in the group

stop()

Set the stop flag so that all managers in the group stop as soon as possible

wait(timeout: float | None = None) → None

Wait for all the managers in the group to finish.

Parameters:

timeout – Overall timeout (in seconds) for all threads to join. The default value indicates an indefinite timeout.

class wptrunner.testrunner.StopFlag(size: int)

Synchronization for coordinating a graceful exit.

class wptrunner.testrunner.TestImplementation(executor_cls, executor_kwargs, browser_cls, browser_kwargs)

browser_cls

Alias for field number 2

browser_kwargs

Alias for field number 3

executor_cls

Alias for field number 0

executor_kwargs

Alias for field number 1

class wptrunner.testrunner.TestRunner(logger, command_queue, result_queue, executor, recording)

Class implementing the main loop for running tests.

This class delegates the job of actually running a test to the executor that is passed in.

Parameters:

• logger – Structured logger

• command_queue – multiprocessing.Queue used to send commands to the process

• result_queue – multiprocessing.Queue used to send results to the parent TestRunnerManager process

• executor – TestExecutor object that will actually run a test.

run()

Main loop accepting commands over the pipe and triggering the associated methods

class wptrunner.testrunner.TestRunnerManager(suite_name, index, test_queue, test_implementations, stop_flag, retry_index=0, rerun=1, pause_after_test=False, pause_on_unexpected=False, restart_on_unexpected=True, debug_info=None, capture_stdio=True, restart_on_new_group=True, recording=None, max_restarts=5)

inject_message(command, *args)

Inject a message to the command queue (from Executor).

restart_runner()

Stop and restart the TestRunner

run_loop(test_queue)

Main loop for the TestRunnerManager.

TestRunnerManagers generally receive commands from their TestRunner updating them on the status of a test. They may also have a stop flag set by the main thread indicating that the manager should shut down the next time the event loop spins.

send_message(command, *args)

Send a message to the remote queue (to Executor).

stop_runner(force=False)

Stop the TestRunner and the browser binary.

test_ended(test, results)

Handle the end of a test.

Output the result of each subtest, and the result of the overall harness to the logs.

wptrunner.testrunner.start_runner(runner_command_queue, runner_result_queue, executor_cls, executor_kwargs, executor_browser_cls, executor_browser_kwargs, capture_stdio, stop_flag, recording)

Launch a TestRunner in a new process