wptrunner Internals¶
-
class
wptrunner.browsers.base.
Browser
(logger)¶ Abstract class serving as the basis for Browser implementations.
The Browser is used in the TestRunnerManager to start and stop the browser process, and to check the state of that process. This class also acts as a context manager, enabling it to do browser-specific setup at the start of the testrun and cleanup after the run is complete.
- Parameters
logger – Structured logger to use for output.
-
check_crash
(process, test)¶ Check if a crash occured and output any useful information to the log. Returns a boolean indicating whether a crash occured.
-
cleanup
()¶ Browser-specific cleanup that is run after the testrun is finished
-
executor_browser
()¶ Returns the ExecutorBrowser subclass for this Browser subclass and the keyword arguments with which it should be instantiated
-
abstract
is_alive
()¶ Boolean indicating whether the browser process is still running
-
abstract
pid
()¶ pid of the browser process or None if there is no pid
-
settings
(test)¶ Dictionary of metadata that is constant for a specific launch of a browser.
This is used to determine when the browser instance configuration changes, requiring a relaunch of the browser. The test runner calls this method for each test, and if the returned value differs from that for the previous test, the browser is relaunched.
-
setup
()¶ Used for browser-specific setup that happens at the start of a test run
-
abstract
start
(group_metadata, **kwargs)¶ Launch the browser object and get it into a state where is is ready to run tests
-
abstract
stop
(force=False)¶ Stop the running browser process.
-
exception
wptrunner.browsers.base.
BrowserError
¶
-
class
wptrunner.browsers.base.
ExecutorBrowser
(**kwargs)¶ View of the Browser used by the Executor object. This is needed because the Executor runs in a child process and we can’t ship Browser instances between processes on Windows.
Typically this will have a few product-specific properties set, but in some cases it may have more elaborate methods for setting up the browser from the runner process.
-
class
wptrunner.browsers.base.
NullBrowser
(logger, **kwargs)¶ -
is_alive
()¶ Boolean indicating whether the browser process is still running
-
pid
()¶ pid of the browser process or None if there is no pid
-
start
(**kwargs)¶ No-op browser to use in scenarios where the TestRunnerManager shouldn’t actually own the browser process (e.g. Servo where we start one browser per test)
-
stop
(force=False)¶ Stop the running browser process.
-
-
wptrunner.browsers.base.
certificate_domain_list
(list_of_domains, certificate_file)¶ Build a list of domains where certificate_file should be used
-
wptrunner.browsers.base.
get_free_port
()¶ Get a random unbound port
-
class
wptrunner.environment.
TestEnvironment
(test_paths, testharness_timeout_multipler, pause_after_test, debug_test, debug_info, options, ssl_config, env_extras, enable_quic=False, mojojs_path=None)¶ Context manager that owns the test environment i.e. the http and websockets servers
-
exception
wptrunner.environment.
TestEnvironmentError
¶
-
class
wptrunner.executors.base.
CallbackHandler
(logger, protocol, test_window)¶ Handle callbacks from testdriver-using tests.
The default implementation here makes sense for things that are roughly like WebDriver. Things that are more different to WebDriver may need to create a fully custom implementation.
-
class
wptrunner.executors.base.
ConnectionlessBaseProtocolPart
(parent)¶ -
execute_script
(script, asynchronous=False)¶ Execute javascript in the current Window.
- Parameters
- Returns
The result of the script execution.
-
load
(url)¶ Load a url in the current browsing context
- Parameters
url – The url to load
-
set_timeout
(timeout)¶ Set the timeout for script execution.
- Parameters
timeout – Script timeout in seconds
-
set_window
(handle)¶ Set the top level browsing context to one specified by a given handle.
- Parameters
handle – A protocol-specific handle identifying a top level browsing context.
-
wait
()¶ Wait indefinitely for the browser to close
-
window_handles
()¶ Get a list of handles to top-level browsing contexts
-
-
class
wptrunner.executors.base.
ConnectionlessProtocol
(executor, browser)¶ -
after_connect
()¶ Run any post-connection steps. This happens after the ProtocolParts are initalized so can depend on a fully-populated object.
-
connect
()¶ Make a connection to the remote browser
-
-
class
wptrunner.executors.base.
CrashtestExecutor
(logger, browser, server_config, timeout_multiplier=1, debug_info=None, **kwargs)¶
-
exception
wptrunner.executors.base.
ExecutorException
(status, message)¶
-
class
wptrunner.executors.base.
PrintRefTestExecutor
(logger, browser, server_config, timeout_multiplier=1, debug_info=None, **kwargs)¶
-
class
wptrunner.executors.base.
RefTestExecutor
(logger, browser, server_config, timeout_multiplier=1, screenshot_cache=None, debug_info=None, **kwargs)¶
-
class
wptrunner.executors.base.
TestExecutor
(logger, browser, server_config, timeout_multiplier=1, debug_info=None, **kwargs)¶ Abstract Base class for object that actually executes the tests in a specific browser. Typically there will be a different TestExecutor subclass for each test type and method of executing tests.
- Parameters
browser – ExecutorBrowser instance providing properties of the browser that will be tested.
server_config – Dictionary of wptserve server configuration of the form stored in TestEnvironment.config
timeout_multiplier – Multiplier relative to base timeout to use when setting test timeout.
-
abstract
do_test
(test)¶ Test-type and protocol specific implementation of running a specific test.
- Parameters
test – The test to run.
-
reset
()¶ Re-initialize internal state to facilitate repeated test execution as implemented by the –rerun command-line argument.
-
run_test
(test)¶ Run a particular test.
- Parameters
test – The test to run
-
setup
(runner)¶ Run steps needed before tests can be started e.g. connecting to browser instance
- Parameters
runner – TestRunner instance that is going to run the tests
-
teardown
()¶ Run cleanup steps after tests have finished
-
class
wptrunner.executors.base.
TestharnessExecutor
(logger, browser, server_config, timeout_multiplier=1, debug_info=None, **kwargs)¶
-
class
wptrunner.executors.base.
WdspecExecutor
(logger, browser, server_config, webdriver_binary, webdriver_args, timeout_multiplier=1, capabilities=None, debug_info=None, environ=None, **kwargs)¶ -
do_test
(test)¶ Test-type and protocol specific implementation of running a specific test.
- Parameters
test – The test to run.
-
-
class
wptrunner.executors.base.
WdspecProtocol
(executor, browser)¶ -
after_connect
()¶ Run any post-connection steps. This happens after the ProtocolParts are initalized so can depend on a fully-populated object.
-
connect
()¶ Connect to browser via the HTTP server.
-
is_alive
()¶ Test that the connection is still alive.
Because the remote communication happens over HTTP we need to make an explicit request to the remote. It is allowed for WebDriver spec tests to not have a WebDriver session, since this may be what is tested.
An HTTP request to an invalid path that results in a 404 is proof enough to us that the server is alive and kicking.
-
teardown
()¶ Run cleanup steps after the tests are finished.
-
-
wptrunner.executors.base.
get_pages
(ranges_value, total_pages)¶ Get a set of page numbers to include in a print reftest.
- Parameters
ranges_value – Parsed page ranges as a list e.g. [[1,2], [4], [6,None]]
total_pages – Integer total number of pages in the paginated output.
- Retval
Set containing integer page numbers to include in the comparison e.g. for the example ranges value and 10 total pages this would be {1,2,4,6,7,8,9,10}
-
wptrunner.executors.base.
hash_screenshots
(screenshots)¶ Computes the sha1 checksum of a list of base64-encoded screenshots.
-
wptrunner.executors.base.
strip_server
(url)¶ Remove the scheme and netloc from a url, leaving only the path and any query or fragment.
url - the url to strip
e.g. http://example.org:8000/tests?id=1#2 becomes /tests?id=1#2
-
wptrunner.wptrunner.
logger
= None¶ Runner for web-platform-tests
The runner has several design goals:
Tests should run with no modification from upstream.
Tests should be regarded as “untrusted” so that errors, timeouts and even crashes in the tests can be handled without failing the entire test run.
For performance tests can be run in multiple browsers in parallel.
The upstream repository has the facility for creating a test manifest in JSON format. This manifest is used directly to determine which tests exist. Local metadata files are used to store the expected test results.
-
wptrunner.wptrunner.
main
()¶ Main entry point when calling from the command line
-
wptrunner.wptrunner.
run_tests
(config, test_paths, product, **kwargs)¶ Set up the test environment, load the list of tests to be executed, and invoke the remainder of the code to execute tests
-
class
wptrunner.testloader.
DirectoryHashChunker
(total_chunks, chunk_number, **kwargs)¶ Like HashChunker except the directory is hashed.
This ensures that all tests in the same directory end up in the same chunk.
-
class
wptrunner.testloader.
TestFilter
(test_manifests, include=None, exclude=None, manifest_path=None, explicit=False)¶ Callable that restricts the set of tests in a given manifest according to initial criteria
-
class
wptrunner.testloader.
TestGroupsFile
(logger, path)¶ Mapping object representing {group name: [test ids]}
-
class
wptrunner.testloader.
TestLoader
(test_manifests, test_types, run_info, manifest_filters=None, chunk_type='none', total_chunks=1, chunk_number=1, include_https=True, include_h2=True, include_quic=False, skip_timeout=False, skip_implementation_status=None, chunker_kwargs=None)¶ Loads tests according to a WPT manifest and any associated expectation files
-
class
wptrunner.testrunner.
ManagerGroup
(suite_name, size, test_source_cls, test_source_kwargs, browser_cls, browser_kwargs, executor_cls, executor_kwargs, rerun=1, pause_after_test=False, pause_on_unexpected=False, restart_on_unexpected=True, debug_info=None, capture_stdio=True, recording=None)¶ Main thread object that owns all the TestRunnerManager threads.
-
run
(test_type, tests)¶ Start all managers in the group
-
stop
()¶ Set the stop flag so that all managers in the group stop as soon as possible
-
wait
()¶ Wait for all the managers in the group to finish
-
-
class
wptrunner.testrunner.
TestRunner
(logger, command_queue, result_queue, executor)¶ Class implementing the main loop for running tests.
This class delegates the job of actually running a test to the executor that is passed in.
- Parameters
logger – Structured logger
command_queue – subprocess.Queue used to send commands to the process
result_queue – subprocess.Queue used to send results to the parent TestRunnerManager process
executor – TestExecutor object that will actually run a test.
-
run
()¶ Main loop accepting commands over the pipe and triggering the associated methods
-
class
wptrunner.testrunner.
TestRunnerManager
(suite_name, test_queue, test_source_cls, browser_cls, browser_kwargs, executor_cls, executor_kwargs, stop_flag, rerun=1, pause_after_test=False, pause_on_unexpected=False, restart_on_unexpected=True, debug_info=None, capture_stdio=True, recording=None)¶ -
inject_message
(command, *args)¶ Inject a message to the command queue (from Executor).
-
restart_runner
()¶ Stop and restart the TestRunner
-
run
()¶ Main loop for the TestRunnerManager.
TestRunnerManagers generally receive commands from their TestRunner updating them on the status of a test. They may also have a stop flag set by the main thread indicating that the manager should shut down the next time the event loop spins.
-
send_message
(command, *args)¶ Send a message to the remote queue (to Executor).
-
stop_runner
(force=False)¶ Stop the TestRunner and the browser binary.
-
test_ended
(test, results)¶ Handle the end of a test.
Output the result of each subtest, and the result of the overall harness to the logs.
-
-
wptrunner.testrunner.
start_runner
(runner_command_queue, runner_result_queue, executor_cls, executor_kwargs, executor_browser_cls, executor_browser_kwargs, capture_stdio, stop_flag)¶ Launch a TestRunner in a new process