testharness.js API¶

testharness.js provides a framework for writing testcases. It is intended to provide a convenient API for making common assertions, and to work both for testing synchronous and asynchronous DOM features in a way that promotes clear, robust, tests.

Markup¶

The test harness script can be used from HTML or SVG documents and workers.

From an HTML or SVG document, start by importing both testharness.js and testharnessreport.js scripts into the document:

<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>

Refer to the Web Workers section for details and an example on testing within a web worker.

Within each file one may define one or more tests. Each test is atomic in the sense that a single test has a single status (PASS/FAIL/TIMEOUT/NOTRUN). Within each test one may have a number of asserts. The test fails at the first failing assert, and the remainder of the test is (typically) not run.

Note: From the point of view of a test harness, each document using testharness.js is a single “test” and each js-defined Test is referred to as a “subtest”.

By default tests must be created before the load event fires. For ways to create tests after the load event, see determining when all tests are complete.

Harness Timeout¶

Execution of tests on a page is subject to a global timeout. By default this is 10s, but a test runner may set a timeout multiplier which alters the value according to the requirements of the test environment (e.g. to give a longer timeout for debug builds).

Long-running tests may opt into a longer timeout by providing a <meta> element:

<meta name="timeout" content="long">

By default this increases the timeout to 60s, again subject to the timeout multiplier.

Tests which define a large number of subtests may need to use the variant feature to break a single test document into several chunks that complete inside the timeout.

Occasionally tests may have a race between the harness timing out and a particular test failing; typically when the test waits for some event that never occurs. In this case it is possible to use Test.force_timeout() in place of assert_unreached(), to immediately fail the test but with a status of TIMEOUT. This should only be used as a last resort when it is not possible to make the test reliable in some other way.

Defining Tests¶

Synchronous Tests¶

test(func, name)¶

Create a synchronous test

Arguments:

func (TestFunction) – Test function. This is executed immediately. If it returns without error, the test status is set to PASS. If it throws an AssertionError(), or any other exception, the test status is set to FAIL (typically from an assert function).
name (String) – Test name. This must be unique in a given file and must be invariant between runs.

A trivial test for the DOM hasFeature() method (which is defined to always return true) would be:

test(function() {
  assert_true(document.implementation.hasFeature());
}, "hasFeature() with no arguments")

Asynchronous Tests¶

Testing asynchronous features is somewhat more complex since the result of a test may depend on one or more events or other callbacks. The API provided for testing these features is intended to be rather low-level but applicable to many situations.

async_test(funcOrName, name)¶

Create an asynchronous test

Arguments:

funcOrName (TestFunction|string) – Initial step function to call immediately with the test name as an argument (if any), or name of the test.
name (String) – Test name (if a test function was provided). This must be unique in a given file and must be invariant between runs.

Returns:

Test – An object representing the ongoing test.

Create a Test:

var t = async_test("DOMContentLoaded")

Code is run as part of the test by calling the step method with a function containing the test assertions:

document.addEventListener("DOMContentLoaded", function(e) {
  t.step(function() {
    assert_true(e.bubbles, "bubbles should be true");
  });
});

When all the steps are complete, the done method must be called:

t.done();

async_test can also takes a function as first argument. This function is called with the test object as both its this object and first argument. The above example can be rewritten as:

async_test(function(t) {
  document.addEventListener("DOMContentLoaded", function(e) {
    t.step(function() {
      assert_true(e.bubbles, "bubbles should be true");
    });
    t.done();
  });
}, "DOMContentLoaded");

In many cases it is convenient to run a step in response to an event or a callback. A convenient method of doing this is through the step_func method which returns a function that, when called runs a test step. For example:

document.addEventListener("DOMContentLoaded", t.step_func(function(e) {
  assert_true(e.bubbles, "bubbles should be true");
  t.done();
}));

As a further convenience, the step_func that calls done can instead use step_func_done, as follows:

document.addEventListener("DOMContentLoaded", t.step_func_done(function(e) {
  assert_true(e.bubbles, "bubbles should be true");
}));

For asynchronous callbacks that should never execute, unreached_func can be used. For example:

document.documentElement.addEventListener("DOMContentLoaded",
  t.unreached_func("DOMContentLoaded should not be fired on the document element"));

Note: the testharness.js doesn’t impose any scheduling on async tests; they run whenever the step functions are invoked. This means multiple tests in the same global can be running concurrently and must take care not to interfere with each other.

Promise Tests¶

promise_test(func, name)¶

Create a promise test.

Promise tests are tests which are represented by a promise object. If the promise is fulfilled the test passes, if it’s rejected the test fails, otherwise the test passes.

Arguments:

func (TestFunction) – Test function. This must return a promise. The test is automatically marked as complete once the promise settles.
name (String) – Test name. This must be unique in a given file and must be invariant between runs.

test_function is a function that receives a new Test as an argument. It must return a promise. The test completes when the returned promise settles. The test fails if the returned promise rejects.

E.g.:

function foo() {
  return Promise.resolve("foo");
}

promise_test(function() {
  return foo()
    .then(function(result) {
      assert_equals(result, "foo", "foo should return 'foo'");
    });
}, "Simple example");

In the example above, foo() returns a Promise that resolves with the string “foo”. The test_function passed into promise_test invokes foo and attaches a resolve reaction that verifies the returned value.

Note that in the promise chain constructed in test_function assertions don’t need to be wrapped in step or step_func calls.

It is possible to mix promise tests with callback functions using step. However this tends to produce confusing tests; it’s recommended to convert any asynchronous behaviour into part of the promise chain. For example, instead of

promise_test(t => {
  return new Promise(resolve => {
    window.addEventListener("DOMContentLoaded", t.step_func(event => {
      assert_true(event.bubbles, "bubbles should be true");
      resolve();
    }));
  });
}, "DOMContentLoaded");

Try,

promise_test(() => {
  return new Promise(resolve => {
    window.addEventListener("DOMContentLoaded", resolve);
  }).then(event => {
    assert_true(event.bubbles, "bubbles should be true");
  });
}, "DOMContentLoaded");

Note: Unlike asynchronous tests, testharness.js queues promise tests so the next test won’t start running until after the previous promise test finishes. When mixing promise-based logic and async steps, the next test may begin to execute before the returned promise has settled. Use add_cleanup to register any necessary cleanup actions such as resetting global state that need to happen consistently before the next test starts.

To test that a promise rejects with a specified exception see promise rejection.

Single Page Tests¶

Sometimes, particularly when dealing with asynchronous behaviour, having exactly one test per page is desirable, and the overhead of wrapping everything in functions for isolation becomes burdensome. For these cases testharness.js support “single page tests”.

In order for a test to be interpreted as a single page test, it should set the single_test setup option to true.

<!doctype html>
<title>Basic document.body test</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<body>
  <script>
    setup({ single_test: true });
    assert_equals(document.body, document.getElementsByTagName("body")[0])
    done()
 </script>

The test title for single page tests is always taken from document.title.

Making assertions¶

Functions for making assertions start assert_. The full list of asserts available is documented in the asserts section. The general signature is:

assert_something(actual, expected, description)

although not all assertions precisely match this pattern e.g. assert_true only takes actual and description as arguments.

The description parameter is used to present more useful error messages when a test fails.

When assertions are violated, they throw an AssertionError exception. This interrupts test execution, so subsequent statements are not evaluated. A given test can only fail due to one such violation, so if you would like to assert multiple behaviors independently, you should use multiple tests.

Note: Unless the test is a single page test, assert functions must only be called in the context of a Test.

Optional Features¶

If a test depends on a specification or specification feature that is OPTIONAL (in the RFC 2119 sense), assert_implements_optional can be used to indicate that failing the test does not mean violating a web standard. For example:

async_test((t) => {
  const video = document.createElement("video");
  assert_implements_optional(video.canPlayType("video/webm"));
  video.src = "multitrack.webm";
  // test something specific to multiple audio tracks in a WebM container
  t.done();
}, "WebM with multiple audio tracks");

A failing assert_implements_optional call is reported as a status of PRECONDITION_FAILED for the subtest. This unusual status code is a legacy leftover; see the RFC that introduced assert_implements_optional.

assert_implements_optional can also be used during test setup. For example:

setup(() => {
  assert_implements_optional("optionalfeature" in document.body,
                             "'optionalfeature' event supported");
});
async_test(() => { /* test #1 waiting for "optionalfeature" event */ });
async_test(() => { /* test #2 waiting for "optionalfeature" event */ });

A failing assert_implements_optional during setup is reported as a status of PRECONDITION_FAILED for the test, and the subtests will not run.

See also the .optional file name convention, which may be preferable if the entire test is optional.

Testing Across Globals¶

Consolidating tests from other documents¶

fetch_tests_from_window(window)¶

Aggregate tests from separate windows or iframes into the current document as if they were all part of the same test file.

The document of the second window (or iframe) should include testharness.js, but not testharnessreport.js, and use test(), async_test(), and promise_test() in the usual manner.

Arguments:

window (Window) – The window to fetch tests from.

Note: By default any markup file referencing testharness.js will be detected as a test. To avoid this, it must be put in a support directory.

The current test suite will not report completion until all fetched tests are complete, and errors in the child contexts will result in failures for the suite in the current context.

Here’s an example that uses window.open.

support/child.html:

<!DOCTYPE html>
<html>
<title>Child context test(s)</title>
<head>
  <script src="/resources/testharness.js"></script>
</head>
<body>
  <div id="log"></div>
  <script>
    test(function(t) {
      assert_true(true, "true is true");
    }, "Simple test");
  </script>
</body>
</html>

test.html:

<!DOCTYPE html>
<html>
<title>Primary test context</title>
<head>
  <script src="/resources/testharness.js"></script>
  <script src="/resources/testharnessreport.js"></script>
</head>
<body>
  <div id="log"></div>
  <script>
    var child_window = window.open("support/child.html");
    fetch_tests_from_window(child_window);
  </script>
</body>
</html>

Web Workers¶

fetch_tests_from_worker(port)¶

Get test results from a worker and include them in the current test.

Arguments:

port (Worker|SharedWorker|ServiceWorker|MessagePort) – Either a worker object or a port connected to a worker which is running tests..

Returns:

Promise – - A promise that’s resolved once all the remote tests are complete.

The testharness.js script can be used from within dedicated workers, shared workers and service workers.

Testing from a worker script is different from testing from an HTML document in several ways:

Workers have no reporting capability since they are running in the background. Hence they rely on testharness.js running in a companion client HTML document for reporting.
Shared and service workers do not have a unique client document since there could be more than one document that communicates with these workers. So a client document needs to explicitly connect to a worker and fetch test results from it using fetch_tests_from_worker. This is true even for a dedicated worker. Once connected, the individual tests running in the worker (or those that have already run to completion) will be automatically reflected in the client document.
The client document controls the timeout of the tests. All worker scripts act as if they were started with the explicit_timeout option.
Dedicated and shared workers don’t have an equivalent of an onload event. Thus the test harness has no way to know when all tests have completed (see Determining when all tests are complete). So these worker tests behave as if they were started with the explicit_done option. Service workers depend on the oninstall event and don’t require an explicit done call.

Here’s an example that uses a dedicated worker.

worker.js:

importScripts("/resources/testharness.js");

test(function(t) {
  assert_true(true, "true is true");
}, "Simple test");

// done() is needed because the testharness is running as if explicit_done
// was specified.
done();

test.html:

<!DOCTYPE html>
<title>Simple test</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<div id="log"></div>
<script>

fetch_tests_from_worker(new Worker("worker.js"));

</script>

fetch_tests_from_worker returns a promise that resolves once all the remote tests have completed. This is useful if you’re importing tests from multiple workers and want to ensure they run in series:

(async function() {
  await fetch_tests_from_worker(new Worker("worker-1.js"));
  await fetch_tests_from_worker(new Worker("worker-2.js"));
})();

Cleanup¶

Occasionally tests may create state that will persist beyond the test itself. In order to ensure that tests are independent, such state should be cleaned up once the test has a result. This can be achieved by adding cleanup callbacks to the test. Such callbacks are registered using the add_cleanup method. All registered callbacks will be run as soon as the test result is known. For example:

  test(function() {
    var element = document.createElement("div");
    element.setAttribute("id", "null");
    document.body.appendChild(element);
    this.add_cleanup(function() { document.body.removeChild(element) });
    assert_equals(document.getElementById(null), element);
  }, "Calling document.getElementById with a null argument.");

If the test was created using the promise_test API, then cleanup functions may optionally return a Promise and delay the completion of the test until all cleanup promises have settled.

All callbacks will be invoked synchronously; tests that require more complex cleanup behavior should manage execution order explicitly. If any of the eventual values are rejected, the test runner will report an error.

AbortSignal support¶

Test.get_signal gives an AbortSignal that is aborted when the test finishes. This can be useful when dealing with AbortSignal-supported APIs.

promise_test(t => {
  // Throws when the user agent does not support AbortSignal
  const signal = t.get_signal();
  const event = await new Promise(resolve => {
    document.body.addEventListener(resolve, { once: true, signal });
    document.body.click();
  });
  assert_equals(event.type, "click");
}, "");

Timers in Tests¶

In general the use of timers (i.e. setTimeout) in tests is discouraged because this is an observed source of instability on test running in CI. In particular if a test should fail when something doesn’t happen, it is good practice to simply let the test run to the full timeout rather than trying to guess an appropriate shorter timeout to use.

In other cases it may be necessary to use a timeout (e.g., for a test that only passes if some event is not fired). In this case it is not permitted to use the standard setTimeout function. Instead use either Test.step_wait(), Test.step_wait_func(), or Test.step_timeout(). Test.step_wait() and Test.step_wait_func() are preferred when there’s a specific condition that needs to be met for the test to proceed. Test.step_timeout() is preferred in other cases.

Note that timeouts generally need to be a few seconds long in order to produce stable results in all test environments.

For single page tests, step_timeout is also available as a global function.

step_timeout(func, timeout)¶

Global version of Test.step_timeout() for use in single page tests.

Arguments:

func (function) – Function to run after the timeout
timeout (number) – Time in ms to wait before running the test step. The actual wait time is timeout x timeout_multiplier.

Harness Configuration¶

Setup¶

setup(funcOrProperties, maybeProperties)¶

Configure the harness

Arguments:

funcOrProperties (function|SettingsObject) – Either a setup function to run, or a set of properties. If this is a function that function is run synchronously. Any exception in the function will set the overall harness status to ERROR.
maybeProperties (SettingsObject) – An object containing the settings to use, if the first argument is a function.

promise_setup(func, properties)¶

Configure the harness, waiting for a promise to resolve before running any promise_test tests.

Arguments:

func (function) – Function returning a promise that’s run synchronously. Promise tests are not run until after this function has resolved.
properties (SettingsObject) – An object containing the harness settings to use.

SettingsObject:

Properties:

single_test (bool) - Use the single-page-test mode. In this mode the Document represents a single Test(). Asserts may be used directly without requiring Test.step() or similar wrappers, and any exceptions set the status of the test rather than the status of the harness.
allow_uncaught_exception (bool) - don’t treat an uncaught exception as an error; needed when e.g. testing the window.onerror handler.
explicit_done (bool) - Wait for a call to done() before declaring all tests complete (this is always true for single-page tests).
hide_test_state (bool) - hide the test state output while the test is running; This is helpful when the output of the test state may interfere the test results.
explicit_timeout (bool) - disable file timeout; only stop waiting for results when the timeout() function is called. This should typically only be set for manual tests, or by a test runner that provides its own timeout mechanism.
timeout_multiplier (Number) - Multiplier to apply to timeouts. This should only be set by a test runner.
output (bool) - (default: true) Whether to output a table containing a summary of test results. This should typically only be set by a test runner, and is typically set to false for performance reasons when running in CI.
output_document (Document) output_document - The document to which results should be logged. By default this is the current document but could be an ancestor document in some cases e.g. a SVG test loaded in an HTML wrapper
debug (bool) - (default: false) Whether to output additional debugging information such as a list of asserts. This should typically only be set by a test runner.

Output¶

If the file containing the tests is a HTML file, a table containing the test results will be added to the document after all tests have run. By default this will be added to a div element with id=log if it exists, or a new div element appended to document.body if it does not. This can be suppressed by setting the output setting to false.

If output is true, the test will, by default, report progress during execution. In some cases this progress report will invalidate the test. In this case the test should set the hide_test_state setting to true.

Determining when all tests are complete¶

By default, tests running in a WindowGlobalScope, which are not configured as a single page test the test harness will assume there are no more results to come when:

There are no Test objects that have been created but not completed
The load event on the document has fired

For single page tests, or when the explicit_done property has been set in the setup, the done function must be used.

done()¶

Mark test loading as complete.

Typically this function is called implicitly on page load; it’s only necessary for users to call this when either the explicit_done or single_page properties have been set via the setup() function.

For single page tests this marks the test as complete and sets its status. For other tests, this marks test loading as complete, but doesn’t affect ongoing tests.

timeout()¶

Timeout the tests.

This only has an effect when explicit_timeout has been set in setup(). In other cases any call is a no-op.

Dedicated and shared workers don’t have an event that corresponds to the load event in a document. Therefore these worker tests always behave as if the explicit_done property is set to true (unless they are defined using the “multi-global” pattern). Service workers depend on the install event which is fired following the completion of running the worker.

Reporting API¶

Callbacks¶

The framework provides callbacks corresponding to 4 events:

start - triggered when the first Test is created
test_state - triggered when a test state changes
result - triggered when a test result is received
complete - triggered when all results are received

add_start_callback(callback)¶

Add a callback that’s triggered when the first Test() is created.

Arguments:

callback (function) – Callback function. This is called without arguments.

add_test_state_callback(callback)¶

Add a callback that’s triggered when a test state changes.

Arguments:

callback (function) – Callback function, called with the Test() as the only argument.

add_result_callback(callback)¶

Add a callback that’s triggered when a test result is received.

Arguments:

callback (function) – Callback function, called with the Test() as the only argument.

add_completion_callback(callback)¶

Add a callback that’s triggered when all tests are complete.

Arguments:

callback (function) – Callback function, called with an array of Test() objects, a TestsStatus() object and an array of AssertRecord() objects. If the debug setting is false the final argument will be an empty array. For performance reasons asserts are only tracked when the debug setting is true. In other cases the array of asserts will be empty.

class TestsStatus()¶

Status of the overall harness

TestsStatus.message¶: Message in case of failure

TestsStatus.stack¶: Stack trace in case of an exception.

TestsStatus.status¶: The status code

TestsStatus.statuses¶

Enum of possible harness statuses.

Values:

OK
ERROR
TIMEOUT
PRECONDITION_FAILED

class AssertRecord(test, assert_name, args)¶

Record of an assert that ran.

Arguments:

test (Test) – The test which ran the assert.
assert_name (string) – The function name of the assert.
args (Any) – The arguments passed to the assert function.

AssertRecord.args¶: Stringification of the arguments that were passed to the assert function

AssertRecord.assert_name¶: Name of the assert that ran

AssertRecord.status¶: Status of the assert

AssertRecord.test¶: Test that ran the assert

External API¶

In order to collect the results of multiple pages containing tests, the test harness will, when loaded in a nested browsing context, attempt to call certain functions in each ancestor and opener browsing context:

start - start_callback
test_state - test_state_callback
result - result_callback
complete - completion_callback

These are given the same arguments as the corresponding internal callbacks described above.

The test harness will also send messages using cross-document messaging to each ancestor and opener browsing context. Since it uses the wildcard keyword (*), cross-origin communication is enabled and script on different origins can collect the results.

This API follows similar conventions as those described above only slightly modified to accommodate message event API. Each message is sent by the harness is passed a single vanilla object, available as the data property of the event object. These objects are structured as follows:

start - { type: "start" }
test_state - { type: "test_state", test: Test }
result - { type: "result", test: Test }
complete - { type: "complete", tests: [Test, ...], status: TestsStatus }

Assert Functions¶

assert_true(actual, description)¶

Assert that actual is strictly true

Arguments:

actual (Any) – Value that is asserted to be true
description (string) – Description of the condition being tested

assert_false(actual, description)¶

Assert that actual is strictly false

Arguments:

actual (Any) – Value that is asserted to be false
description (string) – Description of the condition being tested

assert_equals(actual, expected, description)¶

Assert that actual is the same value as expected.

For objects this compares by object identity; for primitives this distinguishes between 0 and -0, and has correct handling of NaN.

Arguments:

actual (Any) – Test value.
expected (Any) – Expected value.
description (string) – Description of the condition being tested.

assert_not_equals(actual, expected, description)¶

Assert that actual is not the same value as expected.

Comparison is as for assert_equals().

Arguments:

actual (Any) – Test value.
expected (Any) – The value actual is expected to be different to.
description (string) – Description of the condition being tested.

assert_in_array(actual, expected, description)¶

Assert that expected is an array and actual is one of the members. This is implemented using indexOf, so doesn’t handle NaN or ±0 correctly.

Arguments:

actual (Any) – Test value.
expected (Array) – An array that actual is expected to be a member of.
description (string) – Description of the condition being tested.

assert_array_equals(actual, expected, description)¶

Assert that actual and expected are both arrays, and that the array properties of actual and expected are all the same value (as for assert_equals()).

Arguments:

actual (Array) – Test array.
expected (Array) – Array that is expected to contain the same values as actual.
description (string) – Description of the condition being tested.

assert_approx_equals(actual, expected, epsilon, description)¶

Assert that actual is within ± epsilon of expected.

Arguments:

actual (number) – Test value.
expected (number) – Value number is expected to be close to.
epsilon (number) – Magnitude of allowed difference between actual and expected.
description (string) – Description of the condition being tested.

assert_array_approx_equals(actual, expected, epsilon, description)¶

Assert that each array property in actual is a number within ± epsilon of the corresponding property in expected.

Arguments:

actual (Array) – Array of test values.
expected (Array) – Array of values expected to be close to the values in actual.
epsilon (number) – Magnitude of allowed difference between each value in actual and expected.
description (string) – Description of the condition being tested.

assert_less_than(actual, expected, description)¶

Assert that actual is a number less than expected.

Arguments:

actual (number) – Test value.
expected (number) – Number that actual must be less than.
description (string) – Description of the condition being tested.

assert_greater_than(actual, expected, description)¶

Assert that actual is a number greater than expected.

Arguments:

actual (number) – Test value.
expected (number) – Number that actual must be greater than.
description (string) – Description of the condition being tested.

assert_between_exclusive(actual, lower, upper, description)¶

Assert that actual is a number greater than lower and less than upper but not equal to either.

Arguments:

actual (number) – Test value.
lower (number) – Number that actual must be greater than.
upper (number) – Number that actual must be less than.
description (string) – Description of the condition being tested.

assert_less_than_equal(actual, expected, description)¶

Assert that actual is a number less than or equal to expected.

Arguments:

actual (number) – Test value.
expected (number) – Number that actual must be less than or equal to.
description (string) – Description of the condition being tested.

assert_greater_than_equal(actual, expected, description)¶

Assert that actual is a number greater than or equal to expected.

Arguments:

actual (number) – Test value.
expected (number) – Number that actual must be greater than or equal to.
description (string) – Description of the condition being tested.

assert_between_inclusive(actual, lower, upper, description)¶

Assert that actual is a number greater than or equal to lower and less than or equal to upper.

Arguments:

actual (number) – Test value.
lower (number) – Number that actual must be greater than or equal to.
upper (number) – Number that actual must be less than or equal to.
description (string) – Description of the condition being tested.

assert_regexp_match(actual, expected, description)¶

Assert that actual matches the RegExp expected.

Arguments:

actual (String) – Test string.
expected (RegExp) – RegExp actual must match.
description (string) – Description of the condition being tested.

assert_class_string(object, class_string, description)¶

Assert that the class string of object as returned in Object.prototype.toString is equal to class_name.

Arguments:

object (Object) – Object to stringify.
class_string (string) – Expected class string for object.
description (string) – Description of the condition being tested.

assert_own_property(object, property_name, description)¶

Assert that object has an own property with name property_name.

Arguments:

object (Object) – Object that should have the given property.
property_name (string) – Expected property name.
description (string) – Description of the condition being tested.

assert_not_own_property(object, property_name, description)¶

Assert that object does not have an own property with name property_name.

Arguments:

object (Object) – Object that should not have the given property.
property_name (string) – Property name to test.
description (string) – Description of the condition being tested.

assert_inherits(object, property_name, description)¶

Assert that object does not have an own property with name property_name, but inherits one through the prototype chain.

Arguments:

object (Object) – Object that should have the given property in its prototype chain.
property_name (string) – Expected property name.
description (string) – Description of the condition being tested.

assert_idl_attribute(object, property_name, description)¶

Alias for insert_inherits().

Arguments:

object (Object) – Object that should have the given property in its prototype chain.
property_name (string) – Expected property name.
description (string) – Description of the condition being tested.

assert_readonly(object, property_name, description)¶

Assert that object has a property named property_name and that the property is not writable or has no setter.

Arguments:

object (Object) – Object that should have the given (not necessarily own) property.
property_name (string) – Expected property name.
description (string) – Description of the condition being tested.

assert_throws_dom(type, descriptionOrFunc, description)¶

Assert a DOMException with the expected type is thrown.

There are two ways of calling assert_throws_dom:

1) If the DOMException is expected to come from the current global, the second argument should be the function expected to throw and a third, optional, argument is the assertion description.

2) If the DOMException is expected to come from some other global, the second argument should be the DOMException constructor from that global, the third argument the function expected to throw, and the fourth, optional, argument the assertion description.

Arguments:

type (number|string) – The expected exception name or code. See the table of names and codes. If a number is passed it should be one of the numeric code values in that table (e.g. 3, 4, etc). If a string is passed it can either be an exception name (e.g. “HierarchyRequestError”, “WrongDocumentError”) or the name of the corresponding error code (e.g. “HIERARCHY_REQUEST_ERR”, “WRONG_DOCUMENT_ERR”).
descriptionOrFunc (function) – The function expected to throw (if the exception comes from another global), or the optional description of the condition being tested (if the exception comes from the current global).
description (string) – Description of the condition being tested (if the exception comes from another global).

assert_throws_js(constructor, func, description)¶

Assert a JS Error with the expected constructor is thrown.

Arguments:

constructor (object) – The expected exception constructor.
func (function) – Function which should throw.
description (string) – Error description for the case that the error is not thrown.

assert_throws_exactly(exception, func, description)¶

Assert the provided value is thrown.

Arguments:

exception (value) – The expected exception.
func (function) – Function which should throw.
description (string) – Error description for the case that the error is not thrown.

assert_implements(condition, description)¶

Assert that a feature is implemented, based on a ‘truthy’ condition.

This function should be used to early-exit from tests in which there is no point continuing without support for a non-optional spec or spec feature. For example:

assert_implements(window.Foo, ‘Foo is not supported’);

Arguments:

condition (object) – The truthy value to test
description (string) – Error description for the case that the condition is not truthy.

assert_implements_optional(condition, description)¶

Assert that an optional feature is implemented, based on a ‘truthy’ condition.

This function should be used to early-exit from tests in which there is no point continuing without support for an explicitly optional spec or spec feature. For example:

assert_implements_optional(video.canPlayType(“video/webm”),
“webm video playback not supported”);

Arguments:

condition (object) – The truthy value to test
description (string) – Error description for the case that the condition is not truthy.

assert_unreached(description)¶

Asserts if called. Used to ensure that a specific codepath is not taken e.g. that an error event isn’t fired.

Arguments:

description (string) – Description of the condition being tested.

assert_any(assert_func, actual, expected_array, ...args)¶

Asserts that actual matches at least one value of expected according to a comparison defined by assert_func.

Note that tests with multiple allowed pass conditions are bad practice unless the spec specifically allows multiple behaviours. Test authors should not use this method simply to hide UA bugs.

Arguments:

assert_func (AssertFunc) – Function to compare actual and expected. It must throw when the comparison fails and return when the comparison passes.
actual (Any) – Test value.
expected_array (Array) – Array of possible expected values.
args (Array.<Any>) – Additional arguments to pass to assert_func.

Assertions fail by throwing an AssertionError:

class AssertionError(message)¶

Exception type that represents a failing assert.

Arguments:

message (string) – Error message.

Promise Rejection¶

promise_rejects_dom(test, type, promiseOrConstructor, descriptionOrPromise, description)¶

Assert that a Promise is rejected with the right DOMException.

For the remaining arguments, there are two ways of calling promise_rejects_dom:

1) If the DOMException is expected to come from the current global, the third argument should be the promise expected to reject, and a fourth, optional, argument is the assertion description.

2) If the DOMException is expected to come from some other global, the third argument should be the DOMException constructor from that global, the fourth argument the promise expected to reject, and the fifth, optional, argument the assertion description.

Arguments:

test (Test) – the Test to use for the assertion.
type (number|string) – See documentation for assert_throws_dom.
promiseOrConstructor (function) – Either the constructor for the expected exception (if the exception comes from another global), or the promise that’s expected to reject (if the exception comes from the current global).
descriptionOrPromise (function|string) – Either the promise that’s expected to reject (if the exception comes from another global), or the optional description of the condition being tested (if the exception comes from the current global).
description (string) – Description of the condition being tested (if the exception comes from another global).

promise_rejects_js(test, constructor, promise, description)¶

Assert that a Promise is rejected with the right ECMAScript exception.

Arguments:

test (Test) – the Test to use for the assertion.
constructor (function) – The expected exception constructor.
promise (Promise) – The promise that’s expected to reject with the given exception.
description (string) – Error message to add to assert in case of failure.

promise_rejects_exactly(test, exception, promise, description)¶

Assert that a Promise is rejected with the provided value.

Arguments:

test (Test) – the Test to use for the assertion.
exception (Any) – The expected value of the rejected promise.
promise (Promise) – The promise that’s expected to reject.
description (string) – Error message to add to assert in case of failure.

promise_rejects_dom, promise_rejects_js, and promise_rejects_exactly can be used to test Promises that need to reject.

Here’s an example where the bar() function returns a Promise that rejects with a TypeError:

function bar() {
  return Promise.reject(new TypeError());
}

promise_test(function(t) {
  return promise_rejects_js(t, TypeError, bar());
}, "Another example");

Test Objects¶

class Test(name)¶

A single subtest. A Test is not constructed directly but via the test(), async_test() or promise_test() functions.

Arguments:

name (string) – This must be unique in a given file and must be invariant between runs.

Test.message¶: A message indicating the reason for test failure.

Test.name¶: The test name.

Test.stack¶: Stack trace in case of failure.

Test.status¶: The test status code.

Test.statuses¶

Enum of possible test statuses.

Values:

PASS
FAIL
TIMEOUT
NOTRUN
PRECONDITION_FAILED

Test.add_cleanup(callback)¶

Schedule a function to be run after the test result is known, regardless of passing or failing state.

The behavior of this function will not influence the result of the test, but if an exception is thrown, the test harness will report an error.

Arguments:

callback (function) – The cleanup function to run. This is called with no arguments.

Test.done()¶

Mark the test as complete.

This sets the test status to PASS if no other status was already recorded. Any subsequent attempts to run additional test steps will be ignored.

After setting the test status any test cleanup functions will be run.

Test.force_timeout()¶

Manually set the test status to TIMEOUT.

Alias for Test.timeout.

Test.get_signal()¶: Gives an AbortSignal that will be aborted when the test finishes.

Test.step(func, this_obj)¶

Run a single step of an ongoing test.

Arguments:

func (string) – Callback function to run as a step. If this throws an AssertionError(), or any other exception, the Test() status is set to FAIL.
this_obj (Object) – The object to use as the this value when calling func. Defaults to the Test() object.

Test.step_func(func, this_obj)¶

Wrap a function so that it runs as a step of the current test.

This allows creating a callback function that will run as a test step.

Arguments:

func (string) – Function to run as a step. If this throws an AssertionError(), or any other exception, the Test() status is set to FAIL.
this_obj (Object) – The object to use as the this value when calling func. Defaults to the Test() object.

Examples:

let t = async_test("Example");
onload = t.step_func(e => {
  assert_equals(e.name, "load");
  // Mark the test as complete.
  t.done();
})

Test.step_func_done(func, this_obj)¶

Wrap a function so that it runs as a step of the current test, and automatically marks the test as complete if the function returns without error.

Arguments:

func (string) – Function to run as a step. If this throws an AssertionError(), or any other exception, the Test() status is set to FAIL. If it returns without error the status is set to PASS.
this_obj (Object) – The object to use as the this value when calling func. Defaults to the Test() object.

Test.step_timeout(func, timeout)¶

Run a function as a step of the test after a given timeout.

This multiplies the timeout by the global timeout multiplier to account for the expected execution speed of the current test environment. For example test.step_timeout(f, 2000) with a timeout multiplier of 2 will wait for 4000ms before calling f.

In general it’s encouraged to use Test.step_wait() or step_wait_func() in preference to this function where possible, as they provide better test performance.

Arguments:

func (function) – Function to run as a test step.
timeout (number) – Time in ms to wait before running the test step. The actual wait time is timeout x timeout_multiplier.

Test.step_wait(cond, description, timeout, interval)¶

Poll for a function to return true, and resolve a promise once it does, or assert if a timeout is reached. This is preferred over a simple step_timeout whenever possible since it allows the timeout to be longer to reduce intermittents without compromising test execution speed when the condition is quickly met.

Arguments:

cond (function) – A function taking no arguments and returning a boolean or a Promise.
description (string) – Error message to add to assert in case of failure.
timeout (number) – Timeout in ms. This is multiplied by the global timeout_multiplier
interval (number) – Polling interval in ms

Returns:

Promise – Promise resolved once cond is met.

Examples:

promise_test(async t => {
 // …
await t.step_wait(() => frame.contentDocument === null, "Frame navigated to a cross-origin document");
// …
}, "");

Test.step_wait_func(cond, func, description, timeout, interval)¶

Poll for a function to return true, and call a callback function once it does, or assert if a timeout is reached. This is preferred over a simple step_timeout whenever possible since it allows the timeout to be longer to reduce intermittents without compromising test execution speed when the condition is quickly met.

Arguments:

cond (function) – A function taking no arguments and returning a boolean or a Promise. The callback is called when this function returns true, or the returned Promise is resolved with true.
func (function) – A function taking no arguments to call once the condition is met.
description (string) – Error message to add to assert in case of failure.
timeout (number) – Timeout in ms. This is multiplied by the global timeout_multiplier
interval (number) – Polling interval in ms

Test.step_wait_func_done(cond, func, description, timeout, interval)¶

Poll for a function to return true, and invoke a callback followed by this.done() once it does, or assert if a timeout is reached. This is preferred over a simple step_timeout whenever possible since it allows the timeout to be longer to reduce intermittents without compromising test execution speed when the condition is quickly met.

Arguments:

cond (function) – A function taking no arguments and returning a boolean or a Promise. The callback is called when this function returns true, or the returned Promise is resolved with true.
func (function) – A function taking no arguments to call once the condition is met.
description (string) – Error message to add to assert in case of failure.
timeout (number) – Timeout in ms. This is multiplied by the global timeout_multiplier
interval (number) – Polling interval in ms

Examples:

async_test(t => {
 const popup = window.open("resources/coop-coep.py?coop=same-origin&coep=&navigate=about:blank");
 t.add_cleanup(() => popup.close());
 assert_equals(window, popup.opener);

 popup.onload = t.step_func(() => {
   assert_true(popup.location.href.endsWith("&navigate=about:blank"));
   // Use step_wait_func_done as about:blank cannot message back.
   t.step_wait_func_done(() => popup.location.href === "about:blank");
 });
}, "Navigating a popup to about:blank");

Test.timeout()¶: Manually set the test status to TIMEOUT.

Test.unreached_func(description)¶

Return a function that automatically sets the current test to FAIL if it’s called.

Arguments:

description (string) – Error message to add to assert in case of failure.

Helpers¶

Waiting for events¶

class EventWatcher(test, watchedNode, eventTypes, timeoutPromise)¶

Allow DOM events to be handled using Promises.

This can make it a lot easier to test a very specific series of events, including ensuring that unexpected events are not fired at any point.

EventWatcher will assert if an event occurs while there is no wait_for created Promise waiting to be fulfilled, or if the event is of a different type to the type currently expected. This ensures that only the events that are expected occur, in the correct order, and with the correct timing.

Arguments:

test (Test) – The Test to use for the assertion.
watchedNode (EventTarget) – The target expected to receive the events.
eventTypes (Array.<string>) – List of events to watch for.
timeoutPromise (Promise) – Promise that will cause the test to be set to TIMEOUT once fulfilled.

EventWatcher.wait_for(options)¶

Returns a Promise that will resolve after the specified event or series of events has occurred.

Arguments:

options (Object) – An optional options object. If the ‘record’ property on this object has the value ‘all’, when the Promise returned by this function is resolved, all Event objects that were waited for will be returned as an array.

Examples:

const watcher = new EventWatcher(t, div, [ 'animationstart',
                                           'animationiteration',
                                           'animationend' ]);
return watcher.wait_for([ 'animationstart', 'animationend' ],
                        { record: 'all' }).then(evts => {
  assert_equals(evts[0].elapsedTime, 0.0);
  assert_equals(evts[1].elapsedTime, 2.0);
});

EventWatcher.stop_watching()¶: Stop listening for events

Here’s an example of how to use EventWatcher:

var t = async_test("Event order on animation start");

var animation = watchedNode.getAnimations()[0];
var eventWatcher = new EventWatcher(t, watchedNode, ['animationstart',
                                                     'animationiteration',
                                                     'animationend']);

eventWatcher.wait_for('animationstart').then(t.step_func(function() {
  assertExpectedStateAtStartOfAnimation();
  animation.currentTime = END_TIME; // skip to end
  // We expect two animationiteration events then an animationend event on
  // skipping to the end of the animation.
  return eventWatcher.wait_for(['animationiteration',
                                'animationiteration',
                                'animationend']);
})).then(t.step_func(function() {
  assertExpectedStateAtEndOfAnimation();
  t.done();
}));

Loading test data from JSON files¶

fetch_json(resource)¶: Fetches a JSON resource and parses it

Loading test data from a JSON file would normally be accomplished by something like this:

promise_test(() => fetch('resources/my_data.json').then((res) => res.json()).then(runTests));
function runTests(myData) {
  /// ...
}

However, fetch() is not exposed inside ShadowRealm scopes, so if the test is to be run inside a ShadowRealm, use fetch_json() instead:

promise_test(() => fetch_json('resources/my_data.json').then(runTests));

Utility Functions¶

format_value(val)¶

Convert a value to a nice, human-readable string

When many JavaScript Object values are coerced to a String, the resulting value will be "[object Object]". This obscures helpful information, making the coerced value unsuitable for use in assertion messages, test names, and debugging statements. format_value produces more distinctive string representations of many kinds of objects, including arrays and the more important DOM Node types. It also translates String values containing control characters to include human-readable representations.

Arguments:

val (Any) – The value to convert to a string.

Returns:

string – - A string representation of val, optimised for human readability.

Examples:

// "Document node with 2 children"
format_value(document);

// "\"foo\\uffffbar\""
format_value("foo\uffffbar");

// "[-0, Infinity]"
format_value([-0, Infinity]);

Deprecated APIs¶

generate_tests(func, args)¶

Note

Deprecated: generate a list of tests from a function and list of arguments

This is deprecated because it runs all the tests outside of the test functions and as a result any test throwing an exception will result in no tests being run. In almost all cases, you should simply call test within the loop you would use to generate the parameter list array.

Arguments:

func (function) – The function that will be called for each generated tests.
args (Array.<Array.<Any>>) – An array of arrays. Each nested array has the structure [testName, …testArgs]. For each of these nested arrays array, a test is generated with name testName and test function equivalent to func(..testArgs).

on_event(object, event, callback)¶

Note

Deprecated: Register a function as a DOM event listener to the

given object for the event bubbling phase.

Arguments:

object (EventTarget) – Event target
event (string) – Event name
callback (function) – Event handler.

testharness.js API¶

Markup¶

Harness Timeout¶

Defining Tests¶

Synchronous Tests¶

Asynchronous Tests¶

Promise Tests¶

Single Page Tests¶

Making assertions¶

Optional Features¶

Testing Across Globals¶

Consolidating tests from other documents¶

Web Workers¶

Cleanup¶

AbortSignal support¶

Timers in Tests¶

Harness Configuration¶

Setup¶

Output¶

Determining when all tests are complete¶

Reporting API¶

Callbacks¶

External API¶

Assert Functions¶

Promise Rejection¶

Test Objects¶

Helpers¶

Waiting for events¶

Loading test data from JSON files¶

Utility Functions¶

Deprecated APIs¶

web-platform-tests

Navigation

Related Topics