testharness.js tutorial¶

Let’s say you’ve discovered that WPT doesn’t have any tests for how the Fetch API sets cookies from an HTTP response. This tutorial will guide you through the process of writing a test for the web-platform, verifying it, and submitting it back to WPT. Although it includes some very brief instructions on using git, you can find more guidance in the tutorial for git and GitHub.

WPT’s testharness.js is a framework designed to help people write tests for the web platform’s JavaScript APIs. The testharness.js reference page describes the framework in the abstract, but for the purposes of this guide, we’ll only consider the features we need to test the behavior of fetch.

Setting up your workspace¶

To make sure you have the latest code, first type the following into a terminal located in the root of the WPT git repository:

$ git fetch git@github.com:web-platform-tests/wpt.git

Next, we need a place to store the change set we’re about to author. Here’s how to create a new git branch named fetch-cookie from the revision of WPT we just downloaded:

$ git checkout -b fetch-cookie FETCH_HEAD

The tests we’re going to write will rely on special abilities of the WPT server, so you’ll also need to configure your system to run WPT before you continue.

With that out of the way, you’re ready to create your patch.

Writing a subtest¶

The first thing we’ll do is configure the server to respond to a certain request by setting a cookie. Once that’s done, we’ll be able to make the request with fetch and verify that it interpreted the response correctly.

We’ll configure the server with an “asis” file. That’s the WPT convention for controlling the contents of an HTTP response. You can read more about it here, but for now, we’ll save the following text into a file named set-cookie.asis in the fetch/api/basic/ directory of WPT:

HTTP/1.1 204 No Content
Set-Cookie: test1=t1

With this in place, any requests to /fetch/api/basic/set-cookie.asis will receive an HTTP 204 response that sets the cookie named test1. When writing more tests in the future, you may want the server to behave more dynamically. In that case, you can write Python code to control how the server responds.

Now, we can write the test! Create a new file named set-cookie.html in the same directory and insert the following text:

<!DOCTYPE html>
<meta charset="utf-8">
<title>fetch: setting cookies</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>

<script>
promise_test(function() {
  return fetch('set-cookie.asis')
    .then(function() {
        assert_equals(document.cookie, 'test1=t1');
      });
});
</script>

Let’s step through each part of this file.

```
<!DOCTYPE html>
<meta charset="utf-8">
```
We explicitly set the DOCTYPE and character set to be sure that browsers don’t infer them to be something we aren’t expecting. We’re omitting the <html> and <head> tags. That’s a common practice in WPT, preferred because it makes tests more concise.
```
<title>fetch: setting cookies</title>
```
The document’s title should succinctly describe the feature under test.
```
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
```
These two <script> tags retrieve the code that powers testharness.js. A testharness.js test can’t run without them!
```
<script>
promise_test(function() {
  return fetch('thing.asis')
    .then(function() {
        assert_equals(document.cookie, 'test1=t1');
      });
});
</script>
```
This script uses the testharness.js function promise_test to define a “subtest”. We’re using that because the behavior we’re testing is asynchronous. By returning a Promise value, we tell the harness to wait until that Promise settles. The harness will report that the test has passed if the Promise is fulfilled, and it will report that the test has failed if the Promise is rejected.

We invoke the global fetch function to exercise the “behavior under test,” and in the fulfillment handler, we verify that the expected cookie is set. We’re using the testharness.js assert_equals function to verify that the value is correct; the function will throw an error otherwise. That will cause the Promise to be rejected, and that will cause the harness to report a failure.

If you run the server according to the instructions in the guide for local configuration, you can access the test at http://web-platform.test:8000/fetch/api/basic/set-cookie.html. You should see something like this:

../_images/testharness-tutorial-test-screenshot-1.png

Refining the subtest¶

We’d like to test a little more about fetch and cookies, but before we do, there are some improvements we can make to what we’ve written so far.

For instance, we should remove the cookie after the subtest is complete. This ensures a consistent state for any additional subtests we may add and also for any tests that follow. We’ll use the add_cleanup method to ensure that the cookie is deleted even if the test fails.

-promise_test(function() {
+promise_test(function(t) {
+  t.add_cleanup(function() {
+    document.cookie = 'test1=;expires=Thu, 01 Jan 1970 00:00:01 GMT;';
+  });
+
   return fetch('thing.asis')
     .then(function() {
         assert_equals(document.cookie, 'test1=t1');
       });
 });

Although we’d prefer it if there were no other cookies defined during our test, we shouldn’t take that for granted. As written, the test will fail if the document.cookie includes additional cookies. We’ll use slightly more complicated logic to test for the presence of the expected cookie.

 promise_test(function(t) {
   t.add_cleanup(function() {
     document.cookie = 'test1=;expires=Thu, 01 Jan 1970 00:00:01 GMT;';
   });

   return fetch('thing.asis')
     .then(function() {
-        assert_equals(document.cookie, 'test1=t1');
+        assert_true(/(^|; )test1=t1($|;)/.test(document.cookie);
       });
 });

In the screen shot above, the subtest’s result was reported using the document’s title, “fetch: setting cookies”. Since we expect to add another subtest, we should give this one a more specific name:

 promise_test(function(t) {
   t.add_cleanup(function() {
     document.cookie = 'test1=;expires=Thu, 01 Jan 1970 00:00:01 GMT;';
   });

   return fetch('thing.asis')
     .then(function() {
         assert_true(/(^|; )test1=t1($|;)/.test(document.cookie));
       });
-});
+}, 'cookie set for successful request');

Writing a second subtest¶

There are many things we might want to verify about how fetch sets cookies. For instance, it should not set a cookie if the request fails due to cross-origin security restrictions. Let’s write a subtest which verifies that.

We’ll add another <script> tag for a JavaScript support file:

 <!DOCTYPE html>
 <meta charset="utf-8">
 <title>fetch: setting cookies</title>
 <script src="/resources/testharness.js"></script>
 <script src="/resources/testharnessreport.js"></script>
+<script src="/common/get-host-info.sub.js"></script>

get-host-info.sub.js is a general-purpose script provided by WPT. It’s designed to help with testing cross-domain functionality. Since it’s stored in WPT’s common/ directory, tests from all sorts of specifications rely on it.

Next, we’ll define the new subtest inside the same <script> tag that holds our first subtest.

promise_test(function(t) {
  t.add_cleanup(function() {
    document.cookie = 'test1=;expires=Thu, 01 Jan 1970 00:00:01 GMT;';
  });
  const url = get_host_info().HTTP_NOTSAMESITE_ORIGIN +
    '/fetch/api/basic/set-cookie.asis';

  return fetch(url)
    .then(function() {
        assert_unreached('The promise for the aborted fetch operation should reject.');
      }, function() {
        assert_false(/(^|; )test1=t1($|;)/.test(document.cookie));
      });
}, 'no cookie is set for cross-domain fetch operations');

This may look familiar from the previous subtest, but there are some important differences.

```
const url = get_host_info().HTTP_NOTSAMESITE_ORIGIN +
  '/fetch/api/basic/set-cookie.asis';
```
We’re requesting the same resource, but we’re referring to it with an alternate host name. The name of the host depends on how the WPT server has been configured, so we rely on the helper to provide an appropriate value.
```
return fetch(url)
  .then(function() {
      assert_unreached('The promise for the aborted fetch operation should reject.');
    }, function() {
      assert_false(/(^|; )test1=t1($|;)/.test(document.cookie));
    });
```
We’re returning a Promise value, just like the first subtest. This time, we expect the operation to fail, so the Promise should be rejected. To express this, we’ve used assert_unreached in the fulfillment handler. assert_unreached is a testharness.js utility function which always throws an error. With this in place, if fetch does not produce an error, then this subtest will fail.

We’ve moved the assertion about the cookie to the rejection handler. We also switched from assert_true to assert_false because the test should only pass if the cookie is not set. It’s a good thing we have the cleanup logic in the previous subtest, right?

If you run the test in your browser now, you can expect to see both tests reported as passing with their distinct names.

../_images/testharness-tutorial-test-screenshot-2.png

Verifying our work¶

We’re done writing the test, but we should make sure it fits in with the rest of WPT before we submit it.

The lint tool can detect some of the common mistakes people make when contributing to WPT. You enabled it when you configured your system to work with WPT. To run it, open a command-line terminal, navigate to the root of the WPT repository, and enter the following command:

python ./wpt lint fetch/api/basic

If this recognizes any of those common mistakes in the new files, it will tell you where they are and how to fix them. If you do have changes to make, you can run the command again to make sure you got them right.

Now, we’ll run the test using the automated test runner. This is important for testharness.js tests because there are subtleties of the automated test runner which can influence how the test behaves. That’s not to say your test has to pass in all browsers (or even in any browser). But if we expect the test to pass, then running it this way will help us catch other kinds of mistakes.

The tools support running the tests in many different browsers. We’ll use Firefox this time:

python ./wpt run firefox fetch/api/basic/set-cookie.html

We expect this test to pass, so if it does, we’re ready to submit it. If we were testing a web-platform feature that Firefox didn’t support, we would expect the test to fail instead.

There are a few problems to look out for in addition to passing/failing status. The report will describe fewer tests than we expect if the test isn’t run at all. That’s usually a sign of a formatting mistake, so you’ll want to make sure you’ve used the right file names and metadata. Separately, the web browser might crash. That’s often a sign of a browser bug, so you should consider reporting it to the browser’s maintainers!

Submitting the test¶

First, let’s stage the new files for committing:

$ git add fetch/api/basic/set-cookie.asis
$ git add fetch/api/basic/set-cookie.html

We can make sure the commit has everything we want to submit (and nothing we don’t) by using git diff:

$ git diff --staged

On most systems, you can use the arrow keys to navigate through the changes, and you can press the q key when you’re done reviewing.

Next, we’ll create a commit with the staged changes:

$ git commit -m '[fetch] Add test for setting cookies'

And now we can push the commit to our fork of WPT:

$ git push origin fetch-cookie

The last step is to submit the test for review. WPT doesn’t actually need the test we wrote in this tutorial, but if we wanted to submit it for inclusion in the repository, we would create a pull request on GitHub. The guide on git and GitHub has all the details on how to do that.

More practice¶

Here are some ways you can keep experimenting with WPT using this test:

Improve the test’s readability by defining helper functions like cookieIsSet and deleteCookie
Improve the test’s coverage by refactoring it into a “multi-global” test
Improve the test’s coverage by writing more subtests (e.g. the behavior when the fetch operation is aborted by window.stop, or the behavior when the HTTP response sets multiple cookies)