Out-of-band Metadata

WPT uses two kinds of YAML-formatted text files to declare nonessential information about tests: META.yml and WEB_FEATURES.yml.

META.yml

Files with this name may appear in any directory of the web-platform-tests. They may define any of the following properties:

  • spec - a link to the specification covered by the tests in the directory

  • suggested_reviewers - a list of GitHub account username belonging to people who are notified when pull requests modify files in the directory

WEB_FEATURES.yml

Files with this name may appear in any directory that includes tests. They store a mapping between the tests in the local directory and the web-features which those tests validate.

They define one property, rules, which is a list of rules that relate one or more tests in that directory to one or more web-feature. Each mapping rule includes one property. That property’s name is a file-pattern string (matching one or more tests), and that property’s value is a list of web-feature IDs.

Formal [CDDL](https://datatracker.ietf.org/doc/html/rfc8610) schema definition
MappingRules = {
  rules: [*MappingRule],
}

MappingRule = (
  ConciseMappingRule //
  ExtendedMappingRule
)

ConciseMappingRule = {
  *FilePattern => [*text]
}

ExtendedMappingRule = {
  *FilePattern => {
    ids: [*text]
  }
}

FilePattern = text .regexp "[A-Za-z0-9_*.-]+"

These files are used to generate a single JSON-formatted manifest file which relates web-feature IDs to literal test file names (rather than lists of patterns). Such a manifest is automatically generated for every commit made on WPT’s main development branch and included in a corresponding release under the name WEB_FEATURES_MANIFEST (available in a number of encodings).

File patterns

If the property name takes the string value **, this signifies that all tests in the current directory and all subdirectories (if present) belong to the corresponding web-features.

Any other value of the property name is interpreted as a file pattern. The matching set of files will be associated with the corresponding web-feature. While these “patterns” may be literal file names, they also support the “star” (or “glob”) operators. An asterisk appearing anywhere in the string (e.g. foo-*.js) is a placeholder for zero or more other characters. Patterns using a star can therefore describe multiple files in the same directory as the WEB_FEATURES.yml file.

There is no mechanism for matching specific subdirectories (only for matching all subdirectories via **). To define mappings for files in a given subdirectory, write mapping rules in a WEB_FEATURES.yml file within that subdirectory.

These rules are interpreted from top to bottom. When a test file matches a given rule, it is no longer considered when interpreting subsequent rules. For this reason, if an author intends for a test file (or set of test files) to be associated with multiple web-features, the author should write a single rule that associates that file (or set of files) with all the desired web-features.

Caveat: Pattern Matching

Pattern matching allows classifiers to be resilient to expected changes in directory contents. This often occurs when file names have predictable file names according to:

  • their intent (e.g. shape-function-valid.html and shape-function-invalid.html)

  • their membership in a sequence (e.g. float-023.xht, float-024.xht, etc.)

  • their tentative status (e.g. offset_and_page_after_dispatch.tentative.html)

While the asterisk operator can improve concision and robustness in such cases, it also makes it easy to write classifiers that will include unrelated new tests. If there is a reasonable chance that future contributions may unintentionally match a given pattern, then a more restrictive pattern (or a list of literal file names) is likely preferable.

For instance, the pattern b* might match a desired set of tests today, but it is susceptible to matching unrelated new tests added later. If a more restrictive pattern like border-* will suffice, it is generally preferable as a safer alternative.

Caveat: Cross-cutting tests

It is common for tests in web-platform-tests to validate the behavior of more than one web-feature. Rather than including such tests in multiple classifications, it is generally preferable to avoid classifying them at all. (It may be possible to refactor tests like this into multiple files that each focus on a single web-feature, but that work is not considered a high priority.)

Example 1: Mapping an entire directory to a single web-feature

For example, if the directory named fetch/ contained only tests for the fetch web-feature, then that directory might include a WEB_FEATURES.yml file whose content appears as follows:

rules:
- "**": [fetch]

Example 2: Mapping tests within a directory to many web-features

Given a directory with the following entries:

  • crashtests/

  • resources/

  • META.yml

  • WEB_FEATURES.yml

  • aspect-ratio1.html

  • aspect-ratio2.html

  • ar1.html

  • ar1-ref.html

  • ar2.html

  • ar2-ref.html

  • box-sizing-1.html

  • box-sizing-2.html

  • box-sizing-3.js

  • z-index.html

The contents of the file named WEB_FEATURES.yml might appear as follows:

rules:
# The following rule includes all test files whose name begins with "box-sizing":
- box-sizing-*: [box-sizing]

# The following rule includes the test file named "z-index.html"
- z-index.html: [z-index]

# The following rule matches all test files which have not been matched above:
- "*": [aspect-ratio]

Example 3: Mapping the same test file(s) to more than one web-feature

The following WEB_FEATURES.yml file will associate the test files bearing the foo- prefix with the web-feature grape:

rules:
- foo-*: [grape]

The following WEB_FEATURES.yml file is equivalent because the test files prefixed with foo- are matched by the first rule and ignored by the subsequent rule-processing logic:

rules:
# the following rule matches all test files:
- foo-*: [grape]

# there are no unmatched files when this rule is encountered, so the following
# rule matches zero test files:
- foo-*: [orange]

To associate the foo--prefixed test files with the web-features named grape and the web-feature named orange, one must declare both web-feature IDs in the same rule, as in the following WEB_FEATURES.yml file:

rules:
- foo-*: [grape, orange]

Example 4: Excluding test files from all web-features

If the list value reserved for web-feature IDs is an empty list, then the test files matched by the rule will not be associated with any web-feature. This is helpful when the majority of test files within a directory should be classified but a small number should not.

For example, consider a directory where the test-file named c.html should not be associated with any web-feature, but all other web-features should be associated with the background web-feature. Authors may write rules “around” the excluded test file:

rules:
- a.html: [background]
- b.html: [background]
- d.html: [background]
- e.html: [background]

…but the above file does not clearly reflect their intent (that is, “all files except c.html). Instead, authors may write explicit rules to exclude some test files and follow that with a smaller set of more expansive rules:

rules:
- c.html: []
- "*": [background]

The * pattern will not match c.html in this context because that test file has already been matched by a preceding rule.