TestCaseMetadata

From W3C Wiki

Introduction

A test case illustrates a feature/conformance requirement of a specification, and is used as a unit to build up a test suite. It often comes with a set of metadata allowing to track its status, context, link to the specification.

A W3C WG Note “Test Metadata” has been published by the W3C QA WG and summarizing the discussion and the work done in this wiki. You are welcome to add comments and continue the work either here or on the mailing list www-qa@w3.org which has been done in this document.

Usually Test cases fit in a TestSuiteArchitecture and involve a TestDevelopmentMethodologies

Schemas for Test

Technologies-specific metadata

Examples of metadata/description schema specifics to a given technolgoy; can be useful to assess the various needs for a Test Cases Management system for example (QaTools), or to identify patterns that could lead to more generic types.

VoiceXML 2.0 Test API
an XML Schema describes the different input possibles and grammar aspects for the output
RDF Schema for test cases
an RDF Schema with RDF-specific classes (e. g. Negative Entailment Test, Positive Entailment Test), describing the type of the test cases.
OWL Ontology for test cases
an ontology describes types of test cases adapted to OWL (eg. Consistency Test )
CSS Test cases filenames
the test cases in the CSS Test suite carry their metadata in their filenames; they include additional requirements (for the implementations, the tester), the format in which is given the test case (XHTML, XML), and the kind of test [@@@ kind, type... looks like a pattern]
XML 1.1 and Namespaces in XML 1.1 Test suites
[1] each test case is categoried into on of 4 categories (e.g. validating, nonvalidating) and includes a description, an entry describing the section or fules from the XML 1.0 (2nd ed) Rec., a unique test id within a given collection, the collection from which the test originated, and if applicable, the kind of external entities a nonvalidating processor must include.
XSLT/XPath 1.0 test suite
from an OASIS Technical Committee (see the Documents link, particularly the "How to Submit Tests" document) established several bits of test case metadata, concentrating on the ability to filter a single collection based on the version of XSLT under test, errata having been implemented or not, discretionary choices granted to implementers, and other dimensions of variability. In addition, it has a set of scenarios that allow adapting the inputs and outputs expected on a case-by-case basis. The design of the test case catalog is intended to work for a range of test suites, with technology-specific lists (document names, discretionary choices, scenarios, etc.) kept separately from the main catalog specs, yet full validation of a catalog is possible through clever allocation of IDs. -- David Marston
DOM test suite
an XML schema and a DTD describes a well-formed test case for the DOM Test suite
HTML 4.01 Test Suite
the HTML 4.01 Test suites uses a well-defined HTML markup to define the test purposes and the tested assertion
UUAG/HTML Test Suite
uses a DTD to describe its test cases, and then an XSLT style sheet to turn them into HTML
The ValidatorTestSuite
is currently in the process of developing test cases, with an appropriate description language in RDF.
XML Query Test Suite
the Test Task Force of the XML Query Working Group (assisted by the XSL Working Group) issued a test suite with several thousand cases and an XML-based catalog. There is a document about submitting cases that gives a few examples of the catalog format, as well as detailed documentation of the metadata values. For deeper study, the Guidelines for Running the suite show how the catalog data guides the tester in customizing test artifacts.

QA qa.png