TestDevelopmentMethodologies

From W3C Wiki

qa.png

To develop a test suite, you can either wait for the specification to be stabilized and create test cases out of the stable spec (the so called waterfall model) or develop test cases at the same time as you develop your spec, by creating test case each time a new feature is added or a new issue is raised (known as test driven development).

The two models are not mutually exclusive, since you can start your test development with one method, and continue it with the other. (in that case, it probably makes more sense to start with a test driven approach, and then move to a waterfall model once the spec becomes stable).

More generally, it is a good idea, even in the case of the Waterfall model, to put more efforts in the trickiest parts of the specification, or the parts where the interoperability is the most difficult. This sometimes conflicts with expanding the coverage of the test suite (TestCoverageInformation), since putting resources on a specific feature of the spec may remove them from working on non-yet covered features. Points to take into account when trying to define priorities:

  • features where non-interoperability or non-conformance would have the biggest consequences,
  • features that are known to be non-interoperable in current implementations
  • features that have taken the Working Group a long time to decide on or to define

Waterfall model

Pros

  • allows to design a test suite with a global vision of the technology
  • allows to define a plan for covering as many requirements from the spec as possible (see TestCoverageInformation)

Cons

  • takes more time
  • does not necessarily pinpoint the most important issues raised by a spec

Test Driven Development

Pros

  • allows to create the test suite at the same time as the specification itself
  • allows (somewhat) to avoid features creep (since adding a feature requires creating a test case)
  • allows a better understanding of the implication of adding a new feature, or of the resolution of an issue

Cons

  • the specification coverage is very heterogeneous
  • may leave in the shadow difficulties that were never raised as issues