Anatomy of a Test Assertion:

A test assertion (or TA) must at the very minimum:

  • Refer to the specification requirement(s) that it addresses.
  • Characterize theitem under test (or IUT) - the target of the testsupposed to conform to the specification requirement(s).
  • Be uniquely identified by a test assertion identifier (TA id).

In addition, the TA must contain either one or both of the following:

  • Anassertionprose – high-level definition of the test to be performed on the item.
  • Anassertion flow, detailed later.

The assertion prose:

This is a piece of narrativedescribes (a) the operation(s) to be performed on the item under test or on the test environment, (b) the effects to be observed, and how these lead to a “fail” or “pass” outcome. The keywords (RFC2119) MUST, SHOULD, MAY, etc. often found in normative statements of specifications, must not be used in the assertion prose.

Example: An API specificationrequires the following on the method: java.lang.Integer.toString(int i, int radix): “if the radix is smaller than Character.MIN_RADIX or large than Character.MAX_RADIX then the radix 10 is used instead.”

A “prose” test assertion for this requirement can be:

Test Assertion: JAPI-123

Specification requirement: <reference to the API /method description, and to above requirement>

Item under test: an API implementation (or it could be a smaller subset of this API).

Assertion prose: when invoking the Integer.toStringfunctionwith radix < Character.MIN_RADIX, an output showing an actual radix of 10 results in test success. When invoking the toStringfunctionwith radix > Character.MAX_RADIX, an output showing an actual radix of 10 results in test success.

Notes:

  • Although two tests are actually involved, a single TA is appropriate, as these tests are about the handling of a same feature (the radix) by the same function. There are good reasons for always performing these together.An item under test will either pass of fail this TA as a whole, but failure sub-cases may be distinguished in the details of the error messagesby the test case(s)that implement thisTA.
  • The assertion prose can be more specific than the one above (e.g. give a precise value to be used for the radix argument, or be more explicit about failure conditions). But it should not depend on a specific test harness (e.g. it should not tell where to find the argument values to be used – this is the role of a test case). Conversely, in many cases, it is acceptable to be less explicit than above. For example, when the test protocol associated with this type of item under test is already well defined and shared by many test assertions – as often the case with such API tests – the operations and effects associated with test assertion success/failure, may be obvious enough to simply refer to the specification statement narrative as assertion prose.
  • A test assertion ignores the keywords MUST, SHOULD, MAY because its focus is on the feature to be verifiedon the item under test. Had the last part of the above specification requirement been:“…then the radix 10 SHOULD be used instead.”, the assertion prose would not be any different. The SHOULD keyword is to be interpreted outside the TA logic, and would only affect how a “failure” outcome is to be interpreted for conformance, which is outside the scope of defining test assertions.

The assertion flow:

This is a more structured alternative to the assertion prose. It distinguishes four parts (in addition to the three mandatory items: TA id, item under test, specification reference):

  • The pre-condition(optional). This condition acts as a qualifier for the item under test (IUT): it defines which instances of the IUT are qualified for this test, or it defines in which state the test environment must be for an IUT to undergo this test. The pre-condition may be expressed as a predicate or logical expression, of Boolean outcome (true/false). If the pre-condition is false, the TA does not apply. Its outcome is: “not applicable”.
  • The test trigger. This is the first part of the actual test operation. It consists of some test action(s)or external event(s), on the IUT and/or on the test environment. If the test trigger cannot occur for some reason, the TA does not fail nor pass: it is not applicable.
  • The test effect. This is an observable behavior (action, event) or quality (expresses as a predicate) of theIUT– or the combination {IUT, test environment}, when the pre-condition is met and the test trigger occurs. The test effect must be associated with a “pass” or “fail” TA outcome. If the effect is associated with “pass”, it is said to be a positive effect. If the effect is associated with “fail”, it is said to be a negative effect. Either one or both may be defined in a TA. For example, some requirements may only be testable for failure cases (only negative effects may be observable). For any effect outside those described, the TA outcome is “undetermined”.
  • The post-condition (optional). This condition characterizes the final state of the test environment and/or IUT, after the test is complete. If for any reason, the post-condition is “false”, this invalidates whatever result was observed for the test effect. In such a case, the TA outcome is “undetermined”, as it reveals flawed or unreliable conditions under which the test effect was observed.

More examples:

Specification Requirement (req 101): “Purchase Order Receipt documents must be XML documents that are valid against the XML schema po-receipt.xsd, and that must use the approved format for product identifiers.”

The nature of the test(s) associated with this requirement is clear: a schema validation operation, and a verification of the product ID content.

NOTE 1: although we could argue that there are two different tests involved (schema validation, content format), and two requirements actually merged into one, both define the single quality of being a P.O.Receipt: it is unlikely that either one of such requirements will be used separately in the specification. It is then appropriate to consider writing a single Test Assertion for both.

The Test Assertion below – named here TA-101 –is written in a structured way (it defines an assertion flow as opposed to an assertion prose).

Test Assertion: TA-101

Specification requirement: < req 101>

Item under test: A document.

Pre-condition: The po-receipt.xsd schema is available. The rules defining product identifiers - ProductNorm1234 specification, version 2005 - are available. A document that “claims” to be a P.O. Receiptis available (this “claim”may need to be specified further in the context of the test environment,e.g. is it a response document related to a previously P.O.?, Or any XML document with a root element <PurchaseOrderReceipt>? or both?).

Test Trigger: The document is validated against the po-receipt.xsd schema, and then the format of the element <productRef> is verified against the rules in ProductNorm1234.

Test Effect: Successful schema validation – possibly with warnings - together with conforming product reference format, result in a “pass” outcome, otherwise in “fail”.

Post-condition: None.

Notes:

  • The specification requirement req 101 is vague about the product ID format. This is because the specification editor is making two assumptions: (a) the reader knows which field of the document contains the product ID, and (b) the reader knows what the “approved format” is. Both (a) and (b) may have been more precisely specified somewhere else in the specification, or even externally to the specification. But the TA must be explicit and unambiguous on this, as it must be understood by a team of test case writers that does not have enough expertise on the specification domain to infer this.

The item under test (IUT) may depend on the conformance target that is assumed for a specification requirement. For example, req 101 could have been read from the perspective of the entity (message handler, or business application) that produces Purchase Order Receipt documents. A slightly different wording might make this clearer - e.g.: (req 101b): “Purchase Order Receipt documents in messages produced by a business endpoint must be XML documents that are valid ….” The related test assertion below (TA-101b) is defined for verifying conformance of the business endpoint instead of the conformance of the document itself:

Test Assertion: TA-101b

Specification requirement: < req 101>

Item under test: A business endpoint.

Pre-condition: The po-receipt.xsd schema is available. The rules defining product identifiers - ProductNorm1234 specification, version 2005 - are available. The business endpoint produces a document in response to a Purchase Order message.

Test Trigger: The document is validated against the po-receipt.xsd schema, and then the format of the element <productRef> is verified against the rules in ProductNorm1234.

Test Effect: Successful schema validation – possibly with warnings - together with conforming product reference format, result in a “pass” outcome, otherwise in “fail”.

Post-condition: None.