Abstract Procedure Description

Abstract Procedure Description:
An Improved Notation for Writing Abstract Test Suites

HARRY SANTOSO and PANDU E POLUAN

Telematics Laboratory, Dept. of Electrical Engineering
Bandung Institute of Technology
Jl. Ganesha 10, Bandung 40132
INDONESIA
Tel.: +62-816-421-1168

Abstract: - Conformance Testing is based on Abstract Test Suites (ATSs) and Protocol Implementation Conformance Statement (PICS) proformas. Standardized ATSs are usually written in the Tree and Tabular Combined Notation (TTCN). However, TTCN has some serious limitations. This paper proposes a new, improved notation over TTCN, called the Abstract Procedure Description (APD) notation. We found that this new notation does provide for an easier to understand and maintain Test Suite, a case of which was the Abstract Test Suite for the Service-Specific Connection-Oriented Protocol (SSCOP).

Key-Words: - Protocol testing, conformance testing, abstract test suite, tree and tabular combined notation, TTCN.

- 1 -

1. Introduction

Two essential factors for the success of a telecommunications technology are standardization and interoperability. This implies that implementations must be tested to find out if they meet the standards specifications. Furthermore, the implementations should be able to perform under various load conditions.

Thus, three types of testing are defined: conformance testing, interoperability testing, and performance testing. Conformance testing evaluates the conformance of an implementation to the relevant standard specification. Interoperability testing evaluates how well an implementation works with other implementation(s). Performance testing evaluates how an implementation performs under different traffic and load conditions. These tests are independent of each other, and none is a prerequisite for the others.

All these tests must be standardized as well. This is to ensure that the test results gathered by a test laboratory will be accepted as valid by other test laboratories. Standardization of tests also prevents repeated testing of the same implementation, thus saving time, labor, and money.

The methodologies of these three types of testing have been defined by international standardization bodies, such as ITU-T, ISO, IEC, and the ATM Forum. The standardizations cover process standardization, proforma[1] standardization, and procedure standardization.

ISO/IEC Standard 9646 [1], the standard for conformance testing, suggested the use of the TTCN notation for conformance testing. However, we will set out to demonstrate a new, improved notation that is simpler, easier to create, understand, and maintain.

First, we will review the general concept of conformance testing, starting from the process to the architecture. Next, we will review the older TTCN notation, and highlight some of its shortcomings. Finally, we will review the newer APD notation, and give an example of its clear advantage over TTCN.

2. Conformance Testing

Conformance testing is one of three types of testing, which aims to check whether a system implements a standard correctly. Conceptually, this test can be divided into two stages: static and dynamic testing.

Static testing, also known as capability testing, is a test that determines if an implementation being tested implements all mandatory capabilities, and if it also implements optional capabilities. Static testing is carried out by analyzing a Protocol Implementation Conformance Statement (PICS), which is a list of statements concerning the capabilities implemented by an implementation. PICS proforma, which is the “blank form” for a PICS, is always standardized, and usually an integral part of a protocol standard. E.g., the PICS proforma for the Service-Specific Connection-Oriented Protocol (SSCOP) is contained in ITU-T Rec. Q.2110 [2].

Dynamic testing, also known as behavior testing, is a test that determines whether implemented capabilities functions properly/behaves as specified in the relevant standard. Dynamic testing is carried out by executing a set of tests involving the sending of Protocol Data Units (PDUs) and/or Abstract Service Primitives (ASPs). The set of tests may be “adjusted” to truly suit the implementation’s characteristics via the Protocol Implementation Extra Information for Testing (PIXIT).

According to ISO/IEC 9646, the testing procedure in dynamic testing is carried out through test cases, which is a set of actions done to achieve a specific test objective, i.e. to test a specific aspect of the tested capability. Relevant test cases can be grouped into test groups.

The set of test cases to test a specific standard is collected into a Test Suite. Test cases themselves are a sequence of test events, which can be modularized into test steps (similar to statements modularized into subroutines in computer programming).

The process of carrying out a conformance testing is called Conformance Assessment Process.

2.1.  Conformance Assessment Process

The Conformance Assessment Process is the complete process of accomplishing a conformance testing. It can be summarized in this flowchart:

Fig. 1. Conformance Assessment Process

The process begins with static conformance review, which is the static testing described before. In this stage, the test laboratory gathers information about the scope of requirements stated as supported by the Implementation Under Test (IUT).

Based on the result of the static conformance review, and on the answers in PICS and PIXIT, in the test selection and preparation stage, a set of relevant test cases are selected and prepared for execution.

The conformance testing stage is the stage where dynamic testing is carried out. The selected test cases are executed. The observed sequence of test events and other relevant information are recorded in a conformance log.

In the analysis of report stage, the result of each executed test case is analyzed, based on the answers in PIXIT and the conformance log. This is to examine and verify the validity of the test result.

The last stage is the generation of a test report. The test report contains the results of executed tests, and the final verdict that states whether an IUT conforms (or not) with a standard specification. ITU-T recommended a template for test report in Annexes A and B of X.294 [3].

2.2.  Test Method Architecture

The test method for dynamic testing is based on a conceptual testing architecture, as shown below:

Fig. 2. Conceptual testing architecture

Test activities always involve two sets of interactions: one above the IUT, and one below. These interactions are controlled and observed from several Points of Control and Observation (PCOs). The possible locations of PCOs are shown below:

Fig. 3. Possible PCO locations

Each PCO is identified by three factors:

1.  A service boundary at which the test events are controlled and observed.

2.  The set of test events (ASPs or PDUs) that are controlled and observed at this point.

3.  Whether they are controlled and observed within the System Under Test (SUT) or in the test system.

Furthermore, four testing functions are defined:

1.  Lower Tester (LT) is the representation of the means of providing, during test execution, indirect control and observation of the lower service boundary of the IUT via the underlying service-provider. The underlying service-provider is, as illustrated in Figure 3, beneath the protocol that is the focus of testing.

2.  Upper Tester (UT) is the representation of the means of providing, during test execution, control and observation of the upper service boundary of the IUT, as defined by the chosen test method.

3.  Test Coordination Procedures (TCP) define explicitly and implicitly the rules for cooperation between the Lower Tester and the Upper Tester.

4.  Lower Tester Coordination Function (LTCF) coordinates between multiple LTs on multipoint tests.

The position of LT requires the assumption that the underlying service-provider provides error-free service, to ensure the validity of the test result.

2.3.  Test Suites

All Test Suites has a hierarchical structure, as shown below:

Fig. 4. Test Suite structure

The most important element of a Test Suite is a test case. Each test case has a specific purpose (e.g., to check if the IUT supports a certain capability, or if the IUT behave as required).

Test groups are used to gather relevant test cases.

Test events are the smallest unit of event that comprises a test case, such as: a PDU/ASP send/receive event. Several test events can be collected into a test step for modularization purposes.

According to its form, there are two kinds of Test Suite: abstract and executable. An Abstract Test Suite (ATS) is a Test Suite specification that is implementation-dependent, written in a certain high-level notation. The most common standardized notation is the Tree and Tabular Combined Notation (TTCN), as explained in section 3. An Executable Test Suite (ETS) is a realization of an ATS, i.e. a “compiled”, ready-to-run Test Suite to be executed on a certain test system.

2.4.  Test Customization

Conformance testing is built upon two elements: PICS (proforma) and ATS. Therefore, we can create a customized conformance testing for customized standards (e.g. national standards), by creating customized PICS proforma and customized ATS. Thus, if a telecommunications operator needs to specify some additional specifications for some equipment, the tests to check the conformance of the equipment can be standardized as well.

The need to customize a test leads us directly to the need of learning the notation used to compose a Test Suite.

3. Test Notations

All ATS are written in some kind of high-level notation, which is both system-independent and implementation-independent. These properties makes the ATS portable, able to run on a wide variety of test sytems.

3.1.  The Tree and Tabular Combined Notation

The TTCN is a standard notation for creating Test Suites used in conformance testing. This notation is system-independent and implementation-independent. The basic concept of this notation is the tree concept, with branches and attachments.

TTCN is very exhaustive, and covers not only the description of test cases, but also the overall structure of the Test Suite.

TTCN defines the following elements of a Test Suite:

1.  Table of Contents: Test Suite Structure.

2.  Indexes: Test Case Index, Test Step Index, Default Index.

3.  Type definitions, in simple, tabular, or ASN.1 format.

4.  Declarations of Test Suite Operations.

5.  Declarations of parameters, Test Case Selection Expressions, constants, variables, PCOs, timers, ASPs, and PDUs.

6.  Definitions of constraints.

7.  The dynamic behaviour for Test Cases, Test Steps, and Defaults.

3.1.1. Basic Concept

As previously mentioned, the basic concept of TTCN is the tree concept. TTCN statements (also called events) are executed in sequence, with the next statement further indented:

EVENT_A
EVENT_B
EVENT_C

Branches, called alternatives, denote a situation where several outcomes are possible. They are represented as statements with the same indentation level. Control loops from one alternative to the other until one of them is successful. Then control goes to the next statement in sequence following the successful statement:

ALTERNATIVE_A
EVENT_A_1
ALTERNATIVE_B
EVENT_B_1
ALTERNATIVE_C
EVENT_C_1

Oft-executed statements/branches may be collected into a test step, which can be attached to the main procedure:

EVENT_A
+TEST_STEP_1
EVENT_C / ß Attached test step

The definitions for all PDU/ASP structures and data types used in test cases is wholly contained in the ATS, in tables described using TTCN or ASN.1 notations.

The contents of the PDUs/ASPs to be transmitted/received are specified as ‘constraints’, themselves defined in other tables.

For the complete description of the TTCN, the reader is suggested to see ISO/IEC Standard 9646.

3.1.2. Example

There are two styles of presentation for a Test Suite written in TTCN: TTCN.GR (graphical) – a tabular presentation – and TTCN.MP (machine-processible) – a textual presentation. Conversions between one style to the other is standardized in ISO/IEC 9646. Of the two, TTCN.GR is the definitive representation, since a standardized ATS is written in TTCN.GR.

One example of a test case written in TTCN.GR is shown below:

Fig. 5. Example of a Test Case written in TTCN.GR

This test case is a part of the Test Suite for testing an implementation of the Service-Specific Connection-Oriented Protocol (SSCOP). It tests the behaviour of an IUT when it receives a PDU outside of the transmission ‘window’.

This test case has several visible cross-references. One example is the cross-reference in the Constraints Ref column, which is a reference to a certain constraints table. If we look at the specified table, we might find other references to other tables. This causes the definition for a test case to be very explicit.

3.1.3. Limitations

Although TTCN is an internationally standardized notation, TTCN has some shortcomings:

1. It does not conform to programming language concepts

As previously mentioned, the concept of TTCN is the tree concept, not the flowchart concept of ‘normal’ programming languages. Although it is an elegant concept, it made implementing TTCN more difficult, because this concept is not widely used. Most of us are far more familiar with the flowchart concept than with the tree concept.

2. It is difficult to create and trace

Another side effect of the tree concept is the enormous difficulty in creating a test case in TTCN. Because each successive lines need to be indented, writing a TTCN test case becomes very difficult.

Also, it is very difficult to trace a TTCN test case (e.g. when checking for correctness or comprehending a test case). You’ll need a ruler, to properly measure the levels of indentation.

3. It is too verbose

Because TTCN relies on a lot of tables, the complete presentation of a Test Suite can be over-verbose. Besides increasing the amount of paper needed to print, being over-verbose also hinders one in understanding the test cases.

4. It is rigid and inflexible

Generally, TTCN is very rigid and inflexible, in the sense that TTCN is very difficult to be developed further. Thus, TTCN becomes “an end to itself”: an over-engineered, complex, and limited notation.

TTCN’s inflexibility made modifying a test case a very difficult task. Merely changing the contents of a PDU, for instance, may involve 3 or more tables, thus making the job of maintaining a Test Suite very complex.