Chapter 11: Testing

According to the authors, “Testing is the process of finding differences between the behavior specified by the system models and the observed behavior of the system.” In effect, the main purpose of testing is to break the system. During the developmental phases (such as system design, object design, etc.) our main goal was to build a system that was to satisfy the requirements and specifications of the client. Having completed building the system, we now need to break it. The idea is that if we cannot break the system, then it is very likely that we did a good job. That is, the system has met with the requirements and specifications provided by the client. Also, note that tests should be carried out by qualified persons who are somewhat familiar with the whole system. It is better however, that a tester should not be a developer. Furthermore, a tester should be familiar with testing techniques.

There are different types of test used at different “stages” of the application. For example there are:

  1. The Unit tests.
  2. Structural tests.
  3. Functional tests.
  4. Performance tests.

Some techniques to conduct tests that falls under Quality Control Techniques are:

  1. Fault Avoidance Techniques: Tries to prevent the occurrence of errors and failures by finding faults in the system before it is released. It includes

(a)Development methodologies: provides techniques that reduce fault introduction into the system models and code. Includes unambiguous representations of requirements, minimizing coupling and maximizing coherence, use of data abstraction and encapsulation, capture of rationale configuration for maintenance and early definition of subsystem interface.

(b)Configuration management: avoid faults caused by undisciplined change in the system. Need to notify other developers when changes are made so that other parts of the system that may be affected by such changes may be upgraded.

(c)Verification: attempts to find faults before any execution of the system. Must assume that pre conditions are true and verify that post conditions are met.

(d)Review: is manual inspection of the system without actually executing the system. Two types – walkthrough and inspection. Walkthrough is going through the code line by line trying to identify errors in the code. Inspection, which is similar to walkthrough, checks the code against the requirements, checks the algorithm for efficiency and check the comments to see if they are accurate. Developer is not present during inspection.

  1. Fault Detection Techniques: Attempts to find faults of the system during development, and in some cases, after release. These techniques do not attempt to recover from the fault – example a blackbox that may be found on an aircraft. The blackbox would try to record the reasons for a crash. Two types of fault detection techniques:

(a)Debugging: debugger moves through a number of states finally coming to an error and hence be able to fix it. It is finding an error in what is regarded as an unplanned way. Two type of debugging: correctness debugging – find deviation between observed and specified functional requirements and performance debugging – find deviations between observed and nonfunctional requirements, such as response time.

(b)Testing: finding an error in a planned way. Note – a successful test is a test that was able to find errors. The developer tries to find errors before delivery of the system. The idea is to choose test data that would guarantee a greater degree of failure of the system. Testing activities include:

(i)Unit Testing: tries to find faults in participating objects and/or subsystems with respect to use cases from the use case model.

(ii)Integration Testing: the activity of finding faults when testing the individually tested components together. The system structure testing is the culmination of the integration testing whereby all of the components are tested together.

(iii)System Testing: tests all of the components together (seen as a single system) with respect to functional requirements, design, etc. It includes:

  • Functional Testing: tests the requirements from the RAD (Requirements Analysis Document) and if available from the user manual.
  • Performance Testing: checks the nonfunctional requirements and additional design goals. Done by developers.
  • Acceptance testing and Installation testing: check requirements against project agreement. Done by client with support from developer.
  1. Fault Tolerance Technique:

There may exist cases whereby we are unable to prevent errors. If this is the case, then we must be able to recover from any such errors. Fault tolerance techniques are critical in highly reliable systems example, a life saving unit, the space shuttle (which has 5 onboard computers providing a modular operation), etc.

Testing concepts:

  1. A component: a part of the system that can be isolated for testing – could be an object, a group of objects or one or more subsystems.
  1. A fault: bug or defect – a design or coding mistake that may cause abnormal component behavior.
  1. An error: manifestation of a fault during the execution of the system.
  1. A failure: deviation between the specification of a component and its behavior. This is triggered by one or more errors.
  1. A test case: a set of inputs and expected results that exercises a component with the purpose of causing failures and detecting faults.
  1. A test stub: a partial implementation of components on which the test component depends.
  1. A test driver: a partial implementation of a component that depends on the tested component.
  1. A correction: a change to a component – repair a fault.

Faults, errors and failures:

As noted a fault could be the result of bad coding or design. The figure on page 443 – figure 11.3 shows an example of a fault. In this case, the workmen may have miscommunicated thus resulting in the tracks not aligned. In actual software development, this sort of thing can also happen. The programming may be divided into groups, each group responsible for a subsystem. Due to lack of communication, the subsystems cannot be integrated properly, yet each subsystem is working in their own rights.

A fault would become an error only after the piece of code was tested. As in the example given, the line is tested via a use case. A train is run on the track, which would lead to derailment (a failure of the system).

Test cases:

A test case has 5 attributes:

Attributes / Description
Name / Name of test case
Location / Full path of name of executable
Input / Input data or commands
Oracle / Expected test results against which the output of the test is compared
Log / Output produced by the test.

The name of a test case should reflect the component that is tested, thus it should include part of the components name. The location should describe where the test case is found. That is the pathname or URL, as the case may be, of the executable program and its inputs. Input describes set of inputs to be used that will be entered either by the tester or a test driver. The expected behavior (output) is described by the oracle and the log is a correlation of the observed behavior with the expected behavior for various runs.

Testing must be done in some sort of sequence. That is, if a test case is dependent upon the result of another test, then that test should be completed first. This may seem trivial for small applications, but when applications consist of millions of lines of code and involves dozens of programmer, it becomes very important indeed. Coordination is the key.

Test cases are classified into “black-box” and “white-box” tests depending upon the aspect of the system model that is tested. Black-box test deals with the input/output of the component, not with the structure or the behavior of the component. White-box focuses on the internal structure. This makes sure that every state in the dynamic model of the object and every interaction of the objects is tested. Unit testing involves both black-box and white-box testing. It tests for both input/output as well as the structural and dynamic aspects of the component.

Black-box Testing:

Black-box testing focuses on the functional requirements of the software. It allows the tester to use inputs that would test all functional requirements. It attempts to find errors in the following categories:

(a)Incorrect or missing functions.

(b)Interface errors.

(c)Errors in data structures or external data base access.

(d)Performance errors and

(e)Initialization and termination errors.

Black-box testing is deferred to be done later in the software development process, unlike white-box testing which is done earlier. Some questions that are used to guide a black-box test are:

  1. How is functional validity tested?
  2. What classes of input will make good test cases?
  3. Is the system particularly sensitive to certain input values?
  4. How are the boundaries of a data class isolated?
  5. What data rates data volumes can the system tolerate?
  6. What effect will specific combinations of data have on system operation?

The first step in black-box testing is to understand the objects that are modeled in the software and the relationships that connect these objects. Once this has been done, the next step is to define a series of test that will verify “all objects have the expected relationships to one another.” To do this, the software engineer begins to create a graph with a collection of nodes (that represents the objects), links (edges – that represents the relationships) and node weight (that represent the properties of a node). The link could be directed (relationship moves in one direction), bi-directional (moves in both direction) and parallel (a number of different relationships established between two nodes. The figure shows that a menu select on new file generates a document window. The node weight of document window provides a list of attributes that are to be expected when the window is generated. The link weight indicates that the window must be generated in less than 1 second. An undirected link establishes a symmetric relationship between new file menu select and document text and parallel links exist between document window and document text.

White-box testing:

This test uses the control structure to design the test cases. The tests derived to conduct white-box testing:

  1. Guarantee that all independent paths within a module have been exercised at least once.
  2. Exercise all logical decisions on their true and false sides.
  3. Execute all loops at their boundaries and within their operational bounds.
  4. Exercise internal data structures to assure their validity.

One technique used for white-box testing is the basis-path testing technique. The test case designer would derive a logical complexity measure of a design and use this measure as a guide for defining a basis set of execution paths. Test cases derived to exercise the basis set are guaranteed to execute every statement in the program at least one time during testing.

But a flow graph of the application must first be derived. The following graphs give representation of code statement:

If While

Sequence Do

Until Case

These sub-graphs would be used to convert a program into a flow graph. The sub-graph would represent the various coding constructs that were used in the program. The final graph would then be scanned for independent paths. Each independent path would now require a separate test case. In this way, no line of code within the program would escape testing.

An example: check the attached diagrams.

Test stubs and drivers:

When we want to test single components of a system, we need to separate that component from the rest of the system. We do this by creating stubs to represent the related parts to the system and drivers to carry out the test. The stubs actually are used to simulate the parts of the system that are called by the component that is under test. So a stub may consist of values, etc. that may be required by the component that is tested. Such values would actually provide the test for the component. Note: a test stub should simulate the component that it is substituting as close as possible, else the tested component may not be adequately tested. Thus, it is some times even better to use the actual component that would have been simulated by the test stub, to carry out the test.

Corrections:

Once a problem has been found, then corrections are made. Corrections could be simple and applied to only the component that is under test or they could be more involved requiring changes to an entire class or subsystem. There may be cases whereby entire subsystems need to be redesigned, which may eventually introduce new errors. The authors suggested several techniques that can be used to track and handle any new faults that may evolve:

  1. Problem tracking: once documentation of the entire process of finding errors and correction is kept, then it is easy to revise those portions of the system with the intent of finding faults.
  2. Regression testing: re-execution of all prior tests after a change.
  3. Rationale maintenance: justifying the changes that are made with the requirements of the subsystem.

Testing activities:

  1. Inspecting Components:

Inspections find faults in a component by reviewing its source code. It could be done before or after the unit test. Fagan suggested a five-step method for inspection:

(a)Overview: The author of the component briefly presents the purpose and scope of the component and the goal of the inspection.

(b)Preparation: The reviewers become familiar with the implementation of the component.

(c)Inspection meeting: A reader paraphrases the source code (reads each source code statement and explains what the statement should do) of the component and the inspection team raise issues with the component. A moderator keeps the meeting on track.

(d)Rework: The author revises the component.

(e)Follow-up: The moderator checks the quality of the rework and determines the component needs to be re-inspected.

  1. Unit Testing:

This focuses on the building block of the system, that is, the objects and subsystems. There are 3 main reasons for unit test:

(a)It reduces the complexity of the overall test activity enabling focus on smaller units at a time.

(b)It becomes easier to pinpoint and correct errors.

(c)Allow parallelism in testing i.e. it each component is tested independent of each other.

There are many unit testing techniques:

Equivalence testing: a black-box testing technique that minimize the number of test cases. The possible inputs are partitioned into equivalence classes and a test case is selected for each class. Only one member of an equivalence class needs to be tested. The test consists of two steps: identification of the equivalence class and the selection of the test inputs. To identify equivalence class we use:

  • Coverage: Every possible input belongs to one of the equivalence classes.
  • Disjointedness: No input belongs to more than one equivalence class.
  • Representation: If the execution demonstrates an error when a particular member of an equivalence class is used, then the same error should appear if any other member of the class is used.

For each equivalence class, two pieces of data is used – a typical input and an invalid input. The example given for the method that returns the number of days: 3 equivalence classes were found for the month parameter – months with 31 days, with 30 days and February with either 28 or 29 days. Invalid inputs would be non-positive integers and integers bigger than 12. Two equivalence classes were found for the year parameter – leap year and non-leap year. Negative integers are invalid for year. Together, this yield 6 equivalence classes that would need to be tested – table 11.2 page 455.

Boundary testing: Focuses on the conditions at the boundary of the equivalence class. The testing requires that elements be selected from the “edges” of the equivalence class. The above example: generally years that are multiple of 4 are leap years, but note that years that are multiple of 100 are not leap years unless they are multiple of 400, even though they may be multiple of 4. Example – 2000 is a leap year but 1900 was not a leap year, even though it was a multiple of 100 and 4. Hence, both 1900 and 2000 are good boundary cases for year and 0 and 13 would be good boundary cases for month.

Path Testing: A white-box testing technique that identifies faults in the implementation of the component. The idea behind path testing is that each line of code would be tested at least once, and thus if any fault (error) exist in any of the paths tested, it would be found. Thus to carry out this test, flow graph of the source must be developed. In the case of the example method, the flow graph on page 456 was developed. Note the decisions were taken into consideration – no looping structures in this method. There are five if statements in the code (page 457) represented by the diamond shapes and the activities are represented by the rectangular-rounded-edged shapes consisting of 7- two for the exceptions and 5 for the if’s and their else’s. The table on page 458 (table 11.4) shows the test cases and the path – note there are 6 test cases indicating 6 paths.

Note that even though path testing could be used in OO languages, it was developed

specifically for the imperative languages, thus polymorphism for example, will need

more test cases to be used than could be computed from the cyclomatic complexity

formula. Also, because OO methods are shorter, less control faults may be uncovered.

This is because more inheritance exist in OO and thus the tests would require