8 Testing the Programs

Once you have coded your program components, it is time to test them. There are man types of testing, and this chapter and the next will introduce you to several testing approaches that lead to delivering a quality system to your customers Testing is not the first place where fault finding occurs; we have seen how requirements and design reviews help us ferret out problems early in development f But testing is focused on finding faults, and there are many ways we can make our testing effort more efficient and effective, In this chapter, we look at testing components individually and then integrating them to check the interfaces. Then, in Chapter 9, we concentrate on techniques for assessing the system as a whole.

8.1 SOFTWARE FAULTS AND FAILURES

In an ideal situation, we as programmers become so good at our craft that every program we produce works properly every time it is run. Unfortunately, this ideal is not reality The difference between the two is the result of several things. First, many software systems deal with large numbers of states and with complex formulas, activities, and algorithms. In addition to that, we use the tools at our disposal to implement a customer's conception of a system when the customer is sometimes uncertain of exactly what is needed. Finally the size of a project and the number of people involved can add complexity. Thus, the presence of faults is a function not just of the software, but also of user and customer expectations.

what do we mean when we say that our software has failed? Usually, we mean that the software does not do what the requirements describe. For example, the specification may state that the system must respond to a particular query only when the user is authorized to see the data. If the program responds to an unauthorized user, we say that the system has failed. The failure may be the result of any of several reasons:

. The specification may be wrong or have a missing requirement. The specification may not state exactly what the customer wants or needs. In our example, the customer may actually want to have several categories of authorization, with each category having a different kind of access, but has never stated that need explicitly

. The specification may contain a requirement that is impossible to implement, given the prescribed hardware and software.

. The system design may contain a fault. perhaps the database and query-language designs make it impossible to authorize users.

. The program design may contain a fault. The component descriptions may contain an access control algorithm that does not handle this case correctly

. The program code may be wrong. It may implement the algorithm improperly or incompletely

Thus, the failure is the result of one or more faults in some aspect of the system.

No matter how capably we write programs, it is clear from the variety of possible faults that we should check to ensure that our components are coded correctly Many programmers view testing as a demonstration that their programs perform properly However, the idea of demonstrating correctness is really the reverse of what testing is all about. We test a program to demonstrate the existence of a fault. Because our goal is to discover faults, we consider a test successful only when a fault is discovered or a failure occurs as a result of our testing procedures. Fault identification is the process of determining what fault or faults caused the failure, and fault correction or removal is the process of making changes to the system so the faults are removed.

By the time we have coded and are testing program components we hope that the specifications are correct. Moreover, having used the software engineering techniques described in previous chapters, we have tried to assure that design of both the system and its components reflects the requirements and forms a basis for a sound implementation. However, the stages of the software development cycle involve not only our computing skills but also our communication and interpersonal skills. It is entirely possible that a fault in the software can result from a misunderstanding during an earlier development activity

It is important to remember that software faults are different from hardware faults Bridges, buildings, and other engineered constructions may fail because of shoddy materials, poor design, or because their components wear out. But loops do not wear out after several hundred iterations, and arguments are not dropped as they pass from one component to another. If a particular piece of code is not working properly, and if a spurious hardware failure is not the root of the problem, then we can be certain that there is a fault in the code. For this reason, many software engineers refuse to use the term "bug" to describe a software fault; calling a fault a bug implies that the fault wandered into the code from some external source over which the developers have no control. In building software, we use software engineering practices to control the quality of the code we write.

In previous chapters, we examined many of the practices that help minimize the introduction of faults during specification and design. In this chapter, we examine techniques that can minimize the occurrence of faults in the program code itself.

Types of Faults

After coding the program components, we usually examine the code to spot faults and eliminate them right away when no obvious faults exist, we then test our program to see if we can isolate more faults by creating conditions where the code does not react as planned. Thus, it is important to know what kind of faults we are seeking.

An algorithmic fault occurs when a component's algorithm or logic does not produce the proper output for a given input because something is wrong with the processing steps. These faults are sometime easy to spot just by reading through the program (called desk checking) or by submitting input data from each of the different classes of data that we expect the program to receive during its regular working. Typical algorithmic faults include

. branching too soon

. branching too late

. testing for the wrong condition

. forgetting to initialize variables or set loop invariants

. forgetting to test for a particular condition (such as when division by zero might occur)

. comparing variables of inappropriate data types

When checking for algorithmic faults, we may also look for syntax faults. Here, we want to be sure that we have properly used the constructs of the programming language. Sometimes, the presence of a seemingly trivial fault can lead to disastrous results. For example, Myers (1976) points out that the first US. space mission to Venus failed because of a missing comma in a Fortran do loop. Fortunately compilers catch many of our syntax faults for us.

Computation and precision faults occur when a formula's implementation is wrong or does not compute the result to the required degree of accuracy For instance, combining integer and fixed- or floating-point variables in an expression may produce unexpected results. Sometimes, improper use of floating-point data, unexpected truncation, or ordering of operations may result in less-than-acceptable precision.

When the documentation does not match what the program actually does, we say that the program has documentation faults. Often, the documentation is derived from the program design and provides a very dear description of what the programmer would like the program to do, but the implementation of those functions is faulty Such faults can lead to a proliferation of other faults later in the program's life, since many of us tend to believe the documentation when examining the code to make modifications.

The requirements specification usually details the number of users and devices and the need for communication in a system. By using this information, the designer often tailors the system characteristics to handle no more than a maximum load described by the requirements. These characteristics are carried through to the program design as limits on the length of queues, the size of buffers, the dimensions of tables, and so on. Stress or overload faults occur when these data structures are filled past their specified capacity

Similarly capacity or boundary faults occur when the system's performance becomes unacceptable as system activity reaches its specified limit. For instance, if the requirements specify that a system must handle 32 devices, the programs must be tested to monitor system performance when all 32 devices are active. Moreover, the system should also be tested to see what happens when more than 32 devices are active, if such a configuration is possible. By testing and documenting the system's reaction to overloading its stated capacity, the test team can help the maintenance team understand the implications of increasing system capacity in the future. Capacity conditions should also be examined in relation to the number of disk accesses, the number of interrupts, the number of tasks running concurrently and similar system-related measures.

In developing real-time systems, a critical consideration is the coordination of several processes executing simultaneously or in a carefully defined sequence. Timing or coordination faults occur when the code coordinating these events is inadequate. There are two reasons why this kind of fault is hard to identify and correct. First, it is usually difficult for designers and programmers to anticipate all possible system states. Second, because so many factors are involved with timing and processing, it may be impossible to replicate a fault after it has occurred.

Throughput or performance faults occur when the system does not perform at the speed prescribed by the requirements. These are timing problems of a different sort: Time constraints are placed on the system's performance by the customer's requirements, rather than by the need for coordination.

As we saw during design and programming, we take great care to ensure that the system can recover from a variety of failures. Recovery faults can occur when a failure is encountered and the system does not behave as the designers desire or as the customer requires. For example, if a power failure occurs during system processing, the system should recover in an acceptable manner, such as restoring all files to their state just prior to the failure. For some systems, such recovery may mean that the system will continue full processing by using a backup power source; for others, this recovery means that the system keeps a log of transactions, allowing it to continue processing whenever power is restored.

For many systems, some of the hardware and related system software are prescribed in the requirements, and the components are designed according to the specifications of those reused or purchased programs. For example, if a prescribed modem is used for communications, the modem driver generates the commands expected by the modem and reads commands received from the modem. However, hardware and system software faults can arise when the supplied hardware and system software do not actually work according to the documented operating conditions and procedures.

Finally the code should be reviewed to confirm that organizational standards and procedures have been followed. Standards and procedure faults may not always affect the running of the programs, but they may foster an environment where faults are created as the system is tested and modified. By failing to follow the required standards, one programmer may make it difficult for another to understand the code's logic or to find the data descriptions needed for solving a problem.

Orthogonal Defect Classification

It is useful to categorize and track the types of faults we find, not just in code, but anywhere in a software system. Historical information can help us predict what types of faults our code is likely to have (which helps direct our testing efforts), and clusters of certain types of faults can warn us that it may be time to rethink our designs or even our requirements. Many organizations perform statistical fault modeling and causal analysis, both of which depend on understanding the number and distribution of types of faults. For example, IBM's Defect Prevention Process (Mays et al. 1990) seeks and documellts the root cause of every problem that occurs; the information is used to help suggest what types of faults testers should look for, and it has reduced the number of faults injected in the software.

Chillarege et al. (1992) at IBM have developed an approach to fault tracking called orthogonal defect classification, where faults are placed in categories that collectively paint a picture of which parts of the development process need attention because they are responsible for spawning many faults Thus, the classification scheme must be product and organization-independent, and be applicable to all stages of development. Table 8. 1 lists the types of faults that comprise IBM's classification. When using the classification, the developers identify not only the type of fault, but, also whether it is a fault of omission or commission. A fault of omission is one that results when some key aspect of the code is missing; for example, a fault may occur when a variable is not initialized. A fault of commission is one that is incorrect; for example, the variable is initialized to the wrong value.

One of the key features of orthogonal defect classification is its orthogonality. That is, a classification scheme is orthogonal if any item being classified belongs to exactly one category In other words, we want to track the faults in our system in an unambiguous way so the summary information about number of faults in each class is meaningful. We lose the meaning of the measurements if a fault might belong to more than one class. In the same way the fault classification must be clear, so any two developers are likely to classify a particular fault in the same way.