Cleanroom Software Engineering

Derek B. Larson

Software Engineering

University of Wisconsin Platteville

Abstract

The main objective of Cleanroom Software Engineering is to achieve or approach zero defects with certified reliability. Cleanroom Software Engineering provides a complete discipline where software personnel can plan, specify, design, verify, code, test, and certify software. In the Cleanroom development, correctness verification replaces unit testing and debugging. After the coding is complete, the software goes directly to the system testing stage with no debugging. All test errors are then accounted for from the first execution of the program with no private testing allowed. The Cleanroom process places greater emphasis on design and verification rather then testing. In this process errors are detected early in the life cycle of the program, closer to the point of insertion of the error, which makes fixing the errors less expensive.

Introduction

What is Cleanroom Software Engineering? Just imagining those all white rooms in the movies, where all of the people working in the room are dressed in white suits. There is no dust, no germs, and it is clean enough to eat off the floor. Now add software development to the mix. This is the Cleanroom process, IBM developed this process to have the main objective be to approach little to no defects with certified reliability of a system. So the very clean white room means that at the very beginning of the project, all of the details will be worked out. In the Cleanroom process a lot of money is spent up front to prevent defects, or in the case of the example the defects act like the dust and germs.

The common software development model today is usually, code first then fix, or just design the project as you code. Doing a project this way makes the programmer figure out what the program should do, design the program, and debug the program all at the same time. This is very tough to do and makes this development process not very good at preventing defects. There are several different models that separate all of the different parts of the program, for example the water fall model defines a requirements, design, code, and test process. The problem with this is that often the detail of what is done in each step is very informal. This is where the Cleanroom process steps in. “The cleanroom model follows the ideas of manufacturing semiconductors in a "cleanroom" environment” [4]. There are some requirements for using the Cleanroom process

1.  Most spend a lot of time and money at the start of the project for preventing defects.

2.  Most use statistical methods to ensure quality.

3.  Most formally state and “prove” requirement needs.

By following these requirements and adding in some reviews and some different testing techniques is how to develop a project using Cleanroom Software Engineering.

Figure 1: Static and dust free environment.

Cleanroom Process

The Cleanroom process depends on describing the development with three different box structures, black box, state box, and clear box [4]. Black box is a view of an object that hides the implementation process and data. It will describe how that system will respond to stimuli; this is done usually in a formal specification language. State box is where the view of an object shows the data implementation, but does not show the implementation process. It describes how state information is being transformed. Finally clear box shows both the data and process implementation, were the goal is to stepwise refine functions and prove them as being correct. These boxes are used to help test and prove that the system works correctly.

According to Uta.edu, the following process constitutes the Cleanroom approach to software development.

·  Requirements Analysis: Producing and reviewing “informal specifications.”

·  High-level Design: Converting the requirements into state machines and functions.

·  Detailed Design: Further refinement of functions.

·  Coding by increment: Developing code and verifying it using informal methods. Compiling code or unit testing is prohibited.

·  Pretest by increment: Generation of test cases.

·  Statistical Testing by Increment: The code is compiled, linked, and tested. Then the results are validated.

One of the weird points that really differ from most other models is that Cleanroom Software Engineering prohibits unit testing. In a Cleanroom development, correctness verification replaces unit testing and debugging. After coding is complete, the software immediately enters system test with no debugging. All test errors are accounted for from the first execution of the program with no private testing allowed [2]. The role of system testing in the Cleanroom process is to certify the quality of the software with the systems specifications in mind. Not doing unit testing can only be done if the above requirements are followed, that way many of the defects are already found and fixed, so when the system is done coding it should be close to no defects.

Figure 2: The Cleanroom process

From figure 1, the Cleanroom process starts with requirements and initial increment plan. Then the specification increment plan is design, which is done by the specification team. After this is done the project splits into two different phases, Box-Structured Specification and Design Verification and Develop Usage Model Test Planning. The Box-Structured Specification is done by the development team, and uses the boxes that were described earlier in this section. The Develop Usage Model Test Planning is done by the certification team. When both of these phases are done the project goes to the Statistical Testing Certification phase. This phase is performed by the certification team, and it will use some of the boxes to help with the statistical testing. After this phase is complete the project is ready for delivery. For a more in-depth Cleanroom Process see figure 2. This is basically the Cleanroom Software Engineering process, and is usually combined with another model, for instance the spiral model to help to lower the cost and errors of line of code.

Figure 3: In-depth Cleanroom Process

Waterfall Model into a Cleanroom

The waterfall model is a development model that has been around for a very long time. The process of the waterfall model has different phases; each has a different milestone that is met after a phase is done. So we want to take the Cleanroom model and add some milestones to the model. Most customers like to know where in the project the development is at and by having milestones to show the customer how far the development is on the project.

Figure 4: Waterfall Model and Cleanroom Model

As shown in figure 2, the Preliminary Design Review (PDR) is held earlier (in the middle of Increment 1) and addresses the overall CSCI architecture. This is followed by a succession of Critical Design Reviews (CDR), one for each increment. The last CDR is held later than the CDR in the traditional waterfall model, but by this time there are already one or more increments of certified software developed [3]. As you can see that Cleanroom Model starts the testing of the project very early, while the waterfall model does not test until the end of the project. After certification of each increment, an informal Qualification Test (QT) is run, and a final, formal QT is held after the last increment is certified. Despite the incremental development approach, the CSCI schedule duration was effectively unchanged. To develop software in multiple increments results in tested software functionality earlier than the traditional waterfall approach [3]. The reason why the Cleanroom Model starts the testing of the project so early in the model is that this should help find the defects earlier in the system. The main reason for finding the defects of the project earlier means that it will cost less to fix the defects then instead of waiting to the end of the project.

Table 1: Milestones in the Historical and Cleanroom

Milestone / Historical / Cleanroom
Software Specification
Review / Addresses requirements and
external interfaces for the CSCI. / Remains the same with increment plan-mapping
requirements to increment.
Preliminary Design
Review / Top-level architecture of
the CSCI. / Top-level architecture of the CSCI. Updated as
needed in later increments.
Critical Design Review / Detailed Design of CSCI. / Detailed Design of functionality for particular
increment.
Qualification Test (QT) / Verify CSCI requirements. / Verify CSCI requirements. Performed on final CSCI
increment. Earlier increments have informal QT.

As shown in table 1, Cleanroom Software Engineering can be used to have defined milestones. Comparing the Historical and Cleanroom models show that the Cleanroom does every thing the Historical model does, plus the Cleanroom model adds other important steps used to keep defects low.

Case Studies

Cleanroom Software Engineering does not have the long history of use that several other process models have. Nevertheless there are several projects that have reported on their experience with Cleanroom Software Engineering [4]. One program that used Cleanroom Software Engineering was the “STARS” program which was developed by the Department of Defense. The “STARS” program emphasized on three main points, process-driven, re-use based and integrated software engineering environment. This made Cleanroom Software Engineering a clear process to use since it covers all of these points. “STARS and a group at the Picatinny Arsenal have evaluated current "traditional" processes. They have determined that a quality management philosophy (putting decision making in the hands of workers, focusing on processes, quantitative measurements) is critical and that Cleanroom Software Engineering follows this philosophy [4]. We know the government likes to save as much money as they can, so how much money did using the Cleanroom process save them? In the “STARS” project, “Cleanroom was combined with the TRW (spiral) Ada Process Model and some generated and reused code to produce software at a rate of $30-40 per line of code versus industry averages of $130 per line for software of similar complexity and development timeframe (the size of the application is greater than 300 KLOC)” [1]. So by doing some simple subtraction, combine the Cleanroom process with the spiral process helped the Department of Defense save around $100 per line of code. So since the size of the program was greater then 300 thousands lines of code, the Department of Defense saved about $300,000. Now that is a lot of extra spending money to have, even for the government.

Another project that was done using the Cleanroom process was the Tank-automotive and Armaments Command at the United States Army Pictinny Arsenal. “After seven project increments (approximately 90K lines of code), a 4.2:1 productivity increase and a 20:1 return on investment has been documented” [1]. It is easy to see that the Army saved a lot of money by using the Cleanroom process with the 20:1 return on investment. Not only did the United States Army save money, but the error rate on the project was very low. This project experienced an overall testing error rate (represents all errors found in all testing) of 0.5 errors/KLOC [1]. The Army project had an amazing low error rate of 0.5 errors found in one thousand line of code.

By looking at the statistics of these two projects, it is very easy to see that using Cleanroom Software Engineering is very productive. It saved the Department of Defense about $300,000. The United States Army used the Cleanroom process which made the project have an amazing low error rate of 0.5 errors found in one thousand line of code. Some of the reason these projects worked with the Cleanroom process was because being the Department of Defense, they had enough money at the start of the project to complete the requirements for Cleanroom Software Engineering. Smaller companies do not have that much money to invest into a project at the start because they do not know if the project will ever be finished. This is one reason why most companies shy away from Cleanroom Software Engineering.

Conclusion

In summary, Cleanroom Software Engineering focuses on high quality software through preventing defects. Which means the goal of Cleanroom Software Engineering is to have at the end of the project, very few or no defects. Cleanroom Software Engineering requires major changes in the way to approach software. For example, formal correctness proofs and counting compiler errors as defects are the two notable changes. Testing in the Cleanroom process is very different too; it depends on statistical use testing. This is a probabilistic model of how it is used in the project. This testing produces certification of each of the components in the project. As the case studies shows, Cleanroom Software Engineering produces software with drastically less defects and very high productivity from the group members. Since the group members are getting lot of work done, it usually means that the group has high moral. If the group sees that the project is getting done quickly and efficiently, it usually helps keep the group happier.

Taking a look back at some of the projects that have used Cleanroom Software Engineering, it is easy to see how efficient this process is. The Department of Defense’s “STARS” program produced software at a rate of $30-40 per line of code versus the industry average of $130 per line. Since this project was greater than 300 KLOC the Department of Defense saved about $300,000. Another project that was done using the Cleanroom process was the tank-automotive and Armaments Command at the United States Army Pictinny Arsenal. Productivity increased by 4.2:1 and a 20:1 return on investment was documented. There is a lot of money and development time can be saved by either solely using Cleanroom Software Engineering, or by combining Cleanroom Software Engineering with another software development model.