USC-CSE 20041/10/2019
Contextual Models of Cost, Quality and Cycle Time
Authors
Cvetan Redzic, Michael Crowley, Nancy Eickelmann, Jongmoon Baik
Motorola Labs
Schaumburg, IL 60196
We present a value-based approach to controlling software cost, quality and cycle time using multiple models of the software development process. These models are presented in the context of two meta-models, the CMM and Knox model. The Software Engineering Institute (SEI) Capability Maturity Model (CMM) provides a guideline for the required processes that need to be in place for a software development team and their organization to develop quality software, on time and within budget. The CMM decomposes the essential control elements required to conduct software engineering in an efficient and effective manner. These elements include documented requirements, communications with the customer(s) and user(s), agreed-to commitments, planning, documented process and work breakdown structure WBS. These elements are embedded in a five tier ordinal hierarchy comprised of Key Process Areas (KPAs). It is assumed that the higher the maturity level the more cost effective the software development process, specifically the cost of poor quality is incrementally reduced by creating the product “right” the first time, resulting in producing software of quality, on time and on budget.
This paper examines evaluating process improvements based on results, or the improvement in cost structure in four quality cost categories: prevention, appraisal, internal failures and external failures. The Knox model of software quality provides a theoretical account of changing quality cost structure as a function of a maturing software development process. The Knox model has been found to serve as a reasonable predictor of the cost of poor quality in a number of empirical studies. Knox divides the cost of poor quality (CoPQ) into two categories (Internal and External Failure), and adds two categories (Appraisal and Prevention) to complete the Costs of Quality (CoQ). The category Create is not a CoQ category, but rather the total development cost less all CoQ. See Figure 1.
Figure 1 Knox’s Theoretical Model for Cost of Software Quality (Digital Technical Journal, vol.5, No. 4., Fall 1993, Stephen T. Knox.)
Figure 1 provides a graphical representation of a hypothetical cost structure that would correspond to the five levels of the CMM; the x-axis plots the five CMM levels, the y-axis plots the four cost of quality categories and a cumulative total cost of quality (TC0SQ). The two basic assumptions are that total cost of quality is about 60% for a level 1 organization and that this diminishes by about two thirds, to about 20%, for a level 5 organization. A level 1 organization has no defined process and constructs software in an ad hoc manner. The cost structure associated with a level 1 organization has a total cost of quality of approximately 60%, prevention 1%, appraisal 3%, internal failures 22%, and external failures 34%. These percentages are sensitive to the definitions of quality and how failures and appraisal activities are recorded. For instance, the cost of appraisal, if there were no failures would be the cost of executing unique tests once. However, if failures are found this contributes to the cost of internal failure. Each maturity level has a corresponding cost structure for the cost of quality.
The Knox model for cost of software quality was developed using the Cost of Quality model developed in manufacturing environments and extending it across the SEI CMM to produce a theoretical cost of software quality model. The definitions associated with the model are provided in Table 1.
Table 1. Definition of Cost CategoriesCost / Category / Definition / Typical Costs for Software
Conformance / Appraisal / Discovering the condition of the product / Testing and associated activities, product quality audits
Prevention / Efforts to ensure product quality / SQA administration, inspections, process improvements, metrics collection and analysis
Non-conformance / Internal failures / Quality failures detected prior to product shipment / Defect management, rework, retesting
External failures / Quality failures detected after product shipment / Technical support, complaint investigation, defect notification
We now present three models, COQUALMO, COQ-DES, and MotoROI, which are used in an integrated manner to provide and value-based approach to software development and process improvement.
COQUALMO
COQUALMO (COnstructive QUALity MOdel) is one of COCOMO II suite of models; it is be usedto predict the number ofintroduced defects and removed defects by phase and residual defects in a software product development [Boehm et al. ,2000]. It can be applied in the early stage of the software development life cycle such as analysis and design, as well as in the later stages for refining the estimates when more information is available. It enables 'what-if' analyses that demonstrate the impact of various defect removal techniques and the effects of personnel, project, product and platform characteristics of software development. It also provides insights into determining ship time, assessment of payoffs for quality investments and understanding of interactions amongst quality strategies.
COQUALMO is comprised of two separate models, Defect Introduction (DI) and Defect Removal (DR) to estimate the rates of software defect introduction and defect removal. The COQUALMO-DI model uses the same project attributes (size, personnel, platform, project, and product characteristics) as used in COCOMO II to estimate the member of requirements, design, and code defects introduced into the project. Its DR model uses three defect removal profiles, analysis tools, peer reviews, and execution testing, to determine what fraction of the introduced defects are removed.
Figure 2 COQUALMO model of defect introduction and removal. (Software Cost Estimation with COCOMO II; Barry Boehm, et al.; Prentice Hal PPT; 2000)
CoQ-DES
CoQ-DES (Cost of Quality Discrete Event Simulator) is a Monte Carlo simulation of a software development process that produces output distributions, by development phase, of injected and removed faults, and per-phase costs of quality (and hence, costs of poor quality). The inputs to the model are development phase dates, a priori fault injection rate, quantified characterization of peer review and test activities, and hourly loaded engineering costs. The per-phase a priori injection rates are well accounted for by using historical per-phase COQUALMO DI rates applied to a current project estimations; DR rates are currently unused. Calibration is in terms of historical CoQ categories: Create, Prevent, Appraise, Internal and External Failure.
The model is developed as a number of Fault Injecting Phases, Fault Removal Phases, and Customer Use Phases, as shown in figure 3. Because of the modular structure of CoQ-DES, it can be shown to be extensible as to number of phases and non-waterfall methodologies. CoQ-DES is calibrated using the Knox Cost of Quality categories outlined above.
Figure 3. Waterfall instantiation of CoQ-DES.
The “what-if” experiments using CoQ-DES are directed to per-phase fault and cost distributions. The interplay of practitioner cost, phase duration and engineering activity are seen as contributors to cost and savings. Many of the experiments can drive MotoROI, such that an optimizing software development organization can use the tools together to quantify process advances.
MotoROI
The return on investment (ROI) or yield of applying verification and validation (V&V) to a given project consists of the following steps:
1)The Cost of Software Quality/ Cost of Poor Software Quality is calculated using the Knox Model.
2)The costs associated with phase containment or leakage are derived from historical data sets obtained from large industrial projects. Using this historical data, rework costs for V&V applied at different lifecycle phases or at combinations of different phases is calculated.
3)Based on the rework costs and industry guidelines, Potential Maximum Return from investment in V&V is calculated.
4)The actual expected ROI is also calculated based on the investment made in V&V. The expected return is bounded by the actual budgets and V&V scope selected.
By considering the CMM level of the software organization, different phases of the lifecycle as well as different combinations of V&V activities, our ROI model fulfills the needs in a new way. A prototype tool has also been developed that takes the input factors such as CMM Level, Software Budget, V&V Budget, V&V Scope, etc. and outputs the expected ROI and Potential Maximum Return. An example of model output is given below, the output restates the critical input parameters such as CMM level and project budget information. Sample scenarios of ROI estimation and resulting COPQ savings are shown in Table 2. Report 1 on the table presents the results of RM improvements for a CMM level 5 organization with 50% technological effectiveness for the investment in improvement. At high maturity levels it is difficult to achieve cost savings due to the high efficiency level of the organization. Reports 2 through 4 present CMM level 3 results with 100%, 75%, and 50% technology investment effectiveness ratings. Report 2 is generated using 100% effectiveness, the ROI is positive but the likelihood of perfect results through technology insertion is low. Report 3 has a 1:1 ROI and reduces COPQ at 75% effectiveness; this is a highly likely outcome. Report 4 has a negative ROI at 50% effectiveness, which could be used to bound a worst-case scenario when evaluating the
proposed technology change.
Results Summary**
This section will provide several examples from applying the models in an industrial setting resulting in significant savings and cycle time improvements while reducing defect injection.