We Have Studied CMM and Extreme Programming in the Last Couple of Weeks

Lab _003 --- Process Improvement

SE 3730 / CS 5730 – Software Quality

Due: Oct 18NOON

This exercise makes use of your team’s knowledge of metrics and process to optimize a quality assurance process. The objective is to use this knowledge to improve a software processes. Part of this lab is a list of defects, see the link to Lab 003: Defect Data to Analyze. The defect list contains information about Injection Points (Col C).

Definitions:

Injection point: The lifecycle stage in which a defect was introduced into a system.

Detection point: The lifecycle stage in which a defect was discovered.

Standard Cost: The organization’s average cost to do something – in this case it includes the cost of performing a quality assurance technique or the cost of fixing a defect at several lifecycle stages.

Total cost of finding and fixing a defect: The sum of detection and fixing costs.

Here is how this works:

Use appropriate metrics to compute and analyze the total cost of finding and fixing all defects.
See Table 1 below for the standard costs for fixing a defect based on life-cycle detection point. We cannot fix a defect before it is injected and before it is detected.
See Table 2 below for:
The cost of running a particular quality assurance technique (like requirements review, code review, unit testing, …)
The probability of finding a defect using a particular quality assurance technique. Note that performing a quality assurance technique will NOT catch all defects that are in the system, only a certain percentage of them. Also, it can only detect defects that have already been injected into a system.
Example: It cost 100 pieces of gold to perform a requirements review for this hypothetical project and the review will catch50% of defects (table 2). If there were 10 defects to start with, the requirements review would find 5 of them that could be fixed (leaving 5 unfound). We could run another requirements review using another team of independent reviewers and it would find 50% of residual 5 defects or 2 (we round down to the nearest whole defect).

Table 1: Cost of Fixing Defect by Detection Point
Detection Point / Cost Fixing a Defect
(in pieces of gold) / Stage in which we can run these (see Col C of Excel Spreadsheet)
Req. Review / 50 / Rqmts
Design Review / 100 / Design
Unit Test / 200 / Code
Code Review / 210 / Code
Integration Test / 300 / Integration
System Test / 400 / SystemTest
Alpha Test / 800 / SystemTest
Beta Test / 1000 / SystemTest
Customer Report / 4000 / SystemTest
Table 2: Cost of Quality Assurance Techniques and Probability of finding latent defect
Quality Assurance Technique / Can find defects
injected in / Cost of Running Quality Assurance Technique(in pieces of gold) / Probability of finding defect
Req Review / Rqmts / 100 / .50
Design Review / Rqmts, Design / 150 / .40
Unit Test / Rqmts, Design, Code / 350 / .30
Code Review (prior to Integration test) / Rqmts, Design, Code / 300 / .60
Integration Test / Rqmts, Design, Code, Integration / 250 / .40
System Test / Rqmts, Design, Code, Integration, System Test, / 400 / .30
Alpha Test / Rqmts, Design, Code, Integration, System Test, Documentation / 800 / .40
Beta Test / Rqmts, Design, Code, Integration, System Test, Documentation / 1000 / .50
Customer finds after release / Rqmts, Design, Code, Integration, System Test, Documentation / 0 / 1.00 This means that the customer will ALWAYS find all defects left in the product.

DELIVERABLES:

0. Use a Microsoft Excel compatible spreadsheet to show your calculations and formulas. See the REQUIRED format on the last page of this document.

1. Cost of the initial process, assuming each quality assurance technique is performed once and only once. Note: this will be easy –

Find the number of defects injected at each point.
Run the each quality assurance technique once at that point in the lifecycle to determine how many defects were found, and thus could be fixed
Determine the cost of running that quality assurance technique. Note: Techniques must be run in the order given in Table 1
Determine the cost of fixing the discovered defects
Compute how many defects were not found at each detection point (residue defects)
Add the residual defects to the defects injected at the next life-cycle
And repeat a-f until all quality assurance techniques have been used at their appropriate lifecycle stage. Assume that each technique was run only once.

2. Now for the fun part! Engineer a new quality assurance process consisting of up to three iterations of each quality assurance technique that minimizes the total cost of quality.

For example:
Assume we have 9 requirements defects.
The First Requirements Review cost 100 pieces of gold to run and will catch 50% of the defects or 4 (remember we round down assuming we must find all of a defect to be useful).
After fixing the requirements’ defects identified in the initial review which cost 50 pieces of gold per defect or 200; we can do another(Second) Requirements Review at a cost of 100more pieces of gold.
This Second review will find 50% of the remaining (5 * 50% =2.5 OR 2 defects – rounding down ) resulting in 3 remaining defect.
We can then do this a third time costing another 100 pieces of gold to perform which removes 1 more defect. Leaving 2 remaining defects.
Running the second or third requirements review may or may not be cost effective. It may be cheaper to move onto the next quality assurance technique, Design Review.
Note: a quality assurance technique cannot find defects that have not yet been injected. For instance, a Requirements Review cannot find defects that will be injected in the Code Phase. See the second column from the left in Table 2 for guidance.
The goal of this exercise is to engineer our quality assurance process, so as to minimize the Total Cost of Quality (as defined by: total cost of fixing defectsplus performing the QA techniques).
Use the defect data in the spreadsheet to determine a strategy for minimizing our cost. Use the Table on the last page of this document to display your calculations and results in an Excel spreadsheet.

Find the Cost of Detecting and Fixing per Defect for each iteration. For example on third Code Review iteration, if 3 defects were discovered and fixed, we would add the cost of running a Code Review to the cost of fixing three code defects and then divide this quantity by 3 defects. HINT: this will help you answer the question of whether or not to perform the next iteration of a quality assurance technique.

Find the difference between the original process (Deliverable #1)and your new optimized processes Total Cost (Deliverable #2).

Produce a lab report describing what your team did including a discussion of why you did what you did. Time worked by each individuals and a summary of how long (person-minutes) each task took. Copy the grading rubric to the top of your lab report. Email the lab report and your Excel compatible spreadsheet to me by the due time.

GRADING CRITERIA 0-22 POINTS

Task / Points / Continuum of Earned Points / Comments:
0% - no supported and very wrong answer
50% - weakly supported process and an answer based on process – errors in using data.
100% - strongly support and valid process, reasonable answer.
Total Cost of Defects Deliverable 1 / 2
Improved QA

Devise a better QA strategy

/ 10

For each iteration calculatethe cost of detecting and fixing per defect.

/ 2

Calculation of the total cost fixing all defects

/ 2
Calculation of the difference between the original Total Cost and the new Total Cost. / 2
Lab Report and following all instructions. / 4
TOTAL / 22

Lab _003 --- Process Improvement

SE 3730 / CS 5730 – Software Quality

Required Excel Spreadsheet table format.

A / B / C / D / E / F / G / H / I / J / K / L
# Defects Injected this iteration / Current Lifecycle Stage / Defects available to fix now / QA Tech / PerDefect
cost2fix
(Tbl 1) / Type of Defect Review Can Find / QA
Cost
(Tbl 2) / Probability of Detection
(Tbl 2) / Fixed
Defects / Residual
Defects / Cost:
Fix + QA / Cost per defect
Nn / Rqmts / nnn / Req. Review / 50 / Rqmts / 100 / 0.5 / iiii / jjj / kkkkk / llllll

A: The first iteration of a lifecycle stage this is the total defects injected at this point. For subsequent iterations of a technique this will be 0.

B: The current lifecycle stage

C: Col A plus colK of previous row

D: The quality assurance technique being run

E: Cost to fix each defect discovered at this point from table 1

F: List of the Injection points that this technique can find defects in. See table 2.

G: Cost of running the current quality assurance technique. See table 2.

H: Probability of finding a defect using this quality assurance technique. See table 2.

I: Defects found by this quality assurance technique. This is the floor of col C multiplied by col H.

J: Defects not found. This is col C minus col I.

K: Col E * col I + col G

L: Col l / col K

Lab _003 --- Process Improvement

SE 3730 / CS 5730 – Software Quality