SW Metrics Capability Evaluation Guide
Version 2.0, October 20, 1995
SOFTWARE METRICS CAPABILITY EVALUATION GUIDE
Prepared for:
The Software Technology Support Center (STSC)
Ogden Air Logistics Center (OO-ALC/TISE)
Hill Air Force Base, Utah 84056-5205
Prepared by:
Faye Budlong, Draper Laboratory
Judi Peterson, TRW
Representing the
STSC Metrics Team
XXX
SW Metrics Capability Evaluation Guide
Version 2.0, October 20, 1995
TABLE OF CONTENTS
Section Page
1. INTRODUCTION 1
2. EVALUATION APPROACH 1
2.1 Background 1
2.2 Software Metrics Capability Evaluation Process 2
2.2.1 Initial Contact 2
2.2.2 Evaluation Interview 3
2.2.3 Collating and Analyzing the Results 3
2.3 Software Metrics Capability Evaluation Follow-up 4
2.3.1 Metrics Capability Evaluation Report 4
2.3.2 Project Plan and Implementation 5
LIST OF REFERENCES 6
Appendix Page
A. MEASUREMENT THEMES AND RELATIONSHIPS 7
B. SOFTWARE METRICS CAPABILITY QUESTIONNAIRES 8
B.1 Use of Questionnaires and Scoring 8
B.1.1 Use of Questionnaires 8
B.1.2 Scoring 8
B.2 Metrics Customer Profile Form 10
B.3 System Program Office Questionnaire 13
B.3.1 Questions for Metrics Capability Level 2 13
B.3.1.1 Theme 1: Formalization of Source Selection and Contract Monitoring Process 13
B.3.1.2 Theme 2: Formalization of Metrics Process 14
B.3.1.3 Theme 3: Scope of Metrics 15
B.3.1.4 Theme 4: Implementation Support 16
B.3.1.5 Theme 5: Metrics Evolution 17
B.3.1.6 Theme 6: Metrics Support for Management Control 17
B.3.2 Questions for Metrics Capability Level 3 19
B.3.2.1 Theme 1: Formalization of Source Selection and Contract Monitoring Process 19
B.3.2.2 Theme 2: Formalization of Metrics Process 20
B.3.2.3 Theme 3: Scope of Metrics 20
B.3.2.4 Theme 4: Implementation Support 22
B.3.2.5 Theme 5: Metrics Evolution 23
TABLE OF CONTENTS (Continued)
Appendix Page
B.3.2.6 Theme 6: Metrics Support for Management Control 24
B.4 Software Development/Maintenance Organization Questionnaire 25
B.4.1 Questions for Metrics Capability Level 2 25
B.4.1.1 Theme 1: Formalization of the Development Process 25
B.4.1.2 Theme 2: Formalization of Metrics Process 26
B.4.1.3 Theme 3: Scope of Metrics 26
B.4.1.4 Theme 4: Implementation Support 27
B.4.1.5 Theme 5: Metrics Evolution 27
B.4.1.6 Theme 6: Metrics Support for Management Control 28
B.4.2 Questions for Metrics Capability Level 3 29
B.4.2.1 Theme 1: Formalization of the Development Process 29
B.4.2.2 Theme 2: Formalization of Metrics Process 29
B.4.2.3 Theme 3: Scope of Metrics 30
B.4.2.4 Theme 4: Implementation Support 31
B.4.2.5 Theme 5: Metrics Evolution 32
B.4.2.6 Theme 6: Metrics Support for Management Control 33
C. SOFTWARE METRICS CAPABILITY EVALUATION REPORT: ANNOTATED OUTLINE 35
D. ORGANIZATION INFORMATION FORM 38
LIST OF FIGURES
Figure Page
C-1 Software Metrics Capability Evaluation Results and Recommendations Report: Annotated Outline 36
LIST OF TABLES
Table Page
A-1 Themes and Levels of Software Metrics Capability Maturity. 7
XXX
SW Metrics Capability Evaluation Guide
Version 2.0, October 20, 1995
1. INTRODUCTION
In its role as an agent for improving software technology use within the U.S. Air Force, the Software Technology Support Center (STSC) is supporting metrics technology improvement activities for its customers. These activities include: disseminating information regarding the U.S. Air Force Policy on software metrics [AP93M-017], providing metrics information to the public through CrossTalk, conducting customer workshops in software metrics, guiding metrics technology adoption programs at customer locations, researching new and evolving metrics methodologies, etc.
Helping customers become proficient in developing and using software metrics to support their software development and/or management activities is crucial to customer success. The STSC metrics support activities must be tailored to the customer's needs to ensure
a. that the activities are appropriate to the customer's organization and metrics capability maturity, and[1]
b. that the customer is ready to make improvements based on the support obtained.
Customer support needs include activities based on their apparent metrics capability and those that are particularly focused on dealing with the organizational and cultural issues that often need to be addressed to facilitate change.
This guide covers the following:
a. It defines a metrics capability evaluation method that deals specifically with defining a customer's metrics capability.
b. It presents metrics capability questionnaires that help gather metrics capability data.
c. It outlines a metrics capability evaluation report that provides the basis for developing a metrics customer project plan.
d. It provides a metrics customer profile form used to determine the initial information required to prepare for a metrics capability evaluation.
e. It provides a customer organization information form that helps guide the STSC in gathering cultural information about the organization that will help with developing and implementing the metrics customer project plan.
2. EVALUATION APPROACH
2.1 Background
The foundation for the evaluation method is "A Method for Assessing Software Measurement Technology."[DASK90][2] Metrics capability maturity consists of 5 maturity levels that are analogous to the software Capability Maturity Model (CMM) levels defined by the Software Engineering Institute (SEI).[PAUL93] This guide has been designed to cover metrics capability maturity Levels 1 through 3. When metrics capability evaluations show a strong percentage (e.g., 25 percent or more) of organizations at metrics capability maturity Level 3, the scope of the evaluation (and this guide) will be expanded to cover metrics capability maturity Levels 4 and 5.
This guide defines a set of questions to elicit information that will help characterize an organization's metrics capability. The themes used in the questionnaire and their relationships to an organization's metrics capability maturity (for Levels 1 through 3) are shown in Appendix A.
The guide contains two metrics capability questionnaires (one for acquisition organizations and one for software development/maintenance organizations). The questions in the questionnaires are used as the basis for interviews with an organization's representative(s) to help determine their metrics capability maturity. After the interviews are complete, the results are collated and reported in a evaluation report that is delivered to the evaluated organization. Additional work with the evaluated organization will depend on the organization's needs. Section 2.2 discusses the evaluation process. Appendix B contains a brief metrics customer profile form, which is filled out as a precursor to the metrics capability evaluation. Appendix C is an annotated outline of the metrics capability evaluation report, and Appendix D contains the customer organization information form.
2.2 Software Metrics Capability Evaluation Process
The software metrics capability evaluation process consists of the three basic parts:
a. An initial contact, which is performed when it is determined that an organization needs and wants assistance with its metrics capability.
b. The evaluation interview, which is the central activity in the software metrics capability evaluation process.
c. Collating and analyzing the results, which are the transition activities that occur between the evaluation interview and evaluation follow-up.
These sets of activities are discussed in Paragraphs 2.2.1 through 2.2.3.
In addition to evaluation, there may be follow-up activities. These include more detailed work with the customer that will provide a metrics capability improvement strategy and plan when applicable. Paragraph 2.3 discusses the follow-up activities.
2.2.1 Initial Contact
The initial contact with a customer generally is set up through an STSC customer consultant. The customer consultant briefs an assigned member of the STSC metrics team regarding a customer's need for a metrics capability evaluation and provides a contact for the metrics team member at the customer's site.
The metrics team member contacts the customer by phone to gain an initial understanding of the customer's organization and to set up the evaluation interview. The metrics customer profile form is used to help gather that information. Information collected during this initial contact will be used to help determine the proper approach for the introduction briefing presented during the evaluation interview visit. Only the point of contact information must be completed at this time; however, it is highly desirable to include the STSC business information. When the profile is not completed during the initial contact, it needs to be completed prior to (or as an introduction to) the evaluation interview at the customer's site.
2.2.2 Evaluation Interview
Two STSC metrics team members conduct the interviews as a metrics evaluation team. On the same day as the evaluation interview, an introduction briefing is provided to key people within the organization (to be determined jointly by the evaluation team members, the customer consultant assigned to the organization, and the organization’s primary point of contact). The purpose of the briefing is to manage customer expectations. This is accomplished, in part, by providing education with respect to:
a. The concepts ofmetrics maturity.
b. The approach of the metrics evaluation team.
c. What to expect when evaluation results are provided.
The interviews are conducted with the manager most closely associated with the software development activities for the program (or project) under question.[3] One other representative from the program (or project) should participate in the interview (a staff member responsible for metrics analysis and reporting would be most appropriate). The first part of the interview is to complete the metrics customer profile. When this is completed, the metrics capability questionnaire most related to the organization (either acquirer or development/maintenance organization) is used as the input to the remainder of the evaluation process. The questionnaire sections for both Levels 2 and 3 are used regardless of the customer's perceived metrics capability.
The questions in the metrics capability evaluation questionnaires have been formalized to require answers of yes, no, not applicable (NA), or don’t know (?). If an answer is yes, the customer needs to relate examples or otherwise prove performance that fulfills the question. If the answer is no, comments may be helpful but are not required. (If the answer is don't know, a no answer is assumed.) If the answer is NA and it can be shown to be NA, the question is ignored and the answer is not counted as part of the score. The chosen metrics capability evaluation questionnaires need to be completed before the interview is considered complete.
An evaluation interview should not take more than one day for one program (or software project). If an organization is to be assessed, a representative sample of programs (or software projects) need to be assessed and each requires a separate interview.
2.2.3 Collating and Analyzing the Results
The metrics capability questionnaires completed during the interview(s) and their associated examples (or other evidence of metrics capability maturity, see Paragraph B.1) are collated and returned to STSC for analysis. The metrics capability evaluation team that conducted the interview(s) is responsible for analyzing and reporting the results. An assessed program (or software project) is at Level 2 if at least 80% of all Level 2 questions are answered yes. Otherwise the organization is at Level 1, etc. [DASK90] (Scoring is discussed in more detail in Paragraph B.1. The contents of the metrics capability evaluation report are outlined in Appendix C.)
The questions in the metrics capability questionnaires are organized by metrics capability maturity themes to help focus the interviews and the results analysis. (The themes, as defined in [DASK90], and their characteristics at metrics capability maturity Levels 2 and 3 are reported in Appendix A.) The customer's strengths and weaknesses can be addressed directly with the information gathered during the interview session(s). In addition, activities for becoming more effective in implementing and using metrics can be highlighted in the metrics capability evaluation report and in the project plan.
2.3 Software Metrics Capability Evaluation Follow-up
Software metrics capability evaluation follow-up includes two sets of activities:
a. The metrics capability evaluation report.
b. The project plan and implementation.
The report details the evaluation results and provides recommendations for an initial set of improvement activities.
The project plan consists of a customer approved, detailed plan to improve the customer's metrics capability (which may include other aspects of support to the customer such as software process definition, project management support, or requirements management workshops, etc.).
The customer's organizational culture is important in developing the content and phasing of the project plan. Issues such as ability to incorporate change into the organization, management commitment to software technology improvement, etc., often need to be addressed in developing a success-oriented plan.[4]
Metrics capability improvement implementation consists of the physical implementation of the project plan and a periodic evaluation of the customer's status to determine the program's improvement and any required modifications to the plan. The project plan and implementation are described in Paragraph 2.3.2.
2.3.1 Metrics Capability Evaluation Report
The metrics capability evaluation report consists of two parts:
a. The analyzed results of the evaluation.
b. Recommendations for a set of activities that will help improve the customer's metrics capability.
The results portion of the report is organized to discuss the customer's overall software metrics capability and to define the areas of strengths and weaknesses based on each of the measurement themes. The recommendations portion of the report describes an overall improvement strategy that provides a balanced approach toward metrics capability improvement based on the customer's current evaluation results. Appendix C contains an annotated outline of the report.
2.3.2 Project Plan and Implementation
If a customer has the interest to proceed with a project plan, the STSC will develop the plan in conjunction with the customer. The contents of the project plan, the estimates for plan implementation, and the schedule will be developed specifically for each customer's needs. Due to the possible variations in customer needs, it is difficult to determine the exact contents of the plan. At a minimum, the project plan contains the following information:
a. An executive overview, which includes a synopsis of the customer's current software metrics capability maturity and a general outline of the plan to be implemented.
b. Organizational responsibilities for the customer, the customer's interfacing organizations (e.g., a contractor), and the STSC. Issues that arise based on organizational information are highlighted.
c. Improvement objectives.
d. A set of activities to support improvement [e.g., a Work Breakdown Structure (WBS)] and a description of the activities' interrelationships.
e. A schedule for implementation and for periodic evaluation of the customer's progress. (The periodic evaluation may be implemented as additional metrics capability evaluations, as described in this guide.)