COMPOSABLE RISK-DRIVEN PROCESSES FOR DEVELOPING SOFTWARE SYSTEMS FROM COMMERCIAL-OFF-THE-SHELF (COTS) PRODUCTS
by
Ye Yang
A Dissertation Presented to the
FACULTY OF THE GRADUATESCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(COMPUTER SCIENCE)
December 2006
Copyright 2006Ye Yang
Dedication
To my parents.
Acknowledgements
This dissertation could not have been completed without the support of many hearts and minds. First and foremost, I would like to thank my PhD committee members; in particular, my advisor Dr. Barry Boehm. I am deeply indebted to him for his encouragement, advice, mentoring, and research support throughout my academic program. I also truly appreciate his kindness and generosity which made my study life more smoothly and enjoyable. My sincere thanks are also extended to other committee members Dr. George Friedman, Dr. Ming-Deh Huang, Dr. Nenad Medvidovic, and Dr. Stan Settles, for the invaluable guidance on focusing my research and efforts on reviewing drafts of my dissertation.
I would also like to express my thanks and gratitude to several other people. Dr. Daniel Port guided me in identifying potential empirical studies in the USC e-services projects. Mr. Jesal Bhuta has been a wonderful friend and great co-worker for several years. He has provided graciously of his time in helping me succeed. Dr. Elizabeth Clark has been very generous and helpful to share the COCOTS data and provide independent assessment in evaluating the performance of my work. My special thanks to some other colleagues, including Dan Wu, Zhihao Chen, Yue Chen, Ricardo Valerdi, Jo Ann Lane, the discussions with whom have greatly benefited my study. I would also like to thanks Ms. Winsor Brown and Dr. Brad Clark for their help and collaboration in my study.
I also wish to thank Prof. Mingshu Li from Institute of Software Chinese Academy of Sciences, whose early mentoring and support have not only given me the exposure to the field of software engineering, but taught me a lot about professional posture and inter-personal skills.
Lastly, from the bottom of my heart, I would like to thank my family for their unconditional love and support during my study. My parents have inspired me to always strive for the excellence since childhood. My brother, sister-in-law, and niece, were always there every step of the way. Lei, my dear husband, your love is the greatest gift of all, and I could not have done it without you! And my dearest daughter, QianQian, who has just arrived to the world as I completed this dissertation, you are my forever sunshine and strength!
Table of Contents
Dedication ii
Acknowledgements iii
List of Tables
List of Figures
Abstract
Chapter 1 Introduction
1.1 Motivation
1.2 Nature of the problem
1.3 Research Approach and Propositions
1.4 Dissertation Outline
Chapter 2 A Survey of Related Work
2.1 Processes for COTS-Based Development
2.1.1 COTS Selection Process
2.1.2 Waterfall-Based Lifecycle Processes
2.1.3 Spiral-Based Lifecycle Processes
2.1.4 Object-Oriented Process Models
2.2 Costing CBD
2.2.1 COCOTS
2.2.2 Others
2.3 Risks in CBD
2.4 Conclusions
Chapter 3 Composable Processes for Developing CBAs
3.1 Empirical Study Indications
3.1.1 Background
3.1.2 Quantitative CBA Growth Trend
3.1.3 COTS-Related Effort Distribution
3.1.4 Relating COTS Activity Sequences to Risks
3.2 Five principles for CBA development
3.2.1 Process happens where the effort happens
3.2.2 Don’t start with requirements
3.2.3 Avoid premature commitments, but have and use a plan
3.2.4 Buy information early to reduce risk and rework
3.2.5 Prepare for COTS change
3.3 CBA Process Decision Framework
3.3.1 Win Win Spiral Model and its Limitation
3.3.2 CBA Process Decision Framework
3.4 Composable Process Elements
3.4.1 Assessment Process Elements (APE)
3.4.2 Tailoring Process Element (TPE)
3.4.3 Glue Code Process Element
3.5 Generating Composable Process instances
3.5.1 Project Description
3.5.2 Process Description
3.5.3 Spiral Cycle 1
3.5.4 Spiral Cycle 2
3.5.5 Spiral Cycle 3
Chapter 4 COCOTS Risk Analyzer
4.1 Background
4.1.1 COCOTS model
4.1.2 Glue Code Sub-model in COCOTS
4.2 Modeling the COCOTS Risk Analyzer
4.2.1 Constructing the knowledge base
4.2.2 Identifying cost factor’s risk potential rating
4.2.3 Identifying project risk situations
4.2.4 Evaluating the probability of risk
4.2.5 Analyzing the severity of risk
4.2.6 Assessing overall project risk
4.2.7 Providing risk mitigation advices
4.3 Prototype of COCOTS Risk Analyzer
4.4 Evaluations
4.4.1 USC e-services projects
4.4.2 Industry projects
4.4.3 Discussion
Chapter 5 Optimizing CBD Process Decisions
5.1 Project Context
5.2 Risk based prioritization Strategy
5.3 Optimizing process decisions
5.3.1 Establishing COTS evaluation criteria
5.3.2 Scoping COTS assessment process: An example
5.3.3 Sequencing COTS integration activities
5.3.4 Prioritization of top features: Another example
5.4 Discussion
Chapter 6 Validation Results
6.1 Results from Fall 2004
6.1.1 Team Performance
6.1.2 Client Satisfaction
6.1.3 Other Results
6.2 Results from Fall 2005
6.2.1 Pre-experiment survey results
6.2.2 Experiment results
6.2.3 Post-experiment survey results
6.3 Threats to Validity
Chapter 7 Contribution and Future Work
7.1 Contributions
7.2 Areas of Future Work
References
Appendix A Empirical analysis results
Appendix B Guidelines for developing assessment intensive CBAs
Appendix C COCOTS Risk Analyzer Delphi
Appendix D USC E-Services Project COTS Survey
List of Tables
Table 1.1 COTS characteristics and impacts
Table 1.2 Composable Process Elements
Table 3.1 CBA Activity Sequence Examples
Table 3.2 Anticipated COTS activity patterns
Table 3.3 Unanticipated COTS Activity Patterns
Table 3.4 Mapping Win Win Spiral segments to CBA PDF
Table 3.5Comparing CBA APE to ISO/IEC 14598-4
Table 3.6Tailoring methods and common evaluation parameters
Table 3.7 OIV Spiral Cycle 1 – Alternative identification
Table 3.8 OIV Spiral Cycle 1 – Evaluation and elaboration
Table 3.9 OIV Spiral Cycle 2 – Alternative identification
Table 3.10 OIV Spiral Cycle 2 – Evaluation & Elaboration
Table 3.11 OIV Spiral Cycle 3 – Alternative identification
Table 3.12 OIV Spiral Cycle 3 – Evaluation & Elaboration
Table 4.1 COCOTS Glue Code sub-model cost drivers
Table 4.2 Delphi Responses for Size Mapping (Size: KSLOC)
Table 4.3 Mapping between cost factor rating to risk potential rating
Table 4.4 Assignment of Risk Probability Levels
Table 4.5 Risk mitigation advices
Table 4.6 Comparison of USC e-services and industry projects
Table 5.1 Steps of Risk Based Prioritization Strategy
Table 6.1 Comparison of 2004 groups A and B on number of defects
Table 6.2 Comparison of 2004 groups A and B on client satisfaction
Table 6.3 Reported Benefits from the Usage of the Framework
Table 6.4 USC e-services non-CBA projects in Fall 2005
Table 6.5 Comparison of 2005 group A and B project profiles
Table 6.6 Evaluation of COCOTS Risk Analyzer
Table 6.7 Comparison of 2005 groups A and B on number of defects
Table 6.8 Comparison of 2005 groups A and B on client satisfaction
Table A.1 Data on CBA growth trend and CBA classification
Table A.2 Data on a1997 projects
Table A.3 Data on a1998 projects
Table A.4 Data on a1999 projects
Table A.5 Data on a2000 projects
Table A.6 Data on a2001 projects
Table A.7 Data on development effort (Year: 2001-2002)
Table A.8 Data on COTS Activity Sequences (Year: 2001-2002)
Table A.9 Data on COTS activity sequence
Table A.10 Data on COTS effort distribution in USC projects
Table B.1 Project Constraints Specification
Table B.2Business Use Case Description
Table B.3Prioritized System Capability Specification
Table B.4 Levels of Service Specification
Table B.5 Stakeholder Roles / Level of Service Concerns Relationship
Table B.6 Stakeholders and responsibilities
Table B.7 Examples of COTS assessment instruments
Table B.8 Examples of facilities description
Table B.9 Example of list of COTSs
Table B.10 Example of evaluation criteria
Table B.11 Example of evaluation criteria breakdowns
Table B.12 Example of evaluation screen matrix
Table B.13 Example of conclusion-recommendation pairs
List of Figures
Figure 1.1 Required Approach for COTS-Based Systems [Albert 02]
Figure 2.1 A Waterfall-Based CBD Process
Figure 2.2 EPIC Process
Figure 2.3The COCOTS Cost Estimation Model
Figure 3.1 CBA growth trend
Figure 3.2 Effort distribution of USC e-service projects
Figure 3.3 Effort distribution of COCOTS industry projects
Figure 3.4 Elaborated WinWin Spiral Model
Figure 3.5 CBA Process Decision Framework (PDF)
Figure 3.6 Assessment Process Element (APE)
Figure 3.7 Tailoring Process Element (TPE)
Figure 3.8 Glue Code Process Element (GPE)
Figure 4.1 COCOTS Risk Analyzer Workflow
Figure 4.2 Delphi results of risk situations
Figure 4.3 Productivity range of COCOTS cost factors
Figure 4.4Risk quantification equation
Figure 4.5 Input interface
Figure 4.6 Output interface
Figure 4.7 Evaluation results of USC e-services projects
Figure 4.8 Evaluation results of Industry projects
Figure 5.1 Different risk patterns result in different activity sequences
Figure 6.1 Comparison of COTS Impacts in two groups
Figure 6.2 Fall 2005 Experiment Setting
Figure 6.3 Experience with COTS activities
Figure 6.4 COTS activities performed in 2005
Figure 6.5 Activity improved in 2005
Figure B.1 Benefits Realization Approach Results Chain
Figure B.2 COTS assessment context diagram
Figure B.3Elaborated Spiral Model for COTS Assessment Project
Abstract
Research experience has suggested that software processes should be thought of as a kind of software, which can be developed into composable component pieces that can be executed to perform different software lifecycle activities. General experience has indicated that the activities conducted while developing COTS-based applications (CBA)differ greatly from those conducted in traditional custom development. The primary research questions addressed in this dissertation are (1) Can these activity differences be characterized and statistically analyzed? (2) If so, can the primary CBA activity classes be organized into a decision framework for projects developing CBA’s? The resulting research provides a value-based composable set of processes for CBAs that includes an associated Process Decision Framework (PDF), a set of Composable Process Elements (CPEs), and a COCOTS Risk Analyzer.
A composable process implies the ability to construct a specific process from a higher level and broader process framework and a set of reusable process elements. The PDF is a recursive, re-entrant configuration structure, and establishes the relationships among a mix of the CPEs and other process fragments which are extended from the risk-driven WinWin Spiral model. The CPEs includes Assessment, Tailoring, and Glue code development/integration, which are the three primary sources of effort due to CBA development considerations, indicated by empirical analysis on both large industry and small campus e-services CBA projects. Each CPE is a defined, repeatable workflow. While the frameworkprovides a composition basis to support developers for navigating through the option space in developing CBAs, the three process elements establish the basic constituents for composing COTS processes based on common process patterns identified in empirical studies. A technique named COCOTS Risk Analyzer, is also developed and implemented to aid the optimization of process decisions via risk based prioritization strategy. All together, the proposed solution supports flexible composition of process elements with respect to evolving stakeholders’ value propositions, COTS market, and risk considerations.
To validate the value-based set of processes, experiments have been designed and performed on student projects at USC graduate level software engineering class in Fall 2004 and Fall 2005 semesters. The evaluation results show that applying the value-based processes significantly improves the team performance.
1
Chapter 1Introduction
1.1 Motivation
There is no Silver Bullet. [Brooks 86]
Economic imperatives are inexorably changing the nature of software development processes. For software organizations under competitive pressure, developing applications from Commercial-Off-The-Shelf (COTS) products has recently become a widely adapted approach to reduce delivery time and reduce development cost through reuse. Meanwhile, software processes are increasingly moving away from conventional processes to compose pure-custom software from lines of code, and toward processes for assessment, tailoring and integration of COTS or other reusable components. The primary economic drivers of this change are:
- The increasing criticality of software capabilities to a product’s competitive success;
- The ability of COTS or other reusable components to significantly reduce a product’s cost and development time;
- An increasing number of COTS products available to provide needed user and infrastructure capabilities.
However, it did not turn out in practice that COTS is a “silver bullet”. A great deal of experience on the relative advantages and disadvantages of COTS solutions have been summarized and reported [IBM 77, NTIS 80, NASA 80, ICP 80, Auerbach 80, Datapro 80, Boehm 81, Garlan 95, Oberndorf 97, Vigder 98, Voas 98, Abts 01]. From a business perspective, the issues involve the short-term and long-term costs, benefits, evolution and associated risks of using COTS products. Moreover, practitioners are also finding that building systems using COTS products requires new skills, knowledge, and abilities; changed roles and responsibilities; and different processes [SAB 00]. This has led to a consensus that the development and maintenance processes for COTS-based software systems are significantly different from custom-built software systems, and have presented many challenges to software developers. Some of these challenges are:
- The marketplace is characterized by a vast array of products and product claims;
- Extreme quality and capability differences exist between products, and
- There exist many product incompatibilities, even when they purport to adhere to the same standards. [Oberndorf 97]
Itisan increasing awareness that without a well thought out process, many CBA projects will be unsuccessful in meeting their objectives. little chance that a project will be completed. In order to more effectively use COTS products in software development, a well-defined process methodology is required to facilitate the exploitation of COTS advantages and the avoidance of development pitfalls. The motivation behind this research is to make an advance toward organizing software processes into a COTS Process Decision Framework, which can provide effective process decision guidance with respect to project cost estimates as well as risk assessment,to determine how one can realize the cost saving and schedule reduction objectives expected of using COTS; and eventually, to help produce more reliable, higher quality software systems successfully from COTS products.
1.2 Nature of the problem
There is a consensus that special characteristics of COTS software have a pervasive impact on software lifecycle activities and requires software process adjustments and accommodations [Fox 97, Carney 98, Brownsword 98, Vigder 98]. Table 1.1 examines both positive and negative aspects of COTS characteristics, the impact to software lifecycle activities, and required accommodations.
Table 1.1 COTS characteristics and impacts
Characteristics / Description / ImpactsPositive / Affordability / COTS integration can lead toward savings. / Cost analysis
Timeliness / COTS packages are immediately available. / Planning, Requirements
Tailorability / COTS architecture is set for a general domain, and interaction mechanisms are provided for specific customization. / Design, Integration
Negative / Opacity / No white box testing. / Test
Incompatibility / COTS product may not be compatible with exact functionality or other COTS. / Requirements,
Integration, Cost analysis
Supportability / COTS vendor may no longer exist or no longer provide support. / Integration, Maintenance
Uncontrollability / The user has no influence on COTS evolution. / Maintenance
Both / Proliferation / Large numbers of similar COTS are available. / Requirements,
Design
Dynamism / COTS products, vendors, and marketplace are dynamically changing. / Whole life cycle
A software process has been defined as a coherent set of activities and associated results for software production [Sommerville 00]. A number of general software process models such as the Waterfall model [Royce 70] and the Spiral model [Boehm 88] have been widely applied in traditional systems development. However, processes based on these models have met considerable difficulties when applied to CBD, due to the significant CBD impacts to software life cycle activities as listed in Table 1.1 above.
Figure 1.1Required Approach for COTS-Based Systems [Albert 02]
The traditional software processes present a common limitation: most assume the system being built will be coded largely from scratch [Fox 97]. Not surprisingly, this limitation has caused many considerable difficulties and lessons learned within CBD, such as those reported in [Garlan 95, Braun 99, Albert 00, Fox 00, Lewis 01]. In such cases, the potential COTS benefits may be limited significantly and most likely even turn into adverse effects to the system, which if not appropriately handled, could lead a seemingly simple project into a disaster. In the few successful cases reported, the use of COTS products often involves delay and inefficiencies [Sledge 98], excessive integration effort more than expected [Medvidovic 97, Swanson 97, Maxey 03], and significant amount of maintenance effort [Balk 00].
Traditional sequential requirements-design-code-test (waterfall) processes do not work for CBD [Benguria 02], simply because there is a critical relationship among COTS product selection, requirement specification, and architecture design in the front-end. The decision to use a COTS product constitutes acceptance of many of the requirements that led to the product, and to its design and implementation. This leads to a concurrent engineering of requirement, architecture, and COTS selection process, as shown in Figure 1.1 [Albert 02]. Though some Bottom-up design and Object-Oriented (OO) design methods, to some degree, facilitate the maximum use of proven COTS in analysis and design phases by viewing the system as a set of components, the opacity, incompatibility, and volatility of COTS products [Basili 01] often inevitably introduce a great deal of backtrack and retry of early activities in a later phase. These result in high degree of recursion and concurrency of the back-end CBD processes which should be carefully planned and managed through more effective processes and risk management techniques. Additionally, there are a number of new emerging, nontypical activities requiring changes to conventional software processes, such as COTS technology and product evaluation, adaptation, and integration [Brownsword 98].
Meanwhile, as the fraction of CBD projects has increased among USC e-services projects, the developers have encountered increasing conflict between CBD process needs and the USC’s MBASE process and documentation guidelines [Boehm 99]. This has led to a good deal of confusion, frustrating re-work, risky decisions, and a few less than satisfactory products [Boehm 02]. This was evidenced by observing numerous teams succumbing to effort allocation pitfalls. For example performing too much COTS assessment and neglecting to allocate enough time for training, tailoring and integration resulting in project delays, inability to implement desired functional capabilities, and so forth. In some cases some developers did not perform enough assessment, which resulted in problems and delays during construction phases. Such losses may be minor when it comes to implementing student projects as a part of course work, however in case of large scale systems and systems requiring to meet a ‘specific window of opportunity’, such losses may be extremely significant.