-9:58 AM04.10.2018
Survey on Risk Management in OTS-based Development in IT Industries in Several Countries
(Draft Design Version 1.15, 24 May 2004)
Reidar Conradi1,2, Jingyue Li1, Odd Petter N. Slyngstad1,
Vigdis By Kampenes2, Dag Sjøberg2, Maurizio Morisio3, Macro Torchiano3, and Christian Bunse4
1NorwegianUniversity of Science and Technology (NTNU)
NO-7491 Trondheim, Norway
Tel +47 73.593444(rc), Fax +47 73.594466,
{conradi, jingyue, oslyngst}@idi.ntnu.no
2Simula Research Laboratory
P.O.Box 134, 1325 Lysaker, Norway
3Politecnico Di Torino
4Fraunhofer-Institute for Experimental Software Engineering (IESE)
Preliminary abstract
This report describes an initial version of the empirical design on risk management in COTS-based development in IT industry in several countries (Norway, Germany, and Italy). The initial work of this research was prompted by recent papers e.g. by Morisio et al. that seem to indicate new trends in using e.g. COTS components. We designed several hypotheses and did a prestudy in 16 projects in Norway from Oct. 2003 to Feb. 2004 by structured interview. The results of the prestudy showed that some of the hypotheses can be tested and some of them are not easy to test. The results of the open questions in the questionnaire also showed some new research questions and theses. We discovered that all the hypotheses used in the prestudy and the new theses found from the prestudy are relevant to risk-management in COTS-based development. We therefore change the research topic to risk management issues and combine all the research questions in this framework. After a literature survey on risk management in COTS-based development, we also found some other interesting research questions. This empirical study is designed to investigate all these research questions. We plan to design a questionnaire and use the SESE web tool of the Simula Research Lab to collect the data.
Change Log:
Version 1.1: Added the research question relevant to build vs. buy decision;
Merged RQ2 into RQ3 and RQ4;
Made a clear fate between the prestudy and this design
Version 1.2: Revised the project level andOTS componentlevel risks
Completed risk reduction questions
Added analysis methods
Version 1.3: Added software evolution questions to the questionnaire
(also see appendix D)
English language checks & corrections (needs crosscheck)
Version 1.4: Integrated software evolution material from Appendix D in Ver1.3 into the
document, removed Appendix D.
Version 1.5: Add OSS relevant questions based on discussion with Macro.
Merge the questions of evolution and moved them into a separate part.
Version 1.6: Change the definition in the first page of the questionnaire.
Change the direction of the survey from COTS to OTS.
Version 1.7: Simplified questions on evolution and maintenance in section 6 of the
Questionnaire. - Reidar: Added v1.7 and 31.3 in line three.
Version 1.8: Reidar: new file name as cbse-survey-v1.8-2apr04.doc,
reverted many OTS terms to COTS:Have taken abstract and chs. 1-2 from
V1.5, chs. 3-4 from V1.7, References from V1.5, Questionnaire and
Appendices from V1.7 where I also have added hypotheses resume from
3 Feb.04 meeting. Revised questionnaire (two new pages in start, plus lots
of smaller changes) and Appendix B.
Version 1.9, 5apr04: Reidar: Lots of minor changes!
Please recheck questionnaire text vs. design
Version 1.10, 13apr04: Section 4.3 revised, the hypotheses vs. questionnaire questions have been coordinated
Question 2.2 has been added to answer some OSS research question (see chapter 3.2.1)
Question 4.2 is dropped, alternative 4.1.a and 4.1.b are added
Question 5.1 is divided into question 5.1, 5.2 and 5.3
Invitation letter is added in the first page of the questionnaire
Evolution question 6.1 – 6.4 have been revised
Version 1.11, 19apr04:
Change the definition of OTS component because Open Source Component may not be free.
Added some examples of OSS components
Question 1.2.6, 1.2.7 were deleted. Question 5.1 was extended so that respondents can fill in all components used in their project and their relative importance.
Question 2.3 was added to ask the motivation of using COTS component
Question 5.3 was modified so that the OTS component type follows the categories in ComponentSource (The biggest component market in the world).
Question 5.7 was revised to make the alternative a) and b) more flexible.
Version 1.12, 26apr04:
Based on the comments we got from Tore Dybå and Christian Bunse, we changed the version 1.11 a lot. The main purpose was to increase the construct validity of the questionnaire. We current changed only the data analysis chapter and questionnaire. The rest of this design will be changed later.
Question 1.2.5: Alternative “New functionality” was changed to easy to add new functionality. Alternative “Improved functionality” was changed to maintainability.
Question 2.1: Alternative “Better overall system quality” was divided into three separate alternative to ask for “overall system reliability”, “overall system security”, and “overall system performance”.
Question 2.3: Alternative “Reduction of the risk of security because source code is not open” was added.
Question 3.4: deleted because the same question was asked in question 5.7
Question 4.1: changed dramatically. Several new alternatives were added. The alternatives were clustered into different categories of risks (Need future discussion).
Question 4.2: changed dramatically. Some new risk reduction methods were added and they were clustered into 6 categories.
Question 5.1: The version of the OTS component was asked. We asked the respond to fill in one of the most prominent component.
Question 5.2: It was separated into 5.2, 5.3, and 5.4. The scale was changed into five-scale instead of Yes/NO. The number of the component was reduced into one.
Question 5.3: changed into question 5.5. The classification was the same as ComponentSource.
Question 5.4: changed to 5.6.The number of the component was reduced into one.
Question 5.5: changed to 5.7.The number of the component was reduced into one. The alternative Prestudy/Requirements/Analysis phase was divided into two.
Question 5.6: changed to 5.8. The number of the component was reduced into one. The scale was changed to five-scale.
Question 5.7: changed to 5.9. The number of the component was reduced into one. The scale was changed to five-scale. The alternative b) and c) were merged into d). alternatives g) h) i) were moved to question 4.2.
Question 5.10 was added to investigate the effect of the OTS component
Question 6.1: changed from pre-defined scale to the format that the respondent can fill in the exact number.
Question 6.5: changed from pre-defined scale to the format that the respondent can fill in the exact number.
Question 7.1.6 was added. Previous 7.1.6 moved to 7.1.7. Previous 7.1.7 moved to 7.1.8
Data analysis methods were revised:
Hypothesis New Hx2, H10, and possible PHx5 were revised. The variable will be measured by using several items. The reliability and construct validity testing methods were proposed.
Version 1.13 Deleted section 6 (on maintenance/evolution)
Deleted all meta-questions
Deleted all risk relevant hits in question 4.1,4.2,5.8.
Question 1.2.3:Added full description of terms in
Question 2.2 Added alternative f)
Question 2.3 Added alternative e)
Question 3.2:Changed to multiple-selection
Question 4.1: One alternative was added, i.e., the risk of moving from
development environment to production environment
Alternative d) e) f) were revised so that they are relevant to
OTS components
Alternative a,b, c, k, and l were revised to use negative words
Question 4.2: Alternative a) change words to make it clear
Alternative k) was deleted
Question 5.2: Changed format
Question 5.3,5.4: Changed alternatives “hardly ever” to “never”.
Question 5.8, 5.9: Alternatives have been changed from 5-scales to Yes/No
Deleted the risk related words in the question
Other changes after discussion with Reidar Friday morning (21May)
Definition:
OTS component definition was changed based on comments from Macro
Component definition was changed
Changed “executable unit” to “program unit”
Changed “built in house” to “built from scratch”
Platform definition was changed based on comments from Macro
Question 1.2.2: changed to less categories
Question 2.2: added alternatives “It was decided by customer” and “political reasons”
Question 2.2 and 2.3: changed all the term “vendor” to “provider”
Question 4.3: changed the “background” to “role and OTS component relevant experience”.
Question 5.1: slim so that only 5 component information was required
Open issues
Delete section 6 or not? Arguments:
most information of the company can be acquired before the interview start (by company list and internet)
The personal information of the respond can be acquired in the contacting procedure by calling them.
Some respondents don’t want to give personal information by filling in them into the questionnaire
Version 1.14 Revised invitation letter and term definition (Reidar)
The design and hypotheses have been cross checked with the questions
(Possible PHx1 was deleted because it was not possible to be tested and not
question can match it.)
Question 1.2.6 was moved to 1.1.2
1.1.2 to 1.1.3
1.1.3 to 1.1.4
1.1.4 to 1.1.5
1.2.2 was changed to 1.2.3 (Include completed list again because it is
relevant to possible hypotheses PHx2 and PHx3)
1.2.3 to 1.2.4
1.2.4 to 1.2.5
1.2.5 to 1.2.2 (format reason)
2.1 – 2.3 question revised (past tense )
4.1 alternatives i,o,q were deleted to slim the question
4.2 slim the alternatives
4.1 alternative s and 4.2 alternative l were deleted because it is not
possible to answer others based on the question.
5.1 was added to know the whole number of different OTS components
5.1 to 5.2 deleted the last column, only importance of comp. 1 was asked
(see 5.3)
5.2 to 5.4
5.4 to 5.6
5.5 slimmed and changed to 5.7
5.6 to 5.8
5.7 to 5.9
5.8 to 5.10
5.9 to 5.11
5.10 to 5.12
Section 6.2 was deleted because all information can be gathered from
other resource
Open issues: 2.2 – 2.3 what is the meaning of political reasons?
V1.15, 24.5: Reidar - Polished initial definitions a bit more.
Fixed table-of-contents, reinserted 6.2 for the time being.
Table of Contents
1. Research rationale
1.1 Previous studies
1.2 Research motivation
1.3 Literature review
2. Results of the prestudy
3. Research Design
3.1 General design
3.2 Detailed design
3.2.1 RQ1: Motivation of using COTS/OSS components instead of building in-house. (Exploratory study)
3.2.2 RQ2: Actual OTS-based process (Descriptive study)
3.2.3 RQ3: Project level risks (Descriptive study)
3.2.4 RQ4: OTS component-level risks (Descriptive study)
4. Questionnaire design
4.1 General questionnaire design
4.1.1The project background
4.1.2Motivation of using OTS components
4.1.3The project process
4.1.4Project level risks
4.1.5Project level risks management
4.1.6The OTS components background
4.1.7OTS component-level risks
4.1.8OTS component-level risk reduction methods
4.1.9The background information of the respondent
4.2 Data analysis methods
References:
Appendix A. Final questionnaire
Appendix B. Summary of hypotheses and theses in the prestudy
1. Research rationale
1.1 Previous studies
The prestudy was prompted by recent papers e.g. by Morisio et al. [Torchiano04] that seem to indicate new trends in using e.g. COTS components. We have proposed 10 hypotheses based on the theses proposed in that paper. We designed a questionnaire which includes both closed and open questions. We did personal interviews based on this questionnaire in 16 projects in 13 companies in Norway.
The results of the closed questions show that some of the hypotheses are possible to be tested quantitatively by web-survey, but some of the others are not easy to test quantitatively. The results are summarized in the internal report of the prestudy [Conradi04].
One result of the open questions we discovered is a revised COTS-based development process model in the following graph. From these open questions we also found some new theses and research questions.
The revised common COTS-based development process has four phases,P1-P4:
- P1: Decide the whole development process (waterfall, prototyping, etc.) first.
- P2: In one phase of the development process (requirement, design, or implementation), developers will decide whether they should buy a COTS component from a third-party company or should build the component themselves. When to decide this issue is different based on the whole development process and other factors [Li04b]. In this phase, the main issue is to identify and evaluate the possible risks between build and buy.
- P3: After the developers decided to buy COTS components from the market, they should start to perform some risk reduction (mitigation) based on the risks they identified. The next step is to search for and evaluate the COTS components. When to select the COTS components is also different based on the developers experience with various COTS components [Li04b].
- P4: After the COTS components have been selected. The next step is to integrate them into the system.
This process is different from the COTS-based development process proposed by Boehm et al. [Boehm99]. Their spiral model proposed to discover the risk first and decide the waterfall or prototyping process later based on the possible risks [Boehm88]. Our process proposes that the main processes are not decided by the COTS-related risks. They may be decided by other possible risks first. After the project manager has decided to use COTS components, the process may start to be COTS-related risk driven.
This process is similar to that proposed by Morisio et al. [Morisio00]. The difference is that our results showed several possible variances [Li04b]. Morisio et al. pointed at some possible risks in COTS-based development. Our results show that there are two levels of risks in COTS-based development and each level may need a different kind of risk reduction strategies. Before the developers decided to buy COTS components instead of building them in-house, they must identify and compare the possible risks between the two alternatives based on their project context. For example, they need to check if they have such prior experience, and if their requirements are flexible enough etc. After they have decided to buy COTS components, some project level risk reduction should be done immediately, such as re-organizing the group to include experienced personnel. In the COTS componentselection procedure, they also need to do some risk management based on the COTS componentcharacteristics. For example, if they have no previous experience with a specific COTS component, they may need to spend a lot of time on understanding and testing it. On the other hand, they may not need much effort on this issue if they have used the COTS components several times before.
1.2 Research motivation
The purpose of this study includes two parts:
Further investigation of the hypotheses we tested in the prestudy
To study the above process model and related risks management issues we discovered from the prestudy.
From the comments of we got from colleagues, we found that the main limitation of our current study is that those hypotheses we tested in the prestudy are very independent, and some of them have context validity problems because they look very obvious. However, after a careful analysis of these hypotheses and the new findings we gathered from the results of the open questions, we found that all of them are relevant to one common issue: The risk management in the revised COTS-based process model we mentioned above. We therefore decided to merge the two research purposes under the framework of risk management in the COTS-based development process.
1.3 Literature review
We did a literature review of the previous studies on the risk management issues in COTS-based development processes.
Most previous studies are case studies that gathered a lot of different risks [Rose03], [Kotonya01]. These studies proposed risks and strategies on how to resolve them.
- The first limitation is that most of them mixed COTS componentlevel and project level risks together.
- The second limitation is that there is no systematic summary on COTS componentcharacteristics and project contexts of these risks and risk management methods. Their risk reduction strategies are case by case. If the developer just follows their suggestions, it may not be a cost-effective way, for example, to use a very formal evaluation process to evaluate a very familiar and unimportant COTS component.
- Another limitation is that there are too many proposed COTS component related risks and the inter-relationship between these risks is not clear. That two or more risks have an inter-relationship means that the occurrence of one risk might trigger the occurrence of one or several other risks,which might not otherwise occur.It is therefore hard for the developers to catch the real risks and make a proper risk reduction plan. There is no factor analysis to see if some of them have a real inter-relationship.
The most influential study in this field was made by Boehm in the COCOTS project [Abts00] [COCOTS00]:
- The first limitation of their research is that they assumed that every process using COTS components should start with risk analysis. However, our previous result concluded that the risk-driven process starts only after the project members have decided to use a COTS component instead of building the component themselves.
- The second limitation is that they mixed the COTS componentlevel and project level risks together and calculated the possible cost by mixing them together. It does not look reasonable from our prestudy results, because project level risks may have more influence, since they are more general than COTS componentlevel risks, which are more specific.
- Another limitation is that they didn’t touch the risk reduction issues or analyze the reduction strategies to see if they really worked.
- Lastly, the empirical base mostly cover large companies in aerospace and defense,
A major inspiration for this survey is the study by Torchiano and Morisio [Torchiano04]. They interviewed seven small and medium sized software companies in Norway and Italy. Their results are summarizedin six theses, and challenge many “assumed” truths about COTS-based development:
T1: Open source software is often used as closed source.
T2:Integration problems result from lack of compliance with standards; architectural mismatches constitute a secondary issue.
T3: Custom code mainly provides additional functionalities (i.e. addware, not glueware).
T4: Developers seldom use formal selection procedures. Familiarity with either the product or the generic architecture is the leading factor in selection.
T5: Architecture is more important than requirements for product selection.
T6: Integrators tend to influence the vendor on product evolution whenever possible.
However, the study covers few respondents, and there is a need for a more