The Development Risk Premium of IS Projects

And

Consequences for IS Management

By Douglas W. Hubbard 4/3/97

Simplified Version (non-mathematical)

Abstract

We consider the investment in the typical software development project from a risk/return point of view. Specifically, reported cancellation rates and the uncertainties of cost estimates (which we will call the development risk) are considered against the stated risk/return preferences of IT decision by applying the Modern Portfolio Theory (MPT) framework. Even considering only these risks, the result is that risk-adjusted ROI requirements are considerably higher than the typical “hurdle rates” used by IT decision makers (in the range of 15% to 30%). This effect increases rapidly as the size of proposed projects increase. The typical IT decision maker in the average development environment should require a return of well over 100% for the largest projects in the IT portfolio. Many other factors - the uncertainties of benefits, the risk of interference with operations and the undiversifiable “systemic” risk of projects - could be added to this analysis that would “raise the bar” even further. These findings have many consequences for IT decision makers and some of them will be presented here.

Outline

It is not too bold a statement to say that the software development project is one of the riskiest investments a business makes. The chance of a large software project being canceled approaches 50% for the largest projects and software cost estimates are notoriously inaccurate.

Yet, most companies that use ROI analysis do not account for this risk. The typical “hurdle rates” are not adjusted for differences in the risk of IT projects even though risk should be a huge factor in the decision. If the decision maker looked at the software development investment from a risk/return point of view he/she will probably make some very different decisions than would be made with fixed hurdle rates.

Our objective is to show how different some IS decisions would be if risk were considered a factor in the decision. This argument follows this outline:

  1. The risk/return profiles of IT decision makers. Here we show the results of a survey where some decision makers are asked to specify what risks (stated as a probability of a negative return) are acceptable for a given expected return for their entire IT portfolio.
  1. The development risk of IT investments. We review data from Software Productivity Research Inc. that shows cancellation rates and accuracy of cost estimates for over 6,700 IS projects. The risk of cancellation before implementation and the risk of cost overrun are collectively labeled “development risk”. Even without considering uncertainties in other areas (like benefits) this risk is significant.
  1. The MPT model. A non-mathematical overview of Modern Portfolio Theory and its application to this problem is shown.
  1. Summary of Results. The required returns - given the stated risk/return preferences of IT decision makers and the known development risks of IT projects - are shown for projects of various sizes and for investors of various risk aversions.
  1. Consequences for decision makers. The typical hurdle rates are far too low to represent the expected return that would compensate decision makers for the risk involved in IT investments. Several possible risk management strategies are discussed that should help the decision maker manage IT investment risk.

The Risk/Return Profiles Of IT Decision Makers

Thirty-nine individuals went through a one-day seminar called “The Risk/Return Analysis of IS Projects”. This seminar included training in the identification of an individual decision makers “investment boundary”. As in investment theory and utility theory, this is expressed as a function which shows the acceptable risk for a given expected return. These individuals were given a questionnaire to fill out regarding their risk and return preferences.

For the purposes of the survey we defined the “Return” as the average (expected) 5 year internal rate of return (IRR). The “Risk” was defined as the chance of a negative return. A proposed portfolio with a risk of 10% and a return of 50% means that there is a probability distribution of returns a 50% with a 10% chance of a negative return.

After having been trained in these concepts, the participants were asked to fill out a survey to assess their “Risk/Return Profiles”. The participants were asked to estimate the total work-months of current IS development commitments. This gave us an idea of the total size of the IT portfolios that they influenced at their firms. They were also asked for the size in work-months of the single largest project among current development commitments. This would be used to give us some indication of the distribution of project sizes in their IT portfolios. They were then shown the following matrix:

Figure 1. The Risk/Return Preference Matrix in the Survey

In each cell, they could enter how certain they were that they would take the investment. A value of “Y” meant “Yes, I’m fairly sure I would accept this portfolio” while a value of “N” meant “No, this portfolio is a little too risky”. Also a “YY” meant “Yes, I would definitely accept this portfolio” and “NN” meant “No, I definitely would not accept this portfolio. They could also enter a “?” if they were uncertain as to whether that was an acceptable portfolio to them.

17 of the 39 individuals were identified as having significant influence regarding the “IT portfolio” of their firm. Only the responses of these individuals are used in the following summary of results.

The results showed wide variation in all aspects of the investment boundary. People differed on the Risk-free return (the minimum return they would accept even if there is no risk), their maximum acceptable risk and the rate at which they approached maximum risk as they moved past their point of risk-free return. We used a formula that roughly approximates the 10 and 90 percentile bounds of the investment boundary points as well as the median. The sample size of 17 is small but we can still state with statistical confidence that most managers and executive of IT will probably fall between the upper and lower bounds. The following graph depicts some of the actual responses and the approximated upper, median, and lower bounds on a logarithmic scale.

Figure 2. The Risk/Return Boundaries of IT Decision Makers

From the graph we can see that most IT decision-makers would accept a 10% chance of a negative return (on the entire IT portfolio) if they knew that the expected return (the weighted average of all outcomes) was about 200%. On the other hand, some decision makers would accept the same risk for only a 30% expected return. Furthermore, almost anyone would accept a 1% chance of a negative return for an expected return of over 30%.

Participants also responded that their total current commitment in IT projects varied from 100 work months to over 10,000 work months - the largest being Fortune 500 companies. This roughly represents the relative sizes of their total IT portfolio. With this sample size, we were not able to conclude that larger firms are consistently any more or less risk averse than smaller firms. Participants also responded that the largest project in their IT portfolios comprised anywhere from 25% to 100% of the portfolio - meaning that a large part of most portfolios are one or two large projects.

The Risk Of IT Investments

The data regarding the risk of cancellation and the risk of uncertain costs are derived from data provided by Capers Jones of the Software Productivity Institute in a book titled “Applied Software Measurement”. The risks in that book are generally stated in terms of “Function Points” but we convert the data to work-months since many firms will not be familiar with the function point methods.

We apply a conservative assumption that the firm is a moderately effective user of function points or some other software estimating method. According to the data collected by Jones, this still implies an uncertainty that the estimate will be within a factor of 1.5 of the actual development costs 90% of the time. This means that there is still a 10% chance that the estimate could be off by even more than that. This is an average of a few different estimating methods but it should be representative of the uncertainty of a typical metric-oriented development shop. Some methods claim better estimating records (including Jones’ own function point method) but even if it is possible to measure more accurately on a consistent basis it is safe to assume that most firms will not have this level of proficiency.

Not only is there a danger that the development project will cost much more than expected but there is also a danger that the project will simply be canceled before it is ever implemented. Often, quite a bit of unrecoverable investment has been made in the project before cancellation. Jones does not report this data but a variety of sources put the amount of the lost investment at somewhere between 60% and 120% of the originally budgeted amount.

The chance that a project will be canceled is closely related to the size of the project. Figure 2 shows how the chance of cancellation increases with project size.

Figure 3. The Cancellation Rates of Software Projects

Considered together, the risk of cost overrun and the risk of cancellation are what we term the “Development Risk” of a software project. The remainder of this study will show the effects this risk alone (not including risk of uncertain benefits, etc.) have on the required return of software development investments.

An Overview of Modern Portfolio Theory

At this point, we have the “development risk” of a project and the plot of acceptable risks for a given return from various IT decision makers. Now we want to answer the following question: “What should my minimum required return be for a project of this size?”. We do this by first calculating the development risk of a project given the chance of cancellation and the uncertainty of cost estimates. Once we have the development risk we determine what return is required from an IT decision maker in order to accept that level of risk (remember, we have defined risk as the chance of a negative return for this study).

This set of calculations is not entirely obvious. We need to consider how cancellations and cost overruns affect the risk and return of the entire “portfolio” of IT investments. Fortunately, there is a mature and rigorous method to perform this calculation. This method is called Modern Portfolio Theory (MPT).

MPT began in the 1950’s as a way of “optimizing” investment portfolios on a risk and return basis. This method covers a lot of ground, part of which has been superseded by more current quantitative models. However, the relevant part of MPT for this study is still completely valid (in fact, it is provably correct). MPT uses a simple technique from statistics to give us the basis for calculating the risk and return of a portfolio given the risk and return of individual investments in the portfolio.

Summary Of Results

We find that if IT decision makers actually state their risk/return preferences and compare the to the development risk of IT projects that they should be requiring returns far above what they probably require for software projects. Typically, a firm states the minimum required return to approve a project as a fixed “hurdle rate” somewhere between 15% and 30%. This means that an IT decision maker generally would expect a calculated return of 15% to 30% in order to approve a software development project.

Unfortunately, these hurdle rates are not adjusted for risk. We find that when risk is considered minimum returns will be much larger than the typical hurdle rates. Furthermore, since the risk increases rapidly with project size so does the required minimum return. The largest project in a portfolio (typically representing at least half of the cost of a portfolio) will have enough risk to require returns of over 100% for most IT decision makers. Figure 4 shows the minimum return of a software development project given the size of the project and the amount of risk aversions of the decision maker. For this graph we calculated the required returns assuming a proposed IT “portfolio” with a total of 5000 work-months committed to all projects.

Figure 4. Required Returns for IT Projects Given the Risk Aversion of the IT Decision Maker (calculated with a portfolio size of 5000 work-months)

The introduction of other significant factors - such as the uncertainty of benefits - would only raise these minimum returns even higher. In fact, the uncertainty about magnitude of the benefits from the proposed software may often be much higher than the development risk alone. Even a risk tolerant IT manager may find that returns of over 100% are necessary to compensate for the risk of the largest software projects.

Consequences for IT Decision Makers

Obviously, the criterion of using a fixed and relatively small hurdle rate to evaluate software development projects simply does not suffice. It does not change as the risk of the project changes and the risk of the project changes a lot with size. A risk-sensitive criteria is much more appropriate.

Many executives do not rely on fixed hurdle rates to evaluate proposed software projects, anyway. This may, in part, be due to their intuitive sense that the level of risk is not justified simply by surpassing the hurdle rate. Unfortunately, this often results in results in a complete dismissal of any attempt to calculate ROI instead of finding ways to calculate returns and apply a more meaningful risk-adjusted criteria.

If risk-adjusted methods are applied IT executives will find that many of the types of projects they approved in the past are no longer acceptable. There will be a greater incentive to reduce risk by focusing on smaller projects that have a higher chance of being delivered successfully. They will tend to defer major projects until after they have developed the capability of their development staff (increasing the chance of success) or until major organizational changes (one source of cancellations) stabilize. They will be forced to consider how large projects can be broken up into smaller projects that develop incrementally improved versions. Executives should present the risk-adjusted evaluation of the software to the users and developers so that runaway “gold plating” of software will be discouraged more than perhaps “increased cost” arguments alone can do.

Perhaps developing software internally will be less attractive compared to “less perfectly fitted” but working packaged software. Executives may begin to lean toward “hedging” against loss by asking software vendors and consulting firms to assume more of the risk with fixed bids.

Another risk reducing strategy that this may encourage is to seek to reduce uncertainty through further measurement of unknown quantities. The value of better software estimating is magnified when looked at from this point of view.

In conclusion, IT investments are among the riskiest investments a business can make and, therefore, risk must be a significant factor in the analysis. This approach to calculating and adjusting for risk is based on the same methods that any financial portfolio manager may use. IT executives may find that there are already people within their own firm outside of IT that have defined risk/return profiles and use MPT methods. It will only become more critical to incorporate these standard business quantitative methods into IT as risks increase form growing investments in software, increasing complexity of applications and more rapid changing of organizations.