Part a Major Trends in Technology and Information Systems

ROI Analysis 1

ROI Analysis of Software Process Improvement, Online Education, and Telecommuting
David F. Rico


Return on investment or ROI is a widely used approach to measure the economic value of an investment or portfolio of investments, capital improvements, and even organization change. ROI is increasingly used to measure the economic value of software process improvement or SPI, online education or asynchronous learning networks, and telecommuting or teleworking. However, the ROI of software process improvement, online education, and telecommuting is not well known, along with appropriate ways of estimating the economic value of these paradigms. Software process improvement, online education, and telecommuting are major trends, which affect the economy, industry, and society as a whole, signifying the end of the industrial age. Therefore, this paper examines the ROI of software process improvement, online education, and telecommuting, and the hard economic evidence behind these major transformational trends. Additionally, this paper briefly explains return on investment, and provides an in-depth analysis of the management challenges and solutions for the ROI of software process improvement.

ROI is the ratio of adjusted economic benefits to costs, expressed as a percentage. That is, the economic benefits, less the costs, are divided by the costs, and multiplied by 100 percent. ROI is a measure of the magnitude of economic benefits to costs, economic benefits returned above costs, profits returned above expenses, or simply, the value of one or more investments. ROI is a standard tool for analyzing the costs and benefits of a portfolio of capital improvements. Costs, benefits, and ROI are usually monetized, and expressed in economic and monetary units. In other words, ROI is a measure of economic value, which is expressed in a relevant currency. Benefits, which are often based on esoteric statistical models, must be identified and monetized. Costs must be accumulated in painstaking detail to provide the most accurate economic picture. ROI is calculated by dividing the benefits less costs by the costs, and multiplying by 100 percent.

Part A—Major Trends in Technology and Information Systems

ROI of Software Process Improvement

Software process improvement is an outgrowth of the field of software engineering, which was created circa 1968. Software engineering is the professionalization of computer science, and is aimed at successfully managing and engineering large scale, software-intensive systems. Software process improvement is the discipline of perfecting the processes, people, and technology associated with the field of software engineering. There are an infinite variety of software engineering methods, as well as an infinite variety of software process improvement approaches. Since 1968, a small body of literature quantifying the costs, benefits, and ROI of software process improvement methods has been gradually emerging. Usually, a management scientist creates a new software process improvement approach, and then touts its benefits. Rarely, are systematic benefits identified, categorized, and scientifically investigated. And, even more seldom, are software process improvement benefits monetized, or converted to currency, and expressed in terms of ROI. And, few studies exist, which systematically compare the ROI of software process improvement methods, along with other quantitative and economic performance characteristics.

One of the first major studies of the costs, benefits, and ROI of software process improvement was performed by McGibbon (1996). McGibbon’s study identified and compared the costs, benefits, and ROI of the software inspection process, software reuse, and the clean room methodology. The greatest contribution of McGibbon’s study was a technique to quantify and monetize the benefits of various software process improvement methods. He adapted the time proven method of total life cycle cost analysis to the field of software process improvement. In doing so, McGibbon unlocked the secrets of ROI analysis for software process improvement, and opened the door to ROI analysis for a variety of past, present, and future methods. Leaning heavily on detailed cost and benefit analyses, McGibbon’s study revealed an ROI of 7,162% for inspections, 391% for software reuse, and 3,267% for clean room.

About the time McGibbon’s (1996) work appeared, Grady (1997) published one of the first textbooks exhibiting a broad range of costs and benefits for software process improvement. He identified cost reductions of 6% for product definition improvement, 7.5% for detailed design methods, 8.5% for rapid prototyping, 8.5% for systems design improvements, 14% for inspections, and 22.5% for software reuse. He also identified cost reductions of 3.5% for complexity analysis, 6.5% for configuration management, 3.5% for certification processes, 4.5% for software asset management, and 5% for program understanding.

Rico (2000) extended McGibbon’s (1996) methodology in order to evaluate a broad range of software process improvement methods from the past, present, and future. This study examined the costs, benefits, and ROI of PSPsm, clean room, software reuse, defect prevention, inspections, software testing, SW-CMM®, and ISO 9001. Benefit/cost ratios included PSP—1,290:1, Cleanroom—27:1, Reuse—3:1, Defect Prevention—75:1, Inspection—133:1, Test—9:1, CMM—6:1, and ISO 9001—4:1.

Rico (2002) reframed earlier results using scholarly metrics and models for ROI. This study quantified costs and benefits in greater detail, and used conservative assumptions. This resulted in overly conservative ROI values of Inspection—3,555%, PSPsm—3,104%, TSPsm—$1,351%, SW-CMM®—1,330%, ISO 9001—739%, and CMMI®—425%. He implored his readers to pinpoint high ROI factors, target high ROI approaches, minimize cost incurrence, avoid cost intensive approaches, avoid training intensive approaches, look for low cost automated solutions, and use professional methods for analyzing ROI.

Rico (2003), once again, reframed his methods, this time using net present value to temper optimistic ROI values, and added detailed break even point estimates. ROI estimates plunged for Inspection—2,542%, PSPsm—3,217%, TSPsm—2,192%, SW-CMM®—661%, ISO 9001—158%, and CMMI®—114%. He simply said choose a simple set of metrics and models for SPI, don’t spend a lot of money to achieve a high ROI, don’t go bankrupt using expensive methods with a high ROI, and seek low cost automated SPI methods with a high ROI. He also said don’t wait until you’re SW-CMM®/CMMI® Level 5 to use ROI, don’t spend too much money measuring ROI, proactively plan for ROI, use ROI to right size a SPI program for your organization and overhead budget, and apply metrics and models for ROI before it’s too late.

Rico (2004) formulated recommendations for future software process improvement based on detailed cost and benefit analysis of existing software process improvement methods. The future methods included trainingless methods, automated workflow tools, low-cost commercial tools, non-invasive measurement, expert system workflow tools, self documenting approaches, and boundaryless virtual teams.

The impacts of applying the ROI of software process improvement include forcing managers to apply quantitative methods, and collect and analyze cost and benefit data. This implicitly and explicitly forces managers to study, learn, and apply scientific management principles. This often involves continuing education in management science. Managers are now faced with identifying alternative approaches and principles in scientific management. More importantly, managers are faced with comparing and contrasting the qualitative and quantitative costs and benefits of alternative methods in scientific management. Not only must managers learn and apply scientific management principles, but they must also study, master, and apply quantitative decision analysis methods for cost, benefit, and return on investment analysis.

ROI of Online Education

Online education is currently embodied by the use of the Internet, World Wide Web, and technology to enhance the instructional and learning interaction between teacher and student. Online education is also known as online learning, e-learning, asynchronous learning networks, distance education, learning technologies, online course delivery, telelearning, and other names. Early manifestations of technology assisted learning included phonographs, radio, tape recordings, television, videotapes, interactive video disks, mainframes, minicomputers, workstations, personal computers, computer based training, simulations, and video games. The objectives of technology assisted learning included cost effective educational delivery, just-in-time training, mass educational services, and remote educational services. Objectives also included consistency of service, quality, delivery, productivity, and optimal learning experience. Allen and Seaman (2003) define online learning as an online course, which has at least 80% of its course content delivered online, and consists of little or no face-to-face meetings. Studies quantifying the costs, benefits, and ROI of online learning started gaining momentum circa 1999, and then poured in steadily during 2000, 2001, 2002, and 2003.

According to Allen and Seaman (2003), nearly two million students took online courses in 2002. Almost 600,000 students took all of their courses online. There are signs that the growth in online courses may approach 20% per year. 80% of all institutions offer at least one online course, and 34% offer online degree programs. 97% of public institutions offer at least one online course, and 50% of those offer an online degree program. Nearly 60% of institutional leaders believe in online education, and the same amount believes online education is equal to or superior to traditional course delivery. These statistics indicate that the tide is turning in favor of online education, in just a few short years. This is no surprise.

Though not exclusively intended for evaluation and assessment of online education, Phillips (1997) provides an interesting framework for measuring the ROI of training programs. Phillips’ five level ROI model for evaluating training programs consists of measuring student reaction, student learning, student application, business impact, and return on investment. Reaction measures participant reaction to the program and stakeholder satisfaction with the program and the planned implementation (e.g., end-of-course questionnaire/student survey). Learning measures skills, knowledge, or attitude changes related to the program and implementation (e.g., in-process evaluation, quizzes, term papers, oral evaluations, presentations, and end-of-course exam). Application measures changes in behavior on the job and specific application and implementation on the program (e.g., surveys, interviews, focus groups, in-process measurements, audits, and compliance analysis). Impact measures business changes related to the program (e.g., cost, quality, productivity, cycle time metrics and models). Return on Investment compares the monetary value of the business impacts with the costs for the program (e.g., cost, benefit, benefit/cost ratio, and return on investment). The weakness of Phillips’ model is that it is laborious, manual, expensive, invasive, and scientifically complex.

Bishop and SchWeber (2001) have identified four quality indicators of online education, student support, faculty support, curriculum development and delivery, and evaluation and assessment. Student support consists of library reference services, administrative services, bookstore services, and information technology support. Faculty support consists of recruiting, admissions, class size, training, and information technology support. Curriculum delivery consists of course content analysis, curriculum delivery, critical thinking, and application of web resources. Evaluation and assessment consists of course evaluations, program reviews, longitudinal assessments, faculty course assessments, and alumni surveys.

Estabrook (2001) presented the costs and benefits of implementing an asynchronous learning network at the University of Illinois Graduate School of Library and Information Science from 1996 to 2002. The seven year cost amounted to $3,657,800, and the average annual cost was $522,543. Cost categories included, faculty, adjunct faculty, faculty maintenance, summer money, assistant dean, technical support, graduate and teaching assistants, mentors, evaluation, equipment/software, telecommunications, travel, supplies, mail, office, clerk-typist, admissions clerk, promotion, coordinator, special activities, academic outreach, contingency, and campus charges for services. Quantitative benefits included 14.9% higher salaries by program graduates, a 2,952% return on investment by students, and a program growth of 330%. Qualitative benefits included faculty skill modernization, improved quantity and quality of curriculums, more courses offered for dollar invested, information technology infrastructure was modernized more frequently, and the student retention rate improved.

Wentling and Park (2002) conducted a survey of the costs and benefits of e-learning. Some of their reported benefits included student break even points ranging from 20 to 112 students. They also reported a return on investment ranging from 228% to 3,283%.

The impacts of applying the ROI of online education consist of comparing the costs and benefits of online learning with those of traditional methods in training and education. Managers must determine when it is appropriate to use traditional instruction methods, online education, or a mix of educational alternatives. Managers must now budget for these alternatives, and this may or may not include broad based fluctuations in infrastructure budgeting and cost accounting. Managers are faced with the decision to budget for traditional training infrastructures versus outsourced online education. And, they are faced with establishing and applying performance standards for hiring and retention of employees educated using traditional versus online methods.

ROI of Telecommuting

Telecommuting is defined as the use of information technology as a substitute for commuting to and from work using traditional transportation. Telecommuting is a proper subset of teleworking, which is a form of distributed or decentralized work. Teleworking or decentralization is an alternative work style for the information age versus the industrial age, which is characterized by agriculture, administrative bureaucracies, and manufacturing. Modern workers can now access information remotely or electronically, and are no longer required to commute or drive to a central location to access their tools, data, and raw materials. Teleworking eliminates dangerous commutes, inflexible and imprecise work schedules, unproductive meetings, work related stress, expensive office space, inability to attract top talent, and chronic automobile pollution. Teleworking increases productivity, organizational flexibility, response times, morale, management capability, and personal flexibility. Teleworking reduces turnover, eliminates office space, lowers real estate costs, results in a cleaner environment, and reduces energy consumption.

Nilles (1998) provides a broad survey of the costs and benefits of teleworking, primarily focusing on the factors surrounding home based telecommuting. Nilles presents the costs and benefits of teleworking, namely work or social changes, employee profitability, decreases in pollution and energy consumption, typical cost drivers, and qualitative and quantitative benefits. Increases in work or social changes include, liberation—320%, continuity—170%, creativity—190%, personal life—150%, environmental influences—160%, general work life—110%, stress avoidance—90%, interdependence—50%, visibility—50%, belonging—30%, and apprehension—10%. Teleworking increases employee profitability by more than 30%. Annual decreases in pounds of pollution include, 1,000 people—1,720,089, 10,000 people—17,200,893, 100,000 people—172,008,929, 1,000,000 people—1,720,089,286, 10,000,000 people—17,200,892,857, and 100,000,000 people—172,008,928,571. Annual decreases in gallons of gasoline include, 1,000 people—1,560,000, 10,000 people—15,600,000, 100,000 people—156,000,000, 1,000,000 people—1,560,000,000, 10,000,000 people—15,600,000,000, and 100,000,000 people—156,000,000,000. The cost drivers of teleworking consist of additional training, telecommunications equipment, computers, moving expenses, facility leasing, construction costs, furniture, insurance, miscellaneous rentals, project administration, additional travel, and liability. The benefit factors of teleworking consist of increased employee effectiveness, decreased sick leave, decreased medical costs, increased organization effectiveness, decreased turnover, reduced parking requirements, office space savings, and recruiting effectiveness. Quantitative benefits (less costs) of teleworking consist of 1,000 people—$8,417,708, 10,000 people—$84,177,083, 100,000 people—$841,770,833, 1,000,000 people—$8,417,708,333, 10,000,000 people—$84,177,083,333, and 100,000,000 people—$841,770,833,333.

Shafizadeh, Niemeier, Mokhtarian, and Salomon (1998) conducted a broad survey of major studies surrounding the costs and benefits of telecommuting. Their study identified critical factors, telecommuting cost factors, telecommuting benefit factors, annual savings, and annual benefits. Telecommuting cost factors included, marketing/training development, evaluation, ongoing marketing/training, latent demand realization, urban sprawl, planning, marketing/training, equipment, internal program administration, marketing/recruitment, and training. Other cost factors included, equipment maintenance/replacement (less salvage), communication, decreased workplace interaction/immediate access, security of data, equipment, software, stress to perform, communication costs, utility costs, space costs, decreased workplace interaction, loss of support services, and loss of boundary between work and home. Telecommuting benefit factors included, travel reduction (direct), emission reduction (direct), improved highway safety, increased economic development (employment opportunities for underemployed/mobility limited labor segments), increased neighborhood safety, and space cost savings (office and parking). Additional benefit factors included, recruitment (access to best talent and broader labor markets), improved retention, increased productivity, less absenteeism, less sick leave, longer hours, fewer distractions (greater productivity per hour), improved customer service, disaster recovery, and public relations, compliance with air quality and trip reduction regulations, and travel time/stress savings. Still, more benefit factors included, travel cost savings, other cost savings, personal flexibility, reduced work-related stress, ability to get more/better work done, ability to work while mobility limited or physically distant from workplace, and more time with family. Shafizadeh’s, Niemeier’s, Mokhtarian’s, and Salomon’s literature survey identified annual savings of 3.5 billion gallons of gasoline per year, 3.1 billion hours of personal time saved, 1.8 million tons of pollution avoided, and $500,000 in transportation maintenance avoided. Annual benefits per telecommuter consisted of, vehicle miles avoided—1,583 to 3,540, gallons of fuel saved—46 to 177, fuel cost savings—$51 to $280, state and federal taxes avoided—$14 to $32, grams of carbon monoxide not emitted—3,398 to 120,415, grams of nitrogen oxide not emitted—620 to 6,054, and grams of hydrocarbons not emitted—521 to 18,166. Annual benefits per person also included, grams of particulate matter (brake dust) not emitted—33 to 243, monetized carbon monoxide not emitted—$1 to $90, monetized nitrogen oxide not emitted—$1 to $55, monetized hydrocarbons not emitted—$2 to $23, monetized particulate matter (brake dust) not emitted—$5 to $34, time in hours saved—5 to 174, and infrastructure savings—$45 to $69.