Systematic Software Reuse: Architecture, Process and Organization are Crucial
Martin L. Griss, Laboratory Scientist
Software Technology Laboratory, HP Laboratories
Tel: (415) 857-8715
Email:
Introduction
Based on many years researching software reuse and consulting with HP and customers' reuse projects, I have found that the following ten guidelines are key to ensuring systematic reuse:
Long-term management leadership
Systematic, incremental organization and process change
Manage assets as a product family portfolio
Design for domain-specific, architected reuse
Separate: Create/Support/Utilize
Objects alone are not sufficient
Start small pilots, then scale up
Address culture, experience and organization stability
Invest in infrastructure, support and training
Find dedicated reuse champions
This article considers why reuse is so important to competitiveness, and explains and illustrates these ten guidelines. Follow-up articles will begin to address extensions to Fusion for systematic reuse.
What is reuse, and why do you want it?
Reuse is based on a simple, well-known idea. When we build a new firmware or software application, we should use previously developed software components. Doing so will save the cost of developing, testing, documenting and maintaining multiple copies of essentially the same software.
These benefits can be quite significant. Reuse programs within AT&T, GTE, Ericsson, NEC, Toshiba, and HP itself, demonstrate that time to market can be improved by a factor of 1.5 to 2.5 or more. Software or firmware that used to take 24 months or longer to produce, can take only 6 months or less to develop with reuse. Quality can be increased by 5-10 times, and costs of development and maintenance can be reduced. And there are other benefits. Reuse of standard modules can increase the interoperability and consistency of products. Appropriate components and supporting tools can enable a professional services organization to rapidly produce ``almost custom solutions'' cost effectively.
But reuse is not a silver bullet. This simple idea is quite complex in practice. It is not enough to gather interesting pieces of software into a library and offer them to people to reuse. Components have to be carefully designed and carefully developed so that they are high quality and work well together. They have to be documented well so that reusers can understand them. Components have to be carefully chosen so that this extra investment will be repaid by significant reuse. This works best when reusing components between members of a product line or product family (or product ``domain'').
The software development process needs to be changed to support the development and reuse of components. People need to be trained in the use of the components and library systems, in the new technologies and the process. Some people need to take on new roles, such as component developers or librarians. New technologies and methods, such as objects and frameworks, generators and Wizards, require new metrics, and new development models. Mechanisms to fund component development and reuse need to be established. The organization needs to gain experience with these changes.
What seemed like a simple idea has become complex, and needs management attention and organizational support. An incremental plan must be developed and followed. Nevertheless, systematic software reuse is still widely believed to be best way to dramatically improve the software process.
Lessons from HP reuse projects
I have observed the evolution of several HP reuse programs closely, and learned a lot from them. Started around 1990, a Test and Measurement reuse program developed a small number of large components, which fit together within a test instrument architecture targeted to cover about 30% of the firmware within a typical instrument. The reuse team developed an interesting social process and cross-divisional organization to ensure that projects in several divisions were able to share the cost of developing the architecture, components and tools. They piloted the use of the components to ensure that they worked well. As a consequence, the reuse components have been used in many products (17 to date, 10 more underway). In some, they have been used as a “domain-specific kit”[3,4], using all of the components, architecture and generator tools. In other cases, only a few of the components were used. Work continues today within the reuse center to develop new components.
An article in the April issue of the Fusion Newsletter provides an excellent discussion of experiences developing a reusable OO framework using EVO Fusion [5]. Many organizational, management and technical issues had to be addressed. Some were anticipated, while others were discovered along the way. It is important to identify these risks early, and plan to handle them systematically.
Systematic reuse needs a systematic approach
By systematic reuse, we mean an institutionalized organizational approach to product development in which reusable assets are purposely created or acquired, and then consistently used and maintained to obtain high levels of reuse, thereby optimizing the organization's ability to produce quality software products rapidly and effectively.
Over the last ten years, software reuse researchers and practitioners have learned that to succeed at systematic reuse requires careful attention be paid to both technical and non-technical issues[6,7,8,9]. The non-technical issues are more pervasive and complex than at first realized.
In order to get large amounts of reuse, perhaps division or group wide, we need a systematic approach. Without a balanced focus on people, process and product issues, we will not succeed at managing the scope and magnitude of the changes and investment needed. Simply creating and announcing a reusable class library will not work. Without a ``reuse mindset,'' organizational support, and methodical processes directed at the design and construction of reusable assets, the reuse investment will not be profitable. Research and practice have been directed at the development of assessments, guidelines, processes, models, methods and prescriptions that promise more cost-effective reuse.
This requires a significant effort to change culture, organization, and a multitude of other factors. These changes are quite radical, and more widespread than those incremental and ongoing changes associated with CPI (continuous process improvement). Reuse is a business issue. We have to change the way we view software at a fundamental level. It must be viewed as an organizational asset, to be invested in, improved, and leveraged effectively and consistently
Often, sweeping changes in the software development organization will be needed to institute large-scale, systematic reuse. Organization changes separate asset creators from asset users who build applications. Business changes fund product family design and construction with the goals of improved time to market and cost for several related products, investing in reusable assets. Software process changes manage and use a reusable asset repository. Many cultural and management changes create new, unfamiliar roles. The magnitude of the changes, and the issues encountered, are similar to those encountered when doing Business Process Reengineering. We find that the systematic methods and skills used to design and implement such changes can also be applied to software organizations.
Incremental adoption of reuse
Experience shows that it is best to adopt a large scale reuse program in stages, making organization and process changes in a series of incremental steps. Usually, an organization becomes ready for a transition when the business issues it faces motivate a change. Figure 1 illustrates some of the steps that a typical organization may progress through. This figure is based on a synthesis of experiences at several organizations within HP and elsewhere, and as such, is both a summary of history, and perhaps a prescription of how to proceed.
Figure 1. The incremental adoption of reuse is driven by compelling business needs
What this means is that an organization must move incrementally, must start by focusing on a narrow domain and only a part of the complete product architecture. This is best done by using a series of reuse pilots. As the reusable software (components and framework) is being developed, the organization must gain and retain management support, must customize process and organization, and must ensure that the reusers are involved in the dialogue early. In the process, one can not ignore social and cultural realities. We need to consciously address and overcome several issues related to mistrust, perceived loss of creativity and independence. It won't happen fast enough without specific plans and attention, but it is also important to note that most organizations can't move instantaneously either. Certain of the changes take time, perhaps several years.
What we observe is that the benefits due to reuse, such as improved time to market, or higher-quality systems, or lower overall development costs, increase as the levels of reuse and the sophistication of the reuse program increase. Furthermore, it takes time, investment and experience with reuse in your organization to get to those levels of reuse. So what seems to happen is that the organization moves from stage to stage as it consolidates its gains and feels the pressure for more improvement. For example, the transition from no reuse, to informal reuse (sometimes called leverage or cloning) in which chunks of code are copied, adapted slightly and then incorporated into the new system, occurs when developers:
are familiar with each others code, and trust each other
feel the need to reduce time to market, even though they would prefer to rewrite the software.
This strategy works--for a while. Development time is reduced, and testing is often less tedious than with totally new code. But as more products are developed using this approach, maintenance problems increase. Multiple copies of the software, each slightly different, have to be managed. Defects found in one copy have to found and fixed multiple times. This often leads to a ``black box'' code reuse strategy, in which a carefully chosen instance of code is reengineered, tested and documented for reuse. All projects are then encouraged or required to use just this copy without modification.
This works well for a while, and then the issue of dealing with changes to satisfy an increasing number of reusers arises. Should everyone be ``forced'' to use only the standard version? Should multiple versions be maintained? Should adaptation be allowed? Who decides? Should test files, designs and specification files also be reused? All of these issues lead to the creation of a component management process (similar to a preferred parts process), with supporting organization. People need education and help in using these components.
Beyond this point, to get higher levels of reuse, and more coverage of the life-cycle, it is important to architect the components and the systems that will use them. This is the only way to ensure that components fit together. The development and use of a common architecture involves even more organizational commitment and structure.
The reuse process and organization
Among the many issues which must be addressed are the software processes to be followed, and the structure and management of reuse organizations. A well defined software development process is crucial to effective reuse. Since no single software process model is ideal for all situations, different software life-cycles describes specific roles, activities, methods, metrics, responsibilities and deliverables of the developers and managers. A reuse process makes explicit guidelines that will ensure successful reuse. These include explicit separation of component developers (creators) and reusers (utilizers), and explicit reuse-oriented management and support. Effective reuse requires significant organization changes, typically spanning several teams within the company. Such sweeping changes can only be accomplished with informed and explicit management leadership.
Figure 2 shows a basic reuse process model. To explicitly support reuse-enabling guidelines, the model has four distinct process elements:
Figure 2. Ideal reuse process and organization
- Create This process provides reusable assets appropriate for the utilize process. Assets may be new, reengineered or purchased, and of various kinds, such as code, interfaces, architectures, tests, and tools. Activities include inventory and domain analysis of existing applications and assets, assessment of utilizer needs, market and technology trends, and architecture and component definition and construction.
- Utilize This process reuses assets to produce products (applications or systems). Activities include the examination of domain models and assets, the analysis of product requirements, the adaptation of assets, the development of products, and the specification of proposed components.
- SupportThis process supports the overall reuse process, managing and maintaining the asset collection. Activities include the certification of new assets, classification for library storage, matching utilization needs with the assets, providing usage support, and collecting feedback and defect reports.
- Manage This process constrains and controls the other elements. It must plan, initiate, resource, track, prioritize, coordinate and improve the reuse process. Activities include setting priorities and schedules for new asset construction, resolving conflicts when needed assets are not available, establishing training, and setting direction. Coordination can span many levels - within each create and utilize project, across a set of utilizers, and up to the management of a complete ``producer-consumer system'' of projects.
Specific metrics are needed to manage the creation, support and utilization of reusable software, helping utilizers understand the value of the assets, and helping creators prioritize investments.
In a reuse organization operating according to the above process, there is tension between product development objectives (especially capabilities and schedule) and asset development and evolution (especially long-term reusability and quality). Ideally, all members of an organization have a shared reuse mindset, allowing them to operate from knowledge of the process and desired balance between product and reuse objectives. However, in practice, reuse only succeeds within a structure with separate creator and utilizer teams, and high-level management leadership and support.
I have found that the most effective reuse organization structure has four distinct kinds of teams, corresponding to the processes shown in Figure 2. These can sometimes be combined into a smaller set of organizations:
One or more creator teams who produce reusable assets to meet the current and future needs of the utilizer teams.
Several utilizer teams who reuse assets to construct products.
A reuse management team, usually led by a single manager, often assisted by a reuse steering committee of managers and other stakeholders from the other teams. They ensure that the overall process proceeds efficiently, that the construction and support of new assets is funded, and that conflicts between the goals and schedules of the different groups are resolved.
A support team manages the assets, provides utilizer support and training, does some maintenance of assets and provides coordinated feedback on problems. They ensure that compatible assets are delivered to meet utilizer needs and assist with some adaptation consistent with reuse objectives.
There are several issues that compete as a reuse organizational structure is set up. Each structure favors some stability and connections at the expense of some another. The reuse management team and process must be vigilant to ensure appropriate balance. While some structures are easier to establish initially, they are less effective if their creators and supporters do not have full-time, visible and official responsibility for reusable assets alone. Ideally, a single manager is able to coordinate interactions between teams. Whether creators and utilizers are in the same organization or not, they need leadership empowered to help meet both product objectives and organization-wide reuse objectives. If the managers of the other teams do not report to the reuse manager, or a manager close to the reuse effort, some other mechanism is needed to resolve priority and conflict issues, and provide funding for asset creation and support.
On one hand, if creators are embedded as members of product teams or report to the product team manager, they too often get redirected to assist with urgent product construction at the expense of asset creation. On the other hand, independent asset creation teams can easily neglect their responsibility to meet the needs of utilizer teams. They may then prioritize their efforts to produce assets, libraries and tools that are poorly matched to needs, even though technically excellent. Perceived as ``overhead'' or ``unresponsive,'' they get frustrated and either leave the organization, or try to become ``independent component vendors'' to sell the assets or make their own products.