Page 1

A Governance Model For Agile Projects

Alistair Cockburn, Humans and Technology, ,
©Alistair Cockburn
HaT TR 2005.03, Sept. 07, 2005)
(v 0.9 during reader comments period)

Get oversight of a fairly large and complex multi-project that uses incremental and concurrent development in differing amounts on different subprojects, by estimating and then tracking intended versus actual functionality integrated each month or quarter for one or more years at a time..

1. The Project Status Graphic

Figure 1 shows a project status graphic. The first section of this article describes, its elements, how to gather the information and how to read what is happening on the project accordingly.


Figure 1

The graphic contains the following elements:

  1. The names of the components, applications or subsystems being built (I'll call them all components to simplify matters), the names of the leader responsible for the component, whether techical lead, product or project manager, and the place of the component in the system architecture (workstation, mid-layer, back-end)
  2. Which components are codependent with which others (horizontal, vertical, or both).
  3. The intended strategy for developing each component incrementally over the project life.
  4. The expected versus actual amount of work completed to date.
  5. A warning color indicating the alarm level for that component.

(A colleague refers to this as a "Tufte diagram", after Edward Tufte's book The Visual Display of Quantitative Information, because of the number of categories of information captured in the diagram.)

The next sections describe those aspects of the graphic. To simplify description, I cluster them into three categories: the system structure, the declaration of intended strategies, and the expected versus ideal

System structure

The horizontal and vertical rectangles in the graphic show the components to be built as either:

  • horizontal or vertical; and
  • new or legacy.

In the project where I first encountered this situation, there were legal reasons why the legacy system had to be discontinued. However, the new development had to be integrated with the legacy system, at some cost in additional work, for its first two deployments. Thus, it was important to track all work being poured into retrofitting to the legacy system (and to show the legacy system being removed over time).

In the graphic, the yellow components are new development, the blue and green components are legacy. The just-mentioned project would use three of these graphics, one for each deployment, showing the blue/green components being replaced by yellow components.

"Vertical" components are individual applications, probably crossing multiple machines - workstation, middle tier, back end database. "Horizontal" components are service components those that serve multiple applications, such as desktop services, security, auditing, and database. Some of the horizontal components are independent of any application (desktop services, for example), but others contain function or data specifically for certain applications. Tracking the progress of these requires stating how much of their content has been completed for each application (as well as what has been completed that is application-independent).

Putting these together, we see that the graphic shows

  • three horizontal (cross-application) workstation components: a new UI shell, new common UI elements, and some legacy security elements;
  • two new horizontal (cross-application), application-independent, back-end components: a new security module and a new set of common database services;
  • two horizontal back-end components that are application dependent, identified here just as DB1 and DB2. DB1 is new, DB2 is a legacy component being pressed into service for the first deployment.
  • three vertical components (applications), identified here just as App1 and App2, both new, and App3, a legacy application that is being modified slightly to integrate with the new components.

The system sketched in Figure 1 is fairly small as these go. The first project for which I drew this graphic had 17 horizontal and 15 vertical components. It could still be drawn to be readable - though cluttered - on legal-sized paper.

Intended Strategies

Although incremental development is a cornerstone of agile development, it has been around much longer than the agile movement. The question has long been how to graph different incremental strategies for governance purposes, and preferably in a small space.

Many organizations use traditional waterfall style phase completion milestones -- requirements complete, design complete, code complete, testing complete -- as markers of estimated progress through the project. These milestones serve to establish approximate location of the project team in an overall effort. Looking at their historical record and into the future, the project leaders will project that when the requirements are complete, the project will be at about its 15% mark; when the design is complete, the projects will be about 30% along; when the coding is completed, the project is about 90% done ... and here of course comes the big mistake ... that testing and integration will take the last 10% of the time.

While these numbers are not likely to be very accurate on any given project, as markers of forward movement they serve a purpose badly needed by the steering leadership. Any proposal to replace phase completion milestones must provide a similar sense of forward movement. To create a suitable replacement, we need the notions of "running, tested, features" and "internal releases".

Agile- and other incremental-development types of projects measure progress not according to how many requirements have been gathered, but by how much running functionality has been designed, programmed and integrated (what Ron Jeffries neatly calls RTF: Running, Tested, Features). A common way to show this is through burn-up charts, as in Figure 2, which shows the expected versus actual integration of a set of work-flow components by month.


Figure 2

RTFs shown on burn charts are wonderful, but they more than we need and take too much space. To shrink them to the dimension needed for basic project governance, we add the notion of "internal releases."

Some systems deploy to live users every week or two. Many larger systems can't be deployed that often, usually for reasons having to do with user training or the cost of integrating with or replacing existing systems. In these situations, it is valuable to pretend to deploy the system. This allows the team to

  • test, integrate and actually deploy the system to some computer somewhere in the world, exercising those elements of the development process, and
  • put the system in front of one or two friendly but real users so they can comment on what has been built; these comments provide real user feedback, and also to motivate the development team to take their work seriously.

The system gets "deployed" in the sense of exercising the process, but it is not deployed in the sense of being put into real use by all the users of the future system. To capture this sense of real-but-fake deployment, Jeff Patton of Thoughtworks uses the phrase "internal release." Troy MacArthur of IHC uses the term "friendly deployment" as an alternative description. You might find one or the other to work better in differing cultures. I adopt the term "internal release" here.

An "internal release" should occur every one, two or three months, but not longer (keeping them to three months or less was found as a key project success factor in the research behind Surviving Object-Oriented Projects). While there are many valid reasons not to carry out a full deployment every three months, there are almost no valid reasons not to carry out an "internal release" that often. These internal releases fit very neatly into a monthly or quarterly reporting mechanism.

Using RTFs and internal releases, we are finally ready to capture various intended development strategy or strategies that might show up on an incremental-development project.


Figure 3

Figure 3 shows three different incremental development strategies. The small vertical tick-marks show 10% units of completed RTF from left to right. The four triangle milestone markers show the amount of RTF the team intends to have completed and integrated at each internal release milestone. Alert readers will notice that these tick-mark drawings capture the vertical axis of the burn-up charts at the internal-release times. Examining Figure 3, we observe the following three, different strategies:

  • The top team intends to get less than 10% of its functionality in place in the first internal release, and then to integrate functionality in roughly equal thirds after that.
  • The middle team intends to do almost the reverse, and get almost all of the functionality in place by the third internal release (presumably so they have time to fix mistakes in the fourth period).
  • The bottom team expects to get almost 20% completed and integrated in each of the first two periods, and then to speed up and 30% done in each of the last two periods.

These small diagrams let different teams work in different ways and report on their intentions. This is our goal.

Intention-Completion Bars

The final part of the graphic shows expected versus actual accomplishment, and the alarm level the team wishes to register (see Figure 4).




Figure 4

In each of the three reports in Figure 4, the taller vertical bar moves from left to right within each internal release period to show where the current expected accomplishment. It can run at a constant rate within each period according to the calendar, or it can be synchronized with the team's iteration plans (2- to 6-week planning and tracking time windows).

The colored rectangles in the figure show the amount of RTF actually completed and integrated to date (reminder: code that is not integrated and tested does not count as being finished for our purposes). In Figure 4, we see that the top team is delivering according to schedule, the middle team is a little behind, and the bottom team by the middle of the third period has still not finished the work scheduled for the second internal release.

Once the current completion bars get moving, it is possible to associate colors with them, green - yellow - red, to indicate the level of alarm associated with that component. Within the first internal release period, while the current completion bar is too small to be colored, it may be necessary to put a separate alarm marker on the graphic to signal any component that is in trouble (as in: "To meet our deadline, we would have to deliver in this fashion, but we already know we are in trouble!").

Two comments must be made at this point about RTF. The first is that not all final deliverables consist of "features" that "run". Content databases and end-user documentation come to mind as examples. The teams can create intention-completion bars for whatever their final deliverable is, since those bars show growth of accomplishment over time. The second comment is that measures that do not tie to running, tested features have a built-in hazard, namely that it is easy to start tracking completion of artifacts that don't directly sell to or get used by the customers and users. Tieing accomplishments to RTFs makes it easier to see the actual value of what is being created.

Intention-Completion and the Hexagonal Architecture

The Hexagonal Architecture design pattern allows application logic to be developed separately from both the user interface and the back-end data services. This is a useful strategy to decouple development work in those three areas.

When Hexagonal Architecture is used, the development team may develop those three sections of an application at different rates. Figure 1 shows this by showing three different intention-completion bars for those three sections of each application. When the three sections are developed synchronously, one could merge the three and have only one intention-completion graphic for each application. This will simplify the status report.

Horizontal-Vertical Component Interaction

Some horizontal components are independent of each application added to the system; others require specific additional work for each new application. Application-specific content databases are examples of the latter. These are marked as "application-dependent" in Figure 1.

To show the state of application-dependent horizontal components, we need to have an intention-completion graphic for the application-independent portion of the component, and another for each application it must serve. Many modern projects contain one or more of these application-dependent horizontal components. They are typically difficult to report on, since their status is partially independent of applications and partly dependent on them. Having a separate intention-completion graphic for each part allows the development teams to report on how the horizontal and vertical teams are planning their joint work overtime, and how they are doing.

Summarizing the Graphic

At this point we have described all the elements of the graphic. Let us review them briefly:

  • The big rectangles represent components, subsystems or applications. The color and shading on the big rectangles represent new components contrasted with legacy components being updated (this is only important if, for example, it is necessary to remove the legacy systems over time or they use a separate funding pool).
  • Applications are drawn as tall vertical rectangles. Components that get used across multiple applications are drawn as horizontal (this could be reversed for better layout if, for example, there are few cross-application components and many applications). The placement of the big rectangles show the place of the component in the overall architecture: the top set of components reside on the desktop, the middle and bottom sets of components reside as back-end services. The horizontal rectangles that run behind the applications are created independently of the specific applications; the horizontal rectangles that run in front of, and intersect with the applications require application-specific work or content.
  • The component's name and the responsible person's name is put on each big rectangle.
  • Intention-completion markers are created for each component. They show the intended amount of content (preferably RTF) intended for completion at each internal release milestone, the expected and the actual current accomplishment, and the alarm level. An intention-completion marker is created for each component, and also for each intersection of application-dependent components.

2. Gathering the information

Figure 1 is "just a graphic." All of the information in the graphic also fits in a spreadsheet. The spreadsheet version is useful for estimating and planning purposes (since component owners can just submit their spreadsheet rows to the person integrating the spreadsheet), and it is even more useful for tracking status after each iteration, since automated tools can roll up each team's accomplishments into reports at various levels. The highest level is the one that gets "painted" onto the graphic. (The graphic can be generated automatically, too, using markup languages with variables for the intention-completion bars, but that programming effort may take longer than simply coloring the bars each month).

Gathering Estimates

Figure 5 shows a spreadsheet that can be used to capture the estimates. This spreadsheet contains three pieces of information necessary for proper communication about the project that are not in the graphic. They are

  • Units,
  • Total size,
  • Confidence in the estimate.

Project/ Component / Sub-component / Owner / % done in IR1 / Total size / Units / Confidence in Estimate
(L, M, H)
IR 1 / IR 2 / IR 3 / IR 4
Desktop / frame / Mr. A / 0 / 20 / 80 / 100 / 30 / UI widgets / Med :-|
Desktop / APIs / Mr. A / 20 / 50 / 80 / 100 / 60 / API calls / Lo :-(
App 1 / UI / Mr. B / 5 / 60 / 90 / 100 / 450 / UC steps / Lo :-(
App 1 / app / Mr. B / 10 / 60 / 90 / 100 / 450 / UC steps / Hi :-)
App 1 / bus svcs / Mr. B / 5 / 50 / 80 / 100 / 450 / UC steps / Lo :-(
DB 1 / setup / Ms. C / ?? / Lo :-(
DB 1 / App 1 / Ms. C / 60 / codes / Lo :-(
DB 1 / App 2 / Ms. C / 10 / codes / Lo :-(
DB 2 / setup / Ms. C / ?? / Lo :-(
DB 2 / App 1 / Ms. C / 2000 / entity attributes / Lo :-(
DB 2 / App 2 / Ms. C / 1500 / entity attributes / Lo :-(

Figure 5

It is one thing to say, "We intend to be 20% done after the first internal release", but the governance board needs to know, "20% of what?" Being behind on 20% of two use cases is very different than being behind on 20% of 80 use cases. Therefore, the additional information has to do with the size of the work being done.

The first necessary piece of information is "what is the unit of accomplishment?" In some cases, it will be use cases, or more likely, individual steps in use cases. In other cases, it will be something different. For the Desktop component in the table, we decided that the two units of accomplishment are UI widgets such as frames, pull-down lists, or buttons, and API calls used by the applications. For a database, it might be entities and attributes; for a web site it might be articles and images; for a medical system it might be medical codes.

The second necessary piece of information is, "about how many units will you have to create?" At the start of the project, these might be estimated with 50% error bars: "We expect somewhere between 15 and 50 UI widgets, call it 30, plus or minus 50%." This estimate can easily be updated at the start of each successive iteration, with shrinking error bars over time. (Note for the interested reader: burn-up charts can show the moving "100%" mark as a moving ceiling to the graphs, see [ref]). Probably the hardest part about getting this number is dealing with the phobia the development team has to numbers, since they have so often have been punished with them. In part, this is a social problem outside the scope of this article, but also in part, the estimation errors on the size and quantity should less than the estimation errors in time-to-completion.