Microsoft Patent Predisclosure Document

Title of Invention:Process for measuring coding productivity/quality
Date: June 19, 2002
Document Author(s):Rick Andrews

Prior Disclosure

No disclosure outside of Microsoft

Introduction

Maximizer is a new process/algorithm for measuring coding productivity/qualityy that overcomes most of the previous/existing obstacles/objections/problems with measuring coding productivity. I feel that this new process could revolutionize the software development industry and bring our engineering standards up to the standards practiced by other engineering disciplines.

Full discussion of the product can be found at:

Strategic Importance of Invention:

A well known adage for quality engineers is “What you measure, improves”. There has never been a universally accepted standard for measuring software development because tools weren’t available to measure it adequately and simpler methods were met with distain as being unfair to developers. This new method, named “Maximizer”, uses a straight forward method of measuring defective code compared to total code written. Note, it DOES NOT involve bug counts in any way. Counting bug counts has been the major cause of problems in previous measurement schemes.

It’s strategic to Microsoft because it gives us a way to drive improvement far better than any of our competitors.

  1. Is it core technology? No.
  2. Is it a feature that gives Microsoft a competitive advantage? Yes, we can more accurately measure our processes and quality; which means we’ll produce better software than our competitors.
  3. Is it a feature that our competitors would want to copy? Yes, The lack of a way to measure code quality/producitivy is a well known/documented problem throughout the industry and our competitors would pay money to use this tool to measure their own quality and productivity. Furthermore, I think that someday, this could be something our customers expect to see done before purchasing software.
  4. Does it include new APIs, file formats, network protocols, data schema or other components relating to product interoperability? Yes, currently it is dependent on SourceDepot; but this dependency is easily removed.
  5. Is it related to a standard? No, hopefully it will define a new standard.

The most likely competitors for such a product would be companies like McCabe & Associates ( who produce metrics applications for software development.

Motivation for the Invention:

[Describe (1) the problem addressed by the invention (e.g., limitations of prior products of Microsoft, or others), and (2) your solution to the problem (including what “new” things your invention does and a high-level description of how it does them).]

Maximizer is a tool developed by the Microsoft Six Sigma team that is intended to provide a fair and accepted way to measure our coding productivity. Previous methods of counting Bugs/KLOC, etc., received resistance from the developers it was measuring because,

  • They felt it unfairly favored developers working on simple/uncomplicated issues and punished those who were challenged with our toughest problems.
  • The size of bugs was variable. Some bugs only require a single character change, while others require can require 10s of thousands of lines of code. In the grand scheme, of course, the laws of large numbers satisfy this concern, but for the individual developer who may only have a small number of bugs, the laws of large numbers don’t apply.
  • Raid reporting is suspect and it’s often difficult to say accurately that bugs resulted in code changes or not. Again, the laws of large numbers make this valid in the grand scheme, but not for the individual developer being measured.

Regardless, though, of whether the developer concerns are valid or not, the fact remains that during our two years of study of our coding processes, developers have repeatedly shared their distain towards measuring their productivity based on bugs. If they aren’t happy being measured by such metrics, we will, in many people’s opinion, have trouble using these metrics to drive improved productivity. Management, of course, loves these metrics for good reasons; they are easy to calculate and they yield meaningful results when considering large numbers.

About a year ago, we set out on a goal to find a way to measure coding productivity that was not based on bug counts, would be reasonable to calculate, and would be accepted by developers and managers alike. Part of our goal was to say that “code is code is code” so we could measure all code equally regardless of its language, complexity of the problem it is solving, or the process the team uses to develop and ship code. Our answer was to look at the Six Sigma approach of measuring results based on Defects Per Million Opportunities (DPMO). To do this, we needed to define what an “Opportunity for Defect” (OFD) is in the software development industry. We considered several alternatives and finally settled on the OFD being equal to a line of code. In other words, each line of code written by a developer is an opportunity for a defect. Thus, any line of code that needs to be changed later is considered a defect (of some sort). Notice that we no longer care about bugs; we are simply counting the number of lines of code that had to be changed because they were wrong in the first place.

Of course, this solution wasn’t fool proof and we still had a lot of issues to consider (e.g., what if the reason a line of code was changed had nothing to do with a mistake? What if it was planned from the beginning to be changed later?). Hopefully our solution, as described herein, has addressed most of these issues. In the process of developing this solution, several developers and managers where consulted for their opinion and favorable feedback was received by all.

Another goal we faced in developing this solution was to come up with a solution that provided the least impact to the developers writing code. We’ve tried to put most of the work on the server/admin side of the tool and strived to make the tool as transparent as possible to the developer. There is still a simple operation that must be performed on each checkin. It is hoped that teams can build this into their existing automated checkin procedures; it’s a simple script that should be simple to incorporate. This will provide us with the most accurate information, but it could also be performed by the dev lead, Maximizer admin, or scheduled to happen automatically any time after the checkin, with less accurate results.

In summary, the approach used by Maximizer to measure coding productivity is:

  • Code is code is code.
  • Count lines of code changed (including added, deleted, and moved) instead of bugs.
  • Centralize the data in a single location so it can be easily maintained and measure all code the same way.
  • Impact developers as little as we can.

Description of the Invention:

[Describe your proposed implementation of the invention, including the architecture and design details of the implementation. The design details should include a description of the component parts of, and individual operations performed by, your implementation. The use of a specific example, showing how the invention solves the problem being addressed, can be particularly helpful. You should also mention whether you have thought of any other implementations, or applications of, your invention. In most cases, 1-2 pages of description should be adequate to start the patent application process, although a more detailed description may greatly enhance the efficiency of the process.]

Maximizer Tool Overview

The files for the Maximizer tool can be found at \\6sigma\Public\Tools\Metrics\Maximizer

An Excel template for collecting data for Maximizer can be found here.

The following diagram outlines the process used for gathering the necessary data and calculating the success metric. A discussion of the calculations themselves can be found here.

The red and white numbers referenced in the discussion below refer to the numbered blocks in the diagram above.

Preparing to collect the data

SourceDepot- The Maximizer tool assumes that the code you want to measure is stored in a Source Depot (a future version of the tool may address other methods). Also, if you simply want to measure a single file, or set of files, you can use any diff program (such as diff.exe at \\6sigma\cache\metrics\enlistments) to count the number of lines added, deleted, and changed, and then apply the formula described here. The rest of this document assumes your code is stored in a Source Depot.

SourceDepot Enlistment on \\6Sigma– You first need to create an enlistment (and sync) for your project on the \\6Sigma server. This step should be performed by a \\6Sigma administrator with direction from you on precisely what needs to be enlisted. Contact MaxAdmns for help on this step. When creating your enlistment, it’s important that you refine the enlistment as much as possible to only the files you are interested in measuring and further refine, if possible, when you sync.

Register Enlistment with Maximizer– The \\6Sigma administrator will probably do this step for you but, if not,

  1. Go to This is the same as the Maximizer Reporting Tool ().
  2. If your enlistment is not listed in the left pane, click the “New” button at the top of the left pane; if it is listed, then you are already registered and can move to .
  3. A new entry will appear in the list named “New Enlistment”. Click on that new entry. Note in the right pane the ID number in the first cell of the table. YOU WILL NEED THIS NUMBER LATER; make a note of it.
  4. In the right pane, click on the “Edit” Button at the top.
  5. Assign a recognizable name for your enlistment.
  6. Click the “Update” Button.

More later on this tool

Collecting the data

MaximizerGetSDData.js – The \\6Sigma administrator will perform this step, but may need your help. The MaximizerGetSDData.js tool pulls checkin data from the Source Depot enlistment and adds it to the Maximizer DB (). All further processing is done against the Maximizer DB to offload processing against the Source Depot. It is recommended that this tool be scheduled to run automatically on a daily basis. Note, each time the tool is run, it picks up from where it left off on the previous run. If you need to start from a different place (e.g., start over or redo some checkins that you accidentally deleted), you can do so by changing “Last handled raw checkin” and/or “Last handled filtered checkin” (See “Raw vs. Filtered” below for more on “raw” and “filtered”) values by running the Maximizer Reporting Tool () at (more below on how to change these values).

Following are the instructions for running the MaximizerGetSDData.js tool. Note, the tool should be run on the \\6Sigma server itself or via a remote desktop connection to it.

At the command prompt, type:

cscript MaximizerGetSDData.js –e <enlistmentPath> [-r | -f] –s 6Sigma –i <enlistmentID>

usage:

-e enlistmenPath - The path to the enlistment you created in step . Make sure you choose the folder that contains the sd.ini file. This path cannot be a network or internet path; it must be a filesystem path on the local \\6Sigma machine.

[-r] do raw delta – Process the checkins as they are without filtering out blank lines, comments, etc.. I don’t foresee this option being used very often on C/C++ code, but it is the preferred way to run it on html, VB, SQL, etc., code. See “Raw vs. Filtered” below.

[-f] do filtered delta – Process the checkins by first filtering out blank lines, comments, etc., so we only count actual code. Note, this is assuming the files are C/C++ files using C/C++ style comments. See “Raw vs. Filtered” below.

-s the SQL server name – this should always be “6Sigma” unless you set up a different server. Please avoid doing this; we want all the data in a central location for all of Microsoft.

-i the enlistment ID – The ID number assigned to your enlistment in step C.

example:

cscript MaximizerGetSDData.js –e g:\metrics\enlistments\netconfig\net -f –s 6Sigma –i 2

This example gets the data from SourceDepot for the enlistment at g:\metrics\enlistments\netconfig\net on the \\6Sigma server and filters it. It tags each entry with the ID of 2 to match the Enlistment ID assigned by Maximizer when it was registered.

Filter out unwanted checkins – This can be the tricky part and where most of the work will be required. Fortunately, it should only need to be done once, with an occasional update here and there. Not all records/checkins will be appropriate for the metric. Ideally, this will all be taken care of in steps 4 (by defining an enlistment that only contains relevant checkins) and 8 (by marking checkins as Not Applicable when appropriate), but it probably won’t catch everything. E.g., there will likely be checkins that we don’t want to count, such as:

  • Tools added to the depot
  • Build lab checkins unrelated to code fixes
  • Checkins from devs outside the team when you are trying to measure the team’s productivity.
  • Checkins to remove “UNDONE” comments, etc.
  • Checkins before or after the measurement period when we are trying to calculate a metric just for the measurement period.
  • Etc..

The best way to filter out these unwanted checkins is to write simple SQL queries against the “checkins” table of Maximizer DB () to delete them. Since the MaximizerGetSDData.js () tool starts each time from where you left off the previous time, there’s no fear that the deleted records will be brought back (unless you reset the start number as describe in , above. For recurring checkins that you don’t want to count (e.g., a dev you don’t want to count who regularly makes checkins to your depot), write the query to delete his/her records and then schedule it to automatically run immediately after each time you run the MaximizerGetSDData.js ()tool.

Following are some examples of queries that might be helpful:

delete from checkins where enlistmentID = 2 and person = 'REDMOND\rickan'

delete from checkins where enlistmentID = 3 and checkinnum > 123456

Maximizer DB – This is the SQL database on the \\6Sigma server that stores all the data necessary to calculate the Coding Success Metric. It contains 2 tables, “Enlistments” and “Checkins,” as well as several useful stored procedures and functions. The “Checkins” table is the one that will be most interesting for anyone needing to work with Maximizer. It contains a record for each checkin for each project that is registered () with Maximizer. When accessing the database, it may be named “sixsigma” instead of “maximzer”; I am trying to get this renamed.

MaximizerSetCheckinType.js – This tool allows anyone to set the checkin type for a checkin. It is designed to be integrated into the checkin process for your team. I.e., each time a developer performs a checkin, he/she should also run this tool to mark its type for Maximizer. For our purposes, a checkin can be one of 5 types or a combination of them. It’s best if each check falls entirely into one type/category, but this won’t always be the case. Thus, the MaximzerSetCheckinType.js tool provides the functionality to categorize each checkin by one or more types by identifying the percentage of lines in the checkin that fall in to each type. Following are the types we are tracking:

% Bug Fix of Measured Code / Percentage of lines checked in that were due to a bug fix in code previously measured (i.e., not a bug fix of legacy code) where the bug was the result of a mistake by a developer. Note, this includes mistakes such as not reviewing the spec to expectations, relying on an inadequate unit test, as well as simple coding errors. Bugs not related to developer mistake should be categorized as Deficiency or feature, as appropriate.
% Feature Work / Percentage of lines checked in that were due to new feature work.
% Deficiency Work / Percentage of lines checked in that were due to some other dificiency not related to dev mistake. E.g., New requirements are added after the spec is approved and the code is written.
% Bug Fix of Un-Measured Code / Percentage of lines checked in that were due to a bug fix in code that was never measured. In most cases, this should be fixes in legacy code.
% Not Applicable / Percentage of lines checked in that were unrelated to code fixes and/or bugs or were checked in by someone not on the dev team. E.g., Build lab checks in some build script changes, or dev checks in non-code files that we don't care about.

Following are the instructions for running the MaximizerSetCheckinType.js tool. Note, the tool should be run on the developer’s local machine, but can also be run on the \\6Sigma server.

At the command prompt, type:

cscript MaximizerSetCheckinType.js –s 6Sigma –i <enlistmentID> -c <checkin number> [[-b <% bugfix of measured code] | [-u <% bugfix of unmeasured code>] | [-d <% deficiency>] | [-f <% Feature>] | [-n <% NA>]]

usage:

-s the SQL server name (default: 6Sigma) - should always be “6Sigma” unless you set up a different server. Please avoid creating a different server; we want all the data in a central location for all of Microsoft.