Software Craftsmanship
Joseph Sims
Software Engineering
UW-Platteville
Abstract
Software craftsmanship is a modern software development movement that believes the current state of the software industry is troubled, and challenges the notion that “Software Engineering” is always the best approach to software development. Software craftsmanship believes today’s problems in development are mainly sociological, not technological. Software craftsmanship believesadapting the historically proven craftsmanship model can rejuvenate the software industry, by improving quality, and bringing significant rewards to not only the user, but the developers themselves.
Current State of Software Engineering
“Large programming projects suffer management problems different in kind from small ones” wrote Fred Brooks in his seminal software engineering text, “The Mythical Man-Month”. [4] While published over 30 years ago, Brooks’ observations on what doesn’t work in software development seem tohave been ignored by many.
“Software engineering” was a term introduced in 1967 by a NATO study group, and used by Peter Naur and Brian Randell as the title for their report on the 1968 NATO Science Committee conference. They noted that they chose the name to imply software “manufacturing” processes were needed as they existed in other branches of engineering, reflecting upon the general consensus from the conference that such a need existed to prevent a software crisis. The inherent difficulty in defining such a process was immediately evident however, as in the following year’s conference, focused on the technical issues of preventing such a crisis, participants were unable to agree on the details of a defined, quantifiable, and repeatable process, with splits between theory, practice, and management ideologies.[8]
Since 1968 the engineering mindset has stuck, although the scope of software development work has expanded dramatically. When the term was coined, personal computers did not exist, and the Department of Defense (DoD) was one of the biggest players in molding the software industry. The DoD defense projects were extremely large and complex endeavors that involved both software and hardware development, in a time when both fields were relatively immature.One such project was the SAFEGUARD Ballistic Missile Defense System, which involved specially designed computer hardware, and a massive labor effort, taking 5,407 staff years to complete.[15]
Today’s environment is the inverse of the 60s. While large systems engineering projects are still being developed, there is also a large amount of development being done on much smaller projects. One large system engineering effort is done by the “on-board shuttle group”, a Lockheed Martin division responsible for providing the software that controls all aspects of the space shuttle control, is staffed by 260 men and women, at a cost of $35 million a year. The process is rigid and so formal that the manager admits “Yes, the process does stifle creativity”. [10]
On the opposite end of the spectrum are thousands of small projects. In 2006,“Two guys built Twitter in two weeks.” [18].Both projects are huge successes; the shuttle group provides nearly error-free software that allows a spaceship to make an epic journey, while ensuring the safety of the $2billion shuttle and its crew, while Twitter invented a new form of social networking that has allowed people around the world to interact in new ways, including playing a critical part in organization of protestors in the 2009 Iranian presidential election protests [21]. The shuttle group used formal software engineering processes, while the Twitter team did not.
Some of those influential in the creation of software engineering principles now express doubts. Tom DeMarco, author of Controlling Software Projects: Management, Measurement, and Estimation, reflects that the message of his book seemed to be “metrics are good, more would be better, and most would be best”.However, 25 years later, DeMarcosays he is now “coming to the conclusion that software engineering is an idea whose time has come and gone”. He says this is because in the software engineering world, control is one of the most important, if not the most important aspect of any project, but he feels strict control matters less on “useful” projects trying to deliver something of significant value. He believes a project can be managed without controlling when you “manage the people and control the time and money”. DeMarcoconcludes “Software development is and always will be somewhat experimental” [6].
What Is Software Craftsmanship
Software craftsmanship is an alternative to the traditional software engineering model, which is defined by the IEEE as “the application of a systematic, disciplined, quantifiable approach to development, operation and maintenance of software; that is, the application of engineering to software”. [15] Few question the traditionalsoftware engineering’smodel demonstrated success in delivering safety critical systems. However, most developers are not working on safety critical systems. It is unlikely anybody would die because Facebook is down or a store’s inventory system is missing a feature, but people may definitely gain by receiving a slightly imperfect product sooner rather than later. Hunt and Thomas wrote in The Pragmatic Programmer, “Great software today is often preferable to perfect software tomorrow” [13]. So, this raises the question of whether practices used to develop such safety critical systems are applicable or appropriate for many of today’s more common software development projects.
Software craftsmanship believes software engineering unduly chooses process over people. The metaphor of craftsmanship illustrates the importance of people, and allows for the acceptance of the mix of art, science, and engineering that many make believe make up software development. Sociologist C. Wright Mills defined a craftsman as one who does his work for the satisfaction of producing high quality work, which he is able to do because of the freedom to experiment in his work, and allow his skills to continuously develop. [20]. Many believe the educational infrastructure for incoming software developers is inadequate, and a revamp of some kind is required [16]. Software craftsmanship believes adapting the historically proven craftsmanship model to software development can rejuvenate the software industry, by improving quality, reducing cost, and bringing significant rewards to not only the user, but the developers themselves.
History of Craftsmanship
Craftsmanship has a strong history in many industries, tracing back to at least the middle ages. It was in this time that the three tier apprenticeship system gained popularity and acceptance. A master would have at least one journeyman and several apprentices under his control. The apprentices would learn by copying tasks the journeyman demonstrated. Once the apprentice could demonstrate they achieved the required set of skills, they could become a journeyman. As a journeyman, they woulddevelop a deeper understanding of the work and gain managerial experience, with the potential to become a master.[20].
The International Brotherhood of Electrical Workers, an electrician and lineman union, has a strong history of training new members in via an apprenticeship program similar to that practiced centuries ago. An apprenticeship often lasts 5 years, and includes practical experience along with theoretical schooling. Apprentices work with electricians on real jobs to gain experience by pairing up with a journeyman and following the “listen, watch, do, review” cycle. This involves a journeyman discussing a task with an apprentice, demonstrating the task, allowing the apprentice to attempt the task, and providing feedback on the apprentice’s work. Apprentices usually“earn while you learn”, and are paid a percentage of the journeyman’s rate, often starting off at around 50%, and slowly increasing as training continues. Eventually, training completes and their skills can be tested in order to become a journeyman. [17]
History of Software Craftsmanship
The notion of software craftsmanship was sporadically explored in publications from at least the early 90s. One such publication was “Borland Software Craftsmanship: A New Look at Process, Quality, and Productivity”, presented at a Borland conference in 1994. The paper discussed development of Borland Quattro Pro for Windows, which was done under a process research project by AT&T Bell Laboratories. Coding was capped at eight people, and activities were centralized around daily meetings, resulting in productivity the researches believed to be “off the charts”. With a small team and ambitious goals, each developer was brought on board for their recognized expertise with no room for ‘warm bodies’. The researchers found the general purpose talents of all combined with the domain specific knowledge of each resulted in an effective and productive process. And the developers were happy, reporting “We are satisfied by doing real work.”[5]
The origin of what could be considered the most modern outline of software craftsmanship was The Pragmatic Programmer: From Journeyman to Masterby Andy Hunt and Dave Thomas. Hunt and Thomas took a modest approach, suggesting ways an individual developer can improve themselves and slowly improve their work environment at the place they are currently in. Hunt and Thomas emphasized the individuality of developers, but believed all good developers (what they called “Pragmatic Programmers”) could agree on the essentialness of a few certain characteristics: early or fast adopter of new technologies, inquisitive, critical thinker, realistic, and jack of all trades. They felt these characteristics lay the groundwork for productive technical skills development. It is of note that they largely did not address the issue of training new developers from scratch and instead focused on self-improvement. [13]
A year later, Pete McBreen suggested much more dramatic changes in his book, Software Craftsmanship: The New Imperative.McBreen wrote on his belief that the industry is desperately struggling with success, and that was due to fundamental problems with accepted notions within software engineering.He believed that the idea of engineering software was only appropriate for very large projects, and sought an alternative system that would be appropriate for smaller projects. McBreen based his idea on the historic craftsmanship model, which he felt would help emphasize his ideal of people before process.
In 2009, two of the first software craftsmanship conferences took place. Software Craftsmanship 2009 took place in London [11], follow by the Software Craftsmanship North America Conference [1]. Speakers discussed current best practices, new craftsmanship concepts, development strategies, and reflection on learning experiences and processes.
Most recently, members of the software craftsmanship community came together for a first attempt at capturing the essence of the movement in the “Manifesto for Software Craftsmanship”, which so far has been signed by nearly 4000 people:
“As aspiring Software Craftsmen we are raising the bar of professional software development by practicing it and helping others learn the craft. Through this work we have come to value:
Not only working software,
but also well-crafted software
Not only responding to change,
but also steadily adding value
Not only individuals and interactions,
but also a community of professionals
Not only customer collaboration,
but also productive partnerships
That is, in pursuit of the items on the left we have found the items on the right to be indispensable.” [2]
The Apprenticeship Model
Fred Brooks wrote concerning a proposal by Harlan Mills, that a team organized like a surgical team, “rather than a hog-butchering team”,is the more effective approach. That is to say, have one person focus on solving the problem with support from others, versus having everyone focus on solving the problem.In Brook’s idea of software development “surgical team”, the surgeon is an experienced individual who is in charge, writing the specifications, designing, coding, and testing the application. The copilot is able to do any of the same functions as the surgeon, sharing in the design, but is less experienced and is subordinate to the surgeon. An editor maintains documentation. An administrator handles standard management duties to not waste the time of the surgeon. The toolsmith makes sure the team has all the working tools it needs, by ensuring basic tools are available, and developing additional tools as necessary. The tester does the testing.
Ken Auer, the founder of a small software development shop, RoleModel software, has actually been doing apprenticeships at his shop for about a decade. Auer uses this to train new developers up to the point they can become a productive developer, and also uses their low cost labor as a competitive advantage:
We've also found that, in many projects, there is a lot of work that well-trained craftsman find simple. We have found that the age-old technique of apprenticeship is both an effective way to keep our craftsmen fresh (and new craftsmen up and coming), and also a cost-effective way to keep the costs down for our clients by having less experienced developers work on simpler task [sic] while still under supervision of the more experienced craftsmen. [19]
Auer’s apprenticeship system recognizes 3 phases of students, with an expectation of each student spending between 6 and 18 months per phase. A student would start as a novice, one who may have very little software training and understanding. Next step is an apprentice, one who has some software development understanding and can begin to produce some useful work, but still has a lot to learn. Lastly, an internis one who can produce quality work but still requires supervision and guidance. All students are unpaid or receive low pay, which is recognized as a trade off for the education and experience they are getting. [12]
Robert Martin, an award winning software development author, and president and CEO of Object Mentor, developed what he believed would be a good, fully implemented apprenticeship system for software developers.His idea is a team consisting of one master programmer, who has three journeymen, each of which have up to three apprentices.
In his model, the master has at least 20 years of experience and has a deep understanding of software systems. The master makes all the critical decisions and is the ‘boss’ of the project. The master is responsible for writing a lot of the code, and creating the architectural design. They will oversee all the code being added into the system, adding code only when it meets their strict standards. Besides all that, their chief responsibility is to drive the journeymen to ever higher standards of quality and productivity, by establishing and enforcing good practices. A master would be very highly paid (earning $150k to $250k a year or more).
The journeymen have at least five years of experience and can take direction from the master, and provide direction to their apprentices. The journeymen would go through code created by their apprentices, accepting what is good, and giving feedback on what isn’t. The journeymen are the main teachersand mentors of the apprentices, reviewing their work and providing feedback, along with other instruction. The journeymen would also write almost as much code as the master. A journeyman would be paid similar to a developer today (earning $50k to $100k a year).
The apprentice could have little to no experience, andwould be doing this apprenticeship instead of going to university or other educational institution. The apprentices would produce large quantities of work, much of which will not be accepted into the system, but from which they can get feedback and learn from. An apprenticeship would last between 2 and 4 years. Apprentices are paid very little, only around minimum wage, as it is considered they are getting an excellent education and experience which accounts for most of the ‘payment’ they receive for the work they produce.
Under Martin’s system, a team of 13 (one master, three journeymen, and nine apprentices), would cost about $650k, only averaging $50k per person, therefore giving about the same cost as a more traditional model of about the same number of developers at 50k each, but with more productivity and greater rewards to those who have worked longer. [14]
In a craftsmanship system, it is common for journeymen to travel and learn under various masters [20]. This concept was attempted by two Chicago based software development firms, 8th Light and Obtiva, swapped two senior consultants for five days. They tried this despite a series of legal and practical challenges, with grave concerns voiced by co-founder of 8th light Paul Pagel, that “the potential for things to go wrong is catastrophic”.Each of the two swapped consultants felt they each were made “just another member of the team”, seeing themselves as a contributor and also an agitator, stirring up ideas that each side took for granted. The founder of Obtiva, Kevin Taylor also felt it was a cost effective way to train and provide additional experience for his employees, as the only cost was the existing labor costs, with no conference fees, training class costs, or travel expenses. [22]
Craftsmanship Values
In the craftsmanship model, learning is recognized as a lifelong activity. [15] The passion required for this goes beyond a standard ‘9 to 5’ mentality. It is also understood that knowledge and experience are what Hunt and Thomas called “expiring assets”, because with changes in technologies and markets, different knowledge and skills are needed. One must regularly learn from a diverse range of topics, including both stable and risky, to expand one’s horizons and always be able to find the best solution for a given problem.[13]