What Makes an Expert Tester?
James Bach, Satisfice, Inc.
(540) 631-0600
I grant permission to make digital or hard copies of this work for personal or classroom use, provided that (a) Copies are not made or distributed for profit or commercial advantage, (b) Copies bear this notice and full citation on the first page, and if you distribute the work in portions, the notice and citation must appear on the first page of each portion. Abstracting with credit is permitted. The proper citation for this work is Rapid Software Testing (course notes, Fall 2002), (c) Each page that you use from this work must bear the notice "Copyright (c) James Bach, ” or, if you modify the page, "Modified slide, originally from James Bach", and (d) If a substantial portion of a course that you teach is derived from these notes, advertisements of that course must include the statement, "Partially based on materials provided by James Bach." To copy otherwise, to republish or post on servers, or to distribute to lists requires prior specific permission and a fee. Request permission to republish from James Bach, .
What Makes an Expert Tester?
Published: Interface Webzine (Microsoft Internal) 11/99
An interview with James Bach, founder of Satisfice, Inc., a software test consulting and training company.
Software quality assurance great and prolific author James Bach gave an MSTE Talk in August on how an outstanding tester thinks. He later revealed to Interface how his own eclectic intellectual curiosity taught him that the true expert tester is an honorary Epistemologist or lifelong philosopher of knowledge.
Interface: What makes an expert tester?
James Bach: The answer to that question depends on many factors. Somebody may be an expert tester in one context, but not an expert tester in another context. Are we talking about Microsoft in the Outlook Group? Or Boeing in the Avionics Group? The ability to perform and function as an expert in those different environments is quite different. But I would say that there is one thing in common with expert testers everywhere. Expert testers are good at Epistemology. Epistemology is the study of how we know what we know. When I swap war stories with other testers, we tell each other how we avoided being fooled or we laugh about how we were fooled. Expert testers know how to escape from tunnel vision.
Interface: With so many testers at Microsoft, I would imagine that there’s a lot of expertise to share through networking.
James Bach: Oh yes. But I bet testers in one group don’t get a lot of contact with testers in other groups, and often testers are so focused on what they have to do today that it’s hard to step back and think about their general testing skills and how someone in another team might have useful advice. That’s why, in my testing class, I try to show students how they can successfully network with other testers even in a turbulent environment like Microsoft, and how rapidly they can improve themselves if they learn to do that.
Interface: How do you rate Microsoft testing?
James Bach: I would have to answer differently depending on the team. There is tremendous variation from team to team. Microsoft strikes me as a sort of like Europe during the Renaissance. In other words, you’ll find test teams that are like tiny little farm communities that don’t know anything about the outside world and you’ll find test organizations like major urban centers, sophisticated and cosmopolitan. Experience, skills, and practices vary greatly across the company. There are those who say variation is a bad thing. I think they’re flat wrong. I suspect that the variability of Microsoft testing practices is a direct consequence of the fact that each business unit has local control over its process. Local control encourages individual initiative and responsibility, which, in my experience, is the key to flexibility and innovation.
Interface: What could we do better?
James Bach: It would be very easy to say I think you should give your testers a year of paid, intensive hands-on training at The Hawaiian Institute of Testing Excellence but I guess that’s not realistic. How about this: Many times testers do not spend enough time and energy to train themselves in the technologies they are testing, and in the processes of software development. That leads to a knowledge gap between the developers and testers, which makes it hard for developers to respect the testers. I find that when a testers work hard to understand the rudiments of the technologies they’re is testing, the developers appreciate that and tend empathize better with the challenges of testing. Testers then gain more influence in the development process.
I suggest approaching the developer and saying, “ I’m here to make you look good! I’m here to make you a star, I’m here to protect you, I’m like your personal bodyguard! I’m going to find problems in your work, before anyone else gets to see them.” It’s a good bet that a lot of testers at Microsoft would benefit from a stronger connection to the development team.
Interface: You’ve also talked about the importance of watching yourself work, what do you mean by that?
James Bach: I mean every hour or so, stop and say, “What am I doing now? Why am I doing this? Why am I not doing something else?” So much about testing is silent and invisible unless you make a concerted effort to wake up to the dynamics of your own personal process. Here’s a little exercise. Kids, you can try this at home. Open WordPad, type some text, set the point size of that text to 9. Type another line of text, set it to 12 point. Now tell me, did WordPad properly size the text? If your answer is yes (and that’s probably what my answer would be), ask yourself how you know that. Back up and trace your thought process, if you can. Determine exactly how you justify to yourself that WordPad is functioning properly, and how you could be mistaken. What are the limits of your knowledge? How do you decide that your answer is good enough? When I did this exercise for the first time, I discovered I was using a set of unconscious heuristics to evaluate WordPad. Now that I’m aware of those principles, I can talk about them, teach them, and debate them with my friends. I’ve learned some very important lessons by watching myself test.
Interface: You also talked about sleeping lightly on the truth. What did you mean by that?
James Bach: There’s a marvelous tension we maintain as testers between reflex and reflection. This tension is a basic part of our psychomotor system, actually. Once, I was at Universal Studios, at the Terminator 2-3D ride, when something came flying at my eye. I didn’t think to myself, “What’s that? I would not like that to hit my eye.” No. I blinked. I ducked. Simultaneously, my mind reflected on the matter of the flying object. The bureaucracy of my higher faculties processed analyzed the images and figured out what the object was. It was not a stick or a bee, but rather a 3-D illusion. A moment later, my conscious mind reprogrammed my reflexes so that the next time a scary terminator got in my face my ducking reflex was not triggered. Now, pre-defined plans and process are like organizational reflexes. Sleeping lightly on the truth is a principle that says, by all means, have standard operating procedures, have pat [M1]answers for things. But also set up another process that evaluates how those processes fit with the situation at hand. It’s okay to have opinions and conclusions. It’s okay to trust them and it’s okay not to reconsider them every day. But don’t sleep so soundly on them that nothing at all would cause you to wake up and rethink.
Interface: What books do you think testers should read?
James Bach: If we restrict our knowledge of testing, our education about testing, only to testing books, then we are trying to study the ocean by looking at a bucket of water.
My first boss at Apple Computer said, “James, study fields other than testing; see if there are any ideas you can apply here.” I now know that was wonderful advice, but at first I couldn’t find any ideas—other fields looked nothing like testing. Then a colleague, Ed Yourdon, suggested that I study general systems theory. That was the key. I began to learn how to analyze things in terms of general dynamics and patterns. Now I feel like almost anything I read is helpful to testers. Here are a few books that I would recommend to get started in that direction. Quality Software Management, Vol. 1: Systems Thinking, Rethinking Systems Analysis and Design, and An Introduction to General Systems Thinking, all by Gerald M. Weinberg, also read The Fifth Discipline, by Peter Senge. Since testing is thinking, I recommend reading about thinking. One book that blew me away is Cognition in the Wild, by Ed Hutchins, a study of the way naval navigation teams work together and basically how groups of people think. It taught me that expertise is socially situated, even socially constructed, and thus expertise may be meaningless outside of a social context. Oh, and Conjectures and Refutations by Karl Popper, of course. I call Karl Popper the patron saint of software testers. He thought he was writing about testing scientific theories, but his ideas are just as applicable to us.
Interface: When I first started testing, I read some books by Karl Popper. In 1964 before anybody even thought about software testing. He’s writing about quality and you can see direct correlation.
James Bach: Yes, we’re back to Epistemology. Popper’s work is part of a philosophical tradition that goes back at least to Aristotle. Software testing does not belong to Computer Science. Software testing belongs to the philosophy of science itself.
[M1]1The transcriber couldn’t make out this word. We can ask James?