Where Does Exploratory Testing Fit?
James Bach, Satisfice, Inc.
(540) 631-0600
I grant permission to make digital or hard copies of this work for personal or classroom use, provided that (a) Copies are not made or distributed for profit or commercial advantage, (b) Copies bear this notice and full citation on the first page, and if you distribute the work in portions, the notice and citation must appear on the first page of each portion. Abstracting with credit is permitted. The proper citation for this work is Rapid Software Testing (course notes, Fall 2002), (c) Each page that you use from this work must bear the notice "Copyright (c) James Bach, ” or, if you modify the page, "Modified slide, originally from James Bach", and (d) If a substantial portion of a course that you teach is derived from these notes, advertisements of that course must include the statement, "Partially based on materials provided by James Bach." To copy otherwise, to republish or post on servers, or to distribute to lists requires prior specific permission and a fee. Request permission to republish from James Bach, .
Where Does Exploratory Testing Fit?
by James Bach,
This first appeared on as a column feature
If you, like me, find the exploratory approach to testing valuable (see "What is Exploratory Testing"), then the question arises when do you do it? How does it relate to the lifecycle? Yogita Sahoo sent me an insightful list of questions and comments about ET. I’ll be responding to them in future columns. For this installment, she writes “In my personal experience, if I start exploring things when I'm supposed to carry on with my allotted tests, the whole thing gets jumbled up. I can neither concentrate fully on ET nor on my regular test cases. That results in less productivity.”
A simple way to think of ET is concurrent test design and test execution. To help Yogita better, I want to use a more specific definition: Any testing is exploratory to the extent that the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests.
This definition includes pure exploratory testing, where you explore the product and design a test strategy and specific tests based on your understanding of your mission as a tester, but without any specific guidance. It includes chartered exploratory testing, where you have a specific assignment for what to test and what techniques to use, but no procedures. It also includes improvisational testing, where you elaborate on a test procedure or use it to inspire a set of related tests. It includes test procedure creation, too, inasmuch as you perform tests while documenting them.
The question is how to do ET well, not when to do it.
All testing done by humans is exploratory to some degree, because humans are not robots. That means the question is not when do you do exploratory testing, but rather how exploratory is the testing that you do, and howwell do you do it. When I consult about test process, I don’t suggest that my clients perform exploratory testing. Instead, I help them become aware of all the ET that they already do, which is probably mixed up with some form of scripted (or pseudo-scripted testing, whereby testers say they follow the procedures but don’t). Once I can get their exploratory testing to “come out of the closet”, I can help them improve their skills and overall test strategy so that they do ET, and any other testing, better.
Exploratory testing is all you have at the beginning…
ET fits at the beginning of the test project because test procedures don’t yet exist for the new technology being developed. Even if they existed, you have to learn the product (that requires exploring and questioning it) and the procedures would have to be reviewed and upgraded. The process of writing test procedures is exploratory. Watch anyone, or yourself, writing a test script, and you’ll see those thought processes at work.
…and it’s how you create diversity in tests later on.
ET fits into the middle of a test project, even when you have lots of scripted tests. Be exploratory in the sense that a tourist on a tour bus is exploratory. Let your allotted tests take you to visit different parts of the product, then improvise on the theme of those tests, briefly. Spend a few minutes working through variations of the tests, then get back on the tour bus and do the next scripted test.
Assure that there is enough variation and creativity in the test cycle.
Also, throughout the project, I would suggest that you question the value of the tests allotted to you, since no test process provides complete coverage. To improve the breadth and depth of your testing, consider allotting some time, maybe 20% or maybe 80% (there’s no universal right amount of time, other than what fulfills the mission of testing) per test cycle to pure exploratory testing. Pick one or more risk areas in the product and design and execute tests for that, seeking to find important problems fast and collecting information that will help the project evaluate the state of the product.
Another way to fit ET into a project is to dedicate a particular tester or test to continuous duty doing pure exploratory testing. I ran a team like that, once, and I once interviewed a fellow who ran such a team at Nortel. These teams are well trained, and work like a reconnaissance unit, scouring the product, and following up on rumors and risk areas.
Doing exploratory testing well requires skill, no matter when you do it.
Before I made it my goal to master the art of simultaneous test design and test execution, I felt confused and bewildered by the process, too. I eventually developed certain heuristics, notetaking protocols, and skill in modeling, reasoning, communication, and self-management that allow me to be productive under almost any circumstances. The process is creative, but it’s a teachable discipline fits anywhere testers are expected to use their minds.