Monday, March 7, 2011

"Exploratory Testing" is a pleonasm

It's time to stop talking about Exploratory Testing as a subset of testing.  There is no "exploratory testing" and "other testing".  All testing is exploratory.  If it's not exploratory, it's not testing.

Let's go back to Cem Kaner's definition of software testing:
Testing is an empirical investigation conducted to provide stakeholders with information about the quality of the product or service under test.

Let's take the dictionary definition of "empirical" investigation:
Based on, concerned with, or verifiable by observation or experience" (Oxford English Dictionary)

Combine:
Testing is an investigation based on, concerned with, or verifiable by observation or experience conducted to provide stakeholders with information about the quality of the product or service under test.

What is exploratory testing?
"a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project." - Cem Kaner

Or to put it a bit simpler: the simultaneous learning, design, and execution of tests.  The design of the next test is based on the observations of the last test, and the experience the tester has accrued during the testing process.

Exploratory testing is the opposite of scripted testing.  The design of scripted tests is not based on the outcome of the previous test.  The design of scripted tests is not based on, or concerned with observation or experience.  Writing and executing test scripts is not testing.

You could say: "oh, but while we are scripting, we have to explore the application so we know what to script."  Correct.  You are testing, and then scripting.  Two separate activities.

"Oh, but we often deviate from the script if we notice something strange, or want to check something else."  Correct.  You are following a script, and then sometimes you're testing.  Two separate activities.

Sometimes, the test script will be created using exploratory testing, and then will be executed with exploratory testing, and will be called scripted testing.  This just highlights the fact that there is no scripted testing and exploratory testing.  If you're testing, it's exploratory.  It's time to drop the "exploratory" from testing and time to drop the "testing" from scripted.

There is testing, and scripting.  Dropping exploratory from testing means we don't need to justify what we're talking about when we say "exploratory" testing, since most people, even other testers, are ignorant as to what that actually means.  And it should highlight the fact that when you go to an ISTQB course, or even many conferences, there isn't actually much talk about how to test.  And it should immediately nullify any attempt by ignorant testers who say "well, exploratory testing has good and bad sides" or "we don't do exploratory testing".  It sounds much more obviously silly to say "well, testing has good and bad sides" or "we don't do testing."

Seriously, let's get down to what testing really means and begin teaching and talking about it, without the distraction of having to explain ET every time.

17 comments:

  1. Hi Aaron,

    Firstly, fantastic post!

    Just to add to what you've already described fantastically in the difference between scripting and testing.

    The process of generating scripted tests without a system in place, e.g. up front via requirements specification, can also be exploratory testing testing, as long as they are not just copying and pasting rules from the requirements (I've seen this happen).

    Thanks for sharing, I really enjoyed reading this.

    Cheers,

    Darren.

    ReplyDelete
  2. I'd be interested to hear about other ways of generating scripts that doesn't include putting "verify that" in front of every bullet point in the requirements document.

    ReplyDelete
  3. You have some good points, but I can't stop thinking that we have a weak ET definition if it doesn't mean anything (more than "good".)
    Maybe the old one (technique/activity) was better?

    There is also one hole in your argument:
    What about unscripted tests that aren't based on the observations of the last test?

    Some things about other formats than "Verify that..." can be found at
    http://thetesteye.com/blog/2010/11/synthesizing-test-ideas/

    ReplyDelete
  4. >I can't stop thinking that we have a weak ET definition if it doesn't mean anything (more than "good".)

    Or...we now have a stronger "testing" definition?

    >What about unscripted tests that aren't based on the observations of the last test?

    Good point. The error there was saying in the first place that the next test is based on the outcome of the last test, as if that were always the case. I'll try and be more careful next time :)

    ReplyDelete
  5. Hi Aaron,

    You'll find if you write your test conditions from requirements specifications, you actually clarify the scope.

    Whenever I write up front tests for our developers to run prior to committing code, I spend probably half the time communicating with others, identifying gaps, risks and expanding on unclear requirements. I probably spend about 35% of the remaining time thinking and only 15% of my time actually writing any test conditions for the scripts.

    The big thing for us though is that we as testers only write these scripts, we don't execute them, they are essentially acceptance checks which the developer is required to check off before submitting a user story card. We write small use cases to act as regression tests with the view that most of these will later be automated.

    Remember being at the requirements stage the cost to fix is very low, as nothing has been built yet except for the specifications.

    If you haven't read it already I'd recommend having a read of my article on lean test case design (http://www.bettertesting.co.uk/content/?p=253) I go into more detail on the power of up-front test cases there.

    Cheers,

    Darren.

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. >>What about unscripted tests that aren't based on the observations of the last test?

    >Good point. The error there was saying in the first place that the next test is based on the outcome of the last test, as if that were always the case. I'll try and be more careful next time :)

    Actually, having thought about it some more, then, yes, your next test IS always decided by the outcome of the last test. If the outcome of the last test (and the one before that, and the one before that) yields no useful information, then you will use that outcome to decide to do something else, or apply a stopping heuristic and go to lunch.

    Apart from external factors (the building catching on fire, a phone call etc), and apart from scripting / checking, are there any times that the next test you execute is not based on observations of the last test? I'm trying to think of one...

    ReplyDelete
  8. "Apart from external factors (the building catching on fire, a phone call etc), and apart from scripting / checking, are there any times that the next test you execute is not based on observations of the last test?"

    Sure. Imagine that you wanted to act on ten test ideas. On the eighth, you observe something interesting. You consider investigating it right then, right there... but you want to act upon the last two test ideas, so you do.

    In your overall post, though, it seems to me as though you're treating exploratory and scripted as binary; a test is either exploratory or scripted. I don't think it's that clear. A test is scripted to the degree that a) the idea comes from someone else; and to the degree that b) the idea comes from some time in the past. A test is very scripted when the script, the other person, or the past determines the tester's next action. A test is exploratory to the degree that the tester him or her self has the authority to determine the next action, and to the extent that the test idea could come from the most recent instant.

    So, all testing that you do for a client is scripted to some degree, in that you're acting on a mission you've been given. All testing that a human performs is to some degree exploratory, since a human will alter his or her behaviour in ways that the script can't control or anticipate. People tend to work around problems with the script, rather than freezing and turning blue.

    I agree that all testing worthy of the name is exploratory, but I'm not convinced that it's helpful to say you're doing one or the other exclusively. Only machines can perform testing in a completely scripted way (I call that checking). I think it is important, though, to identify the degree to which you're doing one or the other; the degree to which you're free to do what you need to do in the moment; the degree to which you're responsible for it; and the degree to which you're being guided by other people and by the past.

    ReplyDelete
  9. Thanks Michael, that helps a lot in clarifying my thinking.

    ReplyDelete
  10. hahaha, I just happened upon this coincidentally: http://www.satisfice.com/blog/archives/496

    ReplyDelete
  11. From Virdesi: At P&R Bakery and Cafe, our slogan "Stuff as good as your Momma's" means exactly that. bersavto.ru With the new name came the addition of high scores.

    ReplyDelete
  12. By correctly matching the beat of bouncing Thumpies, the player progresses through a song. download apps You can Undo a move by pressing the undo button on top.

    ReplyDelete
  13. The clock as a time machine has become a decisive machine in our civilization. ilovesharewares.com Users may answer multiple concurrent chats and respond to Mobile Text Connect (MTC) inquiries from prospective customers.

    ReplyDelete
  14. Are you on a diet and you want to keep track of your progress. 4demotivator.ru It presents different stretching exercises for your wrists, elbows, shoulders, neck, body, legs, and arms.

    ReplyDelete
  15. ""Like the new voice in the newsest version and the Thai is now much better :) just what I needed, keep up the great work. Welcome to my site. NO REGISTRATION Use guest mode to play poker without registration.

    ReplyDelete