Monday, March 7, 2011

"Exploratory Testing" is a pleonasm

It's time to stop talking about Exploratory Testing as a subset of testing.  There is no "exploratory testing" and "other testing".  All testing is exploratory.  If it's not exploratory, it's not testing.

Let's go back to Cem Kaner's definition of software testing:
Testing is an empirical investigation conducted to provide stakeholders with information about the quality of the product or service under test.

Let's take the dictionary definition of "empirical" investigation:
Based on, concerned with, or verifiable by observation or experience" (Oxford English Dictionary)

Combine:
Testing is an investigation based on, concerned with, or verifiable by observation or experience conducted to provide stakeholders with information about the quality of the product or service under test.

What is exploratory testing?
"a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project." - Cem Kaner

Or to put it a bit simpler: the simultaneous learning, design, and execution of tests.  The design of the next test is based on the observations of the last test, and the experience the tester has accrued during the testing process.

Exploratory testing is the opposite of scripted testing.  The design of scripted tests is not based on the outcome of the previous test.  The design of scripted tests is not based on, or concerned with observation or experience.  Writing and executing test scripts is not testing.

You could say: "oh, but while we are scripting, we have to explore the application so we know what to script."  Correct.  You are testing, and then scripting.  Two separate activities.

"Oh, but we often deviate from the script if we notice something strange, or want to check something else."  Correct.  You are following a script, and then sometimes you're testing.  Two separate activities.

Sometimes, the test script will be created using exploratory testing, and then will be executed with exploratory testing, and will be called scripted testing.  This just highlights the fact that there is no scripted testing and exploratory testing.  If you're testing, it's exploratory.  It's time to drop the "exploratory" from testing and time to drop the "testing" from scripted.

There is testing, and scripting.  Dropping exploratory from testing means we don't need to justify what we're talking about when we say "exploratory" testing, since most people, even other testers, are ignorant as to what that actually means.  And it should highlight the fact that when you go to an ISTQB course, or even many conferences, there isn't actually much talk about how to test.  And it should immediately nullify any attempt by ignorant testers who say "well, exploratory testing has good and bad sides" or "we don't do exploratory testing".  It sounds much more obviously silly to say "well, testing has good and bad sides" or "we don't do testing."

Seriously, let's get down to what testing really means and begin teaching and talking about it, without the distraction of having to explain ET every time.

79 comments:

  1. Hi Aaron,

    Firstly, fantastic post!

    Just to add to what you've already described fantastically in the difference between scripting and testing.

    The process of generating scripted tests without a system in place, e.g. up front via requirements specification, can also be exploratory testing testing, as long as they are not just copying and pasting rules from the requirements (I've seen this happen).

    Thanks for sharing, I really enjoyed reading this.

    Cheers,

    Darren.

    ReplyDelete
  2. I'd be interested to hear about other ways of generating scripts that doesn't include putting "verify that" in front of every bullet point in the requirements document.

    ReplyDelete
  3. You have some good points, but I can't stop thinking that we have a weak ET definition if it doesn't mean anything (more than "good".)
    Maybe the old one (technique/activity) was better?

    There is also one hole in your argument:
    What about unscripted tests that aren't based on the observations of the last test?

    Some things about other formats than "Verify that..." can be found at
    http://thetesteye.com/blog/2010/11/synthesizing-test-ideas/

    ReplyDelete
  4. >I can't stop thinking that we have a weak ET definition if it doesn't mean anything (more than "good".)

    Or...we now have a stronger "testing" definition?

    >What about unscripted tests that aren't based on the observations of the last test?

    Good point. The error there was saying in the first place that the next test is based on the outcome of the last test, as if that were always the case. I'll try and be more careful next time :)

    ReplyDelete
  5. Hi Aaron,

    You'll find if you write your test conditions from requirements specifications, you actually clarify the scope.

    Whenever I write up front tests for our developers to run prior to committing code, I spend probably half the time communicating with others, identifying gaps, risks and expanding on unclear requirements. I probably spend about 35% of the remaining time thinking and only 15% of my time actually writing any test conditions for the scripts.

    The big thing for us though is that we as testers only write these scripts, we don't execute them, they are essentially acceptance checks which the developer is required to check off before submitting a user story card. We write small use cases to act as regression tests with the view that most of these will later be automated.

    Remember being at the requirements stage the cost to fix is very low, as nothing has been built yet except for the specifications.

    If you haven't read it already I'd recommend having a read of my article on lean test case design (http://www.bettertesting.co.uk/content/?p=253) I go into more detail on the power of up-front test cases there.

    Cheers,

    Darren.

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. >>What about unscripted tests that aren't based on the observations of the last test?

    >Good point. The error there was saying in the first place that the next test is based on the outcome of the last test, as if that were always the case. I'll try and be more careful next time :)

    Actually, having thought about it some more, then, yes, your next test IS always decided by the outcome of the last test. If the outcome of the last test (and the one before that, and the one before that) yields no useful information, then you will use that outcome to decide to do something else, or apply a stopping heuristic and go to lunch.

    Apart from external factors (the building catching on fire, a phone call etc), and apart from scripting / checking, are there any times that the next test you execute is not based on observations of the last test? I'm trying to think of one...

    ReplyDelete
  8. "Apart from external factors (the building catching on fire, a phone call etc), and apart from scripting / checking, are there any times that the next test you execute is not based on observations of the last test?"

    Sure. Imagine that you wanted to act on ten test ideas. On the eighth, you observe something interesting. You consider investigating it right then, right there... but you want to act upon the last two test ideas, so you do.

    In your overall post, though, it seems to me as though you're treating exploratory and scripted as binary; a test is either exploratory or scripted. I don't think it's that clear. A test is scripted to the degree that a) the idea comes from someone else; and to the degree that b) the idea comes from some time in the past. A test is very scripted when the script, the other person, or the past determines the tester's next action. A test is exploratory to the degree that the tester him or her self has the authority to determine the next action, and to the extent that the test idea could come from the most recent instant.

    So, all testing that you do for a client is scripted to some degree, in that you're acting on a mission you've been given. All testing that a human performs is to some degree exploratory, since a human will alter his or her behaviour in ways that the script can't control or anticipate. People tend to work around problems with the script, rather than freezing and turning blue.

    I agree that all testing worthy of the name is exploratory, but I'm not convinced that it's helpful to say you're doing one or the other exclusively. Only machines can perform testing in a completely scripted way (I call that checking). I think it is important, though, to identify the degree to which you're doing one or the other; the degree to which you're free to do what you need to do in the moment; the degree to which you're responsible for it; and the degree to which you're being guided by other people and by the past.

    ReplyDelete
  9. Thanks Michael, that helps a lot in clarifying my thinking.

    ReplyDelete
  10. hahaha, I just happened upon this coincidentally: http://www.satisfice.com/blog/archives/496

    ReplyDelete
  11. Essay is that the contemplation of any quite topic wherever an author writes clarification that college papers. Writing essays are a sculpture of enjoying writing. Here you'll get a variety of forms of essays and writing essays service.

    ReplyDelete
  12. When I'm free of expense, I generally scan online journals on the web. I like the look and style of your web journal. I think you furthermore made an incredible showing of offering advantageous data. Keep it up!Best regards, custom writing service


    ReplyDelete
  13. Thank you a lot for taking the time for you personally to share such a nice info. I in fact choose to reading your post.
    Electrical Engineering Project Help

    ReplyDelete
  14. All things considered, nothing I can say except for incredible! I trust you'll make more helpful articles in the up and coming days.
    Project Management Online Help

    ReplyDelete
  15. Stats Homework Help
    That was an really awesome post and i like that.

    ReplyDelete
  16. Thanks a lot for the post. It has helped me get some nice ideas. I hope I will see some really good result soon.
    Management Assignment Help

    ReplyDelete
  17. Those who come to read your article will find lots of helpful and informative tips
    Online Homework Help

    ReplyDelete
  18. Great Information,it has lot for stuff which is informative.I will share the post with my friends.
    C# programming homework help

    ReplyDelete
  19. It is a smart presentation with a unique way of writing and written in well mannered techniques.
    matlab programming help

    ReplyDelete
  20. I personally like your post, you have shared good article. It will help me in great deal.
    C Sharp Homework Help

    ReplyDelete
  21. This is really great work. Thank you for sharing such a useful information here in the blog.
    matlab expert

    ReplyDelete
  22. Awesome work you have done here, I am very happy to read this nice post. You are a great writer and give us much information.
    Homework Help Online

    ReplyDelete
  23. Great info! I recently came across your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have
    affordable website pakistan

    ReplyDelete
  24. All things considered, thank you for making such helpful article.
    Nespresso and the U.S. Market

    ReplyDelete
  25. Great Information,it has lot for stuff which is informative.I will share the post with my friends.
    Buy Harvard Business Case Studies Solutions

    ReplyDelete
  26. No doubt! It will be truly hard enrolling! I wish you good fortunes!
    Case Studies Solutions

    ReplyDelete
  27. This is really great work. Thank you for sharing such a good and useful information here in the blog for students.

    ReplyDelete
  28. I genuinely appreciated understanding it. Sitting tight for some more incredible articles like this from you in the nearing days
    Case Study Homework Help

    ReplyDelete
  29. Dissertation Guidance Provides quality Online Dissertation Help for students.
    Harvard Case solution

    ReplyDelete
  30. I appreciate your efforts in preparing this post. I really like your blog articles.
    Harvard Case Studies Analysis

    ReplyDelete
  31. I loved the way you discuss the topic great work thanks for the share.

    ReplyDelete
  32. EViews Assignment Help
    Really i appreciate the effort you made to share the knowledge. This is really a great stuff for sharing. Keep it up . Thanks for sharing.

    ReplyDelete
  33. John arnold is an academic writer of the Dissertation-Guidance. Who writes quality academic papers for students to help them in accomplishing their goals.
    Finance Assignment Writing Help

    ReplyDelete
  34. HRM homework Help
    John arnold is an academic writer of the Dissertation-Guidance. Who writes quality academic papers for students to help them in accomplishing their goals.

    ReplyDelete
  35. John arnold is an academic writer of the Dissertation-Guidance. Whohelp with java programming
    writes quality academic papers for students to help them in accomplishing their goals.

    ReplyDelete
  36. John arnold is an academic writer of the Dissertation-Guidance. Who writes quality academic papers for students to help them in accomplishing their goals.
    programming help online

    ReplyDelete
  37. SAS Assignment Online
    Great Information,it has lot for stuff which is informative.I will share the post with my friends.

    ReplyDelete
  38. Great Information,it has lot for stuff which is informative.I will share the post with my friends.
    Social Issues Sociology Help

    ReplyDelete
  39. Operations Management Help
    This is great information for students. This article is very helpful i really like this blog thanks. I also have some information relevant for online dissertation help.

    ReplyDelete
  40. It is good to see such a helpful website! It has informative as well as interesting articles. I look forward to visit this site again and again. Students with the assignment writing problem could visit our site and can opt for our assignment writing help.
    Assignment Writing Service

    ReplyDelete
  41. Awesome work you have done here, I am very happy to read this nice post. You are a great writer and give us much information.
    ITT Capstone Project Help

    ReplyDelete
  42. DO My Matlab programming Homework
    This is an excellent post i seen.I have to thanks to you to share it. It is really what I wanted to see hope in future you will continue for sharing such a excellent post.

    ReplyDelete
  43. Thanks for sharing the info, keep up the good work going.... I really enjoyed exploring your site. good resource. 
    Student Essay Writing

    ReplyDelete
  44. Thanks for sharing the info,help with economics
    keep up the good work going.... I really enjoyed exploring your site. good resource. 

    ReplyDelete
  45. John arnold is an academic writer of the Dissertation-Guidance. Who writes quality academic papers for students to help them in accomplishing their goals.
    HRM Dissertation Help

    ReplyDelete
  46. Latent Function Sociology Homework Help
    John arnold is an academic writer of the Dissertation-Guidance. Who writes quality academic papers for students to help them in accomplishing their goals.

    ReplyDelete
  47. John arnold is an academic writer of the Dissertation-Guidance. Who writes quality academic papers for students to help them in accomplishing their goals.
    Depreciation Accounting Homework Help

    ReplyDelete
  48. Thanks for sharing this blog, it’s naturally great blog to get knowledge.

    ReplyDelete
  49. Dissertation Guidance Provides quality Online Dissertation Help for students.
    Programming Assignment Help Service

    ReplyDelete
  50. R Programming Homework Help
    I appreciate your efforts in preparing this post. I really like your blog articles.

    ReplyDelete
  51. Microsoft support team provides best solution of Microsoft products through Microsoft customer service number+1-800-805-7863.

    ReplyDelete
  52. Get the help through Microsoft customer service of windows, office and outlook.

    ReplyDelete
  53. Thank you for some other fantastic article. The place else may anybody get that kind of info in such a perfect way of writing? I’ve a presentation subsequent week, and I’m on the search for such info.
    view site

    ReplyDelete
  54. Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I'll be subscribing to your feed and I hope you post again soon. Big thanks for the useful info.
    check out this

    ReplyDelete
  55. The website is looking bit flashy and it catches the visitors eyes. Design is pretty simple and a good user friendly interface.
    medicinerf.ru

    ReplyDelete
  56. What a fantabulous post this has been. Never seen this kind of useful post. I am grateful to you and expect more number of posts like these. Thank you very much.
    downlodable music

    ReplyDelete
  57. Maintain the nice perform, My partner and i examine handful of content with this internet site and also I do believe your net website will be genuine intriguing and contains received sectors regarding great details.
    download warez

    ReplyDelete
  58. Science Channel’s are giving a complete knowledge to its viewers about every thing students write done dissertation on this subjects and show its importance.Do My Auditing Homework

    ReplyDelete
  59. Thanks for Nice and Informative Post. This article is really contains lot more information about This Topic.
    Do My C Programming Assignment

    ReplyDelete
  60. Hi buddy, your blog' s design is simple and clean and i like it. Your blog posts about Online Dissertation Help are superb. Please keep them coming. Greets!!
    assignment help experts

    ReplyDelete
  61. Good way of telling, good post to take facts regarding my presentation subject matter, which i am going to deliver in my college
    MBA Writing Service

    ReplyDelete
  62. My friend recommended this blog and he was totally right keep up the fantastic work!
    Content Writing Agency

    ReplyDelete
  63. This is really a great stuff for sharing. Keep it up .Thanks for sharing.
    java programming assignments

    ReplyDelete
  64. This is great information for students. This article is very helpful i really like this blog thanks. I also have some information relevant for online dissertation help.
    Help With Law Assignments

    ReplyDelete
  65. Get the dissertation writing service students look for these days with the prime focus being creating a well researched and lively content on any topic.
    Do My Finance Assignment

    ReplyDelete
  66. Well thanks for posting such an outstanding idea. I like this blog & I like the topic and thinking of making it right.
    Online Coursework Homework Help

    ReplyDelete
  67. Great Information,it has lot for stuff which is informative.I will share the post with my friends.
    Do My Psychology Assignments

    ReplyDelete
  68. I personally like your post, you have shared good article. It will help me in great deal. Marketing Research Assignments Help

    ReplyDelete
  69. This was a great and interesting article to read. I have really enjoyed all of this very cool information

    ReplyDelete
  70. I loved the way you discuss the topic great work thanks for the share Your informative post.

    ReplyDelete
  71. Blog gave us useful information to work. You have done an amazing job.
    theclick-to-download.blogspot.com

    ReplyDelete
  72. It is imperative that we read blog post very carefully. I am already done it and find that this post is really amazing.
    Get it!

    ReplyDelete
  73. This was a great and interesting article to read. Social Media Agency Pakistan I have really enjoyed all of this very cool information

    ReplyDelete
  74. Hi buddy, your blog' s design is simple and clean and i like it. Pakistan's Leading Digital & Social Media Agency Your blog posts about Online writing Help are superb. Please keep them coming. Greets!

    ReplyDelete