Showing posts with label Exploratory Testing. Show all posts
Showing posts with label Exploratory Testing. Show all posts

November 16, 2009

Talkin' 'bout test in a differrent light

Pullin' out my big black book
Cause when I need a word defined that's where I look
So I move to the L's quick, fast, in a hurry
Threw on my specs, thought my vision was blurry
I look again but to my dismay
It was black and white with no room for grey
Ya see, a big "V" stood beyond my word
And yo that's when it hit me, that luv is a verb.

Words come easy but they don't mean much
When the words they're sayin' we can put trust in
We're talkin' 'bout love in a different light
And if we all learn to love it would be just right.

- DC Talk


The DC Talk song "Luv is a Verb" points out that love is something to be acted out. Real love is action, not just words and feelings. Love is expressed through action.

Like love, the word test is both a noun and a verb. Also like love, test requires action. Even the noun definitions for test describe action.


test

noun

  • trying something out to find out about it
  • any standardized procedure for measuring sensitivity or memory or intelligence or aptitude or personality, etc.
  • a set of questions or exercises evaluating skill or knowledge
  • the act of undergoing testing
  • the act of testing something
verb

  • put to the test, as for its quality, or give experimental use to
  • test or examine for the presence of disease or infection
  • examine someone's knowledge of something
  • show a certain characteristic when tested
  • achieve a certain score or rating on a test
  • determine the presence or properties of (a substance)
  • undergo a test
from Princeton WordNet

While I don't think we'll find anyone that argues that test is not a verb, people involved in software development seem to use it primarily as a noun. I have nothing against the many things we create that we call tests. Our test cases, test code, test charters, and whatever test things we create can be useful tools -- but they are not the test.


Let's think about test in a different light.

Several years ago, Shrini Kulkarni challenged me in questioning whether there can be such a thing as an automated test. I don't think I disagreed with Shrini. I've not been one to trust testing to machines, but I've been a fan of automation throughout my testing career. I've automated many testing tasks, but not believed I can automate the testing itself.

Earlier this year, Michael Bolton told me of a distinction he was thinking about between checking and testing. While I had no disagreement with this distinction, I wasn't thrilled with the terms. I wanted something more descriptive. I thought Michael was making a distinction I had been trying to make: a distinction between validation and investigation. I've since come to understand that Michael is making a slightly different distinction. Michael has recently written a series of blog posts better describing the Checking vs. Testing distinction. Michael has limited the scope of checking to observations and decisions rules that can be executed without sapience -- without a brain-engaged human. If something requires human sapience, it is testing, not checking.

Yesterday, an insightful tester, Lanette Cream, made a nice attempt at defining test on her blog. In her latest revision, she defines test as follows.



A test is
an action
which produces discoveries
that can be used to evaluate product quality.

I like that this definition identifies action, discovery, and evaluation as being core to testing. However, I'm thinking of pushing, or rather constraining, this just a bit further.

What if we were to say that the evaluation is the action and discovery is the goal?

A test would then be the sapient part of validation or investigation -- the thinking and learning that cannot be automated. All those other things we do to test are really support activities that help us evaluate.

Test is not a document. Test is not code. Test is not executing a program. Test is not applying a procedural decision rule. Test is not anything that can be done by a machine. Test is the act of evaluating. Test requires sapience.

Test is thinking and learning that leads to discovery. We may test by evaluating existing data. We may test by running experiments that produce new data. We may take the output of automated checks to test. We may provide what we learn as input to coding new automated checks. The test is the action we perform in our minds.

This may come across as nitpicking vocabulary. That's not my intent. My goal is not to limit anyone's definition of test, but rather to shed a different light on what I believe sets testing apart from checking, and gives both checking and testing value.

If a check fails in a forest and no one is around to hear it, does it make a sound?

The true value of our checking and testing is in the mind of a sapient tester. What value is there in all the things we call checks and tests without a tester (whatever their role or title) evaluating information and learning?



Test is sapient evaluation that leads to discovery
.
I'm not quite comfortable with this. I want the emphasis to be on the sapient activity; and not generating and collecting data to support the thinking without ignoring that it is a necessary part of testing.

Regardless of where we shine the light or draw lines, let's keep in mind that test is a verb.

What do you think? Testing of my half-baked ideas is welcome and appreciated.

September 30, 2008

The Antonym of Testing


"... one usually encounters a definition such as, 'Testing is the process of confirming that a program is correct. It is the demonstration that errors are not present.' The main trouble with this definition is that it is totally wrong; in fact, it almost defines the antonym of testing."

- Glenford Myers,

Software Reliability: Principles & Practices, 1976

People keep telling me that testing is a validation activity -- that the purpose of testing is to validate that the software meets all the specifications, has no errors, meets performance SLAs, meets expectations of anonymous users, or some other lofty goal.

I read about testing processes designed to validate software. I use testing tools built to support validation. I listen to service companies pitch testing services to validate software. I read about testing metrics built on the assertion that software systems can be proved correct. I attend testing presentations explaining the presenters' best practices for validation.

The trouble is that we cannot prove software correct. We cannot prove the absence of bugs. We cannot test every possible state and input. We cannot evaluate every possible output. We cannot fully understand the desires of stakeholders. We cannot prove that customers will be happy. We cannot prove that a software product will solve the problems it was built to solve. If all this were possible, I suspect insurance companies would find a way to make a profit selling software quality insurance.

"If you think you can fully test a program without testing its response to every possible input, fine. Give us a list of your test cases. We can write a program that will pass all your tests but still fail spectacularly on an input you missed. If we can do this deliberately, our contention is that we or other programmers can do it accidentally."

- Cem Kaner, Jack Falk, and Hung Quoc Nguyen,
Testing Computer Software, Second Edition, 1999

Now, thirty-two years since Glenford Myers called testing to prove correctness the opposite of testing, we're surrounded by testing practices and tools based on proving correctness. The myth of proving correctness is alive and well.

Activities designed to try to prove correctness are the antonym of testing.

So if testing is not validation, what is testing? Testing is investigation; and communicating useful information about quality to decision makers.

"Testing is the process by which we explore and understand the status of the benefits and the risk associated with release of a software system."

- James Bach,
James Bach on Risk-Based Testing, STQE Magazine, Nov 1999


"Testing is done to find information. Critical decisions about the project or the product are made on the basis of that information."

- Cem Kaner, James Bach, Bret Pettichord,
Lessons Learned In Software Testing: A Context-Driven Approach, 2002


"A software tester’s job is to test software, find bugs, and report them so that they can be fixed. An effective software tester focuses on the software product itself and gathers empirical information regarding what it does and doesn’t do. This is a big job all by itself. The challenge is to provide accurate, comprehensive, and timely information, so managers can make informed decisions."

- Brett Pettichord,
Don't Become the Quality Police, StickyMinds.com, 2002


Once we admit that we cannot prove the software correct, we can refocus our efforts on finding useful quality-related information. Instead of pretending to assure quality or validate correctness, we can gather and communicate useful information. Investigate the software. Find information about threats to the quality of the systems under investigation. Communicate that information in terms that matter to stakeholders. Help managers make informed decisions.

May 4, 2008

Terrified by Improvisation

[Improvisational comedy] involves people making very sophisticated decisions on the spur of the moment, without benefit of any kind of script or plot. That's what makes it so compelling -- and to be frank -- terrifying. ... What is terrifying about improv is the fact that it appears utterly random and chaotic. It seems as though you have to get up onstage and make everything up, right there on the spot. But the truth is that improv isn't random or chaotic at all. ... Improv is an art form governed by a set of rules... How good people's decisions are under the fast-moving, high-stress conditions of rapid cognition is a function of training, rules, and rehearsal.


Now, reread the quote above and replace improv with exploratory testing. See a connection? Just as improvisational theater may appear to be random and chaotic (although entertaining) to the ignorant, exploratory testing can appear to be random and chaotic to those that have been taught to rely on scripts. Good improv and exploratory testing is neither. There are rules -- heuristics.

Heuristics are rules of thumb that help solve problems. In improvisational comedy, there are rules. These are not hard rules that guarantee comedy. These are rules that skilled improv actors can use to help keep things funny. Sometimes these rules don't work and actors have to adapt. And, because they aren't following a script, they can adapt when things don't work out. Some parts of improv are scripted. I am a fan of the television show Whose Line is it Anyway. Each comedy sketch in this show is given a structure (think charter) to direct the improvisation. This structure defines and restricts (think script) specific components of each sketch while leaving the bulk of the activity open to each actor to adapt to what happens as the sketch plays itself out. While we do not see it on screen, I suspect that a great deal of training, rules, and rehearsal went into the production of Whose Line. The shows did suffer from an occasional guest participant (usually a trained script actor) that was not as skilled at improv as the regulars. However, other guests (sometimes not actors) who understand the rules of improv have helped produce some of the funniest sketches.

Good exploratory testing works in the same way. Skilled exploratory testers set out with a charter -- a goal for each testing session. Skilled exploratory testers use heuristics to help them learn about the systems they test. Skilled exploratory testers practice.

Improvisation can have scripted aspects and rules that guide it. It is not chaotic and random. It is smart people using simple rules to make quick decisions and adapt to a changing environment under pressure.

November 17, 2007

Finally, a tester certification test that I might like!


I am not a fan of any of the current software tester certification programs. Perhaps it is because I take the word certification too literally -- which means that I expect it to have real meaning. When I think of certification, I usually think in lines with the IEEE's definition of certification.
certification

The process of confirming that a system or component complies with its specified requirements and is acceptable for operational use.
Perhaps my thinking is biased by my past IV&V testing work. If one can go to a weekend class and become certified, then I question the value of that certification. I believe that certifications based on ability to memorize terms and practices free of context are of little value -- and may do more harm than good. Yet, the purpose of this blog post is not to provide arguments against the current crop of tester certification options. If you'd like to see some concerns about certification, read the following.
If we must have a test certification based on a computer-scored multiple-false test, then I think I have found a test that is better than any others I've yet seen. This test covers many aptitudes and skills that I believe are important for testers:

  • Precision reading
  • Requirements interpretation
  • Persistence
  • Exploratory learning
  • Domain knowledge
  • Heuristic based problem solving
  • A good sense of humor
  • Critical analysis
  • Looking at problems from many angles
  • Recognizing context
  • Understanding the software platform
  • Agility
  • Troubleshooting
  • Working under pressure
  • A good memory, or good note taking skills
  • and don't be easily offended
Give it a try.



If I had to select testers based on passing a test, I think I'd take someone that has gotten further than I in this quiz over someone with a software tester certification. I believe this quiz is a better measure of whether or not one is "acceptable for operational use". :)

How far can you get?

And did you notice the bug on question 30?

July 12, 2007

Exploratory Scripted Automated Manual Testing


Exploratory testing is often said to be the opposite of scripted testing. Automated testing is often said to be the opposite of manual testing. Instead of selecting one or the other, I find it helpful to look at these as opposite ends of spectra. I believe good testing contains components spread throughout multiple spectra.

I think we get into trouble when we apply these labels to “testing” as a whole. I believe all software testing has some aspects that are exploratory, some scripted, some automated*, and some manual. Hopefully all are driven by sapience.

Here are some things I remember about one of the best testing projects in my experience:

• Organizational testing processes were well documented
• Requirements were well documented
• We had enforceable quantitative requirements
• Tests were well planned
• Test plans were well documented
• Test cases were well defined and traceable to requirements
• Tests were well scripted
• Automation was an integral part of the testing

All of the above look like attributes of scripted, plan everything first, testing. We did a lot of planning and scripting for this testing. However, the following things are also true for this testing project:

• Process and documentation requirements were adaptable to the context
• Test "script" execution was mostly exploratory and mostly manual
• Bugs were only reported and officially logged if they were not fixed quickly
• Many requirements were qualitative
• Automation was “driven” by human testers


This scripted exploratory automated manual testing worked very well in its context. It was both the most scripted and most exploratory testing I can think of at the moment: it was the best testing.

Exploration does not mean leaving the map at home. Exploration is not the same as wandering. Testing was well planned while being exploratory in nature and practice. Human beings were involved at every step of the test planning and execution. Introducing automation reduced each iteration’s the test execution time from weeks to days without removing the human tester.

This testing was exploratory. It was dynamic. It was also well
scripted. The combination of static and dynamic components was central to the success of the project.

Instead of thinking of performing exploratory or scripted or automated or manual tests, find the balance of the four to create testing that works best in each situation.


* If you think no aspect of your testing is automated, try testing the software without a computer. And now that I wrote that, I am thinking of software tests that may not require a computer or software. It all depends on context. :)

July 8, 2007

Too much testing?

In a recent blog post, Jason Gorman provides some thoughts about the following question:

How much testing is too much?
To me, this is like asking "how much cheese would it take to sink a battleship?" There probably is an answer - a real amount of cheese that really would sink a battleship. But very few of us are ever likely to see that amount of cheese in one place in our lifetimes.
As Jason states, we may never encounter too much testing. However, I believe that we testers often include too much repetition in our testing and miss many bugs that are waiting to be discovered. This becomes especially likely when we limit our testing to scripted testing or put our test plans in freezers. Repeating scripted tests -- whether manual or automated -- is unlikely to find new bugs. To find new bugs, we testers need to step outside the path cleared by previous testing and explore new paths through the subject of our testing.


Executing the same tests over and over again is like a grade school teacher giving a class the same spelling list and test each week. The children will eventually learn to spell the words on the list and ace the test; but this does not help them learn to spell any new words. At some point, the repeated testing stops adding value.
If you have men who will only come if they know there is a good road, I don't want them. I want men who will come if there is no road at all.
Like Dr. Livingstone, I want testers that are willing to explore paths that have not been trampled by the testers that went before them. I want automated tests to go out like probe droids and bring back useful information. I want each manual tester on a team to think and explore the system under test in a different manner than the rest. There is a time and place for repeatable consistency, but that's just a part of testing. Real human users don't follow our test scripts. I don't want testing to be limited to testers and robots (automation) that follow scripts through pre-cleared paths.


Want to learn more about exploratory testing?


Try exploring the web with Google or your favorite search engine: