Terrified by Improvisation

[Improvisational comedy] involves people making very sophisticated decisions on the spur of the moment, without benefit of any kind of script or plot. That's what makes it so compelling -- and to be frank -- terrifying. ... What is terrifying about improv is the fact that it appears utterly random and chaotic. It seems as though you have to get up onstage and make everything up, right there on the spot. But the truth is that improv isn't random or chaotic at all. ... Improv is an art form governed by a set of rules... How good people's decisions are under the fast-moving, high-stress conditions of rapid cognition is a function of training, rules, and rehearsal.

- Malcolm Gladwell, Blink: The Power of Thinking Without Thinking

Now, reread the quote above and replace improv with exploratory testing. See a connection? Just as improvisational theater may appear to be random and chaotic (although entertaining) to the ignorant, exploratory testing can appear to be random and chaotic to those that have been taught to rely on scripts. Good improv and exploratory testing is neither. There are rules -- heuristics.

Heuristics are rules of thumb that help solve problems. In improvisational comedy, there are rules. These are not hard rules that guarantee comedy. These are rules that skilled improv actors can use to help keep things funny. Sometimes these rules don't work and actors have to adapt. And, because they aren't following a script, they can adapt when things don't work out. Some parts of improv are scripted. I am a fan of the television show Whose Line is it Anyway. Each comedy sketch in this show is given a structure (think charter) to direct the improvisation. This structure defines and restricts (think script) specific components of each sketch while leaving the bulk of the activity open to each actor to adapt to what happens as the sketch plays itself out. While we do not see it on screen, I suspect that a great deal of training, rules, and rehearsal went into the production of Whose Line. The shows did suffer from an occasional guest participant (usually a trained script actor) that was not as skilled at improv as the regulars. However, other guests (sometimes not actors) who understand the rules of improv have helped produce some of the funniest sketches.

Good exploratory testing works in the same way. Skilled exploratory testers set out with a charter -- a goal for each testing session. Skilled exploratory testers use heuristics to help them learn about the systems they test. Skilled exploratory testers practice.

Improvisation can have scripted aspects and rules that guide it. It is not chaotic and random. It is smart people using simple rules to make quick decisions and adapt to a changing environment under pressure.

Discombobulation Testing

I do not like software that discombobulates users. I find that software error handling and reporting often confuses users more than it helps. Several months back, I created and wrote about a mnemonic of guideword heuristics that I have been using for testing error handling and reporting. I have found this useful in my own testing and have received some positive feedback from others that are using it. I created the following PowerPoint show to help demonstrate how I use the mnemonic. Take a look. It runs for just under seven minutes.

FAILURE.pps
(For those that don't have PowerPoint: try an experimental Flash version here.)

The next time you encounter an error, try the FAILURE heuristic.

• F Does the error detecting, reporting, and handling function as expected?
• A Is the error reported at the appropriate time in an appropriate manner?
• I Is the impact to the user communicated?
• L Is the error appropriately logged? ... or not logged if that is desired?
• U Is the error appropriately reported in the user interface in terms that the user understands?
• R Is the user told how to recover from the error condition?
• E Does the error reporting evoke the desired emotions? Could changes in the error reporting and handling improve the users emotional response?

Merry Christmas, and Happy Testing!

Solving Intractable Problems

"A solution to a given problem is called optimal if one can prove that no better solution exists. Some skeptics might ask, Why should intuition rely on a rule of thumb instead of the optimal strategy? To solve a problem by optimization -- rather than by a rule of thumb -- implies both that an optimal solution exists and that a strategy exists to find it. Computers would seem to be the ideal tool for finding the best solution to a problem. Yet paradoxically, the advent of high-speed computers has opened our eyes to the fact that the best strategy often cannot be found." - Gerd Gigerenzer, Gut Feelings: The Intelligence of the Unconscious

An intractable problem is a problem for which there is no efficient means of solving. These aren't necessarily problems for which there is no solution. Instead, these are problems that take too long to analyze all the options.

One of the biggest challenges in testing software is to select useful tests from infinite options. Even finite sets of options can create intractable problems.

For example, state model-based test automation generally requires creation of an explicit finite model. Models are less complex than the real thing -- if they aren't, they are copies, not models. Model-based test automation can be used to generate and execute tests for many more paths through a computer program than people are willing or able to try. However, testing all paths in a model for a non-trivial program can easily become an intractable problem.

Gerd Gigerenzer demonstrates this with a challenge in his book Gut Feelings. Gigerenzer asks readers to find the shortest route to visit 50 cities starting and ending at the same city. Think you can find the solution? There are only 12 different routes to visit 5 cities. So how many combinations would you need to check to visit 50 cities? According to Gigerenzer, there are approximately 300,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000 possible routes to visit 50 cities. Even with the help of computers, we do not have the time to calculate the best route.

Even relatively simple computer programs have significantly more data and path possibilities -- especially when we consider retracing paths as part of a larger path. There are potentially infinite path possibilities though even the simplest programs. (As I type each letter of this article into my computer, I am selecting an input from many more than 50 options.)

The problem of visiting 50 cities is solvable if we do not insist on finding the shortest route. Finding the shortest route and trying every possible path through a computer program are not really required to find a satisfactory solution.

Intractable problems like this require that we not consider all possibilities. Trying to consider all the options distracts us from what could be productive testing. Instead of considering all options, we only need to consider enough options to be satisfied. Otherwise, we will spend more time analyzing the possibilities than we do testing. Sometimes we just need to stop analyzing and go with our gut feelings.

"Gut feelings are based on surprisingly little information that makes them look untrustworthy in the eyes of our superego, which has internalized the credo that more is always better. Yet experiments demonstrate the amazing fact that less time and information can improve decisions. Less is more means there is some range of information, time, or alternatives where a smaller amount is better ..." - Gerd Gigerenzer, Gut Feelings: The Intelligence of the Unconscious

Gut feelings -- this unconscious thinking -- is based on heuristics, or rules of thumb. A heuristic is a problem solving device that helps narrow intractable problem into solvable problems. Heuristics are not laws. They do not apply to all situations. Two or more useful heuristics may even contradict one another. Gigerenzer states that the value of a heuristic is dependent on the context in which it is applied.

I have encountered many good testers that just seem to have a gift for breaking software. I once thought that this gift was something that could not be taught to others. I was wrong. Writers like, and including, Gerald Weinberg have shown me that teaching such problem solving is possible. Testing coaches James Bach and Michael Bolton have shown me that intuitive testing can be taught.

The secret to teaching such testing is to help good testers identify the heuristics that they use while testing. Once they are identified, they can be stated in such a way that they make sense to others. Then they can be named to make them easy to remember.*

As (or after) you test, think about why you do what you do and write it down. Don't think in terms of absolutes and programmable logic. Think in terms of rules of thumb. Then give each heuristic a name. Then share it.

If you are having trouble, here is a heuristic I've found useful for starting many things:

Start with what you recognize.

Hopefully starting will trigger many more ideas that are not obvious at the start.


* For some examples, look here for a bunch of links compiled by Brian Marick.

The Bananananananana Principle

... as the little boy said, "Today we learned how to spell 'banana', but we didn't learn when to stop." ... In honor of that little boy, we can elevate his idea to a principle, The Banana Principle:

Heuristic devices don't tell you when to stop.

- Gerald M. Weinberg,
An Introduction to General Systems Thinking

I just had the following exchange with my 12 year old daughter Jessica.

Me: How do software testers know when to stop testing something?

Jessica: When you die! . . . Or when you get really tired of it.

---

The Banana Principle does not mean that heuristics cannot be useful in determining when to stop. It means that heuristics do not tell us when to stop using the heuristic. There is a tendency to start transforming the most useful heuristics into laws -- in our minds. Heuristics should help us think and not replace thinking. This includes continual questioning of even the most useful heuristics.

Finally, a tester certification test that I might like!

I am not a fan of any of the current software tester certification programs. Perhaps it is because I take the word certification too literally -- which means that I expect it to have real meaning. When I think of certification, I usually think in lines with the IEEE's definition of certification.

certification

The process of confirming that a system or component complies with its specified requirements and is acceptable for operational use.

Perhaps my thinking is biased by my past IV&V testing work. If one can go to a weekend class and become certified, then I question the value of that certification. I believe that certifications based on ability to memorize terms and practices free of context are of little value -- and may do more harm than good. Yet, the purpose of this blog post is not to provide arguments against the current crop of tester certification options. If you'd like to see some concerns about certification, read the following.

• Against Certification, James Bach
• AST Certification Debate, Tim Coulter
• Open Certification at ACM SIGCSE, Cem Kaner

If we must have a test certification based on a computer-scored multiple-false test, then I think I have found a test that is better than any others I've yet seen. This test covers many aptitudes and skills that I believe are important for testers:

• Precision reading
• Requirements interpretation
• Persistence
  
• Exploratory learning
• Domain knowledge
• Heuristic based problem solving
  
• A good sense of humor
• Critical analysis
• Looking at problems from many angles
• Recognizing context
• Understanding the software platform
• Agility
• Troubleshooting
• Working under pressure
• A good memory, or good note taking skills
• and don't be easily offended
  

Give it a try.

The Impossible Quiz

If I had to select testers based on passing a test, I think I'd take someone that has gotten further than I in this quiz over someone with a software tester certification. I believe this quiz is a better measure of whether or not one is "acceptable for operational use". :)

How far can you get?

And did you notice the bug on question 30?

Failure Usability

One of my pet peeves about software is bad error messages. In my view, a bad error message is one that does not tell the user how the error impacts them and what they need to do in response to the error. Too many messages fail to communicate this information in terms that the software's user is going to understand. Too many of these messages are written for developers, not users.

There is a place for error logging in terms that help developers and testers troubleshoot and fix problems. This information is often best written to log files, not displayed in the user interface.

Pradeep Soundararajan and I recently discussed some of our experiences with error messages. You can listen to excerpts from this conversation using the link below.

Click to listen: ErroneousErrors.mp3

After the above conversation, Pradeep walked me through an exercise that demonstrated a case in a popular office application in which I, the user, was not sufficiently informed that an error was occurring. It was obvious that the actions I was trying to perform were not functioning but the software gave me no clear indication of why it did not work as I expected.

Good testers recognize the need to include "negative testing" in their search for significant bugs. Testing for errors is a common part of functional testing. Let's go beyond functional testing of errors. Let's test errors for usability.

Here is an error testing mnemonic I created after our conversation.

• Functional
• Appropriate (or Apropos*)
  
• Impact
• Log
• UI
• Recovery
• Emotions
  

Functional
Does the error detection and reporting function as required? Are errors not detected that should be detected? Are errors reported? Do error dialogs function as expected? Do the buttons work?


Appropriate (or Apropos*)
Are errors detected and reported in an accurate and timely manner for the intended audience? Are errors reported as soon as an error condition is met? Are warning messages displayed while there are enough resources to remedy the problem? Is a user allowed to waste time and effort only to be told that their work cannot be applied? Is the text of a message accurate? Does the text convey the situation to the intended audience? Is the error described in terms that will be understood by the intended audience?

Impact
Is the impact of the error sufficiently communicated to the user? Does the message contain too little information for the user to understand what occurred and how it impacts what they were attempting to do? Does the message contain extra information that distracts from communicating the impact?

Log
Does technical information need to be logged for support, system administrators, developers, or testers? Will this log information be available if the user waits to contact support? Are log messages standardized to allow for automated information mining? Are logs detailed enough to facilitate troubleshooting? Are errors logged that add no value? Is there too much logging? Does excessive logging negatively impact performance and disk space? Does excessive logging complicate error investigation?

UI
Are users given some indication that what they attempted to do failed? Are user interface messages worded for the intended audience? Are user interface error messages consistent with the look and feel of the software? Are error messages consistent with other activity that causes the same error elsewhere in the application? Are errors communicated in an efficient manner? Does a user need to click away excessive dialogs? Is this error best communicated as an error dialog? Is this error best communicated as text added to the window? Is this error best communicated audibly?

Recovery
Does the error message tell the user how to recover from the error condition? Does the software facilitate recovery? If needed, is contact information provided? Is the user prompted through the recovery or left to figure it out on their own?

Emotions
What emotions are likely to be raised by the error message? Does the information in the message add to a user's frustration or help quiet it? If a user is being told that they need to pay more to use a feature, does the message encourage them to upgrade or does it encourage them to find a competitor's product? Does the error message cause more confusion?


Here's an error message Pradeep gave me to help test the mnemonic. What faults can you find in the error message using the mnemonic?





The next time you see an error message -- or don't see one that you should -- don't stop at functionality. Check the rest of the FAILURE. (And look here for a PowerPoint show demonstrating the mnemonic.)


PS: While attempting to save this post, Blogger gave me the following error. This error fails many of the tests above. The second save button distracted me from the light grey error message. Trying again did not fix the problem. It was not clear if the error meant that my text had been saved or not. I finally had to copy the HTML of the post and paste it into a new Blogger session. Bad error reporting is easy to find. :)


* UDATE: 18 Aug 2007 @ 6:32 PM: Michael Bolton suggested that "Appropriate" would be more appropriate than "Apropos". After some consideration, I agree. Thanks Michael.

Woodpeckers, Pinatas, and Dead Horses

Here's some short blurbs of a few things I took away from CAST sessions.

From Lee Copeland's keynote address:

• "It's nonsensical to talk about automated tests as if they were automated human testing."

• Write or speak about something you're knowledgeable and passionate about.

• Combine things from multiple disciplines.

From Harry Robinson's keynote address:

• Weinberg's Second Law: If Builders Built Buildings The Way Programmers Write Programs, Then The First Woodpecker That Came Along Would Destroy Civilization.

From Esther Derby's keynote:

• To successfully coach someone, they must want to be coached and want to be coached by you.

From James Bach's tutorial:

• Pinata Heuristic: Keep beating at it until the candy comes out. ... and stop once the candy drops.

unless ...

• Dead Horse Heuristic: You may be beating a dead horse.

yet beware ...

• If it is a pinata, don't stop beating at it until the candy drops; but if it is a dead horse, your beating is bringing no value. It can be a challenge to determine if its a pinata or a dead horse.

From Antti Kervinen's presentation:

• Separate automation models into high level (behavior) and low level (behavior implementation) components to reuse test models on a variety of platforms and configurations.

More from James Bach's tutorial:

• Testing does not break software. Testing dispels illusions.

• Rational Unified Process is none of the three. (attributed to Jerry Weinberg)

From the tester exhibition:

• Testing what can't be fixed or controlled may be of little value. Some things may not be worth testing.

• There is great value in the diversity of approaches and skills on a test team.

• It may be possible to beat a dead horse and test (and analyze) too much. Sometimes we should just stop testing and act on the information we have.
  

From Doug Hoffman's tutorial:

• Record and playback automation can be very useful for testing for the same behavior with many configurations. And, once the script stops finding errors: throw it out.

From Keith Stobie's keynote:

• Reduce the paths though your system to improve quality. Fewer features may be better.

• Free web sites often have higher quality than subscription sites. This is because it is easy to measure the cost of downtime on ad-supported systems.

From David Gilbert's session:

• People expect hurricanes to blow around and change path. We should expect the same with software development projects. (David has some interesting ideas about forecasting in software development.)

• Numbers tell a story only in context. You must understand the story behind the numbers.

One more from James:

• Keep Notes!

What did you take away from CAST?

Heuristics in Test Automation

"A Requirement is a quality or condition that matters to someone who matters."
- Cem Kaner, James Bach, and Bret Pettichord
Lessons Learned In Software Testing

Automated tests are usually coded to perform validations against written requirements. Computers are deterministic in that they need specific instructions regarding what to test, what counts as a passed test, and what counts as a failed test. This is one of the weaknesses of test automation. Many requirements exist beyond the hard written requirements. Test automation can be a great tool for measuring hard requirements. For example, automation can be great for validating mathematical calculations.

Test automation can also be a great tool for testing the fuzzy requirements through the use of heuristics. In addition to coding validations (oracles) for hard facts, automation can be used to report things that require human attention. Automation can report information to help direct the attention of human testers.

"Only weak bugs have logic to them ... Subtle bugs have no definable pattern -- they are wild cards."
- Boris Beizer
Software Testing Techniques

A home-grown (by someone else -- later enhanced by me) test automation tool I used many years ago was built to report "pass", "fail" or "inconclusive" for each test it performed. It had been identified that there were many cases where human judgement and/or investigation was required to determine if something really passed or failed. In some cases, it was just not economical to code a validation for something that humans process better than machines. Therefore, instead of trying to make automation do it all, create automation that does what computers do best and let thinking human testers pick up where the computers stop.

I once automated data validation for a large pricing database. It was suspected that there were numerous errors in this database. Finding and fixing possible errors in millions of records was a daunting task. Instead of creating complex calculations to try to completely automate the validation, I coded simple heuristic rules. When these rules failed, then the "failure" was reported to human testers and data editors for investigation. These rules were things like:

• Suggested retail price is greater than wholesale price
• Current price is within 10% of the previous price
• Generic equivalent price is less than the name-brand price

Were these test oracles always true? No. In some cases it was correct for the data to fail the above tests. However, the automation helped direct the attention of testers. These heuristic validations also helped expose unexpected patterns and led to finding bugs in the software that processed and formatted the data.

Instead of creating complex test automation tools, Harry Robinson suggests that we build "The Simplest Thing That Could Possible Find A Bug". Sometimes this means that we code heuristic validations instead of complex validations that report results with absolute certainty. Let the computers report things that a human tester should investigate. Instead of "inconclusive", Harry uses the term "suspicious". I like that.

The next time you automate testing, in addition to thinking of things that computers can report as "passed" or "failed", think of things that it might be able to report as "suspicious".

Heuristics In Software Testing

The point of philosophy is to start with something so simple as to seem not worth stating, and to end with something so paradoxical that no one will believe it."
- Sir Bertrand Russell

A heuristic is a commonsense guideline used to increase the probability of solving a problem by directing one's attention to things that are likely to matter. The word is derived from the Greek "heurisko" which simply means "I find". The exclamation "eureka", meaning "I found it!", shares roots with heuristic.

Just as the "Eureka!" screaming forty-niners of California's Gold Rush did not have secret knowledge or tools to tell them exactly where all the gold was buried; software testers don't know exactly where the bugs are going to hide. However, both software testers and gold miners know where bugs and gold have been found before. We can use that knowledge of past discoveries and the nature of what we seek to create heuristics that help us narrow in on areas most likely to contain the treasure.

Gold miners and testers can find treasure by accident. However, intentional exploration for bugs and gold are more likely to produce results than aimless wandering. That last statement is a heuristic. It is true most of the time, but sometimes it can be proven false. Sometimes wandering testers and miners stumble into something very important. I just don't want to do all my testing by accident.

Heuristic-based testing may not give us concrete answers, but it can guide us to the important things to test. Heuristics can also be used in automation to provide information to guide human testers.

There was a time that I told developers and project managers that I could not test their products when the requirements did not include straightforward "testable" criteria. I thought that I could not test without being able to report "pass" or "fail" for each test.

A good example was a requirement that stated something like "the user shall not have to wait an unacceptable amount of time". As a good quality school tester working in a factory school organization, I demanded to know how long was acceptable before I could start testing. I wanted to quantify "unacceptable". In this case, the truth is that "unacceptable" will vary based on the user. There were no contractual SLAs to satisfy. I may not be able to report that the requirement is met, but I can provide useful information to the project team to assist in determining if the performance is acceptable.

I have since learned that answers to heuristic questions are useful. It was in that same project that I started applying heuristics to automated data validation. Even without concrete requirements, we testers can provide provide information that is useful in answering important testing questions -- especially the qualitative questions. (As a side note, I am now amazed at how much we who call ourselves "Quality Assurance" like to focus on quantitative requirements and metrics.)

To use heuristics in testing, create a list of open-ended questions and guidelines. This will not be a pass/fail list of test criteria. It will not be a list of specific test steps. Instead it can be used to guide your test scripting and exploration for bugs. You will likely develop general heuristics that you can apply to all your testing and specific heuristics that apply to specific applications.

We need to be careful to apply heuristics as heuristics and not enforceable rules. For example, most people involved in testing web application have heard the heuristic that every page should be within three clicks of any other page. Applying this "rule" to web application design usually results in better usability. However, it does not always improve usability. Sometimes making every page within three clicks of another is not reasonable. Adding too many links are likely to confuse users more than they help. (Another heuristic?) Complex work flows often require that pages be more than three clicks away from others. Common sense needs to be applied to heuristics to ensure they are applied only when they fit the context.

Happy bug prospecting.

Eureka!

Next time.... apply heuristic oracles to test automation.