Stupid Questions

Why do we drive on parkways and park in driveways?

Why do noses run and feet smell?

If Jimmy cracks corn and no one cares, why is there a song about him?

Why do we call them restrooms when no one goes there to rest?

Why do you have to click the "Start" button to stop Windows?

Most of my life, I have been told that there are no such things as stupid questions. This was usually said to encourage me, and others, to not be afraid to learn. However, I am beginning to think that there is such a thing as a stupid question. I don't mean questions like the above. Coming up with the questions above requires some thought and I suspect they all have reasonable answers. The above questions are more silly than stupid.

So what do I consider to be a stupid question? A stupid question is a question that has little basis in intelligent thought. A stupid question is a question without the context required to provide an answer. A stupid question is one that the questioner would have realized has no answer had they thought about it.

(adj) stupid: lacking or marked by lack of intellectual acuity

(noun) question: a sentence of inquiry that asks for a reply

Before I continue, I admit that I have asked my share of stupid questions. I am, however, alarmed at the large number of stupid questions that software testers are asking in Internet discussion forums and newsgroups.

Here are some paraphrases of stupid questions I've recently seen posted online:

• How can all tests be automated?
• What are the limitations of [commercial functional test tool]?
• What is functional testing? I don't want a definition, I want complete details.
• What is the industry standard response time for web applications?
• How much test case detail is required?
• What is the best automation tool?
• How do I test a [development platform] application?
• What is the [one and only] definition for [fuzzy testing term]?
• How do I do software testing?
• What is the standard tester to developer ratio?
• What's the best testing technique?
• What are the CMM procedures for a test team of more than n people?
• What is the role of the QA team?
• How do I create test data?
• How can I do exhaustive testing?
• What is the best way to find bugs?
• How many types of bug resolutions are there?
• Who decides if a bug is resolved?
• What's the difference between a requirement and a specification?
• What is the formula for [magic metric that measures testing value without context]?

Most of these questions are unanswerable because they lack context or are made with the assumption that there is one right context-free answer. These questions may lead to interesting discussions but are not answerable with one-size-fits-all solutions.

Don't get stuck on stupid, reporters. We're moving forward.
... You are stuck on stupid. I'm not going to answer that question.
- Gen. Russel Honore

Many of the "senior" testers in online discussion forums answer stupid questions with the tact of General Honore. They are not trying to be rude. Most are not arrogant. They are experienced. Many have learned through their own failure that there are no magic solutions for general questions. Most of the experienced testers I've interacted with online are very willing to help. They are very willing to answer intelligent questions -- even if they disagree with a premise of the question.

Testing software is a context-sensitive intellectual task. An important aspect of testing is working through ambiguity to find and test what really matters. Testing is not a purely technical domain for which single best ways of doing things can be defined and applied regardless of context. Testers need to think and ask intelligent questions.

I asked plenty of questions when I was new to testing. I was given boundaries in which to work and was given freedom to think and learn within those boundaries. I had some great mentors that taught me a great deal about testing. The mentors provided me with good documentation, answered questions, and exemplified good testing practices. Some of the wisdom of my early mentors did not become clear to me until after I failed on my own. Experience is a great teacher. Sometimes we can learn from other people's successes and failures. Sometimes we have to learn on our own.

If you are new to testing, please ask questions. If you don't understand a term or technical detail, please ask. If a requirement is not clear, please ask. If you don't understand the context, please ask. If you need help, please ask. There are plenty of people able and willing to assist other testers. It would be foolish to pretend to know what you are doing when you do not. Asking for help or clarification is not a sign of weakness, it is a sign of intelligence.

Being ignorant is not so much a shame, as being unwilling to learn.
- Benjamin Franklin

Before asking a broad question, think about it. Ask yourself if it is answerable. Do a little research. Provide some context. Show that you care about the question and the requested answer. Realize that the specificity of your question is directly related to the specificity of the answer. General questions are unlikely to have a single answer. When you get an answer, test it. Try to think of situations in which the answer does not apply. Consider what new problems are created by any solution to an existing problem.

By three methods we may learn wisdom:
First, by reflection, which is noblest;
Second, by imitation, which is easiest;
and third by experience, which is the bitterest.
- Confucius

Now, why do we drive on parkways?

Poka-Yoke

Poka-Yoke is not a dance. Its not an event at a rodeo. Its not what my kids do to each other in the back seat of the car. Poka-Yoke is Japanese for "mistake-proofing". Poka-Yoke was developed by Japanese industrial engineer Shigeo Shingo. He realized that people cannot be expected to work like machines and consistently do everything the same way every time they do it. People make mistakes and poorly designed processes can make it easier for people to err. Poka-Yoke's goal is to make it difficult for people to make mistakes through mistake prevention and detection.

Prevention

Applied poka-yoke gives users warnings about incorrect behavior and directs users towards the correct behavior. Computer PS/2 keyboards and mice share the same physical connector design but the connectors are usually color-coded to indicate which device goes into which port on a computer. Some computing hardware is shipped with warning stickers on top of connectors telling users to read a manual or install software before plugging in the device.

Poka-Yoke also means stopping users from doing the wrong thing. Diesel fuel pump nozzles will not fit in a vehicle that requires gasoline. The ignition key cannot be removed from most cars with automatic transmissions if the car is not in "park". Most cars with manual transmissions cannot be started unless the clutch pedal is pressed. These safety features prevent users from making mistakes.

Detection

Some errors cannot be prevented or are too expensive to prevent. The application of poka-yoke demands that errors be detected when and where they occur so that action can be taken before mistakes become bigger problems. Modern space heaters will automatically shut off if they are kicked over. A great example of automatic error detection and correction is the SawStop table saw that automatically disengages when the blade touches something that conducts electricity -- such as fingers. (See the video below.)


http://SawStop.com


Poka-Yoke Applied to Software

Poka-Yoke has existed in hardware products for decades. Poka-Yoke has improved quality and safety of many devices we use daily. While I do not like the behavior-shaping constraints of poka-yoke applied to intellectual tasks, directing and constraining user behavior is essential for good software. I do not advocate application of poka-yoke to the development process. I do advocate applying poka-yoke thinking to every stage of the software development life cycle to improve the quality of the software products we produce. Designers should think poka-yoke. Coders should think poka-yoke. Testers should think poka-yoke. Thinking about usability can lead to fewer bugs.

We are human and there will be bugs. To the extent that quality assurance fails at its primary purpose -- bug prevention -- it must achieve a secondary goal of bug detection.
- Boris Beizer
Software Testing Techniques

Prevention

Keep it simple. Make it easy for users to identify the expected correct way to use the software. Warn them if they try to do something wrong. Don't overwhelm users with unnecessary options.

When the risk of users not following a warning is great, prevent users from doing bad things. Things like list boxes and radio buttons can prevent users from entering invalid data. Data input constraints keep users and the software on the expected path. The security risks in web applications increase the necessity to prevent users from doing what they are not supposed to do.

Detection

It is especially important to detect errors that get past the warnings and constraints and stop processes before errors develop into bigger problems. The earlier an error is detected the easier it is to recover. Bad data detected when it enters a system does not have a chance to cascade into the rest of the system.

Poke-yoke thinking can improve usability and prevent bugs.

---

Some Poka-yoke resources on the web:

• Mistake-proofing Example Wiki
• Using Poka-Yoke Techniques for Early Defect Detection, Harry Robinson
• John Grout's Poka-Yoke Page
  

Model-Based Test Engine Benefit #3: Automatic handling of application changes and bugs

Automated tests based on models have one important feature that scripted testing cannot: automated handling of application changes and bugs. I do not mean that model-based automation can think and make decisions like a human tester does when they discover something unexpected. Instead, the automated selection of test steps supports working around the unexpected without special exception handling code for each situation.


For example: If there are two methods for logging into an application and one breaks the test engine can try the alternate option to get to the rest of the application. If a traditional scripted automated test encounters an unexpected problem it will not be able to complete.

The model-based test engine (MBTE) can be coded to not try an action after a pre-defined number of failures. The MBTE's selection algorithm can then seek out other options that have not yet been found to fail. This also results in the MBTE reattempting failed actions and exposing failures that only occur after specific sequences of actions.

To facilitate the error detection, each action and validation should return the status to the MBTE framework. This allows for error handling to be built into the framework instead of each test model or script. Standard error codes -- either your own or the tool's built-in codes -- help standardize reporting.

For example: return a zero (0) when an action successfully completes or a validation passes, return a negative number on failure, and return a positive number for inconclusive results that require manual investigation.

Code the test engine to detect the error status of each action and validation and take appropriate action. If an action passes, perform the validations for the action's expected end state. If an action fails, restart the application or do whatever other error recovery fits your situation.

If a validation fails you can either code that the next validation be performed or identify validation failures that should stop further validation.

Validations can also be flagged to be state-changing failures by adding a "fail state" column to the oracle/validation tables. Give this field the name of the state that the application is in if the validation fails. You can even build standard states such as "restart" into the framework to indicate that the state is unknown and the application needs to be restarted. For example, a validation that an HTTP 404 error page is not displayed could have a "fail state" of "restart" defined to indicate that the application should be restarted when this validation fails.

Julian Harty has suggested that validations can be weighted and test execution be varied based on the combined score of failures.


Build error handling into the framework so that you can define the details with data instead of code.

When testers create bugs

How come dumb stuff seems so smart while you're doing it?
- Dennis the Menace

Debasis Pradhan's blog entry Testers don't make Bugs. Oh Really? got me thinking about a time that I as a tester actually introduced a bug into a system. Debasis' post is about bugs that slip by testers and escape into the wild. This is not the case in my story. I asked developers to put a bug in the software and they followed my instructions.

I was testing a data mastering system that assembled and converted data from a data repository's format to a variety of other formats for distribution to customers and inclusion in a variety of software products. I created a data validation tool that was used to inspect the huge volume of transformed data: comparing the actual output of the mastering system to the expected format and presentation. The validation tool also performed some heuristic-based tests that alerted testers and developers to data that may require manual inspection.

Over the course of many months, I reported numerous data transformation defects. Most of these were due to input data that the developers did not expect. (Interesting things happen with the data authors copy and paste text from a variety of other applications.) Some of my reported defects were fixed but many were rejected as "data issues". I eventually figured out that this was the label that development group gave to problems that could be resolved by changing the data. Instead of implementing a lasting fix in the code, they insisted that the users change the input data.

In some cases, another development group created post-processing scripts to fix the data created by the first system -- defeating one of the goals for the development of the first system: consolidating a multitude of mastering processes into a single system.

I continued reporting these data issues and worked with development to fix the most important of the "data issues". One run of the data validation tool reported thousands of instances of a new error. I reported the problem and was told that the data was formatted as I had requested.

Sure enough, I had previously listed the badly formatted data I was seeing as the "expected result" in a bug report. One of the few times that one of these "data issues" was fixed in the code, it was fixed wrong and it was my fault. Normally, the project team would have reviewed it and verified my expected transformation before coding any changes. However, this time my improving credibility with development hurt: I was trusted and my mistyped expected result was implemented. I had worked hard to gain the respect of some of the developers and feared that this mistake could setback some of the good will.

I worked with a developer to undo my mistake. Developers got a good laugh out of it. I was humbled. The bug was removed in a following build.

Be careful what you ask for. You just may get it. Double-check those bug reports. And if you make a mistake, admit it and help fix it.

All men make mistakes, but only wise men learn from their mistakes.
- Winston Churchill

Where No Confabulation Goes Untested

confer

verb

• have a conference in order to talk something over

The Conference of the Association for Software Testing (CAST) is coming this July.

I missed last year's conference but have heard great things about it. Based on all the wonderful things I've heard from those that were there, I am looking forward to this year's conference.

The CAST is different than most conferences where people sit and listen to someone present to an audience without public questioning of what is presented. AST encourages testers to test the presentations. Time is allowed for discussion at every presentation. Challenging ideas is encouraged. I could go on. However, I don't think I can push this conference any better than David Gilbert. Therefore, please take a look at David's blog post: CAST in stone.

CAST early bird registration ends this week. Register at
http://associationforsoftwaretesting.org/conference/registration.html

I hope to see you there.

Ben

Don't Ignore The Little Bugs

And that's how it happened.
Believe me. It's true.
Because . . .
just because . . .
a small bug
went KA-CHOO!

One of my favorite children's books is also one of my favorite testing books. (Thanks go to Rob Sabourin* for alerting me to the testing connection.) Because a Little Bug went Ka-CHOO! tells the story of a multitude of problems that cascade from a little bug's sneeze. A worm gets mad. A turtle gets bopped. A bucket gets stuck. A policeman takes flight -- in a motorcycle sidecar. A boat nearly sinks. Pandemonium ensues.

This book illustrates how things that seem to be insignificant can have substantial lasting impact on a larger system.

Software bugs that appear to be trivial can be a sign of a larger problem. When we testers encounter bugs, we are usually looking at a symptom of a problem and not the underlying error that produced the bug. This requires that we do some investigation to determine if a bug is more serious than it first appears and if it is more wide-spread than it first appears.

After a bug is encountered, it is likely that the system under test is in an unexpected state -- and that unexpected state may lead to a bigger problem. Don't stop testing after you reproduce the bug. Look for bigger problems that may exist only after the bug is encountered.

Even if you can't find a bigger problem related to a bug, report it. Someone else may have knowledge about how it may impact other things in the system. At the very least, MIP it.


*Rob Sabourin wrote a great book titled I Am A Bug about software testing for children and others who may be technically challenged. The book is illustrated by his lovely daughter Catherine. An online version of the book may be viewed here.

Driving for quality

Years ago, I taught defensive driving classes to people cited for violating traffic law in Arizona. The central theme of my classes was that our attitudes behind the wheel often have a bigger impact on safe driving than our skill as drivers. I would start each class by having each student describe what they did to get in my class and what other drivers do that annoy them. We then reviewed the two lists (which usually ended up being identical) and the class discussed whether each item was mostly due to driver skill or attitude. Nearly everything on the lists could be traced to an attitude problem.

We are likely guilty of the same faults that we find in others. I believe the attitudes of those involved in software projects can impact the quality of software more than the skills of the team. And when skill is lacking, the right attitude fosters learning. A little patience and a good attitude can go a long way.

I used to teach the SIPDE mnemonic to my driving students to help improve safety on the road. This mnemonic can also be applied to software testing.

• Scan all that's happening around you -- be aware
• Identify potential hazards
• Predict what hazards are most likely to impact you
• Decide on a safe action to deal with those hazards
• Execute the action

Another method I taught in the driving classes was the "Smith" system. This too can be applied to software testing.

1. Aim high - look ahead
2. Keep your eyes moving - don't get too focused on any single thing
3. Get the big picture - watch all around
4. Make sure others see you - communicate
5. Always leave yourself an out - don't put yourself in a situation that you can't escape

Here's to better driving and better software.

Don't forget to think

This past week at STAR East, James Bach presented a number of questions, magic tricks, games, and riddles to testers that volunteered to be tested. I feel like I did fairly well on some of them and failed miserably on others. James uses these tests to teach testers to think. I thought I had learned some valuable lessons until I was presented with two riddles from children today. If only I could learn to think more like a child. :) I think that I sometimes let my search for hidden meanings keep me from seeing the obvious.


A riddle from my children:

You are blindfolded, placed at the start of a maze, and told to get to the other end. How do you navigate the maze?





You can feel your way through.

or a better answer is

Take off the blindfold.

It can be easy to blindly feel our way through though a challenge -- one obstacle at a time. At times it can be good to isolate problems but problems taken out of their context can be misleading. We need to look at problems in the context of their environment. Sometimes we just need to stop and take off the blindfold and look around.

With this in mind, consider the following riddle passed on by a colleague's grandchild:



How do you put a giraffe into a refrigerator?

You open the door, insert the giraffe, and close the door.

Were you trying to make something simple more complex than it needs to be?

How do you put an elephant into the refrigerator?

Open the door, remove the giraffe, put in the elephant, and close the door.

Did you forget about the giraffe? You need to consider the consequences of your past actions.

The Lion King is hosting a party. All the animals attend except one. Which animal does not attend?

The elephant that is in the refrigerator.

How's your memory? You just put the elephant in there.

There is a river inhabited by crocodiles. How do you cross it?

You jump in and swim across. All the crocodiles are at the lion's party.

Did you learn from your mistake with the elephant?

Let's not forget to think.

Model-Based Test Engine Benefit #2: Simplified test result analysis

Automation is of little value if it does not report useful information that can be quickly reviewed by testers.

Reported results should contain enough information to answer the following questions:

• What happened?
• What is the state of the application?
• How did the application get in that state?
• What automation code was executed?
• What automation data/parameters were used?

Some failures reported by automated tests will be errors in the system under test and others will be errors in the automation model or code. It is important that results point the reader to both.

I have found logging of the following information to be useful:

Test (test configuration information)

• Title
• Start Time
• Script File(s)
• Model Files
• Test Set
• Severity
• Environment
• Object Map
• Action Table(s)
• Oracle Table(s)
• Computer Name
• Operating System
• Tester

Actions (controlling the application)

• Source (where is the action defined?)
  
• Title
• Start Time
• Action Details
• Duration
• State Transition
• Automation Code
• Result Details
• Snapshot (screen capture, saved files, etc)
• Status (Pass, Fail, Inconclusive)

Oracles (validating the results)

• Source (where is the oracle defined?)
  
• Title
• State
• Automation Code
• Error Code / Description
• Validation Details
• Snapshot (screen capture, saved files, etc)
• Status (Pass, Fail, Inconclusive)

Messages (report useful information not directly connected to an action or oracle)

• Message
• Link
• Snapshot

Once you have decided what data to report, it is important to present the data in a manner that is conducive to efficient analysis. Results need to be both comprehensive and summarized (or linked) in ways that aid human testers and toolsmiths in quickly answering the questions listed above. A 10 hour automated test execution may be of little value if it takes another 10 hours to interpret the results.

Standardizing reporting and presentation is the first step to improving results analysis. Do not rely on your tool's built-in reporting. An expensive test automation tool should not be required to view results -- especially the incomplete results reported by many tools. Create a common reporting library that can be used by all your tests and use that library. Users of the reported results will not need to learn new formats for every project or test. Some suggested output formats are:

• HTML: Human users like color-coded well-formed results presented in HTML. A little JavaScript can be added to customize the experience.
• XML: Extensible Markup Language (XML) files can be processed by machines and can be displayed to human users when style sheets are applied.
• Tab-Delimited / Excel: Simple tab-delimited, CSV, or Excel tables are useful reporting formats that are easily processed by both people and machines.
• Database: Results written directly to a database can be easily compared to results from previous test executions.

Determine your needs and select the output formats that best meet those needs. If you standardize your reporting through a single small set of reporting functions, you can easily adapt reporting as your needs change.

STAR East 2007 Conferred

I am sitting in the airport waiting to fly home from STAR-East. The conference was great. It was not great due to the many wonderful presentations. It was great because of what happened outside the scheduled activities. I got to confer with colleagues from around the world.

The best part of conferences such as STAR-East is the opportunity to confer with peers and thought leaders in our industry. It is an opportunity to discover that we are experiencing common problems and share possible solutions. It is an opportunity to learn from the best. I often learn more over dinner and in the hallways than I learn in the presentations.

I was amazed at how quickly the conference attendees disappeared once the scheduled activities were completed. I know that many of us computer geeks are introverts. We may not be the most social bunch of people, but I believe a conference without conferring is a wasted opportunity.

See y'all at CAST.

Model-Based Test Engine Benefit #1: Simplified automation creation and maintenance

Model-Oriented Design

Procedural automated test scripts may be easy to record or script. However, they are difficult to maintain when applications change. They are also difficult to adapt to new test ideas. Maintenance is simplified by automating the procedure generation in addition to the execution. New actions, validations, and data can be added to existing tests. This allows testers to spend more time thinking up new test ideas instead of maintaining procedural scripts.

Simplified GUI Interaction Coding

Most GUI automation tools contain complex vocabularies for controlling objects and retrieving information from those objects. There are usually different methods for interacting with different classes of objects. This requires that toolsmiths learn a class-sensitive vocabulary and be aware of the class as they code tests. There is an easier way: create functions that automatically detect an object's class and apply the appropriate method. This allows for the same command to be used whether you are selecting from a list box or entering text into an edit box. The parameters for the functions can then be specified in tables that are processed by the test generation and execution engine.

Common framework functions for interacting with the applications under test also allows for common solutions to tool bugs and limitations. Workarounds and enhancements can be put in the common framework code instead of being reimplemented for each test script.

Separate result validation from actions

Separating expected results definition from the test action execution simplifies maintenance and supports easy reuse of test oracle code. Validations can be specified at whatever level in the model hierarchy they apply and the test engine automatically applies them to all sub-states.

Faking It

Pradeep Soundararajan recently posted a podcast about fake experience on resumes. [listen] This reminded me of an experience I had with a fake resume.

A colleague came to me, dropped a resume in my hand, and asked if I had worked for a company listed on the resume. I quickly scanned the resume and noticed a former employer listed in the experience. I checked the dates and discovered that they included a period that I worked for that company. I then read details that listed projects in which I had been intimately involved. However, I did not recognize the name at the top of the resume. I then called several people at that company and could not find anyone that knew this person.

The experience listed on the resume was fake. It was a lie.

Lying on your resume can come back to haunt you -- sometimes even many years down the road. Don't fall into that trap.

- Sunder Ramchandran
Resumes: Fake it and you break it

This blatant lie was easily caught. Even if I had not worked for the company listed on the resume, whether or not someone worked for a company is usually easy to check. Former employers may be unlikely to give details about what a person did and why they left, but they will generally confirm whether or not someone was an employee.

Faking it may get your foot in the door, but once you are in you still have to perform. The person that submitted this fake resume was interviewed. It was reported to me that it quickly became clear that the person did not have the amount of experience they claimed.

Job hunting can be tough. Faking it does not help. It only makes it tougher. Tell the truth.

The truth may hurt for a little while but a lie hurts forever.

Automating outside the box

test

1. any standardized procedure for measuring sensitivity or memory or intelligence or aptitude or personality etc • the test was standardized on a large sample of students
2. the act of testing something
3. the act of undergoing testing • he survived the great test of battle
4. trying something to find out about it

automation

1. the act of implementing the control of equipment with advanced technology; usually involving electronic hardware • automation replaces human workers by machines
2. the condition of being automatically operated or controlled • automation increases productivity
   
3. equipment used to achieve automatic control or operation • this factory floor is a showcase for automation and robotic equipment

What is test automation?

I just read some marketing literature from some leading test automation tool vendors. According to one of the vendors, their tool supports the following: Novice testers can create robust and easily maintainable tests that mimic real-life use of an application with a few mouse clicks; and the automation tool will troubleshoot errors without human intervention. (I'd give the actual text if it didn't give away the vendor. You may be guessing correctly as you read this.) If this is test automation, I want some. This is what the tool vendors are telling the executives that authorize spending large sums of money. And when such claims are believed, a very high standard is set for test automation. Most testers that implement these tools quickly learn that the automation nirvana promised by tool traffickers is not available at any price.

Wikipedia currently has a decent definition for test automation -- if you read the whole thing. It starts out with a classic definition of test automation...

Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, ... Commonly, test automation involves automating a manual process ...

There are many processes in testing that are good candidates for some form of automation if we do not try to remove the cognitive aspects of testing. Automation that retraces steps that have already been executed manually and reports "pass" or "fail" is unlikely to find bugs or help testers improve their understanding of a software system under test. The Wikipedia definition for test automation includes the following important aspect.

Another important aspect of test automation is the idea of partial test automation, or automating parts but not all of the software testing process. If, for example, an oracle cannot reasonably be created, or if fully automated tests would be too difficult to maintain, then a software tools engineer can instead create testing tools to help human testers perform their jobs more efficiently. Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with oracles), defect logging, etc., without necessarily automating tests in an end-to-end fashion.

I believe that partial test automation is not just an important aspect. It is essential. It is not possible to replace all aspects of a thinking human being with a machine. Test automation that helps automate testing tasks is likely to be a greater benefit than attempts at complete automation.

Instead of trying to create end-to-end test execution automation, think of how a doctor uses medical tests to help diagnose a patient's problems. No blood test or x-ray can diagnose or heal a patient. Doctors use the information reported by these tests in making a cognitive diagnosis. Look for ways that automated execution can help gather data that is useful in diagnosing software.

Test execution automation can be very useful -- but it may not be the best place to start.

Test automation can also be useful in generating test data and test cases. I believe the potential for automated test generation is often overlooked. Wikipedia mentions test generation automation yet implies that it is more academic than practical.

One way to generate test cases automatically is model-based testing where a model of the system is used for test case generation, but research continues into a variety of methodologies for doing so.

Pairwise testing has become a fairly common implementation of test generation automation. Combinations generated by a pairwise or other orthogonal array data generation tool can even be used for creating tests for manual execution.

I ask you to challenge your assumptions about test automation. Think beyond regression testing. Think beyond test execution. Look for ways that automation can help make you more efficient and put your automation efforts there first.

If you are a toolsmith, talk to and watch manual testers work. Look for ways that tools can help them do their work. You may find that the most beneficial automation has nothing to do with your initial assumptions about test automation.

Distracted by the machinery

Let's not allow the machinery of testing to distract us from the craft of testing.


Over ten years, ago James Bach published his first version of Test Automation Snake Oil.

In this article, James identified eight "reckless assumptions" of the classic arguments for test automation. If we aren't careful, it can be easy to start believing statements based on these assumptions.

1. Testing is a "sequence of actions."
2. Testing means repeating the same actions over and over.
3. We can automate testing actions.
4. An automated test is faster, because it needs no human intervention.
5. Automation reduces human error.
6. We can quantify the costs and benefits of manual vs. automated testing.
7. Automation will lead to "significant labor cost savings."
8. Automation will not harm the test project.

Have you made any of these assumptions? Read the article for details. Then take a look at Sam Burgiss' great review.

Falsifiability Testing

The best experiments deduce an effect from the hypothesis and then isolate it in the very context where it may be disproved.
- Michael Kaplan & Ellen Kaplan,
Chances Are: Adventures In Probability

In Focusing on Falsifiability, Stuart Thompson writes the following about a tester-friend's statement that testers can add value from the start of a project if they understand the project and its direction.

"With a clear understanding of what the software was actually trying to do, his team was able to provide useful feedback to the developers even within the first couple of release cycles."

When testers know the problem that software is trying to solve, they can first focus on the things that matter. They can test the assumptions. They can identify contexts in which the assumptions and the implementation can be disproved. Project managers and developers are usually focused on the solution. Testers can help identify the new problems created by proposed solutions and provide information to help the team determine if the new problems are not as bad as the ones they solve.

EACH SOLUTION IS THE SOURCE OF
THE NEXT PROBLEM
We never get rid of problems. Problems, solutions, and new problems weave an endless chain. The best we can hope for is that the problems we substitute are less troublesome than the ones we "solve".

- Donald C. Gause & Gerald M. Weinberg,
Are Your Lights On? How to Figure Out What the Problem REALLY Is

As Stuart points out, testers often have difficulty proving their worth when they prevent bad things from happening. It is usually in hindsight that we see where bypassed testing could have helped.

An example came to my mind as I read Stuart's post.

A change was made to an application. The testers knew about the problem that forced an update to the software. However, the testers we not told the details of the proposed solution. Shortly before the planned release, the testers discovered that the solution created new problems that were worse than the problem solved. In addition to creating new problems, the solution only worked in one of many likely contexts.

The developers told the testers that the solution was good because the "business" people approved it. The "business" people trusted that the developers knew how to implement the business need. It appears that no one specifically tested the assumptions. No one tried to put the solution in contexts that it may not work. Early information sharing with testers likely would have exposed the new problems created by the solution before time was spent coding, testing, and then re-engineering the solution.

It is ironic -- although common -- that a decision made in haste due to time pressure delayed the update to the software. Time spent on QA and testing early is usually going to save time and money.

Had someone focused on the falsifiability of the solution, the problems caused by the solution would have been prevented before a single line of code was written.

I have heard testers complain that no one invites them to be involved early in the process. I too have joined in that chorus. A colleague recently reminded a group of testers that sometimes we aren't included early because we don't ask. Yes, sometimes getting involved early is as easy as asking.

Ask to be involved early. Seek out ways to focus on falsifiability instead of nit-picking incomplete implementations, and developers are likely to invite you back.

Coffee Break Machine Testing

It's good to have some idea as to what something does before you start testing it -- or eating it.

Cookie Monster does a little testing in a 1967 IBM training film.


Thanks to UtterlyGeek.

What do you call this kind of testing?

Not Gonna Bow

Individuals and interactions
over processes and tools

Working software
over comprehensive documentation

Customer collaboration
over contract negotiation

Responding to change
over following a plan

- Agile Manifesto

Neary 400 years ago, Francis Bacon challenged the status quo in scientific thought in “The New Organon”. James Bach recently pointed out some interesting quotes from this work that apply to software testers. I agree.

Bacon argued that placing our preconceived beliefs over what we observe causes great harm. He went so far as to describe these harmful preconceived notions as “idols”. Bacon put these idols into four categories:

• Idols of the Tribe: Errors common to mankind.


• Idols of the Cave: Errors specific to each individual’s education and experience.


• Idols of the Market Place: Errors formed through association with others — often due to misunderstanding others.


• Idols of the Theater: Errors formed from dogma (institutionalized doctrine) and flawed demonstrations.

All of these exist in software testing. As testers, we should be questioning these “idols”, not worshiping them. Sometimes questioning them may prove them right.

Bacon did not ask anyone to abandon their beliefs without cause. Instead he asks that we not make them idols capable of leading us to ignore what would be obvious if we weren’t looking through the distorted mirror of our idols.

A modern day simplification of Bacon’s arguments may be the Agile Manifesto. We should not let our idols of process, documentation, contracts, and plans prevent us from accomplishing the desired goal. Process, documentation, contracts, and plans are only good in as much as they help. They should not prevent us from seeking improvement.

In some ways I believe that the promotion of testing folklore is the result of an industry-wide desire to show that we are mature — as mature as the engineering of physical products. I believe that eagerness to demonstrate maturity helps lead to the implementation of bad processes and cerfifications. Ironically, enforced process (see the bottom of the FSOP cycle) works best for the immature and gives the impression that anyone that can follow the process can test software.

Don't get me wrong. Process and documentation are good things that help even the smartest people when appropriately applied.

"The only thing that interferes with my learning is my
education." - Albert Einstein

We need to seek continual improvement. It is sad that process and certification often become idols that overshadow the real goals.

Heuristics in Test Automation

"A Requirement is a quality or condition that matters to someone who matters."
- Cem Kaner, James Bach, and Bret Pettichord
Lessons Learned In Software Testing

Automated tests are usually coded to perform validations against written requirements. Computers are deterministic in that they need specific instructions regarding what to test, what counts as a passed test, and what counts as a failed test. This is one of the weaknesses of test automation. Many requirements exist beyond the hard written requirements. Test automation can be a great tool for measuring hard requirements. For example, automation can be great for validating mathematical calculations.

Test automation can also be a great tool for testing the fuzzy requirements through the use of heuristics. In addition to coding validations (oracles) for hard facts, automation can be used to report things that require human attention. Automation can report information to help direct the attention of human testers.

"Only weak bugs have logic to them ... Subtle bugs have no definable pattern -- they are wild cards."
- Boris Beizer
Software Testing Techniques

A home-grown (by someone else -- later enhanced by me) test automation tool I used many years ago was built to report "pass", "fail" or "inconclusive" for each test it performed. It had been identified that there were many cases where human judgement and/or investigation was required to determine if something really passed or failed. In some cases, it was just not economical to code a validation for something that humans process better than machines. Therefore, instead of trying to make automation do it all, create automation that does what computers do best and let thinking human testers pick up where the computers stop.

I once automated data validation for a large pricing database. It was suspected that there were numerous errors in this database. Finding and fixing possible errors in millions of records was a daunting task. Instead of creating complex calculations to try to completely automate the validation, I coded simple heuristic rules. When these rules failed, then the "failure" was reported to human testers and data editors for investigation. These rules were things like:

• Suggested retail price is greater than wholesale price
• Current price is within 10% of the previous price
• Generic equivalent price is less than the name-brand price

Were these test oracles always true? No. In some cases it was correct for the data to fail the above tests. However, the automation helped direct the attention of testers. These heuristic validations also helped expose unexpected patterns and led to finding bugs in the software that processed and formatted the data.

Instead of creating complex test automation tools, Harry Robinson suggests that we build "The Simplest Thing That Could Possible Find A Bug". Sometimes this means that we code heuristic validations instead of complex validations that report results with absolute certainty. Let the computers report things that a human tester should investigate. Instead of "inconclusive", Harry uses the term "suspicious". I like that.

The next time you automate testing, in addition to thinking of things that computers can report as "passed" or "failed", think of things that it might be able to report as "suspicious".

Hey Dad, when I grow up, I also want to be square.

Erkan Yilmaz, a fellow tester blogger from Germany, recently pointed out that the slogans in my post "Slogans are models" may not transmit their message across languages and cultures. He attempted to guess at what some of the slogans meant without the context of American culture and advertising. These slogans that most Americans will instantly understand didn't work very well out of their context.

We both saw this as an example of how recipients of information do not always have the full context in which the information originated. As testers, we need to admit when we don't understand and seek the answers (and context) from those that know. Sometimes we may need to bring subject matter experts into the conversation to fully understand what we are testing. If you don't know, ask questions. If you think you know, ask questions. You are bound to learn something.

I recently sat in some presentations by people from the "business" (as in not IT) side of some projects in which I am involved. I learned a great deal about how customers use our products and the business' vision for the future of the products. This information will help me better test the products. It is good to know more about the context in which the products I test are used.

Know thy user, for he is not thee.
- David S. Platt

After our exchange about the American slogans, Erkan provided the following list of slogans from German-speaking countries for interpretation by those of us that live outside that context.

1. “Hey Dad, when I grow up, I also want to be square.”
2. “We wake up earlier.”
3. “We can do everything but speak Standard German.”
4. “With the second eye you see better.”
5. “Firm as a rock in the surge.”
6. “We demand and bring forward personalities.”
7. “Try it in a gentle way.”
8. “It was never so valuable as in these times.”
9. “If it makes you beautiful…”
10. “Well, is today already christmas ?”

What do you think these mean?

See Erkan's original post of this list here.
And after you have tried to interpret the list, look here for their real meanings.

Viel Spaß!

Ben

Heuristics In Software Testing

The point of philosophy is to start with something so simple as to seem not worth stating, and to end with something so paradoxical that no one will believe it."
- Sir Bertrand Russell

A heuristic is a commonsense guideline used to increase the probability of solving a problem by directing one's attention to things that are likely to matter. The word is derived from the Greek "heurisko" which simply means "I find". The exclamation "eureka", meaning "I found it!", shares roots with heuristic.

Just as the "Eureka!" screaming forty-niners of California's Gold Rush did not have secret knowledge or tools to tell them exactly where all the gold was buried; software testers don't know exactly where the bugs are going to hide. However, both software testers and gold miners know where bugs and gold have been found before. We can use that knowledge of past discoveries and the nature of what we seek to create heuristics that help us narrow in on areas most likely to contain the treasure.

Gold miners and testers can find treasure by accident. However, intentional exploration for bugs and gold are more likely to produce results than aimless wandering. That last statement is a heuristic. It is true most of the time, but sometimes it can be proven false. Sometimes wandering testers and miners stumble into something very important. I just don't want to do all my testing by accident.

Heuristic-based testing may not give us concrete answers, but it can guide us to the important things to test. Heuristics can also be used in automation to provide information to guide human testers.

There was a time that I told developers and project managers that I could not test their products when the requirements did not include straightforward "testable" criteria. I thought that I could not test without being able to report "pass" or "fail" for each test.

A good example was a requirement that stated something like "the user shall not have to wait an unacceptable amount of time". As a good quality school tester working in a factory school organization, I demanded to know how long was acceptable before I could start testing. I wanted to quantify "unacceptable". In this case, the truth is that "unacceptable" will vary based on the user. There were no contractual SLAs to satisfy. I may not be able to report that the requirement is met, but I can provide useful information to the project team to assist in determining if the performance is acceptable.

I have since learned that answers to heuristic questions are useful. It was in that same project that I started applying heuristics to automated data validation. Even without concrete requirements, we testers can provide provide information that is useful in answering important testing questions -- especially the qualitative questions. (As a side note, I am now amazed at how much we who call ourselves "Quality Assurance" like to focus on quantitative requirements and metrics.)

To use heuristics in testing, create a list of open-ended questions and guidelines. This will not be a pass/fail list of test criteria. It will not be a list of specific test steps. Instead it can be used to guide your test scripting and exploration for bugs. You will likely develop general heuristics that you can apply to all your testing and specific heuristics that apply to specific applications.

We need to be careful to apply heuristics as heuristics and not enforceable rules. For example, most people involved in testing web application have heard the heuristic that every page should be within three clicks of any other page. Applying this "rule" to web application design usually results in better usability. However, it does not always improve usability. Sometimes making every page within three clicks of another is not reasonable. Adding too many links are likely to confuse users more than they help. (Another heuristic?) Complex work flows often require that pages be more than three clicks away from others. Common sense needs to be applied to heuristics to ensure they are applied only when they fit the context.

Happy bug prospecting.

Eureka!

Next time.... apply heuristic oracles to test automation.

How Doctors Think

Michael Bolton recently brought Dr. Jerome Groopman’s newest book, “How Doctors Think”, to my attention. Michael suggested to the software-testing list that this book contains information that is relevant to software testing. In this book, Dr Groopman explores how doctors diagnose medical problems and what can be done to reduce bad judgment.

Michael Krasny recently interviewed Dr. Groopman on public radio about this book. You can listen to the interview here. Dr Groopman made a number of statements during this interview that I believe apply to us software testers’ search for and diagnosis of software bugs. I expect to find more in the book.

Here are some gems from the radio interview:

About computer-assisted diagnosis software.

It’s a two-edged sword. On one hand, it can be useful if you feed in the most important and relevant symptoms, but the problem is that you need to be able to illicit them effectively from the patient and so you return to language and you return to some of these thinking errors or these cognitive traps that we can all fall into. So if you put in for, example the first prominent symptom that you hear from someone into the software package, it may be that the fourth thing that the person complains about is the most relevant one. Recently you may have seen, for example, in reading mammograms, that computer assisted diagnosis actually generated more false positives. It caused the radiologist to think that shadows which he or she would normally think were benign were in fact cancer and didn’t add to the diagnosis of cancer –the true positives. So I think this technology is worth exploring, but I think that we have to be very careful about it because it some ways it can seem seductive. You cannot substitute a computer for a human being who listens and filters and judges and elicits from the patient the full story.

The same applies to test automation. It can be a useful tool if we don’t let it lead us astray. We human testers need to be actively involved in listening, filtering, judging, and eliciting the full story.

About time-constraints:

Its one of the biggest issues in the current delivery of medical care. You know, everyone is being told as physicians to see patients in 12 minutes. See patients in 11 minutes. So you’re sitting there with one eye on the clock and you can’t think well in haste. And there’s the natural inclination to anchor onto that very first impression as they stop there and just function as if you’re working on an assembly line in a factory. ... patients are being ill-served.
…
I think everyone in the medical system feels under siege. There’s not a lot of satisfaction but in a way this is penny-wise and pound-foolish because the cost of misdiagnosis is extraordinary. … It also costs in terms of dollars. Its much more
expensive to care for and treat an advanced problem than to make a diagnosis early on.

I think we have to force ourselves to resist and part of that can be done: sometimes we have to extend a visit. But we really are beholden to administrators and managed care plans, so one of the things that I’ve begun to do is if I can’t get to the bottom of a problem in my 15 minutes allotted visit,
then I say to the patient, you know, I haven’t figured it out. I need to think more. ... reschedule an appointment and spend more time.

Testers also make mistakes under time pressure. We also need to be aware that our first impressions may not be right. Testing is not a factory assembly line (even though this is a widely-held view). Testers need to be engaged throughout testing and not just mindlessly follow test scripts. And sometimes we need to lobby to spend more time than the schedule originally gives us.


About the art vs. science of diagnosis and improving the effectiveness of that diagnosis:

There’s a seductive nature of numbers, but statistics from clinical studies and so on are just averages and they may not reflect the individual in front of you. ... So real judgment involves customizing medicine: looking at scientific data but also seeing how it applies or doesn’t apply to the person sitting in front of you.
...
It’s not technology, but its language. Language is really the bedrock of medicine. Most of what we do with a doctor – or should be doing with a doctor – is talking… engaging in a dialog.
...
... all of us are wired to make these kinds of snap judgments to sort of go with our first impressions. to rely very heavily on intuition. … That often works but in medicine unfortunately too often it doesn’t work because we have a tendency to latch on to that first bit of information that we get from a patient and then run with it; as opposed to keeping our mind open and constantly doubting ourselves.

Testing is not an assembly line process. Testers need to keep their minds engaged throughout the testing process. We should not ignore our snap-judgements (blink testing), but we also need to look and think beyond those first impressions. We need to continually question both the software and our own judgement as we test.

We need to communicate more than numbers. We need to communicate stories. We need to translate bugs and metrics into language that matters to the business.

How many tests do you need?

“Testing is potentially an infinite process.” – James Bach

Nearly 30 years ago, Glenford Myers provided a self-assessment test to software testers in his book The Art of Software Testing. Much of this book is outdated; however, his test is still regularly used to illustrate the complexity of testing software. He asked readers to write down the test cases required to adequately test the following program.

The program reads three integer values from a card. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.

Glenford Myers then fills the next page of the book with a list of 14 questions that should be answered by test cases for this simple program. He then makes the point that more complex programs are exponentially more difficult to test.

Elizabeth Hendrickson recently created a modern version of the program (no punch cards are required) and challenged testers to create test cases and test her version. See "Testing Triangles: a Classic Exercise Updated for the Web" in her blog.


Last summer, James Bach offered a new challenge to handful of testers sitting around the table at a Denver eatery. This challenge involves testing an even simpler system. James asked how many times we need to press the button on the following system to test 85% of the possible results.

You are asked to test a black box with a single push button. Pressing the button spins an internal wheel that randomly stops at one of 100 possible positions. If any of the 100 possible results (otherwise unknown to the user) encounter a bug, the box will burst into flames. The system has no other output.

Have an answer? To help, let's consider a couple other systems.

How many times do you need to flip a coin to get 85% of the possible results? A coin has two possible results. The first flip will get you 50% of the possible results. Each additional flip will have a 50% chance of not getting the remaining result. Therefore, it is possible -- although unlikely -- that we will ever get a result of heads and a result of tails.

How many times do you need to roll a die to get 85% of the possible results? A die has six possible results. The firs roll will produce a new value. The chance of getting a new value will drop as each new value is encountered. As with the coin, it is possible that we will never encounter all the possible results.

• 1 value: 83% chance
• 2 values: 67% chance
• 3 values: 50% chance
• 4 values: 33% chance
• 5 values: 17% chance

Let's go back to James' challenge. How many time do you need to press the button to test 85% of the possible results? How about 95% of the results? And I will add the question: Is 100% test coverage feasible?

Click here to test a JavaScript implementation of this black box.

What if this black box had 200 possible values? What if there were 307.200 possible values?

Have an answer? Please share it.

Taxing my badometer

I've used great software. I've used some horrible software. I've used lots of software with annoying bugs. I regularly spend a great deal of time working around bugs in software that I use on a daily basis. However, I think I have recently found the worst yet. It is not the worst because it fails to work. It is not the worst because it is missing features I want. It is the worst because it appears to work on the surface yet generates bad data that has a direct impact on my stress level, my finances, and my interaction with the IRS.

Dealing with the IRS -- or even thinking about it -- is already stressful enough. The tax code is complicated enough that it is easy to accidentally report the wrong income or deductions. If it were easy, we'd see accountants lining up at soup kitchens instead of raking in the cash helping us common folks with our tax returns.

For most of the past 10 years, I have elected to use commercial tax preparation software to help me complete and file my tax returns. I've used some tax software that was good, and some that was bad. I've even tried some freebie and online options. The software I used this year is nothing short of a software atrocity.

This software was swarming with bugs -- although most were not obvious. I suspect that most users didn't even notice.

After installing the software, I dutifully entered more financial data than I care to track. I tried a couple what-if scenarios in the process: such as a last minute IRA contribution. I also entered some estimated values before entering the real data so that I could determine if I should take one option or another. I also entered some data for deductions that I later removed after learning (due to reading the IRS publications) that they not apply to my situation.

I did not follow the interview process from start to finish. I went back and forth in this process entering data, revising data once I clarified the rules or gathered up all my evidence for income and deductions. After several hours, I thought I had a complete return. I had entered and verified all my data in the software's "interview" interface.

Then the automated error check failed. It would not let me continue until I fixed data in a form. This was a form that I had accidentally selected somewhere in the process that contained no data. I then tried to delete the form by unchecking a box that made the options that went the form available. The software then informed me that I could not delete the form there. I went to the portion of the interview where it told me to delete the form and there was no delete option. I then discovered that I could delete the form from the program's "forms" list. However, I later found out that going back to any point in the interview recreated the form. There was no option in the software to permanently get rid of my accidental selection.

I discovered that there was no way to print my return for review until I got to the last step of the eFile process. This required that I go back in the process and tell it that I wanted to file on paper. I printed the return and was surprised to see that the data on the return did not match the current data in the interview interface. Some values that I had entered and then deleted or changed were in the printed return. My return was inaccurate. I was glad that I went through the tedious process of getting to the print feature. If I had eFiled without first printing, I would have submitted a return that did not match the data I entered -- and the data shown to me on the screen.

I then spent several hours deleting and reentering data until the printed forms finally matched the data I entered.

In the midst of my review, I noticed that the software had forced me to take an option that resulted in a lower federal tax bill. However, selecting this option instead of another equally valid option significantly increased my state tax bill. The software forced me to have an overall higher tax bill because it only considered the federal return when deciding which option was best for me. After about an hour, I realized that I could remove a deduction in order to force the software to take the alternate option.

Once everything was in order and the displayed and printed data matched, I selected the eFile option to submit my return. I entered bank account information to pay my taxes online. I then discovered that the eFile service I paid for did not allow me to submit electronic payment to the state. My returns were submitted. My federal payment was sent electronically. However, my state payment still had to be mailed. I could have filed my state taxes and paid online using the state's web site as I had done in previous years. Instead I thought it would be easier to use the integrated system. I wish that the software would have told me that I could not electronically submit my payment before it took my money for the eFile service.

And to top it all off, the software did not tell me how to send in my payment by mail. I spent nearly an hour going through the system's help and the vendor's web site seeking guidance. I finally found an option for an online chat with customer service. It then took customer service a half hour to figure out how and where I should send my payment.

I asked if I could have a refund due to the buggy software and the eFile service that was not as good as the state's free eFile service. I was directed to call them by phone to get a refund because the could not give refunds via the online chat. I called the number and waited on hold of over a half hour before I was disconnected. I called back and got a recording stating that the customer support office was closed.

The bad software was topped off by bad customer service. I was angry. My badometer was pegged.

I then did a little exploratory testing in the application. I discovered additional bugs that created incorrect tax returns. I noticed that the software said it would calculate things for me and then ask me for the value without giving any assistance. I found places where the instructions in the application did not match the IRS publications. I even found a way that I could get it to calculate a refund of any value on the state return without changing the income or deductions.

I understand that tax software companies have a very small window in which to do their development and testing. I am amazed that this software was released. The poor quality -- and the agravation it caused -- have ensured that I will not be a returning customer next year.

I suspect that the software passed all the test cases created by its makers. I suspect that it even passed multiple automated tests. This just goes to show that passing all the preconceived scripted tests does not make a quality product. I do not know what methods this company uses for testing their software, but I suspect that it is primarily scripted testing.

I believe that exploratory testing (and model-based automated testing) would have been more likely to find the bugs I encountered than scripted testing.

As we test software, we need to consider the infinite possibility of data and work flow variations. We can't test all the variations, but we can vary what we test. It is easy for individual testers to select similar test options when they think they are selecting data randomly. Seek out variety. Use random data generation tools to provide more variety.

How would you test tax software? How would you ensure that bugs like the ones I encountered don't aggravate customers?

Performance Testing Lessons Learned

Web and client/server load testing can easily become a complex task. Most people I've met got started in load testing with only minimal training in using the test tools. This is how I got started in load testing -- although I had an advantage in that I had been exposed to load testing of communications systems. I also had experience with automated single-user performance testing. I had led some small-scale manual load tests with multiple testers on a conference call hitting the same client-server application at once. (And we found some show-stopping bugs doing that manual testing.) I had watched others perform load tests. I had read numerous load test plans and reports. However, I had never directly participated in executing automated load tests... then I was asked to lead a load testing project.

Through the years, I have made many mistakes designing, scripting, and executing load tests. Load testing easily becomes complex. Tool sales people sometime tell us that nearly anyone can create tests with their tools. (Yet buying test tools is sometimes just like buying a new car: the salesman tells you that the car is reliable and has a great warranty; then the finance person warns of everything that could go wrong that isn't covered in the warranty and trys to sell you an extended warranty and maintenance contract.) Learning the mechanics of how to use a tool are often the easy part. Its what you do with the tool that matters.

Here is the short list of some of the important performance/load testing lessons I have learned. Some I learned from my own experience. Some I learned from the failures of others.

• Bad assumptions waste time and effort
  
• Ask questions
• Performance testing is often exploratory
• Expect surprises
• Prepare to adapt
  
• Get to know the people in your neighborhood: no single person or group is likely to have all the required information
• Subject-matter experts
• Developers
• System administrators
• Database administrators
• Network engineers
  
• Don’t script too much too soon: you may end up tossing out much of what you script
  
• Applications change
• Real usage is difficult to estimate
• Tool limitations may be discovered
  
• Different processes have different impacts: what users do can be as, or more, important as how many users are doing it
  
• Include high-use processes (80/20 rule)
• Include high-risk processes
• Include “different” processes
  
• Modularize scripts: simplify script maintenance -- but only when you intend to run the script again
  
• Data randomization is not always a good thing: randomization can make result comparison difficult
  
• Code error detection and handling
  
• Don’t assume that your tool will handle errors
• Catch and report errors when and where they happen
• Realize that errors may change simulated user activity
  
• Know your tools and test environment
  
• Tool’s supported protocols and licensing
• Load generator and network resource usage
• Load balancing and caching mechanisms
• Beware of test and production environment differences
  
• Try to translate results into stories that matter to the applicable stakeholders
• Tests are run to answer questions: don't overwhelm your audience with hundreds of numbers if they just want a "yes" or "no" answer

and finally…

• Most performance and load-related problems are due to software code or configuration; not hardware


• Don’t throw more hardware at a software problem

Model-Based Test Engine Benefits

A Model-Based Test Engine (MBTE) is a test automation framework that generates and executes tests based on a behavioral model. Instead of performing scripted test cases, a MBTE generates tests from the model during execution. Instead of implementing models in code, a MBTE can process models defined in tables. Both human testers and computers can understand models defined in tables. A MBTE can be built on top of most existing GUI test automation tools. Combining good automation framework practices with Model-Based Testing (MBT) can transform some common test automation pitfalls to benefits.

Implementing a MBTE can produce the following:

1. Simplified automation creation and maintenance.
2. Simplified test result analysis.
3. Automatic handling of application changes and bugs.
4. Generate and execute new tests – and find new bugs.

More to come...

Get excited about the negative

"It is the peculiar and perpetual error of the human understanding to be more moved and excited by affirmatives than by negatives."

--Francis Bacon

Selective thinking is the practice of giving more weight and credence to information that confirms our beliefs than information that may contradict our beliefs. We are all guilty of this. We tend to easily believe data that confirms our beliefs and experience. We tend to ignore data that does not confirm our beliefs.

This confirmation bias can greatly impact our work as software testers. Testers (and developers) often test (and code) to confirm requirements are met by testing the positives. We often overlook the negatives.

Peter Wason's card problem demonstrates this. The problem involved four cards that each have a number on one side and a letter on the other. They are presented with the following values visible:

A

B

4

7

The other side of the cards is not shown.

The following claim is made: "If a card has a vowel on one side, then it will have an even number on the other."

The following question is then asked: Which cards do you need to turn over to determine if the above claim is true?

Try to solve the problem before continuing.

Research has shown that most people get the answer wrong. The majority of people believe that A and 4 must be turned over to answer the question. This suggests that most people try to confirm the positive when the question requires that we also try to disprove the statement. Both the positive and negative need to be confirmed to answer the question. Click here to see the answer.

When we test software we need verify both the positive and the negative of the requirements. We need to ensure that the software does what it should do and does not do what it should not do.

You're too negative

I believe that some level of pessimism is required to be a good software tester. Some of the best testers I have met are pessimistic towards the systems they test. These black hat testers (of which I am one) consider what might go wrong and ask questions. Although I believe that applied pessimism is a necessary for good testing, this negativity can hurt relationships and the project if it is not constructive. Optimist developers and project managers often have trouble understand us pessimists. The result of our pessimism needs to be better preparedness, not shared depression by all involved in a project. This is the type of pessimism that is the subject of Dr. Julie Norem's book, The Positive Power of Negative Thinking. Dr. Norem defines "defensive pessimism" as

Defensive pessimism is a strategy used by anxious people to help them manage their anxiety so they can work productively. Defensive pessimists lower their expectations to help prepare themselves for the worst. Then, they mentally play through all the bad things that might happen. Though it sounds as if it might be depressing, defensive pessimism actually helps anxious people focus away from their emotions so that they can plan and act effectively.

I do not agree that defensive pessimists are necessarily "anxious people". I see the pessimism as a necessary part of good critical thinking.

Think you might be a defensive pessimist? Take the defensive pessimist quiz.

While the pessimist black hat is a necessary part of testing, Julian Harty argues that we need to try on Ed DeBono's other hats as well: both in analysis and applied methodology. Take a look at Julian's CAST presentation from last year.