Things We Know

I find it at work. I find it in online forums. I find it in books. I find it in papers. I find it in blogs. I find it at conferences.

I hear it from experts. I hear it from freshers. I hear it from friends. I hear it from managers. I sometimes even hear it come out of my own mouth.

It influences testers. It influences developers. It influences managers that influence testers and developers. It impacts customers.

It wastes time. It wastes money. It frustrates developers. It confuses executives. It demeans testers. It decreases quality in the name of improvement.

It permeates the practice of developing and testing software.



What is this ubiquitous it?




It is testing folklore.

It ain’t so much the things we don’t know that gets us in trouble. It’s the things we know that ain’t so.
- Artemus Ward

Here are some examples I pulled off the top of my head:

• There are best practices
• Tool vendors know those best practices
• The right tools make good testing
• Testers are the enemies of developers
• Automated unit testing is the only testing we need
• Written requirements are needed for testing
• It is possible to document unambiguous requirements
• Repeatability is maturity
• Tests can be completely designed and scripted before execution
• Testing is simple if guided by the right process
• Quality can be tested into a product
• Good manual testing can be replaced by automation
• Automation is only good for regression testing
• Test case counts are a good measure of test status
• All web pages should load in under 6 seconds
• Testers need to have development skills
• Good testing can be pre-scripted to be executed by anyone that can follow directions
• Boundaries are easy to identify
• Most bugs occur at boundaries
• Testing is easily outsourced to unintelligent people
• Testing is easily outsourced to tools
• Increased testing effort improves quality

What folklore do you encounter?

It is time to unlearn those things we know that ain't so. Challenge the folklore. Ask questions.

• Who says so?
• How do they know?
• What are they missing?
• Does it apply to my context?
• Does it make sense?

Maybe its time to call Mythbusters.

Woodpeckers, Pinatas, and Dead Horses

Here's some short blurbs of a few things I took away from CAST sessions.

From Lee Copeland's keynote address:

• "It's nonsensical to talk about automated tests as if they were automated human testing."

• Write or speak about something you're knowledgeable and passionate about.

• Combine things from multiple disciplines.

From Harry Robinson's keynote address:

• Weinberg's Second Law: If Builders Built Buildings The Way Programmers Write Programs, Then The First Woodpecker That Came Along Would Destroy Civilization.

From Esther Derby's keynote:

• To successfully coach someone, they must want to be coached and want to be coached by you.

From James Bach's tutorial:

• Pinata Heuristic: Keep beating at it until the candy comes out. ... and stop once the candy drops.

unless ...

• Dead Horse Heuristic: You may be beating a dead horse.

yet beware ...

• If it is a pinata, don't stop beating at it until the candy drops; but if it is a dead horse, your beating is bringing no value. It can be a challenge to determine if its a pinata or a dead horse.

From Antti Kervinen's presentation:

• Separate automation models into high level (behavior) and low level (behavior implementation) components to reuse test models on a variety of platforms and configurations.

More from James Bach's tutorial:

• Testing does not break software. Testing dispels illusions.

• Rational Unified Process is none of the three. (attributed to Jerry Weinberg)

From the tester exhibition:

• Testing what can't be fixed or controlled may be of little value. Some things may not be worth testing.

• There is great value in the diversity of approaches and skills on a test team.

• It may be possible to beat a dead horse and test (and analyze) too much. Sometimes we should just stop testing and act on the information we have.
  

From Doug Hoffman's tutorial:

• Record and playback automation can be very useful for testing for the same behavior with many configurations. And, once the script stops finding errors: throw it out.

From Keith Stobie's keynote:

• Reduce the paths though your system to improve quality. Fewer features may be better.

• Free web sites often have higher quality than subscription sites. This is because it is easy to measure the cost of downtime on ad-supported systems.

From David Gilbert's session:

• People expect hurricanes to blow around and change path. We should expect the same with software development projects. (David has some interesting ideas about forecasting in software development.)

• Numbers tell a story only in context. You must understand the story behind the numbers.

One more from James:

• Keep Notes!

What did you take away from CAST?

Stupid Questions

Why do we drive on parkways and park in driveways?

Why do noses run and feet smell?

If Jimmy cracks corn and no one cares, why is there a song about him?

Why do we call them restrooms when no one goes there to rest?

Why do you have to click the "Start" button to stop Windows?

Most of my life, I have been told that there are no such things as stupid questions. This was usually said to encourage me, and others, to not be afraid to learn. However, I am beginning to think that there is such a thing as a stupid question. I don't mean questions like the above. Coming up with the questions above requires some thought and I suspect they all have reasonable answers. The above questions are more silly than stupid.

So what do I consider to be a stupid question? A stupid question is a question that has little basis in intelligent thought. A stupid question is a question without the context required to provide an answer. A stupid question is one that the questioner would have realized has no answer had they thought about it.

(adj) stupid: lacking or marked by lack of intellectual acuity

(noun) question: a sentence of inquiry that asks for a reply

Before I continue, I admit that I have asked my share of stupid questions. I am, however, alarmed at the large number of stupid questions that software testers are asking in Internet discussion forums and newsgroups.

Here are some paraphrases of stupid questions I've recently seen posted online:

• How can all tests be automated?
• What are the limitations of [commercial functional test tool]?
• What is functional testing? I don't want a definition, I want complete details.
• What is the industry standard response time for web applications?
• How much test case detail is required?
• What is the best automation tool?
• How do I test a [development platform] application?
• What is the [one and only] definition for [fuzzy testing term]?
• How do I do software testing?
• What is the standard tester to developer ratio?
• What's the best testing technique?
• What are the CMM procedures for a test team of more than n people?
• What is the role of the QA team?
• How do I create test data?
• How can I do exhaustive testing?
• What is the best way to find bugs?
• How many types of bug resolutions are there?
• Who decides if a bug is resolved?
• What's the difference between a requirement and a specification?
• What is the formula for [magic metric that measures testing value without context]?

Most of these questions are unanswerable because they lack context or are made with the assumption that there is one right context-free answer. These questions may lead to interesting discussions but are not answerable with one-size-fits-all solutions.

Don't get stuck on stupid, reporters. We're moving forward.
... You are stuck on stupid. I'm not going to answer that question.
- Gen. Russel Honore

Many of the "senior" testers in online discussion forums answer stupid questions with the tact of General Honore. They are not trying to be rude. Most are not arrogant. They are experienced. Many have learned through their own failure that there are no magic solutions for general questions. Most of the experienced testers I've interacted with online are very willing to help. They are very willing to answer intelligent questions -- even if they disagree with a premise of the question.

Testing software is a context-sensitive intellectual task. An important aspect of testing is working through ambiguity to find and test what really matters. Testing is not a purely technical domain for which single best ways of doing things can be defined and applied regardless of context. Testers need to think and ask intelligent questions.

I asked plenty of questions when I was new to testing. I was given boundaries in which to work and was given freedom to think and learn within those boundaries. I had some great mentors that taught me a great deal about testing. The mentors provided me with good documentation, answered questions, and exemplified good testing practices. Some of the wisdom of my early mentors did not become clear to me until after I failed on my own. Experience is a great teacher. Sometimes we can learn from other people's successes and failures. Sometimes we have to learn on our own.

If you are new to testing, please ask questions. If you don't understand a term or technical detail, please ask. If a requirement is not clear, please ask. If you don't understand the context, please ask. If you need help, please ask. There are plenty of people able and willing to assist other testers. It would be foolish to pretend to know what you are doing when you do not. Asking for help or clarification is not a sign of weakness, it is a sign of intelligence.

Being ignorant is not so much a shame, as being unwilling to learn.
- Benjamin Franklin

Before asking a broad question, think about it. Ask yourself if it is answerable. Do a little research. Provide some context. Show that you care about the question and the requested answer. Realize that the specificity of your question is directly related to the specificity of the answer. General questions are unlikely to have a single answer. When you get an answer, test it. Try to think of situations in which the answer does not apply. Consider what new problems are created by any solution to an existing problem.

By three methods we may learn wisdom:
First, by reflection, which is noblest;
Second, by imitation, which is easiest;
and third by experience, which is the bitterest.
- Confucius

Now, why do we drive on parkways?

Don't Ignore The Little Bugs

And that's how it happened.
Believe me. It's true.
Because . . .
just because . . .
a small bug
went KA-CHOO!

One of my favorite children's books is also one of my favorite testing books. (Thanks go to Rob Sabourin* for alerting me to the testing connection.) Because a Little Bug went Ka-CHOO! tells the story of a multitude of problems that cascade from a little bug's sneeze. A worm gets mad. A turtle gets bopped. A bucket gets stuck. A policeman takes flight -- in a motorcycle sidecar. A boat nearly sinks. Pandemonium ensues.

This book illustrates how things that seem to be insignificant can have substantial lasting impact on a larger system.

Software bugs that appear to be trivial can be a sign of a larger problem. When we testers encounter bugs, we are usually looking at a symptom of a problem and not the underlying error that produced the bug. This requires that we do some investigation to determine if a bug is more serious than it first appears and if it is more wide-spread than it first appears.

After a bug is encountered, it is likely that the system under test is in an unexpected state -- and that unexpected state may lead to a bigger problem. Don't stop testing after you reproduce the bug. Look for bigger problems that may exist only after the bug is encountered.

Even if you can't find a bigger problem related to a bug, report it. Someone else may have knowledge about how it may impact other things in the system. At the very least, MIP it.


*Rob Sabourin wrote a great book titled I Am A Bug about software testing for children and others who may be technically challenged. The book is illustrated by his lovely daughter Catherine. An online version of the book may be viewed here.

Driving for quality

Years ago, I taught defensive driving classes to people cited for violating traffic law in Arizona. The central theme of my classes was that our attitudes behind the wheel often have a bigger impact on safe driving than our skill as drivers. I would start each class by having each student describe what they did to get in my class and what other drivers do that annoy them. We then reviewed the two lists (which usually ended up being identical) and the class discussed whether each item was mostly due to driver skill or attitude. Nearly everything on the lists could be traced to an attitude problem.

We are likely guilty of the same faults that we find in others. I believe the attitudes of those involved in software projects can impact the quality of software more than the skills of the team. And when skill is lacking, the right attitude fosters learning. A little patience and a good attitude can go a long way.

I used to teach the SIPDE mnemonic to my driving students to help improve safety on the road. This mnemonic can also be applied to software testing.

• Scan all that's happening around you -- be aware
• Identify potential hazards
• Predict what hazards are most likely to impact you
• Decide on a safe action to deal with those hazards
• Execute the action

Another method I taught in the driving classes was the "Smith" system. This too can be applied to software testing.

1. Aim high - look ahead
2. Keep your eyes moving - don't get too focused on any single thing
3. Get the big picture - watch all around
4. Make sure others see you - communicate
5. Always leave yourself an out - don't put yourself in a situation that you can't escape

Here's to better driving and better software.

Don't forget to think

This past week at STAR East, James Bach presented a number of questions, magic tricks, games, and riddles to testers that volunteered to be tested. I feel like I did fairly well on some of them and failed miserably on others. James uses these tests to teach testers to think. I thought I had learned some valuable lessons until I was presented with two riddles from children today. If only I could learn to think more like a child. :) I think that I sometimes let my search for hidden meanings keep me from seeing the obvious.


A riddle from my children:

You are blindfolded, placed at the start of a maze, and told to get to the other end. How do you navigate the maze?





You can feel your way through.

or a better answer is

Take off the blindfold.

It can be easy to blindly feel our way through though a challenge -- one obstacle at a time. At times it can be good to isolate problems but problems taken out of their context can be misleading. We need to look at problems in the context of their environment. Sometimes we just need to stop and take off the blindfold and look around.

With this in mind, consider the following riddle passed on by a colleague's grandchild:



How do you put a giraffe into a refrigerator?

You open the door, insert the giraffe, and close the door.

Were you trying to make something simple more complex than it needs to be?

How do you put an elephant into the refrigerator?

Open the door, remove the giraffe, put in the elephant, and close the door.

Did you forget about the giraffe? You need to consider the consequences of your past actions.

The Lion King is hosting a party. All the animals attend except one. Which animal does not attend?

The elephant that is in the refrigerator.

How's your memory? You just put the elephant in there.

There is a river inhabited by crocodiles. How do you cross it?

You jump in and swim across. All the crocodiles are at the lion's party.

Did you learn from your mistake with the elephant?

Let's not forget to think.

How Doctors Think

Michael Bolton recently brought Dr. Jerome Groopman’s newest book, “How Doctors Think”, to my attention. Michael suggested to the software-testing list that this book contains information that is relevant to software testing. In this book, Dr Groopman explores how doctors diagnose medical problems and what can be done to reduce bad judgment.

Michael Krasny recently interviewed Dr. Groopman on public radio about this book. You can listen to the interview here. Dr Groopman made a number of statements during this interview that I believe apply to us software testers’ search for and diagnosis of software bugs. I expect to find more in the book.

Here are some gems from the radio interview:

About computer-assisted diagnosis software.

It’s a two-edged sword. On one hand, it can be useful if you feed in the most important and relevant symptoms, but the problem is that you need to be able to illicit them effectively from the patient and so you return to language and you return to some of these thinking errors or these cognitive traps that we can all fall into. So if you put in for, example the first prominent symptom that you hear from someone into the software package, it may be that the fourth thing that the person complains about is the most relevant one. Recently you may have seen, for example, in reading mammograms, that computer assisted diagnosis actually generated more false positives. It caused the radiologist to think that shadows which he or she would normally think were benign were in fact cancer and didn’t add to the diagnosis of cancer –the true positives. So I think this technology is worth exploring, but I think that we have to be very careful about it because it some ways it can seem seductive. You cannot substitute a computer for a human being who listens and filters and judges and elicits from the patient the full story.

The same applies to test automation. It can be a useful tool if we don’t let it lead us astray. We human testers need to be actively involved in listening, filtering, judging, and eliciting the full story.

About time-constraints:

Its one of the biggest issues in the current delivery of medical care. You know, everyone is being told as physicians to see patients in 12 minutes. See patients in 11 minutes. So you’re sitting there with one eye on the clock and you can’t think well in haste. And there’s the natural inclination to anchor onto that very first impression as they stop there and just function as if you’re working on an assembly line in a factory. ... patients are being ill-served.
…
I think everyone in the medical system feels under siege. There’s not a lot of satisfaction but in a way this is penny-wise and pound-foolish because the cost of misdiagnosis is extraordinary. … It also costs in terms of dollars. Its much more
expensive to care for and treat an advanced problem than to make a diagnosis early on.

I think we have to force ourselves to resist and part of that can be done: sometimes we have to extend a visit. But we really are beholden to administrators and managed care plans, so one of the things that I’ve begun to do is if I can’t get to the bottom of a problem in my 15 minutes allotted visit,
then I say to the patient, you know, I haven’t figured it out. I need to think more. ... reschedule an appointment and spend more time.

Testers also make mistakes under time pressure. We also need to be aware that our first impressions may not be right. Testing is not a factory assembly line (even though this is a widely-held view). Testers need to be engaged throughout testing and not just mindlessly follow test scripts. And sometimes we need to lobby to spend more time than the schedule originally gives us.


About the art vs. science of diagnosis and improving the effectiveness of that diagnosis:

There’s a seductive nature of numbers, but statistics from clinical studies and so on are just averages and they may not reflect the individual in front of you. ... So real judgment involves customizing medicine: looking at scientific data but also seeing how it applies or doesn’t apply to the person sitting in front of you.
...
It’s not technology, but its language. Language is really the bedrock of medicine. Most of what we do with a doctor – or should be doing with a doctor – is talking… engaging in a dialog.
...
... all of us are wired to make these kinds of snap judgments to sort of go with our first impressions. to rely very heavily on intuition. … That often works but in medicine unfortunately too often it doesn’t work because we have a tendency to latch on to that first bit of information that we get from a patient and then run with it; as opposed to keeping our mind open and constantly doubting ourselves.

Testing is not an assembly line process. Testers need to keep their minds engaged throughout the testing process. We should not ignore our snap-judgements (blink testing), but we also need to look and think beyond those first impressions. We need to continually question both the software and our own judgement as we test.

We need to communicate more than numbers. We need to communicate stories. We need to translate bugs and metrics into language that matters to the business.

Best Practices Aren’t

The first two of the Seven Basic Principles of the Context-Driven School of software testing are:

1. The value of any practice depends on its context.
2. There are good practices in context, but there are no best practices.

As a former quality school gatekeeper, I understand the value of standards – in both products and processes. However, I am concerned by the current “best practices” trends in software development and testing. The rigidity that we demand in product standards can hurt in process standards. Even the CMM (which is often viewed as a rigid process) has “Optimizing” as the highest level of maturity. A mature process includes continuous learning and adjustment of the process. No process should lock us into one way of doing anything.

Nearly 100 years ago, the industrial efficiency pioneer Frederick Taylor wrote “among the various methods and implements used in each element of each trade there is always one method and one implement which is quicker and better than any of the rest”.

I do not disagree that there may be a best practice for a specific task in a specific context. Taylor broke down existing methods and implements (tools) into small pieces and scientifically evaluated them in search of areas for improvement. The problem is that today's best practices are often applied as one-size-fits-all processes. The best practice for one situation is not necessarily the best for all other contexts. And a "best practice" today may no longer be the best practice tomorrow. This is actually the opposite of what Taylor did. Consultants and tool vendors have discovered that there is money to be made taking "best practices" out of one context and applying to all other contexts. It is harder, and likely less profitable, for the "experts" to seek out the best practices for a specific context. Taylor sought out and applied best practices to small elements. Many of today's "best practices" are applied at a universal level.

I am amazed by what appears to be widespread acceptance of “best practices” by software testers. As testers, it is our job to question. We make a living questioning software. We need to continually do the same for practices. Test your practices.

When presented with a best practice, consider contexts in which the practice is not the best. The broader the scope of the best practice, the more situations it is unlikely to fit. Don’t limit your toolbox to a single practice or set of practices. Be flexible enough to adjust your processes as the context demands. Treat process as an example and apply it where it fits and be willing to deviate from the process -- or even apply an entirely different process -- if it does not fit the context.

No process should replace human intelligence. Let process guide you when it applies. Don’t let a process make decisions for you.

Seek out continuous improvement. Don't let process become a rut.

Process is best used as a map; not as an auto-pilot.