May 21, 2007

Don't forget to think

This past week at STAR East, James Bach presented a number of questions, magic tricks, games, and riddles to testers that volunteered to be tested. I feel like I did fairly well on some of them and failed miserably on others. James uses these tests to teach testers to think. I thought I had learned some valuable lessons until I was presented with two riddles from children today. If only I could learn to think more like a child. :) I think that I sometimes let my search for hidden meanings keep me from seeing the obvious.


A riddle from my children:

You are blindfolded, placed at the start of a maze, and told to get to the other end. How do you navigate the maze?





You can feel your way through.










or a better answer is












Take off the blindfold.



It can be easy to blindly feel our way through though a challenge -- one obstacle at a time. At times it can be good to isolate problems but problems taken out of their context can be misleading. We need to look at problems in the context of their environment. Sometimes we just need to stop and take off the blindfold and look around.

With this in mind, consider the following riddle passed on by a colleague's grandchild:



How do you put a giraffe into a refrigerator?
























You open the door, insert the giraffe, and close the door.



Were you trying to make something simple more complex than it needs to be?





How do you put an elephant into the refrigerator?
























Open the door, remove the giraffe, put in the elephant, and close the door.



Did you forget about the giraffe? You need to consider the consequences of your past actions.




The Lion King is hosting a party. All the animals attend except one. Which animal does not attend?























The elephant that is in the refrigerator.



How's your memory? You just put the elephant in there.




There is a river inhabited by crocodiles. How do you cross it?























You jump in and swim across. All the crocodiles are at the lion's party.




Did you learn from your mistake with the elephant?





Let's not forget to think.

Model-Based Test Engine Benefit #2: Simplified test result analysis


Automation is of little value if it does not report useful information that can be quickly reviewed by testers.

Reported results should contain enough information to answer the following questions:

  • What happened?
  • What is the state of the application?
  • How did the application get in that state?
  • What automation code was executed?
  • What automation data/parameters were used?

Some failures reported by automated tests will be errors in the system under test and others will be errors in the automation model or code. It is important that results point the reader to both.

I have found logging of the following information to be useful:

Test (test configuration information)
  • Title
  • Start Time
  • Script File(s)
  • Model Files
  • Test Set
  • Severity
  • Environment
  • Object Map
  • Action Table(s)
  • Oracle Table(s)
  • Computer Name
  • Operating System
  • Tester

Actions (controlling the application)
  • Source (where is the action defined?)
  • Title
  • Start Time
  • Action Details
  • Duration
  • State Transition
  • Automation Code
  • Result Details
  • Snapshot (screen capture, saved files, etc)
  • Status (Pass, Fail, Inconclusive)

Oracles (validating the results)
  • Source (where is the oracle defined?)
  • Title
  • State
  • Automation Code
  • Error Code / Description
  • Validation Details
  • Snapshot (screen capture, saved files, etc)
  • Status (Pass, Fail, Inconclusive)

Messages (report useful information not directly connected to an action or oracle)
  • Message
  • Link
  • Snapshot

Once you have decided what data to report, it is important to present the data in a manner that is conducive to efficient analysis. Results need to be both comprehensive and summarized (or linked) in ways that aid human testers and toolsmiths in quickly answering the questions listed above. A 10 hour automated test execution may be of little value if it takes another 10 hours to interpret the results.

Standardizing reporting and presentation is the first step to improving results analysis. Do not rely on your tool's built-in reporting. An expensive test automation tool should not be required to view results -- especially the incomplete results reported by many tools. Create a common reporting library that can be used by all your tests and use that library. Users of the reported results will not need to learn new formats for every project or test. Some suggested output formats are:

  • HTML: Human users like color-coded well-formed results presented in HTML. A little JavaScript can be added to customize the experience.
  • XML: Extensible Markup Language (XML) files can be processed by machines and can be displayed to human users when style sheets are applied.
  • Tab-Delimited / Excel: Simple tab-delimited, CSV, or Excel tables are useful reporting formats that are easily processed by both people and machines.
  • Database: Results written directly to a database can be easily compared to results from previous test executions.

Determine your needs and select the output formats that best meet those needs. If you standardize your reporting through a single small set of reporting functions, you can easily adapt reporting as your needs change.

May 19, 2007

STAR East 2007 Conferred

I am sitting in the airport waiting to fly home from STAR-East. The conference was great. It was not great due to the many wonderful presentations. It was great because of what happened outside the scheduled activities. I got to confer with colleagues from around the world.

The best part of conferences such as STAR-East is the opportunity to confer with peers and thought leaders in our industry. It is an opportunity to discover that we are experiencing common problems and share possible solutions. It is an opportunity to learn from the best. I often learn more over dinner and in the hallways than I learn in the presentations.

I was amazed at how quickly the conference attendees disappeared once the scheduled activities were completed. I know that many of us computer geeks are introverts. We may not be the most social bunch of people, but I believe a conference without conferring is a wasted opportunity.

See y'all at CAST.

Model-Based Test Engine Benefit #1: Simplified automation creation and maintenance

Model-Oriented Design

Procedural automated test scripts may be easy to record or script. However, they are difficult to maintain when applications change. They are also difficult to adapt to new test ideas. Maintenance is simplified by automating the procedure generation in addition to the execution. New actions, validations, and data can be added to existing tests. This allows testers to spend more time thinking up new test ideas instead of maintaining procedural scripts.

Simplified GUI Interaction Coding

Most GUI automation tools contain complex vocabularies for controlling objects and retrieving information from those objects. There are usually different methods for interacting with different classes of objects. This requires that toolsmiths learn a class-sensitive vocabulary and be aware of the class as they code tests. There is an easier way: create functions that automatically detect an object's class and apply the appropriate method. This allows for the same command to be used whether you are selecting from a list box or entering text into an edit box. The parameters for the functions can then be specified in tables that are processed by the test generation and execution engine.

Common framework functions for interacting with the applications under test also allows for common solutions to tool bugs and limitations. Workarounds and enhancements can be put in the common framework code instead of being reimplemented for each test script.

Separate result validation from actions

Separating expected results definition from the test action execution simplifies maintenance and supports easy reuse of test oracle code. Validations can be specified at whatever level in the model hierarchy they apply and the test engine automatically applies them to all sub-states.

May 16, 2007

Faking It

Pradeep Soundararajan recently posted a podcast about fake experience on resumes. [listen] This reminded me of an experience I had with a fake resume.

A colleague came to me, dropped a resume in my hand, and asked if I had worked for a company listed on the resume. I quickly scanned the resume and noticed a former employer listed in the experience. I checked the dates and discovered that they included a period that I worked for that company. I then read details that listed projects in which I had been intimately involved. However, I did not recognize the name at the top of the resume. I then called several people at that company and could not find anyone that knew this person.

The experience listed on the resume was fake. It was a lie.

Lying on your resume can come back to haunt you -- sometimes even many years down the road. Don't fall into that trap.


This blatant lie was easily caught. Even if I had not worked for the company listed on the resume, whether or not someone worked for a company is usually easy to check. Former employers may be unlikely to give details about what a person did and why they left, but they will generally confirm whether or not someone was an employee.

Faking it may get your foot in the door, but once you are in you still have to perform. The person that submitted this fake resume was interviewed. It was reported to me that it quickly became clear that the person did not have the amount of experience they claimed.

Job hunting can be tough. Faking it does not help. It only makes it tougher. Tell the truth.

The truth may hurt for a little while but a lie hurts forever.

May 15, 2007

Automating outside the box

test
  1. any standardized procedure for measuring sensitivity or memory or intelligence or aptitude or personality etc • the test was standardized on a large sample of students
  2. the act of testing something
  3. the act of undergoing testing • he survived the great test of battle
  4. trying something to find out about it

automation
  1. the act of implementing the control of equipment with advanced technology; usually involving electronic hardware • automation replaces human workers by machines
  2. the condition of being automatically operated or controlled • automation increases productivity
  3. equipment used to achieve automatic control or operation • this factory floor is a showcase for automation and robotic equipment

What is test automation?


I just read some marketing literature from some leading test automation tool vendors. According to one of the vendors, their tool supports the following: Novice testers can create robust and easily maintainable tests that mimic real-life use of an application with a few mouse clicks; and the automation tool will troubleshoot errors without human intervention. (I'd give the actual text if it didn't give away the vendor. You may be guessing correctly as you read this.) If this is test automation, I want some. This is what the tool vendors are telling the executives that authorize spending large sums of money. And when such claims are believed, a very high standard is set for test automation. Most testers that implement these tools quickly learn that the automation nirvana promised by tool traffickers is not available at any price.

Wikipedia currently has a decent definition for test automation -- if you read the whole thing. It starts out with a classic definition of test automation...

Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, ... Commonly, test automation involves automating a manual process ...

There are many processes in testing that are good candidates for some form of automation if we do not try to remove the cognitive aspects of testing. Automation that retraces steps that have already been executed manually and reports "pass" or "fail" is unlikely to find bugs or help testers improve their understanding of a software system under test. The Wikipedia definition for test automation includes the following important aspect.

Another important aspect of test automation is the idea of partial test automation, or automating parts but not all of the software testing process. If, for example, an oracle cannot reasonably be created, or if fully automated tests would be too difficult to maintain, then a software tools engineer can instead create testing tools to help human testers perform their jobs more efficiently. Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with oracles), defect logging, etc., without necessarily automating tests in an end-to-end fashion.
I believe that partial test automation is not just an important aspect. It is essential. It is not possible to replace all aspects of a thinking human being with a machine. Test automation that helps automate testing tasks is likely to be a greater benefit than attempts at complete automation.

Instead of trying to create end-to-end test execution automation, think of how a doctor uses medical tests to help diagnose a patient's problems. No blood test or x-ray can diagnose or heal a patient. Doctors use the information reported by these tests in making a cognitive diagnosis. Look for ways that automated execution can help gather data that is useful in diagnosing software.

Test execution automation can be very useful -- but it may not be the best place to start.

Test automation can also be useful in generating test data and test cases. I believe the potential for automated test generation is often overlooked. Wikipedia mentions test generation automation yet implies that it is more academic than practical.

One way to generate test cases automatically is model-based testing where a model of the system is used for test case generation, but research continues into a variety of methodologies for doing so.
Pairwise testing has become a fairly common implementation of test generation automation. Combinations generated by a pairwise or other orthogonal array data generation tool can even be used for creating tests for manual execution.

I ask you to challenge your assumptions about test automation. Think beyond regression testing. Think beyond test execution. Look for ways that automation can help make you more efficient and put your automation efforts there first.

If you are a toolsmith, talk to and watch manual testers work. Look for ways that tools can help them do their work. You may find that the most beneficial automation has nothing to do with your initial assumptions about test automation.

May 9, 2007

Distracted by the machinery

Let's not allow the machinery of testing to distract us from the craft of testing.


Over ten years, ago James Bach published his first version of Test Automation Snake Oil.

In this article, James identified eight "reckless assumptions" of the classic arguments for test automation. If we aren't careful, it can be easy to start believing statements based on these assumptions.

    1. Testing is a "sequence of actions."
    2. Testing means repeating the same actions over and over.
    3. We can automate testing actions.
    4. An automated test is faster, because it needs no human intervention.
    5. Automation reduces human error.
    6. We can quantify the costs and benefits of manual vs. automated testing.
    7. Automation will lead to "significant labor cost savings."
    8. Automation will not harm the test project.
Have you made any of these assumptions? Read the article for details. Then take a look at Sam Burgiss' great review.

May 7, 2007

Falsifiability Testing

The best experiments deduce an effect from the hypothesis and then isolate it in the very context where it may be disproved.
- Michael Kaplan & Ellen Kaplan,

Chances Are: Adventures In Probability

In Focusing on Falsifiability, Stuart Thompson writes the following about a tester-friend's statement that testers can add value from the start of a project if they understand the project and its direction.

"With a clear understanding of what the software was actually trying to do, his team was able to provide useful feedback to the developers even within the first couple of release cycles."

When testers know the problem that software is trying to solve, they can first focus on the things that matter. They can test the assumptions. They can identify contexts in which the assumptions and the implementation can be disproved. Project managers and developers are usually focused on the solution. Testers can help identify the new problems created by proposed solutions and provide information to help the team determine if the new problems are not as bad as the ones they solve.

EACH SOLUTION IS THE SOURCE OF
THE NEXT PROBLEM
We never get rid of problems. Problems, solutions, and new problems weave an endless chain. The best we can hope for is that the problems we substitute are less troublesome than the ones we "solve".


As Stuart points out, testers often have difficulty proving their worth when they prevent bad things from happening. It is usually in hindsight that we see where bypassed testing could have helped.

An example came to my mind as I read Stuart's post.

A change was made to an application. The testers knew about the problem that forced an update to the software. However, the testers we not told the details of the proposed solution. Shortly before the planned release, the testers discovered that the solution created new problems that were worse than the problem solved. In addition to creating new problems, the solution only worked in one of many likely contexts.

The developers told the testers that the solution was good because the "business" people approved it. The "business" people trusted that the developers knew how to implement the business need. It appears that no one specifically tested the assumptions. No one tried to put the solution in contexts that it may not work. Early information sharing with testers likely would have exposed the new problems created by the solution before time was spent coding, testing, and then re-engineering the solution.

It is ironic -- although common -- that a decision made in haste due to time pressure delayed the update to the software. Time spent on QA and testing early is usually going to save time and money.

Had someone focused on the falsifiability of the solution, the problems caused by the solution would have been prevented before a single line of code was written.

I have heard testers complain that no one invites them to be involved early in the process. I too have joined in that chorus. A colleague recently reminded a group of testers that sometimes we aren't included early because we don't ask. Yes, sometimes getting involved early is as easy as asking.

Ask to be involved early. Seek out ways to focus on falsifiability instead of nit-picking incomplete implementations, and developers are likely to invite you back.

May 5, 2007

Coffee Break Machine Testing

It's good to have some idea as to what something does before you start testing it -- or eating it.

Cookie Monster does a little testing in a 1967 IBM training film.


Thanks to UtterlyGeek.

What do you call this kind of testing?

Not Gonna Bow

Individuals and interactions
over processes and tools

Working software
over comprehensive documentation

Customer collaboration
over contract negotiation

Responding to change
over following a plan

- Agile Manifesto


Neary 400 years ago, Francis Bacon challenged the status quo in scientific thought in “The New Organon”. James Bach recently pointed out some interesting quotes from this work that apply to software testers. I agree.

Bacon argued that placing our preconceived beliefs over what we observe causes great harm. He went so far as to describe these harmful preconceived notions as “idols”. Bacon put these idols into four categories:

  • Idols of the Tribe: Errors common to mankind.

  • Idols of the Cave: Errors specific to each individual’s education and experience.

  • Idols of the Market Place: Errors formed through association with others — often due to misunderstanding others.

  • Idols of the Theater: Errors formed from dogma (institutionalized doctrine) and flawed demonstrations.


All of these exist in software testing. As testers, we should be questioning these “idols”, not worshiping them. Sometimes questioning them may prove them right.

Bacon did not ask anyone to abandon their beliefs without cause. Instead he asks that we not make them idols capable of leading us to ignore what would be obvious if we weren’t looking through the distorted mirror of our idols.

A modern day simplification of Bacon’s arguments may be the Agile Manifesto. We should not let our idols of process, documentation, contracts, and plans prevent us from accomplishing the desired goal. Process, documentation, contracts, and plans are only good in as much as they help. They should not prevent us from seeking improvement.

In some ways I believe that the promotion of testing folklore is the result of an industry-wide desire to show that we are mature — as mature as the engineering of physical products. I believe that eagerness to demonstrate maturity helps lead to the implementation of bad processes and cerfifications. Ironically, enforced process (see the bottom of the FSOP cycle) works best for the immature and gives the impression that anyone that can follow the process can test software.

Don't get me wrong. Process and documentation are good things that help even the smartest people when appropriately applied.

"The only thing that interferes with my learning is my
education." - Albert Einstein


We need to seek continual improvement. It is sad that process and certification often become idols that overshadow the real goals.

May 4, 2007

Heuristics in Test Automation

"A Requirement is a quality or condition that matters to someone who matters."
- Cem Kaner, James Bach, and Bret Pettichord
Lessons Learned In Software Testing
Automated tests are usually coded to perform validations against written requirements. Computers are deterministic in that they need specific instructions regarding what to test, what counts as a passed test, and what counts as a failed test. This is one of the weaknesses of test automation. Many requirements exist beyond the hard written requirements. Test automation can be a great tool for measuring hard requirements. For example, automation can be great for validating mathematical calculations.

Test automation can also be a great tool for testing the fuzzy requirements through the use of heuristics. In addition to coding validations (oracles) for hard facts, automation can be used to report things that require human attention. Automation can report information to help direct the attention of human testers.

"Only weak bugs have logic to them ... Subtle bugs have no definable pattern -- they are wild cards."
- Boris Beizer
Software Testing Techniques

A home-grown (by someone else -- later enhanced by me) test automation tool I used many years ago was built to report "pass", "fail" or "inconclusive" for each test it performed. It had been identified that there were many cases where human judgement and/or investigation was required to determine if something really passed or failed. In some cases, it was just not economical to code a validation for something that humans process better than machines. Therefore, instead of trying to make automation do it all, create automation that does what computers do best and let thinking human testers pick up where the computers stop.

I once automated data validation for a large pricing database. It was suspected that there were numerous errors in this database. Finding and fixing possible errors in millions of records was a daunting task. Instead of creating complex calculations to try to completely automate the validation, I coded simple heuristic rules. When these rules failed, then the "failure" was reported to human testers and data editors for investigation. These rules were things like:

  • Suggested retail price is greater than wholesale price
  • Current price is within 10% of the previous price
  • Generic equivalent price is less than the name-brand price

Were these test oracles always true? No. In some cases it was correct for the data to fail the above tests. However, the automation helped direct the attention of testers. These heuristic validations also helped expose unexpected patterns and led to finding bugs in the software that processed and formatted the data.

Instead of creating complex test automation tools, Harry Robinson suggests that we build "The Simplest Thing That Could Possible Find A Bug". Sometimes this means that we code heuristic validations instead of complex validations that report results with absolute certainty. Let the computers report things that a human tester should investigate. Instead of "inconclusive", Harry uses the term "suspicious". I like that.

The next time you automate testing, in addition to thinking of things that computers can report as "passed" or "failed", think of things that it might be able to report as "suspicious".

May 2, 2007

Hey Dad, when I grow up, I also want to be square.

Erkan Yilmaz, a fellow tester blogger from Germany, recently pointed out that the slogans in my post "Slogans are models" may not transmit their message across languages and cultures. He attempted to guess at what some of the slogans meant without the context of American culture and advertising. These slogans that most Americans will instantly understand didn't work very well out of their context.

We both saw this as an example of how recipients of information do not always have the full context in which the information originated. As testers, we need to admit when we don't understand and seek the answers (and context) from those that know. Sometimes we may need to bring subject matter experts into the conversation to fully understand what we are testing. If you don't know, ask questions. If you think you know, ask questions. You are bound to learn something.

I recently sat in some presentations by people from the "business" (as in not IT) side of some projects in which I am involved. I learned a great deal about how customers use our products and the business' vision for the future of the products. This information will help me better test the products. It is good to know more about the context in which the products I test are used.

Know thy user, for he is not thee.
-
David S. Platt


After our exchange about the American slogans, Erkan provided the following list of slogans from German-speaking countries for interpretation by those of us that live outside that context.

1. “Hey Dad, when I grow up, I also want to be square.”
2. “We wake up earlier.”
3. “We can do everything but speak Standard German.”
4. “With the second eye you see better.”
5. “Firm as a rock in the surge.”
6. “We demand and bring forward personalities.”
7. “Try it in a gentle way.”
8. “It was never so valuable as in these times.”
9. “If it makes you beautiful…”
10. “Well, is today already christmas ?”



What do you think these mean?

See Erkan's original post of this list here.
And after you have tried to interpret the list, look here for their real meanings.

Viel Spaß!

Ben

April 28, 2007

Heuristics In Software Testing

The point of philosophy is to start with something so simple as to seem not worth stating, and to end with something so paradoxical that no one will believe it."
- Sir Bertrand Russell

A heuristic is a commonsense guideline used to increase the probability of solving a problem by directing one's attention to things that are likely to matter. The word is derived from the Greek "heurisko" which simply means "I find". The exclamation "eureka", meaning "I found it!", shares roots with heuristic.

Just as the "Eureka!" screaming forty-niners of California's Gold Rush did not have secret knowledge or tools to tell them exactly where all the gold was buried; software testers don't know exactly where the bugs are going to hide. However, both software testers and gold miners know where bugs and gold have been found before. We can use that knowledge of past discoveries and the nature of what we seek to create heuristics that help us narrow in on areas most likely to contain the treasure.

Gold miners and testers can find treasure by accident. However, intentional exploration for bugs and gold are more likely to produce results than aimless wandering. That last statement is a heuristic. It is true most of the time, but sometimes it can be proven false. Sometimes wandering testers and miners stumble into something very important. I just don't want to do all my testing by accident.

Heuristic-based testing may not give us concrete answers, but it can guide us to the important things to test. Heuristics can also be used in automation to provide information to guide human testers.

There was a time that I told developers and project managers that I could not test their products when the requirements did not include straightforward "testable" criteria. I thought that I could not test without being able to report "pass" or "fail" for each test.

A good example was a requirement that stated something like "the user shall not have to wait an unacceptable amount of time". As a good quality school tester working in a factory school organization, I demanded to know how long was acceptable before I could start testing. I wanted to quantify "unacceptable". In this case, the truth is that "unacceptable" will vary based on the user. There were no contractual SLAs to satisfy. I may not be able to report that the requirement is met, but I can provide useful information to the project team to assist in determining if the performance is acceptable.

I have since learned that answers to heuristic questions are useful. It was in that same project that I started applying heuristics to automated data validation. Even without concrete requirements, we testers can provide provide information that is useful in answering important testing questions -- especially the qualitative questions. (As a side note, I am now amazed at how much we who call ourselves "Quality Assurance" like to focus on quantitative requirements and metrics.)

To use heuristics in testing, create a list of open-ended questions and guidelines. This will not be a pass/fail list of test criteria. It will not be a list of specific test steps. Instead it can be used to guide your test scripting and exploration for bugs. You will likely develop general heuristics that you can apply to all your testing and specific heuristics that apply to specific applications.

We need to be careful to apply heuristics as heuristics and not enforceable rules. For example, most people involved in testing web application have heard the heuristic that every page should be within three clicks of any other page. Applying this "rule" to web application design usually results in better usability. However, it does not always improve usability. Sometimes making every page within three clicks of another is not reasonable. Adding too many links are likely to confuse users more than they help. (Another heuristic?) Complex work flows often require that pages be more than three clicks away from others. Common sense needs to be applied to heuristics to ensure they are applied only when they fit the context.

Happy bug prospecting.

Eureka!

Next time.... apply heuristic oracles to test automation.

April 26, 2007

How Doctors Think

Michael Bolton recently brought Dr. Jerome Groopman’s newest book, “How Doctors Think”, to my attention. Michael suggested to the software-testing list that this book contains information that is relevant to software testing. In this book, Dr Groopman explores how doctors diagnose medical problems and what can be done to reduce bad judgment.

Michael Krasny recently interviewed Dr. Groopman on public radio about this book. You can listen to the interview here. Dr Groopman made a number of statements during this interview that I believe apply to us software testers’ search for and diagnosis of software bugs. I expect to find more in the book.

Here are some gems from the radio interview:

About computer-assisted diagnosis software.

It’s a two-edged sword. On one hand, it can be useful if you feed in the most important and relevant symptoms, but the problem is that you need to be able to illicit them effectively from the patient and so you return to language and you return to some of these thinking errors or these cognitive traps that we can all fall into. So if you put in for, example the first prominent symptom that you hear from someone into the software package, it may be that the fourth thing that the person complains about is the most relevant one. Recently you may have seen, for example, in reading mammograms, that computer assisted diagnosis actually generated more false positives. It caused the radiologist to think that shadows which he or she would normally think were benign were in fact cancer and didn’t add to the diagnosis of cancer –the true positives. So I think this technology is worth exploring, but I think that we have to be very careful about it because it some ways it can seem seductive. You cannot substitute a computer for a human being who listens and filters and judges and elicits from the patient the full story.

The same applies to test automation. It can be a useful tool if we don’t let it lead us astray. We human testers need to be actively involved in listening, filtering, judging, and eliciting the full story.

About time-constraints:

Its one of the biggest issues in the current delivery of medical care. You know, everyone is being told as physicians to see patients in 12 minutes. See patients in 11 minutes. So you’re sitting there with one eye on the clock and you can’t think well in haste. And there’s the natural inclination to anchor onto that very first impression as they stop there and just function as if you’re working on an assembly line in a factory. ... patients are being ill-served.
…
I think everyone in the medical system feels under siege. There’s not a lot of satisfaction but in a way this is penny-wise and pound-foolish because the cost of misdiagnosis is extraordinary. … It also costs in terms of dollars. Its much more
expensive to care for and treat an advanced problem than to make a diagnosis early on.

I think we have to force ourselves to resist and part of that can be done: sometimes we have to extend a visit. But we really are beholden to administrators and managed care plans, so one of the things that I’ve begun to do is if I can’t get to the bottom of a problem in my 15 minutes allotted visit,
then I say to the patient, you know, I haven’t figured it out. I need to think more. ... reschedule an appointment and spend more time.


Testers also make mistakes under time pressure. We also need to be aware that our first impressions may not be right. Testing is not a factory assembly line (even though this is a widely-held view). Testers need to be engaged throughout testing and not just mindlessly follow test scripts. And sometimes we need to lobby to spend more time than the schedule originally gives us.


About the art vs. science of diagnosis and improving the effectiveness of that diagnosis:

There’s a seductive nature of numbers, but statistics from clinical studies and so on are just averages and they may not reflect the individual in front of you. ... So real judgment involves customizing medicine: looking at scientific data but also seeing how it applies or doesn’t apply to the person sitting in front of you.
...
It’s not technology, but its language. Language is really the bedrock of medicine. Most of what we do with a doctor – or should be doing with a doctor – is talking… engaging in a dialog.
...
... all of us are wired to make these kinds of snap judgments to sort of go with our first impressions. to rely very heavily on intuition. … That often works but in medicine unfortunately too often it doesn’t work because we have a tendency to latch on to that first bit of information that we get from a patient and then run with it; as opposed to keeping our mind open and constantly doubting ourselves.
Testing is not an assembly line process. Testers need to keep their minds engaged throughout the testing process. We should not ignore our snap-judgements (blink testing), but we also need to look and think beyond those first impressions. We need to continually question both the software and our own judgement as we test.

We need to communicate more than numbers. We need to communicate stories. We need to translate bugs and metrics into language that matters to the business.

April 21, 2007

How many tests do you need?

“Testing is potentially an infinite process.” – James Bach

Nearly 30 years ago, Glenford Myers provided a self-assessment test to software testers in his book The Art of Software Testing. Much of this book is outdated; however, his test is still regularly used to illustrate the complexity of testing software. He asked readers to write down the test cases required to adequately test the following program.

The program reads three integer values from a card. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.


Glenford Myers then fills the next page of the book with a list of 14 questions that should be answered by test cases for this simple program. He then makes the point that more complex programs are exponentially more difficult to test.

Elizabeth Hendrickson recently created a modern version of the program (no punch cards are required) and challenged testers to create test cases and test her version. See "Testing Triangles: a Classic Exercise Updated for the Web" in her blog.


Last summer, James Bach offered a new challenge to handful of testers sitting around the table at a Denver eatery. This challenge involves testing an even simpler system. James asked how many times we need to press the button on the following system to test 85% of the possible results.

You are asked to test a black box with a single push button. Pressing the button spins an internal wheel that randomly stops at one of 100 possible positions. If any of the 100 possible results (otherwise unknown to the user) encounter a bug, the box will burst into flames. The system has no other output.

Have an answer? To help, let's consider a couple other systems.

How many times do you need to flip a coin to get 85% of the possible results? A coin has two possible results. The first flip will get you 50% of the possible results. Each additional flip will have a 50% chance of not getting the remaining result. Therefore, it is possible -- although unlikely -- that we will ever get a result of heads and a result of tails.

How many times do you need to roll a die to get 85% of the possible results? A die has six possible results. The firs roll will produce a new value. The chance of getting a new value will drop as each new value is encountered. As with the coin, it is possible that we will never encounter all the possible results.

  • 1 value: 83% chance
  • 2 values: 67% chance
  • 3 values: 50% chance
  • 4 values: 33% chance
  • 5 values: 17% chance

Let's go back to James' challenge. How many time do you need to press the button to test 85% of the possible results? How about 95% of the results? And I will add the question: Is 100% test coverage feasible?

Click here to test a JavaScript implementation of this black box.

What if this black box had 200 possible values? What if there were 307.200 possible values?

Have an answer? Please share it.

April 19, 2007

Taxing my badometer

I've used great software. I've used some horrible software. I've used lots of software with annoying bugs. I regularly spend a great deal of time working around bugs in software that I use on a daily basis. However, I think I have recently found the worst yet. It is not the worst because it fails to work. It is not the worst because it is missing features I want. It is the worst because it appears to work on the surface yet generates bad data that has a direct impact on my stress level, my finances, and my interaction with the IRS.

Dealing with the IRS -- or even thinking about it -- is already stressful enough. The tax code is complicated enough that it is easy to accidentally report the wrong income or deductions. If it were easy, we'd see accountants lining up at soup kitchens instead of raking in the cash helping us common folks with our tax returns.

For most of the past 10 years, I have elected to use commercial tax preparation software to help me complete and file my tax returns. I've used some tax software that was good, and some that was bad. I've even tried some freebie and online options. The software I used this year is nothing short of a software atrocity.

This software was swarming with bugs -- although most were not obvious. I suspect that most users didn't even notice.

After installing the software, I dutifully entered more financial data than I care to track. I tried a couple what-if scenarios in the process: such as a last minute IRA contribution. I also entered some estimated values before entering the real data so that I could determine if I should take one option or another. I also entered some data for deductions that I later removed after learning (due to reading the IRS publications) that they not apply to my situation.

I did not follow the interview process from start to finish. I went back and forth in this process entering data, revising data once I clarified the rules or gathered up all my evidence for income and deductions. After several hours, I thought I had a complete return. I had entered and verified all my data in the software's "interview" interface.

Then the automated error check failed. It would not let me continue until I fixed data in a form. This was a form that I had accidentally selected somewhere in the process that contained no data. I then tried to delete the form by unchecking a box that made the options that went the form available. The software then informed me that I could not delete the form there. I went to the portion of the interview where it told me to delete the form and there was no delete option. I then discovered that I could delete the form from the program's "forms" list. However, I later found out that going back to any point in the interview recreated the form. There was no option in the software to permanently get rid of my accidental selection.

I discovered that there was no way to print my return for review until I got to the last step of the eFile process. This required that I go back in the process and tell it that I wanted to file on paper. I printed the return and was surprised to see that the data on the return did not match the current data in the interview interface. Some values that I had entered and then deleted or changed were in the printed return. My return was inaccurate. I was glad that I went through the tedious process of getting to the print feature. If I had eFiled without first printing, I would have submitted a return that did not match the data I entered -- and the data shown to me on the screen.

I then spent several hours deleting and reentering data until the printed forms finally matched the data I entered.

In the midst of my review, I noticed that the software had forced me to take an option that resulted in a lower federal tax bill. However, selecting this option instead of another equally valid option significantly increased my state tax bill. The software forced me to have an overall higher tax bill because it only considered the federal return when deciding which option was best for me. After about an hour, I realized that I could remove a deduction in order to force the software to take the alternate option.

Once everything was in order and the displayed and printed data matched, I selected the eFile option to submit my return. I entered bank account information to pay my taxes online. I then discovered that the eFile service I paid for did not allow me to submit electronic payment to the state. My returns were submitted. My federal payment was sent electronically. However, my state payment still had to be mailed. I could have filed my state taxes and paid online using the state's web site as I had done in previous years. Instead I thought it would be easier to use the integrated system. I wish that the software would have told me that I could not electronically submit my payment before it took my money for the eFile service.

And to top it all off, the software did not tell me how to send in my payment by mail. I spent nearly an hour going through the system's help and the vendor's web site seeking guidance. I finally found an option for an online chat with customer service. It then took customer service a half hour to figure out how and where I should send my payment.

I asked if I could have a refund due to the buggy software and the eFile service that was not as good as the state's free eFile service. I was directed to call them by phone to get a refund because the could not give refunds via the online chat. I called the number and waited on hold of over a half hour before I was disconnected. I called back and got a recording stating that the customer support office was closed.

The bad software was topped off by bad customer service. I was angry. My badometer was pegged.

I then did a little exploratory testing in the application. I discovered additional bugs that created incorrect tax returns. I noticed that the software said it would calculate things for me and then ask me for the value without giving any assistance. I found places where the instructions in the application did not match the IRS publications. I even found a way that I could get it to calculate a refund of any value on the state return without changing the income or deductions.

I understand that tax software companies have a very small window in which to do their development and testing. I am amazed that this software was released. The poor quality -- and the agravation it caused -- have ensured that I will not be a returning customer next year.

I suspect that the software passed all the test cases created by its makers. I suspect that it even passed multiple automated tests. This just goes to show that passing all the preconceived scripted tests does not make a quality product. I do not know what methods this company uses for testing their software, but I suspect that it is primarily scripted testing.

I believe that exploratory testing (and model-based automated testing) would have been more likely to find the bugs I encountered than scripted testing.

As we test software, we need to consider the infinite possibility of data and work flow variations. We can't test all the variations, but we can vary what we test. It is easy for individual testers to select similar test options when they think they are selecting data randomly. Seek out variety. Use random data generation tools to provide more variety.

How would you test tax software? How would you ensure that bugs like the ones I encountered don't aggravate customers?

April 16, 2007

Performance Testing Lessons Learned

Web and client/server load testing can easily become a complex task. Most people I've met got started in load testing with only minimal training in using the test tools. This is how I got started in load testing -- although I had an advantage in that I had been exposed to load testing of communications systems. I also had experience with automated single-user performance testing. I had led some small-scale manual load tests with multiple testers on a conference call hitting the same client-server application at once. (And we found some show-stopping bugs doing that manual testing.) I had watched others perform load tests. I had read numerous load test plans and reports. However, I had never directly participated in executing automated load tests... then I was asked to lead a load testing project.

Through the years, I have made many mistakes designing, scripting, and executing load tests. Load testing easily becomes complex. Tool sales people sometime tell us that nearly anyone can create tests with their tools. (Yet buying test tools is sometimes just like buying a new car: the salesman tells you that the car is reliable and has a great warranty; then the finance person warns of everything that could go wrong that isn't covered in the warranty and trys to sell you an extended warranty and maintenance contract.) Learning the mechanics of how to use a tool are often the easy part. Its what you do with the tool that matters.

Here is the short list of some of the important performance/load testing lessons I have learned. Some I learned from my own experience. Some I learned from the failures of others.

  • Bad assumptions waste time and effort
    • Ask questions
    • Performance testing is often exploratory
    • Expect surprises
    • Prepare to adapt
  • Get to know the people in your neighborhood: no single person or group is likely to have all the required information
    • Subject-matter experts
    • Developers
    • System administrators
    • Database administrators
    • Network engineers
  • Don’t script too much too soon: you may end up tossing out much of what you script
    • Applications change
    • Real usage is difficult to estimate
    • Tool limitations may be discovered
  • Different processes have different impacts: what users do can be as, or more, important as how many users are doing it
    • Include high-use processes (80/20 rule)
    • Include high-risk processes
    • Include “different” processes
  • Modularize scripts: simplify script maintenance -- but only when you intend to run the script again
  • Data randomization is not always a good thing: randomization can make result comparison difficult
  • Code error detection and handling
    • Don’t assume that your tool will handle errors
    • Catch and report errors when and where they happen
    • Realize that errors may change simulated user activity
  • Know your tools and test environment
    • Tool’s supported protocols and licensing
    • Load generator and network resource usage
    • Load balancing and caching mechanisms
    • Beware of test and production environment differences
  • Try to translate results into stories that matter to the applicable stakeholders
    • Tests are run to answer questions: don't overwhelm your audience with hundreds of numbers if they just want a "yes" or "no" answer


and finally…


  • Most performance and load-related problems are due to software code or configuration; not hardware

    • Don’t throw more hardware at a software problem

          April 15, 2007

          Model-Based Test Engine Benefits

          A Model-Based Test Engine (MBTE) is a test automation framework that generates and executes tests based on a behavioral model. Instead of performing scripted test cases, a MBTE generates tests from the model during execution. Instead of implementing models in code, a MBTE can process models defined in tables. Both human testers and computers can understand models defined in tables. A MBTE can be built on top of most existing GUI test automation tools. Combining good automation framework practices with Model-Based Testing (MBT) can transform some common test automation pitfalls to benefits.

          Implementing a MBTE can produce the following:

          1. Simplified automation creation and maintenance.
          2. Simplified test result analysis.
          3. Automatic handling of application changes and bugs.
          4. Generate and execute new tests – and find new bugs.

          More to come...

          April 7, 2007

          Get excited about the negative

          "It is the peculiar and perpetual error of the human understanding to be more moved and excited by affirmatives than by negatives."

          --Francis Bacon

          Selective thinking is the practice of giving more weight and credence to information that confirms our beliefs than information that may contradict our beliefs. We are all guilty of this. We tend to easily believe data that confirms our beliefs and experience. We tend to ignore data that does not confirm our beliefs.

          This confirmation bias can greatly impact our work as software testers. Testers (and developers) often test (and code) to confirm requirements are met by testing the positives. We often overlook the negatives.

          Peter Wason's card problem demonstrates this. The problem involved four cards that each have a number on one side and a letter on the other. They are presented with the following values visible:


          A

          B

          4

          7

          The other side of the cards is not shown.

          The following claim is made: "If a card has a vowel on one side, then it will have an even number on the other."

          The following question is then asked: Which cards do you need to turn over to determine if the above claim is true?

          Try to solve the problem before continuing.

          Research has shown that most people get the answer wrong. The majority of people believe that A and 4 must be turned over to answer the question. This suggests that most people try to confirm the positive when the question requires that we also try to disprove the statement. Both the positive and negative need to be confirmed to answer the question. Click here to see the answer.

          When we test software we need verify both the positive and the negative of the requirements. We need to ensure that the software does what it should do and does not do what it should not do.

          April 6, 2007

          You're too negative

          I believe that some level of pessimism is required to be a good software tester. Some of the best testers I have met are pessimistic towards the systems they test. These black hat testers (of which I am one) consider what might go wrong and ask questions. Although I believe that applied pessimism is a necessary for good testing, this negativity can hurt relationships and the project if it is not constructive. Optimist developers and project managers often have trouble understand us pessimists. The result of our pessimism needs to be better preparedness, not shared depression by all involved in a project. This is the type of pessimism that is the subject of Dr. Julie Norem's book, The Positive Power of Negative Thinking. Dr. Norem defines "defensive pessimism" as

          Defensive pessimism is a strategy used by anxious people to help them manage their anxiety so they can work productively. Defensive pessimists lower their expectations to help prepare themselves for the worst. Then, they mentally play through all the bad things that might happen. Though it sounds as if it might be depressing, defensive pessimism actually helps anxious people focus away from their emotions so that they can plan and act effectively.
          I do not agree that defensive pessimists are necessarily "anxious people". I see the pessimism as a necessary part of good critical thinking.

          Think you might be a defensive pessimist? Take the defensive pessimist quiz.

          While the pessimist black hat is a necessary part of testing, Julian Harty argues that we need to try on Ed DeBono's other hats as well: both in analysis and applied methodology. Take a look at Julian's CAST presentation from last year.