Showing posts with label Lessons Learned. Show all posts
Showing posts with label Lessons Learned. Show all posts

October 17, 2007

Green Means Go?

Traffic control devices are used on roads to help regulate the flow of traffic. When I taught defensive driving classes, I ensured that each class included a discussion about these devices. It is imperative that all drivers understand what each device means. My first Driver License test (in Germany) required that I properly identify 94 of 100 different signs to pass. There was a time that each local governing authority created its own traffic control devices. However, it was not long after the automobile became common that governments began working together to standardize these safety-critical devices. While there is no universal standard that is really followed (the USA being one of the countries that differs from most), standardization within each country (and some continents) has led to safer streets and highways.

These devices include signs, signals, pavement markings, barricades, and policemen. These devices are tools used to control the flow of traffic. However, these devices cannot really control traffic -- except for massive barricades and armed policemen. They provide information to drivers but they cannot force drivers to be safe or legal. As can be seen here, sometimes drivers ignore the signals.





Traffic lights are mostly standardized around the globe. Green means go. Yellow means caution. Red means stop. ... except for the extraterrestrial visitor in the movie Starman who learned by watching a bad example.

Red means stop;
Green means go;
and Yellow means go very very fast!

- Starman

The red, yellow, and green traffic light colors have become common in software development, testing, and production monitoring. Traffic light colors are regularly used to report the status of projects, systems, and individual tests.

I like simple status indicators -- in context. One of the first test execution tools I helped create used smiley faces to indicate passed tests and fulfilled requirements. I currently use color coding to indicate status in the test automation I develop. Colors help me quickly find test results that need attention. Simple color coding helps communicate test results at a high level.

I like to see green. I don't like to see red. However, I am not a member of the Church of the Green Bar*. I do not worship the green light. I do not trust the green light. I find green lights and bars useful but wrong. Green lights remove the story from the status.

Essentially, all models are wrong, but some are useful.
- George E. P. Box

A green traffic light may tell us that it should be safe to go but it does not guarantee that it is safe to go. A green light does not indicate that the intersection is clear. A green light does not indicate that it is safe to drive through the intersection. Drivers need to wait for the green light but they still need to check the intersection, identify potential risks, and decide if they believe it is safe to drive through the intersection. A green light means go if it has been determined that it is safe to go.

In the same way, a passed automated test does not mean that the software is good. Green simply means that the coded criteria was met.

Every time we test software -- whether with human eyes and mind, or with automation -- we only monitor the things that we choose to monitor. A human tester may notice things about the quality of a product that are not scripted. Automated test execution will only notice things that are coded in the script -- no matter how many time we run the test. If a factor that might indicate a problem is not part of the automation's green/red (pass/fail) criteria, it will not turn the light (or bar, or text) green.

I also find that there is often a misunderstanding of what automation does and does not do. The coder of the automation may know what it does when they code it. But will they really understand what a green light does and does not mean six months later? How about a year later? Five years later? Do other testers understand what the automation does? Does management understand what the automation does do and what it does not do?

Software systems can easily become complex. Computers allow us mortals to create complex systems that are beyond our ability to fully understand. We testers seek out software problems based on what we understand. We cannot completely test the software. We use a variety of tools and approaches to learn as much as possible. However, we are unlikely to completely understand a complex software system.
- Ben Simo^

We mere mortals and the automation we create are unable to monitor every thing that might matter. Therefore I believe it is dangerous to conclude that green means good.

The same goes for project management and system monitoring systems that use quantifiable metrics to set status.

Traffic lights based on the judgment of the people involved in the project are better indicators of status. If a light is green because a person set it to green, that person should be able to tell me the story behind the decision to make the light green.

Beware automated traffic lights.

[Update]

Automated traffic light indicators in testing tools are only badometers+. A badometer tells us when something is suspected to be bad cannot tell us if it is good. Better traffic light indicators are set by people that consider the risks associated with the information reported by our tools.

So, Does green mean go? Yes, but only after a human being has judged it safe to go.


* I don't know who first coined the term, but I first heard it from Brian Marick.
^ Since I regularly quote other people, I think I am entitled to quote myself. :)

+ A term I think I first heard used by Gary McGraw. Or what it Kim Possible?

July 12, 2007

Woodpeckers, Pinatas, and Dead Horses

Here's some short blurbs of a few things I took away from CAST sessions.

From Lee Copeland's keynote address:
  • "It's nonsensical to talk about automated tests as if they were automated human testing."
  • Write or speak about something you're knowledgeable and passionate about.
  • Combine things from multiple disciplines.

From Harry Robinson's keynote address:
  • Weinberg's Second Law: If Builders Built Buildings The Way Programmers Write Programs, Then The First Woodpecker That Came Along Would Destroy Civilization.

From Esther Derby's keynote:
  • To successfully coach someone, they must want to be coached and want to be coached by you.

From James Bach's tutorial:
  • Pinata Heuristic: Keep beating at it until the candy comes out. ... and stop once the candy drops.
unless ...
  • Dead Horse Heuristic: You may be beating a dead horse.
yet beware ...
  • If it is a pinata, don't stop beating at it until the candy drops; but if it is a dead horse, your beating is bringing no value. It can be a challenge to determine if its a pinata or a dead horse.
From Antti Kervinen's presentation:
  • Separate automation models into high level (behavior) and low level (behavior implementation) components to reuse test models on a variety of platforms and configurations.

More from James Bach's tutorial:
  • Testing does not break software. Testing dispels illusions.
  • Rational Unified Process is none of the three. (attributed to Jerry Weinberg)

From the tester exhibition:

  • Testing what can't be fixed or controlled may be of little value. Some things may not be worth testing.
  • There is great value in the diversity of approaches and skills on a test team.
  • It may be possible to beat a dead horse and test (and analyze) too much. Sometimes we should just stop testing and act on the information we have.

From Doug Hoffman's tutorial:
  • Record and playback automation can be very useful for testing for the same behavior with many configurations. And, once the script stops finding errors: throw it out.

From Keith Stobie's keynote:
  • Reduce the paths though your system to improve quality. Fewer features may be better.
  • Free web sites often have higher quality than subscription sites. This is because it is easy to measure the cost of downtime on ad-supported systems.

From David Gilbert's session:
  • People expect hurricanes to blow around and change path. We should expect the same with software development projects. (David has some interesting ideas about forecasting in software development.)
  • Numbers tell a story only in context. You must understand the story behind the numbers.
One more from James:
  • Keep Notes!


What did you take away from CAST?

May 2, 2007

Hey Dad, when I grow up, I also want to be square.

Erkan Yilmaz, a fellow tester blogger from Germany, recently pointed out that the slogans in my post "Slogans are models" may not transmit their message across languages and cultures. He attempted to guess at what some of the slogans meant without the context of American culture and advertising. These slogans that most Americans will instantly understand didn't work very well out of their context.

We both saw this as an example of how recipients of information do not always have the full context in which the information originated. As testers, we need to admit when we don't understand and seek the answers (and context) from those that know. Sometimes we may need to bring subject matter experts into the conversation to fully understand what we are testing. If you don't know, ask questions. If you think you know, ask questions. You are bound to learn something.

I recently sat in some presentations by people from the "business" (as in not IT) side of some projects in which I am involved. I learned a great deal about how customers use our products and the business' vision for the future of the products. This information will help me better test the products. It is good to know more about the context in which the products I test are used.

Know thy user, for he is not thee.
-
David S. Platt


After our exchange about the American slogans, Erkan provided the following list of slogans from German-speaking countries for interpretation by those of us that live outside that context.

1. “Hey Dad, when I grow up, I also want to be square.”
2. “We wake up earlier.”
3. “We can do everything but speak Standard German.”
4. “With the second eye you see better.”
5. “Firm as a rock in the surge.”
6. “We demand and bring forward personalities.”
7. “Try it in a gentle way.”
8. “It was never so valuable as in these times.”
9. “If it makes you beautiful…”
10. “Well, is today already christmas ?”



What do you think these mean?

See Erkan's original post of this list here.
And after you have tried to interpret the list, look here for their real meanings.

Viel Spaß!

Ben

April 28, 2007

Heuristics In Software Testing

The point of philosophy is to start with something so simple as to seem not worth stating, and to end with something so paradoxical that no one will believe it."
- Sir Bertrand Russell

A heuristic is a commonsense guideline used to increase the probability of solving a problem by directing one's attention to things that are likely to matter. The word is derived from the Greek "heurisko" which simply means "I find". The exclamation "eureka", meaning "I found it!", shares roots with heuristic.

Just as the "Eureka!" screaming forty-niners of California's Gold Rush did not have secret knowledge or tools to tell them exactly where all the gold was buried; software testers don't know exactly where the bugs are going to hide. However, both software testers and gold miners know where bugs and gold have been found before. We can use that knowledge of past discoveries and the nature of what we seek to create heuristics that help us narrow in on areas most likely to contain the treasure.

Gold miners and testers can find treasure by accident. However, intentional exploration for bugs and gold are more likely to produce results than aimless wandering. That last statement is a heuristic. It is true most of the time, but sometimes it can be proven false. Sometimes wandering testers and miners stumble into something very important. I just don't want to do all my testing by accident.

Heuristic-based testing may not give us concrete answers, but it can guide us to the important things to test. Heuristics can also be used in automation to provide information to guide human testers.

There was a time that I told developers and project managers that I could not test their products when the requirements did not include straightforward "testable" criteria. I thought that I could not test without being able to report "pass" or "fail" for each test.

A good example was a requirement that stated something like "the user shall not have to wait an unacceptable amount of time". As a good quality school tester working in a factory school organization, I demanded to know how long was acceptable before I could start testing. I wanted to quantify "unacceptable". In this case, the truth is that "unacceptable" will vary based on the user. There were no contractual SLAs to satisfy. I may not be able to report that the requirement is met, but I can provide useful information to the project team to assist in determining if the performance is acceptable.

I have since learned that answers to heuristic questions are useful. It was in that same project that I started applying heuristics to automated data validation. Even without concrete requirements, we testers can provide provide information that is useful in answering important testing questions -- especially the qualitative questions. (As a side note, I am now amazed at how much we who call ourselves "Quality Assurance" like to focus on quantitative requirements and metrics.)

To use heuristics in testing, create a list of open-ended questions and guidelines. This will not be a pass/fail list of test criteria. It will not be a list of specific test steps. Instead it can be used to guide your test scripting and exploration for bugs. You will likely develop general heuristics that you can apply to all your testing and specific heuristics that apply to specific applications.

We need to be careful to apply heuristics as heuristics and not enforceable rules. For example, most people involved in testing web application have heard the heuristic that every page should be within three clicks of any other page. Applying this "rule" to web application design usually results in better usability. However, it does not always improve usability. Sometimes making every page within three clicks of another is not reasonable. Adding too many links are likely to confuse users more than they help. (Another heuristic?) Complex work flows often require that pages be more than three clicks away from others. Common sense needs to be applied to heuristics to ensure they are applied only when they fit the context.

Happy bug prospecting.

Eureka!

Next time.... apply heuristic oracles to test automation.

April 16, 2007

Performance Testing Lessons Learned

Web and client/server load testing can easily become a complex task. Most people I've met got started in load testing with only minimal training in using the test tools. This is how I got started in load testing -- although I had an advantage in that I had been exposed to load testing of communications systems. I also had experience with automated single-user performance testing. I had led some small-scale manual load tests with multiple testers on a conference call hitting the same client-server application at once. (And we found some show-stopping bugs doing that manual testing.) I had watched others perform load tests. I had read numerous load test plans and reports. However, I had never directly participated in executing automated load tests... then I was asked to lead a load testing project.

Through the years, I have made many mistakes designing, scripting, and executing load tests. Load testing easily becomes complex. Tool sales people sometime tell us that nearly anyone can create tests with their tools. (Yet buying test tools is sometimes just like buying a new car: the salesman tells you that the car is reliable and has a great warranty; then the finance person warns of everything that could go wrong that isn't covered in the warranty and trys to sell you an extended warranty and maintenance contract.) Learning the mechanics of how to use a tool are often the easy part. Its what you do with the tool that matters.

Here is the short list of some of the important performance/load testing lessons I have learned. Some I learned from my own experience. Some I learned from the failures of others.

  • Bad assumptions waste time and effort
    • Ask questions
    • Performance testing is often exploratory
    • Expect surprises
    • Prepare to adapt
  • Get to know the people in your neighborhood: no single person or group is likely to have all the required information
    • Subject-matter experts
    • Developers
    • System administrators
    • Database administrators
    • Network engineers
  • Don’t script too much too soon: you may end up tossing out much of what you script
    • Applications change
    • Real usage is difficult to estimate
    • Tool limitations may be discovered
  • Different processes have different impacts: what users do can be as, or more, important as how many users are doing it
    • Include high-use processes (80/20 rule)
    • Include high-risk processes
    • Include “different” processes
  • Modularize scripts: simplify script maintenance -- but only when you intend to run the script again
  • Data randomization is not always a good thing: randomization can make result comparison difficult
  • Code error detection and handling
    • Don’t assume that your tool will handle errors
    • Catch and report errors when and where they happen
    • Realize that errors may change simulated user activity
  • Know your tools and test environment
    • Tool’s supported protocols and licensing
    • Load generator and network resource usage
    • Load balancing and caching mechanisms
    • Beware of test and production environment differences
  • Try to translate results into stories that matter to the applicable stakeholders
    • Tests are run to answer questions: don't overwhelm your audience with hundreds of numbers if they just want a "yes" or "no" answer


and finally…


  • Most performance and load-related problems are due to software code or configuration; not hardware

    • Don’t throw more hardware at a software problem