A Good Practice

The Association for Software Testing (AST) is a professional organization dedicated to advancing the understanding and practice of software testing. The AST provides forums for academics, students, and testing practitioners to discuss testing. AST does this through online forums, workshops, education programs, and conferences. The third annual Conference of the Association for Software Testing (CAST) provides a great forum for face-to-face conferring. This is not your typical conference where experts talk at the masses. This is the software testing conference that puts the confer back in conference.

Ever sit in a presentation about testing and think anything like the following?

• Yeah that works for you but it'll never work in my situation.
• What do you mean by X?
  
• She must work with idiots.
• How does he know what he says? I want to see data.
• My management would never go for it.
  
• What planet is he from?
• You're full of it.
  

Your are not only free to think these things at CAST, but you are free to question presenters. Time is built into the program for facilitated discussion of every presentation.

If you'd like to become a better software tester, join AST and come to CAST.

If you'd like to meet and confer with peers from around the world, join AST and come to CAST.

If you'd like to meet and confer with testing experts, join AST and come to CAST.

If you'd like to be challenged, join AST and and come to CAST.

If you'd like to hear Gerald Weinberg talk about the past, present, and future of software testing: join AST and come to CAST.

If you can't afford those other testing conferences, join AST and come to CAST -- its about half the price of other conferences.

If you'd like to compete against other testers, join AST and come to CAST.

If you care about software testing, join AST and come to CAST.

Only through judgment and skill,
exercised cooperatively throughout the entire project,
are we able to do the right things at the right times
to effectively test our products.
context-driven-testing.com

While I am not a believer in best practices, I believe its a really good practice to participate in CAST 2008. See you in Toronto.

Green Means Go?

Traffic control devices are used on roads to help regulate the flow of traffic. When I taught defensive driving classes, I ensured that each class included a discussion about these devices. It is imperative that all drivers understand what each device means. My first Driver License test (in Germany) required that I properly identify 94 of 100 different signs to pass. There was a time that each local governing authority created its own traffic control devices. However, it was not long after the automobile became common that governments began working together to standardize these safety-critical devices. While there is no universal standard that is really followed (the USA being one of the countries that differs from most), standardization within each country (and some continents) has led to safer streets and highways.

These devices include signs, signals, pavement markings, barricades, and policemen. These devices are tools used to control the flow of traffic. However, these devices cannot really control traffic -- except for massive barricades and armed policemen. They provide information to drivers but they cannot force drivers to be safe or legal. As can be seen here, sometimes drivers ignore the signals.

Traffic lights are mostly standardized around the globe. Green means go. Yellow means caution. Red means stop. ... except for the extraterrestrial visitor in the movie Starman who learned by watching a bad example.

Red means stop;
Green means go;
and Yellow means go very very fast!
- Starman

The red, yellow, and green traffic light colors have become common in software development, testing, and production monitoring. Traffic light colors are regularly used to report the status of projects, systems, and individual tests.

I like simple status indicators -- in context. One of the first test execution tools I helped create used smiley faces to indicate passed tests and fulfilled requirements. I currently use color coding to indicate status in the test automation I develop. Colors help me quickly find test results that need attention. Simple color coding helps communicate test results at a high level.

I like to see green. I don't like to see red. However, I am not a member of the Church of the Green Bar*. I do not worship the green light. I do not trust the green light. I find green lights and bars useful but wrong. Green lights remove the story from the status.

Essentially, all models are wrong, but some are useful.
- George E. P. Box

A green traffic light may tell us that it should be safe to go but it does not guarantee that it is safe to go. A green light does not indicate that the intersection is clear. A green light does not indicate that it is safe to drive through the intersection. Drivers need to wait for the green light but they still need to check the intersection, identify potential risks, and decide if they believe it is safe to drive through the intersection. A green light means go if it has been determined that it is safe to go.

In the same way, a passed automated test does not mean that the software is good. Green simply means that the coded criteria was met.

Every time we test software -- whether with human eyes and mind, or with automation -- we only monitor the things that we choose to monitor. A human tester may notice things about the quality of a product that are not scripted. Automated test execution will only notice things that are coded in the script -- no matter how many time we run the test. If a factor that might indicate a problem is not part of the automation's green/red (pass/fail) criteria, it will not turn the light (or bar, or text) green.

I also find that there is often a misunderstanding of what automation does and does not do. The coder of the automation may know what it does when they code it. But will they really understand what a green light does and does not mean six months later? How about a year later? Five years later? Do other testers understand what the automation does? Does management understand what the automation does do and what it does not do?

Software systems can easily become complex. Computers allow us mortals to create complex systems that are beyond our ability to fully understand. We testers seek out software problems based on what we understand. We cannot completely test the software. We use a variety of tools and approaches to learn as much as possible. However, we are unlikely to completely understand a complex software system.
- Ben Simo^

We mere mortals and the automation we create are unable to monitor every thing that might matter. Therefore I believe it is dangerous to conclude that green means good.

The same goes for project management and system monitoring systems that use quantifiable metrics to set status.

Traffic lights based on the judgment of the people involved in the project are better indicators of status. If a light is green because a person set it to green, that person should be able to tell me the story behind the decision to make the light green.

Beware automated traffic lights.

[Update]

Automated traffic light indicators in testing tools are only badometers+. A badometer tells us when something is suspected to be bad cannot tell us if it is good. Better traffic light indicators are set by people that consider the risks associated with the information reported by our tools.

So, Does green mean go? Yes, but only after a human being has judged it safe to go.


* I don't know who first coined the term, but I first heard it from Brian Marick.
^ Since I regularly quote other people, I think I am entitled to quote myself. :)
+ A term I think I first heard used by Gary McGraw. Or what it Kim Possible?

Best Practices Aren’t

The first two of the Seven Basic Principles of the Context-Driven School of software testing are:

1. The value of any practice depends on its context.
2. There are good practices in context, but there are no best practices.

As a former quality school gatekeeper, I understand the value of standards – in both products and processes. However, I am concerned by the current “best practices” trends in software development and testing. The rigidity that we demand in product standards can hurt in process standards. Even the CMM (which is often viewed as a rigid process) has “Optimizing” as the highest level of maturity. A mature process includes continuous learning and adjustment of the process. No process should lock us into one way of doing anything.

Nearly 100 years ago, the industrial efficiency pioneer Frederick Taylor wrote “among the various methods and implements used in each element of each trade there is always one method and one implement which is quicker and better than any of the rest”.

I do not disagree that there may be a best practice for a specific task in a specific context. Taylor broke down existing methods and implements (tools) into small pieces and scientifically evaluated them in search of areas for improvement. The problem is that today's best practices are often applied as one-size-fits-all processes. The best practice for one situation is not necessarily the best for all other contexts. And a "best practice" today may no longer be the best practice tomorrow. This is actually the opposite of what Taylor did. Consultants and tool vendors have discovered that there is money to be made taking "best practices" out of one context and applying to all other contexts. It is harder, and likely less profitable, for the "experts" to seek out the best practices for a specific context. Taylor sought out and applied best practices to small elements. Many of today's "best practices" are applied at a universal level.

I am amazed by what appears to be widespread acceptance of “best practices” by software testers. As testers, it is our job to question. We make a living questioning software. We need to continually do the same for practices. Test your practices.

When presented with a best practice, consider contexts in which the practice is not the best. The broader the scope of the best practice, the more situations it is unlikely to fit. Don’t limit your toolbox to a single practice or set of practices. Be flexible enough to adjust your processes as the context demands. Treat process as an example and apply it where it fits and be willing to deviate from the process -- or even apply an entirely different process -- if it does not fit the context.

No process should replace human intelligence. Let process guide you when it applies. Don’t let a process make decisions for you.

Seek out continuous improvement. Don't let process become a rut.

Process is best used as a map; not as an auto-pilot.