... as the little boy said, "Today we learned how to spell 'banana', but we didn't learn when to stop." ... In honor of that little boy, we can elevate his idea to a principle, The Banana Principle:
I just had the following exchange with my 12 year old daughter Jessica.
Me: How do software testers know when to stop testing something?
Jessica: When you die!. . . Or when you get really tired of it.
The Banana Principle does not mean that heuristics cannot be useful in determining when to stop. It means that heuristics do not tell us when to stop using the heuristic. There is a tendency to start transforming the most useful heuristics into laws -- in our minds. Heuristics should help us think and not replace thinking. This includes continual questioning of even the most useful heuristics.
For those that put faith in code coverage metrics:
Consider this: If a small change made to the code produces no change in the results of any of the tests, we have evidence of insufficiency of the full set [of tests].
For any system of interesting size it is impossible to test all the different logic paths and all the different input data combinations. Of the infinite number of choices, each one of which is worth of some level of testing, testers can only choose a very small subset because of resource constraints. - Lee Copeland, A Practitioner's Guide to Software Test Design
The complexity of software makes it impossible to test all the possible things we could test for all but the most simple systems. (And I have often argued that even very simple systems cannot be completely tested.) This inability to test everything requires that we testers (and testing developers) identify the things that we believe are most likely to help us fulfill our testing mission. This makes test design very important. We not only need to design our tests in a way that supports our mission -- we need to communicate our testing in a way that supports our mission.
There are many ways that we can design and document tests: from very high level exploratory testing charters to the very specific step-by-step procedures of scripted automation. We often need to communicate a single test using various levels of detail.
A high-level test charter might be fine for communicating to project managers but we may need to describe step-by-step tasks when we document how to reproduce a bug found during exploratory testing. (Tools like Test Explorer can help document exploratory testing.)
High-level test execution steps may be fine for some manual test execution but these steps need to be made explicit for automation. And sometimes the details matter for tests executed by humans.
Detailed test procedures aren't very good for communicating functional coverage to product owners or managers. Sometimes we need to think about even the most scripted tests at a high level and not get bogged down in the details.
Sometimes we need to communicate tests designed and defined at a high level with great detail. Other times we need to communicate low-level automated tests at a high level. Different levels of detail are required by different people at different times.
There was a great deal of discussion at the Agile Alliance Functional Testing Tools Visioning Workshop about the desire to easily define tests using a variety of levels of abstraction and to communicate tests in different ways for different people. We considered how tools could be built to support the disparate needs of people involved in software development.
Elizabeth Hendrickson nicely summed up how tools can help support this by providing "A Place To Put Things". I am all for separating the essence of tests from automation code. Tools like xUnit and FIT aren't great because they do good testing. In fact, these tools don't really do the testing. They are useful tools because they give people a place to put things. When we have a place to put things, we are better organized. When we are better organized, we can communicate better.
Having a place to put things helps keep our testing organized and helps us communicate -- whether we are documenting tests as examples, designing FIT tests, scripting GUI automation, or documenting exploratory testing ideas.
One group of us at the workshop broke off to discuss the things that we testers need a place to put. We considered the possibilities of defining parts of tests at different levels of abstraction.
What if we could easily define tests at the highest possible (or reasonable) level of abstraction and then add details only when and where details are required?
What if a test could be defined at a high enough level that automated test execution engines could run the same tests on different platforms, or with different user roles, or with different data?
We did a little brainstorming and wrote down things we use to document a test and then divided these into three categories -- or levels of abstraction: business requirements (goals), interaction design (activities), and implementation design (tasks). Some items ended up in the twilight zone -- between or occupying multiple levels.
Business Requirements (Goals)
Goal
Expectation
As a ... I want to ... so that ...
Exploratory testing charter
Interaction Design (Activities)
Present / Communicate Results
Communicate test to users, dev, business
Actor
Domain objects
Action
User Preferences
wait for so long
set up pre condition
model-based test generation
Given; When; Then
Wait until
orchestration
Roles
Time Passes...
Branding
Twilight Zone (Somewhere crossing over activities and tasks?)
Domain Models
Verify
System state
data
state transitions
user state
Implementation Design (Tasks)
objects
Check Results
Show
STATES
Do ...
Control GUI, API, test harness
I'm sure that there are many other things we testers would like a place to put that we didn't think of in our few minutes of brainstorming.
Traditionally, defining tests at various levels of abstraction has been difficult. I've seen people try to add abstraction to tests and spend more time maintaining and documenting the various abstractions levels than I think the benefits were worth. I've also successfully used abstraction in test automation to make the same tests executable on multiple platforms.
If we can find the right abstractions to communicate the intent of an example, we might be able to finally break free of the perception of functional tests as brittle, hard-to-understand, write-only artifacts. Even better, we might find a way to layer new tools on top of these abstractions so that, if I want to write my examples in plain text and you want to drag boxes around on a screen and she wants to use the UML, we can each use the form that speaks most clearly to us. ... I want more names for my common things. I want to deal in goals and activities not checkboxes and buttons. I want to give the system a few simple bits of information and have it tell me something I didn't know. I want to show my examples to everyone in the project community and have them lift up their understanding rather than drown it in permutations and edge cases and "what happens if the user types in Kanjii?". - Kevin Lawrence, In Praise of Abstraction
Another group at the workshop worked on devising a framework to give us a common place to put things. If we had a common place to put things, then a variety of tools could use the same data and users could select whatever tools best work for their needs -- and desired level of abstraction. Thanks to Elizabeth for clarifying what I think many were thinking but did not express so clearly: we need a place to put things.
Whether you are trying to create a one-size-fits-many testing framework or a specialized tool to support a specific need: first develop places to put things.
Given the infinite testing possibilities, the best testing tools are those that help us organize, understand, and communicate our tests.
I am not a fan of any of the current software tester certification programs. Perhaps it is because I take the word certification too literally -- which means that I expect it to have real meaning. When I think of certification, I usually think in lines with the IEEE's definition of certification.
certification
The process of confirming that a system or component complies with its specified requirements and is acceptable for operational use.
Perhaps my thinking is biased by my past IV&V testing work. If one can go to a weekend class and become certified, then I question the value of that certification. I believe that certifications based on ability to memorize terms and practices free of context are of little value -- and may do more harm than good. Yet, the purpose of this blog post is not to provide arguments against the current crop of tester certification options. If you'd like to see some concerns about certification, read the following.
If we must have a test certification based on a computer-scored multiple-false test, then I think I have found a test that is better than any others I've yet seen. This test covers many aptitudes and skills that I believe are important for testers:
If I had to select testers based on passing a test, I think I'd take someone that has gotten further than I in this quiz over someone with a software tester certification. I believe this quiz is a better measure of whether or not one is "acceptable for operational use". :)
Traffic control devices are used on roads to help regulate the flow of traffic. When I taught defensive driving classes, I ensured that each class included a discussion about these devices. It is imperative that all drivers understand what each device means. My first Driver License test (in Germany) required that I properly identify 94 of 100 different signs to pass. There was a time that each local governing authority created its own traffic control devices. However, it was not long after the automobile became common that governments began working together to standardize these safety-critical devices. While there is no universal standard that is really followed (the USA being one of the countries that differs from most), standardization within each country (and some continents) has led to safer streets and highways.
These devices include signs, signals, pavement markings, barricades, and policemen. These devices are tools used to control the flow of traffic. However, these devices cannot really control traffic -- except for massive barricades and armed policemen. They provide information to drivers but they cannot force drivers to be safe or legal. As can be seen here, sometimes drivers ignore the signals.
Traffic lights are mostly standardized around the globe. Green means go. Yellow means caution. Red means stop. ... except for the extraterrestrial visitor in the movie Starman who learned by watching a bad example.
Red means stop; Green means go; and Yellow means go very very fast! - Starman
The red, yellow, and green traffic light colors have become common in software development, testing, and production monitoring. Traffic light colors are regularly used to report the status of projects, systems, and individual tests.
I like simple status indicators -- in context. One of the first test execution tools I helped create used smiley faces to indicate passed tests and fulfilled requirements. I currently use color coding to indicate status in the test automation I develop. Colors help me quickly find test results that need attention. Simple color coding helps communicate test results at a high level.
I like to see green. I don't like to see red. However, I am not a member of the Church of the Green Bar*. I do not worship the green light. I do not trust the green light. I find green lights and bars useful but wrong. Green lights remove the story from the status.
Essentially, all models are wrong, but some are useful. - George E. P. Box
A green traffic light may tell us that it should be safe to go but it does not guarantee that it is safe to go. A green light does not indicate that the intersection is clear. A green light does not indicate that it is safe to drive through the intersection. Drivers need to wait for the green light but they still need to check the intersection, identify potential risks, and decide if they believe it is safe to drive through the intersection. A green light means go if it has been determined that it is safe to go.
In the same way, a passed automated test does not mean that the software is good. Green simply means that the coded criteria was met.
Every time we test software -- whether with human eyes and mind, or with automation -- we only monitor the things that we choose to monitor. A human tester may notice things about the quality of a product that are not scripted. Automated test execution will only notice things that are coded in the script -- no matter how many time we run the test. If a factor that might indicate a problem is not part of the automation's green/red (pass/fail) criteria, it will not turn the light (or bar, or text) green.
I also find that there is often a misunderstanding of what automation does and does not do. The coder of the automation may know what it does when they code it. But will they really understand what a green light does and does not mean six months later? How about a year later? Five years later? Do other testers understand what the automation does? Does management understand what the automation does do and what it does not do?
Software systems can easily become complex. Computers allow us mortals to create complex systems that are beyond our ability to fully understand. We testers seek out software problems based on what we understand. We cannot completely test the software. We use a variety of tools and approaches to learn as much as possible. However, we are unlikely to completely understand a complex software system. - Ben Simo^
We mere mortals and the automation we create are unable to monitor every thing that might matter. Therefore I believe it is dangerous to conclude that green means good.
The same goes for project management and system monitoring systems that use quantifiable metrics to set status.
Traffic lights based on the judgment of the people involved in the project are better indicators of status. If a light is green because a person set it to green, that person should be able to tell me the story behind the decision to make the light green.
Beware automated traffic lights.
[Update]
Automated traffic light indicators in testing tools are only badometers+. A badometer tells us when something is suspected to be bad cannot tell us if it is good. Better traffic light indicators are set by people that consider the risks associated with the information reported by our tools.
So, Does green mean go? Yes, but only after a human being has judged it safe to go.
* I don't know who first coined the term, but I first heard it from Brian Marick. ^ Since I regularly quote other people, I think I am entitled to quote myself. :) + A term I think I first heard used by Gary McGraw. Or what it Kim Possible?
In my previous post about the Agile Alliance Functional Testing Tools Workshop , I wrote the following:
After reviewing existing tools used by agile teams: we identified software testing issues that have been solved (yellow), those that have been partially solved (orange), and those that have not been solved (pink). As I recollect, most of the solved issues were technical problems and most of the unsolved problems were people problems. Many of the partially solved problems were those for which I believe we have technical solutions but have not yet been integrated and presented in ways that best support people.
In case you are wondering what problems we wrote down on these notes, Frank Maurer kindly transcribed them for the workshop participants and I have posted them below.
Looking at this list reminds me of the traffic safety problem lists I made in the defensive driving classes I used to teach. As unsafe driving practices were brought up by students, I would add them to a list on the whiteboard. I then asked the students to identify whether each item on my list was primarily due to driver skill or driver attitude. The students usually blamed most of the problems on driver attitude.
Skill and attitude play important roles in software development and testing. Team members need both. Brian Marick addresses this in his guiding values of discipline, skill, ease, and joy. Instead of looking at the functional testing problems as skill or attitude problems, I looked at them as man or machine problems. I asked myself if each problem appeared to be mainly a human problem or a technical problem.
Software development and testing involves a mix of people and technical problems. Interfacing people with people and people with technology is often harder than interfacing technology with technology. Most of the identified problems have both technical and human aspects to the problem and possible solutions. Some are due to the nature of people or the nature of software and will likely never be completely solved.
I find that identifying whether a problem is primarily a people or technology problem helps me identify possible solutions. I quickly scanned this list and identified whether I thought things were primarily human or technical issues. My notes are included to the right of each item. I don't necessarily interpret each item as its author (a human communication problem), and I do not necessarily agree that each item is a problem or belongs in the specified group.
As you review this list, ask yourself if the problem and possible solutions are grounded in people or technology.
Unsolved
Define Test
Human
Unsolved
Organizing large sets of Tests/Expect. Actions/Examples for a large, complex system so you can wrap your head around the whole thing.
Human
Unsolved
Having tests survive handoffs. Project team -> op support -> proj team.
Human and technical
Unsolved
Getting people to care
Human
Unsolved
Transferability of ubiquitous language to other projects
Human
Unsolved
Write SW that is understood
Human
Unsolved
Reducing Uncertainty
Human
Unsolved
Limitations of natural language
Human
Unsolved
How would we act if we really believed code was an asset?
Mostly Human
Unsolved
Allowing Customers to articulate their expectations in a format/tool/way that is comfortable for them
Mostly Human
Unsolved
Multi-model specification text + table + graphic in one test.
Mostly Technical
Unsolved
Domain experts
Human
Unsolved
Fully Automated regr. That does not reduce dev.velocity.
Mostly Technical
Unsolved
Generate a domain model from tests.
Human and technical
Unsolved
Testing usability as part of acceptance testing in incremental development.
Human
Unsolved
Common language to express GUI based tests.
Mostly Human
Unsolved
Conveying "experts" perspective to majority of development team.
Human
Unsolved
Automated Software Development
Technical (Machines aren't creative)
Unsolved
Functional tests that can be easily re-used later in lifecycle.
Mostly Technical
Unsolved
Test business requirements independent of current interaction/Api design.
Mostly Human
Unsolved
Composing tests into useful larger tests
Technical and Human
Unsolved
Test first performance
Human
Unsolved
Getting BA to write the tests
Mostly Human
Unsolved
Having customer to be able to write test and enjoy it.
Mostly Human
Unsolved
Different test notations for different user groups.
Human problem, technical solution?
Unsolved
Acceptance/Functional Tests good for communication and automation.
Human problem, technical solution?
Unsolved
Model the time domain " and 3 months later an email.
Mostly Technical? (Don't want execution to take 3 months.)
Unsolved
Change touch-point dynamically.
?
Unsolved
Terminology (Test or not a Test?)
Human
Partially Solved
Understand what has not been tested.
Human and Technical
Partially Solved
Trace tests into project management tools
Human and Technical
Partially Solved
Getting buy-in for need to automate.(docs,tests,specs)
Satisfying every role's need/desire to be at center, in control.
Human
Partially Solved
Having functional tests specify requirement specifications.
Human and Technical
Partially Solved
Sustain a productive conversation with all stake holders.
Human problem, partial technical solution
Partially Solved
How do we get across what the project would feel like if things were going well.
Human
Partially Solved
Write robust (U.I) tests that are not brittle.
Mostly Technical
Partially Solved
Fragile tests.
Mostly Technical
Partially Solved
Valuing individuals and interactions over processes and tools.
Human
Partially Solved
Describing customer intent.
Human
Partially Solved
Executable (as tests) Models (as specifications)
Human and technical
Partially Solved
Test partitioning
Human and technical
Partially Solved
Tests as support artifacts.
Human and technical
Partially Solved
Test Generation Automation.
Human and technical
Partially Solved
Running tests parallely ( Fast feedback)
Mostly technical
Partially Solved
Reconciling preferred style of abstraction.
Human and technical
Partially Solved
Dealing with size.
Human
Partially Solved
Finding the right words in which to write a test.
Human
Partially Solved
Express requirement in the domain language, graphical, word based , table based.
Mostly Technical
Partially Solved
Functional Test driven development..not just for agilists. How to sell to waterfallists?
Mostly Human
Partially Solved
How to Integrate tools?
Mostly Technical
Partially Solved
Common test case format.
Human and Technical
Partially Solved
Cooperation and collaboration between tool developers (tool stack, tool platform).
Human and Technical
Partially Solved
Change from one notation to another. (graphic -> tabular)
Mostly technical
Partially Solved
Test/Example -> model (generated model based tests)
Mostly technical
Partially Solved
IDE for testers and BAs
Human and technical
Partially Solved
Get all roles actively involved
Mostly Human
Partially Solved
View specs/examples/tests, differently for different roles.
Mostly Technical
Partially Solved
Different editors for different roles?
Mostly Technical
Partially Solved
Super-power IDE
Technical solution to human problems?
Partially Solved
Test Refactoring
Human and technical
Partially Solved
Refactoring tests
Human and technical
Partially Solved
Prioritize and execute tests to get faster feedback.
Mostly Human
Partially Solved
Describe a test at an appropriate level of abstraction
Mostly Technical
Partially Solved
Choosing what to automate (when you can't automate everything)
Mostly Technical
Partially Solved
Tools to support exploratory testing
Human and Technical
Partially Solved
Reusable test artifacts (poor modularity cohesion)
Mostly Technical
Partially Solved
Build community with BAs
Mostly Human
Partially Solved
Allow for refactoring from /to code <-> tests <-> req'ts
Mostly Technical
Partially Solved
Setup a wiki to discuss smaller problem solving.
People
Partially Solved
Capture war stories + testimonials + experiences.
Human, partial technical solution
Partially Solved
Test Maintenance
Human and Technical
Partially Solved
Book: Patterns of Self testing software
?
Partially Solved
Shared vocabulary around parts of a functional testing solution ("Fixture",etc)
Human
Partially Solved
Ensure adequate test coverage.
Mostly Technical
Partially Solved
Communication of what has been tested
Human and technical
Partially Solved
Communication using the ubiquitous language.
Human
Solved
Express automated/able tests in tables
Mostly Technical
Solved
Correctly implementing programmer intent
Mostly Technical
Solved
Deliver SW to test that doesn’t crash immediately
Mostly Technical
Solved
Provide traceability between Story or Requirement and accpetance/functional test
Mostly Technical
Solved
Gui Testing - functional testing is more than GUI testing
Technical
Solved
Data-Driven Testing
Technical
Solved
Driving Apps
Technical
Solved
Integrating test executors/drivers with build process
Technical
Solved
Edit Fit tests from eclipse
Technical
Solved
Report Results
Technical to report, human to be understood
Solved
Express expectations in code
Human and technical
Solved
Unit testing
Mostly Technical
I think we can solve the technical issues and use technical solutions to help people manage some of the people problems. Applying discipline and skill to solve the technical problems may help add to testers' ease and joy.
Where do you think the solutions lie? Have a solution? Please share it.
I spent the second half of last week at the Agile Alliance Functional Testing Tools Visioning Workshop. (How's that for a long name?) Before the workshop, I was thinking that it seemed a little oxymoronic to have an agile workshop with a focus on tools. Perhaps my thinking was triggered by my concerns about those who seem to value "agile" processes and tools (often ones they sell) more than people.
Agile people are supposed to care about people and not care about tools. Right? Wrong.
while there is value in the items on the right, we value the items on the left more
Software is developed by people for people. Agility involves building better software by adapting to the needs of people instead of letting processes and tools lead the way. Process and tools do best when they have a supportive role in software development. Better tools can support agility but they cannot make anyone agile.
The tool-centric discussions at the workshop were driven by a desire to build better software for people that build and test software. It is about people.
It then seems quite apropos that the book I indiscriminately grabbed off the shelf (well, I picked it for its size more than its content) to read on the airplane to and from the workshop is Ben Shneiderman's Leonardo's Laptop: Human Needs and the New Computing Technologies. The first chapter contains the following paragraphs that affirm my thinking about the role of automation in software testing. (Emphasis is mine.)
The first transformation from the old to the new computing is the shift in what users value. Users of the old computing proudly talked about their gigabytes and megahertz, but users of the new computing brag about how many e-mails they sent, how many bids they made in online auctions, and how many discussion groups they posted to. The old computing was about mastering technology; the new computing is about supporting human relationships. The old computing was about formulating query commands for databases; the new computing is about participating in knowledge communities. ...
The second transformation to the new computing is the shift from machine-centered automation to user-centered services and tools. Instead of the machine doing the job, the goal is to enable you to do a better job. Automated medical diagnosis programs that do what doctors do have faded as a strong research topic; however, rapid access to extensive medical lab tests plus patient records for physicians are expected, and online medical support groups for patients are thriving. ... Natural language dialogs with computerized therapists have nearly vanished, but search engines that enable users to specify their information needs are flourishing. The next generation of computers will bring even more powerful tools to enable you to be more creative and then disseminate your work online. This Copernican shift is bringing concerns about users from the periphery to the center. The emerging focus is on what users want to do in their lives.
Although many think of us software testers and developers as eccentric nerds, software developers and testers are human too. Like other humans, we desire tools that help us do a better job. This was the theme of the workshop: envisioning ways that tools can help us do a better job testing software.
After reviewing existing tools used by agile teams: we identified software testing issues that have been solved (yellow), those that have been partially solved (orange), and those that have not been solved (pink). As I recollect, most of the solved issues were technical problems and most of the unsolved problems were people problems. Many of the partially solved problems were those for which I believe we have technical solutions but have not yet been integrated and presented in ways that best support people. Much of the "what's next" discussion at the workshop was focused on how to integrate existing tools that each partially solve problems but together could move problems to the solved group.
Once the technical problems are solved, we can work on the tools to help with the people problems: we can move from old computing to new computing.
In Leonardo's Laptop, Ben Shneiderman presents a framework for integrating creative activities of people. This framework for mega-creativity consists of four activities:
Collect: Learn from what exists
Relate: Consult with peers and mentors
Create: Think: explore solutions
Donate: Disseminate the results and contribute
This is not a waterfall process. It is an interactive iterative framework for innovation. Shneiderman's book focuses on the need to develop software to support this framework. The participants in the functional test tool workshop focused on the need to develop testing software to support those developing software to support this framework. And in doing so, we exhibited this framework in action -- without even identifying the framework. (I read about the framework on the plane home from the workshop.)
Gathering people that are interested in and working on solutions together accelerates the collection, creation, and donation. I expect great things to come from this gathering.
Good testers can be hard to find. It looks like Actel Corp has found a good one. He is young. He is smart. He has excellent growth potential. And he works cheap -- for now.
I suspect that this kid does not know many testing buzzwords. I suspect he doesn't know much about testing tools and processes. However, Carson knows how to ask "why?" and communicate with engineers.
"We would ask what he liked and didn't like about it and he could explain it on a very high-end level." - Mark Nagel, Actel Corp, Field Applications Engineer
A tester that can think, ask questions, and communicate can go far.
Ever thought of a test that you haven't executed? Ever wonder if it might be valuable to add a use case, test case, or test charter? Ever feel like you spend more time talking about how to solve problems than it would take to try some of the proposed solutions?
I've been involved in discussions about software bugs that take more time than it would take to fix and retest. I understand that it is important to consider the risks associated with a bug and any proposed solutions. However, some times we just need to do it.
I've been in test planning meetings that take more time than it would take to execute the proposed tests. I've also wasted time performing unnecessary tests. The problem is that the usefulness of a test is usually not known until we have the results from that test.
When it makes sense, stop hypothesizing and start testing.
I dislike multiple choice tests. The distractors (the wrong answers) tend to be either so wrong that it is easy to find the correct answer or so close to the correct answer that they confuse test takers that know the material. Multiple choice tests make computer scoring of students possible but they are not capable of measuring students' understanding as well as short answer or essay tests. Tests that better measure learning are harder to grade. Tests that better measure learning require sapient human judgment -- not computer-scorable multiple choice tests.
Scripted software tests based on objective pass/fail criteria overly simplify testing. These tests may be easy to execute using automation or cheap labor. These tests may be easy to score. However. they are not able to provide the same value as sapient exploration of the system under test.
If only we could give software multiple-choice tests to measure quality ...
A couple nights ago, my son and I were in a store with a list of grocery items to purchase. My wife had given the list to our son and he was calling out a few items at a time. If he called out too many items, I'd forget them all and ask him to start over. When he called out something that I knew was nearby, we'd head in that direction and get that item. Darting back and forth may not be a very efficient way to shop, but it likely costs me less than systematically going through the store. When I go grocery shopping with an always-hungry 10 year old boy, it is very likely that we will buy twice as much stuff as is on our list.
Although he will claim that I always say "no", my son is skilled at talking his dad into buying things. In this trip to the store, he first asked for some expensive pastries knowing that I would say no -- like I always do. :) Then he proposed the cookies that I think he really wanted and gave me a well-thought-out argument as to why these were better and cheaper. The cookies found their way into our cart.
I wasn't very concerned about the accuracy of our selections because my wife was elsewhere in the store helping our daughter find some new gym pants for school. I knew that my selections would be tested before I paid for anything. Should anything not pass the inspection, I knew I could put it back and replace it with the correct item.
This trip to the store reminded me of a shopping trip a couple months back. My wife gave me a hand-written shopping list containing the text shown below.
DVD Lens Cleaner Toilet Paper - get extra canned dog & cat, kitty litter Tea Tree Oil Lined paper - college ruled Mech. pencil 0.7 hamburger meat, chicken tenders tenderloin cone shaped coffee filters white vinegar 64oz - same isle ^ canned tuna fish crackers - saltines + Ritz Doritos Fig Newtons Maple Syrup reg. coke + apple soda Dairy section coffee creamer, eggs, yogurt cheese variety juices from freezer section plus lemonade Grapes Pears - bartlett
As I worked my way though the list, I noticed that the level of detail specified for each item varied. Some things contained specifics while others did not. However, there is much implied in the list that is not written. How do I know that there is much implied? I've been married to Sophie for 16 years. I know my wife and she knows me. At least that's what I like to think.
She knows that she has to specify regular Coke because I'd buy Diet Coke if she didn't tell me otherwise. She also knows that I know that when she writes "apple soda", she is referring to a specific brand of apple flavored soda pop.
She did not specify what kind of toilet paper to buy but I have learned from experience which brands she will find to be acceptable. I had no idea how much was extra, so I ensured that we would not run out anytime soon.
She wrote "canned dog & cat", knowing that I would joke about not finding canned dog or cat meat in the store while understanding that I should get canned dog food and canned cat food.
She specifically told me what shape coffee filter to buy because she had recently bought a new coffee maker that requires a different shaped filter. Had she not told me which shape, I would have likely bought the wrong kind: the kind I had been buying for years. However, I discovered that there is more than one style and size of cone shaped coffee filter. I did my best to guess as to what size the new coffee maker used. I had not yet used the new coffee maker, but I had seen it. Then, before I left the store, I called home to confirm that I selected the right size.
From a conversation before I went to the store, I knew that the "canned tuna fish" was for the cats. I made an executive decision and bought tuna flavored cat food because it cost less. If I didn't know this, I might have thought that the tuna was to put on the crackers for our lunch.
I knew that "Fig Newtons" did not require that I buy that specific brand of fig bars. I also knew from experience that there were some brands I should not buy.
I am intelligent enough to know that the maple syrup is a separate item from the fig bars. However, I did not know if Sophie wanted real maple syrup or if maple flavored corn syrup would do. I bought both.
The list did not say anything about what kind of cheese I should buy but I knew what kind of cheeses we usually eat and made a guess.
I knew that "coffee creamer" came with many unwritten requirements that I have learned from shopping with my wife.
I knew what "variety juices" my family will drink and what they will not drink.
I knew that if I made a mistake in my selection of items, I might have to make another trip to the store. I was more careful in my selections than I am when Sophie is in the store. I double checked my list before I left the store.
I got most of the items right. The coffee filters fit the new coffee pot. I didn't get the vinegar right in spite of the more detailed directions about type, size, and location. And, I totally forgot the most important item on the list: the chocolate.
So, what do my shopping adventures have to do with software testing? Very much.
Written requirements are only part of the story
Explicit written requirements cannot fully communicate user needs
Requirements may be communicated in passing
Communication before, during, and after implementation is essential
Familiarity improves the success rate -- and makes requirements definition easier
Unit testing by implementers does not ensure success
User acceptance testing earlier in the process costs less than waiting until the end
Context frames everything
After nearly 16 years of marriage, I failed to completely meet expectations implementing something as simple as a shopping list.
And yet we expect people on software projects, that hardly know each other, to understand each other's desires.
One of my pet peeves about software is bad error messages. In my view, a bad error message is one that does not tell the user how the error impacts them and what they need to do in response to the error. Too many messages fail to communicate this information in terms that the software's user is going to understand. Too many of these messages are written for developers, not users.
There is a place for error logging in terms that help developers and testers troubleshoot and fix problems. This information is often best written to log files, not displayed in the user interface.
Pradeep Soundararajan and I recently discussed some of our experiences with error messages. You can listen to excerpts from this conversation using the link below.
After the above conversation, Pradeep walked me through an exercise that demonstrated a case in a popular office application in which I, the user, was not sufficiently informed that an error was occurring. It was obvious that the actions I was trying to perform were not functioning but the software gave me no clear indication of why it did not work as I expected.
Good testers recognize the need to include "negative testing" in their search for significant bugs. Testing for errors is a common part of functional testing. Let's go beyond functional testing of errors. Let's test errors for usability.
Here is an error testing mnemonic I created after our conversation.
Functional
Appropriate(or Apropos*)
Impact
Log
UI
Recovery
Emotions
Functional Does the error detection and reporting function as required? Are errors not detected that should be detected? Are errors reported? Do error dialogs function as expected? Do the buttons work?
Appropriate(or Apropos*) Are errors detected and reported in an accurate and timely manner for the intended audience? Are errors reported as soon as an error condition is met? Are warning messages displayed while there are enough resources to remedy the problem? Is a user allowed to waste time and effort only to be told that their work cannot be applied? Is the text of a message accurate? Does the text convey the situation to the intended audience? Is the error described in terms that will be understood by the intended audience?
Impact Is the impact of the error sufficiently communicated to the user? Does the message contain too little information for the user to understand what occurred and how it impacts what they were attempting to do? Does the message contain extra information that distracts from communicating the impact?
Log Does technical information need to be logged for support, system administrators, developers, or testers? Will this log information be available if the user waits to contact support? Are log messages standardized to allow for automated information mining? Are logs detailed enough to facilitate troubleshooting? Are errors logged that add no value? Is there too much logging? Does excessive logging negatively impact performance and disk space? Does excessive logging complicate error investigation?
UI Are users given some indication that what they attempted to do failed? Are user interface messages worded for the intended audience? Are user interface error messages consistent with the look and feel of the software? Are error messages consistent with other activity that causes the same error elsewhere in the application? Are errors communicated in an efficient manner? Does a user need to click away excessive dialogs? Is this error best communicated as an error dialog? Is this error best communicated as text added to the window? Is this error best communicated audibly?
Recovery Does the error message tell the user how to recover from the error condition? Does the software facilitate recovery? If needed, is contact information provided? Is the user prompted through the recovery or left to figure it out on their own?
Emotions What emotions are likely to be raised by the error message? Does the information in the message add to a user's frustration or help quiet it? If a user is being told that they need to pay more to use a feature, does the message encourage them to upgrade or does it encourage them to find a competitor's product? Does the error message cause more confusion?
Here's an error message Pradeep gave me to help test the mnemonic. What faults can you find in the error message using the mnemonic?
PS: While attempting to save this post, Blogger gave me the following error. This error fails many of the tests above. The second save button distracted me from the light grey error message. Trying again did not fix the problem. It was not clear if the error meant that my text had been saved or not. I finally had to copy the HTML of the post and paste it into a new Blogger session. Bad error reporting is easy to find. :)
* UDATE: 18 Aug 2007 @ 6:32 PM: Michael Bolton suggested that "Appropriate" would be more appropriate than "Apropos". After some consideration, I agree. Thanks Michael.
I have heard a variety of responses from developers in response to bugs I report. Some are good, some bad, and some are just plain ugly. Here are a few handfuls.
That's strange.
How'd you do that?
It works on my machine.
I already fixed that. You'll have it in the next build.
No user would do that.
That's not how you're supposed to do that.
It's a data problem. Tell the users to fix the data.
That's a cool bug! Show me again.
I didn't touch that code.
It works as designed.
It works as coded. [Well, duh. What else would it do?]
That's not a bug, it's a feature.
I can't test everything.
Thank you.
... and my absolute favorite (this came from a development manager)
Don't judge it, just test it.
The difference between the good and bad is often the relationship between developer and tester. Testers need developers to create something to test. Respect your developers. Communicate with respect and help turn the ugly responses into good responses.
As I've heard James Bach say: Testers don't create quality. Developers create quality.
I find it at work. I find it in online forums. I find it in books. I find it in papers. I find it in blogs. I find it at conferences.
I hear it from experts. I hear it from freshers. I hear it from friends. I hear it from managers. I sometimes even hear it come out of my own mouth.
It influences testers. It influences developers. It influences managers that influence testers and developers. It impacts customers.
It wastes time. It wastes money. It frustrates developers. It confuses executives. It demeans testers. It decreases quality in the name of improvement.
It permeates the practice of developing and testing software.
"Ten years ago, there's no way this would have worked. Now there are hardly any barriers."
- Anthony Page
Many of us spend most of our days trapped in a cubical or windowless office. At times I have enjoyed the opportunity to telecommute from home. I've had some good and bad home offices over the years. I've worked with great views and I've worked in basements. I'm a bit envious of James Bach's new digs.
I have the pleasure of working from home one day a week. I look forward to this day because I don't have to deal with traffic, I can work in the comfort of my own home, and I can get work done with fewer interruptions.
Earlier this week, I came across a CNN story about telecommuters that don't work from home. These telecommuters work from wherever they want to be. They are working globetrotters. Today's technology makes it possible for many people to work from anywhere in the world. I think we are still some time away from this being an option for many employees. However, it may be a viable option for contract work. If work can be outsourced to anywhere in the world, why not a beach or mountain top?
"People ask me where I live, and I'm not sure what to say, I'm not sure where I live. I live in the world." - Trygve Inda
Lee Copeland's CAST keynote address referenced in a previous post was not only about books. Good books was one of the items on Lee's list of eight recent innovations in software testing. Lee's complete list is shown below.
Innovations in Software Testing (Lee Copeland's List)
Context-Driven School
Testing Specialties
Test-First Development
Really Good Books
Open Source Tools
Session-Based Test Management
Testing Workshops
Certification
I was glad to see most of the items on this list. I am especially happy to see the Context-Driven School and Session-Based Test Management on the list. I believe that these have had a significant impact on software testing and have great potential that has not yet been realized.
Tester certification may be an innovation but I don't think its impact has been good. In my opinion, the current certification options are bad. (There was a certification debate hosted by AST at CAST this year. Please take a look at Tim Coulter's review: AST Certification Debate.) Most of the certifications show nothing more than one's ability to pass a certification exam. And, many of the certifications are based on context-free and outdated views and techniques. I reviewed some ISTQB sample questions with a group of very smart testers and we could not identify a correct answer for many of the questions. We could, however, make a good guess at what we thought was expected by the exam writers. Matters of opinion and guessing at implied contexts should not be the basis for any exam. I think the following statement summarizes this concern quite well.
We don’t mind that some people hold (and teach) views or techniques that we consider antiquated. We do mind that in prep courses that teach you how to pass “objective” exams, there is no place for presentation of controversy or thoughtful analysis of the fundamentals. - Cem Kaner and Tim Coulter
Now back to Innovations...
In typical CAST style, Lee asked the audience for things they thought he missed. The audience came up with the following additions. Yes, Model-Based Testing was mentioned twice -- followed by applause from Harry Robinson. :)
