Showing posts with label Usability. Show all posts
Showing posts with label Usability. Show all posts

July 26, 2008

Pause at the Pump

  • My fuel gauge is on empty.
  • I don't want to stop, but pull into the gas station.
  • I tell the children to stay in the car.
  • I get out of the car.
  • The children are asking me questions from inside the car that I can't hear well enough to understand.
  • I swipe my credit card in the pump's card reader.

  • The pump responds by prompting me to "SELECT WINDOW OR OUTSIDE".
  • The children are still talking to me. I still don't understand.
  • I pause and stare at the keypad.





Is there a problem here?

  • I pause to think for a moment.
  • I hear the children asking me questions that I still don't understand through the closed car windows.
  • I scan the keypad again.
  • I cant find the "OUTSIDE" button.


Recognizing Bugs

At CAST last week, Pradeep Soundararajan gave a Lighting Talk about the importance of testers being able to recognize a bug. Tests may be of little use if the tester doesn't recognize the bugs triggered by the test.

Sometimes bugs are obvious. Sometimes bugs are not clearly violations of requirements documents. This is especially true when it comes to human computer interaction problems.

Requirements are not always clear and objective.

So, do you recognize why I may have paused when I read the prompt on the gasoline pump? It wasn't the price of the $4 per gallon fuel. It wasn't because I had to think about how I wanted to pay.

I paused because I didn't see a button labeled "OUTSIDE". Plus, I already swiped my credit card indicating that I wanted to pay at the pump. And even if I were to pay at the cashier window, I would still be outside.

I wonder how many minutes are wasted each month prompting customers to select where they want to pay after they have swiped their credit card. I wonder how many other people pause and read twice in search of the button to indicate that they want to pay outside.

The developers and testers of the software in this pump may have not recognized this problem. Maybe the makers executed test scripts -- either automated or manual -- and were blind to the problem. Or maybe they didn't deem it important enough to change.

Sometimes familiarity with the technical details of a system can hide problems that are obvious to those that don't know the technology, the requirements documents, and the test scripts. As testers it is important that we be careful not to let our familiarity with a system make us blind to to bugs -- things that bug our users.

Will you recognize a problem if you see it?

May 10, 2008

Aggravation Testing

An example:



How Long Do I Have To Wait?



A few hours? I don't have hours. I am sitting in the car using borrowed WiFi from a campground. I had to seek out Internet access to use software that came on a CD. I finally find Internet access and now it says I may have to wait a several hours. Can I abort if it takes longer than I have? What happens if I lose my internet access while the firmware update is underway?

I'm already frustrated with this device. I'm already frustrated with the software. I was hoping that a firmware update might fix bugs and usability issues on the device itself. I have reached the tipping point. This thing is going back to the store.

August 18, 2007

Failure Usability

One of my pet peeves about software is bad error messages. In my view, a bad error message is one that does not tell the user how the error impacts them and what they need to do in response to the error. Too many messages fail to communicate this information in terms that the software's user is going to understand. Too many of these messages are written for developers, not users.

There is a place for error logging in terms that help developers and testers troubleshoot and fix problems. This information is often best written to log files, not displayed in the user interface.

Pradeep Soundararajan and I recently discussed some of our experiences with error messages. You can listen to excerpts from this conversation using the link below.



After the above conversation, Pradeep walked me through an exercise that demonstrated a case in a popular office application in which I, the user, was not sufficiently informed that an error was occurring. It was obvious that the actions I was trying to perform were not functioning but the software gave me no clear indication of why it did not work as I expected.

Good testers recognize the need to include "negative testing" in their search for significant bugs. Testing for errors is a common part of functional testing. Let's go beyond functional testing of errors. Let's test errors for usability.

Here is an error testing mnemonic I created after our conversation.

  • Functional
  • Appropriate (or Apropos*)
  • Impact
  • Log
  • UI
  • Recovery
  • Emotions

Functional
Does the error detection and reporting function as required? Are errors not detected that should be detected? Are errors reported? Do error dialogs function as expected? Do the buttons work?


Appropriate (or Apropos*)
Are errors detected and reported in an accurate and timely manner for the intended audience? Are errors reported as soon as an error condition is met? Are warning messages displayed while there are enough resources to remedy the problem? Is a user allowed to waste time and effort only to be told that their work cannot be applied? Is the text of a message accurate? Does the text convey the situation to the intended audience? Is the error described in terms that will be understood by the intended audience?

Impact
Is the impact of the error sufficiently communicated to the user? Does the message contain too little information for the user to understand what occurred and how it impacts what they were attempting to do? Does the message contain extra information that distracts from communicating the impact?

Log
Does technical information need to be logged for support, system administrators, developers, or testers? Will this log information be available if the user waits to contact support? Are log messages standardized to allow for automated information mining? Are logs detailed enough to facilitate troubleshooting? Are errors logged that add no value? Is there too much logging? Does excessive logging negatively impact performance and disk space? Does excessive logging complicate error investigation?

UI
Are users given some indication that what they attempted to do failed? Are user interface messages worded for the intended audience? Are user interface error messages consistent with the look and feel of the software? Are error messages consistent with other activity that causes the same error elsewhere in the application? Are errors communicated in an efficient manner? Does a user need to click away excessive dialogs? Is this error best communicated as an error dialog? Is this error best communicated as text added to the window? Is this error best communicated audibly?

Recovery
Does the error message tell the user how to recover from the error condition? Does the software facilitate recovery? If needed, is contact information provided? Is the user prompted through the recovery or left to figure it out on their own?

Emotions
What emotions are likely to be raised by the error message? Does the information in the message add to a user's frustration or help quiet it? If a user is being told that they need to pay more to use a feature, does the message encourage them to upgrade or does it encourage them to find a competitor's product? Does the error message cause more confusion?


Here's an error message Pradeep gave me to help test the mnemonic. What faults can you find in the error message using the mnemonic?





The next time you see an error message -- or don't see one that you should -- don't stop at functionality. Check the rest of the FAILURE. (And look here for a PowerPoint show demonstrating the mnemonic.)


PS: While attempting to save this post, Blogger gave me the following error. This error fails many of the tests above. The second save button distracted me from the light grey error message. Trying again did not fix the problem. It was not clear if the error meant that my text had been saved or not. I finally had to copy the HTML of the post and paste it into a new Blogger session. Bad error reporting is easy to find. :)


* UDATE: 18 Aug 2007 @ 6:32 PM: Michael Bolton suggested that "Appropriate" would be more appropriate than "Apropos". After some consideration, I agree. Thanks Michael.

June 8, 2007

Bad Messages

I regularly spend a great deal of time tracking down the root cause of software errors -- both on and off the job. Much of the investigation effort could have been avoided if I were not presented with incomplete or incorrect error messages. The text of error messages appears to be commonly overlooked by software developers and testers.

One of my early test automation development tasks was to fix the error messages in a test tool. The tool was used to validate the structure of data exchanged between computer systems. In nearly all cases, this tool displayed an error dialog window stating that the data was not valid. This tool gave the user (a tester) no information about what was wrong with the data. The lack of detail in these messages required that the user manually examine the data -- bit by bit -- with a protocol analyzer. This turned investigation of errors reported by an automation tool into a tedious manual task. The tool encountered a problem but did not report the source of the problem. I reviewed the code and added more detail to the error messages in this test tool. I also added logging of the data exchange in a format that made sense to human beings that could be reviewed without a protocol analyzer. The improved error messages and human-readable logging greatly decreased the troubleshooting and investigation time.

I am amazed at the poor error messages in test automation tools. Test automation tools and frameworks are likely to encounter errors in the software under test. I believe that this requires better error reporting and handling than many other tools. I am also disappointed in how difficult some of the automation tools make it to create scripts that run unattended.

One tool I use regularly often stops executing tests and displays a message like the following.


It took a great deal of effort to figure out how to coax the tool into returning error codes that could be handled in code instead of prompting a user during an automated test execution. The tool vendor's support personnel didn't seem to understand why it was a problem that I could not code their tool to handle an error without human intervention at run time. They kept telling me to fix the application under test to make the test tool's error dialog go away.

I also recently uninstalled a different test tool that had a horrible uninstall interface. It displayed a window with an uninstall progress bar that didn't move. After several minutes, the application displayed an "Install Complete" window underneath the progress window. The progress bar began to slowly move only after I clicked the "Finish" button on the buried completion window. Then it prompted me with something like the following.


Do I want to erase all files? What's the context? Of course, I don't want to erase all my files. I wonder how much the maker of this software spends on support calls about this message.

I used to receive numerous inquires from new computer users about the Windows "program has performed an illegal operation" error message. Many users thought this meant that they did something illegal and the police would soon be knocking on their door. People familiar with computers understand that this error message is about the program's interaction with the operating system and hardware; however this error message is misleading. This error message is designed for developers, not users. Good error messages tell the user what they need to know about the problem and what to do about it. Good error messages explain the problem but do not overwhelm the user with information that is useless to the intended user. This means that different kinds of applications require different kinds of error messages.

If you write software, please provide your users with accurate messages tailored to the user -- not developers. If you create test tools or automation frameworks, provide testers with information that is useful in determining what happened. Please.

People who write framework software should spend more time on useful error messages that show people why the error occurred and give a clue as to how to fix it.
- Eric M. Burke,


Want to practice creating better error messages? Try the Error Message Generator.