Showing posts with label Bad Software. Show all posts
Showing posts with label Bad Software. Show all posts

May 10, 2008

Aggravation Testing

An example:



How Long Do I Have To Wait?



A few hours? I don't have hours. I am sitting in the car using borrowed WiFi from a campground. I had to seek out Internet access to use software that came on a CD. I finally find Internet access and now it says I may have to wait a several hours. Can I abort if it takes longer than I have? What happens if I lose my internet access while the firmware update is underway?

I'm already frustrated with this device. I'm already frustrated with the software. I was hoping that a firmware update might fix bugs and usability issues on the device itself. I have reached the tipping point. This thing is going back to the store.

August 18, 2007

Failure Usability

One of my pet peeves about software is bad error messages. In my view, a bad error message is one that does not tell the user how the error impacts them and what they need to do in response to the error. Too many messages fail to communicate this information in terms that the software's user is going to understand. Too many of these messages are written for developers, not users.

There is a place for error logging in terms that help developers and testers troubleshoot and fix problems. This information is often best written to log files, not displayed in the user interface.

Pradeep Soundararajan and I recently discussed some of our experiences with error messages. You can listen to excerpts from this conversation using the link below.



After the above conversation, Pradeep walked me through an exercise that demonstrated a case in a popular office application in which I, the user, was not sufficiently informed that an error was occurring. It was obvious that the actions I was trying to perform were not functioning but the software gave me no clear indication of why it did not work as I expected.

Good testers recognize the need to include "negative testing" in their search for significant bugs. Testing for errors is a common part of functional testing. Let's go beyond functional testing of errors. Let's test errors for usability.

Here is an error testing mnemonic I created after our conversation.

  • Functional
  • Appropriate (or Apropos*)
  • Impact
  • Log
  • UI
  • Recovery
  • Emotions

Functional
Does the error detection and reporting function as required? Are errors not detected that should be detected? Are errors reported? Do error dialogs function as expected? Do the buttons work?


Appropriate (or Apropos*)
Are errors detected and reported in an accurate and timely manner for the intended audience? Are errors reported as soon as an error condition is met? Are warning messages displayed while there are enough resources to remedy the problem? Is a user allowed to waste time and effort only to be told that their work cannot be applied? Is the text of a message accurate? Does the text convey the situation to the intended audience? Is the error described in terms that will be understood by the intended audience?

Impact
Is the impact of the error sufficiently communicated to the user? Does the message contain too little information for the user to understand what occurred and how it impacts what they were attempting to do? Does the message contain extra information that distracts from communicating the impact?

Log
Does technical information need to be logged for support, system administrators, developers, or testers? Will this log information be available if the user waits to contact support? Are log messages standardized to allow for automated information mining? Are logs detailed enough to facilitate troubleshooting? Are errors logged that add no value? Is there too much logging? Does excessive logging negatively impact performance and disk space? Does excessive logging complicate error investigation?

UI
Are users given some indication that what they attempted to do failed? Are user interface messages worded for the intended audience? Are user interface error messages consistent with the look and feel of the software? Are error messages consistent with other activity that causes the same error elsewhere in the application? Are errors communicated in an efficient manner? Does a user need to click away excessive dialogs? Is this error best communicated as an error dialog? Is this error best communicated as text added to the window? Is this error best communicated audibly?

Recovery
Does the error message tell the user how to recover from the error condition? Does the software facilitate recovery? If needed, is contact information provided? Is the user prompted through the recovery or left to figure it out on their own?

Emotions
What emotions are likely to be raised by the error message? Does the information in the message add to a user's frustration or help quiet it? If a user is being told that they need to pay more to use a feature, does the message encourage them to upgrade or does it encourage them to find a competitor's product? Does the error message cause more confusion?


Here's an error message Pradeep gave me to help test the mnemonic. What faults can you find in the error message using the mnemonic?





The next time you see an error message -- or don't see one that you should -- don't stop at functionality. Check the rest of the FAILURE. (And look here for a PowerPoint show demonstrating the mnemonic.)


PS: While attempting to save this post, Blogger gave me the following error. This error fails many of the tests above. The second save button distracted me from the light grey error message. Trying again did not fix the problem. It was not clear if the error meant that my text had been saved or not. I finally had to copy the HTML of the post and paste it into a new Blogger session. Bad error reporting is easy to find. :)


* UDATE: 18 Aug 2007 @ 6:32 PM: Michael Bolton suggested that "Appropriate" would be more appropriate than "Apropos". After some consideration, I agree. Thanks Michael.

June 27, 2007

Ugly Babies: Another Reason Why We Need Testers

... I got on the train ... And I noticed that the woman across from me in the aisle had her baby with her. Ugly baby. Ugly baby.

From the other end of the coach comes this guy and he was very drunk and he was staring at the baby. ... And the guy said, "I'm looking at that ugly baby. That's a horrible looking baby lady. Where'd you get that baby from?"

And the woman said, "I don't have to take that!" And she snatched the emergency cord and the train came to a screeching halt, and the conductor came running in.

Now this was his moment. At this moment he represented the Pennsylvania Railroad. And he said, "what's going on here?"

And the woman said, "This man just insulted me. I don't have to spend my money and ride this railroad and be insulted. I'd rather walk."

And the conductor said, "calm down! Calm down! Madame there's nothing, nothing that the Pennsylvania Railroad will not do to avoid having situations such as this. Perhaps it would be more to your convenience if we were to ... let you sit somewhere else in the coach. And as a small compensation from the railroad ... we are going to give you a free meal; and maybe we'll find a banana for your monkey."

- Flip Wilson



We are like parents when it comes to our creations: we can be blind to ugly.

I am a believer in developer testing. Good unit testing is essential to good software. Practices like Test Driven Development (TDD) help ensure that software works from the start. Anyone that's been in the software development business for long knows that the cost of defects exponentially grows as a project progresses. Preventing and finding defects early should be a goal in every software project. Early testing by developers does not remove the need for testing by someone other than the coder.

There are a number of reasons for "independent" testing. These reasons often include references to requirements testing, end-to-end testing, integration testing, system testing, and independent verification. These reasons focus on testing things as complete systems or from a user's viewpoint. I have another reason: Creators can be blind to ugly.

Creators have an emotional attachment to their creation. This attachment can get in the way of objective evaluation. It is difficult to be both creator and critic. It is as if creativity and critique are opposite sides of a scale. We can be both, but increasing one can harm the other. Creators need to balance the two without allowing self-criticism to hamper creativity.

I think its good for creators to have an emotional attachment to the product of their thoughts and labor. External critics can help a creator improve their creation without stifling creative thinking.

Creating and critiquing are separate skills. When one group can focus on one and another group can focus on the other: together they are stronger (if the critics are more tactful than the drunk in Flip's story).

At least that's my opinion. Now I open the floor to the critics. :)

June 21, 2007

Marching Through Hell

There is a Churchill quote about when you're marching through hell, just keep marching. Well, that's basically what we're doing. We need to slow this down and get it right.
- Charles Burbridge
CFO, Los Angeles Unified School District

A couple weeks ago, the Los Angeles Unified School District's new computer system incorrectly paid over 32,000 employees. That's almost a third of the district's employees. And the trouble started this past January.


I've been involved in some software projects with a "hell" phase in the SDLC. I've worked on projects that failed. I've been involved with projects that went millions over budget. However, I'm thankful that I have never worked on a project for which tens of thousands of production issues were reported in a handful of months.

Through the replacement of our aging financial, human resources, payroll and
procurement systems, the District will:
  1. Dramatically improve service delivery to schools
  2. Radically improve the efficiency of District operations and our ability to manage them
  3. Reduce/eliminate paperwork and redundant manual processes
  4. Increase accountability and transparency to the public in the use of public funds
  5. Provide better data for decision makers and stakeholders at all levels
- LAUSD ERP Project Vision Statements, March 2004

This enterprise software development project started out with good intentions and lofty goals. However, it appears that something went terribly wrong in implementation. The district's public web site does not include much information about the problems. The latest document I saw with information for employees with payroll problems has a February date. I also find it interesting that a more recent document is missing item #4 from the above list. I guess that when things go wrong, there's no need to keep the public and employees informed. Right? Wrong. I don't know the details of what went wrong, but I do know that when things go wrong people tend to communicate less. I also know that it can be difficult to admit failure. This project reminds me of the six stages of a project that I occasionally encounter in forwarded emails and on bulletin boards. It will be interesting to see what and who is eventually blamed.

The six stages of a project
  1. Enthusiasm
  2. Disillusionment
  3. Panic
  4. Search for the guilty
  5. Punishment of the innocent
  6. Praise and honor for the non participants

Enterprise software systems can be very complex. The vision statement says that they chose a commercial package and partnered with an integrator because they want to configure software and not customize software. A payroll system for over 100,000 school district employees has got to be one of the most complex "configuration" projects around. I believe there were good intentions, but this looks like a game of semantics. Enterprise systems are difficult to configure and deploy -- especially when you have numerous customized needs.

I don't know what kind of testing was performed on this system before it went to production. The original visions document references the need for testing support.
A complete test environment (development, QA, and production instances), including adequate infrastructure, staff (DBA), storage and network capacity, a full volume of test data.
- LAUSD ERP Project Vision Statements, March 2004

I wonder what happened in implementation. Obviously, something was missed or skipped.

I did notice that the project appears to be on schedule. When Churchill said "If you're going through hell, keep going.", the point was to find your way out of the situation -- not ignore it.

I was once offered a "great opportunity" to work on a "high profile" project. I accepted that offer. I learned a great deal from that experience. One of the things I learned is that it may not be a good thing to be visible when there are problems that are out of your control. Yet we have to take risks to gain a reward.

Right now, I am thankful that I am not "marching through hell". And I'm thankful that my paychecks come as expected.

To those in the LAUSD, these problems will pass. Right?
We expect these problems to subside in the very near future.
- Charles Burbridge
CFO, Los Angeles Unified School District
How near is very near?

If you have any more information about the development and testing practices on this project, I'd like to see it.

April 19, 2007

Taxing my badometer

I've used great software. I've used some horrible software. I've used lots of software with annoying bugs. I regularly spend a great deal of time working around bugs in software that I use on a daily basis. However, I think I have recently found the worst yet. It is not the worst because it fails to work. It is not the worst because it is missing features I want. It is the worst because it appears to work on the surface yet generates bad data that has a direct impact on my stress level, my finances, and my interaction with the IRS.

Dealing with the IRS -- or even thinking about it -- is already stressful enough. The tax code is complicated enough that it is easy to accidentally report the wrong income or deductions. If it were easy, we'd see accountants lining up at soup kitchens instead of raking in the cash helping us common folks with our tax returns.

For most of the past 10 years, I have elected to use commercial tax preparation software to help me complete and file my tax returns. I've used some tax software that was good, and some that was bad. I've even tried some freebie and online options. The software I used this year is nothing short of a software atrocity.

This software was swarming with bugs -- although most were not obvious. I suspect that most users didn't even notice.

After installing the software, I dutifully entered more financial data than I care to track. I tried a couple what-if scenarios in the process: such as a last minute IRA contribution. I also entered some estimated values before entering the real data so that I could determine if I should take one option or another. I also entered some data for deductions that I later removed after learning (due to reading the IRS publications) that they not apply to my situation.

I did not follow the interview process from start to finish. I went back and forth in this process entering data, revising data once I clarified the rules or gathered up all my evidence for income and deductions. After several hours, I thought I had a complete return. I had entered and verified all my data in the software's "interview" interface.

Then the automated error check failed. It would not let me continue until I fixed data in a form. This was a form that I had accidentally selected somewhere in the process that contained no data. I then tried to delete the form by unchecking a box that made the options that went the form available. The software then informed me that I could not delete the form there. I went to the portion of the interview where it told me to delete the form and there was no delete option. I then discovered that I could delete the form from the program's "forms" list. However, I later found out that going back to any point in the interview recreated the form. There was no option in the software to permanently get rid of my accidental selection.

I discovered that there was no way to print my return for review until I got to the last step of the eFile process. This required that I go back in the process and tell it that I wanted to file on paper. I printed the return and was surprised to see that the data on the return did not match the current data in the interview interface. Some values that I had entered and then deleted or changed were in the printed return. My return was inaccurate. I was glad that I went through the tedious process of getting to the print feature. If I had eFiled without first printing, I would have submitted a return that did not match the data I entered -- and the data shown to me on the screen.

I then spent several hours deleting and reentering data until the printed forms finally matched the data I entered.

In the midst of my review, I noticed that the software had forced me to take an option that resulted in a lower federal tax bill. However, selecting this option instead of another equally valid option significantly increased my state tax bill. The software forced me to have an overall higher tax bill because it only considered the federal return when deciding which option was best for me. After about an hour, I realized that I could remove a deduction in order to force the software to take the alternate option.

Once everything was in order and the displayed and printed data matched, I selected the eFile option to submit my return. I entered bank account information to pay my taxes online. I then discovered that the eFile service I paid for did not allow me to submit electronic payment to the state. My returns were submitted. My federal payment was sent electronically. However, my state payment still had to be mailed. I could have filed my state taxes and paid online using the state's web site as I had done in previous years. Instead I thought it would be easier to use the integrated system. I wish that the software would have told me that I could not electronically submit my payment before it took my money for the eFile service.

And to top it all off, the software did not tell me how to send in my payment by mail. I spent nearly an hour going through the system's help and the vendor's web site seeking guidance. I finally found an option for an online chat with customer service. It then took customer service a half hour to figure out how and where I should send my payment.

I asked if I could have a refund due to the buggy software and the eFile service that was not as good as the state's free eFile service. I was directed to call them by phone to get a refund because the could not give refunds via the online chat. I called the number and waited on hold of over a half hour before I was disconnected. I called back and got a recording stating that the customer support office was closed.

The bad software was topped off by bad customer service. I was angry. My badometer was pegged.

I then did a little exploratory testing in the application. I discovered additional bugs that created incorrect tax returns. I noticed that the software said it would calculate things for me and then ask me for the value without giving any assistance. I found places where the instructions in the application did not match the IRS publications. I even found a way that I could get it to calculate a refund of any value on the state return without changing the income or deductions.

I understand that tax software companies have a very small window in which to do their development and testing. I am amazed that this software was released. The poor quality -- and the agravation it caused -- have ensured that I will not be a returning customer next year.

I suspect that the software passed all the test cases created by its makers. I suspect that it even passed multiple automated tests. This just goes to show that passing all the preconceived scripted tests does not make a quality product. I do not know what methods this company uses for testing their software, but I suspect that it is primarily scripted testing.

I believe that exploratory testing (and model-based automated testing) would have been more likely to find the bugs I encountered than scripted testing.

As we test software, we need to consider the infinite possibility of data and work flow variations. We can't test all the variations, but we can vary what we test. It is easy for individual testers to select similar test options when they think they are selecting data randomly. Seek out variety. Use random data generation tools to provide more variety.

How would you test tax software? How would you ensure that bugs like the ones I encountered don't aggravate customers?