Retraining the unskilled to code software

I stumbled across a 46 year old newspaper article about how automation is changing business. The following statement caught my attention.

"Unskilled workers can then be retrained to handle peripheral jobs in the EDP system such as coding, card punching and so on."
Computor Invasion Scares The Unskilled,
WINNIPEG FREE PRESS, 1962

Times sure have changed. If retraining unskilled workers to code software was a viable option, then they must have been coding some pretty simple software. Maybe "coding" really refers to the process of entering code designed by someone else. Maybe "coding" meant data entry.

Regardless of what "coding" meant in 1962, computers and the software we create today are more complex than they were 46 years ago. It is too bad that some who would never think of coding as unskilled work still seem to think of testing as work for the not-so-well-skilled. Good developers and testers are good thinkers.

"You've got to be smarter to run a company with a computor than without one. The information on which you will base your decisions comes at you faster. If you're going to take advantage of this, you have to think faster, more decisively and more clearly."
- George Aitken, Vice-President and Comptroller,
Great West Life Assurance Co, 1962
Computor Invasion Scares The Unskilled

Evidence That Quality Has Everything To Do With Value

“Quality is value to some person.”

Gerald Weinberg,
Quality Software Management – Systems Thinking

This morning, Jason Gorman's blog post title Proof That Value Has Little To Do With Quality? caught my attention. This title contradicts my definition of Quality. To me, Quality is all about value to stakeholders.

Quality is not about implementing the best development practices. Quality is not about writing solid code. Quality may not be about impressive features. Quality may have no relation to elegance. Quality may not even be reliable. Quality may be cheap or it may be expensive. Quality may be well planned or it may be haphazard.

Quality is all about value. Quality is about value to people that matter.

Jason references an interesting article about a web site that started as a learning exercise and "seems to come from the Anti-Perfectionist School of Design", yet is profiting its creator millions of dollars annually. In spite of many flaws, this web site is profitable because users find it valuable and users bring advertising dollars. I consider this to be a high quality web site in spite of its obvious flaws because it has value to people who matter. Instead of viewing this as an example of value having little to do with quality, I see this as a great example of Quality having everything to do with value. I suspect that Jason and I define Quality differently. This story is an example of how value sometimes has very little to do with all the other things we often call Quality.

Perhaps my thinking is too touchy-feely for those who think we need to measure and assure Quality through quantitative metrics and processes enforcement. By it's very nature, quality is subjective. Sometimes we can quantify the results of Quality: as in the $10 million in annual advertising profits. I suspect that some of you are subjectively estimating how better metrics and process might lead to better profits.

The real measure of Quality is a measure of value (not necessarily quantitative value) to those who matter.

"As professionals, we have no real control over the ultimate value of the software we create. And neither do our customers, or requirements analysts, or product owners, or whoever it is who's been charged with figuring out what the best use of the budget would be. It's all guesswork, like choosing lottery numbers or selecting which horse to bet on."
- Jason Gorman,
Proof That Value Has Little To Do With Quality?

While there is guesswork in determining who matters and what they value, it is not as random as selecting lottery numbers. Better understanding of who matters and what they value can help us reduce the guesswork. Ongoing dialog can bring better understanding. If Quality is nothing more than a lottery, then we might as well limit ourselves to BUFD and scripted manual testing.

Interactions, collaboration, and responding to our changing understandings can help us take control over Quality.

"Above all, listen to what your customers are telling you about Quality. ... Your customers are in a perfect position to tell you about Quality, because that's all they're really buying. They're not buying a product. They're buying your assurances that their expectations for that product will be met. ... Your customers may not have all the hard business facts. They may not be aware of your specs and your standards and your inspection reports ... They may not be able to give you a precise definition of Quality, but one thing's for certain -- they know it when they see it."

-John Guaspari,
I Know It When I See It: A Modern Fable About Quality

And, as Jason rightly points out, what satisfies users (and the business) today may not satisfy them tomorrow. Keep the dialog going.

Whose side are you on?

Many marriages would be better if the husband and the wife clearly understood that they are on the same side.
- Zig Ziglar

Many software teams would be better if the testers and developers clearly understood that they are on the same side.

No one creates a team to battle their team members. Husbands and wives don't embark on life together so that they can spend their lives battling each other. Software development teams aren't formed to create conflict within a company. Yet, conflict happens. Partners become the victims of friendly fire.

When the going gets tough, the tough need to stop bickering long enough to realize that they are on the same team. Here are a few tips that can help keep a team together.

• Keep the common mission in sight
• Seek first to understand, then to be understood (Stephen Covey)
• Help others overcome obstacles instead of assigning blame
• Praise good work
• Be honest and tactful
  

Although each team member may serve a different role, it is essential that teams value the input of each member and work together to accomplish the common mission. Testers and developers should not be enemies.

Problems: So What's On All Those Sticky Notes?

In my previous post about the Agile Alliance Functional Testing Tools Workshop , I wrote the following:

After reviewing existing tools used by agile teams: we identified software testing issues that have been solved (yellow), those that have been partially solved (orange), and those that have not been solved (pink). As I recollect, most of the solved issues were technical problems and most of the unsolved problems were people problems. Many of the partially solved problems were those for which I believe we have technical solutions but have not yet been integrated and presented in ways that best support people.

In case you are wondering what problems we wrote down on these notes, Frank Maurer kindly transcribed them for the workshop participants and I have posted them below.

Looking at this list reminds me of the traffic safety problem lists I made in the defensive driving classes I used to teach. As unsafe driving practices were brought up by students, I would add them to a list on the whiteboard. I then asked the students to identify whether each item on my list was primarily due to driver skill or driver attitude. The students usually blamed most of the problems on driver attitude.

Skill and attitude play important roles in software development and testing. Team members need both. Brian Marick addresses this in his guiding values of discipline, skill, ease, and joy. Instead of looking at the functional testing problems as skill or attitude problems, I looked at them as man or machine problems. I asked myself if each problem appeared to be mainly a human problem or a technical problem.

Software development and testing involves a mix of people and technical problems. Interfacing people with people and people with technology is often harder than interfacing technology with technology. Most of the identified problems have both technical and human aspects to the problem and possible solutions. Some are due to the nature of people or the nature of software and will likely never be completely solved.

I find that identifying whether a problem is primarily a people or technology problem helps me identify possible solutions. I quickly scanned this list and identified whether I thought things were primarily human or technical issues. My notes are included to the right of each item. I don't necessarily interpret each item as its author (a human communication problem), and I do not necessarily agree that each item is a problem or belongs in the specified group.

As you review this list, ask yourself if the problem and possible solutions are grounded in people or technology.

Unsolved	Define Test	Human
Unsolved	Organizing large sets of Tests/Expect. Actions/Examples for a large, complex system so you can wrap your head around the whole thing.	Human
Unsolved	Having tests survive handoffs. Project team -> op support -> proj team.	Human and technical
Unsolved	Getting people to care	Human
Unsolved	Transferability of ubiquitous language to other projects	Human
Unsolved	Write SW that is understood	Human
Unsolved	Reducing Uncertainty	Human
Unsolved	Limitations of natural language	Human
Unsolved	How would we act if we really believed code was an asset?	Mostly Human
Unsolved	Allowing Customers to articulate their expectations in a format/tool/way that is comfortable for them	Mostly Human
Unsolved	Multi-model specification text + table + graphic in one test.	Mostly Technical
Unsolved	Domain experts	Human
Unsolved	Fully Automated regr. That does not reduce dev.velocity.	Mostly Technical
Unsolved	Generate a domain model from tests.	Human and technical
Unsolved	Testing usability as part of acceptance testing in incremental development.	Human
Unsolved	Common language to express GUI based tests.	Mostly Human
Unsolved	Conveying "experts" perspective to majority of development team.	Human
Unsolved	Automated Software Development	Technical (Machines aren't creative)
Unsolved	Functional tests that can be easily re-used later in lifecycle.	Mostly Technical
Unsolved	Test business requirements independent of current interaction/Api design.	Mostly Human
Unsolved	Composing tests into useful larger tests	Technical and Human
Unsolved	Test first performance	Human
Unsolved	Getting BA to write the tests	Mostly Human
Unsolved	Having customer to be able to write test and enjoy it.	Mostly Human
Unsolved	Different test notations for different user groups.	Human problem, technical solution?
Unsolved	Acceptance/Functional Tests good for communication and automation.	Human problem, technical solution?
Unsolved	Model the time domain " and 3 months later an email.	Mostly Technical? (Don't want execution to take 3 months.)
Unsolved	Change touch-point dynamically.	?
Unsolved	Terminology (Test or not a Test?)	Human
Partially Solved	Understand what has not been tested.	Human and Technical
Partially Solved	Trace tests into project management tools	Human and Technical
Partially Solved	Getting buy-in for need to automate.(docs,tests,specs)	Human and Technical
Partially Solved	Accurately & completely communicating requirements.	Human problem, partial technical solutions
Partially Solved	Satisfying every role's need/desire to be at center, in control.	Human
Partially Solved	Having functional tests specify requirement specifications.	Human and Technical
Partially Solved	Sustain a productive conversation with all stake holders.	Human problem, partial technical solution
Partially Solved	How do we get across what the project would feel like if things were going well.	Human
Partially Solved	Write robust (U.I) tests that are not brittle.	Mostly Technical
Partially Solved	Fragile tests.	Mostly Technical
Partially Solved	Valuing individuals and interactions over processes and tools.	Human
Partially Solved	Describing customer intent.	Human
Partially Solved	Executable (as tests) Models (as specifications)	Human and technical
Partially Solved	Test partitioning	Human and technical
Partially Solved	Tests as support artifacts.	Human and technical
Partially Solved	Test Generation Automation.	Human and technical
Partially Solved	Running tests parallely ( Fast feedback)	Mostly technical
Partially Solved	Reconciling preferred style of abstraction.	Human and technical
Partially Solved	Dealing with size.	Human
Partially Solved	Finding the right words in which to write a test.	Human
Partially Solved	Express requirement in the domain language, graphical, word based , table based.	Mostly Technical
Partially Solved	Functional Test driven development..not just for agilists. How to sell to waterfallists?	Mostly Human
Partially Solved	How to Integrate tools?	Mostly Technical
Partially Solved	Common test case format.	Human and Technical
Partially Solved	Cooperation and collaboration between tool developers (tool stack, tool platform).	Human and Technical
Partially Solved	Change from one notation to another. (graphic -> tabular)	Mostly technical
Partially Solved	Test/Example -> model (generated model based tests)	Mostly technical
Partially Solved	IDE for testers and BAs	Human and technical
Partially Solved	Get all roles actively involved	Mostly Human
Partially Solved	View specs/examples/tests, differently for different roles.	Mostly Technical
Partially Solved	Different editors for different roles?	Mostly Technical
Partially Solved	Super-power IDE	Technical solution to human problems?
Partially Solved	Test Refactoring	Human and technical
Partially Solved	Refactoring tests	Human and technical
Partially Solved	Prioritize and execute tests to get faster feedback.	Mostly Human
Partially Solved	Describe a test at an appropriate level of abstraction	Mostly Technical
Partially Solved	Choosing what to automate (when you can't automate everything)	Mostly Technical
Partially Solved	Tools to support exploratory testing	Human and Technical
Partially Solved	Reusable test artifacts (poor modularity cohesion)	Mostly Technical
Partially Solved	Build community with BAs	Mostly Human
Partially Solved	Allow for refactoring from /to code <-> tests <-> req'ts	Mostly Technical
Partially Solved	Setup a wiki to discuss smaller problem solving.	People
Partially Solved	Capture war stories + testimonials + experiences.	Human, partial technical solution
Partially Solved	Test Maintenance	Human and Technical
Partially Solved	Book: Patterns of Self testing software	?
Partially Solved	Shared vocabulary around parts of a functional testing solution ("Fixture",etc)	Human
Partially Solved	Ensure adequate test coverage.	Mostly Technical
Partially Solved	Communication of what has been tested	Human and technical
Partially Solved	Communication using the ubiquitous language.	Human
Solved	Express automated/able tests in tables	Mostly Technical
Solved	Correctly implementing programmer intent	Mostly Technical
Solved	Deliver SW to test that doesn’t crash immediately	Mostly Technical
Solved	Provide traceability between Story or Requirement and accpetance/functional test	Mostly Technical
Solved	Gui Testing - functional testing is more than GUI testing	Technical
Solved	Data-Driven Testing	Technical
Solved	Driving Apps	Technical
Solved	Integrating test executors/drivers with build process	Technical
Solved	Edit Fit tests from eclipse	Technical
Solved	Report Results	Technical to report, human to be understood
Solved	Express expectations in code	Human and technical
Solved	Unit testing	Mostly Technical

I think we can solve the technical issues and use technical solutions to help people manage some of the people problems. Applying discipline and skill to solve the technical problems may help add to testers' ease and joy.

Where do you think the solutions lie? Have a solution? Please share it.

Better Tools for Individuals through Collaboration

Individuals and
interactions

over
processes and tools

- from the Agile Manifesto

I spent the second half of last week at the Agile Alliance Functional Testing Tools Visioning Workshop. (How's that for a long name?) Before the workshop, I was thinking that it seemed a little oxymoronic to have an agile workshop with a focus on tools. Perhaps my thinking was triggered by my concerns about those who seem to value "agile" processes and tools (often ones they sell) more than people.

Agile people are supposed to care about people and not care about tools. Right? Wrong.

while there is value in the items on
the right, we value the items on the left more

- from the Agile Manifesto

Software is developed by people for people. Agility involves building better software by adapting to the needs of people instead of letting processes and tools lead the way. Process and tools do best when they have a supportive role in software development. Better tools can support agility but they cannot make anyone agile.

The tool-centric discussions at the workshop were driven by a desire to build better software for people that build and test software. It is about people.

It then seems quite apropos that the book I indiscriminately grabbed off the shelf (well, I picked it for its size more than its content) to read on the airplane to and from the workshop is Ben Shneiderman's Leonardo's Laptop: Human Needs and the New Computing Technologies. The first chapter contains the following paragraphs that affirm my thinking about the role of automation in software testing. (Emphasis is mine.)

The first transformation from the old to the new computing is the shift in what users value. Users of the old computing proudly talked about their gigabytes and megahertz, but users of the new computing brag about how many e-mails they sent, how many bids they made in online auctions, and how many discussion groups they posted to. The old computing was about mastering technology; the new computing is about supporting human relationships. The old computing was about formulating query commands for databases; the new computing is about participating in knowledge communities. ...

The second transformation to the new computing is the shift from machine-centered automation to user-centered services and tools. Instead of the machine doing the job, the goal is to enable you to do a better job. Automated medical diagnosis programs that do what doctors do have faded as a strong research topic; however, rapid access to extensive medical lab tests plus patient records for physicians are expected, and online medical support groups for patients are thriving. ... Natural language dialogs with computerized therapists have nearly vanished, but search engines that enable users to specify their information needs are flourishing. The next generation of computers will bring even more powerful tools to enable you to be more creative and then disseminate your work online. This Copernican shift is bringing concerns about users from the periphery to the center. The emerging focus is on what users want to do in their lives.

- Ben Shneiderman, Leonardo's Laptop

Although many think of us software testers and developers as eccentric nerds, software developers and testers are human too. Like other humans, we desire tools that help us do a better job. This was the theme of the workshop: envisioning ways that tools can help us do a better job testing software.

After reviewing existing tools used by agile teams: we identified software testing issues that have been solved (yellow), those that have been partially solved (orange), and those that have not been solved (pink). As I recollect, most of the solved issues were technical problems and most of the unsolved problems were people problems. Many of the partially solved problems were those for which I believe we have technical solutions but have not yet been integrated and presented in ways that best support people. Much of the "what's next" discussion at the workshop was focused on how to integrate existing tools that each partially solve problems but together could move problems to the solved group.

Once the technical problems are solved, we can work on the tools to help with the people problems: we can move from old computing to new computing.

In Leonardo's Laptop, Ben Shneiderman presents a framework for integrating creative activities of people. This framework for mega-creativity consists of four activities:

• Collect: Learn from what exists
  
• Relate: Consult with peers and mentors
• Create: Think: explore solutions
  
• Donate: Disseminate the results and contribute

This is not a waterfall process. It is an interactive iterative framework for innovation. Shneiderman's book focuses on the need to develop software to support this framework. The participants in the functional test tool workshop focused on the need to develop testing software to support those developing software to support this framework. And in doing so, we exhibited this framework in action -- without even identifying the framework. (I read about the framework on the plane home from the workshop.)

Gathering people that are interested in and working on solutions together accelerates the collection, creation, and donation. I expect great things to come from this gathering.

My thanks and appreciation go to the Agile Alliance for sponsoring this workshop; and to Ron Jeffries, Elizabeth Hendrickson, and Jennitta Andrea for organizing it.

Let's keep the innovation ball rolling and build "new computing" tools.

---

Links

• Next Generation Functional Testing, Geoffrey Wiseman
• Envisioning the Next Generation of Functional Testing Tools, Jennitta Andrea
• Functional Test Tools: the Next Generation, Elizabeth Hendrickson
• Portal Transparency, Ward Cunningham

Imaginary Testing

Ever thought of a test that you haven't executed? Ever wonder if it might be valuable to add a use case, test case, or test charter? Ever feel like you spend more time talking about how to solve problems than it would take to try some of the proposed solutions?

I've been involved in discussions about software bugs that take more time than it would take to fix and retest. I understand that it is important to consider the risks associated with a bug and any proposed solutions. However, some times we just need to do it.

I've been in test planning meetings that take more time than it would take to execute the proposed tests. I've also wasted time performing unnecessary tests. The problem is that the usefulness of a test is usually not known until we have the results from that test.

When it makes sense, stop hypothesizing and start testing.

Imaginary testing is unreliable.

Failure Usability

One of my pet peeves about software is bad error messages. In my view, a bad error message is one that does not tell the user how the error impacts them and what they need to do in response to the error. Too many messages fail to communicate this information in terms that the software's user is going to understand. Too many of these messages are written for developers, not users.

There is a place for error logging in terms that help developers and testers troubleshoot and fix problems. This information is often best written to log files, not displayed in the user interface.

Pradeep Soundararajan and I recently discussed some of our experiences with error messages. You can listen to excerpts from this conversation using the link below.

Click to listen: ErroneousErrors.mp3

After the above conversation, Pradeep walked me through an exercise that demonstrated a case in a popular office application in which I, the user, was not sufficiently informed that an error was occurring. It was obvious that the actions I was trying to perform were not functioning but the software gave me no clear indication of why it did not work as I expected.

Good testers recognize the need to include "negative testing" in their search for significant bugs. Testing for errors is a common part of functional testing. Let's go beyond functional testing of errors. Let's test errors for usability.

Here is an error testing mnemonic I created after our conversation.

• Functional
• Appropriate (or Apropos*)
  
• Impact
• Log
• UI
• Recovery
• Emotions
  

Functional
Does the error detection and reporting function as required? Are errors not detected that should be detected? Are errors reported? Do error dialogs function as expected? Do the buttons work?


Appropriate (or Apropos*)
Are errors detected and reported in an accurate and timely manner for the intended audience? Are errors reported as soon as an error condition is met? Are warning messages displayed while there are enough resources to remedy the problem? Is a user allowed to waste time and effort only to be told that their work cannot be applied? Is the text of a message accurate? Does the text convey the situation to the intended audience? Is the error described in terms that will be understood by the intended audience?

Impact
Is the impact of the error sufficiently communicated to the user? Does the message contain too little information for the user to understand what occurred and how it impacts what they were attempting to do? Does the message contain extra information that distracts from communicating the impact?

Log
Does technical information need to be logged for support, system administrators, developers, or testers? Will this log information be available if the user waits to contact support? Are log messages standardized to allow for automated information mining? Are logs detailed enough to facilitate troubleshooting? Are errors logged that add no value? Is there too much logging? Does excessive logging negatively impact performance and disk space? Does excessive logging complicate error investigation?

UI
Are users given some indication that what they attempted to do failed? Are user interface messages worded for the intended audience? Are user interface error messages consistent with the look and feel of the software? Are error messages consistent with other activity that causes the same error elsewhere in the application? Are errors communicated in an efficient manner? Does a user need to click away excessive dialogs? Is this error best communicated as an error dialog? Is this error best communicated as text added to the window? Is this error best communicated audibly?

Recovery
Does the error message tell the user how to recover from the error condition? Does the software facilitate recovery? If needed, is contact information provided? Is the user prompted through the recovery or left to figure it out on their own?

Emotions
What emotions are likely to be raised by the error message? Does the information in the message add to a user's frustration or help quiet it? If a user is being told that they need to pay more to use a feature, does the message encourage them to upgrade or does it encourage them to find a competitor's product? Does the error message cause more confusion?


Here's an error message Pradeep gave me to help test the mnemonic. What faults can you find in the error message using the mnemonic?





The next time you see an error message -- or don't see one that you should -- don't stop at functionality. Check the rest of the FAILURE. (And look here for a PowerPoint show demonstrating the mnemonic.)


PS: While attempting to save this post, Blogger gave me the following error. This error fails many of the tests above. The second save button distracted me from the light grey error message. Trying again did not fix the problem. It was not clear if the error meant that my text had been saved or not. I finally had to copy the HTML of the post and paste it into a new Blogger session. Bad error reporting is easy to find. :)


* UDATE: 18 Aug 2007 @ 6:32 PM: Michael Bolton suggested that "Appropriate" would be more appropriate than "Apropos". After some consideration, I agree. Thanks Michael.

Excuses, Excuses

I have heard a variety of responses from developers in response to bugs I report. Some are good, some bad, and some are just plain ugly. Here are a few handfuls.

• That's strange.
• How'd you do that?
  
• It works on my machine.
• I already fixed that. You'll have it in the next build.
  
• No user would do that.
• That's not how you're supposed to do that.
  
• It's a data problem. Tell the users to fix the data.
  
• That's a cool bug! Show me again.
  
• I didn't touch that code.
• It works as designed.
• It works as coded. [Well, duh. What else would it do?]
  
• That's not a bug, it's a feature.
• I can't test everything.
• Thank you.
  

... and my absolute favorite (this came from a development manager)

• Don't judge it, just test it.

The difference between the good and bad is often the relationship between developer and tester. Testers need developers to create something to test. Respect your developers. Communicate with respect and help turn the ugly responses into good responses.

As I've heard James Bach say: Testers don't create quality. Developers create quality.

What's your favorite bug report response?

Ugly Babies: Another Reason Why We Need Testers

... I got on the train ... And I noticed that the woman across from me in the aisle had her baby with her. Ugly baby. Ugly baby.

From the other end of the coach comes this guy and he was very drunk and he was staring at the baby. ... And the guy said, "I'm looking at that ugly baby. That's a horrible looking baby lady. Where'd you get that baby from?"

And the woman said, "I don't have to take that!" And she snatched the emergency cord and the train came to a screeching halt, and the conductor came running in.

Now this was his moment. At this moment he represented the Pennsylvania Railroad. And he said, "what's going on here?"

And the woman said, "This man just insulted me. I don't have to spend my money and ride this railroad and be insulted. I'd rather walk."

And the conductor said, "calm down! Calm down! Madame there's nothing, nothing that the Pennsylvania Railroad will not do to avoid having situations such as this. Perhaps it would be more to your convenience if we were to ... let you sit somewhere else in the coach. And as a small compensation from the railroad ... we are going to give you a free meal; and maybe we'll find a banana for your monkey."

- Flip Wilson


We are like parents when it comes to our creations: we can be blind to ugly.

I am a believer in developer testing. Good unit testing is essential to good software. Practices like Test Driven Development (TDD) help ensure that software works from the start. Anyone that's been in the software development business for long knows that the cost of defects exponentially grows as a project progresses. Preventing and finding defects early should be a goal in every software project. Early testing by developers does not remove the need for testing by someone other than the coder.

There are a number of reasons for "independent" testing. These reasons often include references to requirements testing, end-to-end testing, integration testing, system testing, and independent verification. These reasons focus on testing things as complete systems or from a user's viewpoint. I have another reason: Creators can be blind to ugly.

Creators have an emotional attachment to their creation. This attachment can get in the way of objective evaluation. It is difficult to be both creator and critic. It is as if creativity and critique are opposite sides of a scale. We can be both, but increasing one can harm the other. Creators need to balance the two without allowing self-criticism to hamper creativity.

I think its good for creators to have an emotional attachment to the product of their thoughts and labor. External critics can help a creator improve their creation without stifling creative thinking.

Creating and critiquing are separate skills. When one group can focus on one and another group can focus on the other: together they are stronger (if the critics are more tactful than the drunk in Flip's story).

At least that's my opinion. Now I open the floor to the critics. :)

Marching Through Hell

There is a Churchill quote about when you're marching through hell, just keep marching. Well, that's basically what we're doing. We need to slow this down and get it right.
- Charles Burbridge
CFO, Los Angeles Unified School District

A couple weeks ago, the Los Angeles Unified School District's new computer system incorrectly paid over 32,000 employees. That's almost a third of the district's employees. And the trouble started this past January.

• Glitches in Los Angeles Payroll System Spark Furor, Education Week
• L.A. Unified payroll snafu's costs soaring, L.A. Times

I've been involved in some software projects with a "hell" phase in the SDLC. I've worked on projects that failed. I've been involved with projects that went millions over budget. However, I'm thankful that I have never worked on a project for which tens of thousands of production issues were reported in a handful of months.

Through the replacement of our aging financial, human resources, payroll and
procurement systems, the District will:

1. Dramatically improve service delivery to schools
2. Radically improve the efficiency of District operations and our ability to manage them
3. Reduce/eliminate paperwork and redundant manual processes
4. Increase accountability and transparency to the public in the use of public funds
5. Provide better data for decision makers and stakeholders at all levels

- LAUSD ERP Project Vision Statements, March 2004

This enterprise software development project started out with good intentions and lofty goals. However, it appears that something went terribly wrong in implementation. The district's public web site does not include much information about the problems. The latest document I saw with information for employees with payroll problems has a February date. I also find it interesting that a more recent document is missing item #4 from the above list. I guess that when things go wrong, there's no need to keep the public and employees informed. Right? Wrong. I don't know the details of what went wrong, but I do know that when things go wrong people tend to communicate less. I also know that it can be difficult to admit failure. This project reminds me of the six stages of a project that I occasionally encounter in forwarded emails and on bulletin boards. It will be interesting to see what and who is eventually blamed.

The six stages of a project

1. Enthusiasm
2. Disillusionment
3. Panic
4. Search for the guilty
5. Punishment of the innocent
6. Praise and honor for the non participants

Enterprise software systems can be very complex. The vision statement says that they chose a commercial package and partnered with an integrator because they want to configure software and not customize software. A payroll system for over 100,000 school district employees has got to be one of the most complex "configuration" projects around. I believe there were good intentions, but this looks like a game of semantics. Enterprise systems are difficult to configure and deploy -- especially when you have numerous customized needs.

I don't know what kind of testing was performed on this system before it went to production. The original visions document references the need for testing support.

A complete test environment (development, QA, and production instances), including adequate infrastructure, staff (DBA), storage and network capacity, a full volume of test data.
- LAUSD ERP Project Vision Statements, March 2004

I wonder what happened in implementation. Obviously, something was missed or skipped.

I did notice that the project appears to be on schedule. When Churchill said "If you're going through hell, keep going.", the point was to find your way out of the situation -- not ignore it.

I was once offered a "great opportunity" to work on a "high profile" project. I accepted that offer. I learned a great deal from that experience. One of the things I learned is that it may not be a good thing to be visible when there are problems that are out of your control. Yet we have to take risks to gain a reward.

Right now, I am thankful that I am not "marching through hell". And I'm thankful that my paychecks come as expected.

To those in the LAUSD, these problems will pass. Right?

We expect these problems to subside in the very near future.
- Charles Burbridge
CFO, Los Angeles Unified School District

How near is very near?

If you have any more information about the development and testing practices on this project, I'd like to see it.

Bad Messages

I regularly spend a great deal of time tracking down the root cause of software errors -- both on and off the job. Much of the investigation effort could have been avoided if I were not presented with incomplete or incorrect error messages. The text of error messages appears to be commonly overlooked by software developers and testers.

One of my early test automation development tasks was to fix the error messages in a test tool. The tool was used to validate the structure of data exchanged between computer systems. In nearly all cases, this tool displayed an error dialog window stating that the data was not valid. This tool gave the user (a tester) no information about what was wrong with the data. The lack of detail in these messages required that the user manually examine the data -- bit by bit -- with a protocol analyzer. This turned investigation of errors reported by an automation tool into a tedious manual task. The tool encountered a problem but did not report the source of the problem. I reviewed the code and added more detail to the error messages in this test tool. I also added logging of the data exchange in a format that made sense to human beings that could be reviewed without a protocol analyzer. The improved error messages and human-readable logging greatly decreased the troubleshooting and investigation time.

I am amazed at the poor error messages in test automation tools. Test automation tools and frameworks are likely to encounter errors in the software under test. I believe that this requires better error reporting and handling than many other tools. I am also disappointed in how difficult some of the automation tools make it to create scripts that run unattended.

One tool I use regularly often stops executing tests and displays a message like the following.

It took a great deal of effort to figure out how to coax the tool into returning error codes that could be handled in code instead of prompting a user during an automated test execution. The tool vendor's support personnel didn't seem to understand why it was a problem that I could not code their tool to handle an error without human intervention at run time. They kept telling me to fix the application under test to make the test tool's error dialog go away.

I also recently uninstalled a different test tool that had a horrible uninstall interface. It displayed a window with an uninstall progress bar that didn't move. After several minutes, the application displayed an "Install Complete" window underneath the progress window. The progress bar began to slowly move only after I clicked the "Finish" button on the buried completion window. Then it prompted me with something like the following.

Do I want to erase all files? What's the context? Of course, I don't want to erase all my files. I wonder how much the maker of this software spends on support calls about this message.

I used to receive numerous inquires from new computer users about the Windows "program has performed an illegal operation" error message. Many users thought this meant that they did something illegal and the police would soon be knocking on their door. People familiar with computers understand that this error message is about the program's interaction with the operating system and hardware; however this error message is misleading. This error message is designed for developers, not users. Good error messages tell the user what they need to know about the problem and what to do about it. Good error messages explain the problem but do not overwhelm the user with information that is useless to the intended user. This means that different kinds of applications require different kinds of error messages.

If you write software, please provide your users with accurate messages tailored to the user -- not developers. If you create test tools or automation frameworks, provide testers with information that is useful in determining what happened. Please.

People who write framework software should spend more time on useful error messages that show people why the error occurred and give a clue as to how to fix it.
- Eric M. Burke, We need better error messages

Want to practice creating better error messages? Try the Error Message Generator.

Poka-Yoke

Poka-Yoke is not a dance. Its not an event at a rodeo. Its not what my kids do to each other in the back seat of the car. Poka-Yoke is Japanese for "mistake-proofing". Poka-Yoke was developed by Japanese industrial engineer Shigeo Shingo. He realized that people cannot be expected to work like machines and consistently do everything the same way every time they do it. People make mistakes and poorly designed processes can make it easier for people to err. Poka-Yoke's goal is to make it difficult for people to make mistakes through mistake prevention and detection.

Prevention

Applied poka-yoke gives users warnings about incorrect behavior and directs users towards the correct behavior. Computer PS/2 keyboards and mice share the same physical connector design but the connectors are usually color-coded to indicate which device goes into which port on a computer. Some computing hardware is shipped with warning stickers on top of connectors telling users to read a manual or install software before plugging in the device.

Poka-Yoke also means stopping users from doing the wrong thing. Diesel fuel pump nozzles will not fit in a vehicle that requires gasoline. The ignition key cannot be removed from most cars with automatic transmissions if the car is not in "park". Most cars with manual transmissions cannot be started unless the clutch pedal is pressed. These safety features prevent users from making mistakes.

Detection

Some errors cannot be prevented or are too expensive to prevent. The application of poka-yoke demands that errors be detected when and where they occur so that action can be taken before mistakes become bigger problems. Modern space heaters will automatically shut off if they are kicked over. A great example of automatic error detection and correction is the SawStop table saw that automatically disengages when the blade touches something that conducts electricity -- such as fingers. (See the video below.)


http://SawStop.com


Poka-Yoke Applied to Software

Poka-Yoke has existed in hardware products for decades. Poka-Yoke has improved quality and safety of many devices we use daily. While I do not like the behavior-shaping constraints of poka-yoke applied to intellectual tasks, directing and constraining user behavior is essential for good software. I do not advocate application of poka-yoke to the development process. I do advocate applying poka-yoke thinking to every stage of the software development life cycle to improve the quality of the software products we produce. Designers should think poka-yoke. Coders should think poka-yoke. Testers should think poka-yoke. Thinking about usability can lead to fewer bugs.

We are human and there will be bugs. To the extent that quality assurance fails at its primary purpose -- bug prevention -- it must achieve a secondary goal of bug detection.
- Boris Beizer
Software Testing Techniques

Prevention

Keep it simple. Make it easy for users to identify the expected correct way to use the software. Warn them if they try to do something wrong. Don't overwhelm users with unnecessary options.

When the risk of users not following a warning is great, prevent users from doing bad things. Things like list boxes and radio buttons can prevent users from entering invalid data. Data input constraints keep users and the software on the expected path. The security risks in web applications increase the necessity to prevent users from doing what they are not supposed to do.

Detection

It is especially important to detect errors that get past the warnings and constraints and stop processes before errors develop into bigger problems. The earlier an error is detected the easier it is to recover. Bad data detected when it enters a system does not have a chance to cascade into the rest of the system.

Poke-yoke thinking can improve usability and prevent bugs.

---

Some Poka-yoke resources on the web:

• Mistake-proofing Example Wiki
• Using Poka-Yoke Techniques for Early Defect Detection, Harry Robinson
• John Grout's Poka-Yoke Page
  

Not Gonna Bow

Individuals and interactions
over processes and tools

Working software
over comprehensive documentation

Customer collaboration
over contract negotiation

Responding to change
over following a plan

- Agile Manifesto

Neary 400 years ago, Francis Bacon challenged the status quo in scientific thought in “The New Organon”. James Bach recently pointed out some interesting quotes from this work that apply to software testers. I agree.

Bacon argued that placing our preconceived beliefs over what we observe causes great harm. He went so far as to describe these harmful preconceived notions as “idols”. Bacon put these idols into four categories:

• Idols of the Tribe: Errors common to mankind.


• Idols of the Cave: Errors specific to each individual’s education and experience.


• Idols of the Market Place: Errors formed through association with others — often due to misunderstanding others.


• Idols of the Theater: Errors formed from dogma (institutionalized doctrine) and flawed demonstrations.

All of these exist in software testing. As testers, we should be questioning these “idols”, not worshiping them. Sometimes questioning them may prove them right.

Bacon did not ask anyone to abandon their beliefs without cause. Instead he asks that we not make them idols capable of leading us to ignore what would be obvious if we weren’t looking through the distorted mirror of our idols.

A modern day simplification of Bacon’s arguments may be the Agile Manifesto. We should not let our idols of process, documentation, contracts, and plans prevent us from accomplishing the desired goal. Process, documentation, contracts, and plans are only good in as much as they help. They should not prevent us from seeking improvement.

In some ways I believe that the promotion of testing folklore is the result of an industry-wide desire to show that we are mature — as mature as the engineering of physical products. I believe that eagerness to demonstrate maturity helps lead to the implementation of bad processes and cerfifications. Ironically, enforced process (see the bottom of the FSOP cycle) works best for the immature and gives the impression that anyone that can follow the process can test software.

Don't get me wrong. Process and documentation are good things that help even the smartest people when appropriately applied.

"The only thing that interferes with my learning is my
education." - Albert Einstein

We need to seek continual improvement. It is sad that process and certification often become idols that overshadow the real goals.

How Doctors Think

Michael Bolton recently brought Dr. Jerome Groopman’s newest book, “How Doctors Think”, to my attention. Michael suggested to the software-testing list that this book contains information that is relevant to software testing. In this book, Dr Groopman explores how doctors diagnose medical problems and what can be done to reduce bad judgment.

Michael Krasny recently interviewed Dr. Groopman on public radio about this book. You can listen to the interview here. Dr Groopman made a number of statements during this interview that I believe apply to us software testers’ search for and diagnosis of software bugs. I expect to find more in the book.

Here are some gems from the radio interview:

About computer-assisted diagnosis software.

It’s a two-edged sword. On one hand, it can be useful if you feed in the most important and relevant symptoms, but the problem is that you need to be able to illicit them effectively from the patient and so you return to language and you return to some of these thinking errors or these cognitive traps that we can all fall into. So if you put in for, example the first prominent symptom that you hear from someone into the software package, it may be that the fourth thing that the person complains about is the most relevant one. Recently you may have seen, for example, in reading mammograms, that computer assisted diagnosis actually generated more false positives. It caused the radiologist to think that shadows which he or she would normally think were benign were in fact cancer and didn’t add to the diagnosis of cancer –the true positives. So I think this technology is worth exploring, but I think that we have to be very careful about it because it some ways it can seem seductive. You cannot substitute a computer for a human being who listens and filters and judges and elicits from the patient the full story.

The same applies to test automation. It can be a useful tool if we don’t let it lead us astray. We human testers need to be actively involved in listening, filtering, judging, and eliciting the full story.

About time-constraints:

Its one of the biggest issues in the current delivery of medical care. You know, everyone is being told as physicians to see patients in 12 minutes. See patients in 11 minutes. So you’re sitting there with one eye on the clock and you can’t think well in haste. And there’s the natural inclination to anchor onto that very first impression as they stop there and just function as if you’re working on an assembly line in a factory. ... patients are being ill-served.
…
I think everyone in the medical system feels under siege. There’s not a lot of satisfaction but in a way this is penny-wise and pound-foolish because the cost of misdiagnosis is extraordinary. … It also costs in terms of dollars. Its much more
expensive to care for and treat an advanced problem than to make a diagnosis early on.

I think we have to force ourselves to resist and part of that can be done: sometimes we have to extend a visit. But we really are beholden to administrators and managed care plans, so one of the things that I’ve begun to do is if I can’t get to the bottom of a problem in my 15 minutes allotted visit,
then I say to the patient, you know, I haven’t figured it out. I need to think more. ... reschedule an appointment and spend more time.

Testers also make mistakes under time pressure. We also need to be aware that our first impressions may not be right. Testing is not a factory assembly line (even though this is a widely-held view). Testers need to be engaged throughout testing and not just mindlessly follow test scripts. And sometimes we need to lobby to spend more time than the schedule originally gives us.


About the art vs. science of diagnosis and improving the effectiveness of that diagnosis:

There’s a seductive nature of numbers, but statistics from clinical studies and so on are just averages and they may not reflect the individual in front of you. ... So real judgment involves customizing medicine: looking at scientific data but also seeing how it applies or doesn’t apply to the person sitting in front of you.
...
It’s not technology, but its language. Language is really the bedrock of medicine. Most of what we do with a doctor – or should be doing with a doctor – is talking… engaging in a dialog.
...
... all of us are wired to make these kinds of snap judgments to sort of go with our first impressions. to rely very heavily on intuition. … That often works but in medicine unfortunately too often it doesn’t work because we have a tendency to latch on to that first bit of information that we get from a patient and then run with it; as opposed to keeping our mind open and constantly doubting ourselves.

Testing is not an assembly line process. Testers need to keep their minds engaged throughout the testing process. We should not ignore our snap-judgements (blink testing), but we also need to look and think beyond those first impressions. We need to continually question both the software and our own judgement as we test.

We need to communicate more than numbers. We need to communicate stories. We need to translate bugs and metrics into language that matters to the business.

Software is more reliable than people!

Software is 100% reliable. It does not break. It does not wear out.

We can depend on software to exactly what it is coded to do every time it does it.

This is why software quality stinks!

The consistent repeatability in software is both a blessing and a curse. We can depend on software processing the same data in the same way every time. We can also rely on software to not do what it does not do every time.

The repeatability of software is both its greatest strength and weakness. A simple mistake in design or implementation will forever repeat itself each time a computer program runs.

Over 50 years ago, three mathematicians wrote:

Those who regularly code for fast electronic computers will have learned from bitter experience that a large fraction of the time spent in preparing calculations for the machine is taken up in removing the blunders that have been made in drawing up the programme. With the aide of common sense and checking subroutines the majority of mistakes are quickly found and rectified. Some errors, however, are sufficiently obscure to escape detection for a surprisingly long time.

[R.A. Brooker, S. Gill, D.J. Wheeler, "The Adventures of a Blunder", Mathematical Tables and Other Aids to Computation, 1952]

The source of the problem is people. We are not reliable. We make mistakes. Software amplifies and repeats our successes and our mistakes equally well.

When our software encounters the unexpected, errors occur. As developers and testers, we need to expect the unexpected. Think about how requirements might be misunderstood and clarify any ambiguity. Think about how users might misuse a system -- either accidentally or intentionally -- and ensure that the system can handle that user behavior.

The software we use today is exponentially more complex than the software being developed 50 years ago. There have got to be more opportunities for blunders in today's software than there were in software development half a century ago. And with that complexity some errors become even more obscure and are more likely to escape detection by developers and testers. Users, however, seem to easily encounter these errors.

A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools.

-- Douglas Adams

Top ten web application security flaws

The topic of tonight's SQuAD meeting was software security. Mike Walters presentation, "Integration of Security into the SDLC", highlighted the need to implement and validate security throughout the software development lifecycle. Mike mentioned that security is moving from the realm of "non-functional" testing to functional testing. Security has become an important functional requirement. Mike stressed the need to define security requirements at the start; get developer buy-in; and provide developers with the tools and training to build secure software. The risks of poor security are often too great to ignore.

Mike recommended the OWASP Top Ten Project as a great starter list of web application security threats to consider.

• A1 Unvalidated Input. Information from web requests is not validated before being used by a web application. Attackers can use these flaws to attack backend components through a web application.
• A2 Broken Access Control. Restrictions on what authenticated users are allowed to do are not properly enforced. Attackers can exploit these flaws to access other users' accounts, view sensitive files, or use unauthorized functions.
• A3 Broken Authentication and Session Management. Account credentials and session
  tokens are not properly protected. Attackers that can compromise passwords, keys, session cookies, or other tokens can defeat authentication restrictions and assume other users' identities.
• A4 Cross Site Scripting. The web application can be used as a mechanism to transport an attack to an end user's browser. A successful attack can disclose the end user's session token, attack the local machine, or spoof content to fool the user.
• A5 Buffer Overflow. Web application components in some languages that do not properly validate input can be crashed and, in some cases, used to take control of a process. These components can include CGI, libraries, drivers, and web application server components.
• A6 Injection Flaws. Web applications pass parameters when they access external systems or the local operating system. If an attacker can embed malicious commands in these parameters, the external system may execute those commands on behalf of the web application.
• A7 Improper Error Handling. Error conditions that occur during normal operation are not handled properly. If an attacker can cause errors to occur that the web application does not handle, they can gain detailed system information, deny service, cause security mechanisms to fail, or crash the server.
• A8 Insecure Storage. Web applications frequently use cryptographic functions to protect information and credentials. These functions and the code to integrate them have proven difficult to code properly, frequently resulting in weak protection.
• A9 Application Denial of Service. Attackers can consume web application resources to a point where other legitimate users can no longer access or use the application. Attackers can also lock users out of their accounts or even cause the entire application to fail.
• A10 Insecure Configuration Management. Having a strong server configuration standard is critical to a secure web application. These servers have many configuration options that affect security and are not secure out of the box.

The next time you are involved in designing, coding, or testing a web application: consider these things.