A Good Practice

The Association for Software Testing (AST) is a professional organization dedicated to advancing the understanding and practice of software testing. The AST provides forums for academics, students, and testing practitioners to discuss testing. AST does this through online forums, workshops, education programs, and conferences. The third annual Conference of the Association for Software Testing (CAST) provides a great forum for face-to-face conferring. This is not your typical conference where experts talk at the masses. This is the software testing conference that puts the confer back in conference.

Ever sit in a presentation about testing and think anything like the following?

• Yeah that works for you but it'll never work in my situation.
• What do you mean by X?
  
• She must work with idiots.
• How does he know what he says? I want to see data.
• My management would never go for it.
  
• What planet is he from?
• You're full of it.
  

Your are not only free to think these things at CAST, but you are free to question presenters. Time is built into the program for facilitated discussion of every presentation.

If you'd like to become a better software tester, join AST and come to CAST.

If you'd like to meet and confer with peers from around the world, join AST and come to CAST.

If you'd like to meet and confer with testing experts, join AST and come to CAST.

If you'd like to be challenged, join AST and and come to CAST.

If you'd like to hear Gerald Weinberg talk about the past, present, and future of software testing: join AST and come to CAST.

If you can't afford those other testing conferences, join AST and come to CAST -- its about half the price of other conferences.

If you'd like to compete against other testers, join AST and come to CAST.

If you care about software testing, join AST and come to CAST.

Only through judgment and skill,
exercised cooperatively throughout the entire project,
are we able to do the right things at the right times
to effectively test our products.
context-driven-testing.com

While I am not a believer in best practices, I believe its a really good practice to participate in CAST 2008. See you in Toronto.

Better Tools for Individuals through Collaboration

Individuals and
interactions

over
processes and tools

- from the Agile Manifesto

I spent the second half of last week at the Agile Alliance Functional Testing Tools Visioning Workshop. (How's that for a long name?) Before the workshop, I was thinking that it seemed a little oxymoronic to have an agile workshop with a focus on tools. Perhaps my thinking was triggered by my concerns about those who seem to value "agile" processes and tools (often ones they sell) more than people.

Agile people are supposed to care about people and not care about tools. Right? Wrong.

while there is value in the items on
the right, we value the items on the left more

- from the Agile Manifesto

Software is developed by people for people. Agility involves building better software by adapting to the needs of people instead of letting processes and tools lead the way. Process and tools do best when they have a supportive role in software development. Better tools can support agility but they cannot make anyone agile.

The tool-centric discussions at the workshop were driven by a desire to build better software for people that build and test software. It is about people.

It then seems quite apropos that the book I indiscriminately grabbed off the shelf (well, I picked it for its size more than its content) to read on the airplane to and from the workshop is Ben Shneiderman's Leonardo's Laptop: Human Needs and the New Computing Technologies. The first chapter contains the following paragraphs that affirm my thinking about the role of automation in software testing. (Emphasis is mine.)

The first transformation from the old to the new computing is the shift in what users value. Users of the old computing proudly talked about their gigabytes and megahertz, but users of the new computing brag about how many e-mails they sent, how many bids they made in online auctions, and how many discussion groups they posted to. The old computing was about mastering technology; the new computing is about supporting human relationships. The old computing was about formulating query commands for databases; the new computing is about participating in knowledge communities. ...

The second transformation to the new computing is the shift from machine-centered automation to user-centered services and tools. Instead of the machine doing the job, the goal is to enable you to do a better job. Automated medical diagnosis programs that do what doctors do have faded as a strong research topic; however, rapid access to extensive medical lab tests plus patient records for physicians are expected, and online medical support groups for patients are thriving. ... Natural language dialogs with computerized therapists have nearly vanished, but search engines that enable users to specify their information needs are flourishing. The next generation of computers will bring even more powerful tools to enable you to be more creative and then disseminate your work online. This Copernican shift is bringing concerns about users from the periphery to the center. The emerging focus is on what users want to do in their lives.

- Ben Shneiderman, Leonardo's Laptop

Although many think of us software testers and developers as eccentric nerds, software developers and testers are human too. Like other humans, we desire tools that help us do a better job. This was the theme of the workshop: envisioning ways that tools can help us do a better job testing software.

After reviewing existing tools used by agile teams: we identified software testing issues that have been solved (yellow), those that have been partially solved (orange), and those that have not been solved (pink). As I recollect, most of the solved issues were technical problems and most of the unsolved problems were people problems. Many of the partially solved problems were those for which I believe we have technical solutions but have not yet been integrated and presented in ways that best support people. Much of the "what's next" discussion at the workshop was focused on how to integrate existing tools that each partially solve problems but together could move problems to the solved group.

Once the technical problems are solved, we can work on the tools to help with the people problems: we can move from old computing to new computing.

In Leonardo's Laptop, Ben Shneiderman presents a framework for integrating creative activities of people. This framework for mega-creativity consists of four activities:

• Collect: Learn from what exists
  
• Relate: Consult with peers and mentors
• Create: Think: explore solutions
  
• Donate: Disseminate the results and contribute

This is not a waterfall process. It is an interactive iterative framework for innovation. Shneiderman's book focuses on the need to develop software to support this framework. The participants in the functional test tool workshop focused on the need to develop testing software to support those developing software to support this framework. And in doing so, we exhibited this framework in action -- without even identifying the framework. (I read about the framework on the plane home from the workshop.)

Gathering people that are interested in and working on solutions together accelerates the collection, creation, and donation. I expect great things to come from this gathering.

My thanks and appreciation go to the Agile Alliance for sponsoring this workshop; and to Ron Jeffries, Elizabeth Hendrickson, and Jennitta Andrea for organizing it.

Let's keep the innovation ball rolling and build "new computing" tools.

---

Links

• Next Generation Functional Testing, Geoffrey Wiseman
• Envisioning the Next Generation of Functional Testing Tools, Jennitta Andrea
• Functional Test Tools: the Next Generation, Elizabeth Hendrickson
• Portal Transparency, Ward Cunningham

Keys to Innovation

Lee Copeland's CAST keynote address referenced in a previous post was not only about books. Good books was one of the items on Lee's list of eight recent innovations in software testing. Lee's complete list is shown below.

Innovations in Software Testing
(Lee Copeland's List)

1. Context-Driven School
2. Testing Specialties
3. Test-First Development
4. Really Good Books
5. Open Source Tools
6. Session-Based Test Management
7. Testing Workshops
8. Certification
   

I was glad to see most of the items on this list. I am especially happy to see the Context-Driven School and Session-Based Test Management on the list. I believe that these have had a significant impact on software testing and have great potential that has not yet been realized.

Tester certification may be an innovation but I don't think its impact has been good. In my opinion, the current certification options are bad. (There was a certification debate hosted by AST at CAST this year. Please take a look at Tim Coulter's review: AST Certification Debate.) Most of the certifications show nothing more than one's ability to pass a certification exam. And, many of the certifications are based on context-free and outdated views and techniques. I reviewed some ISTQB sample questions with a group of very smart testers and we could not identify a correct answer for many of the questions. We could, however, make a good guess at what we thought was expected by the exam writers. Matters of opinion and guessing at implied contexts should not be the basis for any exam. I think the following statement summarizes this concern quite well.

We don’t mind that some people hold (and teach) views or techniques that we consider antiquated. We do mind that in prep courses that teach you how to pass “objective” exams, there is no place for presentation of controversy or thoughtful analysis of the fundamentals.
- Cem Kaner and Tim Coulter

Now back to Innovations...

In typical CAST style, Lee asked the audience for things they thought he missed. The audience came up with the following additions. Yes, Model-Based Testing was mentioned twice -- followed by applause from Harry Robinson. :)

1. Collaborative groups
2. Model-Based Testing - specifically model-based automation
3. Testers help define what is correct - testing is more than comparing dictated expected and actual results
   
4. Model-Based Testing
   
5. Fluidity - not freezing plans - recognizing the need to be adaptable
   
6. Study of software development and testing history - learning from the past
   
7. Toolsmithing
8. Ethnomethodology - the study of common sense (guess who this came from)
   
9. Test management as project management
10. High volume semi-random test automation
11. Academic research in testing techniques
12. Prediction based on source code

What would you add to the innovations list?

What innovations do you think might be just over the horizon?

Woodpeckers, Pinatas, and Dead Horses

Here's some short blurbs of a few things I took away from CAST sessions.

From Lee Copeland's keynote address:

• "It's nonsensical to talk about automated tests as if they were automated human testing."

• Write or speak about something you're knowledgeable and passionate about.

• Combine things from multiple disciplines.

From Harry Robinson's keynote address:

• Weinberg's Second Law: If Builders Built Buildings The Way Programmers Write Programs, Then The First Woodpecker That Came Along Would Destroy Civilization.

From Esther Derby's keynote:

• To successfully coach someone, they must want to be coached and want to be coached by you.

From James Bach's tutorial:

• Pinata Heuristic: Keep beating at it until the candy comes out. ... and stop once the candy drops.

unless ...

• Dead Horse Heuristic: You may be beating a dead horse.

yet beware ...

• If it is a pinata, don't stop beating at it until the candy drops; but if it is a dead horse, your beating is bringing no value. It can be a challenge to determine if its a pinata or a dead horse.

From Antti Kervinen's presentation:

• Separate automation models into high level (behavior) and low level (behavior implementation) components to reuse test models on a variety of platforms and configurations.

More from James Bach's tutorial:

• Testing does not break software. Testing dispels illusions.

• Rational Unified Process is none of the three. (attributed to Jerry Weinberg)

From the tester exhibition:

• Testing what can't be fixed or controlled may be of little value. Some things may not be worth testing.

• There is great value in the diversity of approaches and skills on a test team.

• It may be possible to beat a dead horse and test (and analyze) too much. Sometimes we should just stop testing and act on the information we have.
  

From Doug Hoffman's tutorial:

• Record and playback automation can be very useful for testing for the same behavior with many configurations. And, once the script stops finding errors: throw it out.

From Keith Stobie's keynote:

• Reduce the paths though your system to improve quality. Fewer features may be better.

• Free web sites often have higher quality than subscription sites. This is because it is easy to measure the cost of downtime on ad-supported systems.

From David Gilbert's session:

• People expect hurricanes to blow around and change path. We should expect the same with software development projects. (David has some interesting ideas about forecasting in software development.)

• Numbers tell a story only in context. You must understand the story behind the numbers.

One more from James:

• Keep Notes!

What did you take away from CAST?

Where No Confabulation Goes Untested

confer

verb

• have a conference in order to talk something over

The Conference of the Association for Software Testing (CAST) is coming this July.

I missed last year's conference but have heard great things about it. Based on all the wonderful things I've heard from those that were there, I am looking forward to this year's conference.

The CAST is different than most conferences where people sit and listen to someone present to an audience without public questioning of what is presented. AST encourages testers to test the presentations. Time is allowed for discussion at every presentation. Challenging ideas is encouraged. I could go on. However, I don't think I can push this conference any better than David Gilbert. Therefore, please take a look at David's blog post: CAST in stone.

CAST early bird registration ends this week. Register at
http://associationforsoftwaretesting.org/conference/registration.html

I hope to see you there.

Ben

STAR East 2007 Conferred

I am sitting in the airport waiting to fly home from STAR-East. The conference was great. It was not great due to the many wonderful presentations. It was great because of what happened outside the scheduled activities. I got to confer with colleagues from around the world.

The best part of conferences such as STAR-East is the opportunity to confer with peers and thought leaders in our industry. It is an opportunity to discover that we are experiencing common problems and share possible solutions. It is an opportunity to learn from the best. I often learn more over dinner and in the hallways than I learn in the presentations.

I was amazed at how quickly the conference attendees disappeared once the scheduled activities were completed. I know that many of us computer geeks are introverts. We may not be the most social bunch of people, but I believe a conference without conferring is a wasted opportunity.

See y'all at CAST.