Confidence or Information?

When context-driven software testers get together or talk online about testing, a lot of what we’re really talking about is bug hunting. Finding bugs, figuring out whether they matter, and communicating them efficiently to stakeholders are the main skills that I’ve noticed that people in the community care about. We tend to describe our role on teams as “looking out the window” or “holding the flashlight.” Testers, in our view, gather information and then package it for an audience, generally one or more stakeholders.

But I’m not sure that this is what our clients expect of testing, and even what testers outside of the context-driven conversation see their jobs as. In general, in fact, I think they see the tester’s product not as information, but confidence. They’ll say “test it and make sure it works. Even if they’d agree that’s not possible if you asked about it, I think that phrase reveals a lot about people’s basic beliefs about what testing does, and why they want it.

And honestly, they’re not wrong to want that. The value of a bug free product is obvious; the value of information is a a little more ambiguous. I mean, heck, the reason that the CDT community cares so much about information isn’t that we care about it as an end in itself, it’s that we see information as the primary tool for removing bugs, and making good decisions about the software project.

So maybe there’s no conflict here. Maybe confidence is the ultimate product of all that knowledge of that we’re so good at gathering. We focus on getting a good flashlight and deploying it skillfully, and our clients don’t care as they feel like they know enough about what’s out in the darkness.

I think it goes a little deeper than that, though. In fact, my private theory is that this is one of the big reasons it can be so tough to pry people away from test scripts and massive test plan documentation and certifications. None of those things help you find out about the product, but they’re great at creating the illusion of certainty. People who really want confidence will happily take fake but convincing certainty over real but uncomfortable knowledge.

Is this just a terminology problem? Does good information gathering inevitably lead to confidence? Is this a general problem for exploratory testers, or is this just my particular challenge, due to the way I approach testing? I’m still working on the answers to those questions, and a bunch more, but the more testing I do, the more this issue has been gnawing at me.