The Confidence Game – What is the Mission of Testing?

Doubt is not a pleasant condition, but certainty is absurd. – Voltaire

Maybe it’s due to an extension of my tendency towards skepticism to myself, but I get really uncomfortable telling anyone that something is certain. That is especially true when it comes to software and interpreting the results of testing. There are just too many variables that impact the control and validity of the output, and that’s just limited to what we can know – let alone the things we don’t know! The great “unknown unknowns” loom in the shadows, waiting to rear their head and question our approach and as well – shake our confidence.

By definition, confidence is the quality or state of being certain. It’s knowing that something can be proved true, and is a by-product of actions taken in the process of acquiring that proof. Christopher Chabris and Daniel Simons created a famous experiment in studying inattentional blindness. In their book The Invisible Gorilla, they posit that we should be very unsure of what we are certain we know, and that our confidence or intuition can often mislead us. The idea of questioning the origins of our confidence is also echoed in Blink: The Power of Thinking Without Thinking by Malcolm Gladwell, and Thinking, Fast and Slow by Daniel Kahneman.

So what does that have to do with the mission of testing? It is extremely important that testers understand and adhere to their mission, as to replace it (either willfully or unintentionally) would be directly fogging the headlights on your project. So should the mission of testing be to give confidence? I don’t believe it should. I would agree with my friend Michael Bolton, that making “confidence” your mission in testing is akin to goal displacement, or substituting objectives with those that suit your means as opposed to the end.

I believe the mission of testing is gaining information; but here are some better examples for your reference:

  • Testing is a process of technical investigation, intended to reveal quality-related information about a product (Cem Kaner)
  • Testing is questioning the product in order to evaluate it (James Bach)
  • Gathering information with the intention of informing a decision (Gerald Weinberg)

So what is the problem with making confidence the mission of testing? Shouldn’t we want to have confidence in our products? Isn’t it a good thing to have confidence in our testing? Of course we want confidence in our products and testing, but if you make gaining that confidence your mission, in my opinion, you are intentionally adding confusion to the decision-making process. Aside from trying to hit the bulls-eye on the wrong target, testing for confidence is a slippery slope to ill-informed decisions, misuse of metrics, and a ready candidate for confirmation bias.

Testers should be constantly vigilant against all forms of bias, but especially confirmation bias. Making confidence your mission guarantees you will be seeking information to give your stakeholders certainty – instead of information that should give them pause for thought. Every tester has at times been subject to the “Curse of Cassandra“, or giving a valid warning that is not heeded. But nothing will put you permanently in that place quicker than having things go wrong after you’ve not only provided information to stakeholders – but have made a value judgement on their behalf!

Some may view me as overly skeptical. That’s fine. But I would rather err on the side of caution (and humility) when seeking information for my stakeholders. It’s up to them to decide what to do with what I give them – objectively finding it is hard enough without attempting to gain credibility through inappropriate means. So when someone tells me someone has asked them to give them confidence through testing, my simple advice to them would be this: stick to the mission.