Wait…what? – Tales from the Testing Dark Side

Always do sober what you said you’d do drunk. That will teach you to keep your mouth shut. – Ernest Hemingway

I recently gave a talk at the EuroSTAR conference in Gothenburg, Sweden about how I feel you can re-frame the perception of your testing effort in your organisation. A big part of the philosophy underpinning my approach is to be honest, frank and up front about what is, and is not working with yourself first – then address anything that comes after that.

Part of my talk is about how bias and perceptions are formed, and I take several (hopefully humorous)  pokes at the software testing industry to illustrate my point. I feel strongly, that we accept far too much nonsense and unverified claims about software testing, and in order for there to be fundamental change, those narratives that bounce around the echo chamber of testing conferences, vendors and blogs have to stop.

Well, I am sad to say, that if the state of proposals I have reviewed recently is any sign of the state of our testing union, we are a long ways away from that change. What follows are statements, claims, and general nonsense from a series of test practices (I will not name names to protect the guilty) from firms who represent…wait for it…over 140,000 software testers in the world! In no particular order, here they are things they said about my organisation (without any knowledge of how it runs):

  • Reduction of testing cycles by 15% year on year
  • Committed productivity gains of 23-39%
  • 30% to 40% reduction in TCO
  • Increased testing productivity by 22%
  • 15% decrease in production defects
  • 27% increase in time to market
  • Attain Level 4 benchmark in TMMi in 2 years
  • Quality & Productivity improvement of 20-25%
  • 22% increase in cost savings
  • 20% cost efficiency improvement through standardization


But here are the particularly egregious claims that enter into the realm of pure, unadulterated “WTF??”, including an appearance by my mythical “orthogonal defect predict-o-nator”!

  • Improvement in the defect removal efficiency from 93% to over 97%
  • 20% cost efficiency improvement through standardization
  • Zero Defects in production (Severity 1 & Severity 2 )
  • 90% detection of valid defects in SIT and UAT
  • Decreased testing costs by ~50%
  • Increases to 50% average automation coverage as an industry benchmark
  • Defect prediction model that would predict defects in different stages of testing based on historical data…

This is all good fun, and dismissing them as skilled testers is easy and reminds me of the quote by Neil deGrasse Tyson,  who said, “The good thing about science is that it’s true whether or not you believe in it.” Unfortunately, these companies represent a large part of the global testing population and have collectively a very loud and prominent voice in how people perceive our industry. As well, this doesn’t even get to really nasty stuff in how they under represent what we do as testers and cheapen our value.

Now, I know these proposals are a lot of marketing material that anyone with even a rudimentary set of critical thinking skills could dismantle. But consider this, before my talk in Sweden, I took a walk around the vendor expo before anyone was there, and wrote down a few vendor claims that I shared with the conference during my keynote. After the laughs died down, all in all, there was a fair amount of similarity between those claims and the proposals I’ve seen recently.

As testers, we should be validating what people say their products are claiming, and as an industry, we can (and should) do better to accept responsibility for the narrative and perception of our value. Clearly, others are making that story on our behalf.