Wait…what? – Tales from the Testing Dark Side

Always do sober what you said you’d do drunk. That will teach you to keep your mouth shut. – Ernest Hemingway

I recently gave a talk at the EuroSTAR conference in Gothenburg, Sweden about how I feel you can re-frame the perception of your testing effort in your organisation. A big part of the philosophy underpinning my approach is to be honest, frank and up front about what is, and is not working with yourself first – then address anything that comes after that.

Part of my talk is about how bias and perceptions are formed, and I take several (hopefully humorous)  pokes at the software testing industry to illustrate my point. I feel strongly, that we accept far too much nonsense and unverified claims about software testing, and in order for there to be fundamental change, those narratives that bounce around the echo chamber of testing conferences, vendors and blogs have to stop.

Well, I am sad to say, that if the state of proposals I have reviewed recently is any sign of the state of our testing union, we are a long ways away from that change. What follows are statements, claims, and general nonsense from a series of test practices (I will not name names to protect the guilty) from firms who represent…wait for it…over 140,000 software testers in the world! In no particular order, here they are things they said about my organisation (without any knowledge of how it runs):

  • Reduction of testing cycles by 15% year on year
  • Committed productivity gains of 23-39%
  • 30% to 40% reduction in TCO
  • Increased testing productivity by 22%
  • 15% decrease in production defects
  • 27% increase in time to market
  • Attain Level 4 benchmark in TMMi in 2 years
  • Quality & Productivity improvement of 20-25%
  • 22% increase in cost savings
  • 20% cost efficiency improvement through standardization


But here are the particularly egregious claims that enter into the realm of pure, unadulterated “WTF??”, including an appearance by my mythical “orthogonal defect predict-o-nator”!

  • Improvement in the defect removal efficiency from 93% to over 97%
  • 20% cost efficiency improvement through standardization
  • Zero Defects in production (Severity 1 & Severity 2 )
  • 90% detection of valid defects in SIT and UAT
  • Decreased testing costs by ~50%
  • Increases to 50% average automation coverage as an industry benchmark
  • Defect prediction model that would predict defects in different stages of testing based on historical data…

This is all good fun, and dismissing them as skilled testers is easy and reminds me of the quote by Neil deGrasse Tyson,  who said, “The good thing about science is that it’s true whether or not you believe in it.” Unfortunately, these companies represent a large part of the global testing population and have collectively a very loud and prominent voice in how people perceive our industry. As well, this doesn’t even get to really nasty stuff in how they under represent what we do as testers and cheapen our value.

Now, I know these proposals are a lot of marketing material that anyone with even a rudimentary set of critical thinking skills could dismantle. But consider this, before my talk in Sweden, I took a walk around the vendor expo before anyone was there, and wrote down a few vendor claims that I shared with the conference during my keynote. After the laughs died down, all in all, there was a fair amount of similarity between those claims and the proposals I’ve seen recently.

As testers, we should be validating what people say their products are claiming, and as an industry, we can (and should) do better to accept responsibility for the narrative and perception of our value. Clearly, others are making that story on our behalf.

5 thoughts on “Wait…what? – Tales from the Testing Dark Side

  1. Clearly these firms have given you false metrics… they didn’t go to two decimal places.

    I wept, laughed, then wept again.

    Instead of WTF, I’m angling towards FFS!

  2. Thanks for sharing Keith.
    This is the step one for a new tester stepping into testing.

    To Test, claims made about testing(process,definitions,methodologies,tools,teams involved) itself.

    If Test1 is
    Print (Sanity check is completed on claims)
    Proceed with claims testing

  3. One of the big reasons that numerical claims abound is that many buyers ask for them to justify to their management. There are so many ways surveys go wrong that to do one correctly is very costly. For one, the sample size has to be adequate. I think this is why people who make claims don’t document the sample size, since it’s usually one (or less). Projects I’ve seen never get funding to begin with a scientific baseline and the discipline needed to use valid statistics. Even if it’s real, benefits can usually not be duplicated elsewhere because it’s the specific group of people under the specific set of circumstances is what made it happen. Even though it’s a whole different set of people and technologies, some vendors nevertheless claim the ability to replicate the original group’s success because they have documentation (which are rarely really followed).

  4. Pingback: Testing Bits – 12/8/13 – 12/14/13 | Testing Curator Blog

Leave a Reply

Your email address will not be published. Required fields are marked *