The Great Liberation: Software Quality Management  in the Age of AI

“Do bear in mind that the cynics have a point, of a sort, when they speak of the ‘professional naysayer’.” “To be in opposition is not to be a nihilist. And there is no decent or charted way of making a living at it. It is something you are, and not something you do.” – Christopher Hitchens, Letters to a Young Contrarian, 2001

The pace at which we are incorporating “artificial intelligence” into software testing is unprecedented and presents multiple ethical and technology challenges for the testing industry. As “AI” firms disband their ethics teams and commercial adoption outpaces the regulatory environment, the importance of quality systems couldn’t be higher. At expo:QA this year, I will explore the difficulty of testing networks and algorithms we don’t fully understand, how to refocus your test approach to adapt to AI tools, and a path forward for the testing industry to stay relevant when we need good testing more than ever.

And here . . . we . . . go!

In another failed chapter in the never ending book of “encouraging good behaviour”, the G7 have apparently agreed to a “code of conduct” for companies building #artificialintelligence systems. Per Reuters, “the voluntary code of conduct will set a landmark for how major countries govern AI, amid privacy concerns and security risks…”, now colour me cynical, but I think we’ve seen how this movie has played out before. You don’t need to expend any energy finding the billions being invested for new generative AI and other AI systems, which is only piled on the billions ALREADY spent on systems actively in use.

Continue reading

Head, meet desk . . .

After a long career of reviewing the various outputs from #softwaretesting: strategies, plans, test cases, #testautomation, etc. I’ve never understood why people think conformance to internal/external standards will affect a better outcome. I realise it’s born out of a fundamental misunderstanding of what happens when you test something paired with strong wish thinking that software testing is analogous to manufacturing. But IME, content is key for #testing artefacts and frankly, I’ve been around long enough to see how all these standards usually give a false sense of security and year after year add to the “#softwarequalitymanagement” certification grift.

So I wasn’t that surprised to witness another leg of the race to the #artificialintelligence bottom in testing with talk of creating #AI to check your test strategies, etc for deviations from your internal or external standards. Apparently, all this investment in #ML and #AI is going to be used as some really, really, expensive rube goldberg machine to #automate the lowest value work in testing!

Some day we’ll get something useful for testing from artificial intelligence, but today is not that day…

(old man shouts at cloud rant over 😉)

Inside every cynical person, there is a disappointed idealist. . .

“Inside every cynical person, there is a disappointed idealist.” – George Carlin

Had a great time talking about business risk with the Ministry of Testing folks and how your approach to testing is probably introducing risk to your business. Check out the whole interview or clips on my YouTube channel


#qualityengineering #softwaretesting #riskmanagement #testautomation #testmanagement #softwarequalityassurance

Parts is parts . . .

“Quality is value to some person”. – Jerry Weinberg

Saw this quote misattributed recently, but Jerry Weinberg threw down the gauntlet in his classic book Quality Software Management: Volume 1, Systems Thinking, and the software testing industry has been wrestling with every word in that short sentence ever since. What is “quality”? How do you determine “value”? Who is that “person”?

Continue reading

MoT AMA – Business Risk

Come register to join me here at the Ministry of Testing “Ask Me Anything” on July 20th at 12:30 GMT to talk about testing and business risk. Hope to see you there!


In this hour-long AMA webinar, Keith will answer your questions about business risk management in the context of software testing. Whether you want to know:

  • How to effectively communicate the importance of risk management to stakeholders
  • Common blind spots or overlooked areas of business risk that software testers need to be aware of
  • Emerging trends or future challenges in managing business risk

Here are some reasons why you should register for this webinar:

  • Learn from a leading expert in the field of business risk management
  • Gain insights into how to identify and assess business risks
  • Develop and implement risk management strategies
  • Measure and improve the effectiveness of risk management
  • Network with other software testers and learn from their experiences

A rising tide grifts all bots. . .

“When the tide goes out, you see who has been swimming naked.” – Warren Buffet

I’ve been reviewing a lot of material for a larger piece I’m writing on the use of #chatgpt and various other “#artificialintelligence” tools in #softwaretesting and I have to say one thing as a preview – the state of what the #testing community views as testing is amazingly poor.

Continue reading

“Manual” testing . . .

I worked with a client recently who was frustrated by all the “manual” interventions required to run their “100% fully automated” test suite. Irony aside, as an industry we really need to rethink the amount of sales nonsense we let dominate the public conversations around the value proposition of automation in testing. Terminology in our business is usually polarised and emotionally charged, but I’ve always believed that semantical arguments are worth having even if we can’t agree what to call stuff. Examples like this are just a symptom of a greater problem we have of letting vendors and <shudder> consultants sell us “fully-100%-automated-defect-predictonators” (said in your best Dr. Doofenshmirtz voice) for the last couple decades. We can do better and as well, when these ideas get into the business case, we are just undermining ourselves anyway…

The importance of good resources . . .

I’m doing some work on “documenting” my approach to reviewing software testing operations and I keep coming back to a couple resources. I forget sometimes how much Griffin Jones talk on “What is Good Evidence” and amongst several of his works, James Christies post “Not “right”, but as good as I can do” have influenced my work. They are both brilliant thinkers and contributed a lot to the testing profession…enjoy!

Happy Holidays and 2022 Testers!

Just wanted to send the quality engineering and software testing communities a short holiday note of appreciation for all the work you do.

Testing software is hard. Very hard. And not unlike plumbers, your effort is frequently not appreciated when things are working and first to be criticized when things go wrong.

Years ago I wrote a post trying to define why I like the business of software testing and specifically working with testers and why our work is so difficult.

“Testers spend their days trying to figure out what “might” go wrong by looking for ways a product is already broken – staring into the cosmic abyss of the impossibility of complete testing for all of us takes it toll. All the while competing in an industry teeming with unenlightened vendors, consultants and “experts” undermining their own value proposition by selling “bug free” methodologies, certified super-tester training programs and “automated algorithmic defect predictonators”.”

It doesn’t have to be like this, but it is, and there are lots of us fighting every day to make life better for testers while you make our systems safer, more reliable, and more equitable for the people who use them.

So from me, to you, thank you for all your hard work and know that I see it and appreciate it, even if a lot the world always doesn’t.

Wishing you a happy holiday season and great New Year.